VirtualBox and CONFIG_PAX_RAP

[Ghowe]

Hello.
Was trying to run VirtualBox on Grsecurity hardened kernel (latest patch, 4.7.5 kernel).

Had some troubles with it. VirtualBox was freezing computer completely (with VIRT_HOST=y, VIRT_EPT=y, VIRT_VIRTUALBOX=y).
Eventually I found out, that VirtualBox freezes computer if CONFIG_PAX_RAP option is enabled.

Can you please tell me whether it is a bug or a feature?

[PaX Team]

RAP among others also catches bugs that violate C function pointer related rules, such as type mismatches between a function pointer and target functions. i've fixed up in-tree code for RAP but for out-of-tree code someone else will have to do it.

[Ghowe]

You mean, bugreport about it should be submitted to VirtualBox team?

And one more thing.
Help for CONFIG_PAX_RAP says:
"If you have an amd64 processor that does not support SMEP then you must also enable a KERNEXEC code pointer instrumentation method (see PAX_KERNEXEC_PLUGIN)."
I read online, that PAX_KERNEXEC=y also makes VirtualBox to freeze computer.

Does it mean, that on amd64 CPU with no SMEP (I don't see it among flags) I won't be able to use both RAP and VirtualBox?

[PaX Team]

You mean, bugreport about it should be submitted to VirtualBox team?

yes, but i'm not sure they will care about it since their code 'works', it's just RAP that happens to enforce a C rule while also doing its job.

Does it mean, that on amd64 CPU with no SMEP (I don't see it among flags) I won't be able to use both RAP and VirtualBox?

the KERNEXEC gcc plugin can be enabled without enabling the rest of KERNEXEC.