vmware conflict in stable / test patch + 3/4.X.X kernels

[nixor]

Hi Guys !

Hope you got a spare minute or so to advise on this mystery so far:

- OS: Debian 8

Problem: vmware workstation 11.1 rebooting host instantly the moment any guest is started

Tried: (custom and automatic config of grsec, recompiled vmware modules with each kernel)

- kernel 3.14.40 with grsec patch = reboot
- kernel 3.14.43 with grsec patch = reboot
- kernel 3.2.69 with grsec patch = reboot
- kernel 4.01 with grsec test patch = reboot
- stock kernel 3.16.X without grsec patch = works

-> After the above, i disabled grsecurity and recompiled the kernels from the above list but still result = reboot

So at this point i thought what makes the stock kernel 3.16.x work and all the others don't ? The only thing
i came up with is that despite grsecurity was off the patch was applied to a load of files. So what i did is unpack the
3.14.43 vanilla kernel again but this time did not apply the grsec patch, guess what ? vmware workstation worked ! No reboot occured
when starting any guest !

Now here is where i'm stuck, it seems that the grsec patch when applied and even if it is turned "off" for some reason is causing
the reboot issue.

Kindly advise what this means or what is left to check ? As i really want the grsec patch and not just a vanilla kernel.

Thank you guys !

[PaX Team]

you could try to use the PaX patch alone and/or boot with nopcid to see if that makes a difference. other than that, it'd take some non-trivial debugging to find out the root cause, i'm afraid i won't have time for this anytime soon.

[nixor]

you could try to use the PaX patch alone and/or boot with nopcid to see if that makes a difference. other than that, it'd take some non-trivial debugging to find out the root cause, i'm afraid i won't have time for this anytime soon.

So far i tried nopcid and pax_nouderef but didn't work out, starting any guest in vmware causes an instant reboot. Didn't find anything in the logs as it reboots instantly.
Not sure what could cause this kind of instant reboot by simply trying to start a guest.

If you could make a rough guess without debugging, is there anything i can try to disable / undo done by the patch ? (while grsec / pax are still "off" in the config)

Or is there any way i could gather some data before the instant reboot ? Right now i can't find anything in the logs.

[spender]

Add a serial port to the VM, have it log to a file. Make sure "quiet" is removed from the kernel commandline and add console=ttyS0. Did you try the PaX patch only, from https://grsecurity.net/~paxguy1/ ?

-Brad

[nixor]

Add a serial port to the VM, have it log to a file. Make sure "quiet" is removed from the kernel commandline and add console=ttyS0. Did you try the PaX patch only, from https://grsecurity.net/~paxguy1/ ?

-Brad

Thank you, i'm going to try the pax patch alone and will update here how it went. Btw do you mean removing the quiet from the host kernel commandline and adding console=ttyS0 ? Or the guest ? Normally when starting a guest i see the vmware bios screen flash by but the system reboots so fast that the vmware bios screen doesn't even show up.

[nixor]

OK, results:

- serialport to the VM logging to a file = no data, file doesn't even get created
- applied pax patch alone to vanilla 3.14.43 = same behaviour, instant reboot of host when a guest starts
- no info found in kern.log and other showing what happens

NOTE: pax wasn't even enabled in the config, only applied the patch

I'm guessing that something in the pax patch modifies another "something" causing these instant reboots when a guest is started

[PaX Team]

PaX has several non-configurable changes, some of them affect low-level CPU specific things (my guess would be the changes to the per-cpu GDT) which is what probably makes vmware trigger a triple fault. debugging this would require setting up nested virtualization and catching the first guest's kernel (where the vmware 'host' code runs) as it's about to trigger the triple fault.

[nixor]

PaX has several non-configurable changes, some of them affect low-level CPU specific things (my guess would be the changes to the per-cpu GDT) which is what probably makes vmware trigger a triple fault. debugging this would require setting up nested virtualization and catching the first guest's kernel (where the vmware 'host' code runs) as it's about to trigger the triple fault.

OK, would this work, all 64 bits (never tried this kind of setup btw) :

- Host debian 8 with vmware workstation 11.1 (stock kernel, no grsec/pax patch)
-> install as outer guest KVM hypervisor / (enable option virtualize intel VT-x/EPT or AMD-V/RV for this guest) + grsec/pax kernel
-> install inner guest with debian 8 stock kernel

Not sure if the above is the right setup but appreciate any advise before i go set it up, thank you !

[PaX Team]

the outer guest must run vmware, not kvm as that's what we want to debug after all . i'm not sure that vmware inside vmware works on linux, you may also try vmware inside kvm (the only important bit is that the outer guest should run vmware and a grsec kernel).

[hlein]

I'm having what appears to be the same issue with 4.4.x and 4.5.x kernels. A host running Gentoo's hardened-sources, or vanilla + PAX patches, and vmware workstation: start a VM and the host system reboots.

I filed details at https://bugs.gentoo.org/show_bug.cgi?id=592614 , and found this thread while putting that together.

I'm game to try the vmception or other experiments.