linux-grsec-4.7.7 locks up within 30 minutes

[fatalhalt]

Hello,

I'm experiencing a complete system lockup in most recent version of linux-grsec-4.7.7.r201610101902-1, it happened repeatably after about 30 minutes. I have rolled back to 4.7.6.201609301918 and things are back to normal. There's nothing in systemd journal leading up to the lockup, I'm running Archlinux x86_64, Intel Core i3-2120 Sandybridge CPU, EVGA P67 chipset motherboard, desktop DDR3 RAM, and Radeon HD 5430 GPU.

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation P67 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
02:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
02:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
04:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II / PATA Controller (rev b2)
05:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller (rev 10)
06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
07:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)

[spender]

Can you try booting 4.7.7 with pax_size_overflow_report_only on the kernel commandline?

-Brad

[foxxx0]

I'm having more or less the same issue.

Hardware:
Lenovo Thinkpad X240
Core i5 4200U
8GB RAM
integrated graphics

OS: Archlinux x64

Both 4.7.7.r201610101902 and 4.7.8.r201610161720 are broken on that machine. I have no other 4.7.x kernels available at the moment, maybe I can dig some up in my backups.

Booting with pax_size_overflow_report_only does not help, the whole system just freezes completely. There is nothing in the systemd journal whatsoever and attaching a serial console doesn't help either, the system just HALTs.

Somehow I suspect some interference with the Intel i915 GPU drivers, that have a tendency to break things and in case of failures killing the whole system with it. If I remember correctly a 4.7.6-grsec kernel had the same problem whereas a "vanilla" 4.7.6 kernel from the arch repositories works without problems.

[foxxx0]

Alright, so:

4.6.4.r201607112205 is working just fine

4.7.0.r201608131240 is broken

So the issue (at least for me) was introduced in 4.7.0.
Unfortunately I have no later 4.6.x kernels that I could test, so >4.6.4.r201607112205 and <= 4.7.0.r201608131240 is the range where the issue was introduced.

If I should test something else please let me know.

[KDE]

There is new option CONFIG_PAX_SIZE_OVERFLOW_EXTRA in 4.7.7

[fatalhalt]

I'm on 4.7.8.201610161720-1-grsec now and have appended 'pax_size_overflow_report_only ' to the kernel boot line and system has been stable so far.

Previously I only had 'pax_nouderef' option, now I'm using 'pax_nouderef pax_size_overflow_report_only'

[PaX Team]

I'm on 4.7.8.201610161720-1-grsec now and have appended 'pax_size_overflow_report_only ' to the kernel boot line and system has been stable so far.

can you post your dmesg? it'll have a few size overflow reports probably.

[fatalhalt]

]can you post your dmesg? it'll have a few size overflow reports probably.

Sure can: http://pastebin.com/6UJfTqsB

[PaX Team]

thanks but i'll need dmesg after a size overflow report occurs, so you'll probably have to wait that half an hour you mentioned earlier. you can monitor dmesg for size overflow messages as they all start with "PAX: size overflow detected in function".

[PaX Team]

If I should test something else please let me know.

can you test a vanilla kernel as well? also what does 'broken' mean for you? do you see a kernel crash message at least that you could take a photo of perhaps? how far does the broken system boot? can you tell if it fails in qemu as well?

[foxxx0]

If I should test something else please let me know.

can you test a vanilla kernel as well? also what does 'broken' mean for you? do you see a kernel crash message at least that you could take a photo of perhaps? how far does the broken system boot? can you tell if it fails in qemu as well?

What version of vanilla kernel do you want me to test? I tried 4.7.6 and it runs without problems while 4.7.6-grsec does not work.

"broken" means, that the systems locks up pretty quickly. I am able to reproduce it somewhat consistently by running a grep through the whole filesystem. As soon as the "broken" kernels find something the system just freezes right after grep produced some output. On the working kernels the grep just runs and runs and runs and isn't causing any lock-ups at all (and without grep it runs for numerous hours with varying workload just fine).
But the issue is not limited to my grep command, it is actually pretty accurately described as "within 30 minutes after boot" as the initial post stated.

Unfortunately I am not able (or at least don't know how) to gather any debug information on that, as the whole system instantly freezes. No luck monitoring over SSH and even starting a serial console at boot and monitoring that does not allow me to retrieve any further information, the machine just stops all at once.

Regarding the qemu stuff I don't know what to test because all different 4.7.x grsec kernels that are "broken" on my lenovo x240 are running just fine on other machines. That is precisely why I suspect some interference with the intel i915 gpu drivers in the 4.7.x series when using the grsec patchset.

[PaX Team]

can you use the vga console driver instead of the intel driver to see if it matters? that said, i don't see how a video driver would be affected by filesystem operations. also can you post your kernel config?

[foxxx0]

With 4.7.6.r20160930191

Alright, running the same grep on tty1 without ever having started X leads to same freeze but this time there is at least a stack trace printed on the screen:

https://paste.foxxx0.de/emyG/

Kernel config:

https://paste.foxxx0.de/2IrxH/

[PaX Team]

thanks, this is actually a RAP violation where the wrong type is used for a sysfs callback (i've fixed a bunch of these already but unfortunately it's hard to find them all with static analysis). can you try the following patch and post the output?

Code: Select all

--- a/arch/x86/kernel/smp.c      2016-07-26 23:39:51.990637586 +0200
+++ b/arch/x86/kernel/smp.c       2016-10-21 22:15:47.104213913 +0200
@@ -122,7 +122,7 @@ static bool smp_no_nmi_ipi = false;
 static void native_smp_send_reschedule(int cpu)
 {
        if (unlikely(cpu_is_offline(cpu))) {
-               WARN_ON(1);
+//             WARN_ON(1);
                return;
        }
        apic->send_IPI(cpu, RESCHEDULE_VECTOR);
--- a/lib/kobject.c      2016-07-26 23:40:23.913737096 +0200
+++ b/lib/kobject.c       2016-10-21 16:57:53.946334968 +0200
@@ -779,8 +779,10 @@ static ssize_t kobj_attr_show(struct kob
        ssize_t ret = -EIO;

        kattr = container_of(attr, struct kobj_attribute, attr);
-       if (kattr->show)
+       if (kattr->show) {
+pr_err("PAX: show:%pS\n", kattr->show);
                ret = kattr->show(kobj, kattr, buf);
+}
        return ret;
 }



[spender]

For this testing you should also disable GRKERNSEC_KERN_LOCKOUT

-Brad