"PAX: overwritten function pointer or return address detected" with 4.8.8

[jotik]

This is from Gentoo's sys-kernel/hardened-sources-4.8.8 (grsecurity-3.1-4.8.8-201611150756.patch I think):

Code: Select all

[    2.203772] FS:  0000000000000000(0000) GS:ffff88017fc80000(0000) knlGS:0000000000000000
[    2.203773] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.203775] CR2: 0000033bc77a6630 CR3: 0000000001724000 CR4: 00000000001606b0
[    2.203780] Stack:
[    2.203785]  0000000000000010 ffffc9000000bc58 ffff88017b1bea00 ffffffff81c91d40
[    2.203789]  ffffc9000000bc40 ffffffff8137449e 0000000000000002 0000000000000000
[    2.203792]  0000000000000000 0000000000000000 0232b2ea1b16f224 0000000000000010
[    2.203793] Call Trace:
[    2.203800]  <IRQ>
[    2.203800]  [<ffffffff8137449e>] put_chars+0x4e/0xc0
[    2.203805]  [<ffffffff8136ed4a>] hvc_console_print+0xda/0x130
[    2.203811]  [<ffffffff81096617>] call_console_drivers.isra.15.constprop.27+0xa7/0xc0
[    2.203816]  [<ffffffff81097d7b>] console_unlock+0x26b/0x5a0
[    2.203820]  [<ffffffff810983ae>] vprintk_emit+0x2fe/0x500
[    2.203874]  [<ffffffff810987a3>] vprintk_default+0x23/0x40
[    2.203881]  [<ffffffff810f1cb4>] printk+0x68/0x91
[    2.203886]  [<ffffffff813708df>] crng_fast_load+0x10f/0x130
[    2.203890]  [<ffffffff8137282f>] add_interrupt_randomness+0x1ef/0x230
[    2.203894]  [<ffffffff8109a5be>] handle_irq_event_percpu+0x3e/0x90
[    2.203896]  [<ffffffff8109a654>] handle_irq_event+0x44/0x90
[    2.203901]  [<ffffffff8109e82f>] handle_edge_irq+0xef/0x200
[    2.203905]  [<ffffffff8101375e>] handle_irq+0x7e/0x160
[    2.203910]  [<ffffffff8101a1d9>] ? __exit_idle+0x29/0x40
[    2.203914]  [<ffffffff810133a8>] do_IRQ+0x48/0xf0
[    2.203919]  [<ffffffff8155968e>] common_interrupt+0x8e/0x8e
[    2.203925]  <EOI>
[    2.203926]  [<ffffffff81019f90>] ? arch_remove_reservations+0x110/0x110
[    2.203930]  [<ffffffff81031b36>] ? native_safe_halt+0x6/0x20
[    2.203934]  [<ffffffff81019f99>] default_idle+0x9/0x20
[    2.203937]  [<ffffffff8101a727>] arch_cpu_idle+0x17/0x30
[    2.203940]  [<ffffffff81090e0e>] default_idle_call+0x1e/0x40
[    2.203943]  [<ffffffff81090fcc>] cpu_startup_entry+0x19c/0x260
[    2.203949]  [<ffffffff81026884>] start_secondary+0x1d4/0x210
[    2.203997] Code: e0 06 48 01 d0 49 8b 16 83 e2 03 48 09 d0 49 89 06 41 5e 5d c3 4c 89 e8 48 2b 82 68 02 00 00 48 03 82 10 08 00 00 49 89 c5 eb 85 <0f> 0b 0f 1f 00 cc cc cc cc cc cc 48 b8 81 7e 40 25 00 00 00 00
[    2.204002] RIP  [<ffffffff812e569b>] sg_init_one+0xbb/0xd0
[    2.204003]  RSP <ffffc9000000bbd0>
[    2.204005] ---[ end trace 2724d1860b0ece5b ]---
[    2.204007] Kernel panic - not syncing: Fatal exception in interrupt
[    2.204339] Kernel Offset: disabled
[    2.204356] ------------[ cut here ]------------
[    2.204357] kernel BUG at /usr/src/linux-4.8.8-hardened/include/linux/scatterlist.h:150!
[    2.204360] PAX: overwritten function pointer or return address detected: 0000 [#2] SMP
[    2.204363] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D         4.8.8-hardened #1
[    2.204366] task: ffff88017b09bf00 task.stack: ffffc90000080000
[    2.204371] RIP: 0010:[<ffffffff812e569b>]  [<ffffffff812e569b>] sg_init_one+0xbb/0xd0
[    2.204373] RSP: 0018:ffffc9000000b798  EFLAGS: 00010046
[    2.204374] RAX: 0000000000000000 RBX: 0000000000000010 RCX: 0000000000000028
[    2.204376] RDX: 0000000000000041 RSI: 0000000000000000 RDI: 000041000000b820
[    2.204377] RBP: ffffc9000000b7b8 R08: 0000000000000030 R09: ffffc9000000b7c8
[    2.204379] R10: 0000000000ffff0a R11: 0000000000000166 R12: 0000000000000820
[    2.204380] R13: ffffc9000000b820 R14: ffffc9000000b7c8 R15: 0000000000000000
[    2.204382] FS:  0000000000000000(0000) GS:ffff88017fc80000(0000) knlGS:0000000000000000
[    2.204384] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.204385] CR2: 0000033bc77a6630 CR3: 0000000001724000 CR4: 00000000001606b0
[    2.204389] Stack:
[    2.204393]  0000000000000010 ffffc9000000b820 ffff88017b1bea00 ffffffff81c91d40
[    2.204396]  ffffc9000000b808 ffffffff8137449e 0000000000000002 0000000000000000
[    2.204400]  0000000000000000 0000000000000000 0232b2ea1b16f224 0000000000000010
[    2.204400] Call Trace:
[    2.204405]  <IRQ>
[    2.204405]  [<ffffffff8137449e>] put_chars+0x4e/0xc0
[    2.204409]  [<ffffffff8136ed4a>] hvc_console_print+0xda/0x130
[    2.204414]  [<ffffffff81096617>] call_console_drivers.isra.15.constprop.27+0xa7/0xc0
[    2.204418]  [<ffffffff81097d7b>] console_unlock+0x26b/0x5a0
[    2.204422]  [<ffffffff810988a8>] console_flush_on_panic+0x18/0x30
[    2.204426]  [<ffffffff810f1aba>] panic+0x13a/0x253
[    2.204429]  [<ffffffff810149d3>] oops_end+0xd3/0xf0
[    2.204432]  [<ffffffff81014b26>] die+0x46/0x70
[    2.204435]  [<ffffffff81011d42>] do_trap+0xc2/0x180
[    2.204437]  [<ffffffff81011e9a>] do_error_trap+0x9a/0x130
[    2.204441]  [<ffffffff812e569b>] ? sg_init_one+0xbb/0xd0
[    2.204444]  [<ffffffff8101221a>] do_invalid_op+0x2a/0x40
[    2.204448]  [<ffffffff81559f5e>] invalid_op+0x1e/0x30
[    2.204451]  [<ffffffff812e569b>] ? sg_init_one+0xbb/0xd0
[    2.204454]  [<ffffffff8137449e>] put_chars+0x4e/0xc0
[    2.204457]  [<ffffffff8136ed4a>] hvc_console_print+0xda/0x130
[    2.204462]  [<ffffffff81096617>] call_console_drivers.isra.15.constprop.27+0xa7/0xc0
[    2.204466]  [<ffffffff81097d7b>] console_unlock+0x26b/0x5a0
[    2.204470]  [<ffffffff810983ae>] vprintk_emit+0x2fe/0x500
[    2.204474]  [<ffffffff810987a3>] vprintk_default+0x23/0x40
[    2.204478]  [<ffffffff810f1cb4>] printk+0x68/0x91
[    2.204481]  [<ffffffff813708df>] crng_fast_load+0x10f/0x130
[    2.204484]  [<ffffffff8137282f>] add_interrupt_randomness+0x1ef/0x230
[    2.204486]  [<ffffffff8109a5be>] handle_irq_event_percpu+0x3e/0x90
[    2.204488]  [<ffffffff8109a654>] handle_irq_event+0x44/0x90
[    2.204490]  [<ffffffff8109e82f>] handle_edge_irq+0xef/0x200
[    2.204492]  [<ffffffff8101375e>] handle_irq+0x7e/0x160
[    2.204495]  [<ffffffff8101a1d9>] ? __exit_idle+0x29/0x40
[    2.204497]  [<ffffffff810133a8>] do_IRQ+0x48/0xf0
[    2.204500]  [<ffffffff8155968e>] common_interrupt+0x8e/0x8e
[    2.204504]  <EOI>
[    2.204504]  [<ffffffff81019f90>] ? arch_remove_reservations+0x110/0x110
[    2.204506]  [<ffffffff81031b36>] ? native_safe_halt+0x6/0x20
[    2.204509]  [<ffffffff81019f99>] default_idle+0x9/0x20
[    2.204511]  [<ffffffff8101a727>] arch_cpu_idle+0x17/0x30
[    2.204513]  [<ffffffff81090e0e>] default_idle_call+0x1e/0x40
[    2.204515]  [<ffffffff81090fcc>] cpu_startup_entry+0x19c/0x260
[    2.204518]  [<ffffffff81026884>] start_secondary+0x1d4/0x210
[    2.204544] Code: e0 06 48 01 d0 49 8b 16 83 e2 03 48 09 d0 49 89 06 41 5e 5d c3 4c 89 e8 48 2b 82 68 02 00 00 48 03 82 10 08 00 00 49 89 c5 eb 85 <0f> 0b 0f 1f 00 cc cc cc cc cc cc 48 b8 81 7e 40 25 00 00 00 00
[    2.204547] RIP  [<ffffffff812e569b>] sg_init_one+0xbb/0xd0
[    2.204548]  RSP <ffffc9000000b798>
[    2.204549] ---[ end trace 2724d1860b0ece5c ]---
[    2.204550] Kernel panic - not syncing: Fatal exception in interrupt

Full trace was 1800+ lines before rebooting: http://sprunge.us/ecgZ

I don't know whether this is grsecurity related or not. Please advise further action. FWIW this is fully reproducible and I can add any printouts to the source if this helps. Thanks!

[PaX Team]

can you resolve ffffffff812e569b with addr2line? if it resolves into the BUG_ON in sg_set_buf then we'll need the address of 'buf' too (it's probably in one of the registers but the dump you posted is cut short at the front so we can't see it).

[spender]

Can you apply the patch at:

https://grsecurity.net/~spender/hvcfix.diff

and let me know if it fixes the issue?

Thanks,
-Brad

[jotik]

can you resolve ffffffff812e569b with addr2line?

It just resolved to sg_init_one ??:?

the dump you posted is cut short at the front

It's cut short on the hvc0 terminal, so that's really all I've got:

Code: Select all

[    1.039231] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
[    1.040413] Freeing unused kernel memory: 2536K (ffffffff81a00000 - ffffffff81c7a000)
[    1.040904] Write protecting the kernel read-only data: 8192k
INIT: version 2.88 booting

   OpenRC 0.22.4 is starting up Gentoo Linux (x86_64)

Press I to enter interactive boot mode

 * Mounting /proc ...
[    1.575501] grsec: mount of proc to /proc by /bin/mount[mount:102] uid/euid:0/0 gid/egid:0/0, parent /lib64/rc/sh/init.sh[init.sh:96] uid/euid:0/0 gid/egid:0/0
 [ ok ]
 * Mounting /run ...
[    1.613415] grsec: mount of tmpfs to /run by /bin/mount[mount:108] uid/euid:0/0 gid/egid:0/0, parent /lib64/rc/sh/init.sh[init.sh:96] uid/euid:0/0 gid/egid:0/0
 * /run/openrc: creating directory
 * /run/lock: creating directory
 * /run/lock: correcting owner
 * Caching service dependencies ...
[    1.900707] FS:  0000000000000000(0000) GS:ffff88017fc80000(0000) knlGS:0000000000000000
[    1.900709] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.900710] CR2: 000003bbb0c5dd70 CR3: 0000000001724000 CR4: 00000000001606b0


if it resolves into the BUG_ON in sg_set_buf then we'll need the address of 'buf' too

I added this:

Code: Select all

if (!virt_addr_valid(realbuf))
    printk(KERN_ERR "PAX buf,realbuf: %p,%p\n", buf, realbuf);


before the BUG_ON, and it prints stuff like:

Code: Select all

[    2.558632] PAX buf,realbuf: ffffc9000000b810,ffffc9000000b810
[    2.558948] PAX buf,realbuf: ffffc9000000b3d0,ffffc9000000b3d0
[    2.559288] PAX buf,realbuf: ffffc9000000af90,ffffc9000000af90
[    2.559678] PAX buf,realbuf: ffffc9000000ab50,ffffc9000000ab50
[    2.560085] PAX buf,realbuf: ffffc9000000a710,ffffc9000000a710
[    2.560535] PAX buf,realbuf: ffffc9000000a2d0,ffffc9000000a2d0
[    2.561014] PAX buf,realbuf: ffffc90000009e90,ffffc90000009e90
[    2.561536] PAX buf,realbuf: ffffc90000009a50,ffffc90000009a50
[    2.562088] PAX buf,realbuf: ffffc90000009610,ffffc90000009610
[    2.562683] PAX buf,realbuf: ffffc900000091d0,ffffc900000091d0
[    2.563305] PAX buf,realbuf: ffffc90000008d90,ffffc90000008d90
[    2.563972] PAX buf,realbuf: ffffc90000008950,ffffc90000008950


As for the patch, I get this while compiling:

Code: Select all

  CC      drivers/tty/hvc/hvc_console.o
/usr/src/linux-4.8.8-hardened/drivers/tty/hvc/hvc_console.c: In function 'hvc_console_print':
/usr/src/linux-4.8.8-hardened/drivers/tty/hvc/hvc_console.c:163:5: warning: assignment from incompatible pointer type
   c = &c_stack;
     ^


Other than that, it does make the crash disappear. Thanks!