SOLVED: drbd again

[brainatwork]

Hi

Another round of drbd glitches. As soon as the drbd partner connects and starts to sync it goes BOOM...

I tried 4.7.2-hardened (gentoo) currently not -r1
Any chance this has been already fixed with grsecurity-3.1-4.7.2-201608312326 (in 4.7.2-hardened-r2)?

Thanks

---
krnlm@gentoo-krnlm-15652 ~/kernel/cfg/dom0-kvm-at $ grep ^CONFIG_PAX config-dom0-kvm-at-gw25
CONFIG_PAX_PER_CPU_PGD=y
CONFIG_PAX_USERCOPY_SLABS=y
CONFIG_PAX=y
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_XATTR_PAX_FLAGS=y
CONFIG_PAX_HAVE_ACL_FLAGS=y
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_EMUTRAMP=y
CONFIG_PAX_MPROTECT=y
CONFIG_PAX_KERNEXEC=y
CONFIG_PAX_KERNEXEC_PLUGIN=y
CONFIG_PAX_KERNEXEC_PLUGIN_METHOD_OR=y
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDKSTACK=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
CONFIG_PAX_MEMORY_UDEREF=y
CONFIG_PAX_REFCOUNT=y
CONFIG_PAX_CONSTIFY_PLUGIN=y
CONFIG_PAX_USERCOPY=y
CONFIG_PAX_LATENT_ENTROPY=y
CONFIG_PAX_RAP=y
krnlm@gentoo-krnlm-15652 ~/kernel/cfg/dom0-kvm-at $
---

kernel trace
---
[ 119.282675] kvm: zapping shadow pages for mmio generation wraparound¶
[ 121.958332] kvm: zapping shadow pages for mmio generation wraparound¶
[ 146.334649] PAX: please report this to [email protected]¶
[ 146.347463] BUG: unable to handle kernel NULL pointer dereference at 0000000000000003¶
[ 146.367148] IP: [<ffffffff9d48a2f6>] memcpy_erms+0x6/0x10¶
[ 146.379492] PGD b4202000·¶
[ 146.383502] Oops: 0000 [#1] SMP¶
[ 146.389026] CPU: 1 PID: 5432 Comm: drbd_w_erinome Tainted: G W 4.7.2-hardened-dom0-kvm-at-gw25 #10¶
[ 146.415379] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./J1900N-D3V, BIOS F1 01/22/2014¶
[ 146.441473] task: ffff88041f602300 ti: ffff88041f602978 task.ti: ffff88041f602978¶
[ 146.460016] RIP: 0010:[<ffffffff9d48a2f6>] [<ffffffff9d48a2f6>] memcpy_erms+0x6/0x10¶
[ 146.479651] RSP: 0000:ffffc90009a63ba8 EFLAGS: 00010286¶
[ 146.491682] RAX: ffff8804276d9bf0 RBX: ffffc90009a63cc8 RCX: 00000000000005f0¶
[ 146.509183] RDX: 00000000000005f0 RSI: 0000000000000003 RDI: ffff8804276d9bf0¶
[ 146.526681] RBP: 00000000000007f0 R08: 0000000000000a10 R09: ffff88042c2466c0¶
[ 146.544183] R10: ffff88042069a930 R11: 00000000f8b47059 R12: 8000000000000000¶
[ 146.561684] R13: ffffc90009a63cc8 R14: ffff8804276da1e0 R15: 00000000000005f0¶
[ 146.579185] FS: 0000000000000000(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000¶
[ 146.599548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033¶
[ 146.612881] CR2: 0000000000000003 CR3: 000000001dc4c000 CR4: 00000000001026f0¶
[ 146.630381] Stack:¶
[ 146.632516] ffffffff9d48ec92 0000000000000a10 0000000000000810 0000000000000000¶
[ 146.651006] ffff88041f602300 0000000000000a10 ffffc90009a63cb8 ffff880420a57200¶
[ 146.669498] ffff88042069a800 ffff88041f602600 ffffffff9d82c8d7 000005f001030098¶
[ 146.687985] Call Trace:¶
[ 146.691429] [<ffffffff9d48ec92>] ? copy_from_iter+0x13d/0x32c¶
[ 146.705022] [<ffffffff9d82c8d7>] ? tcp_sendmsg+0x5fa/0xa4f¶
[ 146.717835] [<ffffffff9d773c1c>] ? sock_sendmsg+0x38/0x52¶
[ 146.730386] [<ffffffff9d587bd3>] ? drbd_send+0xa5/0x17a¶
[ 146.742415] [<ffffffff9d587cb1>] ? drbd_send_all+0x9/0x2d¶
[ 146.754968] [<ffffffff9d587cb1>] ? drbd_send_all+0x9/0x2d¶
[ 146.767520] [<ffffffff9d589215>] ? _drbd_no_send_page+0x4e/0x6f¶
[ 146.781637] [<ffffffff9d589660>] ? drbd_send_dblock+0x286/0x4dd¶
[ 146.795752] [<ffffffff9d572632>] ? w_send_dblock+0xda/0x14d¶
[ 146.808825] [<ffffffff9d573760>] ? drbd_worker+0x16c/0x38a¶
[ 146.821636] [<ffffffff9d585cac>] ? drbd_thread_setup+0x55/0x164¶
[ 146.835753] [<ffffffff9d585c57>] ? drbd_rs_del_all+0x3fe/0x3fe¶
[ 146.849606] [<ffffffff9d12b068>] ? kthread+0xd5/0xe7¶
[ 146.860857] [<ffffffff9d8e066e>] ? ret_from_fork+0x1e/0x50¶
[ 146.873670] [<ffffffff9d12af93>] ? __kthread_parkme+0x73/0x73¶
[ 146.887260] Code: 90 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 4c 09 24 24 c3 66 90 48 89 f8 48 89 d1 <f3> a4 4c 09 24 24 c3 0f 1f 00 48 89 f8 48 83 fa 20 72 7e 40 38·¶
[ 146.943305] RIP [<ffffffff9d48a2f6>] memcpy_erms+0x6/0x10¶
[ 146.955909] RSP <ffffc90009a63ba8>¶
[ 146.962470] CR2: 0000000000000003¶
[ 146.968516] ---[ end trace b9b0d5d4b4fecdb0 ]---¶
---

[PaX Team]

this is a NULL ptr dereference, can you enable frame pointers and repost dmesg?

[brainatwork]

hi

I just compiled 4.7.4-hardened with frame pointer but i cannot reproduce the problem. Did you fix it lately?
Thanks

[PaX Team]

no, i didn't touch anything drbd related.

[brainatwork]

...and here it goes...
i hope that helps.

--8<--
[324502.412171] drbd himalia: [drbd_w_himalia/5489] sock_sendmsg time expired, ko = 6
[324503.735791] PAX: please report this to [email protected]
[324503.748607] BUG: unable to handle kernel NULL pointer dereference at 0000000000000003
[324503.768291] IP: [<ffffffffa459bde6>] memcpy_erms+0x6/0x10
[324503.780637] PGD 420943066
[324503.784906] Oops: 0000 [#1] SMP
[324503.790431] CPU: 0 PID: 5489 Comm: drbd_w_himalia Not tainted 4.7.4-hardened-dom0-kvm-at-gw26 #8
[324503.812875] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./J1900N-D3V, BIOS F1 01/22/2014
[324503.838971] task: ffff88042ecd0bc0 ti: ffff88042ecd1338 task.ti: ffff88042ecd1338
[324503.857511] RIP: 0010:[<ffffffffa459bde6>] [<ffffffffa459bde6>] memcpy_erms+0x6/0x10
[324503.877147] RSP: 0000:ffffc90000213ab8 EFLAGS: 00010282
[324503.889178] RAX: ffff8802069ba9d0 RBX: 0000000000000a10 RCX: 0000000000000318
[324503.906680] RDX: 0000000000000318 RSI: 0000000000000003 RDI: ffff8802069ba9d0
[324503.924179] RBP: ffffc90000213b10 R08: 000000000000fe88 R09: 00000000000005f0
[324503.941681] R10: 0000000000000000 R11: 00000000000005a8 R12: 8000000000000000
[324503.959180] R13: ffffc90000213c58 R14: 0000000000000318 R15: 0000000000000318
[324503.976681] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000
[324503.997045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[324504.010378] CR2: 0000000000000003 CR3: 0000000024e5e000 CR4: 00000000001026f0
[324504.027879] Stack:
[324504.030014] ffffffffa45a214f 0000000002400000 00000000000005f0 ffff880084e79100
[324504.048502] 0000000000000000 ffffc90000213c58 ffff8802069bace8 0000000000000a10
[324504.066993] ffff88011c810400 ffff88042ecd1148 ffff880084e79100 ffffc90000213bb0
[324504.085484] Call Trace:
[324504.088925] [<ffffffffa45a214f>] ? copy_from_iter+0x2ef/0x3f0
[324504.102518] [<ffffffffa4a32c72>] tcp_sendmsg+0x2b2/0xc20
[324504.114809] [<ffffffffa4a61151>] inet_sendmsg+0x71/0xc0
[324504.126842] [<ffffffffa4947aa7>] sock_sendmsg+0x47/0x70
[324504.138871] [<ffffffffa4947f8a>] kernel_sendmsg+0x2a/0x50
[324504.151426] [<ffffffffa46d9155>] drbd_send+0xd5/0x210
[324504.162935] [<ffffffffa46dada9>] _drbd_no_send_page.isra.47+0x59/0xa0
[324504.178613] [<ffffffffa46db25e>] drbd_send_dblock+0x27e/0x690
[324504.192206] [<ffffffffa46d04e2>] ? __req_mod+0x252/0xba0
[324504.204499] [<ffffffffa46be437>] w_send_dblock+0x97/0x1d0
[324504.217049] [<ffffffffa46bf935>] drbd_worker+0x115/0x430
[324504.229342] [<ffffffffa46d69d0>] ? drbd_rs_del_all+0x510/0x510
[324504.243197] [<ffffffffa46d6a29>] drbd_thread_setup+0x59/0x180
[324504.256791] [<ffffffffa46d69d0>] ? drbd_rs_del_all+0x510/0x510
[324504.270647] [<ffffffffa415ae02>] kthread+0xe2/0x110
[324504.281635] [<ffffffffa4b1322e>] ret_from_fork+0x1e/0x50
[324504.293927] [<ffffffffa415ad20>] ? __kthread_parkme+0x90/0x90
[324504.307520] Code: ff ff eb e5 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 4c 09 24 24 c3 66 90 48 89 f8 48 89 d1 <f3> a4 4c 09 24 24 c3 0f 1f 00 48 89 f8 48 83 fa 20 72 7e 40 38
[324504.363561] RIP [<ffffffffa459bde6>] memcpy_erms+0x6/0x10
[324504.376166] RSP <ffffc90000213ab8>
[324504.382728] CR2: 0000000000000003
[324504.388776] ---[ end trace 6cdaf9636f88a6d8 ]---

[PaX Team]

can you use addr2line to resolve ffffffffa4a32c72 to a source line?

[brainatwork]

Hi

addr2line -e vmlinux-4.7.4-hardened-dom0-kvm-at-gw26-dbg.x86_64 ffffffffa4a32c72
addr2line: vmlinux-4.7.4-hardened-dom0-kvm-at-gw26-dbg.x86_64: File format not recognized

Sorry most likely i'm doing something wrong here.
I think addr2line does not like the bzimage format? However this is just a wild guess and google didn't help me either.
Debugging kernels is new to me.

[PaX Team]

addr2line needs the vmlinux you can find in the build dir (it's the unstripped uncompressed version that everything else is built from later).

[brainatwork]

ok. i don't have that any more (build image using docker container). will rebuild, reoops and repost.
Be patient...sorry

[brainatwork]

Hi

Still don't get this working
---8<---
[ 3399.839724] PAX: please report this to [email protected]
[ 3399.852525] BUG: unable to handle kernel NULL pointer dereference at 0000000000000003
[ 3399.872208] IP: [<ffffffff964d8306>] memcpy_erms+0x6/0x10
[ 3399.884552] PGD 4215f5066
[ 3399.888823] Oops: 0000 [#1] SMP
[ 3399.894349] CPU: 1 PID: 5571 Comm: drbd_w_metis Tainted: G W 4.7.4-hardened-r1-dom0-kvm-at-gw27-dbg #13
[ 3399.921998] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./J1900N-D3V, BIOS F1 01/22/2014
[ 3399.948096] task: ffff88042eff4c40 ti: ffff88042eff51a8 task.ti: ffff88042eff51a8
[ 3399.966635] RIP: 0010:[<ffffffff964d8306>] [<ffffffff964d8306>] memcpy_erms+0x6/0x10
[ 3399.986272] RSP: 0000:ffffc9000050bad0 EFLAGS: 00010282
[ 3399.998304] RAX: ffff880354baf9d0 RBX: ffffc9000050bc70 RCX: 0000000000000050
[ 3400.015803] RDX: 0000000000000050 RSI: 0000000000000003 RDI: ffff880354baf9d0
[ 3400.033305] RBP: ffffc9000050bb20 R08: 0000000000000599 R09: ffffffff96ee8dc0
[ 3400.050804] R10: ffff88042eff52f0 R11: 0000000065250d28 R12: 8000000000000000
[ 3400.068305] R13: 0000000000000fb0 R14: ffffc9000050bc70 R15: 0000000000000050
[ 3400.085805] FS: 0000000000000000(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
[ 3400.106169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3400.119503] CR2: 0000000000000003 CR3: 0000000016c4a000 CR4: 00000000001026f0
[ 3400.137002] Stack:
[ 3400.139139] ffffffff964dd265 ffffffff968c33a9 0000000000001000 0000000000000050
[ 3400.157628] ffff880354bafa20 0000000000000a10 0000000000000a10 ffffc9000050bc60
[ 3400.176119] ffff880420266040 ffff88039a220e00 ffffc9000050bbc8 ffffffff968b7618
[ 3400.194608] Call Trace:
[ 3400.198053] [<ffffffff964dd265>] ? copy_from_iter+0x147/0x32d
[ 3400.211645] [<ffffffff968c33a9>] ? tcp_cwnd_restart+0x55/0xf2
[ 3400.225239] [<ffffffff968b7618>] tcp_sendmsg+0x617/0xa5f
[ 3400.237531] [<ffffffff968de445>] inet_sendmsg+0x6b/0xad
[ 3400.249561] [<ffffffff967f18a4>] sock_sendmsg+0x3d/0x59
[ 3400.261591] [<ffffffff967f1cfc>] kernel_sendmsg+0x26/0x37
[ 3400.274143] [<ffffffff965e97d6>] drbd_send+0xbc/0x1b3
[ 3400.285654] [<ffffffff965e98da>] drbd_send_all+0xd/0x32
[ 3400.297687] [<ffffffff965eaf0c>] _drbd_no_send_page+0x53/0x76
[ 3400.311279] [<ffffffff965eb350>] drbd_send_dblock+0x28a/0x534
[ 3400.324876] [<ffffffff96153416>] ? __wake_up+0x40/0x53
[ 3400.336646] [<ffffffff965e1e6c>] ? complete_master_bio+0x139/0x150
[ 3400.351540] [<ffffffff965e2860>] ? __req_mod+0x9dd/0x9f9
[ 3400.363833] [<ffffffff965d337a>] w_send_dblock+0xf1/0x17e
[ 3400.376386] [<ffffffff965d4582>] drbd_worker+0x17f/0x3a7
[ 3400.388679] [<ffffffff965e770d>] drbd_thread_setup+0x59/0x16a
[ 3400.402272] [<ffffffff965e76b4>] ? drbd_rs_del_all+0x42b/0x42b
[ 3400.416128] [<ffffffff96138bac>] kthread+0xed/0xff
[ 3400.426858] [<ffffffff9697546e>] ret_from_fork+0x1e/0x50
[ 3400.439148] [<ffffffff96138abf>] ? __kthread_parkme+0x7a/0x7a
[ 3400.452738] Code: 90 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 4c 09 24 24 c3 66 90 48 89 f8 48 89 d1 <f3> a4 4c 09 24 24 c3 0f 1f 00 48 89 f8 48 83 fa 20 72 7e 40 38
[ 3400.508781] RIP [<ffffffff964d8306>] memcpy_erms+0x6/0x10
[ 3400.521387] RSP <ffffc9000050bad0>
[ 3400.527949] CR2: 0000000000000003
[ 3400.533997] ---[ end trace 539f6e00a1a5d619 ]---
---8<---

# addr2line -e vmlinux-img-4.7.4-hardened-r1-dom0-kvm-at-gw27-dbg.x86_64 -fip ffffffff968b7618
?? ??:0

» root@io:~ # cat /proc/config.gz | gzip -d | grep "CONFIG_DEBUG_INFO\|CONFIG_FRAME_POINTER"
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
CONFIG_FRAME_POINTER=y
» root@io:~ #

This is vmlinux image:
http://dl.georgweiss.de/kernel/dom0-kvm ... dbg.x86_64

[PaX Team]

you have KASLR enabled so you'll have to adjust the reported address for its random offset (ffffffff818b7618 in your case). anyway, for some reason tcp_sendmsg calls skb_copy_to_page_nocache with a null buffer pointer but i don't know who the sender of that buffer is as this is just the receiver thread. can you perhaps reproduce this with a vanilla kernel?

[PaX Team]

this is presumably a known upstream bug fixed by https://git.kernel.org/cgit/linux/kerne ... 7376e52e54, can you test it please?

[brainatwork]

This works for me now with 4.8.10-hardened.