SOLVED: Null pointer dereference (tproxy)

[brainatwork]

Hi again

This happens to me starting from 4.7.1 - 4.7.4 on a kvm virtual machine running squid/tproxy. I didn't find a way to trigger it, happens occasionally.Running 4.6.5-hardened-r1 is fine.

--8<--
BUG: unable to handle kernel NULL pointer dereference at 0000000000000441
IP: [<ffffffff8774ded1>] __inet_lookup_listener+0x70/0x198
PGD 6e2a8000
Oops: 0000 [#1] SMP
CPU: 0 PID: 1917 Comm: tor Not tainted 4.7.1-hardened-domU-gw25 #3
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff88007c8d2200 ti: ffff88007c8d2880 task.ti: ffff88007c8d2880
RIP: 0010:[<ffffffff8774ded1>] [<ffffffff8774ded1>] __inet_lookup_listener+0x70/0x198
RSP: 0000:ffff88007fc03a50 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000000411 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000411
RBP: 00000000ffffffff R08: 000000009b02a8c0 R09: 0000000000008aa7
R10: ffff88006cbca400 R11: 000000006608a8c0 R12: 0000000000000000
R13: ffffffff87e5dc80 R14: 0000000000000c39 R15: 0000000000000c39
FS: 00000375398b4700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000441 CR3: 0000000007ced000 CR4: 00000000000006b0
Stack:
ffff88006cbca400 0000003c87e5dc80 00008aa79b02a8c0 ffff88006cbca400
0000000000000c39 ffff88006d788262 ffff88006d78824e ffffffff87e5dc80
0000000000000000 ffffffff876ffc9a 000000006608a8c0 0000000000000c39
Call Trace:
<IRQ>
[<ffffffff876ffc9a>] ? tproxy_tg4+0x38a/0x45e
[<ffffffff877a3fb6>] ? ipt_do_table+0x346/0x5f1
[<ffffffff876e61c9>] ? tcp_new+0x1bb/0x1bb
[<ffffffff876e1834>] ? nf_conntrack_in+0x37a/0x4bd
[<ffffffff876d9391>] ? nf_iterate+0x58/0x8f
[<ffffffff876d93ee>] ? nf_hook_slow+0x26/0x9b
[<ffffffff87743e17>] ? ip_rcv+0x4c5/0x515
[<ffffffff87743431>] ? ip_local_deliver_finish+0x24d/0x24d
[<ffffffff87679532>] ? __netif_receive_skb_core+0x854/0xadb
[<ffffffff87679532>] ? __netif_receive_skb_core+0x854/0xadb
[<ffffffff87679d96>] ? dev_gro_receive+0x3d0/0x5de
[<ffffffff87679d96>] ? dev_gro_receive+0x3d0/0x5de
[<ffffffff876798d5>] ? netif_receive_skb_internal+0x37/0x7b
[<ffffffff876798d5>] ? netif_receive_skb_internal+0x37/0x7b
[<ffffffff8767e14a>] ? napi_gro_receive+0x40/0xb2
[<ffffffff875ec5c8>] ? virtnet_receive+0x8d4/0x8fa
[<ffffffff871022be>] ? __wake_up+0x33/0x4c
[<ffffffff87563ed2>] ? serial8250_tx_chars+0x119/0x2af
[<ffffffff875ec7f3>] ? virtnet_poll+0x13/0x76
[<ffffffff8767e82e>] ? net_rx_action+0x185/0x441
[<ffffffff870d466a>] ? __do_softirq+0xda/0x1e1
[<ffffffff870d4908>] ? irq_exit+0x39/0x80
[<ffffffff8701804a>] ? do_IRQ+0xc1/0xe5
[<ffffffff87833cce>] ? common_interrupt+0x8e/0x8e
<EOI>
Code: e0 04 48 01 c6 48 8b 46 08 48 8d 50 98 48 85 c0 48 0f 45 da 45 31 e4 31 d2 31 f6 31 c9 31 c0 48 85 db 0f 84 0b 01 00 00 83 cd ff <4c> 39 6b 30 0f 85 b1 00 00 00 66 44 39 73 0e 0f 85 a6 00 00 00
RIP [<ffffffff8774ded1>] __inet_lookup_listener+0x70/0x198
RSP <ffff88007fc03a50>
CR2: 0000000000000441
---[ end trace d24122622dce7472 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: 0x6000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: Fatal exception in interrupt
--8<--

[PaX Team]

you know the drill by now , please enable frame pointers and then resolve the address reported for __inet_lookup_listener to a source line (addr2line -e .../vmlinux -fip <addr>). for best results enable DEBUG_INFO in your config, that way we'll see even inlined code.

[brainatwork]

Hi

This is from 4.7.9-hardened

--8<--
PAX: please report this to [email protected]
BUG: unable to handle kernel paging request at 0000000000006dd3
IP: [<ffffffff817915d9>] __inet_lookup_listener+0x73/0x18f
PGD 6c621000
Oops: 0000 [#1] SMP
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.7.9-hardened-domU-gw26-dbg #1
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffffffff81e05e40 ti: ffffffff81e063a8 task.ti: ffffffff81e063a8
RIP: 0010:[<ffffffff817915d9>] [<ffffffff817915d9>] __inet_lookup_listener+0x73/0x18f
RSP: 0000:ffff88007fc03988 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000006da3 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000006da3
RBP: ffff88007fc039c8 R08: 000000009b02a8c0 R09: 000000000000c1e4
R10: ffff88006cae4c00 R11: 000000006608a8c0 R12: 00000000ffffffff
R13: 0000000000000000 R14: ffffffff81e56740 R15: 0000000000000c39
FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000006dd3 CR3: 0000000001cd9000 CR4: 00000000000006b0
Stack:
0000c1e400000c39 0000003c9b02a8c0 ffff88006cae4c00 ffff88006cae4c00
0000000000000c39 ffff88006d1d8862 ffff88006d1d884e 0000000000000000
ffff88007fc03a58 ffffffff8174142f 000000006608a8c0 0000000000000c39
Call Trace:
<IRQ>
[<ffffffff8174142f>] tproxy_tg4+0x39e/0x487
[<ffffffff817270b9>] ? tcp_packet+0xdd8/0xf07
[<ffffffff8174153d>] tproxy_tg4_v1+0x25/0x35
[<ffffffff817eb80a>] ipt_do_table+0x364/0x62c
[<ffffffff817ebeb2>] iptable_mangle_hook+0xf4/0x112
[<ffffffff817ebdbe>] ? iptable_mangle_net_exit+0x3c/0x3c
[<ffffffff81718667>] nf_iterate+0x5d/0x96
[<ffffffff817186dd>] nf_hook_slow+0x3d/0xc9
[<ffffffff81787269>] ip_rcv+0x41a/0x469
[<ffffffff817868ee>] ? ip_local_deliver_finish+0x264/0x264
[<ffffffff816b340f>] __netif_receive_skb_core+0x884/0xb1e
[<ffffffff817c30d6>] ? inet_gro_receive+0x46e/0x49a
[<ffffffff816b3765>] __netif_receive_skb+0x13/0x6f
[<ffffffff816b3765>] ? __netif_receive_skb+0x13/0x6f
[<ffffffff816b380b>] netif_receive_skb_internal+0x4a/0xa3
[<ffffffff816b849b>] napi_gro_receive+0x45/0xb9
[<ffffffff8161fb85>] virtnet_receive+0x8de/0x900
[<ffffffff814f8616>] ? __list_del_entry+0x20/0x61
[<ffffffff8161fdca>] virtnet_poll+0x18/0x80
[<ffffffff816b8bed>] net_rx_action+0x191/0x469
[<ffffffff810d3e72>] __do_softirq+0xdb/0x1e3
[<ffffffff810d412b>] irq_exit+0x3d/0x8c
[<ffffffff81013915>] do_IRQ+0xc7/0xed
[<ffffffff818805ce>] common_interrupt+0x8e/0x8e
<EOI>
[<ffffffff8101a072>] ? hard_enable_TSC+0x37/0x37
[<ffffffff810309ef>] ? native_safe_halt+0x6/0x16
[<ffffffff8101a07b>] default_idle+0x9/0x1b
[<ffffffff8101a696>] arch_cpu_idle+0x21/0x33
[<ffffffff81105a1f>] default_idle_call+0x27/0x37
[<ffffffff81105b91>] cpu_startup_entry+0x162/0x222
[<ffffffff8187a0b4>] rest_init+0x6d/0x7d
[<ffffffff82014d8f>] 0xffffffff82014d8f
[<ffffffff82083951>] ? 0xffffffff82083951
[<ffffffff82013120>] ? 0xffffffff82013120
[<ffffffff8201363e>] 0xffffffff8201363e
[<ffffffff8201363e>] ? 0xffffffff8201363e
[<ffffffff82013766>] 0xffffffff82013766
Code: 04 48 01 c6 48 8b 46 08 48 8d 50 98 48 85 c0 48 0f 45 da 45 31 ed 31 c0 31 f6 31 c9 31 d2 48 85 db 0f 84 03 01 00 00 41 83 cc ff <4c> 39 73 30 0f 85 a5 00 00 00 66 44 39 7b 0e 0f 85 9a 00 00 00
RIP [<ffffffff817915d9>] __inet_lookup_listener+0x73/0x18f
RSP <ffff88007fc03988>
CR2: 0000000000006dd3
---[ end trace b0e080b9f2234122 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt
--8<--

vmlinux image is here:
http://dl.georgweiss.de/kernel/domU/vml ... dbg.x86_64

[brainatwork]

krnlm@gentoo-krnlm-11613 ~/kernel/bin/domU $ addr2line -e vmlinux-img-4.7.9-hardened-domU-gw26-dbg.x86_64 -fip ffffffff8174142f
inet_lookup_listener at /usr/src/linux/include/net/inet_hashtables.h:227
(inlined by) nf_tproxy_get_sock_v4 at /usr/src/linux/net/netfilter/xt_TPROXY.c:123
(inlined by) tproxy_tg4 at /usr/src/linux/net/netfilter/xt_TPROXY.c:333
krnlm@gentoo-krnlm-11613 ~/kernel/bin/domU $

[spender]

Can you resolve ffffffff817915d9 ?

-Brad

[brainatwork]

krnlm@gentoo-krnlm-11613 ~/kernel/bin/domU $ addr2line -e vmlinux-img-4.7.9-hardened-domU-gw26-dbg.x86_64 -fip ffffffff817915d9
compute_score at /usr/src/linux/net/ipv4/inet_hashtables.c:183
(inlined by) __inet_lookup_listener at /usr/src/linux/net/ipv4/inet_hashtables.c:225
krnlm@gentoo-krnlm-11613 ~/kernel/bin/domU $

[brainatwork]

Hi
It's stil broken in 4.7.10-hardened-r1. Any clue? Do you need more informations?

Thanks

[spender]

Have you tried a vanilla kernel of the same version to see if it's an upstream bug?

-Brad

[brainatwork]

This is solved for me with 4.8.6