Kernel panic on first module load

[adrolter]

Linux: 4.8.y (4.8.14 and 4.8.15 tested)
Platform: Amlogic Meson GXBB (Hardkernel ODROID-C2 SBC)

Code: Select all

[    5.233612] kernel BUG at kernel/module.c:1966!
[    5.237049] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    5.242482] Modules linked in:
[    5.245503] CPU: 1 PID: 1 Comm: systemd Not tainted 4.8.15-grsec-gxbb #1
[    5.252141] Hardware name: Hardkernel ODROID-C2 (DT)
[    5.257058] task: ffff800074480000 task.stack: ffff80007446c000
[    5.262930] PC is at frob_writable_data.isra.21+0x38/0x40
[    5.268272] LR is at complete_formation.isra.38+0x120/0x1b0
[    5.273790] pc : [<ffff000008124d68>] lr : [<ffff000008125b70>] pstate: 00000145
[    5.281120] sp : ffff80007446fc90
[    5.284398] x29: ffff80007446fc90 x28: ffff80007446fe68
[    5.289659] x27: ffff80007446fe58 x26: ffff80007446fd50
[    5.294920] x25: ffff000008ca2000 x24: ffff80007446fd50
[    5.300181] x23: ffff000000842b80 x22: 0000000000000001
[    5.305443] x21: ffff000008124b68 x20: 0000000000000000
[    5.310704] x19: ffff000008096c20 x18: 0000000000000000
[    5.315965] x17: 0000ffff94a842d8 x16: ffff0000081282f8
[    5.321226] x15: ffffffffffffffff x14: ffff000000000000
[    5.326488] x13: ffffffffffffffff x12: 0000000000000018
[    5.331749] x11: 0000000000000018 x10: 0101010101010101
[    5.337010] x9 : 0000000000000001 x8 : 7f7f7f7f7f7f7f7f
[    5.342272] x7 : 636e6c2cff646b61 x6 : 000000802e6d6f64
[    5.347533] x5 : 646f6d2e00000000 x4 : 00c8000072c90713
[    5.352794] x3 : 0000000000000080 x2 : ffff000008096c20
[    5.358055] x1 : 0000000000008298 x0 : ffff000000840000
[    5.363316]
[    5.364784] Process systemd (pid: 1, stack limit = 0xffff80007446c020)
[    5.371253] Stack: (0xffff80007446fc90 to 0xffff800074470000)
[    5.376946] fc80:                                   ffff80007446fd50 ffff0000081276a8
[    5.384708] fca0: 0001000000000000 ffff80007446fe58 ffff000000842b98 ffff000000842e98
[    5.392471] fcc0: ffff000000842b80 ffff0000088509e0 000000000000091f ffff000008ca2000
[    5.400233] fce0: 0000000000000c2c ffff000000887ce8 ffff80007446fd10 ffff000000887010
[    5.407996] fd00: ffff000000880001 ffff000000887654 ffff80007446fd50 ffff000008127644
[    5.415758] fd20: ffff0000008877e8 ffff800000000028 ffff000000887a68 ffff000000000028
[    5.423521] fd40: 0000000000000000 ffff000000000000 ffff80007446fe10 ffff0000081283d0
[    5.431283] fd60: 0000000000000000 0000000000000004 0000ffff94a72830 0000ffff949463e4
[    5.439046] fd80: 0000000080000000 0000000000000015 0000000000000120 0000000000000111
[    5.446808] fda0: ffff000008842000 ffff80007446c000 ffff000000000064 ffff00000000006e
[    5.454571] fdc0: ffff00000000003f 0000feff081d83a0 ffff00000089dfd8 ffff000000000018
[    5.462333] fde0: 0000000000000000 ffff000000842e98 0000ffff94a72830 0000ffff949463e4
[    5.470096] fe00: 0000000080000000 0000000000000015 0000000000000000 ffff000008082ef0
[    5.477858] fe20: 0000000000000000 0000ffff94a72830 ffffffffffffffff 0000000000000000
[    5.485621] fe40: ffff80007446feb0 0000000000d29e30 ffff000008e95000 ffff000008e95000
[    5.493383] fe60: 0000000000d29e30 ffff000009bbe070 ffff000009bbde60 ffff0000095822d8
[    5.501146] fe80: 000000000003dce8 000000000004b7d0 0000000000000000 0000000000000000
[    5.508908] fea0: 000000000001a210 0000003600000035 0000000000000021 0000001f0000000c
[    5.516671] fec0: 0000000000000004 0000ffff94a72830 0000000000000000 0000000000000004
[    5.524433] fee0: 0000000000000000 60ceffffffffffff ffffffffffffffff ffffffffffffffff
[    5.532196] ff00: 0000000000000111 0000000000000038 0101010101010101 0000000000000001
[    5.539958] ff20: 0000000000000000 ffffffffffff0000 0000000000000000 0000000000000720
[    5.547721] ff40: 0000ffff949463c0 0000ffff94a842d8 0000000000000000 0000aaaab3d1e4a0
[    5.555483] ff60: 0000ffff94a72830 0000000000000000 0000aaaab3d1e5b0 0000000000020000
[    5.563246] ff80: 0000aaaaaedb5028 0000000000000000 0000000000000000 0000aaaaaedb5058
[    5.571008] ffa0: 0000000000000000 0000ffffc337d7f0 0000ffff94a6b0ac 0000ffffc337d7f0
[    5.578771] ffc0: 0000ffff949463e4 0000000080000000 0000000000000004 0000000000000111
[    5.586533] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    5.594295] Call trace:
[    5.596711] Exception stack(0xffff80007446fac0 to 0xffff80007446fbf0)
[    5.603093] fac0: ffff000008096c20 0001000000000000 ffff80007446fc90 ffff000008124d68
[    5.610856] fae0: ffff80007446faf0 ffff0000083773b0 ffff80007446fb40 ffff000008124ba4
[    5.618618] fb00: ffff0000088508f8 ffff80007446fc68 ffff0000088508d8 0000000000000000
[    5.626381] fb20: ffff000000842b80 ffff000000842b80 ffff0000095890d5 ffff000008ca2000
[    5.634143] fb40: ffff80007446fbf0 ffff000008096b9c ffff00000089d000 000000000001b000
[    5.641906] fb60: ffff000000840000 0000000000008298 ffff000008096c20 0000000000000080
[    5.649668] fb80: 00c8000072c90713 646f6d2e00000000 000000802e6d6f64 636e6c2cff646b61
[    5.657431] fba0: 7f7f7f7f7f7f7f7f 0000000000000001 0101010101010101 0000000000000018
[    5.665193] fbc0: 0000000000000018 ffffffffffffffff ffff000000000000 ffffffffffffffff
[    5.672955] fbe0: ffff0000081282f8 0000ffff94a842d8
[    5.677786] [<ffff000008124d68>] frob_writable_data.isra.21+0x38/0x40
[    5.684169] [<ffff0000081276a8>] load_module+0x1430/0x1ed0
[    5.689602] [<ffff0000081283d0>] sys_finit_module+0xd8/0xe8
[    5.695123] [<ffff000008082ef0>] el0_svc_naked+0x24/0x28
[    5.700384] Code: d65f03c0 d503201f d4210000 d503201f (d4210000)
[    5.706441] ---[ end trace 93bf257b7b30b251 ]---


Pre grsec patching the same kconfig boots fine and loads external modules normally. Panic happens whether or not Grsecurity section is enabled at all in menuconfig.

Thanks!

[PaX Team]

can you print out the offending size (it's layout->size_rw in kernel/module.c:frob_writable_data) and module name?

[adrolter]

This seems to happen when loading literally any module. If I move/rename the /lib/modules/$(uname -r) directory before booting, the kernel boots to completion (albeit with failed units and inoperable devices), but any module I've tried to then manually modprobe/insmod (or systemd tries to autoload for a unit) after moving the directory back causes this panic.

Would you like the name and size_rw value of any module which causes it or, given that it doesn't appear to be module-specific, do you think there's an issue with the build step? I'm also cross compiling from x86_64 to ARM64, if relevant.

[PaX Team]

ok, in this case just pick one module and post the corresponding size_rw value along with its readelf -eW output please.

[adrolter]

I picked the x_tables module from net/netfilter.

readelf -eW net/netfilter/x_tables.ko

Code: Select all

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          598192 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         51
  Section header string table index: 48

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .note.gnu.build-id NOTE            0000000000000000 000040 000024 00   A  0   0  4
  [ 2] .text             PROGBITS        0000000000000000 000068 002bac 00  AX  0   0  8
  [ 3] .rela.text        RELA            0000000000000000 04d640 001758 18   I 49   2  8
  [ 4] .fixup            PROGBITS        0000000000000000 002c14 000010 00  AX  0   0  4
  [ 5] .rela.fixup       RELA            0000000000000000 04ed98 000030 18   I 49   4  8
  [ 6] .text.unlikely    PROGBITS        0000000000000000 002c28 000100 00  AX  0   0  8
  [ 7] .rela.text.unlikely RELA            0000000000000000 04edc8 000078 18   I 49   6  8
  [ 8] .init.text        PROGBITS        0000000000000000 002d28 000170 00  AX  0   0  8
  [ 9] .rela.init.text   RELA            0000000000000000 04ee40 000168 18   I 49   8  8
  [10] .exit.text        PROGBITS        0000000000000000 002e98 000038 00  AX  0   0  8
  [11] .rela.exit.text   RELA            0000000000000000 04efa8 000060 18   I 49  10  8
  [12] .rodata           PROGBITS        0000000000000000 002ed0 00075c 00   A  0   0  8
  [13] .rela.rodata      RELA            0000000000000000 04f008 0003d8 18   I 49  12  8
  [14] .altinstructions  PROGBITS        0000000000000000 00362c 000048 00   A  0   0  1
  [15] .rela.altinstructions RELA            0000000000000000 04f3e0 000120 18   I 49  14  8
  [16] .altinstr_replacement PROGBITS        0000000000000000 003674 000018 00   A  0   0  4
  [17] __ex_table        PROGBITS        0000000000000000 003690 000010 00   A  0   0  8
  [18] .rela__ex_table   RELA            0000000000000000 04f500 000060 18   I 49  17  8
  [19] .modinfo          PROGBITS        0000000000000000 0036a0 0000c2 00   A  0   0  8
  [20] __ksymtab_strings PROGBITS        0000000000000000 003762 00036c 00   A  0   0  1
  [21] __ksymtab         PROGBITS        0000000000000000 003ad0 000110 00   A  0   0  8
  [22] .rela__ksymtab    RELA            0000000000000000 04f560 000330 18   I 49  21  8
  [23] __ksymtab_gpl     PROGBITS        0000000000000000 003be0 0001b0 00   A  0   0  8
  [24] .rela__ksymtab_gpl RELA            0000000000000000 04f890 000510 18   I 49  23  8
  [25] .data             PROGBITS        0000000000000000 003d90 000038 00  WA  0   0  8
  [26] .rela.data        RELA            0000000000000000 04fda0 000018 18   I 49  25  8
  [27] .data..percpu     PROGBITS        0000000000000000 003dc8 000004 00  WA  0   0  8
  [28] .data..read_mostly PROGBITS        0000000000000000 003dd0 000018 00  WA  0   0  8
  [29] .gnu.linkonce.this_module PROGBITS        0000000000000000 003e00 000380 00  WA  0   0 128
  [30] .rela.gnu.linkonce.this_module RELA            0000000000000000 04fdb8 000030 18   I 49  29  8
  [31] .bss              NOBITS          0000000000000000 004180 000008 00  WA  0   0  8
  [32] .debug_info       PROGBITS        0000000000000000 004180 026ded 00      0   0  1
  [33] .rela.debug_info  RELA            0000000000000000 04fde8 038808 18   I 49  32  8
  [34] .debug_abbrev     PROGBITS        0000000000000000 02af6d 000c46 00      0   0  1
  [35] .debug_loc        PROGBITS        0000000000000000 02bbb3 003c64 00      0   0  1
  [36] .rela.debug_loc   RELA            0000000000000000 0885f0 007320 18   I 49  35  8
  [37] .debug_aranges    PROGBITS        0000000000000000 02f817 000080 00      0   0  1
  [38] .rela.debug_aranges RELA            0000000000000000 08f910 000090 18   I 49  37  8
  [39] .debug_ranges     PROGBITS        0000000000000000 02f897 000b40 00      0   0  1
  [40] .rela.debug_ranges RELA            0000000000000000 08f9a0 0018f0 18   I 49  39  8
  [41] .debug_line       PROGBITS        0000000000000000 0303d7 002288 00      0   0  1
  [42] .rela.debug_line  RELA            0000000000000000 091290 000060 18   I 49  41  8
  [43] .debug_str        PROGBITS        0000000000000000 03265f 016706 01  MS  0   0  1
  [44] .comment          PROGBITS        0000000000000000 048d65 00005c 01  MS  0   0  1
  [45] .note.GNU-stack   PROGBITS        0000000000000000 048dc1 000000 00      0   0  1
  [46] .debug_frame      PROGBITS        0000000000000000 048dc8 000ea0 00      0   0  8
  [47] .rela.debug_frame RELA            0000000000000000 0912f0 000bd0 18   I 49  46  8
  [48] .shstrtab         STRTAB          0000000000000000 091ec0 0001ec 00      0   0  1
  [49] .symtab           SYMTAB          0000000000000000 049c68 0029d0 18     50 296  8
  [50] .strtab           STRTAB          0000000000000000 04c638 001008 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

There are no program headers in this file.


Added to top of kernel/module.c:frob_writable_data:

Code: Select all

printk(KERN_ALERT "DEBUG: module_layout->size_rw = %lu", layout->size_rw);


sudo insmod net/netfilter/x_tables.ko

Code: Select all

[  145.101856] DEBUG: module_layout->size_rw = 1032
<segfault>