Panic in prison_alloc() on boot

[Ryan Stone]

I'm getting a persistent panic on boot in prison_allow(). The first
case that I hit is this:

Fatal trap 12: page fault while in kernel mode
cpuid = 10; apic id = 22
fault virtual address = 0x30
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ab9674
stack pointer = 0x28:0xfffffe00f8bb16f0
frame pointer = 0x28:0xfffffe00f8bb16f0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 90 (mount_nullfs)
[ thread pid 90 tid 100913 ]
Stopped at prison_allow+0x4: movq ll+0xf(%rdi),%rax
db> bt
Tracing pid 90 tid 100913 td 0xfffff801a5aab000
prison_allow() at prison_allow+0x4/frame 0xfffffe00f8bb16f0
nullfs_mount() at nullfs_mount+0x31/frame 0xfffffe00f8bb1830
vfs_donmount() at vfs_donmount+0x1415/frame 0xfffffe00f8bb1a80
sys_nmount() at sys_nmount+0x72/frame 0xfffffe00f8bb1ac0
amd64_syscall() at amd64_syscall+0xa48/frame 0xfffffe00f8bb1bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffffffecb0

However if I comment out my nullfs mounts in /etc/fstab, I just get a
panic in prison_allow() called from elsewhere.

I've seen this panic with r328936 (Feb 6), r329091 (Feb 9) and r329142 (Feb 11)

[Konstantin Belousov]

Post by Ryan Stone
I'm getting a persistent panic on boot in prison_allow(). The first
Fatal trap 12: page fault while in kernel mode
cpuid = 10; apic id = 22
fault virtual address = 0x30
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ab9674
stack pointer = 0x28:0xfffffe00f8bb16f0
frame pointer = 0x28:0xfffffe00f8bb16f0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 90 (mount_nullfs)
[ thread pid 90 tid 100913 ]
Stopped at prison_allow+0x4: movq ll+0xf(%rdi),%rax
db> bt
Tracing pid 90 tid 100913 td 0xfffff801a5aab000
prison_allow() at prison_allow+0x4/frame 0xfffffe00f8bb16f0

Can you get the core dump and look at the arguments ?
It is most likely cred->cr_prison which is NULL.

Post by Ryan Stone
nullfs_mount() at nullfs_mount+0x31/frame 0xfffffe00f8bb1830
vfs_donmount() at vfs_donmount+0x1415/frame 0xfffffe00f8bb1a80
sys_nmount() at sys_nmount+0x72/frame 0xfffffe00f8bb1ac0
amd64_syscall() at amd64_syscall+0xa48/frame 0xfffffe00f8bb1bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffffffecb0
However if I comment out my nullfs mounts in /etc/fstab, I just get a
panic in prison_allow() called from elsewhere.
I've seen this panic with r328936 (Feb 6), r329091 (Feb 9) and r329142 (Feb 11)
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

[Ryan Stone]

Sorry for the late reply. Panicking this system is a bit painful, but
I found some time to do it today.

Strangely, it's actually cred that is NULL, not cred->cr_prison:

(kgdb) p cred
$7 = (struct ucred *) 0x0
(kgdb) disassemble
Dump of assembler code for function prison_allow:
0xffffffff80ac33e0 <+0>: push %rbp
0xffffffff80ac33e1 <+1>: mov %rsp,%rbp
=> 0xffffffff80ac33e4 <+4>: mov 0x30(%rdi),%rax
0xffffffff80ac33e8 <+8>: and 0xf8(%rax),%esi
0xffffffff80ac33ee <+14>: mov %esi,%eax
0xffffffff80ac33f0 <+16>: pop %rbp
0xffffffff80ac33f1 <+17>: retq
End of assembler dump.
(kgdb) info reg $rdi
rdi 0x0 0

However, if I go up a frame, things look fine?

(kgdb) up
#13 0xffffffff82c22531 in nullfs_mount (mp=0xfffff801a483d000)
at /usr/src/sys/fs/nullfs/null_vfsops.c:88
88 if (!prison_allow(td->td_ucred, PR_ALLOW_MOUNT_NULLFS))
(kgdb) p td->td_ucred
$8 = (struct ucred *) 0xfffff801854c1700

This appears to be a miscompilation, but I've blown away
/usr/obj/usr/src multiple times and rebuilt and got this same error
every time. But looking at the disassembly, something is definitely
wrong:

0xffffffff82c22517 <+23>: mov %gs:0x0,%r14
0xffffffff82c22520 <+32>: mov 0x150(%r14),%rdi
0xffffffff82c22527 <+39>: mov $0x100,%esi
0xffffffff82c2252c <+44>: callq 0xffffffff80ac33e0 <prison_allow>
=> 0xffffffff82c22531 <+49>: test %eax,%eax

(kgdb) p &((struct thread*)0)->td_ucred
$10 = (struct ucred **) 0x158

It uses offset 0x150 to get the cred, but the debug info claims that
td_ucred is at offset 0x158. If I print out the pointer at that
offset, it looks reasonable:

(kgdb) p *td->td_ucred
$11 = {cr_ref = 107, cr_uid = 0, cr_ruid = 0, cr_svuid = 0, cr_ngroups = 1,
cr_rgid = 0, cr_svgid = 0, cr_uidinfo = 0xfffff80106617000,
cr_ruidinfo = 0xfffff80106617000, cr_prison = 0xffffffff8187cb70 <prison0>,
cr_loginclass = 0xfffff8019fa43b00, cr_flags = 0, cr_pspare2 = {0x0, 0x0},
cr_label = 0x0, cr_audit = {ai_auid = 4294967295, ai_mask = {am_success = 0,
am_failure = 0}, ai_termid = {at_port = 0, at_type = 4, at_addr = {0, 0,
0, 0}}, ai_asid = 0, ai_flags = 0}, cr_groups = 0xfffff801854c179c,
cr_agroups = 16, cr_smallgroups = {0 <repeats 16 times>}}



I'm really at a loss at to what to try next. Build with
MAKEOBJDIRPREFIX set to something else to get rid of any lingering
possibility of an issue in my objdir, I guess?

[Ryan Stone]

To close the loop on this, the root cause ended up being a mistake on
my end. This system had a rather convoluted boot process, and as a
result of that was loading a nullfs.ko built for a months-old kernel.
This setup accidentally worked for some time, but I guess some recent
change to struct thread changed the ABI, causing the old nullfs.ko to
be incompatible and fail to boot.

Sorry for the noise,
Ryan