panic on boot after SVN r328988

[Michael Butler]

This is on a slow (and remote :-() i386

(kgdb) bt
#0 0xc076bfe8 in doadump ()
#1 0xc076c008 in doadump ()
#2 0xc0d00ee0 in suspend_blocked ()
#3 0xcf607548 in ?? ()
#4 0xc076bd8b in kern_reboot ()
#5 0xc076c141 in vpanic ()
#6 0xc076c03b in panic ()
#7 0xc0ab5065 in trap_fatal ()
#8 0xc0ab4de0 in trap ()
#9 <signal handler called>
#10 0xc0991ff0 in dyn_lookup_ipv4_state ()
#11 0xc0992323 in ipfw_dyn_lookup_state ()
#12 0xc098e2b9 in ipfw_chk ()
#13 0xc09983b9 in ipfw_check_packet ()
#14 0xc0879cf8 in pfil_run_hooks ()
#15 0xc08983e4 in ip_input ()
#16 0xc0878e94 in netisr_dispatch_src ()
#17 0xc0879190 in netisr_dispatch ()
#18 0xc0860434 in ether_demux ()
#19 0xc08611c9 in ether_nh_input ()
#20 0xc0878e94 in netisr_dispatch_src ()
#21 0xc0879190 in netisr_dispatch ()
#22 0xc08607aa in ether_input ()
#23 0xc08574e9 in if_input ()
#24 0xc05b5bbe in fxp_intr_body ()
#25 0xc05b4137 in fxp_intr ()
#26 0xc0732da9 in intr_event_execute_handlers ()
#27 0xc073303f in ithread_loop ()
#28 0xc0730351 in fork_exit ()
#29 <signal handler called>

ipfw is compiled into the kernel not loaded as a module.

The best I can do to capture where in the boot process it was comes from
searching the vmcore file for strings ..

[ .. ]

<118>devmatch: Can't read linker hints file.
<118>add host 127.0.0.1: gateway lo0 fib 0: route already in table
<118>add net default: gateway 64.xx.xxx.x
<118>Additional inet routing options: gateway=YES.
<118>add host ::1: gateway lo0 fib 0: route already in table
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>add net default: gateway 2001:xxx:xxxx:xxx::x
<118>Additional inet6 routing options: gateway=YES.
<118>00600 allow ip6 from any to any via lo0
<118>00700 deny ip6 from any to ::1
<118>00800 deny ip6 from ::1 to any
<118>00900 allow ip6 from :: to ff02::/16 ipv6-icmp
<118>01000 allow ip6 from fe80::/10 to fe80::/10 ipv6-icmp
<118>01100 allow ip6 from fe80::/10 to ff02::/16 ipv6-icmp
<118>01200 allow ip6 from any to any proto ipv6-icmp ip6 icmp6types
1,2,135,136
<118>01300 allow ip6 from me6 to any
<110>ipfw: 4900 Deny TCP 77.72.82.80:50823 202.xx.xxx.xx:7696 in via fxp0
<118>Firewall rules loaded.
<5>bridge0: link state changed to DOWN
<6>tap0: promiscuous mode enabled
<110>ipfw: 4900 Deny TCP 77.72.85.24:57077 202.xx.xxx.xx:33970 in via fxp0
<118>Creating and/or trimming log files
<110>ipfw: 4900 Deny TCP 5.188.11.111:59849 202.xx.xxx.xxx:7300 in via fxp0
<118>.
<118>Starting syslogd.
<110>ipfw: 4900 Deny TCP 5.188.11.25:51041 202.xx.xxx.xxx:20826 in via fxp0
<118>Setting date via ntp.
Fatal trap 1: privileged instruction fault while in kernel mode
instruction pointer = 0x20:0xc0991ff0
stack pointer = 0x28:0xcf607704
frame pointer = 0x28:0xcf607728
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (irq7: fxp0 fxp1)
trap number = 1
panic: privileged instruction fault
time = 1518744036
Uptime: 26s
Physical memory: 499 MB
Dumping 51 MB:

[Andrey V. Elsukov]

Post by Michael Butler
ipfw is compiled into the kernel not loaded as a module.

Hi,

do you have some specific optimization flags in make.conf?
Can you show the output of `head -40 /var/run/dmesg.boot`?

[Michael Butler]

Post by Andrey V. Elsukov

Post by Michael Butler
ipfw is compiled into the kernel not loaded as a module.

Hi,
do you have some specific optimization flags in make.conf?
Can you show the output of `head -40 /var/run/dmesg.boot`?

The only relevant flags in /etc/make.conf are ..

CPUTYPE?=pentium3
KERNCONF=SARAH
NO_MODULES=YES

Boot log from last night's failure attached,

imb

[Andrey V. Elsukov]

Post by Michael Butler

Post by Andrey V. Elsukov
do you have some specific optimization flags in make.conf?
Can you show the output of `head -40 /var/run/dmesg.boot`?

The only relevant flags in /etc/make.conf are ..
CPUTYPE?=pentium3
KERNCONF=SARAH
NO_MODULES=YES
Boot log from last night's failure attached,

Ok, it seems ConcurrencyKit was not tested with Pintium3.
Can you show the output from kgdb:

disassemble *0xc0991ff0

[Michael Butler]

Post by Andrey V. Elsukov

Post by Michael Butler

Post by Andrey V. Elsukov
do you have some specific optimization flags in make.conf?
Can you show the output of `head -40 /var/run/dmesg.boot`?

The only relevant flags in /etc/make.conf are ..
CPUTYPE?=pentium3
KERNCONF=SARAH
NO_MODULES=YES
Boot log from last night's failure attached,

Ok, it seems ConcurrencyKit was not tested with Pintium3.
disassemble *0xc0991ff0

That'd do it .. :-(

(kgdb) disassemble 0xc0991ff0
Dump of assembler code for function dyn_lookup_ipv4_state:
0xc0991fa0 <+0>: push %ebp
0xc0991fa1 <+1>: mov %esp,%ebp
0xc0991fa3 <+3>: push %ebx
0xc0991fa4 <+4>: push %edi
0xc0991fa5 <+5>: push %esi
0xc0991fa6 <+6>: sub $0x18,%esp
0xc0991fa9 <+9>: mov 0x10(%ebp),%edx
0xc0991fac <+12>: mov 0xc0d19dd4,%ecx
0xc0991fb2 <+18>: dec %ecx
0xc0991fb3 <+19>: and 0x4(%edx),%ecx
0xc0991fb6 <+22>: mov 0xc0d19dd8,%eax
0xc0991fbb <+27>: mov (%eax,%ecx,4),%eax
0xc0991fbe <+30>: mov %eax,0x8(%edx)
0xc0991fc1 <+33>: mov 0xc0d19ddc,%eax
0xc0991fc6 <+38>: mov (%eax,%ecx,4),%eax
0xc0991fc9 <+41>: mov %eax,-0x14(%ebp)
0xc0991fcc <+44>: mov 0xc0d19de0,%eax
0xc0991fd1 <+49>: mov (%eax,%ecx,4),%ebx
0xc0991fd4 <+52>: test %ebx,%ebx
0xc0991fd6 <+54>: je 0xc09920ba <dyn_lookup_ipv4_state+282>
0xc0991fdc <+60>: mov 0x8(%ebp),%edx
0xc0991fdf <+63>: mov %ecx,-0x10(%ebp)
0xc0991fe2 <+66>: data16 data16 data16 data16 nopw
%cs:0x0(%eax,%eax,1)
0xc0991ff0 <+80>: lfence
0xc0991ff3 <+83>: mov %fs:0x58,%eax
0xc0991ff9 <+89>: mov %eax,-0x18(%ebp)
0xc0991ffc <+92>: mov %ebx,-0x3f409e80(%eax)
0xc0992002 <+98>: mov 0xc0d19ddc,%eax
0xc0992007 <+103>: mov (%eax,%ecx,4),%eax
0xc099200a <+106>: cmp %eax,-0x14(%ebp)
0xc099200d <+109>: jne 0xc099209d <dyn_lookup_ipv4_state+253>
0xc0992013 <+115>: movzbl 0x1(%ebx),%eax
0xc0992017 <+119>: cmp 0xd(%edx),%al
0xc099201a <+122>: jne 0xc0992090 <dyn_lookup_ipv4_state+240>
0xc099201c <+124>: mov 0x10(%ebp),%eax
0xc099201f <+127>: movzwl 0x2(%eax),%eax
0xc0992023 <+131>: test %ax,%ax
0xc0992026 <+134>: je 0xc099202e <dyn_lookup_ipv4_state+142>
0xc0992028 <+136>: cmp %ax,0x2(%ebx)
0xc099202c <+140>: jne 0xc0992090 <dyn_lookup_ipv4_state+240>
0xc099202e <+142>: movzwl 0x4(%ebx),%esi
0xc0992032 <+146>: movzwl 0xa(%edx),%edi
0xc0992036 <+150>: cmp %di,%si
0xc0992039 <+153>: jne 0xc0992063 <dyn_lookup_ipv4_state+195>

[Olivier Houchard]

Hi Michael,

Post by Michael Butler

Post by Andrey V. Elsukov

Post by Michael Butler

Post by Andrey V. Elsukov
do you have some specific optimization flags in make.conf?
Can you show the output of `head -40 /var/run/dmesg.boot`?

The only relevant flags in /etc/make.conf are ..
CPUTYPE?=pentium3
KERNCONF=SARAH
NO_MODULES=YES
Boot log from last night's failure attached,

Ok, it seems ConcurrencyKit was not tested with Pintium3.
disassemble *0xc0991ff0

That'd do it .. :-(

[...]

Sorry about this.
It should be fixed with r329388.
Can you update and let me know how it goes ?

Thanks !

Olivier