Re: 'show alllocks' of completely locked machine [Was: Re: Complete IO lockup, state "ufs" from userland, debuging help wanted]
Bezüglich hiren panchasara's Nachricht vom 06.03.2017 21:10 (localtime): > On 03/06/17 at 08:56P, Harry Schmalzbauer wrote: >> Bez?glich Harry Schmalzbauer's Nachricht vom 05.03.2017 22:59 (localtime): >>> Hello, >>> >>> I can easily lock up FreeBSD stable/11 from userland. Not that I want to... >>> I'm running squid, which starts an authentication helper >>> "*negotiate_kerberos_auth*", which seems to be the culprit. >>> Completely all IO is blocked, there's no way to get anything from any >>> filesystem. >>> All non IO-requesting processes(threads) run well, including sshd and >>> shells. >>> There's no load (neither cpu nor io) just any process requesting io >>> stucks in state "ufs" >>> >>> Can anyone help me finding out what's going wrong? >>> Serial console is available. >> >> Dear hackers, >> >> I managed to get into DDB, but I'm lost from there? >> >> What information could be usefull to find out the cause of this complete >> lockup? >> >> I'd need someone who could guide me through ? I'd pay for a debuging >> lesson! (quiet constrained budget though) >> >> This happens when the machine got stuck: >> >> intr_event_handle() at intr_event_handle+0x9c/frame 0xfe0093dcb7d0 >> intr_execute_handlers() at intr_execute_handlers+0x48/frame >> 0xfe0093dcb800 >> lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfe0093dcb840 >> Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfe0093dcb840 >> --- interrupt, rip = 0x807b9bd6, rsp = 0xfe0093dcb910, rbp = >> 0xfe0093dcb910 --- >> acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfe0093dcb910 >> acpi_cpu_idle() at acpi_cpu_idle+0x2ea/frame 0xfe0093dcb960 >> cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfe0093dcb980 >> cpu_idle() at cpu_idle+0x8f/frame 0xfe0093dcb9a0 >> sched_idletd() at sched_idletd+0x436/frame 0xfe0093dcba70 >> fork_exit() at fork_exit+0x84/frame 0xfe0093dcbab0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfe0093dcbab0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> >> >> db> show alllocks >> Process 1259 (negotiate_kerberos_) thread 0xf80005ddea00 (100096) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1258 (negotiate_kerberos_) thread 0xf80005ddc500 (100252) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1257 (negotiate_kerberos_) thread 0xf80005ddda00 (100247) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1256 (negotiate_kerberos_) thread 0xf80065612500 (100261) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1255 (negotiate_kerberos_) thread 0xf80065612a00 (100260) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1254 (negotiate_kerberos_) thread 0xf80065613000 (100257) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1253 (negotiate_kerberos_) thread 0xf80065614000 (100254) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1252 (negotiate_kerberos_) thread 0xf800651e1000 (100246) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1251 (negotiate_kerberos_) thread 0xf80005ddca00 (100251) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1250 (negotiate_kerberos_) thread 0xf800651e2a00 (100241) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1251 (negotiate_kerberos_) thread 0xf80005ddca00 (100251) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1250 (negotiate_kerberos_) thread 0xf800651e2a00 (100241) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1247 (sqtop) thread 0xf80065650a00 (100259) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1184 (systat) thread 0xf80065613a00 (100255) >> shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >> /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 >> Process 1042 (negotiate_kerberos_) thread
Re: 'show alllocks' of completely locked machine [Was: Re: Complete IO lockup, state "ufs" from userland, debuging help wanted]
hiren panchasara wrote: [sorry, I can't be bothered manually indenting it all, so just see my comment at the end] On 03/06/17 at 08:56P, Harry Schmalzbauer wrote: > Bez?glich Harry Schmalzbauer's Nachricht vom 05.03.2017 22:59 (localtime): > > Hello, > > > > I can easily lock up FreeBSD stable/11 from userland. Not that I want to... > > I'm running squid, which starts an authentication helper > > "*negotiate_kerberos_auth*", which seems to be the culprit. > > Completely all IO is blocked, there's no way to get anything from any > > filesystem. > > All non IO-requesting processes(threads) run well, including sshd and > > shells. > > There's no load (neither cpu nor io) just any process requesting io > > stucks in state "ufs" > > > > Can anyone help me finding out what's going wrong? > > Serial console is available. > > Dear hackers, > > I managed to get into DDB, but I'm lost from there? > > What information could be usefull to find out the cause of this complete > lockup? > > I'd need someone who could guide me through ? I'd pay for a debuging > lesson! (quiet constrained budget though) > > This happens when the machine got stuck: > > intr_event_handle() at intr_event_handle+0x9c/frame 0xfe0093dcb7d0 > intr_execute_handlers() at intr_execute_handlers+0x48/frame > 0xfe0093dcb800 > lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfe0093dcb840 > Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfe0093dcb840 > --- interrupt, rip = 0x807b9bd6, rsp = 0xfe0093dcb910, rbp = > 0xfe0093dcb910 --- > acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfe0093dcb910 > acpi_cpu_idle() at acpi_cpu_idle+0x2ea/frame 0xfe0093dcb960 > cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfe0093dcb980 > cpu_idle() at cpu_idle+0x8f/frame 0xfe0093dcb9a0 > sched_idletd() at sched_idletd+0x436/frame 0xfe0093dcba70 > fork_exit() at fork_exit+0x84/frame 0xfe0093dcbab0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfe0093dcbab0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > > db> show alllocks > Process 1259 (negotiate_kerberos_) thread 0xf80005ddea00 (100096) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1258 (negotiate_kerberos_) thread 0xf80005ddc500 (100252) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1257 (negotiate_kerberos_) thread 0xf80005ddda00 (100247) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1256 (negotiate_kerberos_) thread 0xf80065612500 (100261) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1255 (negotiate_kerberos_) thread 0xf80065612a00 (100260) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1254 (negotiate_kerberos_) thread 0xf80065613000 (100257) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1253 (negotiate_kerberos_) thread 0xf80065614000 (100254) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1252 (negotiate_kerberos_) thread 0xf800651e1000 (100246) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xf80005ddca00 (100251) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xf800651e2a00 (100241) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xf80005ddca00 (100251) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xf800651e2a00 (100241) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1247 (sqtop) thread 0xf80065650a00 (100259) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1184 (systat) thread 0xf80065613a00 (100255) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1042 (negotiate_kerberos_) thread 0xf800651e2500 (100242) > shared
Re: 'show alllocks' of completely locked machine [Was: Re: Complete IO lockup, state "ufs" from userland, debuging help wanted]
On 03/06/17 at 08:56P, Harry Schmalzbauer wrote: > Bez?glich Harry Schmalzbauer's Nachricht vom 05.03.2017 22:59 (localtime): > > Hello, > > > > I can easily lock up FreeBSD stable/11 from userland. Not that I want to... > > I'm running squid, which starts an authentication helper > > "*negotiate_kerberos_auth*", which seems to be the culprit. > > Completely all IO is blocked, there's no way to get anything from any > > filesystem. > > All non IO-requesting processes(threads) run well, including sshd and > > shells. > > There's no load (neither cpu nor io) just any process requesting io > > stucks in state "ufs" > > > > Can anyone help me finding out what's going wrong? > > Serial console is available. > > Dear hackers, > > I managed to get into DDB, but I'm lost from there? > > What information could be usefull to find out the cause of this complete > lockup? > > I'd need someone who could guide me through ? I'd pay for a debuging > lesson! (quiet constrained budget though) > > This happens when the machine got stuck: > > intr_event_handle() at intr_event_handle+0x9c/frame 0xfe0093dcb7d0 > intr_execute_handlers() at intr_execute_handlers+0x48/frame > 0xfe0093dcb800 > lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfe0093dcb840 > Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfe0093dcb840 > --- interrupt, rip = 0x807b9bd6, rsp = 0xfe0093dcb910, rbp = > 0xfe0093dcb910 --- > acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfe0093dcb910 > acpi_cpu_idle() at acpi_cpu_idle+0x2ea/frame 0xfe0093dcb960 > cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfe0093dcb980 > cpu_idle() at cpu_idle+0x8f/frame 0xfe0093dcb9a0 > sched_idletd() at sched_idletd+0x436/frame 0xfe0093dcba70 > fork_exit() at fork_exit+0x84/frame 0xfe0093dcbab0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfe0093dcbab0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > > db> show alllocks > Process 1259 (negotiate_kerberos_) thread 0xf80005ddea00 (100096) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1258 (negotiate_kerberos_) thread 0xf80005ddc500 (100252) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1257 (negotiate_kerberos_) thread 0xf80005ddda00 (100247) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1256 (negotiate_kerberos_) thread 0xf80065612500 (100261) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1255 (negotiate_kerberos_) thread 0xf80065612a00 (100260) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1254 (negotiate_kerberos_) thread 0xf80065613000 (100257) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1253 (negotiate_kerberos_) thread 0xf80065614000 (100254) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1252 (negotiate_kerberos_) thread 0xf800651e1000 (100246) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xf80005ddca00 (100251) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xf800651e2a00 (100241) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1251 (negotiate_kerberos_) thread 0xf80005ddca00 (100251) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1250 (negotiate_kerberos_) thread 0xf800651e2a00 (100241) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1247 (sqtop) thread 0xf80065650a00 (100259) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1184 (systat) thread 0xf80065613a00 (100255) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ > /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/vfs_lookup.c:611 > Process 1042 (negotiate_kerberos_) thread 0xf800651e2500 (100242) > shared lockmgr ufs (ufs) r = 0 (0xf8000523d5f0) locked @ >
