[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #10 from Robert Clausecker --- (In reply to ota from comment #9) Yes, but I recall that I had another lockup after the disk replacement was done. What finally mitigated the problem was locking down the number of simultaneous FTP connections to an unreasonably low number (200) and disabling Apache 2.4. I think this reduced the load sufficiently to avoid the issue. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 o...@j.email.ne.jp changed: What|Removed |Added CC||o...@j.email.ne.jp --- Comment #9 from o...@j.email.ne.jp --- (In reply to Robert Clausecker from comment #7) By the way, were all problems happened during disk replacement? -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #8 from ch...@cretaforce.gr --- Today my backup script hang with chflags process (I chflags recursively a lot of files) in ufs state. I use FreeBSD 12-STABLE (kernel/userland from 9 August) and UFS SU+J. SSH was responsive but server was pingable. The only way was to hard reset the server. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #7 from Robert Clausecker --- Once processes start to lock up, the machine kicks me out of my SSH session when I run "pkg update". I can't log in again (neither from FTP, SSH, or console), but existing connections continue to work. I have to hard reboot the machine then. Weird. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #6 from Robert Clausecker --- The situation has reappeared with a bunch of ftpd instances being stuck in ufs and zfs wait channels. I'll leave the stuck box up for an hour or so in case you need further information. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #5 from Robert Clausecker --- Created attachment 206820 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=206820=edit vmstat -z -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #4 from Robert Clausecker --- Created attachment 206819 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=206819=edit ps -auxc -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #3 from Robert Clausecker --- Created attachment 206816 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=206816=edit pkg version -v -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 --- Comment #2 from Robert Clausecker --- Created attachment 206815 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=206815=edit /var/run/dmesg.boot -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 Kubilay Kocak changed: What|Removed |Added CC||ko...@freebsd.org Keywords||needs-qa Status|New |Open --- Comment #1 from Kubilay Kocak --- Thank you for the repoyrt Robert Can you provide the exact system information (uname -a), and include - pkg version -v output (as an attachment) - /var/run/dmesg.boot output (as an attachment) And when the symptoms are observable: - ps (-aux at least) output (as an attachment ) - vmstat -z output (as an attachment) -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 240047] more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240047 Bug ID: 240047 Summary: more and more processes get stuck waiting for ufs and zfs until system is rendered inaccessible Product: Base System Version: 12.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: b...@freebsd.org Reporter: f...@fuz.su I'm on a conference running an open FTP server. Files are served by FTP via ftpd(8), NFS via nfsd(8), and HTTP via Apache 2.4. The server has its root on UFS and remaining files spread over three ZFS pools, one currently replacing a (working) disk: $ zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAGCAP DEDUP HEALTH ALTROOT disk12 18.1T 14.0T 4.16T- - 3%77% 1.00x ONLINE - da3 9.06T 6.98T 2.08T- - 3%77% diskid/DISK-7JG9E40C%20%20%20%20%20%20%20%20%20%20%20%20 9.06T 6.98T 2.08T - - 3%77% cache - - - - - - ada0p2 170G 3.98G 166G- - 0% 2% disk34 18.1T 14.8T 3.33T- - 4%81% 1.00x ONLINE - da2 9.06T 7.39T 1.67T- - 4%81% da1 9.06T 7.41T 1.66T- - 4%81% cache - - - - - - ada0p5 170G 5.14G 165G- - 0% 3% disk56 18.1T 14.0T 4.15T- - 1%77% 1.00x ONLINE - replacing 9.06T 6.97T 2.10T- - 1%76% da0 - - -- - - - da4 - - -- - - - diskid/DISK-7PGVBGZC%20%20%20%20%20%20%20%20%20%20%20%20 9.06T 7.01T 2.06T - - 1%77% cache - - - - - - ada0p6 170G 6.03G 164G- - 0% 3% $ df -h Filesystem SizeUsed Avail Capacity Mounted on /dev/ada0p4375G 68G278G20%/ devfs 1.0K1.0K 0B 100%/dev tmpfs 33G 76K 33G 0%/var/run tmpfs 33G4.0K 33G 0%/tmp tmpfs 33G156K 33G 0%/var/log fdescfs1.0K1.0K 0B 100%/dev/fd procfs 4.0K4.0K 0B 100%/proc disk12 18T 14T3.6T80%/disk12 disk34 17T 14T2.8T83%/disk34 disk56 18T 14T3.6T80%/disk56 disk34/zeug3.6T864G2.8T23%/usr/home/fuz/zeug :/disk12 18T 14T3.6T80%/export :/disk34 35T 32T2.8T92%/export :/disk56 52T 49T3.6T93%/export Files are served over a 10 GBe connection with an average bandwith of around 200 MB/s, the limit seems to be in the number of IOP/s: $ zpool iostat capacity operationsbandwidth poolalloc free read write read write -- - - - - - - disk12 14.0T 4.16T254 0 34.8M 6.16K disk34 14.8T 3.33T261 29 35.0M 1.20M disk56 14.0T 4.15T882 29 118M 191K -- - - - - - - RAM is about half used and nothing seems to indicate any resource exhaustion. $ vmstat procs memory pagedisks faults cpu r b w avm fre flt re pi pofr sr ad0 da0 insycs us sy id 0 0 0 1.0T 666M 451 1197 436 0 64834 14532 0 0 28631 18084 93822 0 17 83 The only sysctl set is kern.racct.enable=1 After a while, more and more httpd and ftpd processes get stuck in an ufs or zfs wait state. They cannot be killed. I have since rebooted the server a bunch of times and the problem keeps appearing. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
