Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
On Fri, Oct 10, 2008 at 01:07:43PM +0200, Laszlo Nagy wrote: > >> Firstly, I see a periodic(8) job that DOES use find -sx, which means >> your attempt to track it down was faulty, and your syntax should have >> been "find -sx /" not "find / -sx". See here: >> >> /etc/periodic/security/100.chksetuid: find -sx $MP /dev/null -type f \ >> > Thanks for clearing that out. :-) I did not remember what it was and > failed to find it. I believe the reason you saw this process still running at 8-9 in the morning was because of the slowdown induced by lack of dirhash memory. The periodic job runs every day, usually between 0130 and 0200, so the process had been sitting there processing its heart out for 6-7 hours. Since you've tuned the dirhash stuff, I'm betting this periodic job will run much more quickly. >> $MP == mountpoint, e.g. /, /var, or any other mounted filesystem. >> >> So, what you saw was the periodic check looking for setuid-root >> binaries. >> >> Secondly, the kernel does not spawn userland processes like find(1). >> >> Thirdly, dirmem and dirmem_max are *pure* kernel things. What they do >> is control the amount of memory used for directory structure caching; >> rather than continually hit the disk every time and spend all that time >> handling directory contents, the kernel can cache previously-fetched >> contents in memory > Now it stays this value constantly: > > vfs.ufs.dirhash_mem: 44306131 > > I think it is now caching everything. > > Thank you again, and sorry for the dumb questions. You asked absolutely *no* dumb questions, especially given the circumstances! Do not be ashamed, you did the right thing. :-) -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
Firstly, I see a periodic(8) job that DOES use find -sx, which means your attempt to track it down was faulty, and your syntax should have been "find -sx /" not "find / -sx". See here: /etc/periodic/security/100.chksetuid: find -sx $MP /dev/null -type f \ Thanks for clearing that out. :-) I did not remember what it was and failed to find it. $MP == mountpoint, e.g. /, /var, or any other mounted filesystem. So, what you saw was the periodic check looking for setuid-root binaries. Secondly, the kernel does not spawn userland processes like find(1). Thirdly, dirmem and dirmem_max are *pure* kernel things. What they do is control the amount of memory used for directory structure caching; rather than continually hit the disk every time and spend all that time handling directory contents, the kernel can cache previously-fetched contents in memory Now it stays this value constantly: vfs.ufs.dirhash_mem: 44306131 I think it is now caching everything. Thank you again, and sorry for the dumb questions. Laszlo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
On Fri, Oct 10, 2008 at 11:43:39AM +0200, Laszlo Nagy wrote: > If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash >>> shopzeus# sysctl -a | grep dirhash >>> vfs.ufs.dirhash_docheck: 0 >>> vfs.ufs.dirhash_mem: 2095818 >>> vfs.ufs.dirhash_maxmem: 2097152 >>> vfs.ufs.dirhash_minsize: 2560 >>> >>> Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: >>> All right. It is close to it. Which one should I increase? I put this >>> into /etc/sysctl.conf: >>> >>> >>> vfs.ufs.dirhash_maxmem=8228608 >>> >>> Would it be scufficient? >>> >> >> We don't know, and can't tell you. You'll have to monitor >> vfs.ufs.dirhash_mem occasionally to see if you start to reach >> vfs.ufs.dirhash_maxmem. >> >> I have a tendency to use vfs.ufs.dirhash_maxmem=16777216, which is >> 16384*1024 (16MBytes). >> >> I'm not fully confident this is what's causing your problem, but it's >> definitely a recommendation by Johan. >> > Thank you very much! Probably you are right. Our users use shared IMAP > folders and sometimes they keep ten thousands of messages in one folder. > I have increased dirhash_maxmem to 64MB and see what happens. > > Unfortunately, I cannot play with the hardware because it is in a server > park, and it must be up 99.99% on workdays. > > I hope dirhash will solve the problem. I'm setting this to [SOLVED] and > come back if it happens again. (Maybe on monday?) > > By the way, there is nothing in /etc/periodic that would execute "find / > -sx". Can somebody explain what is this for, and why it was started by > root? Is it being used instead for enumerating files in a directory, > when dir hash is full? Firstly, I see a periodic(8) job that DOES use find -sx, which means your attempt to track it down was faulty, and your syntax should have been "find -sx /" not "find / -sx". See here: /etc/periodic/security/100.chksetuid: find -sx $MP /dev/null -type f \ $MP == mountpoint, e.g. /, /var, or any other mounted filesystem. So, what you saw was the periodic check looking for setuid-root binaries. Secondly, the kernel does not spawn userland processes like find(1). Thirdly, dirmem and dirmem_max are *pure* kernel things. What they do is control the amount of memory used for directory structure caching; rather than continually hit the disk every time and spend all that time handling directory contents, the kernel can cache previously-fetched contents in memory. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
Thank you very much! Probably you are right. Our users use shared IMAP folders and sometimes they keep ten thousands of messages in one folder. I have increased dirhash_maxmem to 64MB and see what happens. Unfortunately, I cannot play with the hardware because it is in a server park, and it must be up 99.99% on workdays. I hope dirhash will solve the problem. I'm setting this to [SOLVED] and come back if it happens again. (Maybe on monday?) By the way, there is nothing in /etc/periodic that would execute "find / -sx". Can somebody explain what is this for, and why it was started by root? Is it being used instead for enumerating files in a directory, when dir hash is full? I'm starting to believe that this was the problem. Within an hour, I see this: shopzeus# sysctl vfs.ufs vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 33708867 vfs.ufs.dirhash_maxmem: 134217728 vfs.ufs.dirhash_minsize: 2560 Went up to 32MB! L ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
[SOLVED] Re: 7.1 hangs, shutdown terminated
If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash shopzeus# sysctl -a | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2095818 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: All right. It is close to it. Which one should I increase? I put this into /etc/sysctl.conf: vfs.ufs.dirhash_maxmem=8228608 Would it be scufficient? We don't know, and can't tell you. You'll have to monitor vfs.ufs.dirhash_mem occasionally to see if you start to reach vfs.ufs.dirhash_maxmem. I have a tendency to use vfs.ufs.dirhash_maxmem=16777216, which is 16384*1024 (16MBytes). I'm not fully confident this is what's causing your problem, but it's definitely a recommendation by Johan. Thank you very much! Probably you are right. Our users use shared IMAP folders and sometimes they keep ten thousands of messages in one folder. I have increased dirhash_maxmem to 64MB and see what happens. Unfortunately, I cannot play with the hardware because it is in a server park, and it must be up 99.99% on workdays. I hope dirhash will solve the problem. I'm setting this to [SOLVED] and come back if it happens again. (Maybe on monday?) By the way, there is nothing in /etc/periodic that would execute "find / -sx". Can somebody explain what is this for, and why it was started by root? Is it being used instead for enumerating files in a directory, when dir hash is full? Thanks, Laszo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
