Too many files crashing services
I posted this issue yesterday sometime and have been trying to track down the problem with the help of the Postfix list. There, Weitse has suggested tracking down the culprit using lsof, which I know nothing about. I tried 'lsof|less', but that doesn't even show anything via SSH, related to the mail system. Perhaps I need to be at the console to run this? I tried 'lsof|grep postfix' and get pages of information to open files, but what I'm I looking for? The problem is these errors below keep showing up in the messages log until it starts to crash services, I stop and start Postfix and it goes away, but only for several minutes. I have to stop and start Postfix every 15 minutes to keep the issue at bay. Any ideas what I should be looking for with lsof? esmtp# tail /var/log/messages Dec 4 17:54:33 esmtp master[91678]: service pop3 pid 15695 in READY state: terminated abnormally Dec 4 17:54:33 esmtp master[91678]: service pop3 pid 15696 in READY state: terminated abnormally Dec 4 17:54:33 esmtp master[91678]: service pop3 pid 15697 in READY state: terminated abnormally Dec 4 17:54:34 esmtp kernel: kern.maxfiles limit exceeded by uid 88, please see tuning(7). Dec 4 17:54:39 esmtp last message repeated 5 times Dec 4 17:54:43 esmtp master[91678]: service pop3 pid 15698 in READY state: terminated abnormally I also get these messages: Dec 4 18:02:57 esmtp postfix/smtpd[15794]: fatal: socket: Too many open files Dec 4 18:05:13 esmtp postfix/smtpd[15760]: fatal: socket: Too many open files -- Robert ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Too many files crashing services
Robert Fitzpatrick wrote: [...] Dec 4 17:54:34 esmtp kernel: kern.maxfiles limit exceeded by uid 88, please see tuning(7). man 7 tuning Dec 4 17:54:39 esmtp last message repeated 5 times [...] Dec 4 18:02:57 esmtp postfix/smtpd[15794]: fatal: socket: Too many open files Dec 4 18:05:13 esmtp postfix/smtpd[15760]: fatal: socket: Too many open files ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Too many files crashing services
Norberto Meijome wrote: Robert Fitzpatrick wrote: [...] Dec 4 17:54:34 esmtp kernel: kern.maxfiles limit exceeded by uid 88, please see tuning(7). man 7 tuning So, you think this is what is contributing to my problem? Yeah, I have never ran tunefs on the RAID 5 system running FreeBSD 5.2.1. Is this what you are suggesting? Below is my current partitioning, how risky is running tunefs. I'll have to go to the server location of course and want to be sure of what to do and should I wait until a Saturday in case something were to go wrong. From what I've read, I should run 'tunefs -n enable /filesystem' for soft updates? esmtp# df -la Filesystem 1K-blocks UsedAvail Capacity Mounted on /dev/da0s1a 1982798 579620 124455632%/ devfs 110 100%/dev /dev/da0s1g 17393582 683262 15318834 4%/home /dev/da0s1f988398 161668 74766018%/tmp /dev/da0s1d 19834638 4947542 1330032627%/usr /dev/da0s1e 9914318 8679632 44154295%/var /dev/da0s1h 95950386 88004097 698678693%/data devfs 110 100%/var/chroot/named/dev -- Robert ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Too many files crashing services
Robert Fitzpatrick wrote: Norberto Meijome wrote: Robert Fitzpatrick wrote: [...] Dec 4 17:54:34 esmtp kernel: kern.maxfiles limit exceeded by uid 88, please see tuning(7). man 7 tuning So, you think this is what is contributing to my problem? Yeah, I have never ran tunefs on the RAID 5 system running FreeBSD 5.2.1. Is this what you are suggesting? Below is my current partitioning, how risky is running tunefs. I'll have to go to the server location of course and want to be sure of what to do and should I wait until a Saturday in case something were to go wrong. From what I've read, I should run 'tunefs -n enable /filesystem' for soft updates? No, you are running out of maxfiles, as per "kern.maxfiles limit exceeded by uid ..." . http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kernel-limits.html B ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Too many files crashing services
Norberto Meijome wrote: No, you are running out of maxfiles, as per "kern.maxfiles limit exceeded by uid ..." . errr... of course, i meant to say, you have more open files than allowed by kern.maxfiles. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Too many files crashing services
On Mon, 2005-12-05 at 16:16 +1100, Norberto Meijome wrote: > Norberto Meijome wrote: > > > > No, you are running out of maxfiles, as per "kern.maxfiles limit > > exceeded by uid ..." . > > errr... of course, i meant to say, you have more open files than allowed > by kern.maxfiles. Yeah, still struggling with this issue since upgrading lots of packages. Unless I stop and start Postfix every 10 minutes, the problem starts to build until too many files are open. Messages from lots of different UID's with too many files open, I am not sure how to pinpoint what services is causing the problem. I only assume Postfix since the problem only goes away when stopping and starting. With Weitse help on the Postfix list, I tracked down some errors in the maillog related to too many open files as well, which also points to Postfix. But Weitse had said it may not be Postfix doing it, waiting on his opinion now after sending him some more log info. If I am exceeding the kern.maxfiles, why now? This server has always performed the same mainly mail filtering and other light web and db tasks for over a couple of years. The mail traffic is has not changed. The upgrades were many including Perl, Postfix, amavisd-new, SpamAssassin, ClamAV, but all from recent versions with portupgrade and no errors. The biggest jump was amavisd-new from 2.2.x to 2.3.3. I ran 'perl-after-upgrade -f' and 'pkgdb -F' all good. Ideas? -- Robert ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Too many files crashing services
On Mon, 2005-12-05 at 16:16 +1100, Norberto Meijome wrote: > Norberto Meijome wrote: > > > > No, you are running out of maxfiles, as per "kern.maxfiles limit > > exceeded by uid ..." . > > errr... of course, i meant to say, you have more open files than allowed > by kern.maxfiles. Well, I just did some fsck /var and get a few INCORRECT BLOCK COUNT in Phase 1, lots of these in Phase 2: UNALLOCATED I=967619 OWNER=postfix MODE=100600 SIZE=353 MTIME=Dec 5 18:18 2005 FILE=/spool/postfix/defer/E/E60BEEC3C2 UNEXPECTED SOFT UPDATE INCONSISTENCY REMOVE? no And plenty of this in Phase 4: UNREF FILE I=22 OWNER=vscan MODE=100640 SIZE=80174 MTIME=Jun 20 11:30 2005 RECONNECT? no So on and so on...could these things cause my 'Too many open files' issues and reaching kern.maxfiles? And what are the risks of allowing all these issues to 'REMOVE' and 'RECONNECT' in single user mode? fsck /usr not as bad, but still: esmtp# fsck /usr ** /dev/da0s1d (NO WRITE) ** Last Mounted on /usr ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts LINK COUNT FILE I=2425895 OWNER=pgsql MODE=0 SIZE=0 MTIME=Dec 5 18:23 2005 COUNT 0 SHOULD BE -1 ADJUST? no ** Phase 5 - Check Cyl groups SUMMARY INFORMATION BAD SALVAGE? no ALLOCATED FILE 2425895 MARKED FREE ALLOCATED FRAG 9702124 MARKED FREE BLK(S) MISSING IN BIT MAPS SALVAGE? no 326108 files, 2473634 used, 7443685 free (128341 frags, 914418 blocks, 1.3% fragmentation) -- Robert ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
