Re: 7.1 hangs, shutdown terminated
Johan Hendriks írta: If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash shopzeus# sysctl -a | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2095818 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: All right. It is close to it. Which one should I increase? I put this into /etc/sysctl.conf: vfs.ufs.dirhash_maxmem=8228608 Would it be scufficient? Thanks, Laszlo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 7.1 hangs, shutdown terminated
Johan Hendriks írta: If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash shopzeus# sysctl -a | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2095818 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: All right. It is close to it. Which one should I increase? I put this into /etc/sysctl.conf: vfs.ufs.dirhash_maxmem=8228608 Would it be scufficient? Thanks, Laszlo vfs.ufs.dirhash_maxmem= is the value to adjust. Mine is at 12582912. this happens on Mailservers 'which has a lot of directorys for the mail. The value depends on how many directory's there are on your system. Keep track on the value and when it hits the limit again increase the limit until it stays under the limit. It can take a while to reach the limit again. Regards, Johan Hendriks Double L Automatisering No virus found in this outgoing message. Checked by AVG - http://www.avg.com Version: 8.0.173 / Virus Database: 270.8.0/1717 - Release Date: 9-10-2008 16:56 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.1 hangs, shutdown terminated
On Fri, Oct 10, 2008 at 10:40:01AM +0200, Laszlo Nagy wrote: Hi, A computer hangs every day in the morning at a specific time, between 8 AM and 9 AM. We can ping it. Apparently the console works, also gdm works on it, but we are not able to login at all. ssh accepts connections, but the authentication does not continue (e.g. ssh client waits for the server forever...) I even cannot login on the console as root because it accepts the user name, but does not ask for the password! Pressing Ctrl+Alt+Del on the console waits for about one or two minutes, then I see this on the screen: http://www.imghype.com/viewer.php?imgdata=9d95ee9d1fstrange_shutdown.jpg Here is /var/log/messages just before the crash: Oct 10 01:52:47 shopzeus postgres[81114]: [5-1] WARNING: nonstandard use of escape in a string literal at character 193 Oct 10 01:52:47 shopzeus postgres[81114]: [5-2] HINT: Use the escape string syntax for escapes, e.g., E'\r\n'. Oct 10 01:57:11 shopzeus postgres[84132]: [5-1] WARNING: nonstandard use of escape in a string literal at character 188 Oct 10 01:57:11 shopzeus postgres[84132]: [5-2] HINT: Use the escape string syntax for escapes, e.g., E'\r\n'. Oct 10 02:00:01 shopzeus postfix/postfix-script[86167]: fatal: the Postfix mail system is already running Oct 10 02:30:00 shopzeus postfix/postfix-script[7240]: fatal: the Postfix mail system is already running Oct 10 03:00:00 shopzeus postfix/postfix-script[27437]: fatal: the Postfix mail system is already running Oct 10 04:07:54 shopzeus rc.shutdown: 30 second watchdog timeout expired. Shutdown terminated. Oct 10 04:09:16 shopzeus postgres[30455]: [5-1] FATAL: terminating connection due to administrator command Oct 10 04:09:17 shopzeus syslogd: exiting on signal 15 Oct 10 04:11:31 shopzeus syslogd: kernel boot file is /boot/kernel/kernel Oct 10 04:11:31 shopzeus kernel: Copyright (c) 1992-2008 The FreeBSD Project. Oct 10 04:11:31 shopzeus kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 After rebooting the machine, nothing happens until the next day. Here are some possible problems I can think of: #1. We are using gjournal. It might be that the journal size is too small. Although I do not think this is the case, because we have 40GB journal space for each journaled partition below (except for /home, it has 10GB only, but /home is rarely used) Filesystem 1G-blocks Used Avail Capacity Mounted on /dev/da0s1a 91 714%/ devfs 00 0 100%/dev /dev/da0s1f.journal 140 12 117 9%/home /dev/da0s2d.journal 106889 8%/pgdata0 /dev/da0s1d29026 0%/tmp /dev/da0s2e.journal 585 74 46414%/usr /dev/da0s1e.journal 145 17 11613%/var /dev/da1s1d.journal 4160 383 0%/data Is it possible that gjournal is hanging up the machine? #2. Yesterday when I logged in in the morning, I saw a process running under root, it was something like find / -sx ... and then something. I don't remember but it was scanning the whole filesystem. It was using 100% cpu and 100% disk I/O. I wonder if that might be freezing the computer. I do not know how to disable this maintenance process but I should. After killing this process, the system worked fine. (We have zillions of files on the disks, running find / ... is a bad idea.) This could be a periodic job (since you said this happens daily) which runs early in the morning (2-3am?) and for some reason isn't finishing in a timely manner. You haven't provided any actual ps -auxwww data, so we can't easily discern if it's a periodic job or something amiss on your system (for all we know the system could be compromised). I'm also curious what controller your SCSI disks are attached to. Can you provide that information? dmesg would be useful. I remember hearing some reports about 3Ware controllers locking up due to firmware problems which were later fixed via a f/w upgrade. #3. In the screenshot above, you can see that the IMAP server dovecot was terminated on signal 11. Can it be the problem? I can't believe that dovecot could freeze the whole system. #4. Hardware error. I don't think this is the case since the computer freezes at the same time, every day, so it is more likely a software problem. My vote is on a hardware problem. The watchdog timeout you see indicates a portion of the system is locking up hard. The sig 11 would indicate a sudden segfault, which if unexpected, often indicates bad memory or motherboard. I would recommend you start down the hardware path. Replace the RAM and the mainboard, and see what happens. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | |
Re: 7.1 hangs, shutdown terminated
On Fri, Oct 10, 2008 at 11:13:00AM +0200, Laszlo Nagy wrote: Johan Hendriks írta: If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash shopzeus# sysctl -a | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2095818 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: All right. It is close to it. Which one should I increase? I put this into /etc/sysctl.conf: vfs.ufs.dirhash_maxmem=8228608 Would it be scufficient? We don't know, and can't tell you. You'll have to monitor vfs.ufs.dirhash_mem occasionally to see if you start to reach vfs.ufs.dirhash_maxmem. I have a tendency to use vfs.ufs.dirhash_maxmem=16777216, which is 16384*1024 (16MBytes). I'm not fully confident this is what's causing your problem, but it's definitely a recommendation by Johan. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.1 hangs, shutdown terminated
This could be a periodic job (since you said this happens daily) which runs early in the morning (2-3am?) and for some reason isn't finishing in a timely manner. You haven't provided any actual ps -auxwww data, so we can't easily discern if it's a periodic job or something amiss on your system (for all we know the system could be compromised). I wanted to, but since I could not log in... I'm also curious what controller your SCSI disks are attached to. Can you provide that information? ARECA 1680 ix 12 dmesg would be useful. Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #3: Mon Oct 6 07:50:31 EDT 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SHOPZEUS module_register: module g_journal already exists! Module g_journal failed to register: 17 Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2508.72-MHz K8-class CPU) Origin = GenuineIntel Id = 0x10676 Stepping = 6 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0xce3bdSSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,b19 AMD Features=0x20100800SYSCALL,NX,LM AMD Features2=0x1LAHF Cores per package: 4 usable memory = 8571047936 (8173 MB) avail memory = 8260050944 (7877 MB) ACPI APIC Table: INTEL S5000PSL FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 Version 2.0 irqs 0-23 on motherboard ioapic1 Version 2.0 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: INTEL S5000PSL on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a (3) failed Timecounter ACPI-fast frequency 3579545 Hz quality 1000 acpi_timer0: 24-bit timer at 3.579545MHz port 0x408-0x40b on acpi0 acpi_hpet0: High Precision Event Timer iomem 0xfed0-0xfed003ff on acpi0 Timecounter HPET frequency 14318180 Hz quality 900 acpi_button0: Sleep Button on acpi0 acpi_button1: Power Button on acpi0 pcib0: ACPI Host-PCI bridge port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 pcib1: ACPI PCI-PCI bridge at device 2.0 on pci0 pci1: ACPI PCI bus on pcib1 pcib2: ACPI PCI-PCI bridge irq 16 at device 0.0 on pci1 pci2: ACPI PCI bus on pcib2 pcib3: ACPI PCI-PCI bridge irq 16 at device 0.0 on pci2 pci3: ACPI PCI bus on pcib3 pcib4: ACPI PCI-PCI bridge irq 18 at device 2.0 on pci2 pci4: ACPI PCI bus on pcib4 em0: Intel(R) PRO/1000 Network Connection 6.9.5 port 0x2020-0x203f mem 0xb882-0xb883,0xb840-0xb87f irq 18 at device 0.0 on pci4 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:15:17:81:99:d0 em1: Intel(R) PRO/1000 Network Connection 6.9.5 port 0x2000-0x201f mem 0xb880-0xb881,0xb800-0xb83f irq 19 at device 0.1 on pci4 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:15:17:81:99:d1 pcib5: ACPI PCI-PCI bridge at device 0.3 on pci1 pci5: ACPI PCI bus on pcib5 pcib6: PCI-PCI bridge at device 3.0 on pci0 pci6: PCI bus on pcib6 pcib7: ACPI PCI-PCI bridge at device 4.0 on pci0 pci7: ACPI PCI bus on pcib7 pcib8: ACPI PCI-PCI bridge at device 5.0 on pci0 pci8: ACPI PCI bus on pcib8 pcib9: ACPI PCI-PCI bridge at device 6.0 on pci0 pci9: ACPI PCI bus on pcib9 pcib10: PCI-PCI bridge at device 7.0 on pci0 pci10: PCI bus on pcib10 pci0: base peripheral at device 8.0 (no driver attached) pcib11: ACPI PCI-PCI bridge irq 16 at device 28.0 on pci0 pci11: ACPI PCI bus on pcib11 arcmsr0: Areca SAS Host Adapter RAID Controller (RAID6 capable) mem 0xb8b0-0xb8b01fff irq 16 at device 0.0 on pci11 ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.45 2008-04-29 arcmsr0: [ITHREAD] uhci0: Intel 631XESB/632XESB/3100 USB controller USB-1 port 0x3080-0x309f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: Intel 631XESB/632XESB/3100 USB controller USB-1 on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: Intel 631XESB/632XESB/3100 USB controller USB-2 port 0x3060-0x307f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: Intel 631XESB/632XESB/3100 USB controller USB-2 on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: Intel 631XESB/632XESB/3100 USB controller USB-3 port 0x3040-0x305f irq 23 at
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
On Fri, Oct 10, 2008 at 11:43:39AM +0200, Laszlo Nagy wrote: If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash shopzeus# sysctl -a | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2095818 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: All right. It is close to it. Which one should I increase? I put this into /etc/sysctl.conf: vfs.ufs.dirhash_maxmem=8228608 Would it be scufficient? We don't know, and can't tell you. You'll have to monitor vfs.ufs.dirhash_mem occasionally to see if you start to reach vfs.ufs.dirhash_maxmem. I have a tendency to use vfs.ufs.dirhash_maxmem=16777216, which is 16384*1024 (16MBytes). I'm not fully confident this is what's causing your problem, but it's definitely a recommendation by Johan. Thank you very much! Probably you are right. Our users use shared IMAP folders and sometimes they keep ten thousands of messages in one folder. I have increased dirhash_maxmem to 64MB and see what happens. Unfortunately, I cannot play with the hardware because it is in a server park, and it must be up 99.99% on workdays. I hope dirhash will solve the problem. I'm setting this to [SOLVED] and come back if it happens again. (Maybe on monday?) By the way, there is nothing in /etc/periodic that would execute find / -sx. Can somebody explain what is this for, and why it was started by root? Is it being used instead for enumerating files in a directory, when dir hash is full? Firstly, I see a periodic(8) job that DOES use find -sx, which means your attempt to track it down was faulty, and your syntax should have been find -sx / not find / -sx. See here: /etc/periodic/security/100.chksetuid: find -sx $MP /dev/null -type f \ $MP == mountpoint, e.g. /, /var, or any other mounted filesystem. So, what you saw was the periodic check looking for setuid-root binaries. Secondly, the kernel does not spawn userland processes like find(1). Thirdly, dirmem and dirmem_max are *pure* kernel things. What they do is control the amount of memory used for directory structure caching; rather than continually hit the disk every time and spend all that time handling directory contents, the kernel can cache previously-fetched contents in memory. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
On Fri, Oct 10, 2008 at 01:07:43PM +0200, Laszlo Nagy wrote: Firstly, I see a periodic(8) job that DOES use find -sx, which means your attempt to track it down was faulty, and your syntax should have been find -sx / not find / -sx. See here: /etc/periodic/security/100.chksetuid: find -sx $MP /dev/null -type f \ Thanks for clearing that out. :-) I did not remember what it was and failed to find it. I believe the reason you saw this process still running at 8-9 in the morning was because of the slowdown induced by lack of dirhash memory. The periodic job runs every day, usually between 0130 and 0200, so the process had been sitting there processing its heart out for 6-7 hours. Since you've tuned the dirhash stuff, I'm betting this periodic job will run much more quickly. $MP == mountpoint, e.g. /, /var, or any other mounted filesystem. So, what you saw was the periodic check looking for setuid-root binaries. Secondly, the kernel does not spawn userland processes like find(1). Thirdly, dirmem and dirmem_max are *pure* kernel things. What they do is control the amount of memory used for directory structure caching; rather than continually hit the disk every time and spend all that time handling directory contents, the kernel can cache previously-fetched contents in memory Now it stays this value constantly: vfs.ufs.dirhash_mem: 44306131 I think it is now caching everything. Thank you again, and sorry for the dumb questions. You asked absolutely *no* dumb questions, especially given the circumstances! Do not be ashamed, you did the right thing. :-) -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
Firstly, I see a periodic(8) job that DOES use find -sx, which means your attempt to track it down was faulty, and your syntax should have been find -sx / not find / -sx. See here: /etc/periodic/security/100.chksetuid: find -sx $MP /dev/null -type f \ Thanks for clearing that out. :-) I did not remember what it was and failed to find it. $MP == mountpoint, e.g. /, /var, or any other mounted filesystem. So, what you saw was the periodic check looking for setuid-root binaries. Secondly, the kernel does not spawn userland processes like find(1). Thirdly, dirmem and dirmem_max are *pure* kernel things. What they do is control the amount of memory used for directory structure caching; rather than continually hit the disk every time and spend all that time handling directory contents, the kernel can cache previously-fetched contents in memory Now it stays this value constantly: vfs.ufs.dirhash_mem: 44306131 I think it is now caching everything. Thank you again, and sorry for the dumb questions. Laszlo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: 7.1 hangs, shutdown terminated
Thank you very much! Probably you are right. Our users use shared IMAP folders and sometimes they keep ten thousands of messages in one folder. I have increased dirhash_maxmem to 64MB and see what happens. Unfortunately, I cannot play with the hardware because it is in a server park, and it must be up 99.99% on workdays. I hope dirhash will solve the problem. I'm setting this to [SOLVED] and come back if it happens again. (Maybe on monday?) By the way, there is nothing in /etc/periodic that would execute find / -sx. Can somebody explain what is this for, and why it was started by root? Is it being used instead for enumerating files in a directory, when dir hash is full? I'm starting to believe that this was the problem. Within an hour, I see this: shopzeus# sysctl vfs.ufs vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 33708867 vfs.ufs.dirhash_maxmem: 134217728 vfs.ufs.dirhash_minsize: 2560 Went up to 32MB! L ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 7.1 hangs, shutdown terminated
If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: Regards, Johan Hendriks Double -L Automatisering No virus found in this outgoing message. Checked by AVG - http://www.avg.com Version: 8.0.173 / Virus Database: 270.8.0/1717 - Release Date: 9-10-2008 16:56 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
[SOLVED] Re: 7.1 hangs, shutdown terminated
If find / -sx is running and is consuming all CPU, what is the value of vfs.ufs.dirhash_mem: # sysctl -a | grep dirhash shopzeus# sysctl -a | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2095818 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 Make sure vfs.ufs.dirhash_mem: is not close to vfs.ufs.dirhash_maxmem: All right. It is close to it. Which one should I increase? I put this into /etc/sysctl.conf: vfs.ufs.dirhash_maxmem=8228608 Would it be scufficient? We don't know, and can't tell you. You'll have to monitor vfs.ufs.dirhash_mem occasionally to see if you start to reach vfs.ufs.dirhash_maxmem. I have a tendency to use vfs.ufs.dirhash_maxmem=16777216, which is 16384*1024 (16MBytes). I'm not fully confident this is what's causing your problem, but it's definitely a recommendation by Johan. Thank you very much! Probably you are right. Our users use shared IMAP folders and sometimes they keep ten thousands of messages in one folder. I have increased dirhash_maxmem to 64MB and see what happens. Unfortunately, I cannot play with the hardware because it is in a server park, and it must be up 99.99% on workdays. I hope dirhash will solve the problem. I'm setting this to [SOLVED] and come back if it happens again. (Maybe on monday?) By the way, there is nothing in /etc/periodic that would execute find / -sx. Can somebody explain what is this for, and why it was started by root? Is it being used instead for enumerating files in a directory, when dir hash is full? Thanks, Laszo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
