Re: Panic after hung rsync, probably zfs related
Xin LI wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ben Stuyts wrote: | Hi, | | While doing an rsync from a zfs filesystem to an external usb hd (also | zfs), the rsync processes hung in zfs state. I could not kill these | processes, although the rest of the server seemingly continued to run | fine. The reboot command did not work. Next I tried a shutdown now | command. This caused a panic: Sound like you somehow run out of memory, there is an known issue with ZFS which causes livelock when there is memory pressure. Which rsync version are you using? With rsync 3.x the memory usage would drop drastically which would help to prevent this from happening. This isn't known to cause a double fault though. Unfortunately we probably need more information than was available in order to proceed. Kris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Panic after hung rsync, probably zfs related
On 13 mei 2008, at 21:53, Xin LI wrote: | While doing an rsync from a zfs filesystem to an external usb hd (also | zfs), the rsync processes hung in zfs state. I could not kill these | processes, although the rest of the server seemingly continued to run | fine. The reboot command did not work. Next I tried a shutdown now | command. This caused a panic: Sound like you somehow run out of memory, there is an known issue with ZFS which causes livelock when there is memory pressure. Which rsync version are you using? With rsync 3.x the memory usage would drop drastically which would help to prevent this from happening. I'm running rsync 3.0.0. Two weeks ago I already increased the memory in this machine from 4 to 8 GB, but I did not change any loader.conf variables. BTW, The wired memory goes all over the place in this machine: even when it's mostly idle it varies between 300 MB and 3.5 GB. That's why I added 4 more GB. Ben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Panic after hung rsync, probably zfs related
On Wed, May 14, 2008 at 10:39:36AM +0200, Ben Stuyts wrote: On 13 mei 2008, at 21:53, Xin LI wrote: | While doing an rsync from a zfs filesystem to an external usb hd (also | zfs), the rsync processes hung in zfs state. I could not kill these | processes, although the rest of the server seemingly continued to run | fine. The reboot command did not work. Next I tried a shutdown now | command. This caused a panic: Sound like you somehow run out of memory, there is an known issue with ZFS which causes livelock when there is memory pressure. Which rsync version are you using? With rsync 3.x the memory usage would drop drastically which would help to prevent this from happening. I'm running rsync 3.0.0. Two weeks ago I already increased the memory in this machine from 4 to 8 GB, but I did not change any loader.conf variables. BTW, The wired memory goes all over the place in this machine: even when it's mostly idle it varies between 300 MB and 3.5 GB. That's why I added 4 more GB. 3.5Gb wired ? Do you run amd64 ? Does wired memory drops lower after you change the load ? pgpXo266QLuqV.pgp Description: PGP signature
Re: Panic after hung rsync, probably zfs related
Kostik Belousov wrote: BTW, The wired memory goes all over the place in this machine: even when it's mostly idle it varies between 300 MB and 3.5 GB. That's why I added 4 more GB. 3.5Gb wired ? Do you run amd64 ? Does wired memory drops lower after you change the load ? Either that or some other 64bit Plattform (alpha, sparc64 etc.). Regards Chris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Panic after hung rsync, probably zfs related
On 14 mei 2008, at 10:50, Kostik Belousov wrote: BTW, The wired memory goes all over the place in this machine: even when it's mostly idle it varies between 300 MB and 3.5 GB. That's why I added 4 more GB. 3.5Gb wired ? Do you run amd64 ? Does wired memory drops lower after you change the load ? Yes, it was in my original message. Ben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Panic after hung rsync, probably zfs related
On 14 mei 2008, at 10:15, Kris Kennaway wrote: | While doing an rsync from a zfs filesystem to an external usb hd (also | zfs), the rsync processes hung in zfs state. I could not kill these | processes, although the rest of the server seemingly continued to run | fine. The reboot command did not work. Next I tried a shutdown now | command. This caused a panic: Sound like you somehow run out of memory, there is an known issue with ZFS which causes livelock when there is memory pressure. Which rsync version are you using? With rsync 3.x the memory usage would drop drastically which would help to prevent this from happening. This isn't known to cause a double fault though. Unfortunately we probably need more information than was available in order to proceed. Yes, I agree. But I don't know what to do in this sort of case where the machine is completely wedged. Ben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Panic after hung rsync, probably zfs related
On 14 mei 2008, at 10:50, Kostik Belousov wrote: 3.5Gb wired ? Do you run amd64 ? Yes. Does wired memory drops lower after you change the load ? Sorry, forgot to answer this in my previous msg. It is very unpredictable, and I have not found a pattern. It is a small business server, and both during non-office and regular office hours the wired memory varies between the numbers above. Ben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Panic after hung rsync, probably zfs related
Hi, While doing an rsync from a zfs filesystem to an external usb hd (also zfs), the rsync processes hung in zfs state. I could not kill these processes, although the rest of the server seemingly continued to run fine. The reboot command did not work. Next I tried a shutdown now command. This caused a panic: ... Stopping dhcpd. Shutting down local daemons:. Stopping named. Waiting for PIDS: 106830 second watchdog timeout expired. Shutdown terminated. Tue May 13 21:02:50 CEST 2008 May 13 21:02:50 auth.alert mars init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode panic: vrele: negative ref cnt cpuid = 6 Uptime: 8d3h8m21s Physical memory: 8178 MB Dumping 3679 MB: 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264 3248 atl trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fau =Fta tvailr tduoaulb laed dfraeuslst = r0ixp8 v f0axuflftf fcfofdfef 8 0=6 esbuap9e3r girssopr =r e0axdf fifnfsftfrfufcftbi5o4n8,f fp0a er bnpo t= p0rxe1s0e0n0t xipnusitdr u=c t0i;o na ppioci nitde r= =0 00 8:0x8 stack pointer = 0x10:0xf8c50a50 frame pointer = 0x10:0xfb54c450 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 44 (irq19: uhci1+) trap number = 12 Yes, it printed all that gibberish. (double fault I think it says?) At this point, the server completely hung, and I could not do a bt unfortunately. I had to reset the server. After reboot, savecore did not find a kernel dump. I ran the rsync again and could not reproduce the problem. Although I've had zfs related problems before, this one was new to me. Any idea what happened here? Other info on this machine follows below. Thanks, Ben [mars:~]133: uname -a FreeBSD mars.altus-escon.com 7.0-STABLE FreeBSD 7.0-STABLE #2: Mon Apr 21 08:45:56 CEST 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/ sys/MARS amd64 [mars:~]134: cat /boot/loader.conf console=comconsole zfs_load=YES vfs.root.mountfrom=zfs:tank vfs.zfs.prefetch_disable=1 vm.kmem_size=1536M vm.kmem_size_max=1536M vfs.zfs.arc_max=768M [mars:~]135: dmesg Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #2: Mon Apr 21 08:45:56 CEST 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/MARS Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz (2000.08-MHz K8- class CPU) Origin = GenuineIntel Id = 0x6f7 Stepping = 7 Features = 0xbfebfbff FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP ,MTRR ,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2 =0x4e33dSSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA AMD Features=0x20100800SYSCALL,NX,LM AMD Features2=0x1LAHF Cores per package: 4 usable memory = 8575598592 (8178 MB) avail memory = 8285937664 (7902 MB) ACPI APIC Table: PTLTD APIC FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 Version 2.0 irqs 0-23 on motherboard ioapic1 Version 2.0 irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: PTLTD RSDT on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter ACPI-fast frequency 3579545 Hz quality 1000 acpi_timer0: 24-bit timer at 3.579545MHz port 0x1008-0x100b on acpi0 cpu0: ACPI CPU on acpi0 p4tcc0: CPU Frequency Thermal Control on cpu0 cpu1: ACPI CPU on acpi0 p4tcc1: CPU Frequency Thermal Control on cpu1 cpu2: ACPI CPU on acpi0 p4tcc2: CPU Frequency Thermal Control on cpu2 cpu3: ACPI CPU on acpi0 p4tcc3: CPU Frequency Thermal Control on cpu3 cpu4: ACPI CPU on acpi0 p4tcc4: CPU Frequency Thermal Control on cpu4 cpu5: ACPI CPU on acpi0 p4tcc5: CPU Frequency Thermal Control on cpu5 cpu6: ACPI CPU on acpi0 p4tcc6: CPU Frequency Thermal Control on cpu6 cpu7: ACPI CPU on acpi0 p4tcc7: CPU Frequency Thermal Control on cpu7 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 pcib1: ACPI PCI-PCI bridge at device 2.0 on pci0 pci1: ACPI PCI bus on pcib1 pcib2: ACPI PCI-PCI bridge irq 16 at device 0.0 on pci1 pci2: ACPI PCI bus on pcib2 pcib3: ACPI PCI-PCI bridge irq 16 at device 0.0 on pci2 pci3: ACPI PCI bus on pcib3 pcib4: ACPI PCI-PCI bridge at device 0.0 on pci3 pci4:
Re: Panic after hung rsync, probably zfs related
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ben Stuyts wrote: | Hi, | | While doing an rsync from a zfs filesystem to an external usb hd (also | zfs), the rsync processes hung in zfs state. I could not kill these | processes, although the rest of the server seemingly continued to run | fine. The reboot command did not work. Next I tried a shutdown now | command. This caused a panic: Sound like you somehow run out of memory, there is an known issue with ZFS which causes livelock when there is memory pressure. Which rsync version are you using? With rsync 3.x the memory usage would drop drastically which would help to prevent this from happening. Cheers, - -- ** Help China's quake relief at http://www.redcross.org.cn/ | Xin LI [EMAIL PROTECTED]http://www.delphij.net/ FreeBSD - The Power to Serve! -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgp8bcACgkQi+vbBBjt66DQQgCfSObZqsQNCteT9SsjDTIqAa2E xfgAnAs9reRKfjaJdS9RAco0cnD7JZp0 =J4+1 -END PGP SIGNATURE- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
