Re: Multiple USB drives stability question
I was checking sysctl and noticed skipped_byes, alloc_failures,low_mem all increasing on geom.journal. $sysctl -a | grep geom kern.geom.collectstats: 1 kern.geom.debugflags: 0 kern.geom.label.debug: 0 kern.geom.mirror.sync_requests: 2 kern.geom.mirror.disconnect_on_failure: 1 kern.geom.mirror.idletime: 5 kern.geom.mirror.timeout: 4 kern.geom.mirror.debug: 0 kern.geom.journal.stats.low_mem: 380 kern.geom.journal.stats.journal_full: 0 kern.geom.journal.stats.wait_for_copy: 25 kern.geom.journal.stats.switches: 834 kern.geom.journal.stats.combined_ios: 5612 kern.geom.journal.stats.skipped_bytes: 34684928 kern.geom.journal.cache.alloc_failures: 14726 kern.geom.journal.cache.misses: 13894 kern.geom.journal.cache.switch: 90 kern.geom.journal.cache.divisor: 2 kern.geom.journal.cache.limit: 167772160 kern.geom.journal.cache.used: 79546368 kern.geom.journal.optimize: 1 kern.geom.journal.record_entries: 20 kern.geom.journal.parallel_copies: 16 kern.geom.journal.accept_immediately: 64 kern.geom.journal.parallel_flushes: 16 kern.geom.journal.force_switch: 70 kern.geom.journal.switch_time: 10 kern.geom.journal.debug: 0 kern.geom.virstor.component_watermark: 1 kern.geom.virstor.chunk_watermark: 100 kern.geom.virstor.debug: 2 debug.sizeof.g_geom: 68 $sysctl -a | grep geom kern.geom.collectstats: 1 kern.geom.debugflags: 0 kern.geom.label.debug: 0 kern.geom.mirror.sync_requests: 2 kern.geom.mirror.disconnect_on_failure: 1 kern.geom.mirror.idletime: 5 kern.geom.mirror.timeout: 4 kern.geom.mirror.debug: 0 kern.geom.journal.stats.low_mem: 389 kern.geom.journal.stats.journal_full: 0 kern.geom.journal.stats.wait_for_copy: 28 kern.geom.journal.stats.switches: 838 kern.geom.journal.stats.combined_ios: 5622 kern.geom.journal.stats.skipped_bytes: 35667968 kern.geom.journal.cache.alloc_failures: 15016 kern.geom.journal.cache.misses: 15079 kern.geom.journal.cache.switch: 90 kern.geom.journal.cache.divisor: 2 kern.geom.journal.cache.limit: 167772160 kern.geom.journal.cache.used: 73140224 kern.geom.journal.optimize: 1 kern.geom.journal.record_entries: 20 kern.geom.journal.parallel_copies: 16 kern.geom.journal.accept_immediately: 64 kern.geom.journal.parallel_flushes: 16 kern.geom.journal.force_switch: 70 kern.geom.journal.switch_time: 10 kern.geom.journal.debug: 0 kern.geom.virstor.component_watermark: 1 kern.geom.virstor.chunk_watermark: 100 kern.geom.virstor.debug: 2 debug.sizeof.g_geom: 68 --- On Sat, 8/15/09, Jeff Richards wrote: From: Jeff Richards Subject: Re: Multiple USB drives stability question To: freebsd-stable@freebsd.org Date: Saturday, August 15, 2009, 10:50 AM I am now trying to rsync large files from the 320GB gmirror+gjournal device to the 2nd 1TB gmirror+gjournal device. Using gstat I see the 320GB device active all the time while the 1TB device loads in spurts. There will be periods of multiple seconds where the target providers are completely idle while the source providers are still reporting 100% active. Is there any tuning I should be investigating for these GEOM classes? --- On Fri, 8/14/09, Jeff Richards wrote: From: Jeff Richards Subject: Re: Multiple USB drives stability question To: freebsd-stable@freebsd.org Date: Friday, August 14, 2009, 11:04 PM I just tested my 2nd 1TB gmirror device on another system with FBSD 7.2. I was getting full throughput on the drive and no lockup using bonnie++ and also monitoring with gstat. I then moved those drives back on my main server. When I booted the system I hung on the 320GB gmirror devices. Previously the 1st 1TB gmirror and 320GB gmirror were attached to the integrated USB ports on the motherboard. I moved the 320GB gmirror to a PCI USB adapter. The 2 320GB drives in the gmirror were da5 and da6. Here's what I saw on the console: (da6:umass-sim6:6:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0 (da6:umass-sim6:6:0:0): CAM Status: SCSI Status Error (da6:umass-sim6:6:0:0): SCSI Status: Check Condition (da6:umass-sim6:6:0:0): ILLEGAL REQUEST asc:20,0 (da6:umass-sim6:6:0:0): Invalid command operation mode (da6:umass-sim6:6:0:0): Unretryable error GEOM_MIRROR: Request failed (error=5), da6[READ(offset=512, length=512)] GEOM_MIRROR: Device gm-san: provider da6 disconnected. (da5:umass-sim5:5:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0 (da5:umass-sim5:5:0:0): CAM Status: SCSI Status Error (da5:umass-sim5:5:0:0): SCSI Status: Check Condition (da5:umass-sim5:5:0:0): ILLEGAL REQUEST asc:20,0 (da5:umass-sim5:5:0:0): Invalid command operation mode (da5:umass-sim5:5:0:0): Unretryable error GEOM_JOURNAL: BIO_FLUSH not supported by mirror/gm-san. I waited for a few minutes with no change in the console. I then detached one of the USB drives (which happened to be da6) and saw this: umass6: at uhub7 port 4 (addr 4) disconnected (da6:umass-sim6:6:0:0): lost device Nothing else changed for a few minutes so I powered off the system. When I brought it back up the 320GB gmirror device was out of sync, but apart from tha
Re: Multiple USB drives stability question
I am now trying to rsync large files from the 320GB gmirror+gjournal device to the 2nd 1TB gmirror+gjournal device. Using gstat I see the 320GB device active all the time while the 1TB device loads in spurts. There will be periods of multiple seconds where the target providers are completely idle while the source providers are still reporting 100% active. Is there any tuning I should be investigating for these GEOM classes? --- On Fri, 8/14/09, Jeff Richards wrote: From: Jeff Richards Subject: Re: Multiple USB drives stability question To: freebsd-stable@freebsd.org Date: Friday, August 14, 2009, 11:04 PM I just tested my 2nd 1TB gmirror device on another system with FBSD 7.2. I was getting full throughput on the drive and no lockup using bonnie++ and also monitoring with gstat. I then moved those drives back on my main server. When I booted the system I hung on the 320GB gmirror devices. Previously the 1st 1TB gmirror and 320GB gmirror were attached to the integrated USB ports on the motherboard. I moved the 320GB gmirror to a PCI USB adapter. The 2 320GB drives in the gmirror were da5 and da6. Here's what I saw on the console: (da6:umass-sim6:6:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0 (da6:umass-sim6:6:0:0): CAM Status: SCSI Status Error (da6:umass-sim6:6:0:0): SCSI Status: Check Condition (da6:umass-sim6:6:0:0): ILLEGAL REQUEST asc:20,0 (da6:umass-sim6:6:0:0): Invalid command operation mode (da6:umass-sim6:6:0:0): Unretryable error GEOM_MIRROR: Request failed (error=5), da6[READ(offset=512, length=512)] GEOM_MIRROR: Device gm-san: provider da6 disconnected. (da5:umass-sim5:5:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0 (da5:umass-sim5:5:0:0): CAM Status: SCSI Status Error (da5:umass-sim5:5:0:0): SCSI Status: Check Condition (da5:umass-sim5:5:0:0): ILLEGAL REQUEST asc:20,0 (da5:umass-sim5:5:0:0): Invalid command operation mode (da5:umass-sim5:5:0:0): Unretryable error GEOM_JOURNAL: BIO_FLUSH not supported by mirror/gm-san. I waited for a few minutes with no change in the console. I then detached one of the USB drives (which happened to be da6) and saw this: umass6: at uhub7 port 4 (addr 4) disconnected (da6:umass-sim6:6:0:0): lost device Nothing else changed for a few minutes so I powered off the system. When I brought it back up the 320GB gmirror device was out of sync, but apart from that all devices were online. Below are the kernel messages from the second boot: Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.2-RELEASE #0: Fri May 1 08:49:13 UTC 2009 r...@walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Celeron(R) CPU 2.26GHz (2266.67-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf49 Stepping = 9 Features=0xbfebfbff Features2=0x441d AMD Features2=0x1 real memory = 1877868544 (1790 MB) avail memory = 1826934784 (1742 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a (3) failed acpi0: reservation of 10, 6fde (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_hpet0: iomem 0xfe80-0xfe8003ff on acpi0 device_attach: acpi_hpet0 attach returned 12 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xc000-0xcfff,0xfb00-0xfbff irq 16 at device 0.0 on pci1 pcib2: irq 27 at device 2.0 on pci0 pci2: on pcib2 pcib3: irq 31 at device 3.0 on pci0 pci3: on pcib3 atapci0: port 0xfc00-0xfc07,0xf800-0xf803,0xf400-0xf407,0xf000-0xf003,0xec00-0xec0f,0xe800-0xe8ff irq 21 at device 15.0 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe400-0xe40f at device 15.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] uhci0: port 0xe000-0xe01f irq 20 at device 16.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xdc00-0xdc1f irq 22 at device 16.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xd800-0xd81f irq 21 at device 16.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xd400-0x
Re: Multiple USB drives stability question
0 device da3: 40.000MB/s transfers da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da4 at umass-sim4 bus 4 target 0 lun 0 da4: Fixed Direct Access SCSI-0 device da4: 40.000MB/s transfers da4: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) da5 at umass-sim5 bus 5 target 0 lun 0 da5: Fixed Direct Access SCSI-0 device da5: 40.000MB/s transfers da5: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C) da6 at umass-sim6 bus 6 target 0 lun 0 da6: Fixed Direct Access SCSI-0 device da6: 40.000MB/s transfers da6: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C) GEOM_MIRROR: Device mirror/gm-bacula launched (2/2). GEOM_MIRROR: Device mirror/gm-media launched (2/2). GEOM_MIRROR: Device mirror/gm-san launched (1/2). GEOM_MIRROR: Device gm-san: rebuilding provider da5. GEOM_JOURNAL: Journal 3034050800: mirror/gm-bacula contains data. GEOM_JOURNAL: Journal 3034050800: mirror/gm-bacula contains journal. GEOM_JOURNAL: Journal mirror/gm-bacula clean. GEOM_JOURNAL: Journal 262277634: mirror/gm-media contains data. GEOM_JOURNAL: Journal 262277634: mirror/gm-media contains journal. GEOM_JOURNAL: Journal mirror/gm-media clean. GEOM_LABEL: Label for provider da4s1 is ufsid/49273a95d669d784. GEOM_LABEL: Label for provider da4s1 is ufs/backup. GEOM_JOURNAL: Journal 2270030291: mirror/gm-san contains data. GEOM_JOURNAL: Journal 2270030291: mirror/gm-san contains journal. GEOM_JOURNAL: Journal mirror/gm-san clean. (da6:umass-sim6:6:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0 (da6:umass-sim6:6:0:0): CAM Status: SCSI Status Error (da6:umass-sim6:6:0:0): SCSI Status: Check Condition (da6:umass-sim6:6:0:0): ILLEGAL REQUEST asc:20,0 (da6:umass-sim6:6:0:0): Invalid command operation code (da6:umass-sim6:6:0:0): Unretryable error GEOM_MIRROR: Request failed (error=5). da6[READ(offset=0, length=512)] GEOM_JOURNAL: BIO_FLUSH not supported by mirror/gm-san. GEOM_LABEL: Label for provider mirror/gm-bacula.journal is ufsid/4a509cddbd500a7e. GEOM_LABEL: Label for provider mirror/gm-media.journal is ufsid/4a3f26878cf7f367. GEOM_LABEL: Label for provider mirror/gm-san.journal is ufsid/4a40c57f604c2e44. Trying to mount root from ufs:/dev/ad0s1a GEOM_LABEL: Label ufsid/4a42cfbde524d087 removed. GEOM_LABEL: Label for provider ad0s1a is ufsid/4a42cfbde524d087. GEOM_LABEL: Label ufsid/49273a95d669d784 removed. GEOM_LABEL: Label ufsid/4a42cfbdfcdf27b1 removed. GEOM_LABEL: Label for provider da4s1c is ufsid/49273a95d669d784. GEOM_LABEL: Label for provider ad0s1e is ufsid/4a42cfbdfcdf27b1. GEOM_LABEL: Label ufsid/4a42cfbd65525a3f removed. GEOM_LABEL: Label for provider ad0s1f is ufsid/4a42cfbd65525a3f. GEOM_LABEL: Label ufsid/4a42cfbd75a68b18 removed. GEOM_LABEL: Label for provider ad0s1g is ufsid/4a42cfbd75a68b18. GEOM_LABEL: Label ufsid/4a42cfbdcada79a0 removed. GEOM_LABEL: Label for provider ad0s2d is ufsid/4a42cfbdcada79a0. GEOM_LABEL: Label ufsid/4a42cfc28b730061 removed. GEOM_LABEL: Label for provider ad0s2e is ufsid/4a42cfc28b730061. GEOM_LABEL: Label ufsid/4a42cfc21242e734 removed. GEOM_LABEL: Label for provider ad0s1d is ufsid/4a42cfc21242e734. GEOM_LABEL: Label ufsid/4a42cfc236be6f59 removed. GEOM_LABEL: Label for provider ad0s2f is ufsid/4a42cfc236be6f59. GEOM_LABEL: Label ufsid/4a42cfbde524d087 removed. GEOM_LABEL: Label ufsid/4a42cfbdfcdf27b1 removed. GEOM_LABEL: Label ufsid/4a42cfbd65525a3f removed. GEOM_LABEL: Label ufsid/4a42cfbd75a68b18 removed. GEOM_LABEL: Label ufsid/4a42cfbdcada79a0 removed. GEOM_LABEL: Label ufsid/4a430e552079b936 removed. GEOM_LABEL: Label ufsid/4a42cfc28b730061 removed. GEOM_LABEL: Label ufsid/4a42cfc21242e734 removed. GEOM_LABEL: Label ufsid/4a42cfc236be6f59 removed. GEOM_LABEL: Label ufsid/4a3f26878cf7f367 removed. GEOM_LABEL: Label ufsid/4a40c57f604c2e44 removed. GEOM_LABEL: Label ufsid/49273a95d669d784 removed. fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8 GEOM_LABEL: Label ufsid/4a509cddbd500a7e removed. --- On Fri, 8/14/09, Jeff Richards wrote: From: Jeff Richards Subject: Multiple USB drives stability question To: freebsd-stable@freebsd.org Date: Friday, August 14, 2009, 8:19 PM Is there a practical limit on the number of active USB drives with FreeBSD? I've had stability issues using multiple USB drives as storage. My initial design goal was cheap, hot-swappable storage. I am only using a 100MB network currently so throughput on the storage is not a problem as I can't push the data to/from the drives faster than what my network requests are. I first tried my setup on 7.0, then migrated to a newer PC, then upgraded to 7.2. I have the following USB drive setup: 1 320GB gmirror (320x2) + gjournal + ufs2 1 1TB gmirror (1TBx2) + gjournal + ufs2 1 150GB gjournal + ufs2 I also have another 1TB gmirror (1TBx2) + gjournal but removed it. The system crashed when I used these drives (bacula or bonnie++) so I pulled them to test on another system. Recently my stability issue has been when I have been writing data to the 150GB gjourn
Multiple USB drives stability question
Is there a practical limit on the number of active USB drives with FreeBSD? I've had stability issues using multiple USB drives as storage. My initial design goal was cheap, hot-swappable storage. I am only using a 100MB network currently so throughput on the storage is not a problem as I can't push the data to/from the drives faster than what my network requests are. I first tried my setup on 7.0, then migrated to a newer PC, then upgraded to 7.2. I have the following USB drive setup: 1 320GB gmirror (320x2) + gjournal + ufs2 1 1TB gmirror (1TBx2) + gjournal + ufs2 1 150GB gjournal + ufs2 I also have another 1TB gmirror (1TBx2) + gjournal but removed it. The system crashed when I used these drives (bacula or bonnie++) so I pulled them to test on another system. Recently my stability issue has been when I have been writing data to the 150GB gjournal drive from the 320GB gmirror device (USB device -> USB device). It will be working fine, then all I/O stops on the 150GB drive. The system remains responding to other USB devices etc. for a while. I try rebooting and the system crashes with gjournal errors (didn't write down, but I will later). Every time this happens the 1TB gmirror comes up fine but one of the 320GB providers is missing. No problem after 'gmirror forget' and 'gmirror insert'. Everything rebuilds fine. The 150GB gjournal drive is fine after a 'fsck -y'. I do pair the gmirror drives to the same USB adapter. Found out after initial testing with multiple USB adapters that they do not appear standard enough to cross adapters like I would for a production server at work to prevent SPOF with an adapter. I have tried Linux as well with softraid and LVM2 on the same hardware. It worked fine until I applied software updates and the udev took 30+ minutes to boot. I went back to FreeBSD. Even when I crashed I was back up in 2-5 minutes. I can and will provide more detail if requested. My concern is that the issue seems to continue no matter what hardware/OS changes I try. Thanks in advance. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
'at now' not working as expected
When I try to schedule something immediately with an 'at now' command it appears to queue up but can wait multiple minutes before actually executing. Is there something I have missed with FreeBSD's version of at? I've used 'at now' with AIX, Linux, and OpenBSD and it immediately executes for those systems. I am running FreeBSD 7.0 stable. Thanks in advance. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
gjournal question
I have started experimenting with gjournal filesystems this weekend. I found something that may be a mistake I made. Not sure. To break up my IDE drive into the filesystems I wanted I created multiple slices. On slice 2 I had multiple gjournal filesystems. I tried creating a journal on slice 3 (ad0s3d I think), but it failed. It actually created a journal entry in /dev, but I had trouble using it. I then tried to remove the journal. Then a journal entry in /dev appeared for the slice itself (/dev/ad0s3.journal). If I removed the journal on the slice the partition journal reappeared. I was able to get my filesystems the way I wanted by not using slice 3 and putting the remaining fileystem in slice 2. Are there any internal requirement that a slice must be fully utilized with partitions before a new slice may be used? I think I was able to create a standard UFS2 filesystem on slice 3 and just had trouble with the journal. This was not a major problem for me, just curious. Thanks. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
sshd_config question
Hi. I have configured sshd in OpenBSD to require publickey authentication. I've tried configuring FreeBSD to do the same, but I can still login via keyboard authentication. Here are the options I have in my sshd_config: PasswordAuthentication no ChallengeResponseAuthentication no UsePAM no After setting those options I kill -HUP the sshd process. Is there something simple I am missing? Thanks in advance. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
