Re: serious vinum bug in 4-10 RELEASE?-solved?
On Mon, Jul 12, 2004 at 06:40:01AM +0930, Greg 'groggy' Lehey wrote: > I see no drives. > > > Ideas? > I have concluded that this is the result of somekind of vinum/hardware incompatibility. The problem in question occured during the upgrade to faster disks, specifically, Seagate Cheetah ST318453LC on a DELL 2450. If I swap back the old Quantum Atlas 36G disk, the problem entirely disappears. The new disks function ok with UFS partitions but not vinum. It is 100% repeatable. Don't know why. thanx - steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: serious vinum bug in 4-10 RELEASE?
On Mon, Jul 12, 2004 at 06:40:01AM +0930, Greg 'groggy' Lehey wrote: > > > > I get the following incorrect configuration. Notice that > > drives data0 and data1 are missing and drives data2 and data3 are > > duplicated where data0 and data1 should be. > > I see no drives. Not sure what you mean see below > > > Ideas? > > http://www.vinumvm.org/vinum/how-to-debug.html I recreated a simpler situation with just 2 mirrored drives. After reboot the vinum lv -r -v reports (*missing *) drives. But before reboot they are there. this is the initial vinum lv -r -v. drives data0 and data1 are clearly listed as mq0.p0.s0 and mq0.p1.s0 Volume mq0: Size: 18341345792 bytes (17491 MB) State: up Flags: 2 plexes Read policy: round robin Plex mq0.p0:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Plex mq0.p1:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Subdisk mq0.p0.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p0 at offset 0 (0 B) Drive data0 (/dev/da0d) at offset 135680 (132 kB) Subdisk mq0.p1.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p1 at offset 0 (0 B) Drive data1 (/dev/da1d) at offset 135680 (132 kB) this is the vinum lv -r -v. after reboot drives data0 and data1 are listed as (*missing*) Volume mq0: Size: 18341345792 bytes (17491 MB) State: up Flags: 2 plexes Read policy: round robin Plex mq0.p0:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Plex mq0.p1:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Subdisk mq0.p0.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p0 at offset 0 (0 B) Drive (*missing*) at offset 135680 (132 kB) Subdisk mq0.p1.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p1 at offset 0 (0 B) Drive (*missing*) at offset 135680 (132 kB) Here is the on disk configuration, as per your online script. IN [EMAIL PROTECTED]@^/dev/da0d count=300 vinum no longer detects any volumes, vinum lv is blank, BUT your script to read the vinum config on disk shows the same vinum config above. If I change the fstype from vinum to 4.2BSD and newfs the partition its still there. Is this possible? Also on boot vinum reports vinum: loaded vinum: no drives found vinum_scandisk() returned 2vinum: no drives found ** no drives found **: no such file or directory Any other info required, just ask. -steve "The age of the Internet has a right to its own music" http://www.linuxsuite.org ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
serious vinum bug in 4-10 RELEASE?
Howdy! I have 4 identical disks, labels etc are also identical. It looks like vinum after reboot does not recognize drives properly, as it did immedialtely after initial configuration. One drive/subdisk in each plex isn't recognized, and the other one is duplicated, which destroys the mirror. I created 2 vinum volumes with # vinum create -f /etc/vinum.raid1 and the following config file drive data0 device /dev/da0d drive data1 device /dev/da1d drive data2 device /dev/da2d drive data3 device /dev/da3d volume mq0 setupstate plex org concat sd length 0 drive data0 plex org concat sd length 0 drive data2 volume mq1 setupstate plex org concat sd length 0 drive data1 plex org concat sd length 0 drive data3 after running # vinum lv -r -v I get (correctly) Volume mq0: Size: 18341345792 bytes (17491 MB) State: up Flags: 2 plexes Read policy: round robin Plex mq0.p0:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Plex mq0.p1:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Subdisk mq0.p0.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p0 at offset 0 (0 B) Drive data0 (/dev/da0d) at offset 135680 (132 kB) Subdisk mq0.p1.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p1 at offset 0 (0 B) Drive data2 (/dev/da2d) at offset 135680 (132 kB) Volume mq1: Size: 18341345792 bytes (17491 MB) State: up Flags: 2 plexes Read policy: round robin Plex mq1.p0:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq1 Plex mq1.p1:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq1 Subdisk mq1.p0.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq1.p0 at offset 0 (0 B) Drive data1 (/dev/da1d) at offset 135680 (132 kB) Subdisk mq1.p1.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq1.p1 at offset 0 (0 B) Drive data3 (/dev/da3d) at offset 135680 (132 kB) After rebooting the system and running # vinum lv -r -v I get the following incorrect configuration. Notice that drives data0 and data1 are missing and drives data2 and data3 are duplicated where data0 and data1 should be. Volume mq0: Size: 18341345792 bytes (17491 MB) State: up Flags: 2 plexes Read policy: round robin Plex mq0.p0:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq0 Plex mq0.p1:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: flaky Organization: concat Part of volume mq0 Subdisk mq0.p0.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq0.p0 at offset 0 (0 B) Drive data2 (/dev/da2d) at offset 135680 (132 kB) Subdisk mq0.p1.s0: Size: 18341345792 bytes (17491 MB) State: reborn Plex mq0.p1 at offset 0 (0 B) Drive data2 (/dev/da2d) at offset 135680 (132 kB) Volume mq1: Size: 18341345792 bytes (17491 MB) State: up Flags: 2 plexes Read policy: round robin Plex mq1.p0:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: up Organization: concat Part of volume mq1 Plex mq1.p1:Size: 18341345792 bytes (17491 MB) Subdisks:1 State: flaky Organization: concat Part of volume mq1 Subdisk mq1.p0.s0: Size: 18341345792 bytes (17491 MB) State: up Plex mq1.p0 at offset 0 (0 B) Drive data3 (/dev/da3d) at offset 135680 (132 kB) Subdisk mq1.p1.s0: Size: 18341345792 bytes (17491 MB) State: reborn Plex mq1.p1 at offset 0 (0 B) Dr
Re: Max NFSD processes
On Fri, May 21, 2004 at 01:06:41PM -0500, Eric Anderson wrote: > > > > Disk design issues matter cause it is the ultimate > >bottleneck. RAID is you friend. > > > > Just curious - what RAID level/configuration do you use? I've typically > used RAID5, but RAID0+1 might be acceptable. > Depends. RAID5 has worst right performance, so it might be bad for something like mail. Generally I use RAID0+1 for mail and RAID5 for web data. Also, depending on your budget and/or concerns about failure/downtime, you might want to vinum mirror the entire base system disks. Be sure to have the root partition vinumed also. There is good documentation in the handbook and on gregs page. Actually all my nfs servers are "diskless" except for the data and boot off of boot servers using PXE. The boot servers have vinumed system disks. -steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
On Thu, May 20, 2004 at 10:44:14AM -0500, Eric Anderson wrote: > > > > That's good to hear. Did you do any other tweaks? sysctl settings? > mbufs? net.inet.udp.recvspace=524288 kern.ipc.maxsockbuf=1048576 net.inet.icmp.drop_redirect=1 net.inet.icmp.log_redirect=1 Telus-nfs1:# w 12:41PM up 212 days, 15 hrs, 3 users, load averages: 0.49, 0.62, 0.71 USER TTY FROM LOGIN@ IDLE WHAT nfs1:# netstat -m 441/1488/65536 mbufs in use (current/peak/max): 406 mbufs allocated to data 35 mbufs allocated to packet headers 242/1040/16384 mbuf clusters in use (current/peak/max) 2452 Kbytes allocated to network (4% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines > > > >>Thanks - any help/hints is appreciated. > >> > >> Disk design issues matter cause it is the ultimate bottleneck. RAID is you friend. Lots of RAM helps. I use 4G. and compile a custom kernel with maxusers at 256 and KVA_PAGES at 512. You can check/verify kvm usage with sysclt's vm.kvm_size and vm.kvm_free > >> > > > > You probably also want good nics (fxp0) and to > >increase UDP buffer space. I have found that nfs over udp > >offers supperior performance than tcp on a good LAN > > > > > I'm currently using 3com's (xl0,xl1) and Intel Gigabit cards (em0,em1). > Most of my clients are using udp. > > What did you set your buffer space to? Which sysctl did you change? > udp recvspace. see above. -steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
vinum root problem - "missing unit number"
Howdy! I am setting up a fileserver using two completely mirrored disks using vinum. I am using vinum on the root partition. I booted from a third disk and setup the vinum volumes and mounted them under /mnt, then just did "make installworld DESTDIR=/mnt" Everything was cool, could fsck -n /dev/da1a (the "fake" a partition) no problem. Everything is happy booting from a third drive and mounting vinum volumes. The loader comes up fine and shows the vinum module loaded and the variables vinum_root and vinum_drives correctly as specified in loader.conf. The system is 4.8-STABLE from about 3 weeks ago. /boot/loader.conf (on vinum/root) contains vinum_load="YES"# Concatenated/mirror/raid driver vinum_root="root" # Concatenated/mirror/raid driver vinum_drives="/dev/da0 /dev/da1"# Concatenated/mirror/raid driver Loader is finding the correct "fake" disk partition "a" and everything but when the kernel boots mountroot fails with "missing unit number". Any ideas for debugging, or what I am doing wrong welcome. thanx - steve Additional info follows ... kernel boot ... The Regents of the University of California. All rights reserved. FreeBSD 4.8-STABLE #0: Fri Jul 25 10:41:22 EDT 2003 [EMAIL PROTECTED]:/usr/src/sys/compile/MFS Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 1130456212 Hz CPU: Intel(R) Pentium(R) III CPU family 1133MHz (1130.46-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6b1 Stepping = 1 Features=0x383fbff real memory = 2147418112 (2097088K bytes) avail memory = 2087862272 (2038928K bytes) Preloaded elf kernel "kernel" at 0x4038a000. Preloaded elf module "vinum.ko" at 0x4038a09c. Pentium Pro MTRR support enabled Using $PIR table, 13 entries at 0x400f3b60 [snip] vinum: loaded Mounting root from ufs:/dev/vinum/root missing unit number setrootbyname failed ffs_mountroot: can't find rootvp Root mount failed: 6 Mounting root from ufs:da0s4a Root mount failed: 6 Mounting root from ufs:da0a Root mount failed: 6 mountroot> ? Possibly valid devices for 'ufs' root: "console" "ctty" "mem" "pts" "ptc" "log" "fd" "da" "FD" "pass" "pci" "vinum" "xpt" /etc/fstab in /dev/vinum/root /dev/vinum/swapnone swapsw0 0 /dev/vinum/root/ufs rw1 1 /dev/vinum/usr /usr ufs rw2 2 /dev/vinum/var /var ufs rw2 2 proc /procprocfs rw0 0 disklabels for both mirrored disks 8 partitions: #size offsetfstype [fsize bsize bps/cpg] a: 253671 2814.2BSD 2048 1638497 # (Cyl.0*- 15*) c: 358436700unused0 0 # (Cyl.0 - 2231*) h: 35843654 16 vinum# (Cyl.0*- 2231*) This is vinum configuration file that was used to set up the volume They were originally recongnized as da1 da2, but I moved them to da0 da1. cause I need the other disk to boot and setup other volumes on other boxen. # vinum.cfg # # # drive d0 device /dev/da1h drive d1 device /dev/da2h volume root setupstate plex org concat sd length 253671s driveoffset 265s drive d0 plex org concat sd length 253671s driveoffset 265s drive d1 volume swap setupstate plex org concat sd length 2097152s driveoffset 253936s drive d0 plex org concat sd length 2097152s driveoffset 253936s drive d1 volume usr setupstate plex org concat sd length 2097152s driveoffset 2351088s drive d0 plex org concat sd length 2097152s driveoffset 2351088s drive d1 volume var setupstate plex org concat sd length 409600s driveoffset 4448240s drive d0 plex org concat sd length 409600s driveoffset 4448240s drive d1 volume netb setupstate plex org concat sd length 10485760s driveoffset 4857840s drive d0 plex org concat sd length 10485760s driveoffset 4857840s drive d1 volume netb_root setupstate plex org concat sd length 16777216s driveoffset 15343600s drive d0 plex org concat sd length 16777216s driveoffset 15343600s drive d1 volume netb_host setupstate plex org concat sd length 3722838s driveoffset 32120816s drive d0 plex org concat sd length 3722838s driveoffset 32120816s drive d1 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: cannot boot from SCSI disk
On Thu, Feb 13, 2003 at 03:28:58PM +0100, Roman Neuhauser wrote: > > interesting. all I see is backpedalling from dangerously dedicated > disks in /stand/sysinstall and elsewhere Many IDE drives will not boot if setup in "dangerously dedicated" mode. So, for "normal" people who are installing for personal/desktop use and/or dual boot "home" systems the issue is confusing and often results in failed installlations on many typical "home" installs. > , and the article you linked below says > > "Remember, dedicated mode disks sometimes cannot be booted by the PC > architecture." > This is correct. Particularly for IDE. But the opposite is true for server class SCSI systems. I have had IDE systems that would not boot if setup in "dedicated mode". I have had SCSI systems that would not boot unless they were setup in "dedicated mode". > > In general SCSI disk that only contain FreeBSD should be > > formated/labled in "dedicated mode" IMHO. > > what source do you have this information from? > I have never personally had a SCSI system that *didn't* boot when setup in "dedicated mode". Dedicated mode is simpler so I prefer it. -steve To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
Re: cannot boot from SCSI disk
Howdy! Some hardware will not boot FreeBSD from SCSI disks unless the disk is formated and labled in "dedicated mode". http://www.freebsd.org/doc/en_US.ISO8859-1/articles/formating-media/x65.html In general SCSI disk that only contain FreeBSD should be formated/labled in "dedicated mode" IMHO. -steve To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
Re: NFS Performance woes
Howdy! I have done some simulations with NFS servers - Intel SCB2 (4G RAM) serving files from 500G RAID devices. I created a treed directory structure with 300G of 32k files that approximates our "homedirectory" structure. I had about 6 diskless front ends (tyan 2518 with 2G) that NFS booted and mounted the "homedir" and ran multiple scripts that walked through the directory structure reading files and writing them to /dev/null. All machines have 3 intel 100 NICs. One interface is used to mount the root etc. and the other is used to mount "homedirs". The NFS server serves root from fxp0 and "homedir" data from both fxp1 fxp2. The diskless front ends mount root from fxp0 and mount "homedir" from fxp1. When the simulation was running full out I was serving about 1/3-1/2 data from page cache and 2/3-1/2 from disk. I tried numerous configurations and tuning of network parameters after doing research and discovered... for FreeBSD 4.6.2 1) NFS over UDP outperforms NFS over TCP. I was able to average about 70Mbs over *both* (occasionally they would almost max out ie. 95Mbs) interfaces serving data using UDP mounts with 8K rw. (the default). No matter what I tried with TCP I never got more than half that throughput. 2) The optimal number of nfsd's to run on the server was about 100! If I reduced the number of nfsds below 80 it would start to choke off the data moving through the network. I found that at around 100 there was no more increase. You must make a minor change to source as the max allowed now by default is 20. I was running 8 nfsiod's on the clients. TCP mounts under tested conditions always had much higher loads than UDP. Also it was impossible to do an "ls" on a mounted directory under load with TCP. With UDP there were no such problems. If you are using UDP it is *essential* that you monitor fragments that are being droppend because of timeout. If you have a good network this should not be a problem. For example I have a webserver that has been up 20 days and has moved 1G of fragments but has only dropped about 800. Also TCP mounts will require a remount of the clients if the server should crash/whatever. UDP just keeps on ticking. If you have Gig ether than there is other tuning you *must* do to realize this potential. Personally I think it is better to use multiple 100Mbs NIC's than to use Gig ether if you can get away with it. YMMV. -steve To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
Silly vinum question
Howdy! I have 2 disks on separate SCSI busses that are striped to form a single vinum volume. They are recognized as da0 and da1 I want to add 1 more disk to each SCSI buss and create another striped vinum volume, but keep the initial vinum volume intact. The issue ... When the 2 disks are added to SCSI busses the first 2 disks in the initial volume are not recognized by the system in the same way. For instance ... disk SCSI buss ORIGINAL CONFIGPLANNED CONFIG disk1 1 da0 da0 disk2 2 da1 da2 disk3 1 da1 disk4 2 da3 The original vinum volume is configured on da0 da1 Under the planned config the original vinum volume will now be on da0 da2. While I could wire down the devices in kernel config .. Also I want to stripe to disks on separate busses so moving the secound disk onto the same bus as disk1 is not a good option I am still wondering if vinum will be able to build volumes correctly based only on the information stored in the per disk configuration even if the system sees them differntly. thanx - steve To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
