Re: RAID, persistent superblock on SPARC
On Sun, Jul 09, 2000 at 11:04:20PM -0700, Gregory Leblanc wrote: > What's the current status of RAID on SPARC? I haven't had a chance to keep > up very much, as I wasn't using RAID on SPARCs. I'm about to build a > mirrored system here, and I'd like to make sure that I'm not going to get > hosed because of some bug. Thanks, I think the quick fix (copying 4096 bytes, not sizeof(md_superblock*)) was added, at least to 2.2.16 patches, will have to check 2.4.x (the copy of 2.3.99-pre7 I have on my box does not include it). The larger patch I sent which fixed the on-disk format and allowed RAID arrays to be stored at cylinder 0 of SPARC labeled disks did not make it in into either version (will have to ask Ingo about it). Jakub
Re: Strange quota entries on Sparc64 2.2.16-RAID
On Sat, Jul 08, 2000 at 11:42:14PM -0400, Gustav wrote: > I've started to notice strange entries in my quota tables on the > one of my RAID-1 partitions on a sparc64 2.2.16-RAID box. Basically, > the following entries are appearing, more every few days: What quota package are you using? It has to be patched, so that it reads/writes on sparc64 the same on-disk format as does the kernel (ie. although the utilities may be 32bit, it has to write several entries using long long). Red Hat 6.x quota should work correctly, if you are using some other distribution, check if quota is patched for sparc64. Jakub
RFC: RAID superblock layout fix and reserved-bytes support
Hi! BEWARE: Don't use the following patch in production, it might eat your RAID set for breakfest. Attached is a patch which does 4 things: - tries to solve cleanly the RAID superblock issues on non-x86 architectures (where sizeof(md_super_t) was bigger than 4096, usually 4104) by introducing RAID 0.91.x on-disk format which is binary compatible with the 0.90 for i386 (and at the same time changes it from native-endian to little-endian). - introduces reserved-bytes setting in raidtab, for which the default is auto-probed by mkraid if not specified. If non-zero, the RAID array will make sure first reserved_bytes on the disk are never touched (resynced or whatever). This makes it possible e.g. to place RAID partition to cylinder 0 on a disk with Sun partition table. - in raid1.c raid1_kmalloc allocated a wrong size The patch is against 2.2.14 with 2.2.14-B1 RAID patch, because 2.3.99-pre2 is missing the raid1/5 bits. I can make try to port the remaining files changes to 2.3.99-pre2 though. - raidtab.5 man page fix I'm looking for testers both on x86 and non-x86. Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz/~jj Linux version 2.3.99-pre2 on a sparc64 machine (1343.49 BogoMips) ___ --- linux/arch/sparc64/kernel/ioctl32.c.jj Mon Jan 24 11:36:41 2000 +++ linux/arch/sparc64/kernel/ioctl32.c Fri Mar 10 17:32:37 2000 @@ -2022,12 +2022,14 @@ asmlinkage int sys32_ioctl(unsigned int /* 0x09 */ case /* RAID_VERSION */ _IOR (MD_MAJOR, 0x10, char[12]): - case /* GET_ARRAY_INFO */ _IOR (MD_MAJOR, 0x11, char[72]): + case /* GET_ARRAY_INFO */ _IOR (MD_MAJOR, 0x11, char[128]): + case /* OLD_GET_ARRAY_INFO */ _IOR (MD_MAJOR, 0x11, char[72]): case /* GET_DISK_INFO */_IOR (MD_MAJOR, 0x12, char[20]): case /* CLEAR_ARRAY */ _IO (MD_MAJOR, 0x20): case /* ADD_NEW_DISK */ _IOW (MD_MAJOR, 0x21, char[20]): case /* HOT_REMOVE_DISK */ _IO (MD_MAJOR, 0x22): - case /* SET_ARRAY_INFO */ _IOW (MD_MAJOR, 0x23, char[72]): + case /* SET_ARRAY_INFO */ _IOW (MD_MAJOR, 0x23, char[128]): + case /* OLD_SET_ARRAY_INFO */ _IOW (MD_MAJOR, 0x23, char[72]): case /* SET_DISK_INFO */_IO (MD_MAJOR, 0x24): case /* WRITE_RAID_INFO */ _IO (MD_MAJOR, 0x25): case /* UNPROTECT_ARRAY */ _IO (MD_MAJOR, 0x26): --- linux/drivers/block/md.c.jj Mon Jan 24 11:36:42 2000 +++ linux/drivers/block/md.cThu Mar 16 14:29:46 2000 @@ -11,6 +11,8 @@ - kerneld support by Boris Tobotras <[EMAIL PROTECTED]> - kmod support by: Cyrus Durgin - RAID0 bugfixes: Mark Anthony Lisher <[EMAIL PROTECTED]> + - superblock layout on non-x86 fixes and reserved_bytes support by + Jakub Jelinek <[EMAIL PROTECTED]> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by @@ -299,9 +301,18 @@ static unsigned int calc_dev_sboffset (k return size; } +static inline unsigned int calc_dev_reserved (mddev_t *mddev) +{ + unsigned int reserved = mddev->sb->reserved_bytes; + + reserved += mddev->sb->chunk_size - 1; + reserved &= ~(mddev->sb->chunk_size - 1); + return reserved / 1024; +} + static unsigned int calc_dev_size (kdev_t dev, mddev_t *mddev, int persistent) { - unsigned int size; + unsigned int size, reserved; size = calc_dev_sboffset(dev, mddev, persistent); if (!mddev->sb) { @@ -310,9 +321,25 @@ static unsigned int calc_dev_size (kdev_ } if (mddev->sb->chunk_size) size &= ~(mddev->sb->chunk_size/1024 - 1); + reserved = calc_dev_reserved (mddev); + if (reserved > size) + size = 0; + else + size -= reserved; return size; } +__u64 __inline__ md_read_events (mdp_super_t *sb) +{ + return (((__u64)sb->eventshi) << 32) | sb->eventslo; +} + +void __inline__ md_write_events (__u64 events, mdp_super_t *sb) +{ + sb->eventshi = events >> 32; + sb->eventslo = events; +} + /* * We check wether all devices are numbered from 0 to nb_dev-1. The * order is guaranteed even after device name changes. @@ -376,28 +403,13 @@ abort: return 1; } -static unsigned int zoned_raid_size (mddev_t *mddev) +static inline unsigned int zoned_raid_size (mddev_t *mddev) { - unsigned int mask; mdk_rdev_t * rdev; struct md_list_head *tmp; - if (!mddev->sb) { - MD_BUG(); - return -EINVAL; - } - /* -* do size and offset calculations. -*/ - mask = ~(mddev->sb->chunk_size/1
Re: Help Raid for sparc
On Fri, Nov 26, 1999 at 09:43:06AM +0100, [EMAIL PROTECTED] wrote: > Hallo, > > I have a Sparc 10 with Linux6.1 running I have two disks of 1Gb and > 1.7Gb. > I would like to do a linear raid but when I do "raidstart -a /dev/md0 > into shell I receive -> /dev/md0: Invalid argument <- > and into consolle > (read) sdb1's sb offset:1026048 [events: 20202020] >md: invalid raid superblock magic on sdb1 >md: sdb1 has invalid sb, not importing! >could not import sdb1! >autostart sdb1 failed! > My kernel is 2.2.12-42 and raidtools-0.90 > If I do mkraid /dev/md0 I receive into shell > ->handling MD device /dev/md0 > analyzing super-block > disk 0: /dev/sdb1, 1026144kB, raid superblock at 1026048kB > disk 1: /dev/sdc1, 1720345kB, raid superblock at 1720256kB > /dev/md0: Invalid argument and into consolle I receive some messages > that they are into file attach messages. > My /etc/raidtab is: > raiddev /dev/md0 > raid-level linear > nr-raid-disks 2 > nr-spare-disks 0 > persistent-superblock 1 put chunk-size 8 here and redo mkraid (possibly with -f). It seems that the kernel is checking chunk size always, while raidtools are checking chunk size for raid0,1,4,5 only. IMHO kernel should not check chunk size for other raid levels, but if Ingo thinks it should, then raidtools should either error on not specified chunk-size for other levels as well or supply some default which will not trigger the md.c MD_BUG(). Cheers, Jakub ___ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz/~jj Linux version 2.3.18 on a sparc64 machine (1343.49 BogoMips) ___
Problems with persistant superblocks and drive removal
Hi! Until now I was using non-persistant superblocks, a few days ago I changed a lot of small (4 disk) md devices into a few big ones with persistant superblocks and I must say I'm quite disappointed by totally loosing control over what's going on and kernel doing weird things. The arrays work fine until I try to remove 5 SCSI disks (on which some other raid existed but was commented out already from raidtab). The kernel does not initialize most of the arrays at all. Attached is /proc/scsi/scsi content, /etc/raidtab, kernel messages from the raidstart -a command, /proc/mdstat content and kernel messages from kernel shuting down the arrays from immediately before the disk removal and immediately after it. Although I can hack raid superblocks manually and change device numbers/recalculate checksums for now, I fear as soon as one single disk somewhere dies off, I won't loose just a single raid0 array, but all of them. And I thought persistant superblocks are there exactly so that RAID arrays are independent of drive shuffling. The kernel should try hard to find a raid superblock on some other disk now when this functionality went into the kernel so one cannot provide a translation table where which disk moved to (I have edited raidtab properly but raidstart does not care). Is there some way how can I remove those disks from the system and avoid having to loose all the data on all those raid0 devices? Distribution: RH6.0 + kernel 2.2.12-29smp + rh6.1 raidtools Ultra Enterprise 3500, 43 disks on 5 SCSI HBAs Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz/~jj Linux version 2.3.18 on a sparc64 machine (1343.49 BogoMips) ___ Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST32430N SUN2.1G Rev: 0508 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 02 Lun: 00 Vendor: MICROP Model: 1936-21MZ1074215 Rev: HZ29 Type: Direct-AccessANSI SCSI revision: 01 CCS Host: scsi0 Channel: 00 Id: 04 Lun: 00 Vendor: MICROP Model: 1936-21MZ1074215 Rev: HZ29 Type: Direct-AccessANSI SCSI revision: 01 CCS Host: scsi0 Channel: 00 Id: 05 Lun: 00 Vendor: MICROP Model: 1936-21MZ1074215 Rev: HZ29 Type: Direct-AccessANSI SCSI revision: 01 CCS Host: scsi1 Channel: 00 Id: 06 Lun: 00 Vendor: TOSHIBA Model: XM6201TASUN32XCD Rev: 1103 Type: CD-ROM ANSI SCSI revision: 02 Host: scsi1 Channel: 00 Id: 08 Lun: 00 Vendor: Quantum Model: XP34300W Rev: L912 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi1 Channel: 00 Id: 09 Lun: 00 Vendor: Quantum Model: XP34300W Rev: L912 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi1 Channel: 00 Id: 11 Lun: 00 Vendor: Quantum Model: XP34300W Rev: L912 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 01 Lun: 00 Vendor: Quantum Model: XP34300W Rev: 81HB Type: Direct-AccessANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 02 Lun: 00 Vendor: Quantum Model: XP34300W Rev: 81HB Type: Direct-AccessANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 03 Lun: 00 Vendor: Quantum Model: XP34300W Rev: 81HB Type: Direct-AccessANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 04 Lun: 00 Vendor: Quantum Model: XP34300W Rev: 81HB Type: Direct-AccessANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 05 Lun: 00 Vendor: Quantum Model: XP34300W Rev: 81HB Type: Direct-AccessANSI SCSI revision: 02 Host: scsi3 Channel: 00 Id: 12 Lun: 00 Vendor: Quantum Model: XP34300W Rev: L912 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi3 Channel: 00 Id: 13 Lun: 00 Vendor: SEAGATE Model: ST150176LW Rev: 0002 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi4 Channel: 00 Id: 00 Lun: 00 Vendor: SUN Model: SSA100 Rev: 1.0 Type: ProcessorANSI SCSI revision: 02 Host: scsi4 Channel: 01 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST34371W SUN4.2G Rev: 7462 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi4 Channel: 01 Id: 01 Lun: 00 Vendor: SEAGATE Model: ST34371W SUN4.2G Rev: 7462 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi4 Channel: 01 Id: 02 Lun: 00 Vendor: SEAGATE Model: ST34371W SUN4.2G Rev: 7462 Type: Direct-AccessANSI SCSI revision: 02 Host: scsi4 Channel: 01 Id: 03 Lun: 00 Vendor: SEAGATE Model: ST34371W SUN4.2G Rev: 7462 Type: Direct-Access
Re: mkraid problem on SPARC Linux.
On Fri, Aug 20, 1999 at 11:03:09AM +0900, Akihiro Okamura wrote: > I want to use RAID-1 system on Sparcstaion 5, and I install RedHat6.0 for SPARC > and install raidtools using raidtools-19990724-0.90.tar.gz. > > When I executed mkraid, I received a following message. > > kernel: mkraid[399]: Unimplemented SPARC system call 140 Attached is a fix which has been submitted to Ingo some time ago. Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.3.13 on a sparc64 machine (1343.49 BogoMips) ___ raidtools-19990724-0.90.sparc.diff.gz
Re: Disksuite, Sun SSA & Linux
> > Great! Would either of you care to elaborate? > > Is the Linux server a Sun SPARC, or PC with some kind of > fiber card? Was it all pretty straight-forward, or was > there some wailing and gnashing of teeth to get it to work? I was using Sun SPARCs, with SOC and SOC+ cards. Using it should be straightforward, like for SSA you do modprobe soc; modprobe pluto > > How does the performance of Linux/raidtools/SSA (or A5000) > compare to Solaris/Disksuite/SSA or Solaris/VxVM/SSA? Well, I must tell you I don't know. I don't have the luxury to have a spare SSA or A5000 array which I can play with. I have one production SSA which works, but I cannot play with the driver nor shut it down for performance tests, and I could only play with A5000 in some Sun lab always for a couple of hours, bringing there my own disk with the kernel, testing for a while and coming back, so I reached the point where I succesfully mounted Solaris ufs disks and created some files on it, that's about it. But there are some issues: first, Sun SOC and SOC+ drivers download microcode into the cards, I cannot put that microcode into our drivers unless Sun allows me to do it (they have not yet, but it takes months), and I was told the SOC microcode stored PROM has debugging on, so it is slower and I don't know if SOC+ works at all without that microcode. You can put the microcode in yourself if you extract it from Solaris drivers, there is a simple define in the driver which tells you whether it is provided or not. Second issue is that I could not get scatter-gather to work (the SOC+ manual speaks about it briefly, but does not go into details and all my attempts (there were just a few due to limited time to play with A5000) failed). In fact, Solaris driver (at least the SOC) does not use scatter-gather either, but due to different properties of Solaris buffer cache it is not that big issue (average contiguous buffer in Solaris is much larger than in Linux). Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.3 on a sparc64 machine (3958.37 BogoMips) ___
Re: Disksuite, Sun SSA & Linux (Re: "ckraid" with raidtools)
> > > I'm assuming these were done on SparcLinux, or are the FC/FCAL interfaces > available on X86 also? The drivers/fc4 tree currently works only on SPARCs, as the only low-level drivers written are soc and socal (both SBUS adapters), but I've been trying to separate card specific code from protocol stuff. Of course, it is not fully generic, as it was written from knowledge about just two similar cards, but if new drivers are to be written, the "generic" layer can be fixed/extended. The fcal.c high level driver is also "generic" in that it should work on any low-level adapter driver for any FC-AL topology (well, currently it does not support fabrics). But some other cards, like Qlogic FC, don't take such approach, as they do all the FC4 protocol housekeeping in hardware and are driven as a normal SCSI card with a lot of targets. So, currently, if you want to hook SSA to something, you have to use SPARC box, as I think QlogicFC won't be able to drive point-to-point and SSA specific addressing scheme. For FCAL arrays, like A5?00, you can either hook them to some PCI machine using QlogicFC, or hook them to SOC+ on SBUS. Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.3 on a sparc64 machine (3958.37 BogoMips) ___
Re: Disksuite, Sun SSA & Linux (Re: "ckraid" with raidtools)
> > > What did you use for the FCAL interface? For SSA (that's point-to-point, not AL), I used fc4, soc, pluto for A5000 (and other FCAL disks), I used fc4, socal, fcal Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.3 on a sparc64 machine (3958.37 BogoMips) ___
Re: Disksuite, Sun SSA & Linux (Re: "ckraid" with raidtools)
> > Hmm...Disksuite for Linux...wonder if Sun has thought of that? > > Has anyone tried hanging a Sun SSA or A5000 off of a Linux > box using raidtools? It seems like Sun *has* thought of > this one...this came from someone at Sun: Yep, I'm using full SSA (29 disks) with Linux, raid0 for more than a year and have been testing the A5000 driver with about 5 FC disks in it. Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.3 on a sparc64 machine (3958.37 BogoMips) ___
Re: FC-AL support
> > > > > > > The Sparc port has full support for point to point and full fc-al > > > > fibre channel. This is utilizing Sun's SOC and SOCAL adapter cards. > > > > > > > > > > What about Jaycor (JNI) cards ? They seem to simulate a SCSI > > > interface, but with up to 255 disks connected :-) > > > > 255? Strange. Does it have dual Fibre Channels? Even than that number is > > strange. AFAIK, AL topology limit is 126 devices arbitrating on the loop, > > and one has to be the HBA... > > > > You can stack more on top of the local loop entities if you have fabric > support. Yes, I know, but if you have fabric in (not supported yet by drivers/fc4), I don't know where 255 comes from. You could have thousands of disks behind fabric. Cheers, Jakub ___ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.1 on a sparc64 machine (3958.37 BogoMips) ___
Re: FC-AL support
> > The Sparc port has full support for point to point and full fc-al > > fibre channel. This is utilizing Sun's SOC and SOCAL adapter cards. > > > > What about Jaycor (JNI) cards ? They seem to simulate a SCSI > interface, but with up to 255 disks connected :-) 255? Strange. Does it have dual Fibre Channels? Even than that number is strange. AFAIK, AL topology limit is 126 devices arbitrating on the loop, and one has to be the HBA... Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.1 on a sparc64 machine (3958.37 BogoMips) ___
Re: FC-AL support
> > Hi, > > it might be the wrong list(s) to ask, but I there is still a chance that > someone can help me. > > Does anybody has some experiences with 'Fibre Channel' and know, if > there is some support under Linux? I've just commited to vger CVS drivers for Sun SBUS SOC+ Fibre Channel card, SOC driver is there already for a couple of years. >From the remote side, I currently support SparcSTORAGE Array as a particular example of Point-to-Point topology and Arbitrated Loop topology of any kind. Have a look into drivers/fc4/ and drivers/scsi/ BTW: Actually, there are two types of Fibre Channel drivers: 1) QLogicFC and similar, where it looks like a typical SCSI card to the OS, and FC communication is driven by the card 2) SOC+,SOC,perhaps some Emulex, etc., where you have to talk Fibre Channel protocol to it. drivers/fc4 was created for exactly this kind of devices. Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.1 on a sparc64 machine (3958.37 BogoMips) ___
Re: Sun disklabels (Was: Re: RELEASE: RAID-0,1,4,5 patch...)
> You need to start using data at cylinder 1 on all disks or it will get > nuked. It doesn't happen on the first disk because ext2 skips some > space at the beginning of the volume. > > Swap space has the same problem, you cannot start it at cyliner 0. > So if RAID adds a workaround for it, the problem is still there just > in another case. The solution is to educate people that they need to > be careful when they partition their disks. I don't agree with this solution. New SWAPSPACE2 handles this well (it can sit at cylinder 0), and IMHO RAID should behave like that as well, ie. leave first 1k/4k/8k (someone decide) untouched and not part of the actual raid. Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ UltraLinux | http://ultra.linux.cz/ | http://ultra.penguin.cz/ Linux version 2.2.0 on a sparc64 machine (3958.37 BogoMips) ___
Re: raid1 and sparc-linux
> > I've been trying to set up a raid0 and raid1 array on a sparc linux > box. I was able to build raidtools 0.41 and can initialize the array for > > raid1. Every time I go to mdadd the devices I get the /dev/md0: Invalid > argument for the madd , or mdrun command. I rebuild the raidtools and > did the make install_dev at least twice. No change. I retryed on a SS5 > running kernel rls 2.1.127 and 2.0.35 , a little bit better. the error Just to make sure, are you using sun disklabels? If yes, then you have to make sure the sdb1 and sdc1 partitions don't start at cylinder 0, as raidtools don't leave space at the beginning like ext2/new swap/ufs/isofs/..., so you can smash your partition tables... Cheers, Jakub _______ Jakub Jelinek | [EMAIL PROTECTED] | http://sunsite.mff.cuni.cz Administrator of SunSITE Czech Republic, MFF, Charles University ___ Ultralinux - first 64bit OS to take full power of the UltraSparc Linux version 2.1.128 on a sparc64 machine (498.80 BogoMips). ___