Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
The problems you had with the x8dtn, did the affect the x8dtn+-o that you know of, or just the -6? I'm thinking of building a system around this due to the PCI-X so I can use the AOC-SAT2-MV8. Any thoughts? Thanks. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
On 31 Jan 2010, Jacob Ritorto wrote: I spent *so* many hours looking for that firmware.� Would you please post the link?� Did the firmware dl you found come with fcode? Running blade 2000 here (SPARC). Well, I can't say for sure it's the right firmware for your device. I grabbed the LSI firmware from SuperMicro's FTP site at: --- ftp://ftp.supermicro.com/driver/SAS/LSI/1064_1068/IT/Firmware/B3/L8i/1.26.01/ which appears to be a little newer than even what I grabbed before. I'm not familiar with fcode, so I can't say whether it did. The two things that actually flashed for me were the firmware image for the 1068E itself and the MegaSAS BIOS/ROM image. Both version numbers changed once I had done the update. Now having said all of that, things look a bit flaky still on my end. I'm running a scrub which started earlier today and seems to be hung. I'm seeing a status of: --- pool: san state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: scrub in progress for 20h16m, 21.68% done, 73h15m to go config: NAME STATE READ WRITE CKSUM san ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c7t5d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c7t8d0 ONLINE 0 0 0 2.69M repaired c7t9d0 ONLINE 0 0 0 c7t10d0 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 c7t11d0 ONLINE 0 0 0 c7t12d0 ONLINE 0 0 0 c7t13d0 ONLINE 0 0 0 c7t14d0 ONLINE 0 0 0 c7t15d0 ONLINE 0 0 0 c7t16d0 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 c7t17d0 ONLINE 0 0 0 c7t18d0 ONLINE 0 0 0 c7t19d0 ONLINE 0 0 0 c7t20d0 ONLINE 0 0 0 c7t21d0 ONLINE 0 0 0 c7t22d0 ONLINE 0 0 0 spares c7t23d0AVAIL c7t24d0AVAIL c7t25d0AVAIL errors: No known data errors --- and errors like this: --- Jan 31 20:15:21 goliath scsi: [ID 243001 kern.warning] WARNING: /p...@0,0/pci1002,5...@c/pci15d9,a...@0 (mpt0): Jan 31 20:15:21 goliath mpt_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31123000 Jan 31 20:15:23 goliath scsi: [ID 365881 kern.info] /p...@0,0/pci1002,5...@c/pci15d9,a...@0 (mpt0): Jan 31 20:15:23 goliath Log info 0x31123000 received for target 13. --- With no problems at all, that scrub I don't think takes nearly that long (I think it was less than 12 hours previously) and the percentage is barely moving, although it is increasing. Even still, the exported volumes still appear to be working via iSCSI for now, which is a plus. -- Mark Nipper ni...@bitgnome.net (XMPP) +1 979 575 3193 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
I did see that and confirmed the support has made it into the 130 release I'm testing with. However, the WD10EARS does not expose 4k sectors to the outside world, so it is not identified as supporting it. Correct alignment, to ensure best performance of the internal translation, seems to be the only option available. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
The WD10EARS disks don't work well. I had too many issues with timeouts that disappeared when replacing them with ST32000542AS drives. My next challenge is to get the LSI 3081 to boot off the disk I want it to, and then to get multipath functional. Has anyone else had issues with the LSI IT mode firmware changing the disk order so the bootable disk is no longer the one booted from with expanders? It boots with only two disks installed(bootable zfs mirror). Add some more and the target boot disk moves to one of them. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Update: For the WD10EARS, the blocks appear to be aligned on the 4k boundary when zfs uses the whole disk (whole disk as EFI partition). Part TagFlag First Sector Size Last Sector 0usrwm256 931.51Gb 1953508750 calc256*512/4096=32 Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Mark Bennett mark.benn...@public.co.nz writes: Update: For the WD10EARS, the blocks appear to be aligned on the 4k boundary when zfs uses the whole disk (whole disk as EFI partition). Part TagFlag First Sector Size Last Sector 0usrwm256 931.51Gb 1953508750 calc256*512/4096=32 I'm afraid this isn't enough. if you enable compression, any ZFS write can be unaligned. also, if you're using raid-z with an odd number of data disks, some of (most of?) your stripes will be misaligned. ZFS needs to use 4096 octets as the basic block to fully exploit the performance of these disks. -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
See also PSARC 2008/769 which considers 4 KB blocks for the entire OS in a phased approach. http://arc.opensolaris.org/caselog/PSARC/2008/769/inception.materials/design_doc -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Hey Mark, I spent *so* many hours looking for that firmware. Would you please post the link? Did the firmware dl you found come with fcode? Running blade 2000 here (SPARC). Thx Jake On Jan 26, 2010 11:52 AM, Mark Nipper ni...@bitgnome.net wrote: It may depend on the firmware you're running. We've got a SAS1068E based card in Dell R710 at... Well, I may have misspoke. I just spent a good portion of yesterday upgrading to the latest firmware myself (downloaded from SuperMicro's FTP, version 1.26.00 also; after I figured out I had to pass the -o option to mptutil to force the flash since it was complaining about a mismatched card or some such) and I thought that the machine had locked up again later in the day yesterday because I couldn't ssh into the machine. To my surprise though, I was able to log into the machine just fine this morning directly on the console from the command line. It seems the snv_125 bug with /dev/ptmx bit me (the error: /dev/ptmx: Permission denied problem that required me tracking down the release notes for snv_125 to figure out the problem) and the server was happy otherwise. More importantly, the zpool activity had all finished and I have three clean spares again! Normally this amount of I/O would have totally killed the machine! So somewhere between upgrading the firmware to the latest version and upgrading to snv_131, it looks like the problem may have actually been addressed. I'm guardedly optimistic at this point, given the previous problems I've had so far with this on-board controller. Interesting to hear someone else with the same chip but on an expansion card has no problems (but was with the on-board chip). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zf... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
I do not know if this has anything to do with some of the problems you're seeing. however, there is definitely an issue using /some/ LSI controllers on Supermicro boards. I just rencetly had an issue with using LSI3442 (the next gen over the 1068E chipset) on a Supermicro MB (X8DTN-6). Anyhow, if the LSI board is in the system, the drives are recognized and everything works. HOWEVER, if you have to use the SATA CD-rom/DVD drive - it will not boot from the drive. For whatever strange reason, supermico and I determined that it just didn't like the LSI SAS Controller in the system which caused this. We have no idea why a SAS controller would affect a SATA boot. . . Another issue we have found is the latest x8dtn board does not play nice with the latest release of Solaris (at least for us). It will work fine up to snx129. However 131 (or upgrading from 129 to 131) will fail on boot after GRUB. It'll just cycle/reboot at that stage over and over. Not sure what the issue is between 129 and 131 but that is what we've found on our Supermicro MB and the latest OpenSolaris version. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
I'm looking into the alignment implications for the WD10EARS disks. It may explain my issues. I seem to recall boot issues in some of the LSI release notes affecting other boot devices. I think it takes over boot responsibility. I've encountered this sort of issue over the years with many scsi cards. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Thanks for your answer. I asked primarily because of the mpt timeout issues I saw on the list. Hi Arnaud, I am looking into the LSI SAS 3081 as well. My current understanding with mpt issues is that the sticky part of these problems is rather related to multipath features, that is using port multipliers or sas expanders. Avoiding these one should be fine. I am quite a newbie though. Just judging from what I read here. Regards, Tonmaus -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
On 28/01/10 09:36 PM, Tonmaus wrote: Thanks for your answer. I asked primarily because of the mpt timeout issues I saw on the list. Hi Arnaud, I am looking into the LSI SAS 3081 as well. My current understanding with mpt issues is that the sticky part of these problems is rather related to multipath features, that is using port multipliers or sas expanders. Avoiding these one should be fine. I am quite a newbie though. Just judging from what I read here. I do not think that you are reading the data correctly. The issues that we have seen via this list and storage-discuss have implicated downrev firmware on cards, and the various different disk drives that people choose to attach to those cards. The use of SAS expanders with mpt-based cards is *not* an issue. The use of MPxIO with mpt-based cards is *not* an issue. Both MPxIO and SAS Expanders are an essential part of the whole picture that is the SS7000 appliance series, as well as the J4x00 series. Personally, I'm quite happy with the LSISAS3081E that I have installed in my system, with the attached 320Gb consumer-grade SATA2 disks. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Hi James I do not think that you are reading the data correctly. The issues that we have seen via this list and storage-discuss have implicated downrev firmware on cards, and the various different disk drives that people choose to attach to those cards. Thanks for pointing that out. I have indeed noticed such reports but I didn't see any specific plans or ack's to take these issues from mpt. Thus the question is if these reports justify the assumption that there was anything wrong with mpt in general. The use of SAS expanders with mpt-based cards is *not* an issue. The use of MPxIO with mpt-based cards is *not* an issue. I didn't want to make the point of denying some of mpt's core features. I just saw a couple of reports that involved SAS Expanders, specifically those based on LSI silicon, reported under the mpt problem umbrella, and I understood that these issues were quite obstinate. Personally, I'm quite happy with the LSISAS3081E that I have installed in my system, with the attached 320Gb consumer-grade SATA2 disks. Excellent. That's encouraging. I am planning a similar configuration, with WD RE3 1 TB disks though. Regards, Tonmaus -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
My experience was different again. I have the same timeout issues with both the LSI and Supermicro cards in IT mode. IR mode on the Supermicro card didn't solve the problem, but seems to have reduced it . Server has 1 x 16 bay chassis and 1 x 24 bay chassis (both use expander) test pool has 24 x WD10EARS in 6 disk vdev sets, 1 on 16 bay and 2 on 24 bay. Mark -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Are you using the latest IT mode firmware? (1.26.00 I think, listed above and without checking mine using AOC-USAS-L8i which uses same controller) Also, I noticed you're using 'EARS' series drives. Again, I'm not sure if the WD10EARS drives suffer from a problem mentioned in these posts, but it might be worth looking into -- especially the last link: 1. On synology site, seems like older 4-platter 1.5TB EADS OK (WD15EADS-00R6B0), but newer 3 platter EADS have problems (WD15EADS-00P8B0): http://forum.synology.com/enu/viewtopic.php?f=151t=19131sid=c1c446863595a5addb8652a4af2d09ca 2. A mac user has problems with WD15EARS-00Z5B1: http://community.wdc.com/t5/Desktop/WD-1-5TB-Green-drives-Useful-as-door-stops/td-p/1217/page/2 (WD 1.5TB Green drives - Useful as door stops) http://community.wdc.com/t5/Desktop/WDC-WD15EARS-00Z5B1-awful-performance/m-p/5242 (WDC WD15EARS-00Z5B1 awful performance) Cheers, Simon http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Are you using the latest IT mode firmware? (1.26.00 I think, listed above and without checking mine using AOC-USAS-L8i which uses same controller) Also, I noticed you're using 'EARS' series drives. Again, I'm not sure if the WD10EARS drives suffer from a problem mentioned in these posts, but it might be worth looking into -- especially the last link: 1. On synology site, seems like older 4-platter 1.5TB EADS OK (WD15EADS-00R6B0), but newer 3 platter EADS have problems (WD15EADS-00P8B0): http://forum.synology.com/enu/viewtopic.php?f=151t=19131sid=c1c446863595a5addb8652a4af2d09ca 2. A mac user has problems with WD15EARS-00Z5B1: http://community.wdc.com/t5/Desktop/WD-1-5TB-Green-drives-Useful-as-door-stops/td-p/1217/page/2 (WD 1.5TB Green drives - Useful as door stops) http://community.wdc.com/t5/Desktop/WDC-WD15EARS-00Z5B1-awful-performance/m-p/5242 (WDC WD15EARS-00Z5B1 awful performance) Cheers, Simon http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Also, I noticed you're using 'EARS' series drives. Again, I'm not sure if the WD10EARS drives suffer from a problem mentioned in these posts, but it might be worth looking into -- especially the last link: Aren't the EARS drives the first ones using 4k sectors? Does OpenSolaris support that properly yet? From what I've read using the 512-byte emulation mode in the drives is not good for performance (lots of read/modify/write), though I don't know whether that could cause these kind of problems. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
It may depend on the firmware you're running. We've got a SAS1068E based card in Dell R710 at the moment, connected to an external SAS JBOD, and we did have problems with the as shipped firmware. Well, I may have misspoke. I just spent a good portion of yesterday upgrading to the latest firmware myself (downloaded from SuperMicro's FTP, version 1.26.00 also; after I figured out I had to pass the -o option to mptutil to force the flash since it was complaining about a mismatched card or some such) and I thought that the machine had locked up again later in the day yesterday because I couldn't ssh into the machine. To my surprise though, I was able to log into the machine just fine this morning directly on the console from the command line. It seems the snv_125 bug with /dev/ptmx bit me (the error: /dev/ptmx: Permission denied problem that required me tracking down the release notes for snv_125 to figure out the problem) and the server was happy otherwise. More importantly, the zpool activity had all finished and I have three clean spares again! Normally this amount of I/O would have totally killed the machine! So somewhere between upgrading the firmware to the latest version and upgrading to snv_131, it looks like the problem may have actually been addressed. I'm guardedly optimistic at this point, given the previous problems I've had so far with this on-board controller. Interesting to hear someone else with the same chip but on an expansion card has no problems (but was with the on-board chip). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
As in they work without any possibility of mpt timeout issues? I'm at my wits end with a machine right now that has an integrated 1068E and is dying almost hourly at this point. If I could spend three hundred dollars or so and have my problems magically go away, I'd love to pull the trigger on one of these SAS3081E-R cards right now. Let me know if I can provide any more feedback on my existing 1068E problems. I followed all the threads, but nothing apparently really fixes the problem. I was running 2009.06 (111b I think) and I just updated to 131 earlier today with more of the same problems. My ZFS pool is in such a sad state, it takes an hour or so at most with all the resilvering happening before the machine goes to lunch. All of this on a SuperMicro motherboard and chassis with port expanders. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
It may depend on the firmware you're running. We've got a SAS1068E based card in Dell R710 at the moment, connected to an external SAS JBOD, and we did have problems with the as shipped firmware. However we've upgraded that, and _so far_ haven't had further issues. I didn't do the upgrade myself, so I'm not sure what it shipped with, but we're currently running: Main menu, select an option: [1-99 or e/p/w or 0 to quit] 1 Current active firmware version is 011a (1.26.00) Firmware image's version is MPTFW-01.26.00.00-IT LSI Logic x86 BIOS image's version is MPTBIOS-6.24.00.00 (2008.07.01) FCode image's version is MPT SAS FCode Version 1.00.49 (2007.09.21) We've got multiple ISCSI and NFS clients all running disk benchmarks in loops to produce load, as well as repeated scrubs on the server, and we've been running several days now without fault.We're going to give it several more days before we sign it off as stable tho! Regards, Tristan. On 26/01/2010 10:47 AM, Mark Nipper wrote: As in they work without any possibility of mpt timeout issues? I'm at my wits end with a machine right now that has an integrated 1068E and is dying almost hourly at this point. If I could spend three hundred dollars or so and have my problems magically go away, I'd love to pull the trigger on one of these SAS3081E-R cards right now. Let me know if I can provide any more feedback on my existing 1068E problems. I followed all the threads, but nothing apparently really fixes the problem. I was running 2009.06 (111b I think) and I just updated to 131 earlier today with more of the same problems. My ZFS pool is in such a sad state, it takes an hour or so at most with all the resilvering happening before the machine goes to lunch. All of this on a SuperMicro motherboard and chassis with port expanders. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
Thanks for your answer. I asked primarily because of the mpt timeout issues I saw on the list. I never experienced timeouts with my (personal) usas-l8i (lsi 1068e) but feared this issue might cause some problems with 3081. Anyway, thanks again. Arnaud -Message d'origine- De : james.mcpher...@sun.com [mailto:james.mcpher...@sun.com] De la part de James C. McPherson Envoyé : lundi 11 janvier 2010 23:05 À : Arnaud Brand Cc : zfs-discuss@opensolaris.org Objet : Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ? On 11/01/10 11:57 PM, Arnaud Brand wrote: According to various posts the LSI SAS3081E-R seems to work well with OpenSolaris. But I've got pretty chilled-out from my recent problems with Areca-1680's. Could anyone please confirm that the LSI SAS3081E-R works well ? Is hotplug supported ? Anything else I should know before buying one of these cards ? These cards work very well with OpenSolaris, and attach using the mpt(7d) driver - supports hotplugging and MPxIO too. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
According to various posts the LSI SAS3081E-R seems to work well with OpenSolaris. But I've got pretty chilled-out from my recent problems with Areca-1680's. Could anyone please confirm that the LSI SAS3081E-R works well ? Is hotplug supported ? It works well in Solaris 10 including hotplugging. -- Maurice Volaski, maurice.vola...@einstein.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?
On 11/01/10 11:57 PM, Arnaud Brand wrote: According to various posts the LSI SAS3081E-R seems to work well with OpenSolaris. But I've got pretty chilled-out from my recent problems with Areca-1680's. Could anyone please confirm that the LSI SAS3081E-R works well ? Is hotplug supported ? Anything else I should know before buying one of these cards ? These cards work very well with OpenSolaris, and attach using the mpt(7d) driver - supports hotplugging and MPxIO too. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
