Re: [OpenIndiana-discuss] Only one SSD detected on expander
I'm pretty sure I tried that, but now I can't be sure. I've already returned the drives and ordered the 840 Pros, which are supposed to have WWNs. If it turns out they don't, I'll give this a try before I return them, too. :-/ Thanks! -Scott On Oct 21, 2012, at 7:04 AM, carl brunning ca...@cblinux.co.uk wrote: Not sure if this help you But i had this problem on nexenta and the fix was the following in the /kernel/drv/mpt_sas.conf i unrem the line with mpxio-disable=yes; then reboot the system this fixed it for me and then i could see all my ssd when before only one show in the gui list this was for Samsung disk as well hope it is the same problem for you thanks CArl -Original Message- From: Robin Axelsson [mailto:gu99r...@student.chalmers.se] Sent: 21 October 2012 12:03 To: openindiana-discuss@openindiana.org Subject: Re: [OpenIndiana-discuss] Only one SSD detected on expander The worst case would be if the firmware image is signed and encrypted, which I doubt. There is a workaround for that too if that would be the case anyway. I'm not sure whether those firmware images have a real file system (such as yaffs for Android devices) or are just a solid binary file. Here are a few pages that I found: http://hexblog.com/files/recon%202010%20Skochinsky.pdf http://sviehb.wordpress.com/2011/09/06/reverse-engineering-an-obfuscated-firmware-image-e01-unpacking/ http://www.nslu2-linux.org/wiki/HowTo/RecoverFromABadFlashUsingJTAG On 2012-10-19 14:11, James C. McPherson wrote: On 19/10/12 09:29 PM, Udo Grabowski (IMK) wrote: On 19/10/2012 11:48, Robin Axelsson wrote: Isn't it possible to somehow make the drive dump the firmware somehow, edit it with a HEX editor (and recalculate firmware checksums) and flash the modified .bin file back to the drive? I guess that the WWN must be found in the firmware somewhere. Usually, you can read and write drive firmware with fwflash. You should be able to write new firmware, sure - my colleagues did add support for flashing sd(7d) to fwflash. I would be very, very wary of doing so, however, unless you have documentation from your disk manufacturer which (1) gives you the structure and instructions definitions, and (2) lets you verify that you have a valid image. Reading firmware from the disk is not supported with fwflash. http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/fwfl ash/plugins/transport/common/sd.c#125 James C. McPherson -- Solaris kernel software engineer, system admin and troubleshooter http://www.jmcpdotcom.com/blog Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] Only one SSD detected on expander
FWIW, this is what Samsung tech support had to say about the SSD 840 Pro: The next generation Samsung SSD 840 Pro series will have WWN. The Samsung 840 Pro is aimed for customers who need to use them on servers which is why only the 840 Pro will have the WWN identifier. Guess I'll get in line for a few and find out. -Scott On Oct 17, 2012, at 11:27 AM, Scott Marcy o...@mscott.org wrote: I am returning the 4 drives I planned to use in the expanders, but will keep the two I'm using as my mirrored boot pool. My intention was to use these are ZIL and L2ARC drives, so I can do without them for now. I've had very good luck with these Samsung drives in non-SAS usages—quite a bit more reliable in my (admittedly quite limited) experience than the SanForce-based SSDs. Here's hoping they've fixed this on the 840 Pro models. Thank you all for your help. I learned something new and now understand why it doesn't work. :-) -Scott On Oct 17, 2012, at 11:18 AM, Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Wed, 17 Oct 2012, Scott Marcy wrote: I called Samsung and they basically told me there was nothing they could do. The guy I spoke with said the 830s weren't intended to be used in servers. (He did seem to understand what I was talking about, which was actually more than I expected from simply picking up the phone and getting transferred twice to get to the right department. So at least kudos to Samsung there.) Maybe you should try to return these and get your money back. The problem does not seem to be specific to use in a server. The physical block size also seems to be reported incorrectly. Samsung cut corners by not taking the time to give each device a unique addressable ID as part of their manufacturing process. It may even be that a step was accidentally skipped in the manufacturing process (to make quotas) and that some devices are correctly configured from the factory while others are not. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] Only one SSD detected on expander
So one more data point that seems to suggest that I'm getting duplicate names from these drives. When I attach at least one of the Samsung SSDs to each expander (front and back), I get this: $ cfgadm -al Ap_Id Type Receptacle Occupant Condition c6 scsi-sas connectedconfigured unknown c6::es/ses0ESI connectedconfigured unknown c6::smp/expd0 smp connectedconfigured unknown c6::w5002538043584d30,0disk-pathconnectedconfigured unknown -- SSD c6::w5000c500513a0efe,0disk-pathconnectedconfigured unknown c6::w5000c5005168b2f0,0disk-pathconnectedconfigured unknown c6::w5000c5005169a612,0disk-pathconnectedconfigured unknown c6::w5000c5005169cc6a,0disk-pathconnectedconfigured unknown c6::w5000c500515199bd,0disk-pathconnectedconfigured unknown c6::w5000c500517042d9,0disk-pathconnectedconfigured unknown c8 scsi-sas connectedconfigured unknown c8::es/ses1ESI connectedconfigured unknown c8::smp/expd1 smp connectedconfigured unknown c8::w5002538043584d30,0disk-pathconnectedconfigured unknown -- SSD c9 scsi-sas connectedunconfigured unknown sata0/0::dsk/c4t0d0disk connectedconfigured ok sata0/1::dsk/c4t1d0disk connectedconfigured ok sata0/2sata-portemptyunconfigured ok sata0/3sata-portemptyunconfigured ok sata0/4sata-portemptyunconfigured ok sata0/5sata-portemptyunconfigured ok Note that both c6 and c8 have the exact same name for these drives. I *CAN* access the two drives separately, but why on earth am I getting these duplicates? This means I can only reliably access one Samsung SSD per expander. Thanks. -Scott On Oct 16, 2012, at 7:29 PM, Scott Marcy o...@mscott.org wrote: Hi all, I'm new to the list and OI, but I've been running a Solaris box with some version of ZFS on it for a couple years. I recently acquired a Supermicro 6047R-E1R36N. It has an X9DRi-LN4F+ motherboard, dual E5-2620s and 48GB of RAM. I soon learned that I needed a different HBA (it comes with an LSI 2108-based card that doesn't have any JBOD passthrough mode), so I replaced that with an LSI SAS9211-8i, flashed with the latest IT firmware. Being new to SAS expanders (which this server has two of), I quickly learned that some older 2TB drives I had on my older Solaris box simply will not work reliably on the expander. (They would very quickly lock up the entire file system once the load got heavy.) However, I've had much better luck (so far) with some newer 3TB SATA drives (Seagate ST3000DM001). I'm still not convinced that these are going to be reliable in the long run, having now read about a few SATA-on-SAS expander horror stories (and my own experience with the 2TB drives), but this isn't a production system (yet) so I'm willing to see how it goes. But the one problem I can't seem to figure out are the Samsung 830 SSD drives I have for ZIL and ARC. I have 2 mirrored 128GB 830s as the boot pool, but they're on the built-in SATA ports, not the expander ports. Then I have four other 128GB 830s which I installed into 3.5 hot-swap adapters and these are the ones causing problems. The HBA sees the drives just fine—they're listed during POST and I can access them via the card's built-in BIOS. But once OI (151a5) is up and running, it only sees ONE of the 830s. And here's where it gets weird: if I remove all but one of the SSDs from the expander ports, it will see whichever one SSD is connected—it does not matter which port the drives are on. Also, regardless of the port, the attachment path always seems to be the same in the OS. It's like the OS thinks all four drives are the exact same physical drive. $ cfgadm -al Ap_Id Type Receptacle Occupant Condition c6 scsi-sas connectedconfigured unknown c6::es/ses0ESI connectedconfigured unknown c6::smp/expd0 smp connectedconfigured unknown c6::w5002538043584d30,0disk-pathconnectedconfigured unknown -- this is the SSD c6::w5000c500513a0efe,0disk-pathconnectedconfigured unknown c6::w5000c5005168b2f0,0disk-pathconnectedconfigured unknown c6::w5000c5005169a612,0disk-pathconnectedconfigured unknown c6::w5000c5005169cc6a,0disk-pathconnectedconfigured unknown c6::w5000c500515199bd,0
Re: [OpenIndiana-discuss] Only one SSD detected on expander
Ah, I see now. Yes, all the drives report the same LU WWN Device Id in smartctl, and I see the same constant WWN reported online (for example, http://blog.gmane.org/gmane.linux.utilities.smartmontools.database/page=7). I'll see what Samsung has to say. Thanks! This at least gives me more to work from. -Scott On Oct 17, 2012, at 10:13 AM, Rich rerc...@acm.jhu.edu wrote: SATA devices are not required to provide a WWN - I would guess that they're providing the same WWN, and as a result, your SAS devices are becoming sad. If you could tell the expander(s) to ignore that and assign one, that'd be neat. If there's a firmware update to fix it, even better. If you want to shell out for SAS - SATA interposers, that would probably also solve it, though I can't swear to anything, but that'd probably be more expensive than returning + replacing the SSDs. Don't know. If it were me, I'd complain to Samsung support and see what you got back. - Rich On Wed, Oct 17, 2012 at 11:13 AM, Scott Marcy o...@mscott.org wrote: Well, I enabled multipath support with 'stmsboot -e -D mpt' and rebooted and now it seems I have two paths to what is no doubt two physical devices: $ mpathadm list LU /dev/rdsk/c7t5002538043584D30d0s2 Total Path Count: 2 -- Unlikely that this is correct Operational Path Count: 2 /dev/rdsk/c7t5000C5005168B2F0d0s2 Total Path Count: 1 Operational Path Count: 1 /dev/rdsk/c7t5000C5005169A612d0s2 Total Path Count: 1 Operational Path Count: 1 /dev/rdsk/c7t5000C500517042D9d0s2 Total Path Count: 1 Operational Path Count: 1 /dev/rdsk/c7t5000C5005169CC6Ad0s2 Total Path Count: 1 Operational Path Count: 1 /dev/rdsk/c7t5000C500515199BDd0s2 Total Path Count: 1 Operational Path Count: 1 /dev/rdsk/c7t5000C500513A0EFEd0s2 Total Path Count: 1 Operational Path Count: 1 $ mpathadm show LU /dev/rdsk/c7t5002538043584D30d0s2 Logical Unit: /dev/rdsk/c7t5002538043584D30d0s2 mpath-support: libmpscsi_vhci.so Vendor: ATA Product: SAMSUNG SSD 830 Revision: 3B1Q Name Type: unknown type Name: 5002538043584d30 Asymmetric: no Current Load Balance: none Logical Unit Group ID: NA Auto Failback: on Auto Probing: NA Paths: Initiator Port Name: w500605b0054819b0 Target Port Name: w5002538043584d30 Override Path: NA Path State: OK Disabled: no Initiator Port Name: w500605b0054819b0 Target Port Name: w5002538043584d30 Override Path: NA Path State: OK Disabled: no Target Ports: Name: w5002538043584d30 Relative ID: 0 And cfgadm still only shows one SSD on each expander, when I actually have two on each expander. $ cfgadm -al Ap_Id Type Receptacle Occupant Condition c6 scsi-sas connectedconfigured unknown c6::es/ses0ESI connectedconfigured unknown c6::smp/expd0 smp connectedconfigured unknown c6::w5002538043584d30,0disk-pathconnectedconfigured unknown c6::w5000c500513a0efe,0disk-pathconnectedconfigured unknown c6::w5000c5005168b2f0,0disk-pathconnectedconfigured unknown c6::w5000c5005169a612,0disk-pathconnectedconfigured unknown c6::w5000c5005169cc6a,0disk-pathconnectedconfigured unknown c6::w5000c500515199bd,0disk-pathconnectedconfigured unknown c6::w5000c500517042d9,0disk-pathconnectedconfigured unknown c8 scsi-sas connectedconfigured unknown c8::es/ses1ESI connectedconfigured unknown c8::smp/expd1 smp connectedconfigured unknown c8::w5002538043584d30,0disk-pathconnectedconfigured unknown c9 scsi-sas connectedunconfigured unknown I also wonder if there might be an issue with duplicate initiator-port names: $ mpathadm list initiator-port Initiator Port: w500605b0054819b0 Initiator Port: iqn.1986-03.com.sun:01:bb4db0fb.507ec650,402a00ff Initiator Port: w500605b0054819b0 Although I'm definitely venturing outside my knowledge base here--maybe this is expected and correct. (I assume each initiator-port represents an expander, but both expanders are connected to the one HBA, one to each SFF-8087 port. Also, I do have comstar running, which I expect explains the iSCSI initiator
Re: [OpenIndiana-discuss] Only one SSD detected on expander
I am returning the 4 drives I planned to use in the expanders, but will keep the two I'm using as my mirrored boot pool. My intention was to use these are ZIL and L2ARC drives, so I can do without them for now. I've had very good luck with these Samsung drives in non-SAS usages—quite a bit more reliable in my (admittedly quite limited) experience than the SanForce-based SSDs. Here's hoping they've fixed this on the 840 Pro models. Thank you all for your help. I learned something new and now understand why it doesn't work. :-) -Scott On Oct 17, 2012, at 11:18 AM, Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Wed, 17 Oct 2012, Scott Marcy wrote: I called Samsung and they basically told me there was nothing they could do. The guy I spoke with said the 830s weren't intended to be used in servers. (He did seem to understand what I was talking about, which was actually more than I expected from simply picking up the phone and getting transferred twice to get to the right department. So at least kudos to Samsung there.) Maybe you should try to return these and get your money back. The problem does not seem to be specific to use in a server. The physical block size also seems to be reported incorrectly. Samsung cut corners by not taking the time to give each device a unique addressable ID as part of their manufacturing process. It may even be that a step was accidentally skipped in the manufacturing process (to make quotas) and that some devices are correctly configured from the factory while others are not. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
[OpenIndiana-discuss] Only one SSD detected on expander
Hi all, I'm new to the list and OI, but I've been running a Solaris box with some version of ZFS on it for a couple years. I recently acquired a Supermicro 6047R-E1R36N. It has an X9DRi-LN4F+ motherboard, dual E5-2620s and 48GB of RAM. I soon learned that I needed a different HBA (it comes with an LSI 2108-based card that doesn't have any JBOD passthrough mode), so I replaced that with an LSI SAS9211-8i, flashed with the latest IT firmware. Being new to SAS expanders (which this server has two of), I quickly learned that some older 2TB drives I had on my older Solaris box simply will not work reliably on the expander. (They would very quickly lock up the entire file system once the load got heavy.) However, I've had much better luck (so far) with some newer 3TB SATA drives (Seagate ST3000DM001). I'm still not convinced that these are going to be reliable in the long run, having now read about a few SATA-on-SAS expander horror stories (and my own experience with the 2TB drives), but this isn't a production system (yet) so I'm willing to see how it goes. But the one problem I can't seem to figure out are the Samsung 830 SSD drives I have for ZIL and ARC. I have 2 mirrored 128GB 830s as the boot pool, but they're on the built-in SATA ports, not the expander ports. Then I have four other 128GB 830s which I installed into 3.5 hot-swap adapters and these are the ones causing problems. The HBA sees the drives just fine—they're listed during POST and I can access them via the card's built-in BIOS. But once OI (151a5) is up and running, it only sees ONE of the 830s. And here's where it gets weird: if I remove all but one of the SSDs from the expander ports, it will see whichever one SSD is connected—it does not matter which port the drives are on. Also, regardless of the port, the attachment path always seems to be the same in the OS. It's like the OS thinks all four drives are the exact same physical drive. $ cfgadm -al Ap_Id Type Receptacle Occupant Condition c6 scsi-sas connectedconfigured unknown c6::es/ses0ESI connectedconfigured unknown c6::smp/expd0 smp connectedconfigured unknown c6::w5002538043584d30,0disk-pathconnectedconfigured unknown -- this is the SSD c6::w5000c500513a0efe,0disk-pathconnectedconfigured unknown c6::w5000c5005168b2f0,0disk-pathconnectedconfigured unknown c6::w5000c5005169a612,0disk-pathconnectedconfigured unknown c6::w5000c5005169cc6a,0disk-pathconnectedconfigured unknown c6::w5000c500515199bd,0disk-pathconnectedconfigured unknown c6::w5000c500517042d9,0disk-pathconnectedconfigured unknown c8 scsi-sas connectedconfigured unknown c8::es/ses1ESI connectedconfigured unknown c8::smp/expd1 smp connectedconfigured unknown c9 scsi-sas connectedunconfigured unknown sata0/0::dsk/c4t0d0disk connectedconfigured ok sata0/1::dsk/c4t1d0disk connectedconfigured ok sata0/2sata-portemptyunconfigured ok sata0/3sata-portemptyunconfigured ok sata0/4sata-portemptyunconfigured ok sata0/5sata-portemptyunconfigured ok $ format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t0d0 ATA-SAMSUNGSSD830-3B1Q cyl 19932 alt 2 hd 224 sec 56 /pci@0,0/pci15d9,626@1f,2/disk@0,0 1. c4t1d0 ATA-SAMSUNGSSD830-3B1Q cyl 19932 alt 2 hd 224 sec 56 /pci@0,0/pci15d9,626@1f,2/disk@1,0 2. c7t5000C500513A0EFEd0 ATA-ST3000DM001-9YN1-CC4B-2.73TB /scsi_vhci/disk@g5000c500513a0efe 3. c7t5000C5005168B2F0d0 ATA-ST3000DM001-9YN1-CC4B-2.73TB /scsi_vhci/disk@g5000c5005168b2f0 4. c7t5000C5005169A612d0 ATA-ST3000DM001-9YN1-CC4B-2.73TB /scsi_vhci/disk@g5000c5005169a612 5. c7t5000C5005169CC6Ad0 ATA-ST3000DM001-9YN1-CC4B-2.73TB /scsi_vhci/disk@g5000c5005169cc6a 6. c7t5000C500515199BDd0 ATA-ST3000DM001-9YN1-CC4B-2.73TB /scsi_vhci/disk@g5000c500515199bd 7. c7t5000C500517042D9d0 ATA-ST3000DM001-9YN1-CC4B-2.73TB /scsi_vhci/disk@g5000c500517042d9 8. c7t5002538043584D30d0 ATA-SAMSUNG SSD 830-3B1Q-119.24GB /scsi_vhci/disk@g5002538043584d30 This last one is the SSD. If I remove all the SSDs from the expander ports, disk #8 does not show up (nor does it show in cfgadm). If I put one SSD into any expander port, disk 8 is back. If I add additional SSDs, I still only have disk 8. If I remove the first SSD (while others are still attached), disk 8 still appears. In
