Re: need SSD RAID controller advice
On 04/10/2015 05:21 PM, ToddAndMargo wrote: Hi All, I have a customer interested in high end workstation. She wants RAID 1 to protect her drives. She is also interested in Solid State Drives. (And she knows the difference between RAID and Backup.) What host controller would you guys recommend for a pair of Intel SSDSC2BW480A401, 480 GB 2.5 Internal Solid State Drives in RAID 1? Many thanks, -T Not to ask too stupid a question, but would a hybrid drive, such as http://www.seagate.com/internal-hard-drives/solid-state-hybrid/desktop-solid-state-hybrid-drive/?sku=ST2000DX001 not require TRIM and work with an hardware controller? -T -- ~~ Computers are like air conditioners. They malfunction when you open windows ~~
Re: need SSD RAID controller advice
Hi Kevin K! On 2015.05.06 at 18:52:42 -0500, Kevin K wrote next: On May 6, 2015, at 6:31 PM, Vladimir Mosgalin mosga...@vm10124.spb.edu wrote: Alternative to TRIM-aided GC is Idle Time Garbage Collection (which you could see on Plextor), which actively scans data in background, finds empty blocks (zeroes) and marks them as free. Well probably more complicated than that, but it's the idea. A quick search indicates that Idle TIme Garbage Collection is just another term for garbage collection. And I don’t see how it can work as well without TRIM as with TRIM, unless massively over provisioned. There is still the issue that, at the file system level, you may now that you just deleted a 1GB file that fully fills a whole lot of EBs. Without trim, the controller can’t know that the blocks aren’t in use until the OS attempts to write to those blocks. if the controller needs to make space, it will have to copy the stale data that it doesn’t know isn’t used anymore. Now, a continous write to a whole EB’s worth of data (still stored in DRAM on the drive), it can see that the whole block can be marked as unused, and freed later. I've posted the graph, you've seen it. You can find similar graphs on other sites exploring the matter (anandtech, tom's hardware and so on). There are drives which restore performance to original or near original without aid of TRIM. They are consumer-grade with typical 7-15% over provisioning, just like any other drive. Not enterprise models with 40-50% over provisioning. If you want graph with more drives there is one here http://www.xbitlabs.com/articles/storage/display/plextor-m5-pro-m5s_6.html You can clearly see that some drives can't restore performance without TRIM, and some restore most of it (up to 100% in case of M5P, which isn't here) after some idle even without TRIM. Of course, massive over provisioning is still a great feature of enteprise drives. There is performance degradation on any kind of drive even with the use of TRIM before GC is performed (explored here in detail: http://www.tomshardware.com/reviews/m5s-256gb-ssd-benchmark,3252-7.html) However it isn't so on massively over provisioned drives (Anandtech has excellent graphs of this in articles about enterprise SSD drives, how over provisioning lets drive to show high write IOPS numbers at any moment). Here is another example of drive restoring most of its performance without TRIM: http://www.anandtech.com/show/6337/samsung-ssd-840-250gb-review/11 (as you can see, it behaves in pretty complicated way, however it's clear that GC works very well even without TRIM). I’m not sure what the zero blocks you are referring to means. If you write a whole bunch of zeros, is it marked as unused? Even if contained in a file? You really can’t mark obsolete data as zeros since you can’t rewrite until you erase first. I think I used the wrong wording. I'm sure it would mark zeroes as unused (yes, even in a file - it just erases these blocks and unmaps them, when you read from unmapped LBA the drive returns you zeroes), but generally these algorithms are more complicated. Writing zeroes isn't recommended (http://www.lowlevelformat.info/zero-filling.aspx) - it might work for some SSDs, of course, but still risky. I don't really know the details of how TRIM-less GC works in complicated cases, but it's obvious from performance graphs that it really does work on certain drivers and doesn't work too good on others. Regarding the term Idle Time GC - yes it is used for describing any kind of background GC, whether TRIM-aided or not by some manufacturers. There is nothing I can do about that. I used it the way it's often used on some hardware sites and forums, but there are manufacturers who use this term in a different way... -- Vladimir
Re: need SSD RAID controller advice
On 07/05/15 03:51, Jeff Siddall wrote: On 05/06/2015 09:33 PM, ToddAndMargo wrote: Thank you. I will have to get over my phobia of software RAID. FWIW I have used SW RAID1 for years (maybe decades by now) on all sorts of systems. It has saved me many times and never burned me. If you are going to do anything in SW RAID1 is as easy/good as it gets. +1 ... Same experience here too. I've had mdadm RAID setups with both RAID1 and RAID5 where harddrive failures have happened. Recovered without any issues and no data loss. Performance wise ... I've not done anything testing in particular. But none of my setups have felt sluggish on normal usage. David S.
Re: need SSD RAID controller advice
Hi ToddAndMargo! On 2015.05.06 at 23:45:41 -0700, ToddAndMargo wrote next: Many thanks, -T Not to ask too stupid a question, but would a hybrid drive, such as http://www.seagate.com/internal-hard-drives/solid-state-hybrid/desktop-solid-state-hybrid-drive/?sku=ST2000DX001 not require TRIM and work with an hardware controller? I don't recommend using these at all, if you check the reviews, their performance isn't that much different from regular hard drive in many use cases. They make sense for notebooks but hardly for desktop. Regarding your question, it doesn't make much sense because you aren't going to get the kind of performance you get from SSD with degraded performance from this drive. For OS, it's regular hard drive (without TRIM support) with extra cache to speed up some operations. -- Vladimir
Re: need SSD RAID controller advice
On 04/14/2015 02:29 AM, Vladimir Mosgalin wrote: Hi ToddAndMargo! On 2015.04.13 at 18:26:49 -0700, ToddAndMargo wrote next: -T Wonder if there are any enterprise level SSD's that don't need TRIM? Tons. All Hitachi SSDs, OCZ Interpid and Saber, all Intel DC series SSDs (based on Intel controllers) and many more. Though I'm not sure that 1) you'll like the prices of enteprise SSDs and 2) for workstation (and even for many kinds of servers) there is absolutely no point in using these enterprise SSDs. The controller itself isn't more reliable, just flash chips endure more wear, but the difference (e.g. 100 TB endurance of normal SSD vs 20 PB endurance of enterprise SSD) makes a difference only for certain databases and cache (when data on SSD is constantly refreshed) workloads. So if you exclude the wear factor, the chance of failure depends on the controller/firmware and is about the same whether SSD was marked for enterprise or not. And you already got RAID in mind against these kind of failures anyway. For nearly all other cases, avoiding SandForce-based SSDs is about all you need to care about when picking SSD for systems without TRIM. I listed some model examples in the previous mail. Hi Vladimir, I have look all over Intel's specs for their new SSD DC S3510 series SSDs (older drives are harder to find): http://www.intel.com/content/www/us/en/solid-state-drives/ssd-dc-s3510-spec.html They do not say a thing about Garbage Collection and not having to use TRIM. They do say Data set management Trim command, which means they support TRIM. I have eMailed Intel's tech support twice over two weeks and have been ignored twice by them (which is typical). Also: in my research, I did find mention that TRIM did not effect lifespan -- TRIM only affected write speed, which would slow down to a crawl without it. Can you verify this? Thank you for helping me with this. -T -- ~~ Computers are like air conditioners. They malfunction when you open windows ~~
Re: need SSD RAID controller advice
On 05/06/2015 04:31 PM, Vladimir Mosgalin wrote: Hi ToddAndMargo! On 2015.05.05 at 23:05:36 -0700, ToddAndMargo wrote next: Hi Vladimir, I have look all over Intel's specs for their new SSD DC S3510 series SSDs (older drives are harder to find): http://www.intel.com/content/www/us/en/solid-state-drives/ssd-dc-s3510-spec.html They do not say a thing about Garbage Collection and not having to use TRIM. They do say Data set management Trim command, which means they support TRIM. Well, every modern SSD (or even eMMC device) supports TRIM. I looked at review of this SSD - it's somewhat strange, unlike Intel SSDs of previous generation it doesn't support Idle Garbage Collection. So, indeed, TRIM support would be very useful. Take a look at this graph http://img.fcenter.ru/imgmat/article/SSD/Intel_DC_S3500/207103.png You can see performance degradation right after writes (20-40% of original performance typically - note that this is pretty heavy test, amount of data written is twice the size of SSD). This is expected. Now, there are drives that support Idle Time Garbage Collection. Among these, Plextor M5 are most amazing - they restore to original performance without even need of TRIM at all. As for the other drives, after TRIM they restore performance. There are, however, SandForce controllers (Intel 530 in this example), where performance not just degrades a lot without TRIM, it doesn't restore to original values with TRIM (though, mind you, this is pretty heavy test, in real life this isn't that likely). Also: in my research, I did find mention that TRIM did not effect lifespan -- TRIM only affected write speed, which would slow down to a crawl without it. Can you verify this? TRIM does indirectly affects lifespan: the write speed degradation doesn't appear out of thin air just because you haven't TRIMMED it. It happens because free space becomes fragmented and SSD hardly has any free space left (in its opinion, because without TRIM the OS doesn't report which blocks are actually free and can be reused). But data on SSD can't be written (or, rather, erased) in small blocks, so in order to write new data on heavy fragmented device SSD controller has to move data (read from half-filled blocks and rearrange, writing to different place, to free whole blocks for new data). E.g. SSD might be able to read 4KB blocks but erase only 512KB blocks. Because of that, when SSD is used without TRIM for a long time, this process greatly increases amount of written data - you write 100MB, but SSD had to rearrange data to free bigger blocks and internally had written 300MB. This is called write amplification and it decreases lifespan of SSD much quicker. Also it makes your writes slow (as seen on graph) because at the time you think you've written 100MB SSD had to write 3 times more. Proper TRIM usage makes SSD aware of all the real free blocks (all free space in your filesystem), so it doesn't have to move data anymore and just writes new data there. It reduces write amplification coefficient - and thus prolongs lifespan. Not that I would worry about SSD lifespan even with huge write amplification, mind you. IMHO performance problem is more important. It doesn't TRIM itself that affects write speed or lifespan, it only aids Garbage Collection algorithms of SSD controller. Just keep in mind that getting bad linear write performance usually means big write amplification because SSD doesn't have much free blocks. Alternative to TRIM-aided GC is Idle Time Garbage Collection (which you could see on Plextor), which actively scans data in background, finds empty blocks (zeroes) and marks them as free. Well probably more complicated than that, but it's the idea. So given some idle time (SSD runs this type of GC only when it isn't too busy serving user requests, obviously), it is possible to free blocks for future writings even without OS reporting them using TRIM. There are other alternatives, e.g. SandForce controllers internally compress data to write less, you can say that they might have less-than 1 write amplification coefficient. However, this doesn't help when incompressible data is written. I'm sorry about initial confusion suggesting that Intel DC drive works well without TRIM; I knew that DC drives use good controller and it will give you much better experience than SF-based drive but I didn't check that they actually have Idle GC. Previous generations Intel drives did, but not these. S3500 relies on TRIM for GC. Hi Vladimir, Thank you for all the time you have spend with me on this. Seems to me that I either get over my phobia of software RAID of just get the most reliable SSD I can find and talk the customer out of RAID. -T -- ~~ Computers are like air conditioners. They malfunction when you open windows ~~
Re: need SSD RAID controller advice
On May 6, 2015, at 6:31 PM, Vladimir Mosgalin mosga...@vm10124.spb.edu wrote: Alternative to TRIM-aided GC is Idle Time Garbage Collection (which you could see on Plextor), which actively scans data in background, finds empty blocks (zeroes) and marks them as free. Well probably more complicated than that, but it's the idea. A quick search indicates that Idle TIme Garbage Collection is just another term for garbage collection. And I don’t see how it can work as well without TRIM as with TRIM, unless massively over provisioned. There is still the issue that, at the file system level, you may now that you just deleted a 1GB file that fully fills a whole lot of EBs. Without trim, the controller can’t know that the blocks aren’t in use until the OS attempts to write to those blocks. if the controller needs to make space, it will have to copy the stale data that it doesn’t know isn’t used anymore. Now, a continous write to a whole EB’s worth of data (still stored in DRAM on the drive), it can see that the whole block can be marked as unused, and freed later. I’m not sure what the zero blocks you are referring to means. If you write a whole bunch of zeros, is it marked as unused? Even if contained in a file? You really can’t mark obsolete data as zeros since you can’t rewrite until you erase first.
Re: need SSD RAID controller advice
On 05/06/2015 04:32 PM, Vladimir Mosgalin wrote: Hi ToddAndMargo! On 2015.05.06 at 00:48:30 -0700, ToddAndMargo wrote next: http://www.intel.ie/content/dam/www/public/us/en/documents/white-papers/ssd-dc-s3500-workload-raid-paper.pdf The system used for RAID 1 testing include the following: • LSI MegaRAID 9265-8i* controller card To summarize: • In both RAID 1 and RAID 5, the Intel SSD DC S3500 Series drive shows excellent scalability, performance, and consistency. • Very little latency was introduced by the RAID controller in RAID 1. In RAID 5, the overhead and latency are slightly higher. • In random, mixed read/write workloads, SSDs perform significantly (as much as 100 times) better than HDDs in a similar situation Maybe the S3500 is what I want? They are still available at my distributors. They are fine drives. In fact, I got servers running them; excellent performance, no problem with incompressible data (unlike drives with SF controllers) - this matters to me because I'm using zfs on linux with compression. Sustained write speed 500 MB/s (480GB model). However, like I said before, if long-term performance is needed, It is, but it will be in a high end workstation. 99% of the writes will be OS related and not user related. I advise against using them in hardware RAID controller. This document only shows after-install performance; they didn't do any endurance testing. Without TRIM, performance of this drive will degrade. Buy Plextor M5/M6 if your environment doesn't support TRIM. I'm curious, what stops you from using software RAID with mdadm? For two-disk RAID 1 it provides excellent performance, high compatibility and TRIM support. I could understand if you needed hardware controller for RAID6 with tons of disks, in RAID6 or something, but for two-disk RAID1?.. -- Vladimir Hi Vladimir, I am allergic to software RAID. I adore the way an independent controller works. Thank you for helping me with this, -T -- ~~ Computers are like air conditioners. They malfunction when you open windows ~~
Re: need SSD RAID controller advice
On Wed, May 06, 2015 at 04:46:27PM -0700, ToddAndMargo wrote: ... that I ... get the most reliable SSD I can find and talk the customer out of RAID. This is bad thinking. most reliable does not mean will never fail. (leaving aside the question on how you can tell which brand is more reliable, other than by vendor rosy promises or by counting of stars at newegg). If you are building a system for a customer, you have to have a reasonable answer to the question if this SSD fails, do I lose all of everything?. This means you have to have backups of everything and well tested instructions on how to restore the full working system from these backups. In my experience, mdadm RAID1 is the simplest way to build a system that survives single device failure (SSD or HDD, does not matter), but you still have to have backups and restoration instructions because RAID1 does not protect against filesystem corruption, against accidentally or maliciously deleted or modified files, etc. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: need SSD RAID controller advice
On Mon, Apr 13, 2015 at 03:28:39PM -0700, jdow wrote: The 3Ware RAID cards I have vastly outstrip the motherboard built in Intel RAID implementations for a RAID 5 setup. (I don't consider RAID 1 to be economically sensible for most uses.) A four disk RAID 5 SSD configuration can be breathtaking fast, too. In my experience, all raid cards have rather limited total bandwidth. For example- - consider 6TB disks, each can read data at 160 Mbytes/sec when connected to an on-board (intel) SATA port. - connect 8 of them to an 8-port RAID card. - sure enough, through the raid card you can read each disk at this speed. - now try to read all disks at the same time (8 programs reading from 8 disks) - what I see is the grand total data coming in is around 500-600-700 Mbytes/sec instead of 8*160=1280 Mbytes/sec. - (observe the same with writing instead of reading) - what I have measured is the raid card internal bandwidth (which includes the PCIe bandwidth, you should better be connected at PCIe x4 or better x8 or better x16 link lanes). - some vendors are honest and this bandwidth is written down in spec sheets (making it abvious that the raid cards were designed in the days of 1TB disks that could never reach 100 Mbytes/sec read or write). - now if you put RAID6 on top of this, the rate goes down again. - if instead of software RAID6 you use hardware RAID6, maybe you gain some bandwidth back, I have no data for this. - if instead, I connect all the disks to onboard ports (no raid card), using the sadly discontinued ASUS Z87 mobo with 10 SATA ports (6 intel + 4 marvell), I do see total bandwidths in the 1000 Mbytes/sec range (those measurements were done with 8x4TB disks). All this proves is that intel SATA ports on the intel internal interconnect are much faster than SATA ports connected to a not-super-fast microprocessor on a raid card connected to the system by an x4 or x8 PCIe bus (have not see x16 raid cards yet). Speaking of a theoretical 4xSSD hardware RAID5 configuration, I would be surprised if it reaches 1000 Mbytes/sec read/write speeds (assuming 500 Mbytes/sec SSDs). Software RAID5 with 4xSSDs connected to onboard SATA (with a 4GHz CPU) would probably get all the way to 2000 Mbytes/sec (assuming 500 Mbytes/sec SSDs). -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: need SSD RAID controller advice
From: Nico Kadel-Garcia nka...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com; James M. Pulver jmp...@cornell.edu; scientific-linux-users@fnal.gov scientific-linux-users@fnal.gov Sent: Tuesday, 14 April 2015, 3:02 Subject: Re: need SSD RAID controller advice On Mon, Apr 13, 2015 at 8:16 PM, Konstantin Olchanski olcha...@triumf.ca wrote: On Mon, Apr 13, 2015 at 07:29:18PM -0400, Nico Kadel-Garcia wrote: I am not sure what you refer to. With SL5 and SL6 you have 2 disks, put / on a software RAID1 partitions put grub on both disks ... Until it fails. Grubby, along with the kernel installlation RPM, doesn't know how to update, manage, or keep synchronized this second /boot partition. Hilarity can then ensue, along with making sure that your /etc/fstab doesn't detect the wrong disk and mount it incorrectly as /dev/sda. See, if your first disk dies, unless you're very cautious with /etc/fstab, it's very much a crapshoot if hte right partition will mount as /boot. Been there, done that, gave up on the silliness. You talk about /boot, /dev/sda, grubby, etc, I doubt you have been there, done that. Gave up, sure. With mirrored disks, there is only /dev/md0 (/), /dev/md1 (swap), etc (if you have /home, /data, etc), in /etc/fstab, / is mounted by UUID of the /dev/md0 RAID1 device, which in turn is assembled by UUID of component disks using mdadm magic. Been there, done that. Don't have to make this stuff up, I spent some time *designing* Linux based storage servers, but it was more than 5 years ago. However, software RAID may have improved to the point where /boot doesn't have to be its own partition. So I just tried it on a VM runing SL 6.6, and got unhandled exception, and git it to work flawlessly when I made /boot its own, non-RAID partition.. So I'm not filled with confidence that putting /boot on its partition is not still necessary. Are you finding that it actually works? Would you please post an /etc/fstab from a working system to help verify that it works? More cutting edge linux distributions such as Ubuntu can have /boot as part of a RAID system, and even on an LVM. I think this is a feature of GRUB2. If you wish to RAID /boot on older dists (AFAIR this applies to SL5 and SL6), you can RAID it but it must be mirrored. AFAIK GRUB doesn't understand the RAID but it looks sufficiently like a regular partition/filesystem that it is able to find what it needs. I don't have a fstab to show you from an older config as I migrated to Ubuntu Server for my Xen machines. I thoroughly tested both these configurations and the machine is able to fully boot with any given drive removed. Here is an Ubuntu one, plus supporting information, FWIW... # cat /etc/fstab # /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # file system mount point type options dump pass proc/proc procnodev,noexec,nosuid 0 0 /dev/mapper/xen2-root / ext4errors=remount-ro 0 1 /dev/mapper/xen2-swap noneswapsw 0 0 # cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdc1[2] sda1[0] sdb1[1] 1953258496 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU] unused devices: none root@xen2:~# pvs PV VG Fmt Attr PSize PFree /dev/md0 xen2 lvm2 a- 1.82t 53.18g root@xen2:~# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert --- snip --- rootxen2 -wi-ao 279.39g swapxen2 -wi-ao 7.45g --- snip ---
Re: need SSD RAID controller advice
On 11/04/15 02:21, ToddAndMargo wrote: Hi All, I have a customer interested in high end workstation. She wants RAID 1 to protect her drives. She is also interested in Solid State Drives. (And she knows the difference between RAID and Backup.) What host controller would you guys recommend for a pair of Intel SSDSC2BW480A401, 480 GB 2.5 Internal Solid State Drives in RAID 1? I've followed this discussion for a little while, and there are a lot of good information here - so thanks to everyone sharing their opinions and experiences. What I've not seen discussed here is the purpose of using RAID1. I guess it is related to avoid data loss in case one drive dies. The other thing I'm guessing is that SSD is wanted due to the performance. Considering that the RAID technology is old which was designed for rotating media, maybe that isn't the the best solution for SSD? I'm wondering ... maybe another (possibly better?) approach would be dm-cache? This has been added as a tech-preview in RHEL7. The purpose of dm-cache is to have a faster unit (such as an SSD) to cache contents from a slower device (such as a ordinary hard drive). To have better data security, the ordinary hard drive(s) can be an md-raid device. Maybe dm-cache can provide the best from SDD, HDD and RAID combined? Just my 2 cents. -- kind regards, David Sommerseth
Re: need SSD RAID controller advice
On Tue, Apr 14, 2015 at 06:39:52PM +0300, Vladimir Mosgalin wrote: Considering that the RAID technology is old which was designed for rotating media, maybe that isn't the the best solution for SSD? With SSD, there is still possibility of single-disk failure (SSD bricks itself). When this happens, to protect against data loss you have 2 choices: a) daily (hourly) rsync of SSD to some other storage (on SSD failure, machine will die, require manual recovery) b) RAID1 with another SSD (or HDD, use --write-mostly) (on SSD failure, machine will continue running) SL7 comes with btrfs, which is intended to be better RAID1 (I am presently testing this). But SL7 cannot boot from btrfs, so you still need a /boot partition and RAID1 to protect it against single-disk failure. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: need SSD RAID controller advice
Hi David Sommerseth! On 2015.04.14 at 12:55:14 +0200, David Sommerseth wrote next: Considering that the RAID technology is old which was designed for rotating media, maybe that isn't the the best solution for SSD? I'm wondering ... maybe another (possibly better?) approach would be dm-cache? This has been added as a tech-preview in RHEL7. Heh, RAID is VERY old and doesn't play that good with modern rotating media as well, which is why RAID2 and RAID3 are obsolete for a long time, and RAID5 is obsolete / too dangerous for a few years by now; not to mention write hole problems of RAID5/6, no solution for data inconsistency problems between drives for RAID1/10 and other issues. The proper modern era RAID replacements would be software volume-based solutions, preferable integrated into filesystem (zfs, btrfs) and distributed filesystems like ceph. That aside, using RAID1 or 10 for very small servers to increase reliability somewhat and RAID6 and similar for big file storages and backup servers still does make some sense.. The purpose of dm-cache is to have a faster unit (such as an SSD) to cache contents from a slower device (such as a ordinary hard drive). To have better data security, the ordinary hard drive(s) can be an md-raid device. Maybe dm-cache can provide the best from SDD, HDD and RAID combined? Not really. It was designed for completely different usage (think of it as of ability to greatly increase file cache size, cheaper than adding more RAM). It doesn't solve problems which are solved by RAID and doesn't have its performance characteristics as well. Using second drive for increased reliability to protect against device failure: RAID1 can do this, dm-cache cannot (hard drives breaks - you lose data even if it was somewhat cached on SSD). Using multiple drives for creating storage of a big size - somewhat slow, but with good reliability: RAID5/6/50/60 can do this, dm-cache cannot. Using multiple drives for getting guaranteed high bandwidth (required for video recording from multiple stream sources, video editing and such): RAID0 can do this, dm-cache cannot. But yes, for course it makes sense to use dm-cache with RAID, e.g. on database server where you can't afford to go all-SSD or on file storage where you need extra performance for popular accessed files. -- Vladimir
Re: need SSD RAID controller advice
On Tue, Apr 14, 2015 at 12:55:14PM +0200, David Sommerseth wrote: What I've not seen discussed here is the purpose of using RAID1. I guess it is related to avoid data loss in case one drive dies. The other thing I'm guessing is that SSD is wanted due to the performance. Correcto. SSD, HDD, punch cards, you still need protection against single device failure. These days, hardware is free, compared to cost of down time, cost of manual crash recovery, etc. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: need SSD RAID controller advice
On Tue, Apr 14, 2015 at 12:29:41PM +0300, Vladimir Mosgalin wrote: ... (e.g. 100 TB endurance of normal SSD vs 20 PB endurance of enterprise SSD) These are numbers from vendors. Data on actual performance as seen by end users is much harder to come by. For HDDs, we have periodic reports from backblaze (i.e. https://www.backblaze.com/blog/best-hard-drive/) For SSDs, we have http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead Not much else, in recent years. Both reports do not inspire much confidence in vendor-advertised failure rates or endurance numbers. And direct studies of failure rates between enterprise and consumer storage do not seem to exist (reports of my WD green drive died, but my WD black drive is still ok need not apply). The vendors themselves of course do have this data in the form of warranty return rates, but they are not telling. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: need SSD RAID controller advice
On Fri, Apr 10, 2015 at 05:21:07PM -0700, ToddAndMargo wrote: I have a customer interested in high end workstation. She wants RAID 1 to protect her drives. She is also interested in Solid State Drives. (And she knows the difference between RAID and Backup.) What host controller would you guys recommend for a pair of Intel SSDSC2BW480A401, 480 GB 2.5 Internal Solid State Drives in RAID 1? Since you do not say that hardware raid is required, I assume linux software raid is acceptable? We have been using dual Kingston 120GB SSDs attached to on-mobo sata ports and linux mdadm raid1 for a few years now without any special problems. Some web sites suggest special settings for SSDs in RAID1 configurations, suggest the importance of TRIM, etc, but (as a test) I ignore all that and use them as if they were normal HDDs. See no ill effects. One thing to remember with SSDs - unlike HDDs that seem to soldier on forever (just retired some still working perfectly HDDs from 2004), the SSDs do have finite write capacity and they *will* die eventually (see the eye-opener test of SSD capacity that just completed in February or so). -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: need SSD RAID controller advice
The 3Ware RAID cards I have vastly outstrip the motherboard built in Intel RAID
implementations for a RAID 5 setup. (I don't consider RAID 1 to be economically
sensible for most uses.) A four disk RAID 5 SSD configuration can be
breathtaking fast, too.
{^_^} Joanne
On 2015-04-13 05:10, James M. Pulver wrote:
I would point out that I'm not sure I've ever really seen the benefit of Real
Raid except for the vendor making more money. The only place I've used it is in
iSCSI boxes that run everything in firmware.
On all computers / servers, I've always used MDADM on Linux and ZFS on FreeNAS.
Both have been excellent for my intended use, though FreeNAS is only at home
for ~3 concurrent users, so take that whole thing with a grain of salt. Neither
has lost data due to power outages or drive failures.
--
James Pulver
CLASSE Computer Group
Cornell University
-Original Message-
From: owner-scientific-linux-us...@listserv.fnal.gov
[mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Vladimir
Mosgalin
Sent: Monday, April 13, 2015 5:30 AM
To: scientific-linux-users@fnal.gov
Subject: Re: need SSD RAID controller advice
Hi ToddAndMargo!
On 2015.04.12 at 17:35:04 -0700, ToddAndMargo wrote next:
On 04/12/2015 10:54 AM, Nico Kadel-Garcia wrote:
and I*loved* 3Ware
Me too. LSI gobbled them up.
Well, consolidation is often a good thing.
You can still buy best performing 3Ware 9750 (2011 model!) from LSI, they are
selling them for those who are fine with 6 Gbps speeds. Don't think they'll be
upgrading it to 12Gbps (not that many people are interested in real
RAID-supporting cards at such speeds, these are mostly for connecting external
storages...)
Re: need SSD RAID controller advice
On 04/10/2015 05:21 PM, ToddAndMargo wrote: Hi All, I have a customer interested in high end workstation. She wants RAID 1 to protect her drives. She is also interested in Solid State Drives. (And she knows the difference between RAID and Backup.) What host controller would you guys recommend for a pair of Intel SSDSC2BW480A401, 480 GB 2.5 Internal Solid State Drives in RAID 1? Many thanks, -T Hi All, This from LSI tech support: The deal breaker is TRIM. Once you put drives into the array, TRIM is not supported. So you will have to sacrifice TRIM for RAID and or visa versa. You can use an HBA such as 9207-4i and set up software RAID 1 and use TRIM, but there is no controller cache. Or You can use 9261-4i 9266-4i or 9271-4i with RAID, 1 gig of controller cache but no TRIM. So ... poop! -T Wonder if there are any enterprise level SSD's that don't need TRIM? -- ~~ Computers are like air conditioners. They malfunction when you open windows ~~
Re: need SSD RAID controller advice
On Mon, Apr 13, 2015 at 8:16 PM, Konstantin Olchanski olcha...@triumf.ca wrote: On Mon, Apr 13, 2015 at 07:29:18PM -0400, Nico Kadel-Garcia wrote: I am not sure what you refer to. With SL5 and SL6 you have 2 disks, put / on a software RAID1 partitions put grub on both disks ... Until it fails. Grubby, along with the kernel installlation RPM, doesn't know how to update, manage, or keep synchronized this second /boot partition. Hilarity can then ensue, along with making sure that your /etc/fstab doesn't detect the wrong disk and mount it incorrectly as /dev/sda. See, if your first disk dies, unless you're very cautious with /etc/fstab, it's very much a crapshoot if hte right partition will mount as /boot. Been there, done that, gave up on the silliness. You talk about /boot, /dev/sda, grubby, etc, I doubt you have been there, done that. Gave up, sure. With mirrored disks, there is only /dev/md0 (/), /dev/md1 (swap), etc (if you have /home, /data, etc), in /etc/fstab, / is mounted by UUID of the /dev/md0 RAID1 device, which in turn is assembled by UUID of component disks using mdadm magic. Been there, done that. Don't have to make this stuff up, I spent some time *designing* Linux based storage servers, but it was more than 5 years ago. However, software RAID may have improved to the point where /boot doesn't have to be its own partition. So I just tried it on a VM runing SL 6.6, and got unhandled exception, and git it to work flawlessly when I made /boot its own, non-RAID partition.. So I'm not filled with confidence that putting /boot on its partition is not still necessary. Are you finding that it actually works? Would you please post an /etc/fstab from a working system to help verify that it works?
RE: need SSD RAID controller advice
I would point out that I'm not sure I've ever really seen the benefit of Real Raid except for the vendor making more money. The only place I've used it is in iSCSI boxes that run everything in firmware. On all computers / servers, I've always used MDADM on Linux and ZFS on FreeNAS. Both have been excellent for my intended use, though FreeNAS is only at home for ~3 concurrent users, so take that whole thing with a grain of salt. Neither has lost data due to power outages or drive failures. -- James Pulver CLASSE Computer Group Cornell University -Original Message- From: owner-scientific-linux-us...@listserv.fnal.gov [mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Vladimir Mosgalin Sent: Monday, April 13, 2015 5:30 AM To: scientific-linux-users@fnal.gov Subject: Re: need SSD RAID controller advice Hi ToddAndMargo! On 2015.04.12 at 17:35:04 -0700, ToddAndMargo wrote next: On 04/12/2015 10:54 AM, Nico Kadel-Garcia wrote: and I*loved* 3Ware Me too. LSI gobbled them up. Well, consolidation is often a good thing. You can still buy best performing 3Ware 9750 (2011 model!) from LSI, they are selling them for those who are fine with 6 Gbps speeds. Don't think they'll be upgrading it to 12Gbps (not that many people are interested in real RAID-supporting cards at such speeds, these are mostly for connecting external storages...) -- Vladimir
Re: need SSD RAID controller advice
On Sat, Apr 11, 2015 at 5:36 PM, ToddAndMargo toddandma...@zoho.com wrote: On 04/11/2015 11:22 AM, Nico Kadel-Garcia wrote: RAID is not backup. Hi Nico, She is well aware. And you are correct about goofing a file being far more prevalent. Also, backup needs history, otherwise you can't restore after the goofed file gets backed up. (Often times, the user doesn't know they have goofed it up until they open it again.) Yeah, that's why I'm a big user of rsnapshot. Lightweight, configurable, and doesn't require complex software packages: I especially use it for backing up /etc for systems where the owners believe in reconfiguring things locally with no source control. You have any favorite hardware RAID controller that is SSD friendly? -T I've been dealing more with industrial grade virtualization environments lately, and the vendors or in house support staff for those got really, really touchy when I asked them for part numbers. I had good success, and continue to see good reviews, for Adaptec controllers with *real* RAID, not kind-of-sort-of-software-RAID like a lot of the older, cheaper LSI and Promise cards I ran into more than 5 years back. And I *loved* 3Ware for performance, but rarely had the budget for them. Check the latest reviews for more information, it's been more than 10 years since I had a test rack for this sort of thing.
Re: need SSD RAID controller advice
On 04/11/2015 11:06 AM, Brandon Vincent wrote: IR mode What is IR mode?
Re: need SSD RAID controller advice
On 04/11/2015 11:22 AM, Nico Kadel-Garcia wrote: RAID is not backup. Hi Nico, She is well aware. And you are correct about goofing a file being far more prevalent. Also, backup needs history, otherwise you can't restore after the goofed file gets backed up. (Often times, the user doesn't know they have goofed it up until they open it again.) You have any favorite hardware RAID controller that is SSD friendly? -T
Re: need SSD RAID controller advice
Hi ToddAndMargo! On 2015.04.10 at 17:21:07 -0700, ToddAndMargo wrote next: I have a customer interested in high end workstation. She wants RAID 1 to protect her drives. She is also interested in Solid State Drives. (And she knows the difference between RAID and Backup.) What host controller would you guys recommend for a pair of Intel SSDSC2BW480A401, 480 GB 2.5 Internal Solid State Drives in RAID 1? If you really have no choice in picking any non-SF based SSD, then I would only recommend software RAID with mdadm. It supports TRIM. Otherwise you're going to live with eventual performance degradation, because hardware controllers (at least LSI and Adaptec) do not support TRIM for drives in RAID. You can google yourself on many reports on how performance drops for SandForce SSDs filled with data in absence of TRIM (you may also google about difference between background GC and foreground GC). It's not pretty. If you *do* have a choice in this (way, way more important than picking controller) matter, pick an SSD from other vendor which supports background garbage collection without TRIM and performance doesn't degrade even in absence of TRIM. There are so many options and most vendors put in description that these SSDs are OK to use in RAID or other environment without TRIM support. These include Marvell-based SSDs (Plextor M series, Crucial M/MX series and many others), Samsung SSDs and really many others. If you want to pick SSD for RAID make sure to check some review which covers performance degradation without TRIM. It's not to say that Intel SSDs are bad (some are among very best, but they are expensive enterprise-focused models), but these particular consumer models based on SandForce controllers do not play that good with random RAID controller. They are okay to use in software (mdadm)-based RAID or on Intel software RAID (in Windows) because these support TRIM, also there were specialized RAID-on-board PCIe SSDs based on them which supported TRIM also, but you aren't going to get it with any random controller. Now, as long as you are using non-SF based SSD, you can among various RAID controllers. It's probably easier to list ones which aren't supported.. Good vendors (proven with years) include LSI and Adaptec, basically you can pick nearly any controller with RAID support from these vendors and it will work fine. Note that many other vendors like Intel or Dell sell controllers based on LSI models, often cheaper than originals. While there are occasional glitches (e.g. my favorite low-end RAID controller LSI 9211-8i with buggy P20 firmware, had to downgrade), usually they work great. But really there are tons of options, e.g. if you are using Dell or HP workstation you can go with controller from these vendors (Dell will likely to be LSI-based), they work great as well. It's your choice. (myself, for simple cases like two-disk RAID1 or four-disk RAID10 I always just use software RAID :P) -- Vladimir
Re: need SSD RAID controller advice
On Sat, Apr 11, 2015 at 11:03 AM, Brandon Vincent brandon.vinc...@asu.edu wrote: This is not correct. LSI SAS2008 I/O controllers do support TRIM with drives that support both deterministic trim (DRAT) and deterministic read zero after trim (DZAT). My bad. I didn't see that this thread was about using the controller in IR mode. Vladimir is correct about IR mode not supporting RAID. Sorry for any confusion. Brandon Vincent
Re: need SSD RAID controller advice
On Apr 11, 2015, at 4:39 PM, ToddAndMargo toddandma...@zoho.com wrote: On 04/11/2015 11:06 AM, Brandon Vincent wrote: IR mode What is IR mode? That appears to be when the controller is running in RAID mode. Especially with integrated controllers, there can be the option to run the drives in raid mode or normal mode. In the past when I bothered with this in computers I ran at home, the advantage of raid mode was you could have, say, 2 hard drives treated as 1 by the BIOS and operating system. So you could boot off of the raid system, and have an OS driver take over from it later. In normal mode, if you wanted to RAID, you probably needed at least 3 hard drives. A boot drive, and the others raided together using the OS software. I kind of stopped doing this with a combination of OS software being sufficient to combine drives, and with drives (even SSDs) getting much larger.
