deloptes writes:
Linux-Fan wrote: > deloptes writes: >> Andrei POPESCU wrote: >> >> Each LSI card has a 6 bay cage attached and I have raided 6x2TB WD RED >> >> spinning discs (for data) and 2x1TB WD RED spinning discs (for OS)
[...]
> Sounds OK to me :) From my point of view, I would work towards reducing > the total number of disks, given that spinning disks of 8 TB capacity and > SSDs of 4 TB capacity are readily available today. YMMV > Yes, but the prices of these SSDs are very high compared to spinning disks. When looking at the bigger WD RED disks (3-4TB) few years ago I found out many people complained that disks do not have same quality as 2TB WD RED, so I stick to those despite I could have saved at least 2bays. I still have 4 unused bays anyway. The only thing is the power consumption ... would be at least 10W less ... but it is neglectable.
There was some discussion about newer+larger WD Red drives using SMR rather than CMR recording [1] which indeed makes them almost unusable for RAID purposes. Buying HDDs today is more difficult due to the different recording types -- which are not always identified clearly, not even by the manufacturers themselves...
For my new system, I went with 2x6TB Toshiba N300 which are performing well so far (still not much experience with them, yet). Only odd thing is that they actually make audible sounds every five to ten seconds or so -- it's the same sound that you would hear from accessing a very old IDE/PATA HDD... (rattle rattle). According to Toshiba, it is normal though.
[...]
After this discussion, I understand it is worth considering and it would pay off. This is why I ask for recommendations - might be I correct to replacement for both types. I must admit I now conclude considering putting dedicated disks for development, VMs and OS (like SSDs - may be better buy 2 3TB SSDs and replace the 2x1 and 2x2TB WD RED NAS) or replace just the 2x1TB with SSDs for OS and VMs as for the development I do not care compiling takes 20% more time.
Both sound like solid plans.
> I have had good experience with the following two "consumer-grade" SSDs in > an mdadm RAID 1 (taking the I for inexpensive literally :) ). Both have > about 8000 hours of operation according to SMART and when in use they ran
Correction: They are beyond the 10000 already, see SMART output.
> about 12h/day (i.e. normally not 24/7): > > * Samsung 850 EVO 2TB > * Crucial MX300 2TB > > At the time, these were the cheapest SSDs I could get with 2TB. Despite > their performance being "medicore" (for SSDs, that is), there were no > problems with RAID operation whatsoever. Thank you this is what I am looking for -personal experience. I have been looking at the Samsung 850 EVO 2TB. Can you share the exact model number, please?
Here is the smartctl -a output (shortened by serial numbers and less relevant parts). Not sure if it has a proper "model number".
~~~
# smartctl -a /dev/disk/by-id/ata-Samsung_SSD_850_EVO_2TB_...
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-12-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 850 EVO 2TB
Serial Number: ...
LU WWN Device Id: ...
Firmware Version: EMT02B6Q
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jan 2 17:23:05 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
...
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always -
0
9 Power_On_Hours 0x0032 097 097 000 Old_age Always -
10369
12 Power_Cycle_Count 0x0032 098 098 000 Old_age Always -
1777
177 Wear_Leveling_Count 0x0013 098 098 000 Pre-fail Always
- 30
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always
- 0
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always
- 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always
- 0
183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always
- 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always
- 0
190 Airflow_Temperature_Cel 0x0032 074 055 000 Old_age Always
- 26
195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always
- 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always
- 0
235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always
- 17
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always
- 53692285287
~~~
For comparison, there is also the Crucial MX300 (model number:
CT2050MX300SSD1):
~~~
# smartctl -a /dev/disk/by-id/ata-Crucial_CT2050MX300SSD1_...
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-12-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron MX1/2/300, M5/600, 1100 Client SSDs
Device Model: Crucial_CT2050MX300SSD1
Serial Number: ...
LU WWN Device Id: ...
Firmware Version: M0CR031
User Capacity: 2,050,408,636,416 bytes [2.05 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jan 2 17:40:33 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
...
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always -
104
5 Reallocate_NAND_Blk_Cnt 0x0032 099 099 010 Old_age Always -
90
9 Power_On_Hours 0x0032 100 100 000 Old_age Always -
10193
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always -
1747
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always
- 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always
- 0
173 Ave_Block-Erase_Count 0x0032 098 098 000 Old_age Always
- 33
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always
- 21
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always
- 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always
- 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always
- 56
194 Temperature_Celsius 0x0022 074 051 000 Old_age Always
- 26 (Min/Max 13/49)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always
- 90
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always
- 0
202 Percent_Lifetime_Used 0x0030 098 098 001 Old_age Offline
- 2
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always
- 0
246 Total_Host_Sector_Write 0x0032 100 100 000 Old_age Always
- 54316149898
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always
- 1714640403
248 Bckgnd_Program_Page_Cnt 0x0032 100 100 000 Old_age Always
- 2182825060
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always
- 8994
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always
- 210
~~~
IIRC both drives are rated for 300 TBW i.e. not "write-intensive" workloads.
There are other (more expensive) SSDs rated for higher workloads.
Additionally, I bought them years ago (2016?) -- I would expect that
nowdays, they have been superseded by follow-up models with similar
specifications.
While the SMART output shows that they are now running at SATA III, I remember using them in my old workstation (HP Z400) whith SATA II, too. It worked just as well (despite the obvious limit to 3 Gbit/s from the SATA II interface).
[...]
OK - thank you - this is the most complete answer I accept and I must once again admit, that the discussion helped me put some order in my thoughts.
You're welcome :)
As mentioned I also think of splitting up the disks depending on use. The multimedia would stay on the WD RED, but I will look forward to replace the OS, VM and for development disks.
It's basically what I do, too.
If someone has good experience with SSDs in RAID please share the device model, family and manufacturer.
See above. The drives hold MDADM RAID 1 arrays, one for OS, VMs and Swap respectively:
~~~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.8T 0 disk ├─sda1 8:1 0 256G 0 part │ └─md5 9:5 0 255.9G 0 raid1 / ├─sda2 8:2 0 18G 0 part │ └─md6 9:6 0 18G 0 raid1 [SWAP] └─sda3 8:3 0 1.6T 0 part └─md7 9:7 0 1.6T 0 raid1 /fs/ll sdb 8:16 0 1.9T 0 disk ├─sdb1 8:17 0 256G 0 part │ └─md5 9:5 0 255.9G 0 raid1 / ├─sdb2 8:18 0 18G 0 part │ └─md6 9:6 0 18G 0 raid1 [SWAP] └─sdb3 8:19 0 1.6T 0 part └─md7 9:7 0 1.6T 0 raid1 /fs/ll ~~~ [1] https://blog.westerndigital.com/wd-red-nas-drives/ https://raid.wiki.kernel.org/index.php/Timeout_Mismatch https://hardware.slashdot.org/story/20/06/08/1513222/western-digitals-smr-disks-wont-work-for-zfs-but-theyre-okay-for-most-nases search keywords: <WD RED SMR> or <SMR MDADM RAID> HTH Linux-Fan öö
pgpSgVZ01Qs5A.pgp
Description: PGP signature
