On Tue, Aug 10, 2021 at 03:34:00PM -0700, Yasha Karant wrote:
One SSD had an internal short and turned into a space heater,
luckily there was no fire. End excerpt.
Clearly, there is very poor safety engineering and/or quality
control
you will not be amused to learn how many electronics lack
proper fuses and protections against internal and external
shorts. even here, I have seen good people forget to put fuses
on newly built boards.
(as with certain Li batteries that did similar things in
personal devices being operated by the user).
that's different. SSD stores bits, Li battery stores Joules,
and "bits do not burn".
If that SSD had been inside a laptop (presumably, inside a rack mounted
disk farm and there were fire extinguishers and possibly a machine room fire
suppression system), things could have had a much worse outcome
(most laptops have combustible materials).
tangled server rooms, laptops, men, guns, horses all together.
laptop battery probably will not have enough oomph for a good SSD fire,
cannot supply enough Amps, will shutdown before things get hot. ditto
for laptop power supply (60 W vs 600 W PC power supply).
server chassis with rack mounted SSD in a server room has such good cooling
that the shorted SSD will only get slightly warm. also server power supply
will probably shutdown quickly because of undervoltage condition. so no fire.
in this particular case, the computer was in an experimental area,
that has combustible materials, etc.
As for the small amount of storage, the commentator is at a
reasonably well funded (through government sources and possible
tax-deductible or glamour philanthropy) HEP facility.
We also have a $$$ printing press in our basement (I have a key!) and
we can transmute lead into gold (only slightly radioactive).
K.O.
Much of the world, including non-collaboration funded university research
facilities have rather poor funding at most entities within the USA
(not all faculty members can be at Harvard, Stanford, etc.) --
administrative and some instructional facilities typically can get
much more. Many universities now outsource to paid "cloud" storage,
with all of the issues that may entail.
On 8/10/21 3:08 PM, Konstantin Olchanski wrote:
Hi, Larry, thank you for this information, it is always good to see
how other people do things.
I am surprised at how little storage you have, only a handful of TBs.
Here, for each experiment data acquisition station, we now configure
2x1TB SSD for os, home dirs, apps, etc and 2x8-10-12TB HDD for recording
experiment data. We use "sort by price" NAS CMR HDDs (WD red, etc).
All disks are doubled up as linux mdadm raid1 (mirror) or ZFS mirror. This is
to prevent any disruption of data taking from single-disk failure.
(it is important to configure the boot loader on both SSDs to boot
even if the other SSD is dead).
I am surprised you use 1TB HDDs. We switched to SSD up to 2TB size (WD blue
SATA SSDs).
Failure rates of HDDs, the only reliable data is from backblaze:
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.backblaze.com_b2_hard-2Ddrive-2Dtest-2Ddata.html&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=NXYkiOfF7bPKBqi2iMgqsqrtLHRVdP7lIO-L5J4AmqQ&s=DgUuM1BVcm4jUkUWsi_DNMAjvkuy1zl1oaDQzrC4YAk&e=
and
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.backblaze.com_blog_backblaze-2Ddrive-2Dstats-2Dfor-2Dq2-2D2021_&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=NXYkiOfF7bPKBqi2iMgqsqrtLHRVdP7lIO-L5J4AmqQ&s=lPk2j2mTwp6uDzrZYUsP2rIxyRiacBHZOU0o7R5mUqM&e=
Failure rates of SSDs, seems to be very low, I only have 2-3 failed SSDs. One
SSD had an
internal short and turned into a space heater, luckily there was no fire.
For backups of os and home dirs we use amanda and rsync+zfs snapshots. Backups
of experiment data is not our responibility (many experiments use usb hdds).
K.O.
On Tue, Aug 10, 2021 at 10:55:35AM -0400, Larry Linder wrote:
There are 25 systems in our shop, all linux based, a linux based server,
and synology Disk Station running raid 1. The Disk Station has 12 TB
of space. 6 TB per for each raid level.
We buy only one brand of disk with the black label. They are typically
1 TB.
User boxes has a SSD drive for the OS and a 2 TB disk for the users
space and 32 G RAM. and a quad or six core AMD processor. The graphics
boxes get a Video card with lots of ram. 3 D rendering on a slow video
care wast's a lot of users time.
The server has a SSD for the OS and 6 TB for user apps /
library /usr/local and /opt. It also has a mirror disk that keeps a
copy of the server locally.
These systems are on 24 / 7 and accumulate a lot of hours. No matter
what the make mechanical disks have a life span. For grins I used to do
a post mortum on disk that failed. There were to types of failures,
the spring that returns the arm holding the heads cracks. The second
type of failure is the main bearings. Newer disk seem to have less of a
bearing failure rate.
To prevent operational problems we just swap out the disk on each box at
about 5,000 to 7,000 hr. The manufacturer says they are good for 10,000
hr. See the fine print in the Waranty, You have to remember this is a
money making operation and down time is costly.
Backups run at 12:29 and 0:29 in the AM. At the end of the morning back
up a copy is sent to a remote site.
For security we shut down the network at 6:20 PM, bring it up at 0:01 AM
and shut it down after back up is complete. We bring it back up at 6:45
AM.
10 yeas ago we had a fixed IP and the Chineese found it by just
continually pounding on the door. The return IP was 4 hops to a city
north east of Shanghi. They had installed a root kit on our server,
disabled cron. When you changed the passwd to the server a few
millisecond later it was sent to china. We got rid of the fixed IP and
reloaded all the systems. So when you shout down the network to your
provider the next time your start it you get a different IP.
We don't give the disks away as they contain a lot of design data,
SW,Cad programs, part programs for our mill etc. We donate them to a
charity that drills the disks and recycles the rest.
Larry Linder
