Re: Build servers offline due to failed SSD

[Daniel J. Luke]

On Mar 7, 2021, at 8:30 PM, Todd Doucet  wrote:
> I think one can only get so far with purely qualitative analysis of the 
> characteristics of SSDs and HDs and then the end of that analysis will be 
> one-size-fits all advice, for example "recommended" or "not recommended" for 
> servers.

this +1000

> Surely the answer might vary depending on the particular server usage 
> pattern, the need for performance, the cost of routine maintenance (swapping 
> out aging drives or SSDs), the cost of the devices themselves, etc.

exactly

There's a reason you don't really see 15k enterprise drives anymore.

> It seems to me that a given server operator can tell how long a particular 
> SSD is likely to last.  They do not fail randomly, at least not very much.  
> The fail when they are "used up" and you can figure out well in advance, 
> usually, when you will need to swap the old ones out of service.

Back in 2015 - there's this article 
https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/
 where someone actually bothered to test and report some results.

> HDs fail also, obviously, but tend not to be so predictable about it.  
> Whether it makes sense for a given server to use an SSD really does depend on 
> the numbers.  All drives will fail.  All drives will need to be rotated out 
> of service.  It is a matter of cost, convenience, and performance.
> 
> The only caveat I can think of is that there might be an issue of malicious 
> use--a server with SSDs might be vulnerable to a wear attack, depending on 
> the server services offered, I suppose.

I'm sure there are worst-case scenarios for spinning disks that (in theory) 
could be exploited to wear their mechanisms out as well.

I've personally used both enterprise and consumer SSDs in high-write 
environments where the cost of replacing the SSDs was worthwhile for the 
performance benefits (or otherwise didn't change the overall cost of the 
solution) - and I've been pleasantly surprised with how much more use I've 
gotten from them than I originally calculated (based on the drive specs + the 
planed utilization + over provisioning). 

YMMV of course - but the blanket "you shouldn't use SSDs for servers" or "no 
one uses SSDs for servers" is wrong. For those who are interested in more 
details, there are a bunch of good USENIX and ACM papers where people have 
actually gone and collected data on real-world failure rates.

-- 
Daniel J. Luke

Re: Build servers offline due to failed SSD

[Peter West]

I’ve been looking at VPS providers, and most of them offer SSD-based VPSs, so 
they seem to be increasingly popular. I suspect that most VPSs do not get 
consistently hammered, though.

Peter
—
p...@ehealth.id.au
“Destroy this temple, and in three days I will raise it up.”

> On 8 Mar 2021, at 11:30 am, Todd Doucet  wrote:
> 
> I think one can only get so far with purely qualitative analysis of the 
> characteristics of SSDs and HDs and then the end of that analysis will be 
> one-size-fits all advice, for example "recommended" or "not recommended" for 
> servers.
> 
> Surely the answer might vary depending on the particular server usage 
> pattern, the need for performance, the cost of routine maintenance (swapping 
> out aging drives or SSDs), the cost of the devices themselves, etc.
> 
> It seems to me that a given server operator can tell how long a particular 
> SSD is likely to last.  They do not fail randomly, at least not very much.  
> The fail when they are "used up" and you can figure out well in advance, 
> usually, when you will need to swap the old ones out of service.
> 
> HDs fail also, obviously, but tend not to be so predictable about it.  
> Whether it makes sense for a given server to use an SSD really does depend on 
> the numbers.  All drives will fail.  All drives will need to be rotated out 
> of service.  It is a matter of cost, convenience, and performance.
> 
> The only caveat I can think of is that there might be an issue of malicious 
> use--a server with SSDs might be vulnerable to a wear attack, depending on 
> the server services offered, I suppose.
> 
> 
> 
> 
>> To emphasize again, the reason SSDs aren’t recommended for servers is 
>> because servers—by definition—see much heavier service, and these read/write 
>> cycles are used up more quickly.
>> 
>> For personal use in a PC, or such, SSDs are proving to be the dream they 
>> were promised to be.
>> 
>> As mentioned, given time, the technology will overcome this limitation for 
>> use in servers and these comments will be just so much past history.
>> 
>> Dave C.
>> 
>> - - - 
>> 
>> > The “on/off” switches in SSD’s are fragile and essentially break after 
>> > too many read/write cycles.  As pointed out, it’s a get what you pay for 
>> > world and cheap SSD’s are just that… cheap.   The expensive ones are more 
>> > reliable because they actually make available only a portion of their 
>> > total capacity, reserving the rest as replacements for such failures.  
>> > Intelligent software within the firmware manages this so that the end user 
>> > experiences a much longer device lifespan.
>> > 
>> > There’s lots of technical documentation for such.  Google knows.
>> > 
>> > Regards,
>> > 
>> > 
>> >>> On Mar 7, 2021, at 18:15, Michael A. Leonetti via macports-users 
>> >>> > >>> > wrote:
>> >> I’d really love to know more about what you’re saying here. Up until I 
>> >> just read what you wrote, I thought SSDs were the savior of HDDs.
>> >> Michael A. Leonetti
>> >> As warm as green tea
>> >>> 3/7/21 午後5:26、Dave Horsfall > >>> >のメール:
>> >>> On Sat, 6 Mar 2021, Dave C via macports-users wrote:
>>  Isn’t SSD a bad choice for server duty? No server farms use them, 
>>  apparently due to short lifespan.
>> >>> If you knew how SSDs worked then you wouldn't use them at all without 
>> >>> many backups.  Give me spinning rust any day...
>> >>> -- Dave

Re: Build servers offline due to failed SSD

[Todd Doucet]

I think one can only get so far with purely qualitative analysis of the 
characteristics of SSDs and HDs and then the end of that analysis will be 
one-size-fits all advice, for example "recommended" or "not recommended" for 
servers.

Surely the answer might vary depending on the particular server usage pattern, 
the need for performance, the cost of routine maintenance (swapping out aging 
drives or SSDs), the cost of the devices themselves, etc.

It seems to me that a given server operator can tell how long a particular SSD 
is likely to last.  They do not fail randomly, at least not very much.  The 
fail when they are "used up" and you can figure out well in advance, usually, 
when you will need to swap the old ones out of service.

HDs fail also, obviously, but tend not to be so predictable about it.  Whether 
it makes sense for a given server to use an SSD really does depend on the 
numbers.  All drives will fail.  All drives will need to be rotated out of 
service.  It is a matter of cost, convenience, and performance.

The only caveat I can think of is that there might be an issue of malicious 
use--a server with SSDs might be vulnerable to a wear attack, depending on the 
server services offered, I suppose.




> To emphasize again, the reason SSDs aren’t recommended for servers is because 
> servers—by definition—see much heavier service, and these read/write cycles 
> are used up more quickly.
> 
> For personal use in a PC, or such, SSDs are proving to be the dream they were 
> promised to be.
> 
> As mentioned, given time, the technology will overcome this limitation for 
> use in servers and these comments will be just so much past history.
> 
> Dave C.
> 
> - - - 
> 
> > The “on/off” switches in SSD’s are fragile and essentially break after too 
> > many read/write cycles.  As pointed out, it’s a get what you pay for world 
> > and cheap SSD’s are just that… cheap.   The expensive ones are more 
> > reliable because they actually make available only a portion of their total 
> > capacity, reserving the rest as replacements for such failures.  
> > Intelligent software within the firmware manages this so that the end user 
> > experiences a much longer device lifespan.
> > 
> > There’s lots of technical documentation for such.  Google knows.
> > 
> > Regards,
> > 
> > 
> >>> On Mar 7, 2021, at 18:15, Michael A. Leonetti via macports-users 
> >>>  wrote:
> >> I’d really love to know more about what you’re saying here. Up until I 
> >> just read what you wrote, I thought SSDs were the savior of HDDs.
> >> Michael A. Leonetti
> >> As warm as green tea
> >>> 3/7/21 午後5:26、Dave Horsfall のメール:
> >>> On Sat, 6 Mar 2021, Dave C via macports-users wrote:
>  Isn’t SSD a bad choice for server duty? No server farms use them, 
>  apparently due to short lifespan.
> >>> If you knew how SSDs worked then you wouldn't use them at all without 
> >>> many backups.  Give me spinning rust any day...
> >>> -- Dave
> 
> 

Re: Build servers offline due to failed SSD

[Dave C via macports-users]

To emphasize again, the reason SSDs aren’t recommended for servers is because 
servers—by definition—see much heavier service, and these read/write cycles are 
used up more quickly.

For personal use in a PC, or such, SSDs are proving to be the dream they were 
promised to be.

As mentioned, given time, the technology will overcome this limitation for use 
in servers and these comments will be just so much past history.

Dave C.

- - - 

> The “on/off” switches in SSD’s are fragile and essentially break after too 
> many read/write cycles.  As pointed out, it’s a get what you pay for world 
> and cheap SSD’s are just that… cheap.   The expensive ones are more reliable 
> because they actually make available only a portion of their total capacity, 
> reserving the rest as replacements for such failures.  Intelligent software 
> within the firmware manages this so that the end user experiences a much 
> longer device lifespan.
> 
> There’s lots of technical documentation for such.  Google knows.
> 
> Regards,
> 
> 
>>> On Mar 7, 2021, at 18:15, Michael A. Leonetti via macports-users 
>>>  wrote:
>> I’d really love to know more about what you’re saying here. Up until I just 
>> read what you wrote, I thought SSDs were the savior of HDDs.
>> Michael A. Leonetti
>> As warm as green tea
>>> 3/7/21 午後5:26、Dave Horsfall のメール:
>>> On Sat, 6 Mar 2021, Dave C via macports-users wrote:
 Isn’t SSD a bad choice for server duty? No server farms use them, 
 apparently due to short lifespan.
>>> If you knew how SSDs worked then you wouldn't use them at all without many 
>>> backups.  Give me spinning rust any day...
>>> -- Dave

Re: Build servers offline due to failed SSD

[John Chivian]

The “on/off” switches in SSD’s are fragile and essentially break after too many 
read/write cycles.  As pointed out, it’s a get what you pay for world and cheap 
SSD’s are just that… cheap.   The expensive ones are more reliable because they 
actually make available only a portion of their total capacity, reserving the 
rest as replacements for such failures.  Intelligent software within the 
firmware manages this so that the end user experiences a much longer device 
lifespan.

There’s lots of technical documentation for such.  Google knows.

Regards,


> On Mar 7, 2021, at 18:15, Michael A. Leonetti via macports-users 
>  wrote:
> 
> I’d really love to know more about what you’re saying here. Up until I just 
> read what you wrote, I thought SSDs were the savior of HDDs.
> 
> Michael A. Leonetti
> As warm as green tea
> 
>> 3/7/21 午後5:26、Dave Horsfall のメール:
>> 
>> On Sat, 6 Mar 2021, Dave C via macports-users wrote:
>> 
>>> Isn’t SSD a bad choice for server duty? No server farms use them, 
>>> apparently due to short lifespan.
>> 
>> If you knew how SSDs worked then you wouldn't use them at all without many 
>> backups.  Give me spinning rust any day...
>> 
>> -- Dave
> 

Re: Build servers offline due to failed SSD

[Michael A. Leonetti via macports-users]

I’d really love to know more about what you’re saying here. Up until I just 
read what you wrote, I thought SSDs were the savior of HDDs.

Michael A. Leonetti
As warm as green tea

> 3/7/21 午後5:26、Dave Horsfall のメール:
> 
> On Sat, 6 Mar 2021, Dave C via macports-users wrote:
> 
>> Isn’t SSD a bad choice for server duty? No server farms use them, apparently 
>> due to short lifespan.
> 
> If you knew how SSDs worked then you wouldn't use them at all without many 
> backups.  Give me spinning rust any day...
> 
> -- Dave

Re: Build servers offline due to failed SSD

[Dave Horsfall]

On Sat, 6 Mar 2021, Dave C via macports-users wrote:

Isn’t SSD a bad choice for server duty? No server farms use them, 
apparently due to short lifespan.


If you knew how SSDs worked then you wouldn't use them at all without many 
backups.  Give me spinning rust any day...


-- Dave

Re: Warning: cltversion again

[Chris Jones]

Hi,

> On 7 Mar 2021, at 6:18 pm, Fielding, Eric J (US 329A) via macports-users 
>  wrote:
> 
> 
> I am getting this warning again from running ‘port upgrade outdated’:
>  
> Warning: cltversion: The Command Line Tools are installed, but MacPorts 
> cannot determine the version.
> Warning: cltversion: For a possible fix, please see: 
> https://trac.macports.org/wiki/ProblemHotlist#reinstall-clt
>  
> I recently opened Xcode and it said it needed to update some tools, so I 
> guess it installed a different version of the Command Line Tools (CLT) in a 
> way that MacPorts can’t determine the version. Is it worth doing the CLT 
> reinstallation described on the Hotlist page or should I just ignore it? It 
> seems that everything is building well despite the warning. I went through 
> the reinstallation procedure a month or two ago.

You should rerun the fixes. The issues sadly reappears on macOS/Xcode updates.

Chris

>  
> I am on Catalina (10.15.7), in case that makes a difference.
>  
> ++Eric

Warning: cltversion again

[Fielding, Eric J (US 329A) via macports-users]

I am getting this warning again from running ‘port upgrade outdated’:

Warning: cltversion: The Command Line Tools are installed, but MacPorts cannot 
determine the version.
Warning: cltversion: For a possible fix, please see: 
https://trac.macports.org/wiki/ProblemHotlist#reinstall-clt

I recently opened Xcode and it said it needed to update some tools, so I guess 
it installed a different version of the Command Line Tools (CLT) in a way that 
MacPorts can’t determine the version. Is it worth doing the CLT reinstallation 
described on the Hotlist page or should I just ignore it? It seems that 
everything is building well despite the warning. I went through the 
reinstallation procedure a month or two ago.

I am on Catalina (10.15.7), in case that makes a difference.

++Eric

Re: Build servers offline due to failed SSD

[Ryan Schmidt]

On Mar 7, 2021, at 00:20, Dave C wrote:

> Isn’t SSD a bad choice for server duty?

My opinion is that it is a good choice in terms of performance. When I first up 
this incarnation of our buildbot system in 2016 I had the workers running on 
SSDs so that builds would be fast (our previous buildbot setup at Apple's macOS 
forge used a very expensive very-many-hard-disk RAID; we were in no position to 
purchase any equivalent type of hardware once we left macOS forge) and I had 
the master and distfiles/packages storage on a hard disk RAID for reliability. 
The specific RAID that I have turned out to be too slow. Web requests could 
take many seconds to respond. GitHub Web Hooks being delivered to the server 
could be marked as failed because GitHub didn't always wait long enough for our 
server to respond. This was unsatisfying so I moved the buildmaster to an old 
SSD while keeping the large files on the RAID. This was much faster, though not 
as fast as if I had used a new SSD, which is what I will ultimately be using. 
For now, the buildmaster is temporarily running off a USB hard drive and is 
slow as molasses. This is a terrible choice but all drive bays are already 
occupied by the RAID.

All of the SSDs we used for the workers have failed as well, 2 last year and 
the last one last month. In response to these failures, someone else also 
suggested that we should not use SSDs. I've run one of the workers off of three 
independent hard disks for the past year, and my opinion is that the 
performance and power consumption of SSDs is much better and I will switch the 
hard disk-based worker back to an SSD in the future. You can read this 
discussion here:

https://trac.macports.org/ticket/60178