Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[jdow]

On 2017-04-11 12:57, Michael Tiernan wrote:

On 4/11/17 3:41 PM, jdow wrote:

So I suppose the extended downtime while several terabytes of data are
restored after it's loss due to filesystem malfunction is of no consequence to
you.

While not trying to dowse the fire with gasoline, I'd like to remind folks that
data loss isn't the only problem, data corruption is also a problem.

Oh, and while one's trying to manipulate serveral terabytes of data, the system
slowing to a crawl because of excessive thrashing or fs overhead is also bad.


The reduced utility and accessibility of the data being tested and restored is 
the sticky point as I see it. This can be a serious cost. Having dozens of 
backups is fine. If none of the backups are online on different disks available 
to take over the load while the main server's data is restored, you find 
yourself dead in the water for however long it takes to test and restore two or 
three digit number of Terabytes. (I wonder how NSA handles this in their new 
data center)


{^_^}

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Konstantin Olchanski]

On Tue, Apr 11, 2017 at 10:47:09PM +0200, David Sommerseth wrote:
> 
> So lets flip this around ... Why isn't btrfs enabled by default in RHEL...
>

I already wrote about BTRFS in this thread. I did some extensive
tests of BTRFS and the performance is quite acceptable (close to hardware
capability, about the same as raid6+xfs), the interesting features
all work, raid5/raid6 is more flexible compared to zfs.

But for production use, they are missing one small feature:

there is no handling whatsoever of disk failures. If you unplug a disk
(or if a disk unplugs itself by giving up the ghost), BTRFS does
absolutely nothing other than filling the syslog with error messages.

Then there are small bugs, for example, if BTRFS is in degraded
mode (one or more dead disk) el7 will not boot (systemd will wait
forever for the missing disk). Suggested workaround do not work,
and this means if one disk dies, you have to manually replace
it and do the rebuild before rebooting. If you do reboot, you will have
to go into single user mode, install the replacement disk,
run the btrfs rebuild, only then you can boot normally. The server
is down all this time.

My conclusion is BTRFS in the "this is a toy" departement,
not in the "this is unstable" or "preview" departement.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[David Sommerseth]

On 11/04/17 20:48, Konstantin Olchanski wrote:
> 
> So the choice is between you saying "I trust XFS, I never fsck it",
> and me saying "I do not trust ZFS, I do not trust the hardware, I run
> ZFS scrub daily, I have backups, I have archives".

So lets flip this around ... Why isn't btrfs enabled by default in RHEL,
but still being a tech-preview which explicitly labels it "unsuitable
for production"?  And why haven't RHEL seen any active involvement from
RH and/or the Fedora community to add ZFS to the distro?  And why isn't
Fedora still using btrfs as the default file system, despite being
suggested a few times already?

  * Fedora 16

  * Fedora 17
  
  * Fedora 23



Part of the answer is definitely that the btrfs file system is not
considered ready for prime time production.  For ZFS, the licensing is
probably quite an issue as well.

As I already said in an earlier mail:

   "Once Red Hat enabling ZFS on a kernel where it is native in the
upstream kernel, not being labelled tech-preview - that day I will
consider ZFS for production.  If btrfs reaches this stage earlier
[than ZFS], then I will consider btrfs instead."

I trust the expertise RH have in the file system area.  In fact, I have
spent some time discussing this topic with a few of their FS developers
face-to-face several times, in addition to some of their storage driver
maintainers.  So when RH is not pushing customers unto these new file
systems, it is for a good reason.  And I choose to take RH's advise.
What you do is your decision - and I don't have a need to convince you
to change your opinion.


As I've now iterated a few times already things I've already said before
... I'm letting this be my last reply to this mail thread.


-- 
kind regards,

David Sommerseth

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Michael Tiernan]

On 4/11/17 3:41 PM, jdow wrote:
So I suppose the extended downtime while several terabytes of data are 
restored after it's loss due to filesystem malfunction is of no 
consequence to you.
While not trying to dowse the fire with gasoline, I'd like to remind 
folks that data loss isn't the only problem, data corruption is also a 
problem.


Oh, and while one's trying to manipulate serveral terabytes of data, the 
system slowing to a crawl because of excessive thrashing or fs overhead 
is also bad.

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[jdow]

On 2017-04-11 09:44, Konstantin Olchanski wrote:

On Tue, Apr 11, 2017 at 11:13:25AM +0200, David Sommerseth wrote:


But that aside, according to [1], ZFS on Linux was considered stable in
2013.  That is still fairly fresh, and my concerns regarding the time it
takes to truly stabilize file systems for production [2] still stands.



Why do you worry about filesystem stability?

So I suppose the extended downtime while several terabytes of data are restored 
after it's loss due to filesystem malfunction is of no consequence to you. 
Others find extended downtime both extremely frustrating and expensive. And that 
does ignore the last few {interval between backups} worth of data loss, which 
can also be expensive.


{o.o}   Joanne

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Konstantin Olchanski]

On Tue, Apr 11, 2017 at 07:47:46PM +0200, David Sommerseth wrote:
> 
> ... newer file systems have more bugs than older file systems. ...
>

I do not think that is necessarily true.

Newer filesystems have consistly added functionality to detect
and prevent data corruption, to preserve and ensure data integrity.
It is hard to quantify how this is offset by the (unmeasureable)
increase in number of bugs.

The 1st generation filesystems are quite well debugged (msdos/fat/vfat,
ext2/3/4) but do not have any features to preserve data integrity.

The 2nd generation filesystems (like SGI XFS) added some built-in data
integrity checks (the first release of XFS did not even have an fsck
because "it will never corrupt". the second release added an fsck,
because bad hardware does corrupt anything).

But they lack data checksums and "online fsck" features - you have to take
the server offline to confirm filesystem consistency.

They also tend to have bad interaction with the RAID layers (all the stripes
and stuff has to line-up correctly or performance is very bad).

The 3rd generation filesystems (zfs, btrfs) add data checksums, online-fsck,
"integrated raid".

So the choice is between you saying "I trust XFS, I never fsck it",
and me saying "I do not trust ZFS, I do not trust the hardware, I run
ZFS scrub daily, I have backups, I have archives".

>
> Consider the
> amount of users using ext4 and XFS, vs btrfs/zfs ... and in which of
> these groups do you hear more about users experiencing data loss?
> 

Ah, the science of it. "which do I hear more...". There is a difference
between the actual number of faults, reported number of faults
and "I read about it on slashdot" number of faults.

>
> And I do care about data loss.  Because if that happens, I need to start
> running restore jobs from backups and recover files and file systems.
> So having a stable and rock solid file system reduces the chances of
> extra work for me.  And my users are much more happy too.
> 

I think it is a good thing if you have to restore files from backup and archive
at least once a year. How else do you know your backups actually work?

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[David Sommerseth]

On 11/04/17 18:44, Konstantin Olchanski wrote:
> On Tue, Apr 11, 2017 at 11:13:25AM +0200, David Sommerseth wrote:
>>
>> But that aside, according to [1], ZFS on Linux was considered stable in
>> 2013.  That is still fairly fresh, and my concerns regarding the time it
>> takes to truly stabilize file systems for production [2] still stands.
>>
> 
> Why do you worry about filesystem stability?
> 
> So what if it eats your data every 100 years due to a rare bug? You do have
> backups (filesystem stability does not protect you against
> fat-fingering the "rm" command), and you do archive your data, yes?
> You do have a hot-spare server (filesystem stability does not protect
> you against power supply fire) and you do have disaster mitigation plans
> (filesystem stability does not protect you against "server is down,
> you are fired!").
> 
> So what if it eats your data every 1 week due to a frequent bug? How is that
> different from faulty hardware eating your data? (like the cheap intel pcie 
> ssd
> eating all data on xfs and ext4 within 10 seconds of booting). You build
> a system, you burn it in, you test it, if it works, it works, if it does not,
> you throw zfs (or the hardware) into the dumpster, start again with yfs, qfs, 
> whatever.
> 
> How else can you build something that works reliably? Using only components 
> annointed
> by the correct penguin can only take you so far.

It's called risk assessment and management.  You consider the risks, and
newer file systems have more bugs than older file systems.  Consider the
amount of users using ext4 and XFS, vs btrfs/zfs ... and in which of
these groups do you hear more about users experiencing data loss?

And I do care about data loss.  Because if that happens, I need to start
running restore jobs from backups and recover files and file systems.
So having a stable and rock solid file system reduces the chances of
extra work for me.  And my users are much more happy too.

Why do you think I'm running RAID 6?  To reduce the risk of data loss if
even 2 drives decides to bail out.  And if all that should explode, then
I have backups to restore.  And if the local backup fails, I have an
offsite backup as well.  Because data loss is really annoying for me and
my users.

But even with all that ... RAID 6 won't save me from a file system going
crazy and leaving data into the void.  So the stability and matureness
of the file system is equally important.

If you don't need to care for you data in 1-2 months ... then I
understand you being willing to use less stable and mature file systems.


-- 
kind regards,

David Sommerseth

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Konstantin Olchanski]

On Tue, Apr 11, 2017 at 11:13:25AM +0200, David Sommerseth wrote:
> 
> But that aside, according to [1], ZFS on Linux was considered stable in
> 2013.  That is still fairly fresh, and my concerns regarding the time it
> takes to truly stabilize file systems for production [2] still stands.
> 

Why do you worry about filesystem stability?

So what if it eats your data every 100 years due to a rare bug? You do have
backups (filesystem stability does not protect you against
fat-fingering the "rm" command), and you do archive your data, yes?
You do have a hot-spare server (filesystem stability does not protect
you against power supply fire) and you do have disaster mitigation plans
(filesystem stability does not protect you against "server is down,
you are fired!").

So what if it eats your data every 1 week due to a frequent bug? How is that
different from faulty hardware eating your data? (like the cheap intel pcie ssd
eating all data on xfs and ext4 within 10 seconds of booting). You build
a system, you burn it in, you test it, if it works, it works, if it does not,
you throw zfs (or the hardware) into the dumpster, start again with yfs, qfs, 
whatever.

How else can you build something that works reliably? Using only components 
annointed
by the correct penguin can only take you so far.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Kraus, Dave (GE Healthcare)]

On 4/11/17, 10:08 AM, "owner-scientific-linux-us...@listserv.fnal.gov on behalf 
of Tom H"  wrote:

On Tue, Apr 11, 2017 at 4:30 AM, Jose Marques  
wrote:
>> On 10 Apr 2017, at 18:23, David Sommerseth 
 wrote:
>>
>> But I'll give you that Oracle is probably a very different beast on
>> the legal side and doesn't have a too good "open source karma".
>
> ZFS on Linux is based on OpenZFS
> (). Oracle has no input into its
> development as far as I can tell.

I'm not sure that David S is referring to. Sun open-sourced zfs and it
was at zpool version 28 when Oracle closed-sourced it. So the cat's
out of the bag up to v28. But the Solaris version's currently 36
(IIRC) and, in openzfs, you can enable extra, post-v28 features on a
case by case basis.

[Someone said up-thread that you couldn't expand a pool (or add disks
to a pool). That's incorrect. You can add a same-type vdev to a pool.]


FWIW, we’ve been shipping a current version of ZFS with our SL spin for a few 
years, and use it for several of our internal storage servers. It’s been quite 
stable in our environment with the proper care and feeding. But it’s still a 
storage-server environment (I don’t call it a filesystem), not a generic 
substitute for EXT4 or XFS. Tools for the job and all that.

Not going to comment much on the licensing aspects other than we’re fairly 
paranoid and have deemed the licenses acceptable for our business and 
redistribution purposes. Most if not all of the Oracle CDDL(?)-licensed code 
has been excised as far as I know. Others here track that, so I personally 
don’t think about it much.

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Tom H]

On Tue, Apr 11, 2017 at 4:30 AM, Jose Marques  wrote:
>> On 10 Apr 2017, at 18:23, David Sommerseth  
>> wrote:
>>
>> But I'll give you that Oracle is probably a very different beast on
>> the legal side and doesn't have a too good "open source karma".
>
> ZFS on Linux is based on OpenZFS
> (). Oracle has no input into its
> development as far as I can tell.

I'm not sure that David S is referring to. Sun open-sourced zfs and it
was at zpool version 28 when Oracle closed-sourced it. So the cat's
out of the bag up to v28. But the Solaris version's currently 36
(IIRC) and, in openzfs, you can enable extra, post-v28 features on a
case by case basis.

[Someone said up-thread that you couldn't expand a pool (or add disks
to a pool). That's incorrect. You can add a same-type vdev to a pool.]

Re: RAID 6 array and failing harddrives

[Tom H]

On Mon, Apr 10, 2017 at 1:23 PM, David Sommerseth
 wrote:
> On 10/04/17 09:15, Tom H wrote:
>>
>> zfs'll never be in-tree for licensing reasons.
>
> Well, "never" might be a too strong word. Stranger things have
> happened, like Microsoft embracing Linux and open source; even
> starting to open up some of their closed projects as open source ;-)
>
> But I'll give you that Oracle is probably a very different beast on
> the legal side and doesn't have a too good "open source karma".

No one can see Oracle changing the zfs license to one that's
gpl-compatible. The cddl is supposed to be gpl-incompatible by design.

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[David Sommerseth]

On 11/04/17 10:30, Jose Marques wrote:
>> On 10 Apr 2017, at 18:23, David Sommerseth  
>> wrote:
>>
>> But I'll give you that Oracle is probably a very different beast on the
>> legal side and doesn't have a too good "open source karma".
> 
> ZFS on Linux is based on OpenZFS (). 
> Oracle has no input into its development as far as I can tell.

Yeah, that /sounds/ like a clean implementation.  IANAL and also cannot
predict if the Oracle lawyers agree.

I have few doubts of the F/L/OSS "pureness" of many of the developers
hired by Oracle, working with various upstream communities.  But I do
fear the intentions and attitudes of the Oracle business side people,
including the CEO.  ANd also consider that the ZFS name is a registered
trademark.

But that aside, according to [1], ZFS on Linux was considered stable in
2013.  That is still fairly fresh, and my concerns regarding the time it
takes to truly stabilize file systems for production [2] still stands.

[1] 
[2]



-- 
kind regards,

David Sommerseth

Re: [SCIENTIFIC-LINUX-USERS] RAID 6 array and failing harddrives

[Jose Marques]

> On 10 Apr 2017, at 18:23, David Sommerseth  
> wrote:
> 
> But I'll give you that Oracle is probably a very different beast on the
> legal side and doesn't have a too good "open source karma".

ZFS on Linux is based on OpenZFS (). Oracle 
has no input into its development as far as I can tell.

The University of St Andrews is a charity registered in Scotland, No. SC013532.

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 10/04/17 09:15, Tom H wrote:
> zfs'll never be in-tree for licensing reasons.

Well, "never" might be a too strong word.  Stranger things have
happened, like Microsoft embracing Linux and open source; even starting
to open up some of their closed projects as open source ;-)

But I'll give you that Oracle is probably a very different beast on the
legal side and doesn't have a too good "open source karma".


-- 
kind regards,

David Sommerseth

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 10/04/17 10:43, Tom H wrote:
> On Thu, Apr 6, 2017 at 5:01 PM, David Sommerseth
>  wrote:
> 
> 
>> I would never run btrfs on *any* production server, regardless of
>> currently available kernel versions. Because it is not deemed ready
>> for production yet. For testing I would be willing to experiment with
>> it, as there I can tolerate data loss. But never ever in production.
> 
> It's being used in production at Facebook and in Oracle Linux and SUSE Linux.
> 
> I don't know how SUSE does it but Oracle Oracle has kernel and
> kernel-uek packages. The kernel package is the same as SL's and the
> kernel-uek package provides a more recent kernel that goes
> hand-in-hand with the more recent btrfs tools.

Last time I checked, SUSE Linux typically had disabled all of the really
useful btrfs features which makes it more interesting than XFS/ext4.
But I'll admit my reference point is a few years old by now (approx as
old as RHEL7 is).

What Facebook does, I have no idea how they use it.  It might even be
used in areas where data loss is acceptable due to data being replicated
elsewhere too.  Or again, that some features are disabled again.
Sailfish OS (by Jolla) have used btrfs by default on their mobile OS,
that worked fine there too.  But I doubt that also used the most
interesting features.

In regards to Oracle, IIRC, they also employ many of the upstream btrfs
developers.  But I wonder how well they support it for customers
installing it in production, and which features are fully supported.

That is one of the true things which makes it hard to compare btrfs too
even between distros.  Each distro have their own opinion of which
features could/should be enabled.


-- 
kind regards,

David Sommerseth

Re: RAID 6 array and failing harddrives

[Tom H]

On Thu, Apr 6, 2017 at 5:01 PM, David Sommerseth
 wrote:


> I would never run btrfs on *any* production server, regardless of
> currently available kernel versions. Because it is not deemed ready
> for production yet. For testing I would be willing to experiment with
> it, as there I can tolerate data loss. But never ever in production.

It's being used in production at Facebook and in Oracle Linux and SUSE Linux.

I don't know how SUSE does it but Oracle Oracle has kernel and
kernel-uek packages. The kernel package is the same as SL's and the
kernel-uek package provides a more recent kernel that goes
hand-in-hand with the more recent btrfs tools.


>> But the "kmod" release of ZFS seems to work well enough. I do have to
>> disable automatic kernel updates, but that may be wise in some
>> configurations anyway, YMMV.
>
> By all means, if you feel confident you will get the needed help
> sorting out issues you hit on ZFS, go ahead. I would never do that,
> even for ZFS. Once Red Hat enabling ZFS on a kernel where it is native
> in the upstream kernel, not being labelled tech-preview - that day I
> will consider ZFS for production. If btrfs reaches this stage earlier,
> then I will consider btrfs instead.

As I said in a previous email, zfs isn't going to be in-tree (unless,
of course, Oracle changes its license...).

Other than having installed Ubuntu on btrfs twice in a VM and played
around with it, my knowledge of btrfs comes from gentoo-user@. Two
things stand out from those threads. The btrfs developers seem to be
more interested in adding features than in stability (in the words of
committed btrfs users; more interested in features doesn't mean that
stability is ignored) and, except for one or two kernels where there
were regressions, every new iteration seems to bring better stability.


> For me, the safest option _now_ is mdraid + LVM + XFS/ext4. Which
> combined is somewhat closer (but not directly comparable) to the
> features I like in both btrfs and ZFS. In addition, I know and
> understand the toolset needed for mdraid + LVM + XFS/ext4 fairly well,
> so I don't have to spend much extra time figure out the tooling if
> something bad happens. But I'm also not going to spend time learning
> btrfs/zfs tooling until it begins to be ready for production use.

Not quite. Unlike btrfs, zfs is stable but it doesn't fulfill your
in-tree criterion...

Re: RAID 6 array and failing harddrives

[Tom H]

On Thu, Apr 6, 2017 at 8:15 AM, David Sommerseth
 wrote:
> On 06/04/17 10:54, Tom H wrote:
>> On Wed, Apr 5, 2017 at 5:50 AM, David Sommerseth
>>  wrote:
>>>
>>> ZFS looks great, so does btrfs - on the paper. But until ZFS is native
>>> in Linux or btrfs stabilizes on the same level as ext4 and XFS, I'm
>>> not going that path for production environments.
>>
>> What do you mean by "native?"
>>
>> The upstream deb and rpm files use dkms (as well as kmod for RHEL and
>> clones) and Ubuntu ships zfs pre-compiled. This is "native" in my
>> book.
>>
>> I've used and am using zfs in production on Linux and it's good and stable.
>
> My meaning of "native" is that it is included in the upstream Linux
> kernel, not a side-loaded product/project/kernel module.

Thanks. I read a later email in which you said that.

zfs'll never be in-tree for licensing reasons.

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 06/04/17 21:17, Konstantin Olchanski wrote:
>> ... until ZFS is native in Linux ...
>> ... My meaning of "native" is that it is included in the upstream Linux
>> kernel, not a side-loaded product/project/kernel module.
> 
> At least for BTRFS, "native" seems to be a bad thing. The BTRFS version that 
> comes
> with el7 is quite behind the current development version and unlike the 
> "non-native" ZFS,

Perhaps you should re-read my previous mail [0] where I warn against
comparing btrfs against any other stabilized file systems?

Yes, ZFS is probably closest to btrfs on the feature whitepaper.  But
btrfs is not bleeding edge on EL7, partly due to the reasoning in the
other mail [0]; it is currently not really ready for proper production.
Several features are even disabled to reduce the risk of data loss even
further.  To my knowledge, btrfs is still a technical preview in RHEL 7 [1].

Red Hat won't introduce the latest and greatest upstream changes in an
enterprise distribution, especially not on a fresh and new file system.
And yes, 4 years since the on-disk format is labelled stable _is_ very
fresh.  Again, the "Technology Preview" label means:

   "However, these features are not fully supported under Red Hat
Subscription Level Agreements, may not be functionally complete,
and are not intended for production use." [2]

And as a reference point, I've also heard people bragging about openSUSE
shipping btrfs by default too.  But again, they have also disabled most
of the unstable features - which are the one which makes it a real
alternative to ZFS.

[0]

[1]

[2] 

> BTRFS is not packaged for easy installation into "my" linux. I guess I could 
> run el7
> with a "non-native" kernel. (but how is that better than running "non-native" 
> ZFS).

I would never run btrfs on *any* production server, regardless of
currently available kernel versions.  Because it is not deemed ready for
production yet.  For testing I would be willing to experiment with it,
as there I can tolerate data loss.  But never ever in production.

> But the "kmod" release of ZFS seems to work well enough. I do have to disable 
> automatic
> kernel updates, but that may be wise in some configurations anyway, YMMV.

By all means, if you feel confident you will get the needed help sorting
out issues you hit on ZFS, go ahead.  I would never do that, even for
ZFS.  Once Red Hat enabling ZFS on a kernel where it is native in the
upstream kernel, not being labelled tech-preview - that day I will
consider ZFS for production.  If btrfs reaches this stage earlier, then
I will consider btrfs instead.

For me, the safest option _now_ is mdraid + LVM + XFS/ext4.  Which
combined is somewhat closer (but not directly comparable) to the
features I like in both btrfs and ZFS.  In addition, I know and
understand the toolset needed for mdraid + LVM + XFS/ext4 fairly well,
so I don't have to spend much extra time figure out the tooling if
something bad happens.  But I'm also not going to spend time learning
btrfs/zfs tooling until it begins to be ready for production use.


-- 
kind regards,

David Sommerseth

Re: RAID 6 array and failing harddrives

[Konstantin Olchanski]

> ... until ZFS is native in Linux ...
> ... My meaning of "native" is that it is included in the upstream Linux
> kernel, not a side-loaded product/project/kernel module.

At least for BTRFS, "native" seems to be a bad thing. The BTRFS version that 
comes
with el7 is quite behind the current development version and unlike the 
"non-native" ZFS,
BTRFS is not packaged for easy installation into "my" linux. I guess I could 
run el7
with a "non-native" kernel. (but how is that better than running "non-native" 
ZFS).

In practice, it looks like the "dkms" stuff is junk (*for me*, it always breaks
between kernel releases - the problem it was supposed to solve).

But the "kmod" release of ZFS seems to work well enough. I do have to disable 
automatic
kernel updates, but that may be wise in some configurations anyway, YMMV.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 06/04/17 10:54, Tom H wrote:
> On Wed, Apr 5, 2017 at 5:50 AM, David Sommerseth
>  wrote:
>>
>> ZFS looks great, so does btrfs - on the paper. But until ZFS is native
>> in Linux or btrfs stabilizes on the same level as ext4 and XFS, I'm
>> not going that path for production environments.
> 
> What do you mean by "native?"
> 
> The upstream deb and rpm files use dkms (as well as kmod for RHEL and
> clones) and Ubuntu ships zfs pre-compiled. This is "native" in my
> book.
> 
> I've used and am using zfs in production on Linux and it's good and stable.

My meaning of "native" is that it is included in the upstream Linux
kernel, not a side-loaded product/project/kernel module.


-- 
kind regards,

David Sommerseth

Re: RAID 6 array and failing harddrives

[Tom H]

On Wed, Apr 5, 2017 at 5:50 AM, David Sommerseth
 wrote:
>
> ZFS looks great, so does btrfs - on the paper. But until ZFS is native
> in Linux or btrfs stabilizes on the same level as ext4 and XFS, I'm
> not going that path for production environments.

What do you mean by "native?"

The upstream deb and rpm files use dkms (as well as kmod for RHEL and
clones) and Ubuntu ships zfs pre-compiled. This is "native" in my
book.

I've used and am using zfs in production on Linux and it's good and stable.

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 05/04/17 11:50, David Sommerseth wrote:
> ZFS looks great, so does btrfs - on the paper.  But until ZFS is native
> in Linux or btrfs stabilizes on the same level as ext4 and XFS, I'm not
> going that path for production environments :)

Just to be clear, when I say "ZFS is native in Linux" - I do mean
included and available as a build option in the upstream Linux kernel.
In my opinion, the OpenZFS [1] effort is not "native" in this regard.

[1] 


-- 
kind regards,

David Sommerseth

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 05/04/17 05:03, Graham Allan wrote:
> 
>> BTRFS is billed as "open source replacement for ZFS", but after
>> testing it,
>> my impression is that it is only used by a couple of enthusiasts
>> in single-disk laptop configurations. In a single-disk system, it is not
>> clear how btrfs/zfs is better than old-plain ext4/xfs.
> 
> I've never seen any good success stories for btrfs but to be fair I have
> not followed it closely.

To be honest, it is fairly unfair to start comparing btrfs stability
with other file systems now. Even though, "ZFS on Linux" is probably the
closest to the best one to compare it against currently.  And I say "ZFS
on Linux" as there are several Linux implementations, which is different
from the original OpenSolaris implementation.

The reason is:  It takes incredibly long time to stabilize a new file
system.  And the more complex and feature rich it is, the longer it takes.

Take the history of ext4, which is based upon ext3 which is derived from
ext2.  In fact you can upgrade ext2 -> ext3 -> ext4 quite successfully.
So the whole ext2/3/4 file systems shares a lot of code.  ext2 got
introduced in 1994, ext3 in 2001 and ext4 in 2006.

For XFS, it arrived in IRIX in 1994 and arrived in Linux in 2001.

ZFS arrived in OpenSolaris in 2005.  And the Linux ports (depending on
which approach you look at) varies between 2006 and 2010.

Btrfs arrived in Linux in 2009.  And first in 2014(!) the on-disk format
was considered stable.

So btrfs do have a long journey ahead to get stabilized and become
really useful and truly integrated in larger deployments.  So expecting
btrfs to be feature rich and stable _now_ will not give any good
experiences.


-- 
kind regards,

David Sommerseth

Re: RAID 6 array and failing harddrives

[David Sommerseth]

On 05/04/17 00:58, Steven Haigh wrote:
> On 05/04/17 05:44, Konstantin Olchanski wrote:
[...snip...]
>> ZFS is also scary, but seems to behave well, even in the presence
>> of "bad disks". ZFS scrub seems to overcome/rewrite bad sectors,
>> bad data on disk (if I poop on a disk using "dd"), all without corrupting
>> data files (I compute and check my own sha-512 checksums for each file).
> 
> Heh - another soon to be victim of ZFS on linux :)

ZFS looks great, so does btrfs - on the paper.  But until ZFS is native
in Linux or btrfs stabilizes on the same level as ext4 and XFS, I'm not
going that path for production environments :)

Most of the nice features I need can in most cases be handled with
md-raid, LVM and XFS.  And those have worked very well over the last
decade for me.  I have not experienced a single data loss despite
failing drives, plus I have replaced drives and expanded the mount
points dynamically with close to no downtime (some hosts don't have
hot-swap, which will require some downtime).

[...snip...]
> 
> However, BTRFS is very stable if you use it as a simple filesystem. You
> will get more flexible results in using mdadm with btrfs on top of it.

This is my understanding as well, in addition to this is what several
file system gurus also say.

> mdadm can be a pain to tweak - but almost all problems are well known
> and documented - and unless you really lose all your parity, you'll be
> able to recover with much less data loss than most other concoctions.

I believe I found out what my issue was.  It really seems to have been
caused by poor cable seating.  After taking server down for "physical
maintenance" (vacuum cleaning, cable/drive re-seating, etc) no errors
occurred again.  I had failures happening regularly at at often around
30 minutes intervals or so before this.  Now the server have been up for
14 hours without a single failure.  Before it didn't even manage to
complete several of the longer smartd self-tests.  But after this
maintenance round, all tests completed without any issues or error reports.

Re-adding (mdadm --re-add) the failed drives also worked flawlessly.
RAID 6 recovered incredibly quickly (despite it being 4TB) - took about
10 minutes.  So this time, it was yet another md-raid + LVM + XFS success.

But what I really learnt was that the SMART pre-failure reports from
smartctl may not always be as bad as it sounds.  I discovered that
Seagate puts a lot of additional data encoded into these fields and
these numbers must be decoded properly.  In addition it's key to
understand the calculations done for these reports too.  And in
particular the RAW_VALUE column can be very misleading.  One of the more
interesting reads I found was this one:



-- 
kind regards,

David Sommerseth



>> On Tue, Apr 04, 2017 at 04:17:22PM +0200, David Sommerseth wrote:
>>> Hi,
>>>
>>> I just need some help to understand what might be the issue on a SL7.3
>>> server which today decided to disconnect two drives from a RAID 6 setup.
>>>
>>> First some gory details
>>>
>>> - smartctl + mdadm output
>>> 
>>>
>>> - kernel log messages
>>> https://paste.fedoraproject.org/paste/mkyjZINKnkD4SQcXTSxyt15M1UNdIGYhyRLivL9gydE=
>>>
>>>
>>> The server is setup with 2x WD RE4 harddrives and 2x Seagate
>>> Constellation ES.3 drives.  All 4TB, all was bought brand new.  They're
>>> installed in a mixed pattern (sda: RE4, sdb: ES3, sdc: RE4, sdd: ES3)
>>> ... and the curious devil in the detail ... there are no /dev/sde
>>> installed on this system - never have been even, at least not on that
>>> controller.  (Later today, I attached a USB drive to make some backups -
>>> which got designated /dev/sde)
>>>
>>> This morning *both* ES.3 drives (sdb, sdd) got disconnected and removed
>>> from the mdraid setup.  With just minutes in between.  On drives which
>>> have been in production for less than 240 days or so.
>>>
>>> lspci details:
>>> 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
>>> Family SATA AHCI Controller (rev 05)
>>>
>>> Server: HP ProLiant MicroServer Gen8 (F9A40A)
>>>
>>> 
>>>
>>>
>>> Have any one else experienced such issues?  Several places on the net,
>>> the ata kernel error messages have been resolved by checking SATA cables
>>> and their seating.  It just sounds a bit too incredible that two
>>> harddrives of the same brand and type in different HDD slots have the
>>> same issues but not at the exact same time (but close, though).  And I
>>> struggle to believe two identical drives just failing so close in time.
>>>
>>> What am I missing? :)  Going to shut down the server soon (after last
>>> backup round) and will double check all the HDD seating and cabling.
>>> But I'm not convinced

Re: RAID 6 array and failing harddrives

[Graham Allan]

On 4/4/2017 6:59 PM, Konstantin Olchanski wrote:

Moving to ZFS...
ZFS is also scary...


Heh - another soon to be victim of ZFS on linux :)



No kidding. Former victim of XLV+XFS (remember XLV?), former
victim of LV+EFS, former victim of ext2, ext3, reiserfs, former
victim of LVM, current victim of mdadm/raid5/6/ext4/xfs.



You'll quickly realise that the majority of major features you'd expect
to work - don't.


I am not big on "features". For me the main features is 
open()/read()/write()/close(),
mkdir()/rmdir()/readdir() and those seem to work on all filesystems. Next 
features are:
a) non-scary raid rebuild after a crash or disk failure,
b) "online" fsck



You can't grow a ZFS 'raid'. You're stuck with the number of disks you first 
start with.


(I know this is 2 quotes back) That's kind of unfair since here you're 
talking about a feature that was never offered, it's just an incorrect 
assumption. It took me a while to understand what ZFS does and does not 
offer as well - I missed many things from (ancient history) Digital's 
advfs - but ZFS does lots of things quite well. There really is no such 
thing as "a ZFS raid", that's probably most analogous to a zfs pool made 
of a single raidz vdev, but that's a very simple case. What other system 
lets you make large reliable storage pools from hundreds of drives on a 
single server? I built some with 200+ 4TB drives some years back.



We only have a few hardware configurations, all with fixed number of disks, so 
not a problem:

a) single 120GB ssd for OS (/home on NFS)
b) single SSD for OS, dual 4-6-8 TB HDD for data, RAID1 configuration to 
protect against single disk failure
c) dual SSD for OS and /home, dual HDD for data, both RAID1 configuration to 
protect against single disk failure
d) single SSD for OS, multiple (usually 8) 6-8 TB HDDs for data, mdadm 
raid6+xfs and now raidz2 ZFS (protection against single disk failure + failure 
of second disk during raid rebuild).


For case (b) for your data storage, you can expand a ZFS mirror 
reasonably easily.
For case (c) I don't know how hard it is to use ZFS for the OS drive on 
linux; I only used it on BSD. But mdadm on linux is ok for that role.
For case (d) it is true that you cannot expand a ZFS RAIDZ(2) vdev, but 
that's ok if you know that going in.



BTRFS is billed as "open source replacement for ZFS", but after testing it,
my impression is that it is only used by a couple of enthusiasts
in single-disk laptop configurations. In a single-disk system, it is not
clear how btrfs/zfs is better than old-plain ext4/xfs.


I've never seen any good success stories for btrfs but to be fair I have 
not followed it closely.


zfs can still give you some useful things on a single drive system: you 
get the data checksums, useful snapshots (as opposed to LVM), volume 
manager features, etc.


By default the checksums would only warn you of problems with a single 
drive, but you can tell zfs to keep multiple copies of your data ("zfs 
set copies=n") so that might well also let it recover from bad blocks.


G.
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu

Re: RAID 6 array and failing harddrives

[Konstantin Olchanski]

> > Moving to ZFS...
> > ZFS is also scary...
>
> Heh - another soon to be victim of ZFS on linux :)
> 

No kidding. Former victim of XLV+XFS (remember XLV?), former
victim of LV+EFS, former victim of ext2, ext3, reiserfs, former
victim of LVM, current victim of mdadm/raid5/6/ext4/xfs.

>
> You'll quickly realise that the majority of major features you'd expect
> to work - don't.
>

I am not big on "features". For me the main features is 
open()/read()/write()/close(),
mkdir()/rmdir()/readdir() and those seem to work on all filesystems. Next 
features are:
a) non-scary raid rebuild after a crash or disk failure,
b) "online" fsck

>
> You can't grow a ZFS 'raid'. You're stuck with the number of disks you first 
> start with.
>

We only have a few hardware configurations, all with fixed number of disks, so 
not a problem:

a) single 120GB ssd for OS (/home on NFS)
b) single SSD for OS, dual 4-6-8 TB HDD for data, RAID1 configuration to 
protect against single disk failure
c) dual SSD for OS and /home, dual HDD for data, both RAID1 configuration to 
protect against single disk failure
d) single SSD for OS, multiple (usually 8) 6-8 TB HDDs for data, mdadm 
raid6+xfs and now raidz2 ZFS (protection against single disk failure + failure 
of second disk during raid rebuild).

Storage requirements are always known ahead of time, there is no need to add 
storage space
until time of major refresh when we install centos N+1, replace 2TB disks with 
6TB disks,
4TB disks with 8TB disks and so forth, always in pairs.

>
> You'll find out more as you go down this rabbit hole.
> 

There is no rabbit hole for me. If me no like zfs, I roll out a new machine
with a new-fs and a new-raid, rsync copy the data, and say bye, bye zfs.

>
> > BTRFS is even better (on paper), but not usable in el7.3...
> 
> DO NOT USE RAID5/6 WITHIN BTRFS.
> 

BTRFS is billed as "open source replacement for ZFS", but after testing it,
my impression is that it is only used by a couple of enthusiasts
in single-disk laptop configurations. In a single-disk system, it is not
clear how btrfs/zfs is better than old-plain ext4/xfs. And anyway,
single-disk system is not appropriate for production use as there is no
protection against single-disk failure. (based on existing failure-rate
data one can make the argument that SSDs never fail and do not require
protection against single-disk failure).


> I have tried this before and have the many Gb of lost data when it goes
> wrong. In fact, I discovered several new bugs that I lodged with the
> BTRFS guys - which led to warnings of DO NOT USE PARITY BASED RAID
> LEVELS IN BTRFS becoming the official line.

No lost data here, yet. All home directories are backed-up, all data directories
the experiments are required to make second copies of their data (usually
on cloud-type storage).


> However, BTRFS is very stable if you use it as a simple filesystem. You
> will get more flexible results in using mdadm with btrfs on top of it.

For simple filesystem for single-disk use we already have ext4 and xfs,
both work well enough, thank you very much.

For multiple-disk installations (pairs of RAID1 for single-disk-failure
protection and 3-4-8-disk sets of RAID6 (6-8-10 TB HDDs) for large capacity),
md raid does not cut it anymore because raid rebuild takes unreasonable
amounts of time (days) and lacks self-healing functions (i.e. no rewriting
of bad sectors, no reasonable handling of disk errors, etc). If lights blink
during rebuild (or you bump the power bar, or ...), you have some exciting
time recovering the raid array without any guarranty of data integrity
(no per-file checksums).


> mdadm can be a pain to tweak - but almost all problems are well known
> and documented - and unless you really lose all your parity, you'll be
> able to recover with much less data loss than most other concoctions.


mdadm does not cut it for 8x10TB RAID6. raid rebuild takes *days*.


K.O.



> 
> > 
> > K.O.
> > 
> > 
> > On Tue, Apr 04, 2017 at 04:17:22PM +0200, David Sommerseth wrote:
> >> Hi,
> >>
> >> I just need some help to understand what might be the issue on a SL7.3
> >> server which today decided to disconnect two drives from a RAID 6 setup.
> >>
> >> First some gory details
> >>
> >> - smartctl + mdadm output
> >> 
> >>
> >> - kernel log messages
> >> https://paste.fedoraproject.org/paste/mkyjZINKnkD4SQcXTSxyt15M1UNdIGYhyRLivL9gydE=
> >>
> >>
> >> The server is setup with 2x WD RE4 harddrives and 2x Seagate
> >> Constellation ES.3 drives.  All 4TB, all was bought brand new.  They're
> >> installed in a mixed pattern (sda: RE4, sdb: ES3, sdc: RE4, sdd: ES3)
> >> ... and the curious devil in the detail ... there are no /dev/sde
> >> installed on this system - never have been even, at least not on that
> >> controller.  (Later today, I attached a USB drive to make some backups -
> >> which got designated /dev/sde)
> >>
> >> This morning *both

Re: RAID 6 array and failing harddrives

[Steven Haigh]

On 05/04/17 05:44, Konstantin Olchanski wrote:
> Moving to ZFS because of issues like this. RAID6 rebuild with 4-6-8-10TB disks
> has become too scary. If there is any transient error during the rebuild,
> the md driver starts kicking disks out, getting into funny states with many
> "missing" disks, recovery is only via "mdadm --assemble --force" and without
> per-file checksums in ext4/xfs there is no confidence whatsoever that data
> was not subtly corrupted.
> 
> ZFS is also scary, but seems to behave well, even in the presence
> of "bad disks". ZFS scrub seems to overcome/rewrite bad sectors,
> bad data on disk (if I poop on a disk using "dd"), all without corrupting
> data files (I compute and check my own sha-512 checksums for each file).

Heh - another soon to be victim of ZFS on linux :)

You'll quickly realise that the majority of major features you'd expect
to work - don't. You can't grow a ZFS 'raid'. You're stuck with the
number of disks you first start with. You'll find out more as you go
down this rabbit hole.

> BTRFS is even better (on paper), but not usable in el7.3 because it has no
> concept of "failed disk". If you pull a disk on btrfs, it will fill /var/log
> with disk error messages, will not take any mitigation/recovery action
> (drop disk from array, rerun the data balancer, etc).

DO NOT USE RAID5/6 WITHIN BTRFS.

I have tried this before and have the many Gb of lost data when it goes
wrong. In fact, I discovered several new bugs that I lodged with the
BTRFS guys - which led to warnings of DO NOT USE PARITY BASED RAID
LEVELS IN BTRFS becoming the official line.

However, BTRFS is very stable if you use it as a simple filesystem. You
will get more flexible results in using mdadm with btrfs on top of it.

mdadm can be a pain to tweak - but almost all problems are well known
and documented - and unless you really lose all your parity, you'll be
able to recover with much less data loss than most other concoctions.

> 
> K.O.
> 
> 
> On Tue, Apr 04, 2017 at 04:17:22PM +0200, David Sommerseth wrote:
>> Hi,
>>
>> I just need some help to understand what might be the issue on a SL7.3
>> server which today decided to disconnect two drives from a RAID 6 setup.
>>
>> First some gory details
>>
>> - smartctl + mdadm output
>> 
>>
>> - kernel log messages
>> https://paste.fedoraproject.org/paste/mkyjZINKnkD4SQcXTSxyt15M1UNdIGYhyRLivL9gydE=
>>
>>
>> The server is setup with 2x WD RE4 harddrives and 2x Seagate
>> Constellation ES.3 drives.  All 4TB, all was bought brand new.  They're
>> installed in a mixed pattern (sda: RE4, sdb: ES3, sdc: RE4, sdd: ES3)
>> ... and the curious devil in the detail ... there are no /dev/sde
>> installed on this system - never have been even, at least not on that
>> controller.  (Later today, I attached a USB drive to make some backups -
>> which got designated /dev/sde)
>>
>> This morning *both* ES.3 drives (sdb, sdd) got disconnected and removed
>> from the mdraid setup.  With just minutes in between.  On drives which
>> have been in production for less than 240 days or so.
>>
>> lspci details:
>> 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
>> Family SATA AHCI Controller (rev 05)
>>
>> Server: HP ProLiant MicroServer Gen8 (F9A40A)
>>
>> 
>>
>>
>> Have any one else experienced such issues?  Several places on the net,
>> the ata kernel error messages have been resolved by checking SATA cables
>> and their seating.  It just sounds a bit too incredible that two
>> harddrives of the same brand and type in different HDD slots have the
>> same issues but not at the exact same time (but close, though).  And I
>> struggle to believe two identical drives just failing so close in time.
>>
>> What am I missing? :)  Going to shut down the server soon (after last
>> backup round) and will double check all the HDD seating and cabling.
>> But I'm not convinced that's all just yet.
>>
>>
>> -- 
>> kind regards,
>>
>> David Sommerseth
> 

-- 
Steven Haigh

Email: net...@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897



signature.asc
Description: OpenPGP digital signature

Re: RAID 6 array and failing harddrives

[Konstantin Olchanski]

Moving to ZFS because of issues like this. RAID6 rebuild with 4-6-8-10TB disks
has become too scary. If there is any transient error during the rebuild,
the md driver starts kicking disks out, getting into funny states with many
"missing" disks, recovery is only via "mdadm --assemble --force" and without
per-file checksums in ext4/xfs there is no confidence whatsoever that data
was not subtly corrupted.

ZFS is also scary, but seems to behave well, even in the presence
of "bad disks". ZFS scrub seems to overcome/rewrite bad sectors,
bad data on disk (if I poop on a disk using "dd"), all without corrupting
data files (I compute and check my own sha-512 checksums for each file).

BTRFS is even better (on paper), but not usable in el7.3 because it has no
concept of "failed disk". If you pull a disk on btrfs, it will fill /var/log
with disk error messages, will not take any mitigation/recovery action
(drop disk from array, rerun the data balancer, etc).


K.O.


On Tue, Apr 04, 2017 at 04:17:22PM +0200, David Sommerseth wrote:
> Hi,
> 
> I just need some help to understand what might be the issue on a SL7.3
> server which today decided to disconnect two drives from a RAID 6 setup.
> 
> First some gory details
> 
> - smartctl + mdadm output
> 
> 
> - kernel log messages
> https://paste.fedoraproject.org/paste/mkyjZINKnkD4SQcXTSxyt15M1UNdIGYhyRLivL9gydE=
> 
> 
> The server is setup with 2x WD RE4 harddrives and 2x Seagate
> Constellation ES.3 drives.  All 4TB, all was bought brand new.  They're
> installed in a mixed pattern (sda: RE4, sdb: ES3, sdc: RE4, sdd: ES3)
> ... and the curious devil in the detail ... there are no /dev/sde
> installed on this system - never have been even, at least not on that
> controller.  (Later today, I attached a USB drive to make some backups -
> which got designated /dev/sde)
> 
> This morning *both* ES.3 drives (sdb, sdd) got disconnected and removed
> from the mdraid setup.  With just minutes in between.  On drives which
> have been in production for less than 240 days or so.
> 
> lspci details:
> 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
> Family SATA AHCI Controller (rev 05)
> 
> Server: HP ProLiant MicroServer Gen8 (F9A40A)
> 
> 
> 
> 
> Have any one else experienced such issues?  Several places on the net,
> the ata kernel error messages have been resolved by checking SATA cables
> and their seating.  It just sounds a bit too incredible that two
> harddrives of the same brand and type in different HDD slots have the
> same issues but not at the exact same time (but close, though).  And I
> struggle to believe two identical drives just failing so close in time.
> 
> What am I missing? :)  Going to shut down the server soon (after last
> backup round) and will double check all the HDD seating and cabling.
> But I'm not convinced that's all just yet.
> 
> 
> -- 
> kind regards,
> 
> David Sommerseth

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

RAID 6 array and failing harddrives

[David Sommerseth]

Hi,

I just need some help to understand what might be the issue on a SL7.3
server which today decided to disconnect two drives from a RAID 6 setup.

First some gory details

- smartctl + mdadm output


- kernel log messages
https://paste.fedoraproject.org/paste/mkyjZINKnkD4SQcXTSxyt15M1UNdIGYhyRLivL9gydE=


The server is setup with 2x WD RE4 harddrives and 2x Seagate
Constellation ES.3 drives.  All 4TB, all was bought brand new.  They're
installed in a mixed pattern (sda: RE4, sdb: ES3, sdc: RE4, sdd: ES3)
... and the curious devil in the detail ... there are no /dev/sde
installed on this system - never have been even, at least not on that
controller.  (Later today, I attached a USB drive to make some backups -
which got designated /dev/sde)

This morning *both* ES.3 drives (sdb, sdd) got disconnected and removed
from the mdraid setup.  With just minutes in between.  On drives which
have been in production for less than 240 days or so.

lspci details:
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
Family SATA AHCI Controller (rev 05)

Server: HP ProLiant MicroServer Gen8 (F9A40A)




Have any one else experienced such issues?  Several places on the net,
the ata kernel error messages have been resolved by checking SATA cables
and their seating.  It just sounds a bit too incredible that two
harddrives of the same brand and type in different HDD slots have the
same issues but not at the exact same time (but close, though).  And I
struggle to believe two identical drives just failing so close in time.

What am I missing? :)  Going to shut down the server soon (after last
backup round) and will double check all the HDD seating and cabling.
But I'm not convinced that's all just yet.


-- 
kind regards,

David Sommerseth