Have you opened a ticket with Lenovo and/or IBM about this?
If there is a genuine bug here (and it seems there might be), that's the way to
get it fixed.
Generally find the disk hospital very reliable and it takes disks out of
"rotation" (pun intended) for slow performance long before they cause any
problems, but have yet to see it the other way round - although if it's not
reporting things we could be missing them...
Cheers,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
-----Original Message-----
From: gpfsug-discuss <gpfsug-discuss-boun...@gpfsug.org> On Behalf Of Jonathan
Buzzard
Sent: Monday, June 24, 2024 1:52 PM
To: Achim Rehor <achim.re...@de.ibm.com>; gpfsug-discuss@gpfsug.org
Subject: Re: [gpfsug-discuss] Bad disk but not failed in DSS-G
CAUTION: This email originated from outside the organisation. Do not click
links or open attachments unless you recognise the sender and know the content
is safe.
On 24/06/2024 13:16, Achim Rehor wrote:
> CAUTION: This email originated outside the University. Check before
> clicking links or attachments.
> well ... not necessarily 😄
> but on the disk ... just as i expected ... taking it out helps a lot.
>
> Now on taking it out automatically when raising too many errors was a
> discussion i had several times with the GNR development.
> The issue really is .. I/O errors on disks (as seen in the
> mmlsrecoverygroupevent logs) can be due to several issues (the disk
> itself,
> the expander, the IOM, the adapter, the cable ... )
> in case of a more general part serving like 5 or more pdisks, that would
> risk the FT , if we took them out automatically.
> Thus ... we dont do that ..
>
When smartctl for the disk says
Error counter log:
Errors Corrected by Total Correction
Gigabytes Total
ECC rereads/ errors algorithm
processed uncorrected
fast | delayed rewrites corrected invocations [10^9
bytes] errors
read: 0 33839 32 0 0 137434.705
32
write: 0 36 0 0 0 178408.893
0
Non-medium error count: 0
A disk with 32 read errors in smartctl is fubar, no ifs no buts.
Whatever the balance in ejecting bad disks is, IMHO currently it's in
the wrong place because it failed to eject an actual bad disk.
At an absolute bare minimum mmhealth should be not be saying everything
is fine and dandy because clearly it was not. That's the bigger issue. I
can live with them not been taken out automatically, it is unacceptable
that mmhealth was giving false and inaccurate information about the
state of the filesystem. Had it even just changed something to a
"degraded" state the problems could have been picked up much much sooner.
Presumably the disk category was still good because the vdisk's where
theoretically good. I suggest renaming that to VDISK to more accurately
reflect what it is about and add a PDISK category. Then when a pdisk
starts showing IO errors you can increment the number of disks in a
degraded state and it can be picked up without end users having to roll
their own monitoring.
> The idea is to improve the disk hospital more and more, so that the
> decision to switch a disk back to OK is more accurate, over time.
>
> Until then .. it might always be a good idea to scan the event log for
> pdisk errors ...
>
That is my conclusion, that mmhealth is as useful as a chocolate teapot
because you can't rely on it to provide correct information and I need
to do my own health monitoring of the system.
JAB.
--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
