Hi Benoît,
On Mon, Nov 8, 2021 at 4:31 PM Benoît Knecht wrote:
>
> Hi Dan,
>
> On Thursday, November 4th, 2021 at 11:33, Dan van der Ster
> wrote:
> > - Are we running the same firmware as you? (We have 0104). I wonder if
> > Toshiba has changed the implementation of the cache in the
Thanks Mark.
With the help of the crowd on Telegram, we found that (at least here)
the drive cache needs to be disabled like this:
```
for x in /sys/class/scsi_disk/*/cache_type; do echo 'write through' > $x; done
```
This disables the cache (confirmed afterwards with hdparm) but more
Hi Dan,
I can't speak for those specific Toshiba drives, but we have absolutely
seen very strange behavior (sometimes with cache enabled and sometimes
not) with different drives and firmwares over the years from various
manufacturers. There was one especially bad case from back in the
Hello Benoît, (and others in this great thread),
Apologies for replying to this ancient thread.
We have been debugging similar issues during an ongoing migration to
new servers with TOSHIBA MG07ACA14TE hdds.
We see a similar commit_latency_ms issue on the new drives (~60ms in
our env vs ~20ms
Am 24.06.2020 um 18:08 schrieb Marc Roos:
I can remember reading this before. I was hoping you maybe had some
setup with systemd scripts or maybe udev.
We use udev to disable write cache once a suitable disk is detected,
base on the MODEL_ID from udev environment:
ACTION=="add",
On 25/06/2020 5:10 pm, Frank Schilder wrote:
I was pondering with that. The problem is, that on Centos systems it seems to
be ignored, in general it does not apply to SAS drives, for example, and that
it has no working way of configuring which drives to exclude.
For example, while for data
me it. At time of writing, I don't
know of any.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Marc Roos
Sent: 25 June 2020 00:01:51
To: paul.emmerich; vitalif
Cc: bknecht; ceph-users; s.priebe
Subje
FS=' '
arr=($line); service ceph-osd@${arr[1]} stop && smartctl -s wcache,off
${arr[0]} && service ceph-osd@${arr[1]} start ;done
-Original Message-
To: Paul Emmerich
Cc: Benoît Knecht; s.pri...@profihost.ag; ceph-users@ceph.io
Subject: [ceph-users] Re: High ceph_osd_commit_
Hi, https://yourcmc.ru/wiki/Ceph_performance author here %)
Disabling write cache is REALLY bad for SSDs without capacitors
[consumer SSDs], also it's bad for HDDs with firmwares that don't have
this bug-o-feature. The bug is really common though. I have no idea
where it comes from, but it's
On 25/06/2020 3:17 am, dhils...@performair.com wrote:
Completely non-portable, but...
Couldn't you write a script to issue the necessary commands to the desired
drives, then create a system unit that calls it before OSD initialization?
Couldn't we just set (uncomment)
write_cache = off
in
The benefit of disabling on-drive cache may be at least partly dependent on the
HBA; I’ve done testing of one specific drive model and found no difference,
where someone else reported a measurable difference for the same model.
> Good to know that we're not alone :) I also looked for a newer
T Risø Campus
Bygning 109, rum S14
From: Marc Roos
Sent: 24 June 2020 18:08:49
To: Frank Schilder; paul.emmerich
Cc: bknecht; ceph-users; s.priebe
Subject: RE: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
MG07ACA14TE HDDs
> Sorry for the sp
,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Frank Schilder
Sent: 24 June 2020 18:00:19
To: Marc Roos; paul.emmerich
Cc: bknecht; ceph-users; s.priebe
Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
MG07ACA14TE HDDs
I
9, rum S14
From: Marc Roos
Sent: 24 June 2020 17:55:35
To: Frank Schilder; paul.emmerich
Cc: bknecht; ceph-users; s.priebe
Subject: RE: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
MG07ACA14TE HDDs
> I run the corresponding smartctl com
Emmerich
Sent: 24 June 2020 17:39:16
To: Frank Schilder
Cc: Frank R; Benoît Knecht; s.pri...@profihost.ag; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
MG07ACA14TE HDDs
Well, what I was saying was "does it hurt to unconditionally run hdparm -W 0 on
.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Paul Emmerich
Sent: 24 June 2020 17:30:51
To: Frank R
Cc: Benoît Knecht; s.pri...@profihost.ag; ceph-users@ceph.io
Subject: [ceph-users] Re: High
Hi Igor,
Igor Fedotov wrote:
> for the sake of completeness one more experiment please if possible:
>
> turn off write cache for HGST drives and measure commit latency once again.
I just did the same experiment with HGST drives, and disabling the write cache
on those drives brought the latency
>> I can remember reading this before. I was hoping you maybe had some
>> setup with systemd scripts or maybe udev.
>
> Yeah, doing this on boot up would be ideal. I was looking really hard into
> tuned and other services that claimed can do it, but required plugins or
> other stuff did/does
rformAir.com
>
>
>
> -Original Message-
> From: Frank Schilder [mailto:fr...@dtu.dk]
> Sent: Wednesday, June 24, 2020 9:15 AM
> To: Marc Roos; paul.emmerich
> Cc: bknecht; ceph-users; s.priebe
> Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
&g
; paul.emmerich
Cc: bknecht; ceph-users; s.priebe
Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
MG07ACA14TE HDDs
> I can remember reading this before. I was hoping you maybe had some
> setup with systemd scripts or maybe udev.
Yeah, doing this on boot up would be ideal. I was l
Thank you all for your answers, this was really helpful!
Stefan Priebe wrote:
> yes we have the same issues and switched to seagate for those reasons.
> you can fix at least a big part of it by disabling the write cache of
> those drives - generally speaking it seems the toshiba firmware is
>
> Sorry for the spam, but I need to add this disclaimer:
> Although it is documented as safe to disable volatile write cache on a
disk in use, I would
> probably not do it. The required cache flush might be erroneous in the
firmware.
I can remember reading this before. I was hoping you
Just throwing my hat in here with a small bit of anecdotal experience.
In the early days of experimenting with ceph, I had 24x 8T disk, all behind
RAID controllers as R0 vd's with no BBU (so controller cache is WT, default
value), and pdcache (disk write cache) enabled (default value).
We had
> I run the corresponding smartctl command on every drive just before
OSD daemon start.
How/where did you do this?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
> From: Paul Emmerich
> Sent: 24 June 2020 17:30:51
> To: Frank R
> Cc: Benoît Knecht; s.pri...@profihost.ag; ceph-users@ceph.io
> Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba
> MG07ACA14TE HDDs
>
> Has anyone ever enc
Has anyone ever encountered a drive with a write cache that actually
*helped*?
I haven't.
As in: would it be a good idea for the OSD to just disable the write cache
on startup? Worst case it doesn't do anything, best case it improves
latency.
Paul
--
Paul Emmerich
Looking for help with your
fyi, there is an interesting note on disabling the write cache here:
https://yourcmc.ru/wiki/index.php?title=Ceph_performance=toggle_view_desktop#Drive_cache_is_slowing_you_down
On Wed, Jun 24, 2020 at 9:45 AM Benoît Knecht wrote:
>
> Hi Igor,
>
> Igor Fedotov wrote:
> > for the sake of
Benoit, thanks for the update.
for the sake of completeness one more experiment please if possible:
turn off write cache for HGST drives and measure commit latency once again.
Kind regards,
Igor
On 6/24/2020 3:53 PM, Benoît Knecht wrote:
Thank you all for your answers, this was really
Benoit, wondering what are the write cache settings in your case?
And do you see any difference after disabling it if any?
Thanks,
Igor
On 6/24/2020 3:16 PM, Mark Nelson wrote:
This isn't the first time I've seen drive cache cause problematic
latency issues, and not always from the same
This isn't the first time I've seen drive cache cause problematic
latency issues, and not always from the same manufacturer. Unfortunately
it seems like you really have to test the drives you want to use before
deploying them them to make sure you don't run into issues.
Mark
On 6/24/20
HI Ben,
yes we have the same issues and switched to seagate for those reasons.
you can fix at least a big part of it by disabling the write cache of
those drives - generally speaking it seems the toshiba firmware is broken.
I was not able to find a newer one.
Greets,
Stefan
Am 24.06.20 um
31 matches
Mail list logo
