Re: [ceph-users] OSD tuning no longer required?
Hi there,
I would like just to note, that for some scenarios defaults are not good enough.
Recently we upgraded one of our clusters from Jewel to Luminous, during the
upgrade we removed all the custom tuning we had done on it over the years from
ceph.conf -I was extremely excited to get rid of all those parameters!!- but
performance after the upgrade was really bad.
The cluster runs mainly over write cache backed HDDs (via dedicated SAS
controller), so looks like the defaults for HDDs are for really low performance
HDD setups.
So, at the end we had to reapply our previous config, and tune some other -not
as well documented as I would like- parameters. And now, we have even more
tuning parameters than we did… :/
I think the main issue is the lack of a updated guide to tune ceph OSD
performance. Or at least the proper documentation in order to know what each
parameter really does (And I’m running ceph clusters since cuttlefish, so I did
spend my time checking documentation, presentations, etc…)
It’s a pity so many new users become disappointed with ceph, because they don’t
get even near to the performance they are expecting.
I still have to try the new default values in a pure NVMe cluster, I hope the
result will be better.
I think a good document about how to tune OSD performance, would really help
Ceph :)
Cheers!
Saludos Cordiales,
Xavier Trilla P.
Clouding.io<https://clouding.io/>
¿Un Servidor Cloud con SSDs, redundado
y disponible en menos de 30 segundos?
¡Pruébalo ahora en Clouding.io<https://clouding.io/>!
De: ceph-users En nombre de Robert Stanford
Enviado el: lunes, 16 de julio de 2018 21:40
Para: Gregory Farnum
CC: ceph-users ; Konstantin Shalygin
Asunto: Re: [ceph-users] OSD tuning no longer required?
Golden advice. Thank you Greg
On Mon, Jul 16, 2018 at 1:45 PM, Gregory Farnum
mailto:gfar...@redhat.com>> wrote:
On Fri, Jul 13, 2018 at 2:50 AM Robert Stanford
mailto:rstanford8...@gmail.com>> wrote:
This is what leads me to believe it's other settings being referred to as well:
https://ceph.com/community/new-luminous-rados-improvements/
"There are dozens of documents floating around with long lists of Ceph
configurables that have been tuned for optimal performance on specific hardware
or for specific workloads. In most cases these ceph.conf fragments tend to
induce funny looks on developers’ faces because the settings being adjusted
seem counter-intuitive, unrelated to the performance of the system, and/or
outright dangerous. Our goal is to make Ceph work as well as we can out of the
box without requiring any tuning at all, so we are always striving to choose
sane defaults. And generally, we discourage tuning by users. "
To me it's not just bluestore settings / sdd vs. hdd they're talking about
("dozens of documents floating around"... "our goal... without any tuning at
all". Am I off base?
Ceph is *extremely* tunable, because whenever we set up a new behavior
(snapshot trimming sleeps, scrub IO priorities, whatever) and we're not sure
how it should behave we add a config option. Most of these config options we
come up with some value through testing or informed guesswork, set it in the
config, and expect that users won't ever see it. Some of these settings we
don't know what they should be, and we really hope the whole mechanism gets
replaced before users see it, but they don't. Some of the settings should be
auto-tuning or manually set to a different value for each deployment to get
optimal performance.
So there are lots of options for people to make things much better or much
worse for themselves.
However, by far the biggest impact and most common tunables are those that
basically vary on if the OSD is using a hard drive or an SSD for its local
storage — those are order-of-magnitude differences in expected latency and
throughput. So we now have separate default tunables for those cases which are
automatically applied.
Could somebody who knows what they're doing tweak things even better for a
particular deployment? Undoubtedly. But do *most* people know what they're
doing that well? They don't.
In particular, the old "fix it" configuration settings that a lot of people
were sharing and using starting in the Cuttlefish days are rather dangerously
out of date, and we no longer have defaults that are quite as stupid as some of
those were.
So I'd generally recommend you remove any custom tuning you've set up unless
you have specific reason to think it will do better than the defaults for your
currently-deployed release.
-Greg
Regards
On Thu, Jul 12, 2018 at 9:12 PM, Konstantin Shalygin
mailto:k0...@k0ste.ru>> wrote:
I saw this in the Luminous release notes:
"Each OSD now adjusts its default configuration based on whether the
backing device is an HDD or SSD. Manual tuning generally not required"
Which tuning in particular? The ones i
Re: [ceph-users] OSD tuning no longer required?
Golden advice. Thank you Greg
On Mon, Jul 16, 2018 at 1:45 PM, Gregory Farnum wrote:
> On Fri, Jul 13, 2018 at 2:50 AM Robert Stanford
> wrote:
>
>>
>> This is what leads me to believe it's other settings being referred to
>> as well:
>> https://ceph.com/community/new-luminous-rados-improvements/
>>
>> *"There are dozens of documents floating around with long lists of Ceph
>> configurables that have been tuned for optimal performance on specific
>> hardware or for specific workloads. In most cases these ceph.conf
>> fragments tend to induce funny looks on developers’ faces because the
>> settings being adjusted seem counter-intuitive, unrelated to the
>> performance of the system, and/or outright dangerous. Our goal is to make
>> Ceph work as well as we can out of the box without requiring any tuning at
>> all, so we are always striving to choose sane defaults. And generally, we
>> discourage tuning by users. "*
>>
>> To me it's not just bluestore settings / sdd vs. hdd they're talking
>> about ("dozens of documents floating around"... "our goal... without any
>> tuning at all". Am I off base?
>>
>
> Ceph is *extremely* tunable, because whenever we set up a new behavior
> (snapshot trimming sleeps, scrub IO priorities, whatever) and we're not
> sure how it should behave we add a config option. Most of these config
> options we come up with some value through testing or informed guesswork,
> set it in the config, and expect that users won't ever see it. Some of
> these settings we don't know what they should be, and we really hope the
> whole mechanism gets replaced before users see it, but they don't. Some of
> the settings should be auto-tuning or manually set to a different value for
> each deployment to get optimal performance.
> So there are lots of options for people to make things much better or much
> worse for themselves.
>
> However, by far the biggest impact and most common tunables are those that
> basically vary on if the OSD is using a hard drive or an SSD for its local
> storage — those are order-of-magnitude differences in expected latency and
> throughput. So we now have separate default tunables for those cases which
> are automatically applied.
>
> Could somebody who knows what they're doing tweak things even better for a
> particular deployment? Undoubtedly. But do *most* people know what they're
> doing that well? They don't.
> In particular, the old "fix it" configuration settings that a lot of
> people were sharing and using starting in the Cuttlefish days are rather
> dangerously out of date, and we no longer have defaults that are quite as
> stupid as some of those were.
>
> So I'd generally recommend you remove any custom tuning you've set up
> unless you have specific reason to think it will do better than the
> defaults for your currently-deployed release.
> -Greg
>
>
>>
>> Regards
>>
>> On Thu, Jul 12, 2018 at 9:12 PM, Konstantin Shalygin
>> wrote:
>>
>>> I saw this in the Luminous release notes:
"Each OSD now adjusts its default configuration based on whether the
backing device is an HDD or SSD. Manual tuning generally not required"
Which tuning in particular? The ones in my configuration are
osd_op_threads, osd_disk_threads, osd_recovery_max_active,
osd_op_thread_suicide_timeout, and osd_crush_chooseleaf_type, among
others. Can I rip these out when I upgrade to
Luminous?
>>>
>>> This mean that some "bluestore_*" settings tuned for nvme/hdd separately.
>>>
>>> Also with Luminous we have:
>>>
>>> osd_op_num_shards_(ssd|hdd)
>>>
>>> osd_op_num_threads_per_shard_(ssd|hdd)
>>>
>>> osd_recovery_sleep_(ssd|hdd)
>>>
>>>
>>>
>>>
>>> k
>>>
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD tuning no longer required?
On Fri, Jul 13, 2018 at 2:50 AM Robert Stanford
wrote:
>
> This is what leads me to believe it's other settings being referred to as
> well:
> https://ceph.com/community/new-luminous-rados-improvements/
>
> *"There are dozens of documents floating around with long lists of Ceph
> configurables that have been tuned for optimal performance on specific
> hardware or for specific workloads. In most cases these ceph.conf
> fragments tend to induce funny looks on developers’ faces because the
> settings being adjusted seem counter-intuitive, unrelated to the
> performance of the system, and/or outright dangerous. Our goal is to make
> Ceph work as well as we can out of the box without requiring any tuning at
> all, so we are always striving to choose sane defaults. And generally, we
> discourage tuning by users. "*
>
> To me it's not just bluestore settings / sdd vs. hdd they're talking about
> ("dozens of documents floating around"... "our goal... without any tuning
> at all". Am I off base?
>
Ceph is *extremely* tunable, because whenever we set up a new behavior
(snapshot trimming sleeps, scrub IO priorities, whatever) and we're not
sure how it should behave we add a config option. Most of these config
options we come up with some value through testing or informed guesswork,
set it in the config, and expect that users won't ever see it. Some of
these settings we don't know what they should be, and we really hope the
whole mechanism gets replaced before users see it, but they don't. Some of
the settings should be auto-tuning or manually set to a different value for
each deployment to get optimal performance.
So there are lots of options for people to make things much better or much
worse for themselves.
However, by far the biggest impact and most common tunables are those that
basically vary on if the OSD is using a hard drive or an SSD for its local
storage — those are order-of-magnitude differences in expected latency and
throughput. So we now have separate default tunables for those cases which
are automatically applied.
Could somebody who knows what they're doing tweak things even better for a
particular deployment? Undoubtedly. But do *most* people know what they're
doing that well? They don't.
In particular, the old "fix it" configuration settings that a lot of people
were sharing and using starting in the Cuttlefish days are rather
dangerously out of date, and we no longer have defaults that are quite as
stupid as some of those were.
So I'd generally recommend you remove any custom tuning you've set up
unless you have specific reason to think it will do better than the
defaults for your currently-deployed release.
-Greg
>
> Regards
>
> On Thu, Jul 12, 2018 at 9:12 PM, Konstantin Shalygin
> wrote:
>
>> I saw this in the Luminous release notes:
>>>
>>> "Each OSD now adjusts its default configuration based on whether the
>>> backing device is an HDD or SSD. Manual tuning generally not required"
>>>
>>> Which tuning in particular? The ones in my configuration are
>>> osd_op_threads, osd_disk_threads, osd_recovery_max_active,
>>> osd_op_thread_suicide_timeout, and osd_crush_chooseleaf_type, among
>>> others. Can I rip these out when I upgrade to
>>> Luminous?
>>>
>>
>> This mean that some "bluestore_*" settings tuned for nvme/hdd separately.
>>
>> Also with Luminous we have:
>>
>> osd_op_num_shards_(ssd|hdd)
>>
>> osd_op_num_threads_per_shard_(ssd|hdd)
>>
>> osd_recovery_sleep_(ssd|hdd)
>>
>>
>>
>>
>> k
>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD tuning no longer required?
This is what leads me to believe it's other settings being referred to as
well:
https://ceph.com/community/new-luminous-rados-improvements/
*"There are dozens of documents floating around with long lists of Ceph
configurables that have been tuned for optimal performance on specific
hardware or for specific workloads. In most cases these ceph.conf
fragments tend to induce funny looks on developers’ faces because the
settings being adjusted seem counter-intuitive, unrelated to the
performance of the system, and/or outright dangerous. Our goal is to make
Ceph work as well as we can out of the box without requiring any tuning at
all, so we are always striving to choose sane defaults. And generally, we
discourage tuning by users. "*
To me it's not just bluestore settings / sdd vs. hdd they're talking about
("dozens of documents floating around"... "our goal... without any tuning
at all". Am I off base?
Regards
On Thu, Jul 12, 2018 at 9:12 PM, Konstantin Shalygin wrote:
> I saw this in the Luminous release notes:
>>
>> "Each OSD now adjusts its default configuration based on whether the
>> backing device is an HDD or SSD. Manual tuning generally not required"
>>
>> Which tuning in particular? The ones in my configuration are
>> osd_op_threads, osd_disk_threads, osd_recovery_max_active,
>> osd_op_thread_suicide_timeout, and osd_crush_chooseleaf_type, among
>> others. Can I rip these out when I upgrade to
>> Luminous?
>>
>
> This mean that some "bluestore_*" settings tuned for nvme/hdd separately.
>
> Also with Luminous we have:
>
> osd_op_num_shards_(ssd|hdd)
>
> osd_op_num_threads_per_shard_(ssd|hdd)
>
> osd_recovery_sleep_(ssd|hdd)
>
>
>
>
> k
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD tuning no longer required?
I saw this in the Luminous release notes: "Each OSD now adjusts its default configuration based on whether the backing device is an HDD or SSD. Manual tuning generally not required" Which tuning in particular? The ones in my configuration are osd_op_threads, osd_disk_threads, osd_recovery_max_active, osd_op_thread_suicide_timeout, and osd_crush_chooseleaf_type, among others. Can I rip these out when I upgrade to Luminous? This mean that some "bluestore_*" settings tuned for nvme/hdd separately. Also with Luminous we have: osd_op_num_shards_(ssd|hdd) osd_op_num_threads_per_shard_(ssd|hdd) osd_recovery_sleep_(ssd|hdd) k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSD tuning no longer required?
I saw this in the Luminous release notes: "Each OSD now adjusts its default configuration based on whether the backing device is an HDD or SSD. Manual tuning generally not required" Which tuning in particular? The ones in my configuration are osd_op_threads, osd_disk_threads, osd_recovery_max_active, osd_op_thread_suicide_timeout, and osd_crush_chooseleaf_type, among others. Can I rip these out when I upgrade to Luminous? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
