[ceph-users] Re: BlueFS spillover warning gone after upgrade to Quincy
Hi Benoit and Peter,
looks like your findings are valid and spillover alert is broken for
now. I've just created https://tracker.ceph.com/issues/58440 to track this.
Thanks,
Igor
On 1/13/2023 9:54 AM, Benoît Knecht wrote:
Hi Peter,
On Thursday, January 12th, 2023 at 15:12, Peter van Heusden
wrote:
I have a Ceph installation where some of the OSDs were misconfigured to use
1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS spillover
detected"). I recently upgraded to quincy using cephadm (17.2.5) the
spillover warning vanished. This is
despite bluestore_warn_on_bluefs_spillover still being set to true.
I noticed this on Pacific as well, and I think it's due to this commit:
https://github.com/ceph/ceph/commit/d17cd6604b4031ca997deddc5440248aff451269.
It removes the logic that would normally update the spillover health check, so
it never triggers anymore.
As others mentioned, you can get the relevant metrics from Prometheus and setup
alerts there instead. But it does make me wonder how many people might have
spillover in their clusters and not even realize it, since there's no warning by
default.
Cheers,
--
Igor Fedotov
Ceph Lead Developer
--
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web |
Youtube |
Twitter
Meet us at the SC22 Conference! Learn more
Technology Fast50 Award Winner by Deloitte
!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: BlueFS spillover warning gone after upgrade to Quincy
Hi Peter,
On Thursday, January 12th, 2023 at 15:12, Peter van Heusden
wrote:
> I have a Ceph installation where some of the OSDs were misconfigured to use
> 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS spillover
> detected"). I recently upgraded to quincy using cephadm (17.2.5) the
> spillover warning vanished. This is
> despite bluestore_warn_on_bluefs_spillover still being set to true.
I noticed this on Pacific as well, and I think it's due to this commit:
https://github.com/ceph/ceph/commit/d17cd6604b4031ca997deddc5440248aff451269.
It removes the logic that would normally update the spillover health check, so
it never triggers anymore.
As others mentioned, you can get the relevant metrics from Prometheus and setup
alerts there instead. But it does make me wonder how many people might have
spillover in their clusters and not even realize it, since there's no warning by
default.
Cheers,
--
Ben
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: BlueFS spillover warning gone after upgrade to Quincy
Thanks. The command definitely shows "slow_bytes":
"db_total_bytes": 1073733632,
"db_used_bytes": 240123904,
"slow_total_bytes": 4000681103360,
"slow_used_bytes": 8355381248,
So I am not sure why the warnings are no longer appearing.
Peter
On Thu, 12 Jan 2023 at 17:41, Eugen Block wrote:
> Hi,
>
> I usually look for this:
>
> [ceph: root@storage01 /]# ceph daemon osd.0 perf dump bluefs | grep -E
> "db_|slow_"
> "db_total_bytes": 21470642176,
> "db_used_bytes": 179699712,
> "slow_total_bytes": 0,
> "slow_used_bytes": 0,
>
> If you have spillover I would expect the "slow_bytes" values to be >
> 0. Is it possible that the OSDs were compacted during/after the
> upgrade so the spillover would have been corrected (temporarily)? Do
> you know how much spillover you had before? And how big was the db
> when you had the warnings?
>
> Regards,
> Eugen
>
> Zitat von Peter van Heusden :
>
> > Hello everyone
> >
> > I have a Ceph installation where some of the OSDs were misconfigured to
> use
> > 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS
> *spillover*
> > detected"). I recently upgraded to quincy using cephadm (17.2.5) the
> > spillover warning vanished. This is
> > despite bluestore_warn_on_bluefs_spillover still being set to true.
> >
> > Is there a way to investigate the current state of the DB to see if
> > spillover is, indeed, still happening?
> >
> > Thank you,
> > Peter
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: BlueFS spillover warning gone after upgrade to Quincy
If you have prometheus enabled, the metrics should be in there I think?
Thanks,
Kevin
From: Peter van Heusden
Sent: Thursday, January 12, 2023 6:12 AM
To: ceph-users@ceph.io
Subject: [ceph-users] BlueFS spillover warning gone after upgrade to Quincy
Check twice before you click! This email originated from outside PNNL.
Hello everyone
I have a Ceph installation where some of the OSDs were misconfigured to use
1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover*
detected"). I recently upgraded to quincy using cephadm (17.2.5) the
spillover warning vanished. This is
despite bluestore_warn_on_bluefs_spillover still being set to true.
Is there a way to investigate the current state of the DB to see if
spillover is, indeed, still happening?
Thank you,
Peter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: BlueFS spillover warning gone after upgrade to Quincy
Hi,
I usually look for this:
[ceph: root@storage01 /]# ceph daemon osd.0 perf dump bluefs | grep -E
"db_|slow_"
"db_total_bytes": 21470642176,
"db_used_bytes": 179699712,
"slow_total_bytes": 0,
"slow_used_bytes": 0,
If you have spillover I would expect the "slow_bytes" values to be >
0. Is it possible that the OSDs were compacted during/after the
upgrade so the spillover would have been corrected (temporarily)? Do
you know how much spillover you had before? And how big was the db
when you had the warnings?
Regards,
Eugen
Zitat von Peter van Heusden :
Hello everyone
I have a Ceph installation where some of the OSDs were misconfigured to use
1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover*
detected"). I recently upgraded to quincy using cephadm (17.2.5) the
spillover warning vanished. This is
despite bluestore_warn_on_bluefs_spillover still being set to true.
Is there a way to investigate the current state of the DB to see if
spillover is, indeed, still happening?
Thank you,
Peter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
