Re: [ceph-users] OSD Full Ratio Luminous - Unset
On Tue, Jul 11, 2017 at 12:46 PM, Ashley Merrick wrote:
> Hello,
>
> Perfect thanks that fixed my issue!
>
> Still seems to be a bug on the ceph pg dump unless it has been moved out of
> the PG and directly into the OSD?
I am looking into this issue and have been trying to bisect it but it has proved
challenging.
Please feel free to open a tracker as we have discussed or I will open one once
I have more information.
Edward, if you feel this could be improved please also open a separate tracker
for the ratio ordering issue outlining the steps involved and noting the issues
encountered.
>
> ,Ashley
>
> -Original Message-
> From: Edward R Huyer [mailto:erh...@rit.edu]
> Sent: Tuesday, 11 July 2017 7:53 AM
> To: Ashley Merrick
> Cc: ceph-users@lists.ceph.com
> Subject: [ceph-users] OSD Full Ratio Luminous - Unset
>
> I just now ran into the same problem you did, though I managed to get it
> straightened out.
>
> It looks to me like the "ceph osd set-{full,nearfull,backfillfull}-ratio"
> commands *do* work, with two caveats.
>
> Caveat 1: "ceph pg dump" doesn't reflect the change for some reason.
> Caveat 2: There doesn't seem to be much of a safety measure to prevent you
> from entering incoherent values. In particular, if backfillfull-ratio is
> smaller than nearfull-ratio, you will continue to have the ERR health
> message. Backfillfull must be greater than (or equal to?) nearfull. So, for
> instance, my cluster at the moment has a full ratio of .9, a backfillfull
> ratio of 0.85, and a nearfull ratio of 0.8.
>
> So there appears to be a bug when upgrading from Jewel where the full ratios
> don't carry over properly, a display bug of some kind with "ceph pg dump",
> and a general issue with lack of information about full ratios being out of
> order. Oh, and the set ratio commands allow incoherent values, though that
> might be intentional.
>
> -
> Edward Huyer
> School of Interactive Games and Media
> Rochester Institute of Technology
> Golisano 70-2373
> 152 Lomb Memorial Drive
> Rochester, NY 14623
> 585-475-6651
> mailto:erh...@rit.edu
>
> Obligatory Legalese:
> The information transmitted, including attachments, is intended only for the
> person(s) or entity to which it is addressed and may contain confidential
> and/or privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon this information by
> persons or entities other than the intended recipient is prohibited. If you
> received this in error, please contact the sender and destroy any copies of
> this information.
>
> So using the commands given I checked all the mon's and a couple of OSD's
> that are not backfilling due to saying they are backfill_full.
>
> However when running the command the value's I have been trying to set via
> the set commands are correctly set and reporting via the admin socket.
>
> However "ceph pg dump | head" still shows 0 for them all and the PG's still
> aren't back filling, so apart from ceph pg dump showing incorrect values the
> admin socket shows correct however cluster is still stuck in non-backfilling
> state / ERR_health due to messages.
>
> ,Ashley
>
>
> From: Brad Hubbard
> Sent: 07 July 2017 10:31:01
> To: Ashley Merrick
> Cc: ceph-users at ceph.com
> Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
>
> On Fri, Jul 7, 2017 at 4:49 PM, Ashley Merrick
> wrote:
>> After looking into this further it seem's none of the :
>>
>>
>> ceph osd set-{full,nearfull,backfillfull}-ratio
>>
>>
>> Commands seem to be taking any effect on the cluster including the
>> backfillfull ratio, this command looks to have been added/changed
>> since Jewel, and a different way of setting the above. However does
>> not seem to be giving the expected results.
>
> Hi Ashley,
>
> Please do open a tracker for this including commands used and resulting
> output so we can investigate this fully.
>
> In the meantime you should be able to view and manipulate individual full
> ratio values via the admin socket.
>
> ceph daemon osd.0 config show|grep ratio ceph daemon mon.a config show|grep
> ratio ceph daemon osd.0 help|grep "config set "
>
>>
>>
>> ,Ashley
>>
>>
>> From: ceph-users on behalf of
>> Ashley Merrick
>> Sent: 06 July 2017 12:44:09
>> To: ceph-users at ceph.com
>> Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
>>
>> Anyone have some feedback on
Re: [ceph-users] OSD Full Ratio Luminous - Unset
Hello,
Perfect thanks that fixed my issue!
Still seems to be a bug on the ceph pg dump unless it has been moved out of the
PG and directly into the OSD?
,Ashley
-Original Message-
From: Edward R Huyer [mailto:erh...@rit.edu]
Sent: Tuesday, 11 July 2017 7:53 AM
To: Ashley Merrick
Cc: ceph-users@lists.ceph.com
Subject: [ceph-users] OSD Full Ratio Luminous - Unset
I just now ran into the same problem you did, though I managed to get it
straightened out.
It looks to me like the "ceph osd set-{full,nearfull,backfillfull}-ratio"
commands *do* work, with two caveats.
Caveat 1: "ceph pg dump" doesn't reflect the change for some reason.
Caveat 2: There doesn't seem to be much of a safety measure to prevent you from
entering incoherent values. In particular, if backfillfull-ratio is smaller
than nearfull-ratio, you will continue to have the ERR health message.
Backfillfull must be greater than (or equal to?) nearfull. So, for instance,
my cluster at the moment has a full ratio of .9, a backfillfull ratio of 0.85,
and a nearfull ratio of 0.8.
So there appears to be a bug when upgrading from Jewel where the full ratios
don't carry over properly, a display bug of some kind with "ceph pg dump", and
a general issue with lack of information about full ratios being out of order.
Oh, and the set ratio commands allow incoherent values, though that might be
intentional.
-
Edward Huyer
School of Interactive Games and Media
Rochester Institute of Technology
Golisano 70-2373
152 Lomb Memorial Drive
Rochester, NY 14623
585-475-6651
mailto:erh...@rit.edu
Obligatory Legalese:
The information transmitted, including attachments, is intended only for the
person(s) or entity to which it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination or other
use of, or taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you received this
in error, please contact the sender and destroy any copies of this information.
So using the commands given I checked all the mon's and a couple of OSD's that
are not backfilling due to saying they are backfill_full.
However when running the command the value's I have been trying to set via the
set commands are correctly set and reporting via the admin socket.
However "ceph pg dump | head" still shows 0 for them all and the PG's still
aren't back filling, so apart from ceph pg dump showing incorrect values the
admin socket shows correct however cluster is still stuck in non-backfilling
state / ERR_health due to messages.
,Ashley
From: Brad Hubbard
Sent: 07 July 2017 10:31:01
To: Ashley Merrick
Cc: ceph-users at ceph.com
Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
On Fri, Jul 7, 2017 at 4:49 PM, Ashley Merrick wrote:
> After looking into this further it seem's none of the :
>
>
> ceph osd set-{full,nearfull,backfillfull}-ratio
>
>
> Commands seem to be taking any effect on the cluster including the
> backfillfull ratio, this command looks to have been added/changed
> since Jewel, and a different way of setting the above. However does
> not seem to be giving the expected results.
Hi Ashley,
Please do open a tracker for this including commands used and resulting output
so we can investigate this fully.
In the meantime you should be able to view and manipulate individual full ratio
values via the admin socket.
ceph daemon osd.0 config show|grep ratio ceph daemon mon.a config show|grep
ratio ceph daemon osd.0 help|grep "config set "
>
>
> ,Ashley
>
> ____________
> From: ceph-users on behalf of
> Ashley Merrick
> Sent: 06 July 2017 12:44:09
> To: ceph-users at ceph.com
> Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
>
> Anyone have some feedback on this? Happy to log a bug ticket if it is
> one, but want to make sure not missing something Luminous change related.
>
> ,Ashley
>
> Sent from my iPhone
>
> On 4 Jul 2017, at 3:30 PM, Ashley Merrick wrote:
>
> Okie noticed their is a new command to set these.
>
>
> Tried these and still showing as 0 and error on full ratio out of
> order "ceph osd set-{full,nearfull,backfillfull}-ratio"
>
>
> ,Ashley
>
>
> From: ceph-users on behalf of
> Ashley Merrick
> Sent: 04 July 2017 05:55:10
> To: ceph-users at ceph.com
> Subject: [ceph-users] OSD Full Ratio Luminous - Unset
>
>
> Hello,
>
>
> On a Luminous upgraded from Jewel I am seeing the following in ceph -s :
> "Full ratio(s) out of order"
>
>
> and
>
>
> ceph pg dump | head
> dumped all
> version 44281
> stamp
Re: [ceph-users] OSD Full Ratio Luminous - Unset
So using the commands given I checked all the mon's and a couple of OSD's that
are not backfilling due to saying they are backfill_full.
However when running the command the value's I have been trying to set via the
set commands are correctly set and reporting via the admin socket.
However "ceph pg dump | head" still shows 0 for them all and the PG's still
aren't back filling, so apart from ceph pg dump showing incorrect values the
admin socket shows correct however cluster is still stuck in non-backfilling
state / ERR_health due to messages.
,Ashley
From: Brad Hubbard
Sent: 07 July 2017 10:31:01
To: Ashley Merrick
Cc: ceph-us...@ceph.com
Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
On Fri, Jul 7, 2017 at 4:49 PM, Ashley Merrick wrote:
> After looking into this further it seem's none of the :
>
>
> ceph osd set-{full,nearfull,backfillfull}-ratio
>
>
> Commands seem to be taking any effect on the cluster including the
> backfillfull ratio, this command looks to have been added/changed since
> Jewel, and a different way of setting the above. However does not seem to be
> giving the expected results.
Hi Ashley,
Please do open a tracker for this including commands used and
resulting output so we can investigate this fully.
In the meantime you should be able to view and manipulate individual
full ratio values via the admin socket.
ceph daemon osd.0 config show|grep ratio
ceph daemon mon.a config show|grep ratio
ceph daemon osd.0 help|grep "config set "
>
>
> ,Ashley
>
>
> From: ceph-users on behalf of Ashley
> Merrick
> Sent: 06 July 2017 12:44:09
> To: ceph-us...@ceph.com
> Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
>
> Anyone have some feedback on this? Happy to log a bug ticket if it is one,
> but want to make sure not missing something Luminous change related.
>
> ,Ashley
>
> Sent from my iPhone
>
> On 4 Jul 2017, at 3:30 PM, Ashley Merrick wrote:
>
> Okie noticed their is a new command to set these.
>
>
> Tried these and still showing as 0 and error on full ratio out of order
> "ceph osd set-{full,nearfull,backfillfull}-ratio"
>
>
> ,Ashley
>
>
> From: ceph-users on behalf of Ashley
> Merrick
> Sent: 04 July 2017 05:55:10
> To: ceph-us...@ceph.com
> Subject: [ceph-users] OSD Full Ratio Luminous - Unset
>
>
> Hello,
>
>
> On a Luminous upgraded from Jewel I am seeing the following in ceph -s :
> "Full ratio(s) out of order"
>
>
> and
>
>
> ceph pg dump | head
> dumped all
> version 44281
> stamp 2017-07-04 05:52:08.337258
> last_osdmap_epoch 0
> last_pg_scan 0
> full_ratio 0
> nearfull_ratio 0
>
> I have tried to inject the values however makes no effect, these where
> previously non 0 values and the issue only showed after running "ceph osd
> require-osd-release luminous"
>
>
> Thanks,
>
> Ashley
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Full Ratio Luminous - Unset
On Fri, Jul 7, 2017 at 4:49 PM, Ashley Merrick wrote:
> After looking into this further it seem's none of the :
>
>
> ceph osd set-{full,nearfull,backfillfull}-ratio
>
>
> Commands seem to be taking any effect on the cluster including the
> backfillfull ratio, this command looks to have been added/changed since
> Jewel, and a different way of setting the above. However does not seem to be
> giving the expected results.
Hi Ashley,
Please do open a tracker for this including commands used and
resulting output so we can investigate this fully.
In the meantime you should be able to view and manipulate individual
full ratio values via the admin socket.
ceph daemon osd.0 config show|grep ratio
ceph daemon mon.a config show|grep ratio
ceph daemon osd.0 help|grep "config set "
>
>
> ,Ashley
>
>
> From: ceph-users on behalf of Ashley
> Merrick
> Sent: 06 July 2017 12:44:09
> To: ceph-us...@ceph.com
> Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
>
> Anyone have some feedback on this? Happy to log a bug ticket if it is one,
> but want to make sure not missing something Luminous change related.
>
> ,Ashley
>
> Sent from my iPhone
>
> On 4 Jul 2017, at 3:30 PM, Ashley Merrick wrote:
>
> Okie noticed their is a new command to set these.
>
>
> Tried these and still showing as 0 and error on full ratio out of order
> "ceph osd set-{full,nearfull,backfillfull}-ratio"
>
>
> ,Ashley
>
>
> From: ceph-users on behalf of Ashley
> Merrick
> Sent: 04 July 2017 05:55:10
> To: ceph-us...@ceph.com
> Subject: [ceph-users] OSD Full Ratio Luminous - Unset
>
>
> Hello,
>
>
> On a Luminous upgraded from Jewel I am seeing the following in ceph -s :
> "Full ratio(s) out of order"
>
>
> and
>
>
> ceph pg dump | head
> dumped all
> version 44281
> stamp 2017-07-04 05:52:08.337258
> last_osdmap_epoch 0
> last_pg_scan 0
> full_ratio 0
> nearfull_ratio 0
>
> I have tried to inject the values however makes no effect, these where
> previously non 0 values and the issue only showed after running "ceph osd
> require-osd-release luminous"
>
>
> Thanks,
>
> Ashley
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Full Ratio Luminous - Unset
After looking into this further it seem's none of the :
ceph osd set-{full,nearfull,backfillfull}-ratio
Commands seem to be taking any effect on the cluster including the backfillfull
ratio, this command looks to have been added/changed since Jewel, and a
different way of setting the above. However does not seem to be giving the
expected results.
,Ashley
From: ceph-users on behalf of Ashley
Merrick
Sent: 06 July 2017 12:44:09
To: ceph-us...@ceph.com
Subject: Re: [ceph-users] OSD Full Ratio Luminous - Unset
Anyone have some feedback on this? Happy to log a bug ticket if it is one, but
want to make sure not missing something Luminous change related.
,Ashley
Sent from my iPhone
On 4 Jul 2017, at 3:30 PM, Ashley Merrick
mailto:ash...@amerrick.co.uk>> wrote:
Okie noticed their is a new command to set these.
Tried these and still showing as 0 and error on full ratio out of order "ceph
osd set-{full,nearfull,backfillfull}-ratio"
,Ashley
From: ceph-users
mailto:ceph-users-boun...@lists.ceph.com>>
on behalf of Ashley Merrick
mailto:ash...@amerrick.co.uk>>
Sent: 04 July 2017 05:55:10
To: ceph-us...@ceph.com<mailto:ceph-us...@ceph.com>
Subject: [ceph-users] OSD Full Ratio Luminous - Unset
Hello,
On a Luminous upgraded from Jewel I am seeing the following in ceph -s : "Full
ratio(s) out of order"
and
ceph pg dump | head
dumped all
version 44281
stamp 2017-07-04 05:52:08.337258
last_osdmap_epoch 0
last_pg_scan 0
full_ratio 0
nearfull_ratio 0
I have tried to inject the values however makes no effect, these where
previously non 0 values and the issue only showed after running "ceph osd
require-osd-release luminous"
Thanks,
Ashley
___
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Full Ratio Luminous - Unset
Anyone have some feedback on this? Happy to log a bug ticket if it is one, but
want to make sure not missing something Luminous change related.
,Ashley
Sent from my iPhone
On 4 Jul 2017, at 3:30 PM, Ashley Merrick
mailto:ash...@amerrick.co.uk>> wrote:
Okie noticed their is a new command to set these.
Tried these and still showing as 0 and error on full ratio out of order "ceph
osd set-{full,nearfull,backfillfull}-ratio"
,Ashley
From: ceph-users
mailto:ceph-users-boun...@lists.ceph.com>>
on behalf of Ashley Merrick
mailto:ash...@amerrick.co.uk>>
Sent: 04 July 2017 05:55:10
To: ceph-us...@ceph.com
Subject: [ceph-users] OSD Full Ratio Luminous - Unset
Hello,
On a Luminous upgraded from Jewel I am seeing the following in ceph -s : "Full
ratio(s) out of order"
and
ceph pg dump | head
dumped all
version 44281
stamp 2017-07-04 05:52:08.337258
last_osdmap_epoch 0
last_pg_scan 0
full_ratio 0
nearfull_ratio 0
I have tried to inject the values however makes no effect, these where
previously non 0 values and the issue only showed after running "ceph osd
require-osd-release luminous"
Thanks,
Ashley
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Full Ratio Luminous - Unset
Okie noticed their is a new command to set these.
Tried these and still showing as 0 and error on full ratio out of order "ceph
osd set-{full,nearfull,backfillfull}-ratio"
,Ashley
From: ceph-users on behalf of Ashley
Merrick
Sent: 04 July 2017 05:55:10
To: ceph-us...@ceph.com
Subject: [ceph-users] OSD Full Ratio Luminous - Unset
Hello,
On a Luminous upgraded from Jewel I am seeing the following in ceph -s : "Full
ratio(s) out of order"
and
ceph pg dump | head
dumped all
version 44281
stamp 2017-07-04 05:52:08.337258
last_osdmap_epoch 0
last_pg_scan 0
full_ratio 0
nearfull_ratio 0
I have tried to inject the values however makes no effect, these where
previously non 0 values and the issue only showed after running "ceph osd
require-osd-release luminous"
Thanks,
Ashley
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
