Re: [ceph-users] Luminous scrub catch-22

[Roger Brown]

Thank you. That was it. They were installed, but hadn't been restarted
since upgrading. Solved with: sudo systemctl restart ceph-mgr.target


On Thu, Aug 3, 2017 at 1:56 PM Gregory Farnum  wrote:

> I believe that command should still work, but it looks like it requires a
> working manager daemon. Did you set one up yet?
>
> On Thu, Aug 3, 2017 at 7:31 AM Roger Brown  wrote:
>
>> I'm running Luminous 12.1.2 and I seem to be in a catch-22. I've got pgs
>> that report they need to be scrubbed, however the command to scrub them
>> seems to have gone away. The flapping OSD is an issue for another thread.
>> Please advise.
>>
>> Example:
>>
>> roger@desktop:~$ ceph --version
>> ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous
>> (rc)
>>
>> roger@desktop:~$ ceph health detail
>> HEALTH_WARN 1 osds down; 1 host (1 osds) down; Degraded data redundancy:
>> 54278/162834 objects degraded (33.333%), 372 pgs unclean, 372 pgs degraded,
>> 372 pgs undersized; 119 pgs not deep-scrubbed for 86400; 128 pgs not
>> scrubbed for 86400; 10 slow requests are blocked > 32 sec
>> OSD_DOWN 1 osds down
>> osd.0 (root=default,host=osd3) is down
>> OSD_HOST_DOWN 1 host (1 osds) down
>> host osd3 (root=default) (1 osds) is down
>> PG_DEGRADED Degraded data redundancy: 54278/162834 objects degraded
>> (33.333%), 372 pgs unclean, 372 pgs degraded, 372 pgs undersized
>> pg 103.cd is stuck unclean for 1351336.704366, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,4,3]
>> pg 103.ce is stuck undersized for 155.107291, current state
>> active+undersized+degraded, last acting [4,3,2147483647
>> <(214)%20748-3647>]
>> pg 103.cf is stuck undersized for 155.107098, current state
>> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
>> ,3]
>> pg 103.d0 is stuck undersized for 155.106886, current state
>> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
>> ,3]
>> pg 103.d1 is stuck undersized for 155.106692, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,4,3]
>> pg 103.d2 is stuck undersized for 155.106511, current state
>> active+undersized+degraded, last acting [4,3,2147483647
>> <(214)%20748-3647>]
>> pg 103.d3 is stuck undersized for 155.106333, current state
>> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
>> ,3]
>> pg 103.d4 is stuck undersized for 155.107740, current state
>> active+undersized+degraded, last acting [4,3,2147483647
>> <(214)%20748-3647>]
>> pg 103.d5 is stuck undersized for 155.122380, current state
>> active+undersized+degraded, last acting [3,2147483647 <(214)%20748-3647>
>> ,4]
>> pg 103.d6 is stuck undersized for 155.122118, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,3,4]
>> pg 103.d7 is stuck undersized for 155.122251, current state
>> active+undersized+degraded, last acting [3,4,2147483647
>> <(214)%20748-3647>]
>> pg 103.d8 is stuck undersized for 155.121991, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,3,4]
>> pg 103.d9 is stuck undersized for 155.121842, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,3,4]
>> pg 103.da is stuck undersized for 155.126263, current state
>> active+undersized+degraded, last acting [3,2147483647 <(214)%20748-3647>
>> ,4]
>> pg 103.db is stuck undersized for 155.107793, current state
>> active+undersized+degraded, last acting [3,4,2147483647
>> <(214)%20748-3647>]
>> pg 103.dc is stuck undersized for 155.105822, current state
>> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
>> ,3]
>> pg 103.dd is stuck undersized for 155.105619, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,4,3]
>> pg 103.de is stuck undersized for 155.095456, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,4,3]
>> pg 103.df is stuck undersized for 155.121684, current state
>> active+undersized+degraded, last acting [3,4,2147483647
>> <(214)%20748-3647>]
>> pg 103.e0 is stuck undersized for 155.115082, current state
>> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
>> ,3,4]
>> pg 103.e1 is stuck undersized for 155.100069, current state
>> active+undersized+degraded, last acting [4,3,2147483647
>> <(214)%20748-3647>]
>> pg 103.e2 is stuck undersized for 155.095089, current state
>> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
>> ,3]
>> pg 103.e3 is stuck undersized for 155.100056, current state
>> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
>> ,3]
>> pg 103.e4 is stuck undersized for 155.100750, current state
>> active+undersized+degraded, last acting [4,2147483647 

Re: [ceph-users] Luminous scrub catch-22

[Gregory Farnum]

I believe that command should still work, but it looks like it requires a
working manager daemon. Did you set one up yet?

On Thu, Aug 3, 2017 at 7:31 AM Roger Brown  wrote:

> I'm running Luminous 12.1.2 and I seem to be in a catch-22. I've got pgs
> that report they need to be scrubbed, however the command to scrub them
> seems to have gone away. The flapping OSD is an issue for another thread.
> Please advise.
>
> Example:
>
> roger@desktop:~$ ceph --version
> ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous
> (rc)
>
> roger@desktop:~$ ceph health detail
> HEALTH_WARN 1 osds down; 1 host (1 osds) down; Degraded data redundancy:
> 54278/162834 objects degraded (33.333%), 372 pgs unclean, 372 pgs degraded,
> 372 pgs undersized; 119 pgs not deep-scrubbed for 86400; 128 pgs not
> scrubbed for 86400; 10 slow requests are blocked > 32 sec
> OSD_DOWN 1 osds down
> osd.0 (root=default,host=osd3) is down
> OSD_HOST_DOWN 1 host (1 osds) down
> host osd3 (root=default) (1 osds) is down
> PG_DEGRADED Degraded data redundancy: 54278/162834 objects degraded
> (33.333%), 372 pgs unclean, 372 pgs degraded, 372 pgs undersized
> pg 103.cd is stuck unclean for 1351336.704366, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,4,3]
> pg 103.ce is stuck undersized for 155.107291, current state
> active+undersized+degraded, last acting [4,3,2147483647 <(214)%20748-3647>
> ]
> pg 103.cf is stuck undersized for 155.107098, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.d0 is stuck undersized for 155.106886, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.d1 is stuck undersized for 155.106692, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,4,3]
> pg 103.d2 is stuck undersized for 155.106511, current state
> active+undersized+degraded, last acting [4,3,2147483647 <(214)%20748-3647>
> ]
> pg 103.d3 is stuck undersized for 155.106333, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.d4 is stuck undersized for 155.107740, current state
> active+undersized+degraded, last acting [4,3,2147483647 <(214)%20748-3647>
> ]
> pg 103.d5 is stuck undersized for 155.122380, current state
> active+undersized+degraded, last acting [3,2147483647 <(214)%20748-3647>
> ,4]
> pg 103.d6 is stuck undersized for 155.122118, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,3,4]
> pg 103.d7 is stuck undersized for 155.122251, current state
> active+undersized+degraded, last acting [3,4,2147483647 <(214)%20748-3647>
> ]
> pg 103.d8 is stuck undersized for 155.121991, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,3,4]
> pg 103.d9 is stuck undersized for 155.121842, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,3,4]
> pg 103.da is stuck undersized for 155.126263, current state
> active+undersized+degraded, last acting [3,2147483647 <(214)%20748-3647>
> ,4]
> pg 103.db is stuck undersized for 155.107793, current state
> active+undersized+degraded, last acting [3,4,2147483647 <(214)%20748-3647>
> ]
> pg 103.dc is stuck undersized for 155.105822, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.dd is stuck undersized for 155.105619, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,4,3]
> pg 103.de is stuck undersized for 155.095456, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,4,3]
> pg 103.df is stuck undersized for 155.121684, current state
> active+undersized+degraded, last acting [3,4,2147483647 <(214)%20748-3647>
> ]
> pg 103.e0 is stuck undersized for 155.115082, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,3,4]
> pg 103.e1 is stuck undersized for 155.100069, current state
> active+undersized+degraded, last acting [4,3,2147483647 <(214)%20748-3647>
> ]
> pg 103.e2 is stuck undersized for 155.095089, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.e3 is stuck undersized for 155.100056, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.e4 is stuck undersized for 155.100750, current state
> active+undersized+degraded, last acting [4,2147483647 <(214)%20748-3647>
> ,3]
> pg 103.e5 is stuck undersized for 155.117161, current state
> active+undersized+degraded, last acting [2147483647 <(214)%20748-3647>
> ,3,4]
> pg 103.e6 is stuck undersized for 155.101932, current state
> active+undersized+degraded, last acting [4,3,2147483647 

[ceph-users] Luminous scrub catch-22

[Roger Brown]

I'm running Luminous 12.1.2 and I seem to be in a catch-22. I've got pgs
that report they need to be scrubbed, however the command to scrub them
seems to have gone away. The flapping OSD is an issue for another thread.
Please advise.

Example:

roger@desktop:~$ ceph --version
ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)

roger@desktop:~$ ceph health detail
HEALTH_WARN 1 osds down; 1 host (1 osds) down; Degraded data redundancy:
54278/162834 objects degraded (33.333%), 372 pgs unclean, 372 pgs degraded,
372 pgs undersized; 119 pgs not deep-scrubbed for 86400; 128 pgs not
scrubbed for 86400; 10 slow requests are blocked > 32 sec
OSD_DOWN 1 osds down
osd.0 (root=default,host=osd3) is down
OSD_HOST_DOWN 1 host (1 osds) down
host osd3 (root=default) (1 osds) is down
PG_DEGRADED Degraded data redundancy: 54278/162834 objects degraded
(33.333%), 372 pgs unclean, 372 pgs degraded, 372 pgs undersized
pg 103.cd is stuck unclean for 1351336.704366, current state
active+undersized+degraded, last acting [2147483647,4,3]
pg 103.ce is stuck undersized for 155.107291, current state
active+undersized+degraded, last acting [4,3,2147483647]
pg 103.cf is stuck undersized for 155.107098, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.d0 is stuck undersized for 155.106886, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.d1 is stuck undersized for 155.106692, current state
active+undersized+degraded, last acting [2147483647,4,3]
pg 103.d2 is stuck undersized for 155.106511, current state
active+undersized+degraded, last acting [4,3,2147483647]
pg 103.d3 is stuck undersized for 155.106333, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.d4 is stuck undersized for 155.107740, current state
active+undersized+degraded, last acting [4,3,2147483647]
pg 103.d5 is stuck undersized for 155.122380, current state
active+undersized+degraded, last acting [3,2147483647,4]
pg 103.d6 is stuck undersized for 155.122118, current state
active+undersized+degraded, last acting [2147483647,3,4]
pg 103.d7 is stuck undersized for 155.122251, current state
active+undersized+degraded, last acting [3,4,2147483647]
pg 103.d8 is stuck undersized for 155.121991, current state
active+undersized+degraded, last acting [2147483647,3,4]
pg 103.d9 is stuck undersized for 155.121842, current state
active+undersized+degraded, last acting [2147483647,3,4]
pg 103.da is stuck undersized for 155.126263, current state
active+undersized+degraded, last acting [3,2147483647,4]
pg 103.db is stuck undersized for 155.107793, current state
active+undersized+degraded, last acting [3,4,2147483647]
pg 103.dc is stuck undersized for 155.105822, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.dd is stuck undersized for 155.105619, current state
active+undersized+degraded, last acting [2147483647,4,3]
pg 103.de is stuck undersized for 155.095456, current state
active+undersized+degraded, last acting [2147483647,4,3]
pg 103.df is stuck undersized for 155.121684, current state
active+undersized+degraded, last acting [3,4,2147483647]
pg 103.e0 is stuck undersized for 155.115082, current state
active+undersized+degraded, last acting [2147483647,3,4]
pg 103.e1 is stuck undersized for 155.100069, current state
active+undersized+degraded, last acting [4,3,2147483647]
pg 103.e2 is stuck undersized for 155.095089, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.e3 is stuck undersized for 155.100056, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.e4 is stuck undersized for 155.100750, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.e5 is stuck undersized for 155.117161, current state
active+undersized+degraded, last acting [2147483647,3,4]
pg 103.e6 is stuck undersized for 155.101932, current state
active+undersized+degraded, last acting [4,3,2147483647]
pg 103.e7 is stuck undersized for 155.115349, current state
active+undersized+degraded, last acting [3,4,2147483647]
pg 103.e8 is stuck undersized for 155.099699, current state
active+undersized+degraded, last acting [2147483647,4,3]
pg 103.e9 is stuck undersized for 155.114433, current state
active+undersized+degraded, last acting [3,2147483647,4]
pg 103.ea is stuck undersized for 155.099736, current state
active+undersized+degraded, last acting [4,2147483647,3]
pg 103.eb is stuck undersized for 155.106121, current state
active+undersized+degraded, last acting [4,3,2147483647]
pg 103.ec is stuck undersized for 155.091681, current state
active+undersized+degraded, last acting [2147483647,4,3]
pg 103.ed is stuck undersized for 155.115500, current state
active+undersized+degraded, last acting [2147483647,3,4]
pg 103.ee is stuck undersized for 155.114273, current