Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

Hey Tom,

Ok, yeah, I've used those steps before myself for other operations like cluster 
software updates.  I'll try them.

As for the query, not sure how I missed that, and think I've used it before!  
Unfortunately it just hangs similar to the other daemon operation:
root@rook-tools:/# ceph pg 19.1fdf query

Thanks,
-Bryan


From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 5:36 PM
To: Bryan Banister ; ceph-users@lists.ceph.com
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email

Hi Bryan,

Try "ceph pg xx.xx query", with xx.xx of course being the PG number. With some 
luck this will give you the state of that individual pg and which OSDs or 
issues may be blocking the peering from completing, which can be used as a clue 
perhaps as to the cause. If you can find a point of two OSDs unable to peer, 
you then at least have a pathway to begin testing connectivity again.

For pausing your cluster, my method (never tested in an environment with more 
than a few osds or in production) is:
ceph osd set noout
ceph osd set nobackfill
ceph osd set norecover
ceph osd set nodown
ceph osd set norebalance
ceph osd pause

To return, just reverse the order with "ceph osd pause" becoming "ceph osd 
unpause" I believe. All the above flags will stop most activity and just get 
things peered and not much else, once they are successfully peered, you can 
slowly begin to unset the above (and I do recommend going slowly..). You don't 
have any significant misplaced/degraded objects so you won't likely see much 
recovery activity, but as scrubbing kicks in there might be significant amounts 
of inconsistent PGs and backfill/recovery going on. It might be best to limit 
the impact of these from going 0 to 100 with these parameters (1 backfill at a 
time, wait 0.1 seconds between recovery op per OSD).

ceph tell osd.* injectargs '--osd-max-backfills 1'
ceph tell osd.* injectargs '--osd-recovery-sleep 0.1'

Tom



From: Bryan Banister 
mailto:bbanis...@jumptrading.com>>
Sent: 17 July 2018 23:22
To: Tom W mailto:to...@ukfast.co.uk>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Thanks Tom,

Yes, we can try pausing I/O to give the cluster time to recover.  I assume that 
you're talking about using `ceph osd set pause` for this?

We did finally get some health output, which seems to indicate everything is 
basically stuck:
2018-07-17 21:00:00.000107 mon.rook-ceph-mon7 [WRN] overall HEALTH_WARN 
nodown,noout flag(s) set; 1/8884349 objects misplaced (0.000%); Reduced data 
availability: 10907 pgs inactive, 6354 pgs down, 4553 pgs peering; Degraded 
data redundancy: 10/8884349 objects degraded (0.000%), 6 pgs degraded, 80 pgs 
undersized; 135 slow requests are blocked > 32 sec
2018-07-17 22:00:00.000124 mon.rook-ceph-mon7 [WRN] overall HEALTH_WARN 
nodown,noout flag(s) set; 1/8884349 objects misplaced (0.000%); Reduced data 
availability: 10907 pgs inactive, 6354 pgs down, 4553 pgs peering; Degraded 
data redundancy: 10/8884349 objects degraded (0.000%), 6 pgs degraded, 80 pgs 
undersized; 135 slow requests are blocked > 32 sec

I can't seem to find a command to run a query on a specific PG, though I'm 
really new to ceph so sorry if that's an obvious thing.  What would I run to 
query the status and condition of a PG?

I'll talk with our kubernetes team to see if they can also help rule out any 
networking related issues.

Cheers,
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 5:06 PM
To: Bryan Banister 
mailto:bbanis...@jumptrading.com>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email

Hi Bryan,

That's unusual, and not something I can really begin to unravel. As some other 
pointers, perhaps run a PG query on some of the inactive and peering PGs for 
any potentially useful output?

I suspect from what you've put that most PGs are simply in a down and peering 
state, and it can't peer as they are down still. The nodown flag doesn't seem 
to have fixed that, but then again it can't peer if they actually are down 
which nodown will mask.

Is pausing all cluster IO an option for you? My thinking here is to pause all 
IO, completely restart and verify all OSDs are back up and operational? If they 
fail to come up during paused IO, it will rule out any spiking load, but this 
seems to be more of a network issue, as even peering would normally generate 
some volume of traffic as it cycles to reattempt.

I'm not familiar at all with Rook or Kubernetes at this stage so I

Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

Thanks Tom,

Yes, we can try pausing I/O to give the cluster time to recover.  I assume that 
you're talking about using `ceph osd set pause` for this?

We did finally get some health output, which seems to indicate everything is 
basically stuck:
2018-07-17 21:00:00.000107 mon.rook-ceph-mon7 [WRN] overall HEALTH_WARN 
nodown,noout flag(s) set; 1/8884349 objects misplaced (0.000%); Reduced data 
availability: 10907 pgs inactive, 6354 pgs down, 4553 pgs peering; Degraded 
data redundancy: 10/8884349 objects degraded (0.000%), 6 pgs degraded, 80 pgs 
undersized; 135 slow requests are blocked > 32 sec
2018-07-17 22:00:00.000124 mon.rook-ceph-mon7 [WRN] overall HEALTH_WARN 
nodown,noout flag(s) set; 1/8884349 objects misplaced (0.000%); Reduced data 
availability: 10907 pgs inactive, 6354 pgs down, 4553 pgs peering; Degraded 
data redundancy: 10/8884349 objects degraded (0.000%), 6 pgs degraded, 80 pgs 
undersized; 135 slow requests are blocked > 32 sec

I can't seem to find a command to run a query on a specific PG, though I'm 
really new to ceph so sorry if that's an obvious thing.  What would I run to 
query the status and condition of a PG?

I'll talk with our kubernetes team to see if they can also help rule out any 
networking related issues.

Cheers,
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 5:06 PM
To: Bryan Banister ; ceph-users@lists.ceph.com
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email

Hi Bryan,

That's unusual, and not something I can really begin to unravel. As some other 
pointers, perhaps run a PG query on some of the inactive and peering PGs for 
any potentially useful output?

I suspect from what you've put that most PGs are simply in a down and peering 
state, and it can't peer as they are down still. The nodown flag doesn't seem 
to have fixed that, but then again it can't peer if they actually are down 
which nodown will mask.

Is pausing all cluster IO an option for you? My thinking here is to pause all 
IO, completely restart and verify all OSDs are back up and operational? If they 
fail to come up during paused IO, it will rule out any spiking load, but this 
seems to be more of a network issue, as even peering would normally generate 
some volume of traffic as it cycles to reattempt.

I'm not familiar at all with Rook or Kubernetes at this stage so I also have 
concern over how the networking stack there would work. MTU has been a problem 
in the past but this would only affect performance and not operation in my 
mind. Also perhaps being able to reach other nodes on the right interfaces, so 
can you definitely traverse the public and cluster networks successfully?

Tom



From: Bryan Banister 
mailto:bbanis...@jumptrading.com>>
Sent: 17 July 2018 22:36
To: Tom W mailto:to...@ukfast.co.uk>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Hi Tom,

I tried to check out the ops in flight as you suggested but this seems to just 
hang:
root@rook-ceph-osd-carg-kubelet-osd02-m9rhx:/# ceph --admin-daemon 
/var/lib/rook/osd238/rook-osd.238.asok daemon osd.238 dump_ops_in_flight

Nothing returns and don't get a prompt back.

The cluster is somewhat new, but has been running without any major issues for 
more than a week or so.  We're not even sure how this all started.

I'm happy to provide more details of our deployment if you or others need 
anything.

We haven't changed anything today/recently.  I think you're correct that 
unsetting 'nodown' will just return things to the previous state.

Thanks!
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 4:19 PM
To: Bryan Banister 
mailto:bbanis...@jumptrading.com>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email

Hi Bryan,

OSDs may not truly be up, this flag merely prevents them being marked as down 
even if they are unresponsive. It may be worth unsetting nodown as soon as you 
are confident, but unsetting it before anything changes will just return to the 
previous state. Perhaps not harmful, but I have no oversight on your deployment 
nor am I an expert in any regards.

Find an OSD which is up and having issues peering, and perhaps try something 
like this

ceph daemon osd.x dump_ops_in_flight

Replacing x with the OSD number, I am curious to see what may be holding it up. 
I assume you have already done the usual tests to ensure it is traversing the 
right interface, correct VLANs, reachable via ICMP, perhaps even run an iperf 
and tpcdump to be certain the flow is as expected.

Tom

Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

Hi Tom,

I tried to check out the ops in flight as you suggested but this seems to just 
hang:
root@rook-ceph-osd-carg-kubelet-osd02-m9rhx:/# ceph --admin-daemon 
/var/lib/rook/osd238/rook-osd.238.asok daemon osd.238 dump_ops_in_flight

Nothing returns and don't get a prompt back.

The cluster is somewhat new, but has been running without any major issues for 
more than a week or so.  We're not even sure how this all started.

I'm happy to provide more details of our deployment if you or others need 
anything.

We haven't changed anything today/recently.  I think you're correct that 
unsetting 'nodown' will just return things to the previous state.

Thanks!
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 4:19 PM
To: Bryan Banister ; ceph-users@lists.ceph.com
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email

Hi Bryan,

OSDs may not truly be up, this flag merely prevents them being marked as down 
even if they are unresponsive. It may be worth unsetting nodown as soon as you 
are confident, but unsetting it before anything changes will just return to the 
previous state. Perhaps not harmful, but I have no oversight on your deployment 
nor am I an expert in any regards.

Find an OSD which is up and having issues peering, and perhaps try something 
like this

ceph daemon osd.x dump_ops_in_flight

Replacing x with the OSD number, I am curious to see what may be holding it up. 
I assume you have already done the usual tests to ensure it is traversing the 
right interface, correct VLANs, reachable via ICMP, perhaps even run an iperf 
and tpcdump to be certain the flow is as expected.

Tom

From: Bryan Banister 
mailto:bbanis...@jumptrading.com>>
Sent: 17 July 2018 22:03
To: Tom W mailto:to...@ukfast.co.uk>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Hi Tom,

Decided to try your suggestion of the 'nodown' setting and this indeed has 
gotten all of the OSDs up and they haven't failed out like before.  However the 
PGs are in bad states and Ceph doesn't seem interested in starting recovery 
over the last 30 minues since the latest health message was reported:

2018-07-17 20:29:00.638398 mon.rook-ceph-mon7 [WRN] Health check update: 
1/8884343 objects misplaced (0.000%) (OBJECT_MISPLACED)
2018-07-17 20:29:00.864863 mon.rook-ceph-mon7 [INF] osd.221 
7.129.220.49:6957/30346 boot
2018-07-17 20:29:01.907855 mon.rook-ceph-mon7 [INF] Health check cleared: 
OSD_DOWN (was: 1 osds down)
2018-07-17 20:29:02.598518 mon.rook-ceph-mon7 [INF] osd.238 
7.129.220.49:6923/30330 boot
2018-07-17 20:29:02.988546 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10895 pgs inactive, 6514 pgs down, 4391 pgs peering, 
2 pgs stale (PG_AVAILABILITY)
2018-07-17 20:29:04.380454 mon.rook-ceph-mon7 [WRN] Health check update: 
Degraded data redundancy: 10/8884349 objects degraded (0.000%), 6 pgs degraded, 
80 pgs undersized (PG_DEGRADED)
2018-07-17 20:29:08.319073 mon.rook-ceph-mon7 [WRN] Health check update: 
1/8884349 objects misplaced (0.000%) (OBJECT_MISPLACED)
2018-07-17 20:29:08.319103 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10893 pgs inactive, 6391 pgs down, 4515 pgs peering, 
1 pg stale (PG_AVAILABILITY)
2018-07-17 20:29:13.319406 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10893 pgs inactive, 6354 pgs down, 4552 pgs peering 
(PG_AVAILABILITY)
2018-07-17 20:29:14.044696 mon.rook-ceph-mon7 [WRN] Health check update: 123 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 20:29:20.277493 mon.rook-ceph-mon7 [WRN] Health check update: 129 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 20:29:27.344834 mon.rook-ceph-mon7 [WRN] Health check update: 135 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 20:29:54.516115 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10899 pgs inactive, 6354 pgs down, 4552 pgs peering 
(PG_AVAILABILITY)
2018-07-17 20:30:03.322101 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10907 pgs inactive, 6354 pgs down, 4553 pgs peering 
(PG_AVAILABILITY)

Nothing since then, which was 30 min ago.  Hosts are basically idle.

I'm thinking of unsetting the 'nodown" now to see what it does, but is there 
any other recommendations here before I do that?

Thanks again!
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 1:58 PM
To: Bryan Banister 
mailto:bbanis...@jumptrading.com>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: Re: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External 

Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

Hi Tom,

Decided to try your suggestion of the 'nodown' setting and this indeed has 
gotten all of the OSDs up and they haven't failed out like before.  However the 
PGs are in bad states and Ceph doesn't seem interested in starting recovery 
over the last 30 minues since the latest health message was reported:

2018-07-17 20:29:00.638398 mon.rook-ceph-mon7 [WRN] Health check update: 
1/8884343 objects misplaced (0.000%) (OBJECT_MISPLACED)
2018-07-17 20:29:00.864863 mon.rook-ceph-mon7 [INF] osd.221 
7.129.220.49:6957/30346 boot
2018-07-17 20:29:01.907855 mon.rook-ceph-mon7 [INF] Health check cleared: 
OSD_DOWN (was: 1 osds down)
2018-07-17 20:29:02.598518 mon.rook-ceph-mon7 [INF] osd.238 
7.129.220.49:6923/30330 boot
2018-07-17 20:29:02.988546 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10895 pgs inactive, 6514 pgs down, 4391 pgs peering, 
2 pgs stale (PG_AVAILABILITY)
2018-07-17 20:29:04.380454 mon.rook-ceph-mon7 [WRN] Health check update: 
Degraded data redundancy: 10/8884349 objects degraded (0.000%), 6 pgs degraded, 
80 pgs undersized (PG_DEGRADED)
2018-07-17 20:29:08.319073 mon.rook-ceph-mon7 [WRN] Health check update: 
1/8884349 objects misplaced (0.000%) (OBJECT_MISPLACED)
2018-07-17 20:29:08.319103 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10893 pgs inactive, 6391 pgs down, 4515 pgs peering, 
1 pg stale (PG_AVAILABILITY)
2018-07-17 20:29:13.319406 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10893 pgs inactive, 6354 pgs down, 4552 pgs peering 
(PG_AVAILABILITY)
2018-07-17 20:29:14.044696 mon.rook-ceph-mon7 [WRN] Health check update: 123 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 20:29:20.277493 mon.rook-ceph-mon7 [WRN] Health check update: 129 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 20:29:27.344834 mon.rook-ceph-mon7 [WRN] Health check update: 135 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 20:29:54.516115 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10899 pgs inactive, 6354 pgs down, 4552 pgs peering 
(PG_AVAILABILITY)
2018-07-17 20:30:03.322101 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10907 pgs inactive, 6354 pgs down, 4553 pgs peering 
(PG_AVAILABILITY)

Nothing since then, which was 30 min ago.  Hosts are basically idle.

I'm thinking of unsetting the 'nodown" now to see what it does, but is there 
any other recommendations here before I do that?

Thanks again!
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 1:58 PM
To: Bryan Banister ; ceph-users@lists.ceph.com
Subject: Re: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email

Prior to the OSD being marked as down by the cluster, do you note the PGs 
become inactive on it? Using a flag such as nodown may prevent OSDs flapping if 
it helps reduce the IO load to see if things stabilise out, but be wary of this 
flag as I believe PGs using the OSD as the primary will not failover to another 
OSD while nodown is set.

My thoughts here, albeit I am shooting in the dark a little with this theory, 
is perhaps individual OSDs being overloaded and not returning a heartbeat as a 
result of the load. When OSDs are marked as down and new maps are distributed 
this would add further load so while it keeps recalculating it may be a vicious 
cycle which may be alleviated if it could stabilise.

With networks mainly idle, do you see any spikes at all? Perhaps an OSD coming 
online, OSD attempts backfill/recovery and QoS dropping the heartbeat packets 
if it overloads the link?

Just spitballing some ideas here until somebody more qualified may have an idea.

____
From: Bryan Banister 
mailto:bbanis...@jumptrading.com>>
Sent: 17 July 2018 19:18:15
To: Bryan Banister; Tom W; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: RE: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

I didn't find anything obvious in the release notes about this issue we see to 
have, but I don't understand it really.

We have seen logs indicating some kind of heartbeat issue with OSDs, but we 
don't believe there is any issues with the networking between the nodes, which 
are mostly idle as well:

2018-07-17 17:41:32.903871 I | osd12: 2018-07-17 17:41:32.903793 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6866 osd.219 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:41:12.903604)
2018-07-17 17:41:32.903875 I | osd12: 2018-07-17 17:41:32.903795 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6922 osd.220 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:4

Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

I didn't find anything obvious in the release notes about this issue we see to 
have, but I don't understand it really.

We have seen logs indicating some kind of heartbeat issue with OSDs, but we 
don't believe there is any issues with the networking between the nodes, which 
are mostly idle as well:

2018-07-17 17:41:32.903871 I | osd12: 2018-07-17 17:41:32.903793 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6866 osd.219 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:41:12.903604)
2018-07-17 17:41:32.903875 I | osd12: 2018-07-17 17:41:32.903795 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6922 osd.220 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:41:12.903604)
2018-07-17 17:41:32.903878 I | osd12: 2018-07-17 17:41:32.903798 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6901 osd.221 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:41:12.903604)
2018-07-17 17:41:32.903880 I | osd12: 2018-07-17 17:41:32.903800 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6963 osd.222 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:41:12.903604)
2018-07-17 17:41:32.903884 I | osd12: 2018-07-17 17:41:32.903803 7fffef198700 
-1 osd.12 4296 heartbeat_check: no reply from 7.129.220.44:6907 osd.224 ever on 
either front or back, first ping sent 2018-07-17 17:41:09.893761 (cutoff 
2018-07-17 17:41:12.903604)

Is there a way to resolve this issue, which seems to be the root cause of the 
OSDs being marked as failed.

Thanks in advance for any help,
-Bryan

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Tuesday, July 17, 2018 12:08 PM
To: Tom W ; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs 
failed, then marked down, then booted, then failed again

Note: External Email

Hi Tom,

We're apparently running ceph version 12.2.5 on a Rook based cluster.  We have 
EC pools on large 8TB HDDs and metadata on bluestore OSDs on NVMe drives.

I'll look at the release notes.

Thanks!
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 12:05 PM
To: Bryan Banister 
mailto:bbanis...@jumptrading.com>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: Re: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email


Hi Bryan,



What version of Ceph are you currently running on, and do you run any erasure 
coded pools or bluestore OSDs? Might be worth having a quick glance over the 
recent changelogs:



http://docs.ceph.com/docs/master/releases/luminous/



Tom


From: ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
on behalf of Bryan Banister 
mailto:bbanis...@jumptrading.com>>
Sent: 17 July 2018 18:00:05
To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs 
failed, then marked down, then booted, then failed again

Hi all,

We're still very new to managing Ceph and seem to have cluster that is in an 
endless loop of failing OSDs, then marking them down, then booting them again:

Here are some example logs:
2018-07-17 16:48:28.976673 mon.rook-ceph-mon7 [INF] osd.83 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491973 >= grace 20.010293)
2018-07-17 16:48:28.976730 mon.rook-ceph-mon7 [INF] osd.84 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491916 >= grace 20.010293)
2018-07-17 16:48:28.976785 mon.rook-ceph-mon7 [INF] osd.85 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491870 >= grace 20.011151)
2018-07-17 16:48:28.976843 mon.rook-ceph-mon7 [INF] osd.86 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491828 >= grace 20.010293)
2018-07-17 16:48:28.976890 mon.rook-ceph-mon7 [INF] Marking osd.1 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976913 mon.rook-ceph-mon7 [INF] Marking osd.2 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976933 mon.rook-ceph-mon7 [INF] Marking osd.3 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976954 mon.rook-ceph-mon7 [INF] Marking osd.4 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976979 mon.rook-ceph-mon7 [INF] Marking osd.9 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.977000 mon.rook-ceph-mon7 [INF] Marking osd.10 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977020 mon.rook-ceph-mon7 [INF] Marking osd.11 out (has 
been down f

Re: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

Hi Tom,

We're apparently running ceph version 12.2.5 on a Rook based cluster.  We have 
EC pools on large 8TB HDDs and metadata on bluestore OSDs on NVMe drives.

I'll look at the release notes.

Thanks!
-Bryan

From: Tom W [mailto:to...@ukfast.co.uk]
Sent: Tuesday, July 17, 2018 12:05 PM
To: Bryan Banister ; ceph-users@lists.ceph.com
Subject: Re: Cluster in bad shape, seemingly endless cycle of OSDs failed, then 
marked down, then booted, then failed again

Note: External Email


Hi Bryan,



What version of Ceph are you currently running on, and do you run any erasure 
coded pools or bluestore OSDs? Might be worth having a quick glance over the 
recent changelogs:



http://docs.ceph.com/docs/master/releases/luminous/



Tom


From: ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
on behalf of Bryan Banister 
mailto:bbanis...@jumptrading.com>>
Sent: 17 July 2018 18:00:05
To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: [ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs 
failed, then marked down, then booted, then failed again

Hi all,

We're still very new to managing Ceph and seem to have cluster that is in an 
endless loop of failing OSDs, then marking them down, then booting them again:

Here are some example logs:
2018-07-17 16:48:28.976673 mon.rook-ceph-mon7 [INF] osd.83 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491973 >= grace 20.010293)
2018-07-17 16:48:28.976730 mon.rook-ceph-mon7 [INF] osd.84 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491916 >= grace 20.010293)
2018-07-17 16:48:28.976785 mon.rook-ceph-mon7 [INF] osd.85 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491870 >= grace 20.011151)
2018-07-17 16:48:28.976843 mon.rook-ceph-mon7 [INF] osd.86 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491828 >= grace 20.010293)
2018-07-17 16:48:28.976890 mon.rook-ceph-mon7 [INF] Marking osd.1 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976913 mon.rook-ceph-mon7 [INF] Marking osd.2 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976933 mon.rook-ceph-mon7 [INF] Marking osd.3 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976954 mon.rook-ceph-mon7 [INF] Marking osd.4 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976979 mon.rook-ceph-mon7 [INF] Marking osd.9 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.977000 mon.rook-ceph-mon7 [INF] Marking osd.10 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977020 mon.rook-ceph-mon7 [INF] Marking osd.11 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977040 mon.rook-ceph-mon7 [INF] Marking osd.12 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977059 mon.rook-ceph-mon7 [INF] Marking osd.13 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977079 mon.rook-ceph-mon7 [INF] Marking osd.14 out (has 
been down for 605 seconds)
2018-07-17 16:48:30.889316 mon.rook-ceph-mon7 [INF] osd.55 
7.129.218.12:6920/90761 boot
2018-07-17 16:48:31.113052 mon.rook-ceph-mon7 [WRN] Health check update: 
4946/8854434 objects misplaced (0.056%) (OBJECT_MISPLACED)
2018-07-17 16:48:31.113087 mon.rook-ceph-mon7 [WRN] Health check update: 
Degraded data redundancy: 7951/8854434 objects degraded (0.090%), 88 pgs 
degraded, 273 pgs undersized (PG_DEGRADED)
2018-07-17 16:48:32.763546 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10439 pgs inactive, 8994 pgs down, 1639 pgs peering, 
88 pgs incomplete, 3430 pgs stale (PG_AVAILABILITY)
2018-07-17 16:48:32.763578 mon.rook-ceph-mon7 [WRN] Health check update: 29 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 16:48:34.096178 mon.rook-ceph-mon7 [INF] osd.88 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
66.612054 >= grace 20.010283)
2018-07-17 16:48:34.108020 mon.rook-ceph-mon7 [WRN] Health check update: 112 
osds down (OSD_DOWN)
2018-07-17 16:48:38.736108 mon.rook-ceph-mon7 [WRN] Health check update: 
4946/8843715 objects misplaced (0.056%) (OBJECT_MISPLACED)
2018-07-17 16:48:38.736140 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10415 pgs inactive, 9000 pgs down, 1635 pgs peering, 
88 pgs incomplete, 3418 pgs stale (PG_AVAILABILITY)
2018-07-17 16:48:38.736166 mon.rook-ceph-mon7 [WRN] Health check update: 
Degraded data redundancy: 7949/8843715 objects degraded (0.090%), 86 pgs 
degraded, 267 pgs undersized (PG_DEGRADED)
2018-07-17 16:48:40.430146 mon.rook-ceph-mon7 [WRN] Health check update: 111 
osds down (OSD_DOWN)
2018-07-17 16:48:40.812579 mon.rook-ceph-mon7 [INF] osd.117 
7.129.217.10:6833/98090 boot
2018-07-17 16:48:42.427204 mon.rook-ceph-mon7 [INF] osd.115 
7.129.217.10:6940/98114 boot
2018-07-17 16:48:42.427297 mon.rook-c

[ceph-users] Cluster in bad shape, seemingly endless cycle of OSDs failed, then marked down, then booted, then failed again

[Bryan Banister]

Hi all,

We're still very new to managing Ceph and seem to have cluster that is in an 
endless loop of failing OSDs, then marking them down, then booting them again:

Here are some example logs:
2018-07-17 16:48:28.976673 mon.rook-ceph-mon7 [INF] osd.83 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491973 >= grace 20.010293)
2018-07-17 16:48:28.976730 mon.rook-ceph-mon7 [INF] osd.84 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491916 >= grace 20.010293)
2018-07-17 16:48:28.976785 mon.rook-ceph-mon7 [INF] osd.85 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491870 >= grace 20.011151)
2018-07-17 16:48:28.976843 mon.rook-ceph-mon7 [INF] osd.86 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
61.491828 >= grace 20.010293)
2018-07-17 16:48:28.976890 mon.rook-ceph-mon7 [INF] Marking osd.1 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976913 mon.rook-ceph-mon7 [INF] Marking osd.2 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976933 mon.rook-ceph-mon7 [INF] Marking osd.3 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976954 mon.rook-ceph-mon7 [INF] Marking osd.4 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.976979 mon.rook-ceph-mon7 [INF] Marking osd.9 out (has been 
down for 605 seconds)
2018-07-17 16:48:28.977000 mon.rook-ceph-mon7 [INF] Marking osd.10 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977020 mon.rook-ceph-mon7 [INF] Marking osd.11 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977040 mon.rook-ceph-mon7 [INF] Marking osd.12 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977059 mon.rook-ceph-mon7 [INF] Marking osd.13 out (has 
been down for 605 seconds)
2018-07-17 16:48:28.977079 mon.rook-ceph-mon7 [INF] Marking osd.14 out (has 
been down for 605 seconds)
2018-07-17 16:48:30.889316 mon.rook-ceph-mon7 [INF] osd.55 
7.129.218.12:6920/90761 boot
2018-07-17 16:48:31.113052 mon.rook-ceph-mon7 [WRN] Health check update: 
4946/8854434 objects misplaced (0.056%) (OBJECT_MISPLACED)
2018-07-17 16:48:31.113087 mon.rook-ceph-mon7 [WRN] Health check update: 
Degraded data redundancy: 7951/8854434 objects degraded (0.090%), 88 pgs 
degraded, 273 pgs undersized (PG_DEGRADED)
2018-07-17 16:48:32.763546 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10439 pgs inactive, 8994 pgs down, 1639 pgs peering, 
88 pgs incomplete, 3430 pgs stale (PG_AVAILABILITY)
2018-07-17 16:48:32.763578 mon.rook-ceph-mon7 [WRN] Health check update: 29 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-17 16:48:34.096178 mon.rook-ceph-mon7 [INF] osd.88 failed 
(root=default,host=carg-kubelet-osd04) (3 reporters from different host after 
66.612054 >= grace 20.010283)
2018-07-17 16:48:34.108020 mon.rook-ceph-mon7 [WRN] Health check update: 112 
osds down (OSD_DOWN)
2018-07-17 16:48:38.736108 mon.rook-ceph-mon7 [WRN] Health check update: 
4946/8843715 objects misplaced (0.056%) (OBJECT_MISPLACED)
2018-07-17 16:48:38.736140 mon.rook-ceph-mon7 [WRN] Health check update: 
Reduced data availability: 10415 pgs inactive, 9000 pgs down, 1635 pgs peering, 
88 pgs incomplete, 3418 pgs stale (PG_AVAILABILITY)
2018-07-17 16:48:38.736166 mon.rook-ceph-mon7 [WRN] Health check update: 
Degraded data redundancy: 7949/8843715 objects degraded (0.090%), 86 pgs 
degraded, 267 pgs undersized (PG_DEGRADED)
2018-07-17 16:48:40.430146 mon.rook-ceph-mon7 [WRN] Health check update: 111 
osds down (OSD_DOWN)
2018-07-17 16:48:40.812579 mon.rook-ceph-mon7 [INF] osd.117 
7.129.217.10:6833/98090 boot
2018-07-17 16:48:42.427204 mon.rook-ceph-mon7 [INF] osd.115 
7.129.217.10:6940/98114 boot
2018-07-17 16:48:42.427297 mon.rook-ceph-mon7 [INF] osd.100 
7.129.217.10:6899/98091 boot
2018-07-17 16:48:42.427502 mon.rook-ceph-mon7 [INF] osd.95 
7.129.217.10:6901/98092 boot

Not sure this is going to fix itself.  Any ideas on how to handle this 
situation??

Thanks in advance!
-Bryan




Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential, or privileged information and/or 
personal data. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, or copying of this email is strictly 
prohibited, and requested to notify the sender immediately and destroy this 
email and any attachments. Email transmission cannot be guaranteed to be secure 
or error-free. The Company, therefore, does not make any guarantees as to the 
completeness or accuracy of this email or any attachments. This email is for 
informational purposes only and does not constitute a recommendation, offer, 
request, or solicitation of any kind to buy, sell, subscribe, redeem, or 
perform any type of transaction of a financial product. Personal data, as 
defined by applicable data privacy laws, contained in this 

[ceph-users] Many inconsistent PGs in EC pool, is this normal?

[Bryan Banister]

Hi all,

We started running a EC pool based object store, set up with a 4+2 
configuration, and we seem to be getting an almost constant report of 
inconsistent PGs during scrub operations.  For example:
root@rook-tools:/# ceph pg ls inconsistent
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG  
DISK_LOG STATE STATE_STAMPVERSION   
REPORTED  UP  UP_PRIMARY ACTING  
ACTING_PRIMARY LAST_SCRUB SCRUB_STAMPLAST_DEEP_SCRUB 
DEEP_SCRUB_STAMP
19.26   104  00 0   0 436207616 1537
 1537 active+clean+inconsistent 2018-06-28 15:04:54.054227 1811'3137 2079:5075 
[206,116,68,31,193,156]206 [206,116,68,31,193,156]206  
1811'3137 2018-06-27 21:00:17.011611   1811'3137 2018-06-27 21:00:17.011611
19.234   98  00 0   0 406847488 1581
 1581 active+clean+inconsistent 2018-06-28 15:05:18.077003 1811'2981 2080:4822  
[28,131,229,180,84,68] 28  [28,131,229,180,84,68] 28  
1811'2981 2018-06-28 14:09:54.092401   1811'2981 2018-06-28 14:09:54.092401
19.2a8  116  00 0   0 486539264 1561
 1561 active+clean+inconsistent 2018-06-28 15:04:54.073762 1811'3161 2079:4825 
[177,68,222,13,131,107]177 [177,68,222,13,131,107]177  
1811'3161 2018-06-28 07:51:21.109587   1811'3161 2018-06-28 07:51:21.109587
19.406  126  00 0   0 520233399 1557
 1557 active+clean+inconsistent 2018-06-28 15:04:57.142651 1811'3057 2080:4944  
[230,199,128,68,92,11]230  [230,199,128,68,92,11]230  
1811'3057 2018-06-27 18:36:18.497899   1811'3057 2018-06-27 18:36:18.497899
19.46b  109  00 0   0 449840274 1558
 1558 active+clean+inconsistent 2018-06-28 15:04:54.227970 1811'3058 2079:4986  
[18,68,130,94,181,225] 18  [18,68,130,94,181,225] 18  
1811'3058 2018-06-27 14:32:17.800961   1811'3058 2018-06-27 14:32:17.800961
[snip]

We sometimes see that running a deep scrub on the PG resolves the issue but not 
all the time.

We have been running the PG repair operation on them (e.g. ceph pg repair ), which clears the issue.  Is this the correct way to resolve this issue?

Is this a normal behavior for a Ceph cluster?

If so, why doesn't it attempt to repair itself automatically?

Thanks for the help understanding Ceph, we are very new to it still!
-Bryan





Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential, or privileged information and/or 
personal data. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, or copying of this email is strictly 
prohibited, and requested to notify the sender immediately and destroy this 
email and any attachments. Email transmission cannot be guaranteed to be secure 
or error-free. The Company, therefore, does not make any guarantees as to the 
completeness or accuracy of this email or any attachments. This email is for 
informational purposes only and does not constitute a recommendation, offer, 
request, or solicitation of any kind to buy, sell, subscribe, redeem, or 
perform any type of transaction of a financial product. Personal data, as 
defined by applicable data privacy laws, contained in this email may be 
processed by the Company, and any of its affiliated or related companies, for 
potential ongoing compliance and/or business-related purposes. You may have 
rights regarding your personal data; for information on exercising these rights 
or the Company's treatment of personal data, please email 
datareque...@jumptrading.com.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph tech talk on deploy ceph with rook on kubernetes

[Bryan Banister]

Hi Sage,

Please provide a link to the youtube video once it's posted, thanks!!
-Bryan

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sage 
Weil
Sent: Thursday, May 24, 2018 12:04 PM
To: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph tech talk on deploy ceph with rook on kubernetes

Note: External Email
-

Starting now!

https://redhat.bluejeans.com/967991495/

It'll be recorded and go up on youtube shortly as well.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Increasing number of PGs by not a factor of two?

[Bryan Banister]

+1

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Kai 
Wagner
Sent: Thursday, May 17, 2018 4:20 PM
To: David Turner 
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Increasing number of PGs by not a factor of two?


Great summary David. Wouldn't this be worth a blog post?

On 17.05.2018 20:36, David Turner wrote:
By sticking with PG numbers as a base 2 number (1024, 16384, etc) all of your 
PGs will be the same size and easier to balance and manage.  What happens when 
you have a non base 2 number is something like this.  Say you have 4 PGs that 
are all 2GB in size.  If you increase pg(p)_num to 6, then you will have 2 PGs 
that are 2GB and 4 PGs that are 1GB as you've split 2 of the PGs into 4 to get 
to the 6 total.  If you increase the pg(p)_num to 8, then all 8 PGs will be 
1GB.  Depending on how you manage your cluster, that doesn't really matter, but 
for some methods of balancing your cluster, that will greatly imbalance things.

This would be a good time to go to a base 2 number.  I think you're thinking 
about Gluster where if you have 4 bricks and you want to increase your 
capacity, going to anything other than a multiple of 4 (8, 12, 16) kills 
performance (worse than increasing storage already does) and takes longer as it 
has to weirdly divide the data instead of splitting a single brick up to 
multiple bricks.

As you increase your PGs, do this slowly and in a loop.  I like to increase my 
PGs by 256, wait for all PGs to create, activate, and peer, rinse/repate until 
I get to my target.  [1] This is an example of a script that should accomplish 
this with no interference.  Notice the use of flags while increasing the PGs.  
It will make things take much longer if you have an OSD OOM itself or die for 
any reason by adding to the peering needing to happen.  It will also be wasted 
IO to start backfilling while you're still making changes; it's best to wait 
until you finish increasing your PGs and everything peers before you let data 
start moving.

Another thing to keep in mind is how long your cluster will be moving data 
around.  Increasing your PG count on a pool full of data is one of the most 
intensive operations you can tell a cluster to do.  The last time I had to do 
this, I increased pg(p)_num by 4k PGs from 16k to 32k, let it backfill, 
rinse/repeat until the desired PG count was achieved.  For me, that 4k PGs 
would take 3-5 days depending on other cluster load and how full the cluster 
was.  If you do decide to increase your PGs by 4k instead of the full increase, 
change the 16384 to the number you decide to go to, backfill, continue.


[1]
# Make sure to set pool variable as well as the number ranges to the 
appropriate values.
flags="nodown nobackfill norecover"
for flag in $flags; do
  ceph osd set $flag
done
pool=rbd
echo "$pool currently has $(ceph osd pool get $pool pg_num) PGs"
# The first number is your current PG count for the pool, the second number is 
the target PG count, and the third number is how many to increase it by each 
time through the loop.
for num in {7700..16384..256}; do
  ceph osd pool set $pool pg_num $num
  while sleep 10; do
ceph osd health | grep -q 'peering\|stale\|activating\|creating\|inactive' 
|| break
  done
  ceph osd pool set $pool pgp_num $num
  while sleep 10; do
ceph osd health | grep -q 'peering\|stale\|activating\|creating\|inactive' 
|| break
  done
done
for flag in $flags; do
  ceph osd unset $flag
done

On Thu, May 17, 2018 at 9:27 AM Kai Wagner 
> wrote:
Hi Oliver,

a good value is 100-150 PGs per OSD. So in your case between 20k and 30k.

You can increase your PGs, but keep in mind that this will keep the
cluster quite busy for some while. That said I would rather increase in
smaller steps than in one large move.

Kai


On 17.05.2018 01:29, Oliver Schulz wrote:
> Dear all,
>
> we have a Ceph cluster that has slowly evolved over several
> years and Ceph versions (started with 18 OSDs and 54 TB
> in 2013, now about 200 OSDs and 1.5 PB, still the same
> cluster, with data continuity). So there are some
> "early sins" in the cluster configuration, left over from
> the early days.
>
> One of these sins is the number of PGs in our CephFS "data"
> pool, which is 7200 and therefore not (as recommended)
> a power of two. Pretty much all of our data is in the
> "data" pool, the only other pools are "rbd" and "metadata",
> both contain little data (and they have way too many PGs
> already, another early sin).
>
> Is it possible - and safe - to change the number of "data"
> pool PGs from 7200 to 8192 or 16384? As we recently added
> more OSDs, I guess it would be time to increase the number
> of PGs anyhow. Or would we have to go to 14400 instead of
> 16384?
>
>
> Thanks for any advice,
>
> Oliver
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

HI David [Resending with smaller message size],

I tried setting the OSDs down and that does clear the blocked requests 
momentarily but they just return back to the same state.  Not sure how to 
proceed here, but one thought was just to do a full cold restart of the entire 
cluster.  We have disabled our backups so the cluster is effectively down.  Any 
recommendations on next steps?

This also seems like a pretty serious issue, given that making this change has 
effectively broken the cluster.  Perhaps Ceph should not allow you to increase 
the number of PGs so drastically or at least make you put in a 
‘--yes-i-really-mean-it’ flag?

Or perhaps just some warnings on the docs.ceph.com placement groups page 
(http://docs.ceph.com/docs/master/rados/operations/placement-groups/ ) and the 
ceph command man page?

Would be good to help other avoid this pitfall.

Thanks again,
-Bryan

From: David Turner [mailto:drakonst...@gmail.com]
Sent: Friday, February 16, 2018 3:21 PM
To: Bryan Banister <bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>>
Cc: Bryan Stillwell <bstillw...@godaddy.com<mailto:bstillw...@godaddy.com>>; 
Janne Johansson <icepic...@gmail.com<mailto:icepic...@gmail.com>>; Ceph Users 
<ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email

That sounds like a good next step.  Start with OSDs involved in the longest 
blocked requests.  Wait a couple minutes after the osd marks itself back up and 
continue through them.  Hopefully things will start clearing up so that you 
don't need to mark all of them down.  There is usually a only a couple OSDs 
holding everything up.

On Fri, Feb 16, 2018 at 4:15 PM Bryan Banister 
<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> wrote:
Thanks David,

Taking the list of all OSDs that are stuck reports that a little over 50% of 
all OSDs are in this condition.  There isn’t any discernable pattern that I can 
find and they are spread across the three servers.  All of the OSDs are online 
as far as the service is concern.

I have also taken all PGs that were reported the health detail output and 
looked for any that report “peering_blocked_by” but none do, so I can’t tell if 
any OSD is actually blocking the peering operation.

As suggested, I got a report of all peering PGs:
[root@carf-ceph-osd01 ~]# ceph health detail | grep "pg " | grep peering | sort 
-k13
pg 14.fe0 is stuck peering since forever, current state peering, last 
acting [104,94,108]
pg 14.fe0 is stuck unclean since forever, current state peering, last 
acting [104,94,108]
pg 14.fbc is stuck peering since forever, current state peering, last 
acting [110,91,0]
pg 14.fd1 is stuck peering since forever, current state peering, last 
acting [130,62,111]
pg 14.fd1 is stuck unclean since forever, current state peering, last 
acting [130,62,111]
pg 14.fed is stuck peering since forever, current state peering, last 
acting [32,33,82]
pg 14.fed is stuck unclean since forever, current state peering, last 
acting [32,33,82]
pg 14.fee is stuck peering since forever, current state peering, last 
acting [37,96,68]
pg 14.fee is stuck unclean since forever, current state peering, last 
acting [37,96,68]
pg 14.fe8 is stuck peering since forever, current state peering, last 
acting [45,31,107]
pg 14.fe8 is stuck unclean since forever, current state peering, last 
acting [45,31,107]
pg 14.fc1 is stuck peering since forever, current state peering, last 
acting [59,124,39]
pg 14.ff2 is stuck peering since forever, current state peering, last 
acting [62,117,7]
pg 14.ff2 is stuck unclean since forever, current state peering, last 
acting [62,117,7]
pg 14.fe4 is stuck peering since forever, current state peering, last 
acting [84,55,92]
pg 14.fe4 is stuck unclean since forever, current state peering, last 
acting [84,55,92]
pg 14.fb0 is stuck peering since forever, current state peering, last 
acting [94,30,38]
pg 14.ffc is stuck peering since forever, current state peering, last 
acting [96,53,70]
pg 14.ffc is stuck unclean since forever, current state peering, last 
acting [96,53,70]

Some have common OSDs but some OSDs only listed once.

Should I try just marking OSDs with stuck requests down to see if that will 
re-assert them?

Thanks!!
-Bryan

From: David Turner [mailto:drakonst...@gmail.com<mailto:drakonst...@gmail.com>]
Sent: Friday, February 16, 2018 2:51 PM

To: Bryan Banister <bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>>
Cc: Bryan Stillwell <bstillw...@godaddy.com<mailto:bstillw...@godaddy.com>>; 
Janne Johansson <icepic...@gmail.com<mailto:icepic...@gmail.com>>; Ceph Users 
<ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>>
Subject: Re: [

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Thanks David,

Taking the list of all OSDs that are stuck reports that a little over 50% of 
all OSDs are in this condition.  There isn’t any discernable pattern that I can 
find and they are spread across the three servers.  All of the OSDs are online 
as far as the service is concern.

I have also taken all PGs that were reported the health detail output and 
looked for any that report “peering_blocked_by” but none do, so I can’t tell if 
any OSD is actually blocking the peering operation.

As suggested, I got a report of all peering PGs:
[root@carf-ceph-osd01 ~]# ceph health detail | grep "pg " | grep peering | sort 
-k13
pg 14.fe0 is stuck peering since forever, current state peering, last 
acting [104,94,108]
pg 14.fe0 is stuck unclean since forever, current state peering, last 
acting [104,94,108]
pg 14.fbc is stuck peering since forever, current state peering, last 
acting [110,91,0]
pg 14.fd1 is stuck peering since forever, current state peering, last 
acting [130,62,111]
pg 14.fd1 is stuck unclean since forever, current state peering, last 
acting [130,62,111]
pg 14.fed is stuck peering since forever, current state peering, last 
acting [32,33,82]
pg 14.fed is stuck unclean since forever, current state peering, last 
acting [32,33,82]
pg 14.fee is stuck peering since forever, current state peering, last 
acting [37,96,68]
pg 14.fee is stuck unclean since forever, current state peering, last 
acting [37,96,68]
pg 14.fe8 is stuck peering since forever, current state peering, last 
acting [45,31,107]
pg 14.fe8 is stuck unclean since forever, current state peering, last 
acting [45,31,107]
pg 14.fc1 is stuck peering since forever, current state peering, last 
acting [59,124,39]
pg 14.ff2 is stuck peering since forever, current state peering, last 
acting [62,117,7]
pg 14.ff2 is stuck unclean since forever, current state peering, last 
acting [62,117,7]
pg 14.fe4 is stuck peering since forever, current state peering, last 
acting [84,55,92]
pg 14.fe4 is stuck unclean since forever, current state peering, last 
acting [84,55,92]
pg 14.fb0 is stuck peering since forever, current state peering, last 
acting [94,30,38]
pg 14.ffc is stuck peering since forever, current state peering, last 
acting [96,53,70]
pg 14.ffc is stuck unclean since forever, current state peering, last 
acting [96,53,70]

Some have common OSDs but some OSDs only listed once.

Should I try just marking OSDs with stuck requests down to see if that will 
re-assert them?

Thanks!!
-Bryan

From: David Turner [mailto:drakonst...@gmail.com]
Sent: Friday, February 16, 2018 2:51 PM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: Bryan Stillwell <bstillw...@godaddy.com>; Janne Johansson 
<icepic...@gmail.com>; Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email

The questions I definitely know the answer to first, and then we'll continue 
from there.  If an OSD is blocking peering but is online, when you mark it as 
down in the cluster it receives a message in it's log saying it was wrongly 
marked down and tells the mons it is online.  That gets it to stop what it was 
doing and start talking again.  I referred to that as re-asserting.  If the OSD 
that you marked down doesn't mark itself back up within a couple minutes, 
restarting the OSD might be a good idea.  Then again actually restarting the 
daemon could be bad because the daemon is doing something.  With as much 
potential for places to work with to get things going, actually restarting the 
daemons is probably something I would wait to do for now.

The reason the cluster doesn't know anything about the PG is because it's still 
creating and hasn't actually been created.  Starting with some of the OSDs that 
you see with blocked requests would be a good idea.  Eventually you'll down an 
OSD that when it comes back up things start looking much better as things start 
peering and getting better.  Below are the list of OSDs you had from a previous 
email that if they're still there with stuck requests then they'll be good to 
start doing this to.  On closer review, it's almost all of them... but you have 
to start somewhere.  Another possible place to start with these is to look at a 
list of all of the peering PGs and see if there are any common OSDs when you 
look at all of them at once.  Some patterns may emerge and would be good 
options to try.

osds 7,39,60,103,133 have stuck requests > 67108.9 sec
osds 
5,12,13,28,33,40,55,56,61,64,69,70,75,83,92,96,110,114,119,122,123,129,131 have 
stuck requests > 134218 sec
osds 
4,8,10,15,16,20,27,29,30,31,34,37,38,42,43,44,47,48,49,51,52,57,66,68,73,81,84,85,87,90,95,97,99,100,102,105,106,107,108,111,112,113,121,124,127,130,132
 have stuck requests > 268435 sec


On Fri, Feb 16, 2018 at 2:53 PM

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Thanks David,

I have set the nobackfill, norecover, noscrub, and nodeep-scrub options at this 
point and the backfills have stopped.  I’ll also stop the backups from pushing 
into ceph for now.

I don’t want to make things worse, so ask for some more guidance now.


1)  In looking at a PG that is still peering or one that is “unknown”, Ceph 
complains that it doesn’t have that pgid:
pg 14.fb0 is stuck peering since forever, current state peering, last 
acting [94,30,38]
[root@carf-ceph-osd03 ~]# ceph pg 14.fb0 query
Error ENOENT: i don't have pgid 14.fb0
[root@carf-ceph-osd03 ~]#


2)  One that is activating shows this for the recovery_state:
[root@carf-ceph-osd03 ~]# ceph pg 14.fe1 query | less
[snip]
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2018-02-13 14:33:21.406919",
"might_have_unfound": [
{
"osd": "84(0)",
"status": "not queried"
}
],
"recovery_progress": {
"backfill_targets": [
"56(0)",
"87(1)",
"88(2)"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
},
"scrub": {
"scrubber.epoch_start": "0",
"scrubber.active": false,
"scrubber.state": "INACTIVE",
"scrubber.start": "MIN",
"scrubber.end": "MIN",
"scrubber.subset_last_update": "0'0",
"scrubber.deep": false,
"scrubber.seed": 0,
"scrubber.waiting_on": 0,
"scrubber.waiting_on_whom": []
}
},
{
"name": "Started",
    "enter_time": "2018-02-13 14:33:17.491148"
}
],


Sorry for all the hand holding, but how do I determine if I need to set an OSD 
as ‘down’ to fix the issues, and how does it go about re-asserting itself?

I again tried looking at the ceph docs on troubleshooting OSDs but didn’t find 
any details.  Man page also has no details.

Thanks again,
-Bryan

From: David Turner [mailto:drakonst...@gmail.com]
Sent: Friday, February 16, 2018 1:21 PM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: Bryan Stillwell <bstillw...@godaddy.com>; Janne Johansson 
<icepic...@gmail.com>; Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email

Your problem might have been creating too many PGs at once.  I generally 
increase pg_num and pgp_num by no more than 256 at a time.  Making sure that 
all PGs are creating, peered, and healthy (other than backfilling).

To help you get back to a healthy state, let's start off by getting all of your 
PGs peered.  Go ahead and put a stop to backfilling, recovery, scrubbing, etc.  
Those are all hindering the peering effort right now.  The more clients you can 
disable is also better.

ceph osd set nobackfill
ceph osd set norecovery
ceph osd set noscrubbing
ceph osd set nodeep-scrubbing

After that look at your peering PGs and find out what is blocking their 
peering.  This is where you might need to be using `ceph osd down 23` (assuming 
you needed to kick osd.23) to mark them down in the cluster and let them 
re-assert themselves.  Once you have all PGs done with peering, go ahead and 
unset nobackfill and norecovery and let the cluster start moving data around.  
Leaving noscrubbing and nodeep-scrubbing off is optional and up to you.  I'll 
never say it's better to leave them off, but scrubbing does use a fair bit of 
spindles while you're trying to backfill.

On Fri, Feb 16, 2018 at 2:12 PM Bryan Banister 
<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> wrote:
Well I decided to try the increase in PGs to 4096 and that seems to have caused 
some issues:

2018-02-16 12:38:35.798911 mon.carf-ceph-osd01 [ERR] overall HEALTH_ERR 

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Well I decided to try the increase in PGs to 4096 and that seems to have caused 
some issues:

2018-02-16 12:38:35.798911 mon.carf-ceph-osd01 [ERR] overall HEALTH_ERR 
61802168/241154376 objects misplaced (25.628%); Reduced data availability: 2081 
pgs inactive, 322 pgs peering; Degraded data redundancy: 552/241154376 objects 
degraded (0.000%), 3099 pgs unclean, 38 pgs degraded; 163 stuck requests are 
blocked > 4096 sec

The cluster is actively backfilling misplaced objects, but not all PGs are 
active at this point and may are stuck peering, stuck unclean, or have a state 
of unknown:
PG_AVAILABILITY Reduced data availability: 2081 pgs inactive, 322 pgs peering
pg 14.fae is stuck inactive for 253360.025730, current state 
activating+remapped, last acting [85,12,41]
pg 14.faf is stuck inactive for 253368.511573, current state unknown, last 
acting []
pg 14.fb0 is stuck peering since forever, current state peering, last 
acting [94,30,38]
pg 14.fb1 is stuck inactive for 253362.605886, current state 
activating+remapped, last acting [6,74,34]
[snip]

The health also shows a large number of degraded data redundancy PGs:
PG_DEGRADED Degraded data redundancy: 552/241154376 objects degraded (0.000%), 
3099 pgs unclean, 38 pgs degraded
pg 14.fc7 is stuck unclean for 253368.511573, current state unknown, last 
acting []
pg 14.fc8 is stuck unclean for 531622.531271, current state 
active+remapped+backfill_wait, last acting [73,132,71]
pg 14.fca is stuck unclean for 420540.396199, current state 
active+remapped+backfill_wait, last acting [0,80,61]
pg 14.fcb is stuck unclean for 531622.421855, current state 
activating+remapped, last acting [70,26,75]
[snip]

We also now have a number of stuck requests:
REQUEST_STUCK 163 stuck requests are blocked > 4096 sec
69 ops are blocked > 268435 sec
66 ops are blocked > 134218 sec
   28 ops are blocked > 67108.9 sec
osds 7,39,60,103,133 have stuck requests > 67108.9 sec
osds 
5,12,13,28,33,40,55,56,61,64,69,70,75,83,92,96,110,114,119,122,123,129,131 have 
stuck requests > 134218 sec
osds 
4,8,10,15,16,20,27,29,30,31,34,37,38,42,43,44,47,48,49,51,52,57,66,68,73,81,84,85,87,90,95,97,99,100,102,105,106,107,108,111,112,113,121,124,127,130,132
 have stuck requests > 268435 sec

I tried looking through the mailing list archive on how to solve the stuck 
requests, and it seems that restarting the OSDs is the right way?

At this point we have just been watching the backfills running and see a steady 
but slow decrease of misplaced objects.  When the cluster is idle, the overall 
OSD disk utilization is not too bad at roughly 40% on the physical disks 
running these backfills.

However we still have our backups trying to push new images to the cluster.  
This worked ok for the first few days, but yesterday we were getting failure 
alerts.  I checked the status of the RGW service and noticed that 2 of the 3 
RGW civetweb servers where not responsive.  I restarted the RGWs on the ones 
that appeared hung and that got them working for a while, but then the same 
condition happened.  The RGWs seem to have recovered on their own now, but 
again the cluster is idle and only backfills are currently doing anything (that 
I can tell).  I did see these log entries:
2018-02-15 16:46:07.541542 7fffe6c56700  1 heartbeat_map is_healthy 
'RGWAsyncRadosProcessor::m_tp thread 0x7fffcec26700' had timed out after 600
2018-02-15 16:46:12.541613 7fffe6c56700  1 heartbeat_map is_healthy 
'RGWAsyncRadosProcessor::m_tp thread 0x7fffdbc40700' had timed out after 600
2018-02-15 16:46:12.541629 7fffe6c56700  1 heartbeat_map is_healthy 
'RGWAsyncRadosProcessor::m_tp thread 0x7fffcec26700' had timed out after 600
2018-02-15 16:46:17.541701 7fffe6c56700  1 heartbeat_map is_healthy 
'RGWAsyncRadosProcessor::m_tp thread 0x7fffdbc40700' had timed out after 600

At this point we do not know to proceed with recovery efforts.  I tried looking 
at the ceph docs and mail list archives but wasn’t able to determine the right 
path forward here.

Any help is appreciated,
-Bryan


From: Bryan Stillwell [mailto:bstillw...@godaddy.com]
Sent: Tuesday, February 13, 2018 2:27 PM
To: Bryan Banister <bbanis...@jumptrading.com>; Janne Johansson 
<icepic...@gmail.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email

It may work fine, but I would suggest limiting the number of operations going 
on at the same time.

Bryan

From: Bryan Banister 
<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>>
Date: Tuesday, February 13, 2018 at 1:16 PM
To: Bryan Stillwell <bstillw...@godaddy.com<mailto:bstillw...@godaddy.com>>, 
Janne Johansson <icepic...@gmail.com<mailto:icepic...@gmail.com>>
Cc: Ceph Users <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>>
Subject:

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Thanks for the response Bryan!

Would it be good to go ahead and do the increase up to 4096 PGs for thee pool 
given that it's only at 52% done with the rebalance backfilling operations?

Thanks in advance!!
-Bryan

-Original Message-
From: Bryan Stillwell [mailto:bstillw...@godaddy.com]
Sent: Tuesday, February 13, 2018 12:43 PM
To: Bryan Banister <bbanis...@jumptrading.com>; Janne Johansson 
<icepic...@gmail.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email
-

Bryan,

Based off the information you've provided so far, I would say that your largest 
pool still doesn't have enough PGs.

If you originally had only 512 PGs for you largest pool (I'm guessing 
.rgw.buckets has 99% of your data), then on a balanced cluster you would have 
just ~11.5 PGs per OSD (3*512/133).  That's way lower than the recommended 100 
PGs/OSD.

Based on the number of disks and assuming your .rgw.buckets pool has 99% of the 
data, you should have around 4,096 PGs for that pool.  You'll still end up with 
an uneven distribution, but the outliers shouldn't be as far out.

Sage recently wrote a new balancer plugin that makes balancing a cluster 
something that happens automatically.  He gave a great talk at LinuxConf 
Australia that you should check out, here's a link into the video where he 
talks about the balancer and the need for it:

https://youtu.be/GrStE7XSKFE?t=20m14s

Even though your objects are fairly large, they are getting broken up into 
chunks that are spread across the cluster.  You can see how large each of your 
PGs are with a command like this:

ceph pg dump | grep '[0-9]*\.[0-9a-f]*' | awk '{ print $1 "\t" $7 }' |sort -n 
-k2

You'll see that within a pool the PG sizes are fairly close to the same size, 
but in your cluster the PGs are fairly large (~200GB would be my guess).

Bryan

From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of Bryan 
Banister <bbanis...@jumptrading.com>
Date: Monday, February 12, 2018 at 2:19 PM
To: Janne Johansson <icepic...@gmail.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Hi Janne and others,

We used the “ceph osd reweight-by-utilization “ command to move a small amount 
of data off of the top four OSDs by utilization.  Then we updated the pg_num 
and pgp_num on the pool from 512 to 1024 which started moving roughly 50% of 
the objects around as a result.  The unfortunate issue is that the weights on 
the OSDs are still roughly equivalent and the OSDs that are nearfull were still 
getting allocated objects during the rebalance backfill operations.

At this point I have made some massive changes to the weights of the OSDs in an 
attempt to stop Ceph from allocating any more data to OSDs that are getting 
close to full.  Basically the OSD with the lowest utilization remains weighted 
at 1 and the rest of the OSDs are now reduced in weight based on the percent 
usage of the OSD + the %usage of the OSD with the amount of data (21% at the 
time).  This means the OSD that is at the most full at this time at 86% full 
now has a weight of only .33 (it was at 89% when reweight was applied).  I’m 
not sure this is a good idea, but it seemed like the only option I had.  Please 
let me know if I’m making a bad situation worse!

I still have the question on how this happened in the first place and how to 
prevent it from happening going forward without a lot of monitoring and 
reweighting on weekends/etc to keep things balanced.  It sounds like Ceph is 
really expecting that objects stored into a pool will roughly have the same 
size, is that right?

Our backups going into this pool have very large variation in size, so would it 
be better to create multiple pools based on expected size of objects and then 
put backups of similar size into each pool?

The backups also have basically the same names with the only difference being 
the date which it was taken (e.g. backup name difference in subsequent days can 
be one digit at times), so does this mean that large backups with basically the 
same name will end up being placed in the same PGs based on the CRUSH 
calculation using the object name?

Thanks,
-Bryan

From: Janne Johansson [mailto:icepic...@gmail.com]
Sent: Wednesday, January 31, 2018 9:34 AM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email



2018-01-31 15:58 GMT+01:00 Bryan Banister <mailto:bbanis...@jumptrading.com>:


Given that this will move data around (I think), should we increase the pg_num 
and pgp_num first and then see how it looks?


I guess adding pgs and pgps will move stuff around too, but if the PGCALC 
formula says you should have more then that wo

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Hi Janne and others,

We used the “ceph osd reweight-by-utilization “ command to move a small amount 
of data off of the top four OSDs by utilization.  Then we updated the pg_num 
and pgp_num on the pool from 512 to 1024 which started moving roughly 50% of 
the objects around as a result.  The unfortunate issue is that the weights on 
the OSDs are still roughly equivalent and the OSDs that are nearfull were still 
getting allocated objects during the rebalance backfill operations.

At this point I have made some massive changes to the weights of the OSDs in an 
attempt to stop Ceph from allocating any more data to OSDs that are getting 
close to full.  Basically the OSD with the lowest utilization remains weighted 
at 1 and the rest of the OSDs are now reduced in weight based on the percent 
usage of the OSD + the %usage of the OSD with the amount of data (21% at the 
time).  This means the OSD that is at the most full at this time at 86% full 
now has a weight of only .33 (it was at 89% when reweight was applied).  I’m 
not sure this is a good idea, but it seemed like the only option I had.  Please 
let me know if I’m making a bad situation worse!

I still have the question on how this happened in the first place and how to 
prevent it from happening going forward without a lot of monitoring and 
reweighting on weekends/etc to keep things balanced.  It sounds like Ceph is 
really expecting that objects stored into a pool will roughly have the same 
size, is that right?

Our backups going into this pool have very large variation in size, so would it 
be better to create multiple pools based on expected size of objects and then 
put backups of similar size into each pool?

The backups also have basically the same names with the only difference being 
the date which it was taken (e.g. backup name difference in subsequent days can 
be one digit at times), so does this mean that large backups with basically the 
same name will end up being placed in the same PGs based on the CRUSH 
calculation using the object name?

Thanks,
-Bryan

From: Janne Johansson [mailto:icepic...@gmail.com]
Sent: Wednesday, January 31, 2018 9:34 AM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email



2018-01-31 15:58 GMT+01:00 Bryan Banister 
<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>>:


Given that this will move data around (I think), should we increase the pg_num 
and pgp_num first and then see how it looks?


I guess adding pgs and pgps will move stuff around too, but if the PGCALC 
formula says you should have more then that would still be a good
start. Still, a few manual reweights first to take the 85-90% ones down might 
be good, some move operations are going to refuse adding things
to too-full OSDs, so you would not want to get accidentally bumped above such a 
limit due to some temp-data being created during moves.

Also, dont bump pgs like crazy, you can never move down. Aim for getting ~100 
per OSD at most, and perhaps even then in smaller steps so
that the creation (and evening out of data to the new empty PGs) doesn't kill 
normal client I/O perf in the meantime.

--
May the most significant bit of your life be positive.



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Thanks for the response, Janne!

Here is what test-reweight-by-utilization gives me:

[root@carf-ceph-osd01 ~]# ceph osd test-reweight-by-utilization
no change
moved 12 / 4872 (0.246305%)
avg 36.6316
stddev 5.37535 -> 5.29218 (expected baseline 6.02961)
min osd.48 with 25 -> 25 pgs (0.682471 -> 0.682471 * mean)
max osd.90 with 48 -> 48 pgs (1.31034 -> 1.31034 * mean)

oload 120
max_change 0.05
max_change_osds 4
average_utilization 0.5273
overload_utilization 0.6327
osd.113 weight 1. -> 0.9500
osd.87 weight 1. -> 0.9500
osd.29 weight 1. -> 0.9500
osd.52 weight 1. -> 0.9500

I tried looking for documentation on this command to see if there is a way to 
increase the max_change or max_change_osd’s but can’t find any docs on how to 
do this!

Man:
   Subcommand reweight-by-utilization reweight OSDs by utilization 
[overload-percentage-for-consideration, default 120].

   Usage:

  ceph osd reweight-by-utilization {<int[100-]>}
  {--no-increasing}

The `ceph –h` output:
  osd reweight-by-utilization {} {} {} 
{--no-increasing}

What do those optional parameters do (e.g. {} {} {} 
{--no-increasing} )??

We could keep running this multiple times, but would be nice to just rebalance 
everything in one shot so that the rebalance gets things back to pretty even.

Yes, these backup images do vary greatly in size, but I expected that just 
through random PG allocation that all OSDs would have still accumulated roughly 
the same number of small and large objects that the usage would be much closer 
to even.  This usage is way imbalanced!  So I still need to know how to 
mitigate this going forward.  Should we increase the number of PGs in this 
pool??

[root@carf-ceph-osd01 ~]# ceph osd pool ls detail
[snip]
pool 14 'carf01.rgw.buckets.data' erasure size 3 min_size 2 crush_rule 7 
object_hash rjenkins pg_num 512 pgp_num 512 last_change 3187 lfor 0/1005 flags 
hashpspool,nearfull stripe_width 8192 application rgw
[snip]

Given that this will move data around (I think), should we increase the pg_num 
and pgp_num first and then see how it looks?

Thanks,
-Bryan

From: Janne Johansson [mailto:icepic...@gmail.com]
Sent: Wednesday, January 31, 2018 7:53 AM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email
________
2018-01-30 17:24 GMT+01:00 Bryan Banister 
<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>>:
Hi all,

We are still very new to running a Ceph cluster and have run a RGW cluster for 
a while now (6-ish mo), it mainly holds large DB backups (Write once, read 
once, delete after N days).  The system is now warning us about an OSD that is 
near_full and so we went to look at the usage across OSDs.  We are somewhat 
surprised at how imbalanced the usage is across the OSDs, with the lowest usage 
at 22% full, the highest at nearly 90%, and an almost linear usage pattern 
across the OSDs (though it looks to step in roughly 5% increments):

[root@carf-ceph-osd01 ~]# ceph osd df | sort -nk8
ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
77   hdd 7.27730  1.0 7451G 1718G 5733G 23.06 0.43  32
73   hdd 7.27730  1.0 7451G 1719G 5732G 23.08 0.43  31

I noticed that the PGs (the last column there, which counts PGs per OSD I 
gather) was kind of even,
so perhaps the objects that get into the PGs are very unbalanced in size?

But yes, using reweight to compensate for this should work for you.


ceph osd test-reweight-by-utilization

should be worth testing.

--
May the most significant bit of your life be positive.



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help rebalancing OSD usage, Luminous 12.2.2

[Bryan Banister]

Sorry I hadn't RTFML archive before posting this... Looking at the following 
thread for guidance:  
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008626.html

Not the exact same situation (e.g. didn't add larger OSD disks later on) but 
seems like the same recommendations from this nearly 2yr old thread would still 
apply?

Thanks again,
-Bryan

From: Bryan Banister
Sent: Tuesday, January 30, 2018 10:26 AM
To: Bryan Banister <bbanis...@jumptrading.com>; Ceph Users 
<ceph-users@lists.ceph.com>
Subject: RE: Help rebalancing OSD usage, Luminous 12.2.2

Sorry, obviously should have been Luminous 12.2.2,
-B

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Tuesday, January 30, 2018 10:24 AM
To: Ceph Users <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>>
Subject: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email

Hi all,

We are still very new to running a Ceph cluster and have run a RGW cluster for 
a while now (6-ish mo), it mainly holds large DB backups (Write once, read 
once, delete after N days).  The system is now warning us about an OSD that is 
near_full and so we went to look at the usage across OSDs.  We are somewhat 
surprised at how imbalanced the usage is across the OSDs, with the lowest usage 
at 22% full, the highest at nearly 90%, and an almost linear usage pattern 
across the OSDs (though it looks to step in roughly 5% increments):

[root@carf-ceph-osd01 ~]# ceph osd df | sort -nk8
ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
77   hdd 7.27730  1.0 7451G 1718G 5733G 23.06 0.43  32
73   hdd 7.27730  1.0 7451G 1719G 5732G 23.08 0.43  31
  3   hdd 7.27730  1.0 7451G 2059G 5392G 27.63 0.52  27
46   hdd 7.27730  1.0 7451G 2060G 5391G 27.65 0.52  32
48   hdd 7.27730  1.0 7451G 2061G 5390G 27.66 0.52  25
127   hdd 7.27730  1.0 7451G 2066G 5385G 27.73 0.52  31
42   hdd 7.27730  1.0 7451G 2067G 5384G 27.74 0.52  42
107   hdd 7.27730  1.0 7451G 2402G 5049G 32.24 0.61  34
56   hdd 7.27730  1.0 7451G 2405G 5046G 32.28 0.61  37
51   hdd 7.27730  1.0 7451G 2406G 5045G 32.29 0.61  30
106   hdd 7.27730  1.0 7451G 2408G 5043G 32.31 0.61  29
81   hdd 7.27730  1.0 7451G 2408G 5043G 32.32 0.61  25
123   hdd 7.27730  1.0 7451G 2411G 5040G 32.37 0.61  35
47   hdd 7.27730  1.0 7451G 2412G 5039G 32.37 0.61  29
122   hdd 7.27730  1.0 7451G 2749G 4702G 36.90 0.69  30
84   hdd 7.27730  1.0 7451G 2750G 4701G 36.91 0.69  35
114   hdd 7.27730  1.0 7451G 2751G 4700G 36.92 0.69  26
82   hdd 7.27730  1.0 7451G 2751G 4700G 36.92 0.69  43
103   hdd 7.27730  1.0 7451G 2753G 4698G 36.94 0.69  39
36   hdd 7.27730  1.0 7451G 2752G 4699G 36.94 0.69  37
105   hdd 7.27730  1.0 7451G 2754G 4697G 36.97 0.69  26
14   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  31
  2   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  43
  8   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  37
20   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  28
60   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  29
69   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  37
110   hdd 7.27730  1.0 7451G 3093G 4358G 41.51 0.78  38
68   hdd 7.27730  1.0 7451G 3092G 4358G 41.51 0.78  34
76   hdd 7.27730  1.0 7451G 3093G 4358G 41.51 0.78  28
99   hdd 7.27730  1.0 7451G 3092G 4358G 41.51 0.78  34
50   hdd 7.27730  1.0 7451G 3095G 4356G 41.54 0.78  35
95   hdd 7.27730  1.0 7451G 3095G 4356G 41.54 0.78  31
  0   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  36
125   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  34
128   hdd 7.27730  1.0 7451G 3095G 4355G 41.55 0.78  37
94   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  33
63   hdd 7.27730  1.0 7451G 3096G 4355G 41.56 0.78  41
30   hdd 7.27730  1.0 7451G 3100G 4351G 41.60 0.78  31
26   hdd 7.27730  1.0 7451G 3435G 4015G 46.11 0.87  30
64   hdd 7.27730  1.0 7451G 3435G 4016G 46.11 0.87  42
57   hdd 7.27730  1.0 7451G 3437G 4014G 46.12 0.87  29
33   hdd 7.27730  1.0 7451G 3437G 4014G 46.13 0.87  27
65   hdd 7.27730  1.0 7451G 3439G 4012G 46.15 0.87  29
109   hdd 7.27730  1.0 7451G 3439G 4012G 46.16 0.87  39
11   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  32
121   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  46
78   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  36
13   hdd 7.27730  1.0 7451G 3442G 4009G 46.19 0.87  40
115   hdd 7.27730  1.0 7451G 3443G 4008G 46.21 0.87  33
41   hdd 7.27730  1.0 7451G 3444G 4007G 46.22 0.87  37
49   hdd 7.27730  1.0 7451G 3776G 3674G 50.68 0.95  34
71   hdd 7.27730  1.0 7451G 3776G 3675G 50.68 0.95  36
97   hdd 7.27730  1.0 7451G 3776G 3675G 50.68 0.95  26
17   hdd 7.27730  1.0 7451G 3777G 3674G 50.70 0.95  35
75   hdd 7.27730  1.0 7451G 3778G 3673G 50.70 0.95  41
  1   hdd 7.27730  1.0 7451G 3779G

Re: [ceph-users] Help rebalancing OSD usage, Luminous 12.2.2

[Bryan Banister]

Sorry, obviously should have been Luminous 12.2.2,
-B

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Tuesday, January 30, 2018 10:24 AM
To: Ceph Users <ceph-users@lists.ceph.com>
Subject: [ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

Note: External Email

Hi all,

We are still very new to running a Ceph cluster and have run a RGW cluster for 
a while now (6-ish mo), it mainly holds large DB backups (Write once, read 
once, delete after N days).  The system is now warning us about an OSD that is 
near_full and so we went to look at the usage across OSDs.  We are somewhat 
surprised at how imbalanced the usage is across the OSDs, with the lowest usage 
at 22% full, the highest at nearly 90%, and an almost linear usage pattern 
across the OSDs (though it looks to step in roughly 5% increments):

[root@carf-ceph-osd01 ~]# ceph osd df | sort -nk8
ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
77   hdd 7.27730  1.0 7451G 1718G 5733G 23.06 0.43  32
73   hdd 7.27730  1.0 7451G 1719G 5732G 23.08 0.43  31
  3   hdd 7.27730  1.0 7451G 2059G 5392G 27.63 0.52  27
46   hdd 7.27730  1.0 7451G 2060G 5391G 27.65 0.52  32
48   hdd 7.27730  1.0 7451G 2061G 5390G 27.66 0.52  25
127   hdd 7.27730  1.0 7451G 2066G 5385G 27.73 0.52  31
42   hdd 7.27730  1.0 7451G 2067G 5384G 27.74 0.52  42
107   hdd 7.27730  1.0 7451G 2402G 5049G 32.24 0.61  34
56   hdd 7.27730  1.0 7451G 2405G 5046G 32.28 0.61  37
51   hdd 7.27730  1.0 7451G 2406G 5045G 32.29 0.61  30
106   hdd 7.27730  1.0 7451G 2408G 5043G 32.31 0.61  29
81   hdd 7.27730  1.0 7451G 2408G 5043G 32.32 0.61  25
123   hdd 7.27730  1.0 7451G 2411G 5040G 32.37 0.61  35
47   hdd 7.27730  1.0 7451G 2412G 5039G 32.37 0.61  29
122   hdd 7.27730  1.0 7451G 2749G 4702G 36.90 0.69  30
84   hdd 7.27730  1.0 7451G 2750G 4701G 36.91 0.69  35
114   hdd 7.27730  1.0 7451G 2751G 4700G 36.92 0.69  26
82   hdd 7.27730  1.0 7451G 2751G 4700G 36.92 0.69  43
103   hdd 7.27730  1.0 7451G 2753G 4698G 36.94 0.69  39
36   hdd 7.27730  1.0 7451G 2752G 4699G 36.94 0.69  37
105   hdd 7.27730  1.0 7451G 2754G 4697G 36.97 0.69  26
14   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  31
  2   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  43
  8   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  37
20   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  28
60   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  29
69   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  37
110   hdd 7.27730  1.0 7451G 3093G 4358G 41.51 0.78  38
68   hdd 7.27730  1.0 7451G 3092G 4358G 41.51 0.78  34
76   hdd 7.27730  1.0 7451G 3093G 4358G 41.51 0.78  28
99   hdd 7.27730  1.0 7451G 3092G 4358G 41.51 0.78  34
50   hdd 7.27730  1.0 7451G 3095G 4356G 41.54 0.78  35
95   hdd 7.27730  1.0 7451G 3095G 4356G 41.54 0.78  31
  0   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  36
125   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  34
128   hdd 7.27730  1.0 7451G 3095G 4355G 41.55 0.78  37
94   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  33
63   hdd 7.27730  1.0 7451G 3096G 4355G 41.56 0.78  41
30   hdd 7.27730  1.0 7451G 3100G 4351G 41.60 0.78  31
26   hdd 7.27730  1.0 7451G 3435G 4015G 46.11 0.87  30
64   hdd 7.27730  1.0 7451G 3435G 4016G 46.11 0.87  42
57   hdd 7.27730  1.0 7451G 3437G 4014G 46.12 0.87  29
33   hdd 7.27730  1.0 7451G 3437G 4014G 46.13 0.87  27
65   hdd 7.27730  1.0 7451G 3439G 4012G 46.15 0.87  29
109   hdd 7.27730  1.0 7451G 3439G 4012G 46.16 0.87  39
11   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  32
121   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  46
78   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  36
13   hdd 7.27730  1.0 7451G 3442G 4009G 46.19 0.87  40
115   hdd 7.27730  1.0 7451G 3443G 4008G 46.21 0.87  33
41   hdd 7.27730  1.0 7451G 3444G 4007G 46.22 0.87  37
49   hdd 7.27730  1.0 7451G 3776G 3674G 50.68 0.95  34
71   hdd 7.27730  1.0 7451G 3776G 3675G 50.68 0.95  36
97   hdd 7.27730  1.0 7451G 3776G 3675G 50.68 0.95  26
17   hdd 7.27730  1.0 7451G 3777G 3674G 50.70 0.95  35
75   hdd 7.27730  1.0 7451G 3778G 3673G 50.70 0.95  41
  1   hdd 7.27730  1.0 7451G 3779G 3672G 50.71 0.95  40
79   hdd 7.27730  1.0 7451G 3778G 3672G 50.71 0.95  42
54   hdd 7.27730  1.0 7451G 3779G 3672G 50.72 0.95  39
58   hdd 7.27730  1.0 7451G 3780G 3670G 50.74 0.95  41
  7   hdd 7.27730  1.0 7451G 3781G 3670G 50.74 0.95  40
21   hdd 7.27730  1.0 7451G 3783G 3668G 50.77 0.95  27
31   hdd 7.27730  1.0 7451G 3783G 3668G 50.77 0.95  34
67   hdd 7.27730  1.0 7451G 3784G 3667G 50.79 0.95  33
43   hdd 7.27730  1.0 7451G 4119G 3332G 55.28 1.04  36
72   hdd 7.27730  1.0 7451G 4120G 3331G 55.30 1.04  45
74   hdd 7.27730  1.0 7451G 4121G 3330G 55.31 1.04  32
102   hdd 7

[ceph-users] Help rebalancing OSD usage, Luminus 1.2.2

[Bryan Banister]

Hi all,

We are still very new to running a Ceph cluster and have run a RGW cluster for 
a while now (6-ish mo), it mainly holds large DB backups (Write once, read 
once, delete after N days).  The system is now warning us about an OSD that is 
near_full and so we went to look at the usage across OSDs.  We are somewhat 
surprised at how imbalanced the usage is across the OSDs, with the lowest usage 
at 22% full, the highest at nearly 90%, and an almost linear usage pattern 
across the OSDs (though it looks to step in roughly 5% increments):

[root@carf-ceph-osd01 ~]# ceph osd df | sort -nk8
ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
77   hdd 7.27730  1.0 7451G 1718G 5733G 23.06 0.43  32
73   hdd 7.27730  1.0 7451G 1719G 5732G 23.08 0.43  31
  3   hdd 7.27730  1.0 7451G 2059G 5392G 27.63 0.52  27
46   hdd 7.27730  1.0 7451G 2060G 5391G 27.65 0.52  32
48   hdd 7.27730  1.0 7451G 2061G 5390G 27.66 0.52  25
127   hdd 7.27730  1.0 7451G 2066G 5385G 27.73 0.52  31
42   hdd 7.27730  1.0 7451G 2067G 5384G 27.74 0.52  42
107   hdd 7.27730  1.0 7451G 2402G 5049G 32.24 0.61  34
56   hdd 7.27730  1.0 7451G 2405G 5046G 32.28 0.61  37
51   hdd 7.27730  1.0 7451G 2406G 5045G 32.29 0.61  30
106   hdd 7.27730  1.0 7451G 2408G 5043G 32.31 0.61  29
81   hdd 7.27730  1.0 7451G 2408G 5043G 32.32 0.61  25
123   hdd 7.27730  1.0 7451G 2411G 5040G 32.37 0.61  35
47   hdd 7.27730  1.0 7451G 2412G 5039G 32.37 0.61  29
122   hdd 7.27730  1.0 7451G 2749G 4702G 36.90 0.69  30
84   hdd 7.27730  1.0 7451G 2750G 4701G 36.91 0.69  35
114   hdd 7.27730  1.0 7451G 2751G 4700G 36.92 0.69  26
82   hdd 7.27730  1.0 7451G 2751G 4700G 36.92 0.69  43
103   hdd 7.27730  1.0 7451G 2753G 4698G 36.94 0.69  39
36   hdd 7.27730  1.0 7451G 2752G 4699G 36.94 0.69  37
105   hdd 7.27730  1.0 7451G 2754G 4697G 36.97 0.69  26
14   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  31
  2   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  43
  8   hdd 7.27730  1.0 7451G 3091G 4360G 41.49 0.78  37
20   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  28
60   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  29
69   hdd 7.27730  1.0 7451G 3092G 4359G 41.50 0.78  37
110   hdd 7.27730  1.0 7451G 3093G 4358G 41.51 0.78  38
68   hdd 7.27730  1.0 7451G 3092G 4358G 41.51 0.78  34
76   hdd 7.27730  1.0 7451G 3093G 4358G 41.51 0.78  28
99   hdd 7.27730  1.0 7451G 3092G 4358G 41.51 0.78  34
50   hdd 7.27730  1.0 7451G 3095G 4356G 41.54 0.78  35
95   hdd 7.27730  1.0 7451G 3095G 4356G 41.54 0.78  31
  0   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  36
125   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  34
128   hdd 7.27730  1.0 7451G 3095G 4355G 41.55 0.78  37
94   hdd 7.27730  1.0 7451G 3096G 4355G 41.55 0.78  33
63   hdd 7.27730  1.0 7451G 3096G 4355G 41.56 0.78  41
30   hdd 7.27730  1.0 7451G 3100G 4351G 41.60 0.78  31
26   hdd 7.27730  1.0 7451G 3435G 4015G 46.11 0.87  30
64   hdd 7.27730  1.0 7451G 3435G 4016G 46.11 0.87  42
57   hdd 7.27730  1.0 7451G 3437G 4014G 46.12 0.87  29
33   hdd 7.27730  1.0 7451G 3437G 4014G 46.13 0.87  27
65   hdd 7.27730  1.0 7451G 3439G 4012G 46.15 0.87  29
109   hdd 7.27730  1.0 7451G 3439G 4012G 46.16 0.87  39
11   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  32
121   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  46
78   hdd 7.27730  1.0 7451G 3441G 4010G 46.18 0.87  36
13   hdd 7.27730  1.0 7451G 3442G 4009G 46.19 0.87  40
115   hdd 7.27730  1.0 7451G 3443G 4008G 46.21 0.87  33
41   hdd 7.27730  1.0 7451G 3444G 4007G 46.22 0.87  37
49   hdd 7.27730  1.0 7451G 3776G 3674G 50.68 0.95  34
71   hdd 7.27730  1.0 7451G 3776G 3675G 50.68 0.95  36
97   hdd 7.27730  1.0 7451G 3776G 3675G 50.68 0.95  26
17   hdd 7.27730  1.0 7451G 3777G 3674G 50.70 0.95  35
75   hdd 7.27730  1.0 7451G 3778G 3673G 50.70 0.95  41
  1   hdd 7.27730  1.0 7451G 3779G 3672G 50.71 0.95  40
79   hdd 7.27730  1.0 7451G 3778G 3672G 50.71 0.95  42
54   hdd 7.27730  1.0 7451G 3779G 3672G 50.72 0.95  39
58   hdd 7.27730  1.0 7451G 3780G 3670G 50.74 0.95  41
  7   hdd 7.27730  1.0 7451G 3781G 3670G 50.74 0.95  40
21   hdd 7.27730  1.0 7451G 3783G 3668G 50.77 0.95  27
31   hdd 7.27730  1.0 7451G 3783G 3668G 50.77 0.95  34
67   hdd 7.27730  1.0 7451G 3784G 3667G 50.79 0.95  33
43   hdd 7.27730  1.0 7451G 4119G 3332G 55.28 1.04  36
72   hdd 7.27730  1.0 7451G 4120G 3331G 55.30 1.04  45
74   hdd 7.27730  1.0 7451G 4121G 3330G 55.31 1.04  32
102   hdd 7.27730  1.0 7451G 4123G 3328G 55.33 1.04  35
34   hdd 7.27730  1.0 7451G 4123G 3328G 55.33 1.04  37
111   hdd 7.27730  1.0 7451G 4123G 3327G 55.34 1.04  40
44   hdd 7.27730  1.0 7451G 4123G 3328G 55.34 1.04  41
27   hdd 7.27730  1.0 7451G 4124G 3327G 55.35 1.04  44
39   hdd 7.27730  1.0 7451G 4124G 3327G 55.35 1.04  36
55   hdd 

[ceph-users] Simple RGW Lifecycle processing questions (luminous 12.2.2)

[Bryan Banister]

Hi All,

Hey all,

How often does the "lc process" run on RGW buckets in a cluster?

Also is it configurable per bucket or anything?

Tried searching man pages and ceph docs with no luck, so any help appreciated!

Thanks!
-Bryan





Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any way to get around selinux-policy-base dependency

[Bryan Banister]

Thanks Ken, that's understandable,
-Bryan

-Original Message-
From: Ken Dreyer [mailto:kdre...@redhat.com]
Sent: Wednesday, December 06, 2017 12:03 PM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>; Rafael Suarez 
<rsua...@jumptrading.com>
Subject: Re: [ceph-users] Any way to get around selinux-policy-base dependency

Note: External Email
-

Hi Bryan,

Why not upgrade to RHEL 7.4? We don't really build Ceph to run on
older RHEL releases.

- Ken

On Mon, Dec 4, 2017 at 11:26 AM, Bryan Banister
<bbanis...@jumptrading.com> wrote:
> Hi all,
>
>
>
> I would like to upgrade to the latest Luminous release but found that it
> requires the absolute latest selinux-policy-base.  We aren’t using selinux,
> so was wondering if there is a way around this dependency requirement?
>
>
>
> [carf-ceph-osd15][WARNIN] Error: Package: 2:ceph-selinux-12.2.2-0.el7.x86_64
> (ceph)
>
> [carf-ceph-osd15][WARNIN]Requires: selinux-policy-base >=
> 3.13.1-166.el7_4.5
>
> [carf-ceph-osd15][WARNIN]Installed:
> selinux-policy-targeted-3.13.1-102.el7_3.13.noarch
> (@rhel7.3-rhn-server-production/7.3)
>
>
>
> Thanks for any help!
>
> -Bryan
>
>
> 
>
> Note: This email is for the confidential use of the named addressee(s) only
> and may contain proprietary, confidential or privileged information. If you
> are not the intended recipient, you are hereby notified that any review,
> dissemination or copying of this email is strictly prohibited, and to please
> notify the sender immediately and destroy this email and any attachments.
> Email transmission cannot be guaranteed to be secure or error-free. The
> Company, therefore, does not make any guarantees as to the completeness or
> accuracy of this email or any attachments. This email is for informational
> purposes only and does not constitute a recommendation, offer, request or
> solicitation of any kind to buy, sell, subscribe, redeem or perform any type
> of transaction of a financial product.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Any way to get around selinux-policy-base dependency

[Bryan Banister]

Hi all,

I would like to upgrade to the latest Luminous release but found that it 
requires the absolute latest selinux-policy-base.  We aren't using selinux, so 
was wondering if there is a way around this dependency requirement?

[carf-ceph-osd15][WARNIN] Error: Package: 2:ceph-selinux-12.2.2-0.el7.x86_64 
(ceph)
[carf-ceph-osd15][WARNIN]Requires: selinux-policy-base >= 
3.13.1-166.el7_4.5
[carf-ceph-osd15][WARNIN]Installed: 
selinux-policy-targeted-3.13.1-102.el7_3.13.noarch 
(@rhel7.3-rhn-server-production/7.3)

Thanks for any help!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Help with full osd and RGW not responsive

[Bryan Banister]

Hi all,

Still a real novice here and we didn't set up our initial RGW cluster very 
well.  We have 134 osds and set up our RGW pool with only 64 PGs, thus not all 
of our OSDs got data and now we have one that is 95% full.

This apparently has put the cluster into a HEALTH_ERR condition:
[root@carf-ceph-osd01 ~]# ceph health detail
HEALTH_ERR full flag(s) set; 1 full osd(s); 1 pools have many more objects per 
pg than average; application not enabled on 6 pool(s); too few PGs per OSD (26 
< min 30)
OSDMAP_FLAGS full flag(s) set
OSD_FULL 1 full osd(s)
osd.5 is full
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
pool carf01.rgw.buckets.data objects per pg (602762) is more than 18.3752 
times cluster average (32803)

There is plenty of space on most of the OSDs and don't know how to go about 
fixing this situation.  If we update the pg_num and pgp_num settings for this 
pool, can we rebalance the data across the OSDs?

Also, seems like this is causing a problem with the RGWs, which was reporting 
this error in the logs:
2017-10-16 16:36:47.534461 7fffe6c5c700  1 heartbeat_map is_healthy 
'RGWAsyncRadosProcessor::m_tp thread 0x7fffdc447700' had timed out after 600

After trying to restart the RGW, we see this now:
2017-10-17 10:40:38.517002 7fffe6c5c700  1 heartbeat_map is_healthy 
'RGWAsyncRadosProcessor::m_tp thread 0x7fffddc4a700' had timed out after 600
2017-10-17 10:40:42.124046 77fd4e00  0 deferred set uid:gid to 167:167 
(ceph:ceph)
2017-10-17 10:40:42.124162 77fd4e00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 65313
2017-10-17 10:40:42.245259 77fd4e00  0 client.769905.objecter  FULL, paused 
modify 0x5662fb00 tid 0
2017-10-17 10:45:42.124283 7fffe7bcf700 -1 Initialization timeout, failed to 
initialize
2017-10-17 10:45:42.353496 77fd4e00  0 deferred set uid:gid to 167:167 
(ceph:ceph)
2017-10-17 10:45:42.353618 77fd4e00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 71842
2017-10-17 10:45:42.388621 77fd4e00  0 client.769986.objecter  FULL, paused 
modify 0x5662fb00 tid 0
2017-10-17 10:50:42.353731 7fffe7bcf700 -1 Initialization timeout, failed to 
initialize

Seems pretty evident that the "FULL, paused" is a problem.  So if I fix the 
first issue the RGW should be ok after?

Thanks in advance,
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph mgr dashboard, no socket could be created

[Bryan Banister]

Hi John,



Yes, it was working for some time and then I tried updating the run_dir on the 
cluster for another reason so I had to restart the cluster.  Now I get the 
issue with the socket creation.



I tried reverting the run_dir configuration to default and restarted but the 
issue persists.



I looked for anything listening on that port but didn't find anything:

[root@carf-ceph-osd15 ~]# netstat -a | grep -i 7000

tcp0  0 carf-ceph-osd15.w:37000 carf-ceph-osd17.w2:6894 ESTABLISHED

tcp0  0 carf-ceph-osd15.w2:6960 carf-ceph-osd17.w:37000 ESTABLISHED

tcp0  0 carf-ceph-osd15.w2:6863 carf-ceph-osd16.w:37000 ESTABLISHED

tcp0  0 carf-ceph-osd15.w2:6855 carf-ceph-osd17.w:57000 ESTABLISHED



And lsof didn't return anything either.



Guess I could try a reboot... but would prefer to figure it out,

-Bryan



-Original Message-
From: John Spray [mailto:jsp...@redhat.com]
Sent: Friday, September 22, 2017 2:32 AM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph mgr dashboard, no socket could be created



Note: External Email

-



On Thu, Sep 21, 2017 at 3:29 PM, Bryan Banister

<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> wrote:

> I’m not sure what happened but the dashboard module can no longer startup

> now:



I'm curious about the "no longer" part -- from the log it looks like

you only just enabled the dashboard module ("module list changed...")?

Was it definitely working before?



>

>

> 2017-09-21 09:28:34.646369 7fffef2e6700 -1 mgr got_mgr_map mgrmap module

> list changed to (dashboard), respawn

>

> 2017-09-21 09:28:34.646372 7fffef2e6700  1 mgr respawn  e:

> '/usr/bin/ceph-mgr'

>

> 2017-09-21 09:28:34.646374 7fffef2e6700  1 mgr respawn  0:

> '/usr/bin/ceph-mgr'

>

> 2017-09-21 09:28:34.646375 7fffef2e6700  1 mgr respawn  1: '-f'

>

> 2017-09-21 09:28:34.646376 7fffef2e6700  1 mgr respawn  2: '--cluster'

>

> 2017-09-21 09:28:34.646376 7fffef2e6700  1 mgr respawn  3: 'ceph'

>

> 2017-09-21 09:28:34.646377 7fffef2e6700  1 mgr respawn  4: '--id'

>

> 2017-09-21 09:28:34.646378 7fffef2e6700  1 mgr respawn  5: 'carf-ceph-osd15'

>

> 2017-09-21 09:28:34.646379 7fffef2e6700  1 mgr respawn  6: '--setuser'

>

> 2017-09-21 09:28:34.646379 7fffef2e6700  1 mgr respawn  7: 'ceph'

>

> 2017-09-21 09:28:34.646380 7fffef2e6700  1 mgr respawn  8: '--setgroup'

>

> 2017-09-21 09:28:34.646380 7fffef2e6700  1 mgr respawn  9: 'ceph'

>

> 2017-09-21 09:28:34.646398 7fffef2e6700  1 mgr respawn respawning with exe

> /usr/bin/ceph-mgr

>

> 2017-09-21 09:28:34.646399 7fffef2e6700  1 mgr respawn  exe_path

> /proc/self/exe

>

> 2017-09-21 09:28:34.670145 77fdd480  0 ceph version 12.1.4

> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown),

> pid 11714

>

> 2017-09-21 09:28:34.670163 77fdd480  0 pidfile_write: ignore empty

> --pid-file

>

> 2017-09-21 09:28:34.689929 77fdd480  1 mgr send_beacon standby

>

> 2017-09-21 09:28:35.647600 7fffef2e6700  1 mgr handle_mgr_map Activating!

>

> 2017-09-21 09:28:35.647762 7fffef2e6700  1 mgr handle_mgr_map I am now

> activating

>

> 2017-09-21 09:28:35.684790 7fffeb2de700  1 mgr init Loading python module

> 'dashboard'

>

> 2017-09-21 09:28:35.774439 7fffeb2de700  1 mgr load Constructed class from

> module: dashboard

>

> 2017-09-21 09:28:35.774448 7fffeb2de700  1 mgr start Creating threads for 1

> modules

>

> 2017-09-21 09:28:35.774489 7fffeb2de700  1 mgr send_beacon active

>

> 2017-09-21 09:28:35.880124 7fffd3742700 -1 mgr serve dashboard.serve:

>

> 2017-09-21 09:28:35.880132 7fffd3742700 -1 mgr serve Traceback (most recent

> call last):

>

>   File "/usr/lib64/ceph/mgr/dashboard/module.py", line 989, in serve

>

> cherrypy.engine.start()

>

>   File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line

> 250, in start

>

> raise e_info

>

> ChannelFailures: error('No socket could be created',)

>

>

>

> Any ideas what causes this?



I would look for other processes already listening on the same port

that dashboard is configured to listen on (7000 is the default), then

check for any security configuration changes (e.g. related to selinux)

that might be preventing it listening.



John



>

>

>

> Thanks,

>

> -Bryan

>

>

> 

>

> Note: This email is for the confidential use of the named addressee(s) only

> and may contain proprietary, confidential or privileged information. If you

> are not the inte

[ceph-users] Ceph mgr dashboard, no socket could be created

[Bryan Banister]

I'm not sure what happened but the dashboard module can no longer startup now:

2017-09-21 09:28:34.646369 7fffef2e6700 -1 mgr got_mgr_map mgrmap module list 
changed to (dashboard), respawn
2017-09-21 09:28:34.646372 7fffef2e6700  1 mgr respawn  e: '/usr/bin/ceph-mgr'
2017-09-21 09:28:34.646374 7fffef2e6700  1 mgr respawn  0: '/usr/bin/ceph-mgr'
2017-09-21 09:28:34.646375 7fffef2e6700  1 mgr respawn  1: '-f'
2017-09-21 09:28:34.646376 7fffef2e6700  1 mgr respawn  2: '--cluster'
2017-09-21 09:28:34.646376 7fffef2e6700  1 mgr respawn  3: 'ceph'
2017-09-21 09:28:34.646377 7fffef2e6700  1 mgr respawn  4: '--id'
2017-09-21 09:28:34.646378 7fffef2e6700  1 mgr respawn  5: 'carf-ceph-osd15'
2017-09-21 09:28:34.646379 7fffef2e6700  1 mgr respawn  6: '--setuser'
2017-09-21 09:28:34.646379 7fffef2e6700  1 mgr respawn  7: 'ceph'
2017-09-21 09:28:34.646380 7fffef2e6700  1 mgr respawn  8: '--setgroup'
2017-09-21 09:28:34.646380 7fffef2e6700  1 mgr respawn  9: 'ceph'
2017-09-21 09:28:34.646398 7fffef2e6700  1 mgr respawn respawning with exe 
/usr/bin/ceph-mgr
2017-09-21 09:28:34.646399 7fffef2e6700  1 mgr respawn  exe_path /proc/self/exe
2017-09-21 09:28:34.670145 77fdd480  0 ceph version 12.1.4 
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), 
pid 11714
2017-09-21 09:28:34.670163 77fdd480  0 pidfile_write: ignore empty 
--pid-file
2017-09-21 09:28:34.689929 77fdd480  1 mgr send_beacon standby
2017-09-21 09:28:35.647600 7fffef2e6700  1 mgr handle_mgr_map Activating!
2017-09-21 09:28:35.647762 7fffef2e6700  1 mgr handle_mgr_map I am now 
activating
2017-09-21 09:28:35.684790 7fffeb2de700  1 mgr init Loading python module 
'dashboard'
2017-09-21 09:28:35.774439 7fffeb2de700  1 mgr load Constructed class from 
module: dashboard
2017-09-21 09:28:35.774448 7fffeb2de700  1 mgr start Creating threads for 1 
modules
2017-09-21 09:28:35.774489 7fffeb2de700  1 mgr send_beacon active
2017-09-21 09:28:35.880124 7fffd3742700 -1 mgr serve dashboard.serve:
2017-09-21 09:28:35.880132 7fffd3742700 -1 mgr serve Traceback (most recent 
call last):
  File "/usr/lib64/ceph/mgr/dashboard/module.py", line 989, in serve
cherrypy.engine.start()
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 250, 
in start
raise e_info
ChannelFailures: error('No socket could be created',)

Any ideas what causes this?

Thanks,
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Possible to change the location of run_dir?

[Bryan Banister]

But the socket files are not group writeable:

srwxr-xr-x 1 ceph ceph 0 Aug 21 15:58 
/var/run/ceph/ceph-mgr.carf-ceph-osd16.asok

So users in that group can’t write to that admin socket, which is required for 
running `ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok perf dump` (what 
telegraf is trying to run)?

Thanks,
-Bryan

From: David Turner [mailto:drakonst...@gmail.com]
Sent: Wednesday, September 20, 2017 1:34 PM
To: Bryan Banister <bbanis...@jumptrading.com>; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Possible to change the location of run_dir?

Note: External Email

You can always add the telegraf user to the ceph group.  That change will 
persist on reboots and allow the user running the commands to read any 
folder/file that is owned by the group ceph.  I do this for Zabbix and Nagios 
now that the /var/lib/ceph folder is not public readable.

On Wed, Sep 20, 2017 at 2:27 PM Bryan Banister 
<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> wrote:
We are running telegraf and would like to have the telegraf user read the admin 
sockets from ceph, which is required for the ceph telegraf plugin to apply the 
ceph  related tags to the data.  The ceph admin sockets are by default stored 
in /var/run/ceph, but this is recreated at boot time, so we can’t set 
permissions on these sockets which will persist.

We would like to change the run_dir for ceph to be a persistent directory.  Is 
there a way to do this?

Would be nice if there was a [global] config option or something we could put 
in the /etc/sysconfig/ceph file.

Thanks,
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Possible to change the location of run_dir?

[Bryan Banister]

We are running telegraf and would like to have the telegraf user read the admin 
sockets from ceph, which is required for the ceph telegraf plugin to apply the 
ceph  related tags to the data.  The ceph admin sockets are by default stored 
in /var/run/ceph, but this is recreated at boot time, so we can't set 
permissions on these sockets which will persist.

We would like to change the run_dir for ceph to be a persistent directory.  Is 
there a way to do this?

Would be nice if there was a [global] config option or something we could put 
in the /etc/sysconfig/ceph file.

Thanks,
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Bryan Banister]

Very new to Ceph but long time but long time sys admin who is jaded/opinionated.

My 2 cents:
1) This sounds like a perfect thing to put in a poll and ask/beg people to 
vote.  Hopefully that will get you more of a response from a larger number of 
users.
2) Given that the value of the odd releases is "dubious", maybe those that are 
using these releases can give reason why they feel they need/want them?

I, personally, think having the even releases on a shorter cadence with the 
most users on each would be best, but I'm still new to this game,
-Bryan

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sage 
Weil
Sent: Wednesday, September 06, 2017 10:24 AM
To: ceph-de...@vger.kernel.org; ceph-maintain...@ceph.com; ceph-us...@ceph.com
Subject: [ceph-users] Ceph release cadence

Note: External Email
-

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year,
and every other such release has been an "LTS" release, with fixes
backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of
releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis,
kraken).  This limits the value of actually making them.  It also means
that those who *do* run them are running riskier code (fewer users -> more
bugs).

- The more recent requirement that upgrading clusters must make a stop at
each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
-> lumninous) has been hugely helpful on the development side by reducing
the amount of cross-version compatibility code to maintain and reducing
the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always
seems to be some "must-have" thing that delays the release a bit.  This
doesn't happen as much with the odd releases, but it definitely happens
with the LTS releases.  When the next LTS is a year away, it is hard to
suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular
cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

  + eliminate the confusing odd releases with dubious value

* Drop the odd releases, and aim for a ~9 month cadence. This splits the
difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

[Bryan Banister]

Looking at the systemd service it does show that twice, at roughly the same 
time and one day apart, the service did receive a HUP signal:

Aug 29 16:31:02 carf-ceph-osd02 radosgw[130050]: 2017-08-29 16:31:02.528559 
7fffc641c700 -1 received  signal: Hangup from  PID: 73176 task name: killall -q 
-1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0

Aug 30 16:32:02 carf-ceph-osd02 radosgw[130050]: 2017-08-30 16:32:02.529825 
7fffc641c700 -1 received  signal: Hangup from  PID: 48062 task name: killall -q 
-1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0



Any idea what would do this?



I'll be updating the version to 12.2.0 shortly,

-Bryan



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Wednesday, August 30, 2017 3:42 PM
To: Yehuda Sadeh-Weinraub <yeh...@redhat.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4



Note: External Email

-



We are not sending a HUP signal that we know about.  We were not modifying our 
configuration.  However all user accounts in the RGW were lost!

-Bryan



-Original Message-

From: Yehuda Sadeh-Weinraub [mailto:yeh...@redhat.com]

Sent: Wednesday, August 30, 2017 3:30 PM

To: Bryan Banister <bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>>

Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>

Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4



Note: External Email

-



On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister

<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> wrote:

> Not sure what’s happening but we started to but a decent load on the RGWs we

> have setup and we were seeing failures with the following kind of

> fingerprint:

>

>

>

> 2017-08-29 17:06:22.072361 7ffdc501a700  1 rgw realm reloader: Frontends

> paused

>



Are you modifying configuration? Could be that something is sending

HUP singal to the radosgw process. We disabled this behavior (process

dynamic reconfig after HUP) in 12.2.0.



Yehuda



> 2017-08-29 17:06:22.072359 7fffacbe9700  1 civetweb: 0x56add000:

> 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT

> /blah?partNumber=8=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0

> - Minio (linux; amd64) minio-go/3.0.0

>

> 2017-08-29 17:06:22.072438 7fffcb426700  0 ERROR: failed to clone shard,

> completion_mgr.get_next() returned ret=-125

>

> 2017-08-29 17:06:23.689610 7ffdc501a700  1 rgw realm reloader: Store closed

>

> 2017-08-29 17:06:24.117630 7ffdc501a700  1 failed to decode the mdlog

> history: buffer::end_of_buffer

>

> 2017-08-29 17:06:24.117635 7ffdc501a700  1 failed to read mdlog history: (5)

> Input/output error

>

> 2017-08-29 17:06:24.118711 7ffdc501a700  1 rgw realm reloader: Creating new

> store

>

> 2017-08-29 17:06:24.118901 7ffdc501a700  1 mgrc service_daemon_register

> rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4

> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)

> Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat

> Enterprise Linux Server 7.3

> (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80

> num_threads=1024,frontend_type#0=civetweb,hos

>

> tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT

> 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}

>

> 2017-08-29 17:06:24.118925 7ffdc501a700  1 rgw realm reloader: Finishing

> initialization of new store

>

> 2017-08-29 17:06:24.118927 7ffdc501a700  1 rgw realm reloader:  - REST

> subsystem init

>

> 2017-08-29 17:06:24.118943 7ffdc501a700  1 rgw realm reloader:  - user

> subsystem init

>

> 2017-08-29 17:06:24.118947 7ffdc501a700  1 rgw realm reloader:  - user

> subsystem init

>

> 2017-08-29 17:06:24.118950 7ffdc501a700  1 rgw realm reloader:  - usage

> subsystem init

>

> 2017-08-29 17:06:24.118985 7ffdc501a700  1 rgw realm reloader: Resuming

> frontends with new realm configuration.

>

> 2017-08-29 17:06:24.119018 7fffad3ea700  1 == starting new request

> req=0x7fffad3e4190 =

>

> 2017-08-29 17:06:24.119039 7fffacbe9700  1 == starting new request

> req=0x7fffacbe3190 =

>

> 2017-08-29 17:06:24.120163 7fffacbe9700  1 == req done

> req=0x7fffacbe3190 op status=0 http_status=403 ==

>

> 2017-08-29 17:06:24.120200 7fffad3ea700  1 == req d

Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

[Bryan Banister]

We are not sending a HUP signal that we know about.  We were not modifying our 
configuration.  However all user accounts in the RGW were lost!
-Bryan

-Original Message-
From: Yehuda Sadeh-Weinraub [mailto:yeh...@redhat.com]
Sent: Wednesday, August 30, 2017 3:30 PM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

Note: External Email
-

On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister
<bbanis...@jumptrading.com> wrote:
> Not sure what’s happening but we started to but a decent load on the RGWs we
> have setup and we were seeing failures with the following kind of
> fingerprint:
>
>
>
> 2017-08-29 17:06:22.072361 7ffdc501a700  1 rgw realm reloader: Frontends
> paused
>

Are you modifying configuration? Could be that something is sending
HUP singal to the radosgw process. We disabled this behavior (process
dynamic reconfig after HUP) in 12.2.0.

Yehuda

> 2017-08-29 17:06:22.072359 7fffacbe9700  1 civetweb: 0x56add000:
> 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT
> /blah?partNumber=8=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0
> - Minio (linux; amd64) minio-go/3.0.0
>
> 2017-08-29 17:06:22.072438 7fffcb426700  0 ERROR: failed to clone shard,
> completion_mgr.get_next() returned ret=-125
>
> 2017-08-29 17:06:23.689610 7ffdc501a700  1 rgw realm reloader: Store closed
>
> 2017-08-29 17:06:24.117630 7ffdc501a700  1 failed to decode the mdlog
> history: buffer::end_of_buffer
>
> 2017-08-29 17:06:24.117635 7ffdc501a700  1 failed to read mdlog history: (5)
> Input/output error
>
> 2017-08-29 17:06:24.118711 7ffdc501a700  1 rgw realm reloader: Creating new
> store
>
> 2017-08-29 17:06:24.118901 7ffdc501a700  1 mgrc service_daemon_register
> rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4
> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)
> Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat
> Enterprise Linux Server 7.3
> (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80
> num_threads=1024,frontend_type#0=civetweb,hos
>
> tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT
> 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
>
> 2017-08-29 17:06:24.118925 7ffdc501a700  1 rgw realm reloader: Finishing
> initialization of new store
>
> 2017-08-29 17:06:24.118927 7ffdc501a700  1 rgw realm reloader:  - REST
> subsystem init
>
> 2017-08-29 17:06:24.118943 7ffdc501a700  1 rgw realm reloader:  - user
> subsystem init
>
> 2017-08-29 17:06:24.118947 7ffdc501a700  1 rgw realm reloader:  - user
> subsystem init
>
> 2017-08-29 17:06:24.118950 7ffdc501a700  1 rgw realm reloader:  - usage
> subsystem init
>
> 2017-08-29 17:06:24.118985 7ffdc501a700  1 rgw realm reloader: Resuming
> frontends with new realm configuration.
>
> 2017-08-29 17:06:24.119018 7fffad3ea700  1 == starting new request
> req=0x7fffad3e4190 =
>
> 2017-08-29 17:06:24.119039 7fffacbe9700  1 == starting new request
> req=0x7fffacbe3190 =
>
> 2017-08-29 17:06:24.120163 7fffacbe9700  1 == req done
> req=0x7fffacbe3190 op status=0 http_status=403 ==
>
> 2017-08-29 17:06:24.120200 7fffad3ea700  1 == req done
> req=0x7fffad3e4190 op status=0 http_status=403 ==
>
>
>
> Any help understanding how to fix this would be greatly appreciated!
>
> -Bryan
>
>
> 
>
> Note: This email is for the confidential use of the named addressee(s) only
> and may contain proprietary, confidential or privileged information. If you
> are not the intended recipient, you are hereby notified that any review,
> dissemination or copying of this email is strictly prohibited, and to please
> notify the sender immediately and destroy this email and any attachments.
> Email transmission cannot be guaranteed to be secure or error-free. The
> Company, therefore, does not make any guarantees as to the completeness or
> accuracy of this email or any attachments. This email is for informational
> purposes only and does not constitute a recommendation, offer, request or
> solicitation of any kind to buy, sell, subscribe, redeem or perform any type
> of transaction of a financial product.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

___

Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

[Bryan Banister]

Looks like my RGW users also got deleted/lost?!

[root@carf-ceph-osd01 ~]# radosgw-admin user list
[]

Yikes!!

Any thoughts?
-Bryan

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Wednesday, August 30, 2017 9:45 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

Note: External Email

Not sure what's happening but we started to but a decent load on the RGWs we 
have setup and we were seeing failures with the following kind of fingerprint:

2017-08-29 17:06:22.072361 7ffdc501a700  1 rgw realm reloader: Frontends paused
2017-08-29 17:06:22.072359 7fffacbe9700  1 civetweb: 0x56add000: 
7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT 
/blah?partNumber=8=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0 - 
Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700  0 ERROR: failed to clone shard, 
completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700  1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700  1 failed to decode the mdlog history: 
buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700  1 failed to read mdlog history: (5) 
Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700  1 rgw realm reloader: Creating new 
store
2017-08-29 17:06:24.118901 7ffdc501a700  1 mgrc service_daemon_register 
rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4 
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R) Xeon(R) 
CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat Enterprise 
Linux Server 7.3 (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80 
num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT 
2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700  1 rgw realm reloader: Finishing 
initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700  1 rgw realm reloader:  - REST 
subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700  1 rgw realm reloader:  - user 
subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700  1 rgw realm reloader:  - user 
subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700  1 rgw realm reloader:  - usage 
subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700  1 rgw realm reloader: Resuming 
frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700  1 == starting new request 
req=0x7fffad3e4190 =
2017-08-29 17:06:24.119039 7fffacbe9700  1 == starting new request 
req=0x7fffacbe3190 =
2017-08-29 17:06:24.120163 7fffacbe9700  1 == req done req=0x7fffacbe3190 
op status=0 http_status=403 ==
2017-08-29 17:06:24.120200 7fffad3ea700  1 == req done req=0x7fffad3e4190 
op status=0 http_status=403 ==

Any help understanding how to fix this would be greatly appreciated!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Repeated failures in RGW in Ceph 12.1.4

[Bryan Banister]

Not sure what's happening but we started to but a decent load on the RGWs we 
have setup and we were seeing failures with the following kind of fingerprint:

2017-08-29 17:06:22.072361 7ffdc501a700  1 rgw realm reloader: Frontends paused
2017-08-29 17:06:22.072359 7fffacbe9700  1 civetweb: 0x56add000: 
7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT 
/blah?partNumber=8=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0 - 
Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700  0 ERROR: failed to clone shard, 
completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700  1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700  1 failed to decode the mdlog history: 
buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700  1 failed to read mdlog history: (5) 
Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700  1 rgw realm reloader: Creating new 
store
2017-08-29 17:06:24.118901 7ffdc501a700  1 mgrc service_daemon_register 
rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4 
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R) Xeon(R) 
CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat Enterprise 
Linux Server 7.3 (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80 
num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT 
2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700  1 rgw realm reloader: Finishing 
initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700  1 rgw realm reloader:  - REST 
subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700  1 rgw realm reloader:  - user 
subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700  1 rgw realm reloader:  - user 
subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700  1 rgw realm reloader:  - usage 
subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700  1 rgw realm reloader: Resuming 
frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700  1 == starting new request 
req=0x7fffad3e4190 =
2017-08-29 17:06:24.119039 7fffacbe9700  1 == starting new request 
req=0x7fffacbe3190 =
2017-08-29 17:06:24.120163 7fffacbe9700  1 == req done req=0x7fffacbe3190 
op status=0 http_status=403 ==
2017-08-29 17:06:24.120200 7fffad3ea700  1 == req done req=0x7fffad3e4190 
op status=0 http_status=403 ==

Any help understanding how to fix this would be greatly appreciated!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help with down OSD with Ceph 12.1.4 on Bluestore back

[Bryan Banister]

Found some bad stuff in the messages file about SCSI block device fails... I 
think I found my smoking gun...
-B

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Tuesday, August 29, 2017 5:02 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Help with down OSD with Ceph 12.1.4 on Bluestore back

Note: External Email

Hi all,

Not sure what to do with this down OSD:

-2> 2017-08-29 16:55:34.588339 72d58700  1 -- 7.128.13.57:6979/18818 
--> 7.128.13.55:0/52877 -- osd_ping(ping_reply e935 stamp 2017-08-29 
16:55:34.587991) v4 -- 0x67397000 con 0
-1> 2017-08-29 16:55:34.588351 72557700  1 -- 7.128.13.57:6978/18818 
--> 7.128.13.55:0/52877 -- osd_ping(ping_reply e935 stamp 2017-08-29 
16:55:34.587991) v4 -- 0x67395000 con 0
 0> 2017-08-29 16:55:34.650061 7fffecd93700 -1 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.4/rpm/el7/BUILD/ceph-12.1.4/src/os/bluestore/KernelDevice.cc:
 In function 'void KernelDevice::_aio_thread()' thread 7fffecd93700 time 
2017-08-29 16:55:34.648642
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.4/rpm/el7/BUILD/ceph-12.1.4/src/os/bluestore/KernelDevice.cc:
 372: FAILED assert(r >= 0)

ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) 
[0x55fb4420]
2: (KernelDevice::_aio_thread()+0x4b5) [0x55f59ce5]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f5e3cd]
4: (()+0x7dc5) [0x75635dc5]
5: (clone()+0x6d) [0x7472a73d]
NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
[snip]

Any help with recovery would be greatly appreciated, thanks!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Help with down OSD with Ceph 12.1.4 on Bluestore back

[Bryan Banister]

Hi all,

Not sure what to do with this down OSD:

-2> 2017-08-29 16:55:34.588339 72d58700  1 -- 7.128.13.57:6979/18818 
--> 7.128.13.55:0/52877 -- osd_ping(ping_reply e935 stamp 2017-08-29 
16:55:34.587991) v4 -- 0x67397000 con 0
-1> 2017-08-29 16:55:34.588351 72557700  1 -- 7.128.13.57:6978/18818 
--> 7.128.13.55:0/52877 -- osd_ping(ping_reply e935 stamp 2017-08-29 
16:55:34.587991) v4 -- 0x67395000 con 0
 0> 2017-08-29 16:55:34.650061 7fffecd93700 -1 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.4/rpm/el7/BUILD/ceph-12.1.4/src/os/bluestore/KernelDevice.cc:
 In function 'void KernelDevice::_aio_thread()' thread 7fffecd93700 time 
2017-08-29 16:55:34.648642
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.4/rpm/el7/BUILD/ceph-12.1.4/src/os/bluestore/KernelDevice.cc:
 372: FAILED assert(r >= 0)

ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) 
[0x55fb4420]
2: (KernelDevice::_aio_thread()+0x4b5) [0x55f59ce5]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f5e3cd]
4: (()+0x7dc5) [0x75635dc5]
5: (clone()+0x6d) [0x7472a73d]
NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
[snip]

Any help with recovery would be greatly appreciated, thanks!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Anybody gotten boto3 and ceph RGW working?

[Bryan Banister]

That was the problem, thanks again,
-Bryan

From: Bryan Banister
Sent: Wednesday, August 23, 2017 9:06 AM
To: Bryan Banister <bbanis...@jumptrading.com>; Abhishek Lekshmanan 
<abhis...@suse.com>; ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Anybody gotten boto3 and ceph RGW working?

Looks like I found the problem:  
https://github.com/snowflakedb/snowflake-connector-python/issues/1

I’ll try the fixed version of botocore 1.4.87+,
-Bryan

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Wednesday, August 23, 2017 9:01 AM
To: Abhishek Lekshmanan <abhis...@suse.com<mailto:abhis...@suse.com>>; 
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Anybody gotten boto3 and ceph RGW working?

Note: External Email


Here is the error I get:



# python3 boto3_test.py

Traceback (most recent call last):

  File "boto3_test.py", line 15, in 

for bucket in s3.list_buckets():

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/client.py",
 line 251, in _api_call

return self._make_api_call(operation_name, kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/client.py",
 line 526, in _make_api_call

operation_model, request_dict)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 141, in make_request

return self._send_request(request_dict, operation_model)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 170, in _send_request

success_response, exception):

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 249, in _needs_retry

caught_exception=caught_exception, request_dict=request_dict)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/hooks.py",
 line 227, in emit

return self._emit(event_name, kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/hooks.py",
 line 210, in _emit

response = handler(**kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 183, in __call__

if self._checker(attempts, response, caught_exception):

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 251, in __call__

caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 269, in _should_retry

return self._checker(attempt_number, response, caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 317, in __call__

caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 223, in __call__

attempt_number, caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 359, in _check_caught_exception

raise caught_exception

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 204, in _get_response

proxies=self.proxies, timeout=self.timeout)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py",
 line 573, in send

r = adapter.send(request, **kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py",
 line 370, in send

timeout=timeout

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py",
 line 544, in urlopen

body=body, headers=headers)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py",
 line 349, in _make_request

conn.request(method, url, **httplib_request_kw)

  File "/jump/software/rhel7/Python-3.6.0/lib/python3.6/http/client.py", line 
1239, in request

self._send_request(method, url, body, headers, encode_chunked)

TypeError: _send_request() takes 5 positional arguments but 6 were given



Here is the simple code:

import boto3



access_key = ""

secret_key = ""

gateway = "http://carf-ceph-osd15;



s3 = boto3.client('s3', 'us-east-1',

aws_access_key_id=access_key,

aws_secret_acce

Re: [ceph-users] Anybody gotten boto3 and ceph RGW working?

[Bryan Banister]

Looks like I found the problem:  
https://github.com/snowflakedb/snowflake-connector-python/issues/1

I’ll try the fixed version of botocore 1.4.87+,
-Bryan

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Bryan 
Banister
Sent: Wednesday, August 23, 2017 9:01 AM
To: Abhishek Lekshmanan <abhis...@suse.com>; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Anybody gotten boto3 and ceph RGW working?

Note: External Email


Here is the error I get:



# python3 boto3_test.py

Traceback (most recent call last):

  File "boto3_test.py", line 15, in 

for bucket in s3.list_buckets():

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/client.py",
 line 251, in _api_call

return self._make_api_call(operation_name, kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/client.py",
 line 526, in _make_api_call

operation_model, request_dict)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 141, in make_request

return self._send_request(request_dict, operation_model)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 170, in _send_request

success_response, exception):

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 249, in _needs_retry

caught_exception=caught_exception, request_dict=request_dict)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/hooks.py",
 line 227, in emit

return self._emit(event_name, kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/hooks.py",
 line 210, in _emit

response = handler(**kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 183, in __call__

if self._checker(attempts, response, caught_exception):

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 251, in __call__

caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 269, in _should_retry

return self._checker(attempt_number, response, caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 317, in __call__

caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 223, in __call__

attempt_number, caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 359, in _check_caught_exception

raise caught_exception

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 204, in _get_response

proxies=self.proxies, timeout=self.timeout)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py",
 line 573, in send

r = adapter.send(request, **kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py",
 line 370, in send

timeout=timeout

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py",
 line 544, in urlopen

body=body, headers=headers)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py",
 line 349, in _make_request

conn.request(method, url, **httplib_request_kw)

  File "/jump/software/rhel7/Python-3.6.0/lib/python3.6/http/client.py", line 
1239, in request

self._send_request(method, url, body, headers, encode_chunked)

TypeError: _send_request() takes 5 positional arguments but 6 were given



Here is the simple code:

import boto3



access_key = ""

secret_key = ""

gateway = "http://carf-ceph-osd15;



s3 = boto3.client('s3', 'us-east-1',

aws_access_key_id=access_key,

aws_secret_access_key=secret_key,

endpoint_url=gateway,

use_ssl=False

   )

# config=boto3.session.Config(signature_version='s3v2')



for bucket in s3.list_buckets():

    for key in bucket.objects.all():

print(key.key)



Thanks in advance for any help!!

-Bryan



-Original Message-
From: Abhishek Lekshmanan [mailto:abhis...@suse.com]
Sent: Wed

Re: [ceph-users] Anybody gotten boto3 and ceph RGW working?

[Bryan Banister]

Here is the error I get:



# python3 boto3_test.py

Traceback (most recent call last):

  File "boto3_test.py", line 15, in 

for bucket in s3.list_buckets():

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/client.py",
 line 251, in _api_call

return self._make_api_call(operation_name, kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/client.py",
 line 526, in _make_api_call

operation_model, request_dict)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 141, in make_request

return self._send_request(request_dict, operation_model)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 170, in _send_request

success_response, exception):

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 249, in _needs_retry

caught_exception=caught_exception, request_dict=request_dict)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/hooks.py",
 line 227, in emit

return self._emit(event_name, kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/hooks.py",
 line 210, in _emit

response = handler(**kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 183, in __call__

if self._checker(attempts, response, caught_exception):

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 251, in __call__

caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 269, in _should_retry

return self._checker(attempt_number, response, caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 317, in __call__

caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 223, in __call__

attempt_number, caught_exception)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/retryhandler.py",
 line 359, in _check_caught_exception

raise caught_exception

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/endpoint.py",
 line 204, in _get_response

proxies=self.proxies, timeout=self.timeout)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py",
 line 573, in send

r = adapter.send(request, **kwargs)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py",
 line 370, in send

timeout=timeout

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py",
 line 544, in urlopen

body=body, headers=headers)

  File 
"/jump/software/rhel7/python36_botocore-1.4.85/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py",
 line 349, in _make_request

conn.request(method, url, **httplib_request_kw)

  File "/jump/software/rhel7/Python-3.6.0/lib/python3.6/http/client.py", line 
1239, in request

self._send_request(method, url, body, headers, encode_chunked)

TypeError: _send_request() takes 5 positional arguments but 6 were given



Here is the simple code:

import boto3



access_key = ""

secret_key = ""

gateway = "http://carf-ceph-osd15;



s3 = boto3.client('s3', 'us-east-1',

aws_access_key_id=access_key,

aws_secret_access_key=secret_key,

endpoint_url=gateway,

use_ssl=False

   )

# config=boto3.session.Config(signature_version='s3v2')



for bucket in s3.list_buckets():

for key in bucket.objects.all():

print(key.key)



Thanks in advance for any help!!

-Bryan



-Original Message-
From: Abhishek Lekshmanan [mailto:abhis...@suse.com]
Sent: Wednesday, August 23, 2017 4:07 AM
To: Bryan Banister <bbanis...@jumptrading.com>; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Anybody gotten boto3 and ceph RGW working?



Note: External Email

-



Bryan Banister <bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> 
writes:



> Hello,

>

> I have the boto python API working with our ceph cluster but haven't figured 
> out a way t

[ceph-users] Anybody gotten boto3 and ceph RGW working?

[Bryan Banister]

Hello,

I have the boto python API working with our ceph cluster but haven't figured 
out a way to get boto3 to communicate yet to our RGWs.  Anybody have a simple 
example?

Cheers for any help!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help with file system with failed mds daemon

[Bryan Banister]

All sounds right to me... looks like this is a little too bleeding edge for my 
taste!  I'll probably drop it at this point and just wait till we are actually 
on a 4.8 kernel before checking on status again.

Thanks for your help!
-Bryan

-Original Message-
From: John Spray [mailto:jsp...@redhat.com]
Sent: Tuesday, August 22, 2017 2:56 PM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Help with file system with failed mds daemon

Note: External Email
-

On Tue, Aug 22, 2017 at 8:49 PM, Bryan Banister
<bbanis...@jumptrading.com> wrote:
> Hi John,
>
>
>
> Seems like you're right... strange that it seemed to work with only one mds
> before I shut the cluster down.  Here is the `ceph fs get` output for the
> two file systems:
>
>
>
> [root@carf-ceph-osd15 ~]# ceph fs get carf_ceph_kube01
>
> Filesystem 'carf_ceph_kube01' (2)
>
> fs_name carf_ceph_kube01
>
> epoch   22
>
> flags   8
>
> created 2017-08-21 12:10:57.948579
>
> modified2017-08-21 12:10:57.948579
>
> tableserver 0
>
> root0
>
> session_timeout 60
>
> session_autoclose   300
>
> max_file_size   1099511627776
>
> last_failure0
>
> last_failure_osd_epoch  1218
>
> compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
> uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
>
> max_mds 1
>
> in  0
>
> up  {}
>
> failed  0
>
> damaged
>
> stopped
>
> data_pools  [23]
>
> metadata_pool   24
>
> inline_data disabled
>
> balancer
>
> standby_count_wanted0
>
> [root@carf-ceph-osd15 ~]#
>
> [root@carf-ceph-osd15 ~]# ceph fs get carf_ceph02
>
> Filesystem 'carf_ceph02' (1)
>
> fs_name carf_ceph02
>
> epoch   26
>
> flags   8
>
> created 2017-08-18 14:20:50.152054
>
> modified2017-08-18 14:20:50.152054
>
> tableserver 0
>
> root0
>
> session_timeout 60
>
> session_autoclose   300
>
> max_file_size   1099511627776
>
> last_failure0
>
> last_failure_osd_epoch  1198
>
> compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
> uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
>
> max_mds 1
>
> in  0
>
> up  {0=474299}
>
> failed
>
> damaged
>
> stopped
>
> data_pools  [21]
>
> metadata_pool   22
>
> inline_data disabled
>
> balancer
>
> standby_count_wanted0
>
> 474299: 7.128.13.69:6800/304042158 'carf-ceph-osd15' mds.0.23 up:active seq
> 5

In that instance, it's not complaining because one of the filesystems
has never had an MDS.


> I also looked into trying to specify the mds_namespace option to the mount
> operation (http://docs.ceph.com/docs/master/cephfs/kernel/) but that doesn’t
> seem to be valid:
>
> [ceph-admin@carf-ceph-osd04 ~]$ sudo mount -t ceph carf-ceph-osd15:6789:/
> /mnt/carf_ceph02/ -o
> mds_namespace=carf_ceph02,name=cephfs.k8test,secretfile=k8test.secret
>
> mount error 22 = Invalid argument

It's likely that you are using an older kernel that doesn't have
support for the feature.  It was added in linux 4.8.

John

>
>
> Thanks,
>
> -Bryan
>
>
>
> -Original Message-
> From: John Spray [mailto:jsp...@redhat.com]
> Sent: Tuesday, August 22, 2017 11:18 AM
> To: Bryan Banister <bbanis...@jumptrading.com>
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Help with file system with failed mds daemon
>
>
>
> Note: External Email
>
> -
>
>
>
> On Tue, Aug 22, 2017 at 4:58 PM, Bryan Banister
>
> <bbanis...@jumptrading.com> wrote:
>
>> Hi all,
>
>>
>
>>
>
>>
>
>> I’m still new to ceph and cephfs.  Trying out the multi-fs configuration
>> on
>
>> at Luminous test cluster.  I shutdown the cluster to do an upgrade and
>> when
>
>> I brought the cluster back up I now have a warnings that one of the file
>
>> systems has a failed mds daemon:
>
>>
>
>>
>
>>
>
>> 2017-08-21 17:00:00.81 mon.carf-ceph-osd15 [WRN] overall HEALTH_WARN 1
>
>> filesystem is degraded; 1 filesystem is have a failed mds daemon; 1 pools
>
>> have many more objects per pg than average; application not enabled on 9
>
>> pool(s)
>
>>
>
>>
>
>>
&g

Re: [ceph-users] Help with file system with failed mds daemon

[Bryan Banister]

Hi John,



Seems like you're right... strange that it seemed to work with only one mds 
before I shut the cluster down.  Here is the `ceph fs get` output for the two 
file systems:



[root@carf-ceph-osd15 ~]# ceph fs get carf_ceph_kube01

Filesystem 'carf_ceph_kube01' (2)

fs_name carf_ceph_kube01

epoch   22

flags   8

created 2017-08-21 12:10:57.948579

modified2017-08-21 12:10:57.948579

tableserver 0

root0

session_timeout 60

session_autoclose   300

max_file_size   1099511627776

last_failure0

last_failure_osd_epoch  1218

compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses 
versioned encoding,6=dirfrag is stored in omap,8=file layout v2}

max_mds 1

in  0

up  {}

failed  0

damaged

stopped

data_pools  [23]

metadata_pool   24

inline_data disabled

balancer

standby_count_wanted0

[root@carf-ceph-osd15 ~]#

[root@carf-ceph-osd15 ~]# ceph fs get carf_ceph02

Filesystem 'carf_ceph02' (1)

fs_name carf_ceph02

epoch   26

flags   8

created 2017-08-18 14:20:50.152054

modified2017-08-18 14:20:50.152054

tableserver 0

root0

session_timeout 60

session_autoclose   300

max_file_size   1099511627776

last_failure0

last_failure_osd_epoch  1198

compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses 
versioned encoding,6=dirfrag is stored in omap,8=file layout v2}

max_mds 1

in  0

up  {0=474299}

failed

damaged

stopped

data_pools  [21]

metadata_pool   22

inline_data disabled

balancer

standby_count_wanted0

474299: 7.128.13.69:6800/304042158 'carf-ceph-osd15' mds.0.23 up:active seq 5



I also looked into trying to specify the mds_namespace option to the mount 
operation (http://docs.ceph.com/docs/master/cephfs/kernel/) but that doesn’t 
seem to be valid:

[ceph-admin@carf-ceph-osd04 ~]$ sudo mount -t ceph carf-ceph-osd15:6789:/ 
/mnt/carf_ceph02/ -o 
mds_namespace=carf_ceph02,name=cephfs.k8test,secretfile=k8test.secret

mount error 22 = Invalid argument



Thanks,

-Bryan



-Original Message-
From: John Spray [mailto:jsp...@redhat.com]
Sent: Tuesday, August 22, 2017 11:18 AM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Help with file system with failed mds daemon



Note: External Email

-



On Tue, Aug 22, 2017 at 4:58 PM, Bryan Banister

<bbanis...@jumptrading.com<mailto:bbanis...@jumptrading.com>> wrote:

> Hi all,

>

>

>

> I’m still new to ceph and cephfs.  Trying out the multi-fs configuration on

> at Luminous test cluster.  I shutdown the cluster to do an upgrade and when

> I brought the cluster back up I now have a warnings that one of the file

> systems has a failed mds daemon:

>

>

>

> 2017-08-21 17:00:00.81 mon.carf-ceph-osd15 [WRN] overall HEALTH_WARN 1

> filesystem is degraded; 1 filesystem is have a failed mds daemon; 1 pools

> have many more objects per pg than average; application not enabled on 9

> pool(s)

>

>

>

> I tried restarting the mds service on the system and it doesn’t seem to

> indicate any problems:

>

> 2017-08-21 16:13:40.979449 7fffed8b0700  1 mds.0.20 shutdown: shutting down

> rank 0

>

> 2017-08-21 16:13:41.012167 77fde1c0  0 set uid:gid to 167:167

> (ceph:ceph)

>

> 2017-08-21 16:13:41.012180 77fde1c0  0 ceph version 12.1.4

> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown),

> pid 16656

>

> 2017-08-21 16:13:41.014105 77fde1c0  0 pidfile_write: ignore empty

> --pid-file

>

> 2017-08-21 16:13:45.541442 710b7700  1 mds.0.23 handle_mds_map i am now

> mds.0.23

>

> 2017-08-21 16:13:45.541449 710b7700  1 mds.0.23 handle_mds_map state

> change up:boot --> up:replay

>

> 2017-08-21 16:13:45.541459 710b7700  1 mds.0.23 replay_start

>

> 2017-08-21 16:13:45.541466 710b7700  1 mds.0.23  recovery set is

>

> 2017-08-21 16:13:45.541475 710b7700  1 mds.0.23  waiting for osdmap 1198

> (which blacklists prior instance)

>

> 2017-08-21 16:13:45.565779 7fffea8aa700  0 mds.0.cache creating system inode

> with ino:0x100

>

> 2017-08-21 16:13:45.565920 7fffea8aa700  0 mds.0.cache creating system inode

> with ino:0x1

>

> 2017-08-21 16:13:45.571747 7fffe98a8700  1 mds.0.23 replay_done

>

> 2017-08-21 16:13:45.571751 7fffe98a8700  1 mds.0.23 making mds journal

> writeable

>

> 2017-08-21 16:13:46.542148 710b7700  1 mds.0.23 handle_mds_map i am now

> mds.0.23

>

> 2017-08-21 16:13:46.542149 710b7700  1 mds.0.23 handle_mds_map state

> change up:replay --> 

[ceph-users] Help with file system with failed mds daemon

[Bryan Banister]

Hi all,

I'm still new to ceph and cephfs.  Trying out the multi-fs configuration on at 
Luminous test cluster.  I shutdown the cluster to do an upgrade and when I 
brought the cluster back up I now have a warnings that one of the file systems 
has a failed mds daemon:

2017-08-21 17:00:00.81 mon.carf-ceph-osd15 [WRN] overall HEALTH_WARN 1 
filesystem is degraded; 1 filesystem is have a failed mds daemon; 1 pools have 
many more objects per pg than average; application not enabled on 9 pool(s)

I tried restarting the mds service on the system and it doesn't seem to 
indicate any problems:
2017-08-21 16:13:40.979449 7fffed8b0700  1 mds.0.20 shutdown: shutting down 
rank 0
2017-08-21 16:13:41.012167 77fde1c0  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-21 16:13:41.012180 77fde1c0  0 ceph version 12.1.4 
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), 
pid 16656
2017-08-21 16:13:41.014105 77fde1c0  0 pidfile_write: ignore empty 
--pid-file
2017-08-21 16:13:45.541442 710b7700  1 mds.0.23 handle_mds_map i am now 
mds.0.23
2017-08-21 16:13:45.541449 710b7700  1 mds.0.23 handle_mds_map state change 
up:boot --> up:replay
2017-08-21 16:13:45.541459 710b7700  1 mds.0.23 replay_start
2017-08-21 16:13:45.541466 710b7700  1 mds.0.23  recovery set is
2017-08-21 16:13:45.541475 710b7700  1 mds.0.23  waiting for osdmap 1198 
(which blacklists prior instance)
2017-08-21 16:13:45.565779 7fffea8aa700  0 mds.0.cache creating system inode 
with ino:0x100
2017-08-21 16:13:45.565920 7fffea8aa700  0 mds.0.cache creating system inode 
with ino:0x1
2017-08-21 16:13:45.571747 7fffe98a8700  1 mds.0.23 replay_done
2017-08-21 16:13:45.571751 7fffe98a8700  1 mds.0.23 making mds journal writeable
2017-08-21 16:13:46.542148 710b7700  1 mds.0.23 handle_mds_map i am now 
mds.0.23
2017-08-21 16:13:46.542149 710b7700  1 mds.0.23 handle_mds_map state change 
up:replay --> up:reconnect
2017-08-21 16:13:46.542158 710b7700  1 mds.0.23 reconnect_start
2017-08-21 16:13:46.542161 710b7700  1 mds.0.23 reopen_log
2017-08-21 16:13:46.542171 710b7700  1 mds.0.23 reconnect_done
2017-08-21 16:13:47.543612 710b7700  1 mds.0.23 handle_mds_map i am now 
mds.0.23
2017-08-21 16:13:47.543616 710b7700  1 mds.0.23 handle_mds_map state change 
up:reconnect --> up:rejoin
2017-08-21 16:13:47.543623 710b7700  1 mds.0.23 rejoin_start
2017-08-21 16:13:47.543638 710b7700  1 mds.0.23 rejoin_joint_start
2017-08-21 16:13:47.543666 710b7700  1 mds.0.23 rejoin_done
2017-08-21 16:13:48.544768 710b7700  1 mds.0.23 handle_mds_map i am now 
mds.0.23
2017-08-21 16:13:48.544771 710b7700  1 mds.0.23 handle_mds_map state change 
up:rejoin --> up:active
2017-08-21 16:13:48.544779 710b7700  1 mds.0.23 recovery_done -- successful 
recovery!
2017-08-21 16:13:48.544924 710b7700  1 mds.0.23 active_start
2017-08-21 16:13:48.544954 710b7700  1 mds.0.23 cluster recovered.

This seems like an easy problem to fix.  Any help is greatly appreciated!
-Bryan



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any experience with multiple cephfs instances in one ceph cluster? How experimental is this?

[Bryan Banister]

Thanks for the response John.

Maybe I'm not understanding this correctly, but I thought clients could be 
restricted to specific file systems by limiting access to the underlying ceph 
pools used in each file system?

client.cephfs.test1
key: AQDuQpdZp90MHhAAkYE6P5XYzsoswgEkZy6RLw==
caps: [mds] allow
caps: [mon] allow r
caps: [osd] allow rw pool cephfs01_data

client.cephfs.test2
key: AQDuQpdZp90MHhAAkYE6P5XYzsoswgEkZy6RLw==
caps: [mds] allow
caps: [mon] allow r
caps: [osd] allow rw pool cephfs02_data

Would these two clients keys which only have access to specific data pools 
restrict their access?

Or I guess with mds allow on both, then they could mount the file system but 
only reading/writing the data in the file systems would be restricted?

Thanks!
-Bryan

-Original Message-
From: John Spray [mailto:jsp...@redhat.com]
Sent: Monday, August 21, 2017 8:48 AM
To: Bryan Banister <bbanis...@jumptrading.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Any experience with multiple cephfs instances in one 
ceph cluster? How experimental is this?

Note: External Email
-

On Mon, Aug 21, 2017 at 2:35 PM, Bryan Banister
<bbanis...@jumptrading.com> wrote:
> Hi all,
>
>
>
> I’m very new to ceph and cephfs, so I’m just starting to play around with
> the Luminous release.  There are some very concerning warnings about
> deploying multiple cephfs instances in the same cluster:
>
> “There are no known bugs, but any failures which do result from having
> multiple active filesystems in your cluster will require manual intervention
> and, so far, will not have been experienced by anybody else – knowledgeable
> help will be extremely limited. You also probably do not have the security
> or isolation guarantees you want or think you have upon doing so.”

The sort of literal translation of this is:
 - the automated tests for systems with multiple filesystems are not
very comprehensive
 - a client that can access one filesystem can access all of them

If you're adventurous enough to be running upstream Ceph packages, and
you have at least some level of test/staging environment to try it in,
then I'd not be too scared about trying it out.

> And Redhat says:
>
> “Creating multiple Ceph File Systems in one cluster is not fully supported
> yet and can cause the MDS or client nodes to terminate unexpectedly.”

I don't know who wrote that text, but I do not believe that there are
any known issues involving MDS or client nodes terminating
unexpectedly.

John

>
>
>
> Is anybody deploying multiple cephfs instances and have there been any
> issues like the warnings indicate?
>
>
>
> Thanks!
>
> -Bryan
>
>
>
>
>
>
> 
>
> Note: This email is for the confidential use of the named addressee(s) only
> and may contain proprietary, confidential or privileged information. If you
> are not the intended recipient, you are hereby notified that any review,
> dissemination or copying of this email is strictly prohibited, and to please
> notify the sender immediately and destroy this email and any attachments.
> Email transmission cannot be guaranteed to be secure or error-free. The
> Company, therefore, does not make any guarantees as to the completeness or
> accuracy of this email or any attachments. This email is for informational
> purposes only and does not constitute a recommendation, offer, request or
> solicitation of any kind to buy, sell, subscribe, redeem or perform any type
> of transaction of a financial product.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Any experience with multiple cephfs instances in one ceph cluster? How experimental is this?

[Bryan Banister]

Hi all,

I'm very new to ceph and cephfs, so I'm just starting to play around with the 
Luminous release.  There are some very concerning warnings about deploying 
multiple cephfs instances in the same cluster:
"There are no known bugs, but any failures which do result from having multiple 
active filesystems in your cluster will require manual intervention and, so 
far, will not have been experienced by anybody else - knowledgeable help will 
be extremely limited. You also probably do not have the security or isolation 
guarantees you want or think you have upon doing so."

And Redhat says:
"Creating multiple Ceph File Systems in one cluster is not fully supported yet 
and can cause the MDS or client nodes to terminate unexpectedly."

Is anybody deploying multiple cephfs instances and have there been any issues 
like the warnings indicate?

Thanks!
-Bryan





Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com