Re: [ceph-users] Sudden omap growth on some OSDs

Hi Greg, I have re-introduced the OSD that was taken out (the one that used to be a primary). I have kept debug 20 logs from both the re-introduced primary and the outgoing primary. I have used ceph-post-file to upload these, tag: 5b305f94-83e2-469c-a301-7299d2279d94 Hope this helps, let me

Re: [ceph-users] Sudden omap growth on some OSDs

From: Gregory Farnum Date: Tuesday, 12 December 2017 at 19:24 To: "Vasilakakos, George (STFC,RAL,SC)" Cc: "ceph-users@lists.ceph.com" Subject: Re: [ceph-users] Sudden omap growth on some OSDs On Tue, Dec 12, 2017 at

Re: [ceph-users] Sudden omap growth on some OSDs

On 11 Dec 2017, at 18:24, Gregory Farnum > wrote: Hmm, this does all sound odd. Have you tried just restarting the primary OSD yet? That frequently resolves transient oddities like this. If not, I'll go poke at the kraken source and one of the

Re: [ceph-users] Sudden omap growth on some OSDs

From: Gregory Farnum [gfar...@redhat.com] Sent: 07 December 2017 21:57 To: Vasilakakos, George (STFC,RAL,SC) Cc: drakonst...@gmail.com; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Sudden omap growth on some OSDs On Thu, Dec 7, 2017 at 4:41 AM

Re: [ceph-users] Sudden omap growth on some OSDs

From: Gregory Farnum [gfar...@redhat.com] Sent: 06 December 2017 22:50 To: David Turner Cc: Vasilakakos, George (STFC,RAL,SC); ceph-users@lists.ceph.com Subject: Re: [ceph-users] Sudden omap growth on some OSDs On Wed, Dec 6, 2017 at 2:35 PM David Turner

[ceph-users] Sudden omap growth on some OSDs

Hi ceph-users, We have a Ceph cluster (running Kraken) that is exhibiting some odd behaviour. A couple weeks ago, the LevelDBs on some our OSDs started growing large (now at around 20G size). The one thing they have in common is the 11 disks with inflating LevelDBs are all in the set for one

Re: [ceph-users] OSDs in EC pool flapping

No, nothing like that. The cluster is in the process of having more OSDs added and, while that was ongoing, one was removed because the underlying disk was throwing up a bunch of read errors. Shortly after, the first three OSDs in this PG started crashing with error messages about corrupted EC

[ceph-users] OSDs in EC pool flapping

Hey folks, I'm staring at a problem that I have found no solution for and which is causing major issues. We've had a PG go down with the first 3 OSDs all crashing and coming back only to crash again with the following error in their logs: -1> 2017-08-22 17:27:50.961633 7f4af4057700 -1

Re: [ceph-users] Speeding up backfill after increasing PGs and or adding OSDs

@Christian: I think the slow tail end is just the fact that there is contention for the same OSDs. @David: Yes that's what I did, used shell/awk/python to grab and compare the set of OSDs locked for backfilling versus the ones waiting. From: Christian

Re: [ceph-users] Speeding up backfill after increasing PGs and or adding OSDs

Thanks for your response David. What you've described has been what I've been thinking about too. We have 1401 OSDs in the cluster currently and this output is from the tail end of the backfill for +64 PG increase on the biggest pool. The problem is we see this cluster do at most 20 backfills

[ceph-users] Speeding up backfill after increasing PGs and or adding OSDs

Hey folks, We have a cluster that's currently backfilling from increasing PG counts. We have tuned recovery and backfill way down as a "precaution" and would like to start tuning it to bring up to a good balance between that and client I/O. At the moment we're in the process of bumping up PG

Re: [ceph-users] Ceph and IPv4 -> IPv6

Good to know. Frankly, the RGW isn’t my major concern at the moment, it seems to be able to handle things well enough. It’s the RBD/CephFS side of things for one cluster where we will eventually need to support IPv6 clients but will not necessarily be able to switch everyone to IPv6 in one go.

Re: [ceph-users] Ceph and IPv4 -> IPv6

> I don't think either. I don't think there is another way then just 'hacky' > changing the MONMaps. There have been talks of being able to make Ceph > dual-stack, but I don't think there is any code in the source right now. Yeah, that's what I'd like to know. What do the Ceph team think of

Re: [ceph-users] Ceph and IPv4 -> IPv6

Hey Wido, Thanks for your suggestion. It sounds like the process might be feasible but I'd be looking for an "official" thing to do to a production cluster. Something that's documented ceph.com/docs, tested and "endorsed" if you will by the Ceph team. We could try this on a pre-prod

Re: [ceph-users] Ceph and IPv4 -> IPv6

I don't think you can do that, it would require running a mixed cluster which, going by the docs, doesn't seem to be supported. From: Jake Young [jak3...@gmail.com] Sent: 27 June 2017 22:42 To: Wido den Hollander; ceph-users@lists.ceph.com; Vasilakakos, George

[ceph-users] Ceph and IPv4 -> IPv6

Hey Ceph folks, I was wondering what the current status/roadmap/intentions etc. are on the possibility of providing a way of transitioning a cluster from IPv4 to IPv6 in the future. My current understanding is that this not possible at the moment and that one should deploy initially with the

Re: [ceph-users] Large OSD omap directories (LevelDBs)

Hi Greg, > This does sound weird, but I also notice that in your earlier email you > seemed to have only ~5k PGs across ~1400 OSDs, which is a pretty > low number. You may just have a truly horrible PG balance; can you share > more details (eg ceph osd df)? Our distribution is pretty bad,

Re: [ceph-users] Large OSD omap directories (LevelDBs)

Hi Wido, I see your point. I would expect OMAPs to grow with the number of objects but multiple OSDs getting to multiple tens of GBs for their omaps seems excessive. I find it difficult to believe that not sharding the index for a bucket of 500k objects in RGW causes the 10 largest OSD omaps

Re: [ceph-users] Large OSD omap directories (LevelDBs)

> Your RGW buckets, how many objects in them, and do they have the index > sharded? > I know we have some very large & old buckets (10M+ RGW objects in a > single bucket), with correspondingly large OMAPs wherever that bucket > index is living (sufficently large that trying to list the entire

[ceph-users] Large OSD omap directories (LevelDBs)

Hello Ceph folk, We have a Ceph cluster (info at bottom) with some odd omap directory sizes in our OSDs. We're looking at 1439 OSDs were the most common omap sizes are 15-40MB. However a quick sampling reveals some outliers, looking at around the 100 largest omaps one can see sizes go to a few

Re: [ceph-users] PG stuck peering after host reboot

Since we need this pool to work again, we decided to take the data loss and try to move on. So far, no luck. We tried a force create but, as expected, with a PG that is not peering this did absolutely nothing. We also tried rm-past-intervals and remove from ceph-objectstore-tool and manually

Re: [ceph-users] PG stuck peering after host reboot

So what I see there is this for osd.307: "empty": 1, "dne": 0, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } } last_epoch_started is 0 and empty is 1. The other OSDs are reporting

Re: [ceph-users] PG stuck peering after host reboot

Brad Hubbard pointed out on the bug tracker (http://tracker.ceph.com/issues/18960) that, for EC, we need to add the shard suffix to the PGID parameter in the command, e.g. --pgid 1.323s0 The command now works and produces the same output as PG query. To avoid spamming the list I've put the

Re: [ceph-users] PG stuck peering after host reboot

I have noticed something odd with the ceph-objectstore-tool command: It always reports PG X not found even on healthly OSDs/PGs. The 'list' op works on both and unhealthy PGs. From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of

Re: [ceph-users] PG stuck peering after host reboot

> Can you for the sake of redundancy post your sequence of commands you > executed and their output? [root@ceph-sn852 ~]# systemctl stop ceph-osd@307 [root@ceph-sn852 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-307 --op info --pgid 1.323 PG '1.323' not found [root@ceph-sn852

Re: [ceph-users] PG stuck peering after host reboot

Hi Wido, Just to make sure I have everything straight, > If the PG still doesn't recover do the same on osd.307 as I think that 'ceph > pg X query' still hangs? > The info from ceph-objectstore-tool might shed some more light on this PG. You mean run the objectstore command on 307, not remove

Re: [ceph-users] PG stuck peering after host reboot

On 17/02/2017, 12:00, "Wido den Hollander" wrote: > >> Op 17 februari 2017 om 11:09 schreef george.vasilaka...@stfc.ac.uk: >> >> >> Hi Wido, >> >> In an effort to get the cluster to complete peering that PG (as we need to >> be able to use our pool) we have removed osd.595

Re: [ceph-users] PG stuck peering after host reboot

Hi Wido, In an effort to get the cluster to complete peering that PG (as we need to be able to use our pool) we have removed osd.595 from the CRUSH map to allow a new mapping to occur. When I left the office yesterday osd.307 had replaced osd.595 in the up set but the acting set had

Re: [ceph-users] PG stuck peering after host reboot

Hi folks, I have just made a tracker for this issue: http://tracker.ceph.com/issues/18960 I used ceph-post-file to upload some logs from the primary OSD for the troubled PG. Any help would be appreciated. If we can't get it to peer, we'd like to at least get it unstuck, even if it means data

Re: [ceph-users] PG stuck peering after host reboot

Hi Brad, I'll be doing so later in the day. Thanks, George From: Brad Hubbard [bhubb...@redhat.com] Sent: 13 February 2017 22:03 To: Vasilakakos, George (STFC,RAL,SC); Ceph Users Subject: Re: [ceph-users] PG stuck peering after host reboot I'd suggest

Re: [ceph-users] PG stuck peering after host reboot

OK, I've had a look. Haven't been able to take a proper look at the network yet but here's what I've gathered on other fronts so far: * Marking either osd.595 or osd.7 out results in this: $ ceph health detail | grep -v stuck | grep 1.323 pg 1.323 is remapped+peering, acting

Re: [ceph-users] PG stuck peering after host reboot

Hey Greg, Thanks for your quick responses. I have to leave the office now but I'll look deeper into it tomorrow to try and understand what's the cause of this. I'll try to find other peerings between these two hosts and check those OSDs' logs for potential anomalies. I'll also have a look at

Re: [ceph-users] PG stuck peering after host reboot

Hi Greg, > Yes, "bad crc" indicates that the checksums on an incoming message did > not match what was provided — ie, the message got corrupted. You > shouldn't try and fix that by playing around with the peering settings > as it's not a peering bug. > Unless there's a bug in the messaging layer

Re: [ceph-users] PG stuck peering after host reboot

Hi Corentin, I've tried that, the primary hangs when trying to injectargs so I set the option in the config file and restarted all OSDs in the PG, it came up with: pg 1.323 is remapped+peering, acting [595,1391,2147483647,127,937,362,267,320,7,634,716] Still can't query the PG, no error

[ceph-users] PG stuck peering after host reboot

Hi Ceph folks, I have a cluster running Jewel 10.2.5 using a mix EC and replicated pools. After rebooting a host last night, one PG refuses to complete peering pg 1.323 is stuck inactive for 73352.498493, current state peering, last acting [595,1391,240,127,937,362,267,320,7,634,716]

[ceph-users] PGs of EC pool stuck in peering state

Hi Ceph folks, I’ve just posted a bug report http://tracker.ceph.com/issues/18508 I have a cluster (Jewel 10.2.3, SL7) that has trouble creating PGs in EC pools. Essentially, I’ll get a lot of CRUSH_ITEM_NONE (2147483647) in there and PGs will stay in peering states. This sometimes affects

Re: [ceph-users] Incomplete PGs, how do I get them back without data loss?

What exactly do you mean by log? As in a journal of the actions taken or logging done by a daemon. I'm making the same guess but I'm not sure what else I can try at this point. The PG I've been working on actively reports it needs to probe 4 OSDs (the new set and the old primary) which are all

Re: [ceph-users] Incomplete PGs, how do I get them back without data loss?

Hey Dan, This is on Hammer 0.94.5. osd.52 was always on a problematic machine and when this happened had less data on its local disk than the other OSDs. I've tried adapting that blog post's solution to this situation to no avail. I've tried things like looking at all probing OSDs in the query

[ceph-users] Incomplete PGs, how do I get them back without data loss?

Hi folks, I've got a serious issue with a Ceph cluster that's used for RBD. There are 4 PGs stuck in an incomplete state and I'm trying to repair this problem to no avail. Here's ceph status: health HEALTH_WARN 4 pgs incomplete 4 pgs stuck inactive 4 pgs

Re: [ceph-users] EXT :Re: ceph auth list - access denied

For future reference, You can reset your keyring's permissions using a keyring located on the monitors at /var/lib/ceph/mon/your-mon/keyring. Specify the -k option for the ceph command and the full path to the keyring and you can correct this without having to restart the cluster a couple of