[ceph-users] Re: rgw index shard much larger than others

2020-12-08 Thread Dan van der Ster
Hi Eric & Matt, I'm working on this again, and was able to reproduce with a versioned test bucket in v14.2.11. I put a test file "passwd", then deleted it, then let the lc trim the versions. The exact lc and resulting bi list are at: https://stikked.web.cern.ch/stikked/view/raw/cc748686 > an

[ceph-users] Re: rgw index shard much larger than others

2020-10-02 Thread Dan van der Ster
Hi Eric, So yes we're hit by this. We have around 1.6M entries in shard 0 with an empty key, e.g.: { "type": "olh", "idx": "<80>1001_02/5f/025f8e0fc8234530d6ae7302adf682509f0f7fb68666391122e16d00bd7107e3/2018_11_14/2625203/3034777/metadata.gz", "entry": {

[ceph-users] Re: rgw index shard much larger than others

2020-10-01 Thread Eric Ivancich
Hi Dan, One way to tell would be to do a: radosgw-admin bi list —bucket= And see if any of the lines output contains (perhaps using `grep`): "type": "olh", That would tell you if there were any versioned objects in the bucket. The “fix” we currently have only prevents this

[ceph-users] Re: rgw index shard much larger than others

2020-10-01 Thread Dan van der Ster
Thanks Matt and Eric, Sorry for the basic question, but how can I as a ceph operator tell if a bucket is versioned? And for fixing this current situation, I would wait for the fix then reshard? (We want to reshard this bucket anyway because listing perf is way too slow for the user with 512

[ceph-users] Re: rgw index shard much larger than others

2020-10-01 Thread Eric Ivancich
Hi Matt and Dan, I too suspect it’s the issue Matt linked to. That bug only affects versioned buckets, so I’m guessing your bucket is versioned, Dan. This bug is triggered when the final instance of an object in a versioned bucket is deleted, but for reasons we do not yet understand, the

[ceph-users] Re: rgw index shard much larger than others

2020-10-01 Thread Matt Benjamin
Hi Dan, Possibly you're reproducing https://tracker.ceph.com/issues/46456. That explains how the underlying issue worked, I don't remember how a bucked exhibiting this is repaired. Eric? Matt On Thu, Oct 1, 2020 at 8:41 AM Dan van der Ster wrote: > > Dear friends, > > Running 14.2.11, we