[ceph-users] Re: radosgw-octopus latest - NoSuchKey Error - some buckets lose their rados objects, but not the bucket index

So it seems like a bucket still has objects listed in the bucket index but the underlying data objects are no longer there. Since you made reference to a customer, I’m guessing the customer does not have direct access to the cluster via `rados` commands, so there’s no chance that they could have

[ceph-users] Re: Missing object in bucket list

A bug was reported recently where if a put object occurs when bucket resharding is finishing up, it would write to the old bucket shard rather than the new one. From your logs there is evidence that resharding is underway alongside the put object. A fix for that bug is on main and pacific, and

[ceph-users] Re: RGW archive zone lifecycle

> On Feb 7, 2023, at 6:07 AM, ond...@kuuk.la wrote: > > Hi, > > I have two Ceph clusters in a multi-zone setup. The first one (master zone) > would be accessible to users for their interaction using RGW. > The second one is set to sync from the master zone with the tier type of the > zone set a

[ceph-users] Re: Undo "radosgw-admin bi purge"

When the admin runs “bi purge” they have the option of supplying a bucket_id with the “--bucket-id” command-line argument. This was useful back when resharding did not automatically remove the older bucket index shards (which it now does), which had a different bucket_id from the current bucket

[ceph-users] Re: Undo "radosgw-admin bi purge"

Everything you say is to be expected. I was not aware `reshard` could be run when the prior shards are removed, but apparently it can, and it creates new bucket index shards that are empty. Normally `reshard` reads entries from the old shards and copies their data to the new shards but since the

[ceph-users] Re: Undo "radosgw-admin bi purge"

: Versioned buckets will likely require some additional steps, but I’d need to refresh my memory on some of the details. Eric (he/him) > On Feb 23, 2023, at 4:51 AM, Robert Sander > wrote: > > Hi, > > On 22.02.23 17:45, J. Eric Ivancich wrote: > >> You also asked w

[ceph-users] Re: Undo "radosgw-admin bi purge"

rently it does not work for versioned buckets. And it is experimental. If anyone is able to try it I’d be curious about your experiences. Eric (he/him) > On Feb 23, 2023, at 11:20 AM, J. Eric Ivancich wrote: > > Off the top of my head: > > 1. The command would take a bucket

[ceph-users] Re: Not all Bucket Shards being used

1. I recommend that you *not* issue another bucket reshard until you figure out what’s going on. 2. Which version of Ceph are you using? 3. Can you issue a `radosgw-admin metadata get bucket:` so we can verify what the current marker is? 4. After you resharded previously, did you get command-line

[ceph-users] Re: LARGE_OMAP_OBJECTS warning and bucket has lot of unknown objects and 1999 shards.

There are a couple of potential explanations. 1) Do you have versioning turned on? 1a) And do you write the same file over and over, such as a heartbeat file? 2) Do you have lots of incomplete multipart uploads? If you wouldn’t mind, please run: `radosgw-admin bi list —bucket=epbucket --max-ent

[ceph-users] Re: Not all Bucket Shards being used

Thank you for the information, Christian. When you reshard the bucket id is updated (with most recent versions of ceph, a generation number is incremented). The first bucket id matches the bucket marker, but after the first reshard they diverge. The bucket id is in the names of the currently us

[ceph-users] Re: Unable to delete bucket - endless multipart uploads?

I wonder if this will be resolved by: https://github.com/ceph/ceph/pull/39358 Deleting a bucket invokes an unordered listing, so that the objects in the bucket can be removed. There was a bug that caused this to loop back the objects. Eric >

[ceph-users] Re: unknown object

There could be a couple of things going on here. When you copy an object to a new bucket, it creates what’s widely known as a “shallow” copy. The head object gets a true copy, but all tail objects are shared between the two copies. There could also be occasional bugs or somehow an object delete

[ceph-users] Re: not so empty bucket

You’ve provided convincing evidence that the bucket index is not correctly reflecting the data objects. So the next step would be to remove the bucket index entries for these 39 objects. It looks like you’ve already mapped which entries go to which bucket index shards (or you could redo your co

[ceph-users] Re: Ceph Octopus RGW - files vanished from rados while still in bucket index

There is no known bug that would cause the rados objects underlying an RGW object to be removed without a user requesting the RGW object be deleted. There is a known bug where the bucket index might not get updated correctly after user-requested operations. So perhaps the user removed the rgw ob

[ceph-users] Re: Ceph Octopus RGW - files vanished from rados while still in bucket index

3.3.106.11 <http://3.3.106.11/>), > but so small multiparts are very strange. I guess you can really screw up > configs but who am I to judge. > > Am Di., 14. Juni 2022 um 00:29 Uhr schrieb J. Eric Ivancich > mailto:ivanc...@redhat.com>>: > There is no known b

[ceph-users] Re: Downside of many rgw bucket shards?

Generally it’s a good thing. There’s less contention for bucket index updates when, for example, lots of writes are happening together. Dynamic resharding will take things up to 1999 shards on its own with the default config. Given that we use hashing of objet names to determine which shard they

[ceph-users] Re: S3 Object Returns Days after Deletion

A couple of questions, Alex. Is it the case that the object does not appear when you list the RGW bucket it was in? You referred to "one side of my cluster”. Does that imply you’re using multisite? And just for completeness, this is not a versioned bucket? With a size of 6252 bytes, it wouldn

[ceph-users] Re: [EXTERNAL] S3 Object Returns Days after Deletion

t; if it will repro. > > I'm using the aws s3api CLI tool to get the info I've shared. > > Thanks for the info regarding the multi-part and tail objects, good to know > that this won't be the cause. > > Kindest regards, > Alex > > -Ori

[ceph-users] Re: Wrong size actual?

You could use `rgw-orphan-list` to determine rados objects that aren’t referenced by any bucket indices. Those objects could be removed after verification since this is an experimental feature. Eric (he/him) > On Sep 5, 2022, at 10:44 AM, Ulrich Klein wrote: > > Looks like the old problem of

[ceph-users] Re: RGW problems after upgrade to 16.2.10

What jumps out to me is: a. The -13 error code represents permission denied b. You’ve commented out the keyring configuration in ceph.conf So do your RGWs have appropriate credentials? Eric (he/him) > On Sep 7, 2022, at 3:04 AM, Rok Jaklič wrote: > > Hi, > > after upgrading t

[ceph-users] Re: Wrong size actual?

tps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XOQXZYOWYMMQBWFXMHYDQUJ7LZZPFLSU>) > in any Ceph version since at least Pacific > > Ciao, Uli > > >> On 6. Sep 2022, at 15:48, J. Eric Ivancich > <mailto:ivanc...@redhat.com>> wrote: >> &

[ceph-users] Re: How to remove remaining bucket index shard objects

I don’t believe there is any tooling to find and clean orphaned bucket index shards. So if you’re certain they’re no longer needed, you can use `rados` commands to remove the objects. Eric (he/him) > On Sep 27, 2022, at 2:37 AM, Yuji Ito (伊藤 祐司) wrote: > > Hi, > > I have encountered a proble

[ceph-users] Re: Bucket rename with

ize of the PR -- 22 commits and 32 files altered -- my guess is that it will not be backported to Nautilus. However I'll invite the principals to weigh in. Best, Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA ___ ce

[ceph-users] Re: squid 19.2.0 QE validation status

Still looking at the rgw failures. One caught in rgw testing looks to be in core, a valgrind mismatched delete[] in libceph, and I think this squid PR is addressing: https://github.com/ceph/ceph/pull/58991 Here's the valgrind error: https://qa-proxy.ceph.com/teuthology/yuriw-2024-08-29_20:04:

[ceph-users] Re: squid 19.2.0 QE validation status

rgw — approved Eric (he/him) > On Aug 30, 2024, at 10:42 AM, Yuri Weinstein wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/67779#note-1 > > Release Notes - TBD > Gibba upgrade -TBD > LRC upgrade - TBD > > It was decided and agreed upon that there