[ceph-users] Re: squid 19.2.0 QE validation status

rgw — approved Eric (he/him) > On Aug 30, 2024, at 10:42 AM, Yuri Weinstein wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/67779#note-1 > > Release Notes - TBD > Gibba upgrade -TBD > LRC upgrade - TBD > > It was decided and agreed upon that there

[ceph-users] Re: squid 19.2.0 QE validation status

Still looking at the rgw failures. One caught in rgw testing looks to be in core, a valgrind mismatched delete[] in libceph, and I think this squid PR is addressing: https://github.com/ceph/ceph/pull/58991 Here's the valgrind error: https://qa-proxy.ceph.com/teuthology/yuriw-2024-08-29_20:04:

[ceph-users] Re: Not all Bucket Shards being used

Thank you for the information, Christian. When you reshard the bucket id is updated (with most recent versions of ceph, a generation number is incremented). The first bucket id matches the bucket marker, but after the first reshard they diverge. The bucket id is in the names of the currently us

[ceph-users] Re: LARGE_OMAP_OBJECTS warning and bucket has lot of unknown objects and 1999 shards.

There are a couple of potential explanations. 1) Do you have versioning turned on? 1a) And do you write the same file over and over, such as a heartbeat file? 2) Do you have lots of incomplete multipart uploads? If you wouldn’t mind, please run: `radosgw-admin bi list —bucket=epbucket --max-ent

[ceph-users] Re: Not all Bucket Shards being used

1. I recommend that you *not* issue another bucket reshard until you figure out what’s going on. 2. Which version of Ceph are you using? 3. Can you issue a `radosgw-admin metadata get bucket:` so we can verify what the current marker is? 4. After you resharded previously, did you get command-line

[ceph-users] Re: Undo "radosgw-admin bi purge"

rently it does not work for versioned buckets. And it is experimental. If anyone is able to try it I’d be curious about your experiences. Eric (he/him) > On Feb 23, 2023, at 11:20 AM, J. Eric Ivancich wrote: > > Off the top of my head: > > 1. The command would take a bucket

[ceph-users] Re: Undo "radosgw-admin bi purge"

: Versioned buckets will likely require some additional steps, but I’d need to refresh my memory on some of the details. Eric (he/him) > On Feb 23, 2023, at 4:51 AM, Robert Sander > wrote: > > Hi, > > On 22.02.23 17:45, J. Eric Ivancich wrote: > >> You also asked w

[ceph-users] Re: Undo "radosgw-admin bi purge"

Everything you say is to be expected. I was not aware `reshard` could be run when the prior shards are removed, but apparently it can, and it creates new bucket index shards that are empty. Normally `reshard` reads entries from the old shards and copies their data to the new shards but since the

[ceph-users] Re: Undo "radosgw-admin bi purge"

When the admin runs “bi purge” they have the option of supplying a bucket_id with the “--bucket-id” command-line argument. This was useful back when resharding did not automatically remove the older bucket index shards (which it now does), which had a different bucket_id from the current bucket

[ceph-users] Re: RGW archive zone lifecycle

> On Feb 7, 2023, at 6:07 AM, ond...@kuuk.la wrote: > > Hi, > > I have two Ceph clusters in a multi-zone setup. The first one (master zone) > would be accessible to users for their interaction using RGW. > The second one is set to sync from the master zone with the tier type of the > zone set a

[ceph-users] Re: Missing object in bucket list

A bug was reported recently where if a put object occurs when bucket resharding is finishing up, it would write to the old bucket shard rather than the new one. From your logs there is evidence that resharding is underway alongside the put object. A fix for that bug is on main and pacific, and

[ceph-users] Re: radosgw-octopus latest - NoSuchKey Error - some buckets lose their rados objects, but not the bucket index

So it seems like a bucket still has objects listed in the bucket index but the underlying data objects are no longer there. Since you made reference to a customer, I’m guessing the customer does not have direct access to the cluster via `rados` commands, so there’s no chance that they could have

[ceph-users] Re: How to remove remaining bucket index shard objects

I don’t believe there is any tooling to find and clean orphaned bucket index shards. So if you’re certain they’re no longer needed, you can use `rados` commands to remove the objects. Eric (he/him) > On Sep 27, 2022, at 2:37 AM, Yuji Ito (伊藤 祐司) wrote: > > Hi, > > I have encountered a proble

[ceph-users] Re: Wrong size actual?

tps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XOQXZYOWYMMQBWFXMHYDQUJ7LZZPFLSU>) > in any Ceph version since at least Pacific > > Ciao, Uli > > >> On 6. Sep 2022, at 15:48, J. Eric Ivancich > <mailto:ivanc...@redhat.com>> wrote: >> &

[ceph-users] Re: RGW problems after upgrade to 16.2.10

What jumps out to me is: a. The -13 error code represents permission denied b. You’ve commented out the keyring configuration in ceph.conf So do your RGWs have appropriate credentials? Eric (he/him) > On Sep 7, 2022, at 3:04 AM, Rok Jaklič wrote: > > Hi, > > after upgrading t

[ceph-users] Re: Wrong size actual?

You could use `rgw-orphan-list` to determine rados objects that aren’t referenced by any bucket indices. Those objects could be removed after verification since this is an experimental feature. Eric (he/him) > On Sep 5, 2022, at 10:44 AM, Ulrich Klein wrote: > > Looks like the old problem of

[ceph-users] Re: [EXTERNAL] S3 Object Returns Days after Deletion

t; if it will repro. > > I'm using the aws s3api CLI tool to get the info I've shared. > > Thanks for the info regarding the multi-part and tail objects, good to know > that this won't be the cause. > > Kindest regards, > Alex > > -Ori

[ceph-users] Re: S3 Object Returns Days after Deletion

A couple of questions, Alex. Is it the case that the object does not appear when you list the RGW bucket it was in? You referred to "one side of my cluster”. Does that imply you’re using multisite? And just for completeness, this is not a versioned bucket? With a size of 6252 bytes, it wouldn

[ceph-users] Re: Downside of many rgw bucket shards?

Generally it’s a good thing. There’s less contention for bucket index updates when, for example, lots of writes are happening together. Dynamic resharding will take things up to 1999 shards on its own with the default config. Given that we use hashing of objet names to determine which shard they

[ceph-users] Re: Ceph Octopus RGW - files vanished from rados while still in bucket index

3.3.106.11 <http://3.3.106.11/>), > but so small multiparts are very strange. I guess you can really screw up > configs but who am I to judge. > > Am Di., 14. Juni 2022 um 00:29 Uhr schrieb J. Eric Ivancich > mailto:ivanc...@redhat.com>>: > There is no known b

[ceph-users] Re: Ceph Octopus RGW - files vanished from rados while still in bucket index

There is no known bug that would cause the rados objects underlying an RGW object to be removed without a user requesting the RGW object be deleted. There is a known bug where the bucket index might not get updated correctly after user-requested operations. So perhaps the user removed the rgw ob

[ceph-users] Re: not so empty bucket

You’ve provided convincing evidence that the bucket index is not correctly reflecting the data objects. So the next step would be to remove the bucket index entries for these 39 objects. It looks like you’ve already mapped which entries go to which bucket index shards (or you could redo your co

[ceph-users] Re: unknown object

There could be a couple of things going on here. When you copy an object to a new bucket, it creates what’s widely known as a “shallow” copy. The head object gets a true copy, but all tail objects are shared between the two copies. There could also be occasional bugs or somehow an object delete

[ceph-users] Re: Unable to delete bucket - endless multipart uploads?

I wonder if this will be resolved by: https://github.com/ceph/ceph/pull/39358 Deleting a bucket invokes an unordered listing, so that the objects in the bucket can be removed. There was a bug that caused this to loop back the objects. Eric >

[ceph-users] Re: [Suspicious newsletter] Re: Unable to reshard bucket

> Istvan Szabo > Senior Infrastructure Engineer -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW Data Loss Bug in Octopus 15.2.0 through 15.2.6

Tracking down this bug was a group effort and many people participated. See the master branch tracker for that history. Thanks to everyone who helped out. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA ___ ceph-users mailing

[ceph-users] Re: Unable to reshard bucket

emove entry from reshard log, oid=reshard.09 > tenant= bucket=foo > > Is there anything else that I should look for? It looks like the cancel > process thinks > that reshard.09 is present (and probably blocking my attempts at > resharding) but > it's not act

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

jects in general. The > affected objects are simply missing all shadow objects, leaving us with > nothing but the empty head object and a few metadata. > > > > On 13/11/2020 20:14, Eric Ivancich wrote: >> Thank you for the answers to those questions, Janek. >>

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

shadow_) objects, then the initial data is stored in the head object. So this test would not be truly diagnostic. This could be done with a large object, for example, with `s3cmd put --disable-multipart …`. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA ___

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on

[ceph-users] Re: rgw index shard much larger than others

ted in the versioned bucket. I hope that’s informative, if not what you were hoping to hear. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA > On Oct 1, 2020, at 10:53 AM, Dan van der Ster wrote: > > Thanks Matt and Eric, > > Sorry for the basi

[ceph-users] Re: rgw index shard much larger than others

Hi Matt and Dan, I too suspect it’s the issue Matt linked to. That bug only affects versioned buckets, so I’m guessing your bucket is versioned, Dan. This bug is triggered when the final instance of an object in a versioned bucket is deleted, but for reasons we do not yet understand, the object

[ceph-users] Re: Unable to reshard bucket

Perhaps the next step is to examine the generated logs from: radosgw-admin reshard status --bucket=foo --debug-rgw=20 --debug-ms=1 radosgw-admin reshard cancel --bucket foo --debug-rgw=20 --debug-ms=1 Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan

[ceph-users] Re: RGW and the orphans

Hi Katarzyna, Incomplete multipart uploads are not considered orphans. With respect to the 404s…. Which version of ceph are you running? What tooling are you using to list and cancel? Can you provide a console transcript of the listing and cancelling? Thanks, Eric -- J. Eric Ivancich he

[ceph-users] Re: RGW and the orphans

. I look forward to your report! And please feel free to post additional questions in this forum. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA > On Apr 20, 2020, at 6:18 AM, Katarzyna Myrek wrote: > > Hi Eric, > > I will try your tool t

[ceph-users] Re: RGW and the orphans

tures. Instead it involved internal reorganization of the code. I suspect it would work, but if you try it, please report back what you find. Of course this is currently an experimental feature and care (e.g., sanity checking) should be taken before using the list produced to feed into a massive delete pr

[ceph-users] Re: RGW and the orphans

complete on master, so that can be backported as well. See: https://github.com/ceph/ceph/pull/34127 <https://github.com/ceph/ceph/pull/34127> -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, US ___ ceph-users mailing list --

[ceph-users] Re: RGW and the orphans

There is currently a PR for an “orphans list” capability. I’m currently working on the testing side to make sure it’s part of our teuthology suite. See: https://github.com/ceph/ceph/pull/34148 Eric > On Apr 16, 2020, at 9:26 AM, Katarzyna Myrek wrote

[ceph-users] Re: Problems with ragosgw

ot thread_join’d by their parents. a. This seems unlikely as this appears to happen during start-up before threads are likely done with their work. 3. Florian Weimer identifies a kernel bug. I suspect it’s #1, so you might want to try reducing the number of threads rgw uses by lowering the valu

[ceph-users] Re: Bucket rename with

ize of the PR -- 22 commits and 32 files altered -- my guess is that it will not be backported to Nautilus. However I'll invite the principals to weigh in. Best, Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA ___ ce