Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-02 Thread Sam Wouters
Thanks! Playing around with max_keys in bucket listing retrieval actually gives me results or not, this gives me a way to list the content until the bug is fixed. Is it possible somehow to copy the objects to a new bucket (with versioning disabled), and rename the current one? I don't think the

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Sam Wouters
Sorry, forgot to mention: - yes, filtered by thread - the "is not valid" line occurred when performing the bucket --check - when doing a bucket listing, I also get an "is not valid", but on a different object: 7fe4f1d5b700 20 cls/rgw/cls_rgw.cc:460: entry abc_econtract/data/6scbrrlo4vttk72melewiz

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Sam Wouters
Hi, see inline On 01-09-15 20:14, Yehuda Sadeh-Weinraub wrote: > I assume you filtered the log by thread? I don't see the response > messages. For the bucket check you can run radosgw-admin with > --log-to-stderr. nothing is logged to the console when I do that > > Can you also set 'debug objclas

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Yehuda Sadeh-Weinraub
I assume you filtered the log by thread? I don't see the response messages. For the bucket check you can run radosgw-admin with --log-to-stderr. Can you also set 'debug objclass = 20' on the osds? You can do it by: $ ceph tell osd.\* injectargs --debug-objclass 20 Also, it'd be interesting to ge

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Sam Wouters
not sure where I can find the logs for the bucket check, I can't really filter them out in the radosgw log. -Sam On 01-09-15 19:25, Sam Wouters wrote: > It looks like it, this is what shows in the logs after bumping the debug > and requesting a bucket list. > > 2015-09-01 17:14:53.008620 7fccb17c

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Sam Wouters
It looks like it, this is what shows in the logs after bumping the debug and requesting a bucket list. 2015-09-01 17:14:53.008620 7fccb17ca700 10 cls_bucket_list aws-cmis-prod(@{i=.be-east.rgw.buckets.index}.be-east.rgw.buckets[be-east.5436.1]) start abc_econtract/data/6shflrwbwwcm6dsemrpjit2li3v9

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Yehuda Sadeh-Weinraub
Can you bump up debug (debug rgw = 20, debug ms = 1), and see if the operations (bucket listing and bucket check) go into some kind of infinite loop? Yehuda On Tue, Sep 1, 2015 at 1:16 AM, Sam Wouters wrote: > Hi, I've started the bucket --check --fix on friday evening and it's > still running.

Re: [ceph-users] Troubleshooting rgw bucket list

2015-09-01 Thread Sam Wouters
Hi, I've started the bucket --check --fix on friday evening and it's still running. 'ceph -s' shows the cluster health as OK, I don't know if there is anything else I could check? Is there a way of finding out if its actually doing something? We only have this issue on the one bucket with versioni

Re: [ceph-users] Troubleshooting rgw bucket list

2015-08-31 Thread Gregory Farnum
This generally shouldn't be a problem at your bucket sizes. Have you checked that the cluster is actually in a healthy state? The sleeping locks are normal but should be getting woken up; if they aren't it means the object access isn't working for some reason. A down PG or something would be the si

Re: [ceph-users] Troubleshooting rgw bucket list

2015-08-28 Thread Sam Wouters
Ok, maybe I'm to impatient. It would be great if there were some verbose or progress logging of the radosgw-admin tool. I will start a check and let it run over the weekend. tnx, Sam On 28-08-15 18:16, Sam Wouters wrote: > Hi, > > this bucket only has 13389 objects, so the index size shouldn't be

Re: [ceph-users] Troubleshooting rgw bucket list

2015-08-28 Thread Ben Hines
Still, i'd strongly recommend sharding your big bucket before it gets much bigger. Typically it's during OSD recovery that you will encounter problems as it moves the index and locks all writes to it, it will start returning 500s. Problem is, you need to recreate the bucket before sharding it, it

Re: [ceph-users] Troubleshooting rgw bucket list

2015-08-28 Thread Sam Wouters
Hi, this bucket only has 13389 objects, so the index size shouldn't be a problem. Also, on the same cluster we have an other bucket with 1200543 objects (but no versioning configured), which has no issues. when we run a radosgw-admin bucket --check (--fix), nothing seems to be happening. Putting

Re: [ceph-users] Troubleshooting rgw bucket list

2015-08-28 Thread Ben Hines
How many objects in the bucket? RGW has problems with index size once number of objects gets into the 90+ level. The buckets need to be recreated with 'sharded bucket indexes' on: rgw override bucket index max shards = 23 You could also try repairing the index with: radosgw-admin bucket ch

[ceph-users] Troubleshooting rgw bucket list

2015-08-28 Thread Sam Wouters
Hi, we have a rgw bucket (with versioning) where PUT and GET operations for specific objects succeed, but retrieving an object list fails. Using python-boto, after a timeout just gives us an 500 internal error; radosgw-admin just hangs. Also a radosgw-admin bucket check just seems to hang... cep