On 5/30/19 2:45 PM, Wido den Hollander wrote:
>
>
> On 5/29/19 11:22 PM, J. Eric Ivancich wrote:
>> Hi Wido,
>>
>> When you run `radosgw-admin gc list`, I assume you are *not* using the
>> "--include-all" flag, right? If you're not using that flag, then
>> everything listed should be expired and be ready for clean-up. If after
>> running `radosgw-admin gc process` the same entries appear in
>> `radosgw-admin gc list` then gc apparently stalled.
>>
>
> Not using the --include-all in both cases.
>
> GC seems to stall and doesn't do anything when looking at it with
> --debug-rados=10
>
>> There were a few bugs within gc processing that could prevent it from
>> making forward progress. They were resolved with a PR (master:
>> https://github.com/ceph/ceph/pull/26601 ; mimic backport:
>> https://github.com/ceph/ceph/pull/27796). Unfortunately that code was
>> backported after the 13.2.5 release, but it is in place for the 13.2.6
>> release of mimic.
>>
>
> Thanks! I'll might grab some packages from Shaman to give GC a try.
>
I've set up a temporary machine next to the 13.2.5 cluster with the
13.2.6 packages from Shaman.
On that machine I'm running:
$ radosgw-admin gc process
That seems to work as intended! So the PR seems to have fixed it.
Should be fixed permanently when 13.2.6 is officially released.
Wido
> Wido
>
>> Eric
>>
>>
>> On 5/29/19 3:19 AM, Wido den Hollander wrote:
>>> Hi,
>>>
>>> I've got a Ceph cluster with this status:
>>>
>>> health: HEALTH_WARN
>>> 3 large omap objects
>>>
>>> After looking into it I see that the issue comes from objects in the
>>> '.rgw.gc' pool.
>>>
>>> Investigating it I found that the gc.* objects have a lot of OMAP keys:
>>>
>>> for OBJ in $(rados -p .rgw.gc ls); do
>>> echo $OBJ
>>> rados -p .rgw.gc listomapkeys $OBJ|wc -l
>>> done
>>>
>>> I then found out that on average these objects have about 100k of OMAP
>>> keys each, but two stand out and have about 3M OMAP keys.
>>>
>>> I can list the GC with 'radosgw-admin gc list' and this yields a JSON
>>> which is a couple of MB in size.
>>>
>>> I ran:
>>>
>>> $ radosgw-admin gc process
>>>
>>> That runs for hours and then finishes, but the large list of OMAP keys
>>> stays.
>>>
>>> Running Mimic 13.3.5 on this cluster.
>>>
>>> Has anybody seen this before?
>>>
>>> Wido
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
