Thanks Frédéric, we’ve done that in the meantime to work around issue #47866.

The error has been reproduced and there’s a PR associated with the issue:

https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866>

Cheers,

Denis

> On 23 Nov 2020, at 11:56, Frédéric Nass <frederic.n...@univ-lorraine.fr> 
> wrote:
> 
> Hi Denis,
> 
> You might want to look at rgw_gc_obj_min_wait from [1] and try increasing the 
> default value of 7200s (2 hours) to whatever suits your need < 2^64.
> Just remind that at some point you'll have to get these objects processed by 
> the gc. Or manually through the API [2].
> 
> One thing that comes to mind regarding the "last night's missing object" is 
> maybe it was multi-part re-written and the re-write failed somehow and the 
> object was then enlisted by the gc. But that supposes this particular object 
> sometimes gets re-written which may not be the case.
> 
> Regards,
> 
> Frédéric.
> 
> [1] https://docs.ceph.com/en/latest/radosgw/config-ref/
> [2] 
> https://docs.ceph.com/en/latest/dev/radosgw/admin/adminops_nonimplemented/#manually-processes-garbage-collection-items
> 
> Le 18/11/2020 à 11:27, Denis Krienbühl a écrit :
>> By the way, since there’s some probability that this is a GC refcount issue, 
>> would it be possible and sane to somehow slow the GC down or disable it 
>> altogether? Is that something we could implement on our end as a stop-gap 
>> measure to prevent dataloss?
>> 
>>> On 18 Nov 2020, at 10:46, Denis Krienbühl <de...@href.ch> wrote:
>>> 
>>> I can now confirm that last night’s missing object was a multi-part file.
>>> 
>>>> On 18 Nov 2020, at 10:01, Janek Bevendorff 
>>>> <janek.bevendo...@uni-weimar.de> wrote:
>>>> 
>>>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME 
>>>> --object=OBJECTNAME (forgot the "object" there)
>>>> 
>>>> On 18/11/2020 09:58, Janek Bevendorff wrote:
>>>>>> The object, a Docker layer, that went missing has not been touched in 2 
>>>>>> months. It worked for a while, but then suddenly went missing.
>>>>> Was the object a multipart object? You can check by running radosgw-admin 
>>>>> stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something 
>>>>> "ns": "multipart" in the output. If it says "ns": "shadow", it's a 
>>>>> single-part object.
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to