Hi there all,
Perhaps someone can help.
We tried to free some storage so we deleted a lot S3 objects. The bucket
has also valuable data so we can't delete the whole bucket.
Everything went fine, but used storage space doesn't get less. We are
expecting several TB of data to be freed.
We then learned of garbage collection. So we thought let's wait. But
even day's later no real change.
We started " radosgw-admin gc process ", that never finished , or
displayed any error or anything.
Could find anything like -verbose or debug for this command or find a
place with log to debug what is going on when radosgw-admin is working
We tried to change the default settings, we got from old posting.
We have put them in global and tried also in [client.rgw..]
rgw_gc_max_objs =7877 ( but also rgw_gc_max_objs =200 or
rgw_gc_max_objs =1000)
rgw_lc_max_objs = 7877
rgw_gc_obj_min_wait = 300
rgw_gc_processor_period = 600
rgw_gc_processor_max_time = 600
We restarted the ceph-radosgw several times, the computers, all over
period of days etc . Tried radosgw-admin gc process a few times etc.
Did not find any references in radosgw logs like gc:: delete etc. But we
don't know what to look for
System is well, nor errors or warnings. But system is in use ( we are
loading up data) -> Will GC only run when idle?
When we count them with "radosgw-admin gc list | grep oid | wc -l" we get
11:00 18.086.665 objects
13:00 18.086.665 objects
15:00 18.086.665 objects
so no change in objects after hours
When we list "radosgw-admin gc list" we get files like
radosgw-admin gc list | more
[
{
"tag": "b5687590-473f-4386-903f-d91a77b8d5cd.7354141.21122\u0000",
"time": "2017-12-06 11:04:56.0.459704s",
"objs": [
{
"pool": "default.rgw.buckets.data",
"oid":
"b5687590-473f-4386-903f-d91a77b8d5cd.44121.4__shadow_.5OtA02n_GU8TkP08We_SLrT5GL1ihuS_1",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid":
"b5687590-473f-4386-903f-d91a77b8d5cd.44121.4__shadow_.5OtA02n_GU8TkP08We_SLrT5GL1ihuS_2",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid":
"b5687590-473f-4386-903f-d91a77b8d5cd.44121.4__shadow_.5OtA02n_GU8TkP08We_SLrT5GL1ihuS_3",
"key": "",
"instance": ""
},
A few questions ->
Who purges the gc list. Is it on the the radosgw machines. Or is it done
distributed on the OSD's?
Where do i have to change default "rgw_gc_max_objs =1000". We tried
everywhere. We have used "tell" to change them in OSD and MON systems
and also on the RGW endpoint's which we restarted.
We have two radosgw endpoints. Is there a lock that only one will act,
or will they both try to delete. Can we free / display such a lock
How can I debug the application radosgw-admin. In which log files to
look, what would be example of message.
If I know an oid like above. Can I manually delete such an oid.
Suppose we would delete the complete bucket with "radosgw-admin bucket
rm --bucket=mybucket --purge-objects --inconsistent-index" would that
also get rid of the GC files that allready there?
Thanks ahead for your time,
JW Michels
q
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
