It appears we eventually got 'data sync init' working. At least, it's worked on 5 of the 6 sync directions in our 3-node cluster. The sixth has not run without an error returned, although 'sync status' does say "preparing for full sync".
Thanks, Trey On Wed, Mar 6, 2019 at 1:22 PM Trey Palmer <nerdmagic...@gmail.com> wrote: > Casey, > > This was the result of trying 'data sync init': > > root@c2-rgw1:~# radosgw-admin data sync init > ERROR: source zone not specified > root@c2-rgw1:~# radosgw-admin data sync init --source-zone=<master zone > uuid> > WARNING: cannot find source zone id for name=<master zone uuid> > ERROR: sync.init_sync_status() returned ret=-2 > root@c2-rgw1:~# radosgw-admin data sync init --source-zone=c1-zone > ERROR: sync.init() returned ret=-5 > 2019-03-06 10:14:59.815735 7fecb214fe40 0 data sync: ERROR: failed to > fetch datalog info > root@c2-rgw1:~# > > Do you have any further advice or info? > > Thanks again, > > Trey > > > On Wed, Mar 6, 2019 at 11:47 AM Casey Bodley <cbod...@redhat.com> wrote: > >> Hi Trey, >> >> I think it's more likely that these stale metadata entries are from >> deleted buckets, rather than accidental bucket reshards. When a bucket >> is deleted in a multisite configuration, we don't delete its bucket >> instance because other zones may still need to sync the object deletes - >> and they can't make progress on sync if the bucket metadata disappears. >> These leftover bucket instances look the same to the 'reshard >> stale-instances' commands, but I'd be cautious about using that to >> remove them in multisite, as it may cause more sync errors and >> potentially leak storage if they still contain objects. >> >> Regarding 'datalog trim', that alone isn't safe because it could trim >> entries that hadn't been applied on other zones yet, causing them to >> miss some updates. What you can do is run 'data sync init' on each zone, >> and restart gateways. This will restart with a data full sync (which >> will scan all buckets for changes), and skip past any datalog entries >> from before the full sync. I was concerned that the bug in error >> handling (ie "ERROR: init sync on...") would also affect full sync, but >> that doesn't appear to be the case - so I do think that's worth trying. >> >> On 3/5/19 6:24 PM, Trey Palmer wrote: >> > Casey, >> > >> > Thanks very much for the reply! >> > >> > We definitely have lots of errors on sync-disabled buckets and the >> > workaround for that is obvious (most of them are empty anyway). >> > >> > Our second form of error is stale buckets. We had dynamic resharding >> > enabled but have now disabled it (having discovered it was on by >> > default, and not supported in multisite). >> > >> > We removed several hundred stale buckets via 'radosgw-admin sharding >> > stale-instances rm', but they are still giving us sync errors. >> > >> > I have found that these buckets do have entries in 'radosgw-admin >> > datalog list', and my guess is this could be fixed by doing a >> > 'radosgw-admin datalog trim' for each entry on the master zone. >> > >> > Does that sound right? :-) >> > >> > Thanks again for the detailed explanation, >> > >> > Trey Palmer >> > >> > On Tue, Mar 5, 2019 at 5:55 PM Casey Bodley <cbod...@redhat.com >> > <mailto:cbod...@redhat.com>> wrote: >> > >> > Hi Christian, >> > >> > I think you've correctly intuited that the issues are related to >> > the use >> > of 'bucket sync disable'. There was a bug fix for that feature in >> > http://tracker.ceph.com/issues/26895, and I recently found that a >> > block >> > of code was missing from its luminous backport. That missing code is >> > what handled those "ERROR: init sync on <bucket instance> failed, >> > retcode=-2" errors. >> > >> > I included a fix for that in a later backport >> > (https://github.com/ceph/ceph/pull/26549), which I'm still working >> to >> > get through qa. I'm afraid I can't really recommend a workaround >> > for the >> > issue in the meantime. >> > >> > Looking forward though, we do plan to support something like s3's >> > cross >> > region replication so you can enable replication on a specific >> bucket >> > without having to enable it globally. >> > >> > Casey >> > >> > >> > On 3/5/19 2:32 PM, Christian Rice wrote: >> > > >> > > Much appreciated. We’ll continue to poke around and certainly >> will >> > > disable the dynamic resharding. >> > > >> > > We started with 12.2.8 in production. We definitely did not >> > have it >> > > enabled in ceph.conf >> > > >> > > *From: *Matthew H <matthew.he...@hotmail.com >> > <mailto:matthew.he...@hotmail.com>> >> > > *Date: *Tuesday, March 5, 2019 at 11:22 AM >> > > *To: *Christian Rice <cr...@pandora.com >> > <mailto:cr...@pandora.com>>, ceph-users >> > > <ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>> >> > > *Cc: *Trey Palmer <nerdmagic...@gmail.com >> > <mailto:nerdmagic...@gmail.com>> >> > > *Subject: *Re: radosgw sync falling behind regularly >> > > >> > > Hi Christian, >> > > >> > > To be on the safe side and future proof yourself will want to go >> > ahead >> > > and set the following in your ceph.conf file, and then issue a >> > restart >> > > to your RGW instances. >> > > >> > > rgw_dynamic_resharding = false >> > > >> > > There are a number of issues with dynamic resharding, multisite >> rgw >> > > problems being just one of them. However I thought it was disabled >> > > automatically when multisite rgw is used (but I will have to >> double >> > > check the code on that). What version of Ceph did you initially >> > > install the cluster with? Prior to v12.2.2 this feature was >> > enabled by >> > > default for all rgw use cases. >> > > >> > > Thanks, >> > > >> > > >> > >> ------------------------------------------------------------------------ >> > > >> > > *From:*Christian Rice <cr...@pandora.com <mailto: >> cr...@pandora.com>> >> > > *Sent:* Tuesday, March 5, 2019 2:07 PM >> > > *To:* Matthew H; ceph-users >> > > *Subject:* Re: radosgw sync falling behind regularly >> > > >> > > Matthew, first of all, let me say we very much appreciate your >> help! >> > > >> > > So I don’t think we turned dynamic resharding on, nor did we >> > manually >> > > reshard buckets. Seems like it defaults to on for luminous but the >> > > mimic docs say it’s not supported in multisite. So do we need to >> > > disable it manually via tell and ceph.conf? >> > > >> > > Also, after running the command you suggested, all the stale >> > instances >> > > are gone…these from my examples were in output: >> > > >> > > "bucket_instance": >> > > >> > >> "sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.303", >> > > >> > > "bucket_instance": >> > > >> > >> "sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299", >> > > >> > > "bucket_instance": >> > > >> > >> "sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.301", >> > > >> > > Though we still get lots of log messages like so in rgw: >> > > >> > > 2019-03-05 11:01:09.526120 7f64120ae700 0 ERROR: failed to get >> > bucket >> > > instance info for >> > > >> > >> >> .bucket.meta.sysad_task:sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > >> > > 2019-03-05 11:01:09.528664 7f63e5016700 1 civetweb: >> > 0x55976f1c2000: >> > > 172.17.136.17 - - [05/Mar/2019:10:54:06 -0800] "GET >> > > >> > >> >> /admin/metadata/bucket.instance/sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299?key=sysad_task%2Fsysad-task%3A1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299&rgwx-zonegroup=de6af748-1a2f-44a1-9d44-30799cf1313e >> > >> > > HTTP/1.1" 404 0 - - >> > > >> > > 2019-03-05 11:01:09.529648 7f64130b0700 0 meta sync: ERROR: can't >> > > remove key: >> > > >> > >> >> bucket.instance:sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > >> > > ret=-2 >> > > >> > > 2019-03-05 11:01:09.530324 7f64138b1700 0 ERROR: failed to get >> > bucket >> > > instance info for >> > > >> > >> >> .bucket.meta.sysad_task:sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > >> > > 2019-03-05 11:01:09.530345 7f6405094700 0 data sync: ERROR: >> > failed to >> > > retrieve bucket info for >> > > >> > >> bucket=sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > >> > > 2019-03-05 11:01:09.531774 7f6405094700 0 data sync: WARNING: >> > > skipping data log entry for missing bucket >> > > >> sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > >> > > 2019-03-05 11:01:09.571680 7f6405094700 0 data sync: ERROR: >> > init sync >> > > on >> > > >> > sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.302 >> > > failed, retcode=-2 >> > > >> > > 2019-03-05 11:01:09.573179 7f6405094700 0 data sync: WARNING: >> > > skipping data log entry for missing bucket >> > > >> sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.302 >> > > >> > > 2019-03-05 11:01:13.504308 7f63f903e700 1 civetweb: >> > 0x55976f0f2000: >> > > 10.105.18.20 - - [05/Mar/2019:11:00:57 -0800] "GET >> > > >> > >> >> /admin/metadata/bucket.instance/sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299?key=sysad_task%2Fsysad-task%3A1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299&rgwx-zonegroup=de6af748-1a2f-44a1-9d44-30799cf1313e >> > >> > > HTTP/1.1" 404 0 - - >> > > >> > > *From: *Matthew H <matthew.he...@hotmail.com >> > <mailto:matthew.he...@hotmail.com>> >> > > *Date: *Tuesday, March 5, 2019 at 10:03 AM >> > > *To: *Christian Rice <cr...@pandora.com >> > <mailto:cr...@pandora.com>>, ceph-users >> > > <ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>> >> > > *Subject: *Re: radosgw sync falling behind regularly >> > > >> > > Hi Christian, >> > > >> > > You have stale bucket instances that need to be clean up, which is >> > > what 'radosgw-admin reshard stale-instances list' is showing >> > you. Have >> > > you or were you manually resharding your buckets? The errors you >> > are >> > > seeing in the logs are related to these stale instances being kept >> > > around. >> > > >> > > In v12.2.11 this command along with 'radosgw-admin reshard >> > > stale-instance rm' was introduced [1]. >> > > >> > > Hopefully this helps. >> > > >> > > [1] >> > > >> > > https://ceph.com/releases/v12-2-11-luminous-released/ >> > > >> > < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__ceph.com_releases_v12-2D2-2D11-2Dluminous-2Dreleased_&d=DwMF-g&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=NE1NbWtVhgG-K7YvLdoLZigfFx8zGPwOGk6HWpYK04I&m=vdtYIn6lEKaWD9wW297aHjQLpmQdHZrOVpOhmCBqkqo&s=nGCpS4p5jnaSpPUFlziSi3Y3pFijhVDy6e3867jA9BE&e= >> > >> > > >> > > /"There have been fixes to RGW dynamic and manual resharding, >> > which no >> > > longer >> > > leaves behind stale bucket instances to be removed manually. For >> > > finding and >> > > cleaning up older instances from a reshard a radosgw-admin >> > command reshard >> > > stale-instances list and reshard stale-instances rm should do the >> > > necessary >> > > cleanup."/ >> > > >> > > >> > >> ------------------------------------------------------------------------ >> > > >> > > *From:*Christian Rice <cr...@pandora.com <mailto: >> cr...@pandora.com>> >> > > *Sent:* Tuesday, March 5, 2019 11:34 AM >> > > *To:* Matthew H; ceph-users >> > > *Subject:* Re: radosgw sync falling behind regularly >> > > >> > > The output of “radosgw-admin reshard stale-instances list” shows >> > 242 >> > > entries, which might embed too much proprietary info for me to >> > list, >> > > but here’s a tiny sample: >> > > >> > > >> > >> "sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.303", >> > > >> > > >> > >> "sysad_task/sysad_task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.281", >> > > >> > > >> > >> "sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299", >> > > >> > > >> > >> "sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.301", >> > > >> > > Some of appear repeatedly in the radosgw error logs like so: >> > > >> > > 2019-03-05 08:13:08.929206 7f6405094700 0 data sync: ERROR: >> > init sync >> > > on >> > > >> > sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.302 >> > > failed, retcode=-2 >> > > >> > > 2019-03-05 08:13:08.930581 7f6405094700 0 data sync: WARNING: >> > > skipping data log entry for missing bucket >> > > >> sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.302 >> > > >> > > 2019-03-05 08:13:08.972053 7f6405094700 0 data sync: ERROR: >> > init sync >> > > on >> > > >> > sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > failed, retcode=-2 >> > > >> > > 2019-03-05 08:13:08.973442 7f6405094700 0 data sync: WARNING: >> > > skipping data log entry for missing bucket >> > > >> sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > >> > > 2019-03-05 08:13:19.528295 7f6406897700 0 data sync: ERROR: >> > init sync >> > > on >> > > >> > sysad_task/sysad-task:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18330.299 >> > > failed, retcode=-2 >> > > >> > > Notably, “Sync is disabled for bucket sysad-task.” We use “bucket >> > > sync disable” A LOT. It’s the only way we’ve been able to use >> > > multisite and a single namespace and not replicate things that are >> > > unneeded to every zone. Perhaps there’s a bug in the >> > implementation >> > > that’s tripping us up now, with the new sync multisite sync code >> > from >> > > 12.2.9 onward? >> > > >> > > What might we do with stale bucket instances, then? >> > > >> > > Of note, our master zone endpoint, which was timing out health >> > checks >> > > most of the day after the upgrade (was running but overworked by >> > > cluster confusion, so we couldn’t create new buckets or do user >> > ops), >> > > has returned to availability late last night. There’s a lot of >> > data >> > > to look at, but in my estimation, due to lack of user complaints >> > (or >> > > their unawareness of specific issues), it seems the zones are >> > > nominally available, even with all the errors and warnings being >> > > logged. We’ve tested simple zone replication by creating a few >> > files >> > > in one zone and seeing them show up elsewhere… >> > > >> > > here’s “period get” output: >> > > >> > > sv5-ceph-rgw1 >> > > >> > > { >> > > >> > > "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53", >> > > >> > > "epoch": 16, >> > > >> > > "predecessor_uuid": "926c74c7-c1a7-46b1-9f25-eb5c392a7fbb", >> > > >> > > "sync_status": [], >> > > >> > > "period_map": { >> > > >> > > "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53", >> > > >> > > "zonegroups": [ >> > > >> > > { >> > > >> > > "id": "de6af748-1a2f-44a1-9d44-30799cf1313e", >> > > >> > > "name": "us", >> > > >> > > "api_name": "us", >> > > >> > > "is_master": "true", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv5-ceph-rgw1.savagebeast.com:8080"; >> > > >> > > ], >> > > >> > > "hostnames": [], >> > > >> > > "hostnames_s3website": [], >> > > >> > > "master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "zones": [ >> > > >> > > { >> > > >> > > "id": "107d29a0-b732-4bf1-a26e-1f64f820e839", >> > > >> > > "name": "dc11-prod", >> > > >> > > "endpoints": [ >> > > >> > > "http://dc11-ceph-rgw1:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > }, >> > > >> > > { >> > > >> > > "id": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "name": "sv5-corp", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv5-ceph-rgw1.savagebeast.com:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > }, >> > > >> > > { >> > > >> > > "id": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8", >> > > >> > > "name": "sv3-prod", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv3-ceph-rgw1:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > } >> > > >> > > ], >> > > >> > > "placement_targets": [ >> > > >> > > { >> > > >> > > "name": "default-placement", >> > > >> > > "tags": [] >> > > >> > > } >> > > >> > > ], >> > > >> > > "default_placement": "default-placement", >> > > >> > > "realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd" >> > > >> > > } >> > > >> > > ], >> > > >> > > "short_zone_ids": [ >> > > >> > > { >> > > >> > > "key": "107d29a0-b732-4bf1-a26e-1f64f820e839", >> > > >> > > "val": 1720993486 >> > > >> > > }, >> > > >> > > { >> > > >> > > "key": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "val": 2301637458 >> > > >> > > }, >> > > >> > > { >> > > >> > > "key": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8", >> > > >> > > "val": 1449486239 >> > > >> > > } >> > > >> > > ] >> > > >> > > }, >> > > >> > > "master_zonegroup": "de6af748-1a2f-44a1-9d44-30799cf1313e", >> > > >> > > "master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "period_config": { >> > > >> > > "bucket_quota": { >> > > >> > > "enabled": false, >> > > >> > > "check_on_raw": false, >> > > >> > > "max_size": -1, >> > > >> > > "max_size_kb": 0, >> > > >> > > "max_objects": -1 >> > > >> > > }, >> > > >> > > "user_quota": { >> > > >> > > "enabled": false, >> > > >> > > "check_on_raw": false, >> > > >> > > "max_size": -1, >> > > >> > > "max_size_kb": 0, >> > > >> > > "max_objects": -1 >> > > >> > > } >> > > >> > > }, >> > > >> > > "realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd", >> > > >> > > "realm_name": "savagebucket", >> > > >> > > "realm_epoch": 2 >> > > >> > > } >> > > >> > > sv3-ceph-rgw1 >> > > >> > > { >> > > >> > > "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53", >> > > >> > > "epoch": 16, >> > > >> > > "predecessor_uuid": "926c74c7-c1a7-46b1-9f25-eb5c392a7fbb", >> > > >> > > "sync_status": [], >> > > >> > > "period_map": { >> > > >> > > "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53", >> > > >> > > "zonegroups": [ >> > > >> > > { >> > > >> > > "id": "de6af748-1a2f-44a1-9d44-30799cf1313e", >> > > >> > > "name": "us", >> > > >> > > "api_name": "us", >> > > >> > > "is_master": "true", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv5-ceph-rgw1.savagebeast.com:8080"; >> > > >> > > ], >> > > >> > > "hostnames": [], >> > > >> > > "hostnames_s3website": [], >> > > >> > > "master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "zones": [ >> > > >> > > { >> > > >> > > "id": "107d29a0-b732-4bf1-a26e-1f64f820e839", >> > > >> > > "name": "dc11-prod", >> > > >> > > "endpoints": [ >> > > >> > > "http://dc11-ceph-rgw1:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > }, >> > > >> > > { >> > > >> > > "id": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "name": "sv5-corp", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv5-ceph-rgw1.savagebeast.com:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > }, >> > > >> > > { >> > > >> > > "id": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8", >> > > >> > > "name": "sv3-prod", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv3-ceph-rgw1:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > } >> > > >> > > ], >> > > >> > > "placement_targets": [ >> > > >> > > { >> > > >> > > "name": "default-placement", >> > > >> > > "tags": [] >> > > >> > > } >> > > >> > > ], >> > > >> > > "default_placement": "default-placement", >> > > >> > > "realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd" >> > > >> > > } >> > > >> > > ], >> > > >> > > "short_zone_ids": [ >> > > >> > > { >> > > >> > > "key": "107d29a0-b732-4bf1-a26e-1f64f820e839", >> > > >> > > "val": 1720993486 >> > > >> > > }, >> > > >> > > { >> > > >> > > "key": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "val": 2301637458 >> > > >> > > }, >> > > >> > > { >> > > >> > > "key": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8", >> > > >> > > "val": 1449486239 >> > > >> > > } >> > > >> > > ] >> > > >> > > }, >> > > >> > > "master_zonegroup": "de6af748-1a2f-44a1-9d44-30799cf1313e", >> > > >> > > "master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "period_config": { >> > > >> > > "bucket_quota": { >> > > >> > > "enabled": false, >> > > >> > > "check_on_raw": false, >> > > >> > > "max_size": -1, >> > > >> > > "max_size_kb": 0, >> > > >> > > "max_objects": -1 >> > > >> > > }, >> > > >> > > "user_quota": { >> > > >> > > "enabled": false, >> > > >> > > "check_on_raw": false, >> > > >> > > "max_size": -1, >> > > >> > > "max_size_kb": 0, >> > > >> > > "max_objects": -1 >> > > >> > > } >> > > >> > > }, >> > > >> > > "realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd", >> > > >> > > "realm_name": "savagebucket", >> > > >> > > "realm_epoch": 2 >> > > >> > > } >> > > >> > > dc11-ceph-rgw1 >> > > >> > > { >> > > >> > > "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53", >> > > >> > > "epoch": 16, >> > > >> > > "predecessor_uuid": "926c74c7-c1a7-46b1-9f25-eb5c392a7fbb", >> > > >> > > "sync_status": [], >> > > >> > > "period_map": { >> > > >> > > "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53", >> > > >> > > "zonegroups": [ >> > > >> > > { >> > > >> > > "id": "de6af748-1a2f-44a1-9d44-30799cf1313e", >> > > >> > > "name": "us", >> > > >> > > "api_name": "us", >> > > >> > > "is_master": "true", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv5-ceph-rgw1.savagebeast.com:8080"; >> > > >> > > ], >> > > >> > > "hostnames": [], >> > > >> > > "hostnames_s3website": [], >> > > >> > > "master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "zones": [ >> > > >> > > { >> > > >> > > "id": "107d29a0-b732-4bf1-a26e-1f64f820e839", >> > > >> > > "name": "dc11-prod", >> > > >> > > "endpoints": [ >> > > >> > > "http://dc11-ceph-rgw1:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > }, >> > > >> > > { >> > > >> > > "id": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "name": "sv5-corp", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv5-ceph-rgw1.savagebeast.com:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > }, >> > > >> > > { >> > > >> > > "id": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8", >> > > >> > > "name": "sv3-prod", >> > > >> > > "endpoints": [ >> > > >> > > "http://sv3-ceph-rgw1:8080"; >> > > >> > > ], >> > > >> > > "log_meta": "false", >> > > >> > > "log_data": "true", >> > > >> > > "bucket_index_max_shards": 0, >> > > >> > > "read_only": "false", >> > > >> > > "tier_type": "", >> > > >> > > "sync_from_all": "true", >> > > >> > > "sync_from": [] >> > > >> > > } >> > > >> > > ], >> > > >> > > "placement_targets": [ >> > > >> > > { >> > > >> > > "name": "default-placement", >> > > >> > > "tags": [] >> > > >> > > } >> > > >> > > ], >> > > >> > > "default_placement": "default-placement", >> > > >> > > "realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd" >> > > >> > > } >> > > >> > > ], >> > > >> > > "short_zone_ids": [ >> > > >> > > { >> > > >> > > "key": "107d29a0-b732-4bf1-a26e-1f64f820e839", >> > > >> > > "val": 1720993486 >> > > >> > > }, >> > > >> > > { >> > > >> > > "key": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "val": 2301637458 >> > > >> > > }, >> > > >> > > { >> > > >> > > "key": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8", >> > > >> > > "val": 1449486239 >> > > >> > > } >> > > >> > > ] >> > > >> > > }, >> > > >> > > "master_zonegroup": "de6af748-1a2f-44a1-9d44-30799cf1313e", >> > > >> > > "master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04", >> > > >> > > "period_config": { >> > > >> > > "bucket_quota": { >> > > >> > > "enabled": false, >> > > >> > > "check_on_raw": false, >> > > >> > > "max_size": -1, >> > > >> > > "max_size_kb": 0, >> > > >> > > "max_objects": -1 >> > > >> > > }, >> > > >> > > "user_quota": { >> > > >> > > "enabled": false, >> > > >> > > "check_on_raw": false, >> > > >> > > "max_size": -1, >> > > >> > > "max_size_kb": 0, >> > > >> > > "max_objects": -1 >> > > >> > > } >> > > >> > > }, >> > > >> > > "realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd", >> > > >> > > "realm_name": "savagebucket", >> > > >> > > "realm_epoch": 2 >> > > >> > > } >> > > >> > > *From: *Matthew H <matthew.he...@hotmail.com >> > <mailto:matthew.he...@hotmail.com>> >> > > *Date: *Tuesday, March 5, 2019 at 4:31 AM >> > > *To: *Christian Rice <cr...@pandora.com >> > <mailto:cr...@pandora.com>>, ceph-users >> > > <ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>> >> > > *Subject: *Re: radosgw sync falling behind regularly >> > > >> > > Hi Christian, >> > > >> > > You haven't resharded any of your buckets have you? You can run >> > the >> > > command below in v12.2.11 to list stale bucket instances. >> > > >> > > radosgw-admin reshard stale-instances list >> > > >> > > Can you also send the output from the following command on each >> rgw? >> > > >> > > radosgw-admin period get >> > > >> > > >> > > _______________________________________________ >> > > ceph-users mailing list >> > > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
