Hello, We have 2 Ceph clusters running in two separate data centers, each one with 3 mons, 3 rgws, and 5 osds. I am attempting to get bi-directional multi-site replication setup as described in the ceph documentation here: http://docs.ceph.com/docs/jewel/radosgw/multisite/
We are running Jewel v 10.2.2: rpm -qa | grep ceph ceph-base-10.2.2-0.el7.x86_64 ceph-10.2.2-0.el7.x86_64 ceph-radosgw-10.2.2-0.el7.x86_64 libcephfs1-10.2.2-0.el7.x86_64 python-cephfs-10.2.2-0.el7.x86_64 ceph-selinux-10.2.2-0.el7.x86_64 ceph-mon-10.2.2-0.el7.x86_64 ceph-osd-10.2.2-0.el7.x86_64 ceph-release-1-1.el7.noarch ceph-common-10.2.2-0.el7.x86_64 ceph-mds-10.2.2-0.el7.x86_64 It appears syncing is happening, however it is not able to sync the metadata, and therefore no users/buckets from the primary are going to the secondary. Primary sync status: * radosgw-admin sync status* * realm 3af93a86-916a-490f-b38f-17922b472b19 (my_realm)* * zonegroup 235b010c-22e2-4b43-8fcc-8ae01939273e (us)* * zone 6c830b44-4e39-4e19-9bd8-03c37c2021f2 (us-dfw)* * metadata sync no sync (zone is master)* * data sync source: 58aa3eef-fc1f-492c-a08e-9c6019e7c266 (us-phx)* * syncing* * full sync: 0/128 shards* * incremental sync: 128/128 shards* * data is caught up with source* *radosgw-admin data sync status --source-zone=us-phx{ "sync_status": { "info": { "status": "sync", "num_shards": 128 }, "markers": [...}radosgw-admin metadata sync status{ "sync_status": { "info": { "status": "init", "num_shards": 0, "period": "", "realm_epoch": 0 }, "markers": [] }, "full_sync": { "total": 0, "complete": 0 }}* Secondary sync status: *radosgw-admin sync status* * realm 3af93a86-916a-490f-b38f-17922b472b19 (pardot)* * zonegroup 235b010c-22e2-4b43-8fcc-8ae01939273e (us)* * zone 58aa3eef-fc1f-492c-a08e-9c6019e7c266 (us-phx)* * metadata sync failed to read sync status: (2) No such file or directory* * data sync source: 6c830b44-4e39-4e19-9bd8-03c37c2021f2 (us-dfw)* * syncing* * full sync: 0/128 shards* * incremental sync: 128/128 shards* * data is behind on 10 shards* * oldest incremental change not applied: 2016-09-20 15:00:17.0.330225s* *radosgw-admin data sync status --source-zone=us-dfw* *{* * "sync_status": {* * "info": {* * "status": "building-full-sync-maps",* * "num_shards": 128* * },* *....* *}* *radosgw-admin metadata sync status --source-zone=us-dfwERROR: sync.read_sync_status() returned ret=-2* In the logs I am seeing (date stamps not in order to pick out non-dupes): Primary logs: 2016-09-20 15:02:44.313204 7f2a2dffb700 0 ERROR: client_io->complete_request() returned -5 2016-09-20 10:31:57.501247 7faf4bfff700 0 ERROR: failed to wait for op, ret=-11: POST http://pardot0-cephrgw1-3-phx.ops.sfdc.net:80/admin/realm/period?period=385c44c7-0506-4204-90d7-9d26a6cbaad2&epoch=12&rgwx-zonegroup=f46ce11b-ee5d-489b-aa30-752fc5353931 2016-09-20 10:32:03.391118 7fb12affd700 0 ERROR: failed to fetch datalog info 2016-09-20 10:32:03.491520 7fb12affd700 0 ERROR: lease cr failed, done early Secondary logs; 2016-09-20 10:28:15.290050 7faab2fed700 0 ERROR: failed to get bucket instance info for bucket id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1 2016-09-20 10:28:15.290108 7faab5ff3700 0 ERROR: failed to get bucket instance info for bucket id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1 2016-09-20 10:28:15.290571 7faab77f6700 0 ERROR: failed to get bucket instance info for bucket id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1 2016-09-20 10:28:15.304619 7faaad7e2700 0 ERROR: failed to get bucket instance info for bucket id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1 2016-09-20 10:28:38.169629 7fa98bfff700 0 ERROR: failed to distribute cache for .rgw.root:periods.385c44c7-0506-4204-90d7-9d26a6cbaad2.12 2016-09-20 10:28:38.169642 7fa98bfff700 -1 period epoch 12 is not newer than current epoch 12, discarding update 2016-09-21 03:19:01.550808 7fe10bfff700 0 rgw meta sync: ERROR: failed to fetch mdlog info 2016-09-21 15:45:09.799195 7fcd677fe700 0 ERROR: failed to fetch remote data log info: ret=-11 Each of those logs are repeated several times each, and constantly. Any help would be greatly appreciated. Thanks!
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com