Re: [ceph-users] Multisite Ceph setup sync issue
Hi Amit, Still same. Please see the below output. Anything else I can try? Please update [cephuser@zabbix-client ~]$ radosgw-admin period update --commit 2>/dev/null Sending period to new master zone 71931e0e-1be6-449f-af34-edb4166c4e4a [cephuser@zabbix-client ~]$ sudo systemctl start ceph-radosgw@rgw.`hostname -s` [cephuser@zabbix-client ~]$ sudo systemctl restart ceph-radosgw@rgw.`hostname -s` [cephuser@zabbix-client ~]$ sudo systemctl status ceph-radosgw@rgw.`hostname -s` ● ceph-radosgw@rgw.zabbix-client.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-01-30 22:38:15 IST; 7s ago Main PID: 8234 (radosgw) CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.zabbix-client.service └─8234 /usr/bin/radosgw -f --cluster ceph --name client.rgw.zabbix-client --setuser ceph --setgroup ceph Jan 30 22:38:15 zabbix-client systemd[1]: Started Ceph rados gateway. [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) metadata sync failed to read sync status: (2) No such file or directory data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) failed to retrieve sync info: (5) Input/output error [cephuser@zabbix-client ~]$ /Krishna From: Amit Ghadge Sent: Wednesday, January 30, 2019 9:17 PM To: Krishna Verma Cc: Casey Bodley ; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Multisite Ceph setup sync issue EXTERNAL MAIL Have you commit your changes on slave gateway? First, run commit command on slave gateway and then try. -AmitG On Wed, 30 Jan 2019, 21:06 Krishna Verma, mailto:kve...@cadence.com>> wrote: Hi Casey, Thanks for your reply, however I tried with "--source-zone" option with sync command but getting below error: Sync status From slave gateway to master zone "noida1" [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) metadata sync failed to read sync status: (2) No such file or directory data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) failed to retrieve sync info: (5) Input/output error [cephuser@zabbix-client ~]$ Sync status From Master Gateway to slave zone " san-jose": [cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone san-jose 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) metadata sync no sync (zone is master) [cephuser@zabbix-server ~]$ Zone detail from master gateway : [cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ { "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", "name": "noida1", "endpoints": [ "http:\/\/vlno-ceph01:7480" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" } [cephuser@zabbix-server ~]$ Zone detail from slave gateway: [cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1b
Re: [ceph-users] Multisite Ceph setup sync issue
Hi Casey, Thanks for your reply, however I tried with "--source-zone" option with sync command but getting below error: Sync status From slave gateway to master zone "noida1" [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) metadata sync failed to read sync status: (2) No such file or directory data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) failed to retrieve sync info: (5) Input/output error [cephuser@zabbix-client ~]$ Sync status From Master Gateway to slave zone " san-jose": [cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone san-jose 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) metadata sync no sync (zone is master) [cephuser@zabbix-server ~]$ Zone detail from master gateway : [cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ { "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", "name": "noida1", "endpoints": [ "http:\/\/vlno-ceph01:7480" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" } [cephuser@zabbix-server ~]$ Zone detail from slave gateway: [cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ { "id": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac", "name": "san-jose", "endpoints": [ "http:\/\/zabbix-client:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false" }, { "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", "name": "noida1", "endpoints": [ "http:\/\/vlno-ceph01:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" } [cephuser@zabbix-client ~] I need your expert advice. /Krishna -Original Message- From: Casey Bodley Sent: Wednesday, January 30, 2019 1:54 AM To: Krishna Verma Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Multisite Ceph setup sync issue EXTERNAL MAIL On Tue, Jan 29, 2019 at 12:24 PM Krishna Verma wrote: > > Hi Ceph Users, > > > > I need your to fix sync issue in multisite setup. > > > > I have 2 cluster in different datacenter that we want to use for > bidirectional data replication. By followed the documentation > https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ceph.com_do
[ceph-users] Multisite Ceph setup sync issue
Hi Ceph Users, I need your to fix sync issue in multisite setup. I have 2 cluster in different datacenter that we want to use for bidirectional data replication. By followed the documentation http://docs.ceph.com/docs/master/radosgw/multisite/ I have setup the gateway on each site but when I am checking the sync status its getting failed as below: Admin node at master : [cephuser@vlno-ceph01 cluster]$ radosgw-admin data sync status ERROR: source zone not specified [cephuser@vlno-ceph01 cluster]$ radosgw-admin realm list { "default_info": "1102c891-d81c-480e-9487-c9f874287d13", "realms": [ "georep", "geodata" ] } [cephuser@vlno-ceph01 cluster]$ radosgw-admin zonegroup list read_default_id : 0 { "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "zonegroups": [ "noida" ] } [cephuser@vlno-ceph01 cluster]$ radosgw-admin zone list { "default_info": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ "noida1" ] } [cephuser@vlno-ceph01 cluster]$ [cephuser@vlno-ceph01 cluster]$ cat ceph.conf [global] fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55 mon_initial_members = vlno-ceph01 mon_host = 172.23.16.67 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.23.16.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [cephuser@vlno-ceph01 cluster]$ On Master Gateway : [cephuser@zabbix-server ~]$ cat /etc/ceph/ceph.conf [global] fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55 mon_initial_members = vlno-ceph01 mon_host = 172.23.16.67 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.23.16.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [client.rgw.zabbix-server] host = zabbix-server rgw frontends = "civetweb port=7480" rgw_zone=noida1 [cephuser@zabbix-server ~]$ On Secondary site admin node. [cephuser@vlsj-kverma1 cluster]$ radosgw-admin realm list { "default_info": "1102c891-d81c-480e-9487-c9f874287d13", "realms": [ "georep" ] } [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zonegroup list read_default_id : 0 { "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "zonegroups": [ "noida", "default" ] } [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zone list { "default_info": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac", "zones": [ "san-jose" ] } [cephuser@vlsj-kverma1 cluster]$ [cephuser@vlsj-kverma1 cluster]$ cat ceph.conf [global] fsid = c626be3a-4536-48b9-8db8-470437052313 mon_initial_members = vlsj-kverma1 mon_host = 172.18.84.131 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.18.84.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [cephuser@vlsj-kverma1 cluster]$ [cephuser@vlsj-kverma1 cluster]$ radosgw-admin data sync status 2019-01-28 10:33:12.163298 7f11c24c79c0 1 Cannot find zone id=45c690a8-f39c-4b1d-9faf-e0e991ceaaac (name=san-jose), switching to local zonegroup configuration ERROR: source zone not specified [cephuser@vlsj-kverma1 cluster]$ On Secondary site Gateway host: [cephuser@zabbix-client ceph]$ cat /etc/ceph/ceph.conf [global] fsid = c626be3a-4536-48b9-8db8-470437052313 mon_initial_members = vlsj-kverma1 mon_host = 172.18.84.131 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.18.84.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [client.rgw.zabbix-client] host = zabbix-client rgw frontends = "civetweb port=7480" rgw_zone=san-jose [cephuser@zabbix-client ceph]$ Appreciate any help in the setup. /Krishna ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com