Re: [ceph-users] Multisite Ceph setup sync issue
Hi Amit, Still same. Please see the below output. Anything else I can try? Please update [cephuser@zabbix-client ~]$ radosgw-admin period update --commit 2>/dev/null Sending period to new master zone 71931e0e-1be6-449f-af34-edb4166c4e4a [cephuser@zabbix-client ~]$ sudo systemctl start ceph-radosgw@rgw.`hostname -s` [cephuser@zabbix-client ~]$ sudo systemctl restart ceph-radosgw@rgw.`hostname -s` [cephuser@zabbix-client ~]$ sudo systemctl status ceph-radosgw@rgw.`hostname -s` ● ceph-radosgw@rgw.zabbix-client.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-01-30 22:38:15 IST; 7s ago Main PID: 8234 (radosgw) CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.zabbix-client.service └─8234 /usr/bin/radosgw -f --cluster ceph --name client.rgw.zabbix-client --setuser ceph --setgroup ceph Jan 30 22:38:15 zabbix-client systemd[1]: Started Ceph rados gateway. [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) metadata sync failed to read sync status: (2) No such file or directory data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) failed to retrieve sync info: (5) Input/output error [cephuser@zabbix-client ~]$ /Krishna From: Amit Ghadge Sent: Wednesday, January 30, 2019 9:17 PM To: Krishna Verma Cc: Casey Bodley ; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Multisite Ceph setup sync issue EXTERNAL MAIL Have you commit your changes on slave gateway? First, run commit command on slave gateway and then try. -AmitG On Wed, 30 Jan 2019, 21:06 Krishna Verma, mailto:kve...@cadence.com>> wrote: Hi Casey, Thanks for your reply, however I tried with "--source-zone" option with sync command but getting below error: Sync status From slave gateway to master zone "noida1" [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) metadata sync failed to read sync status: (2) No such file or directory data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) failed to retrieve sync info: (5) Input/output error [cephuser@zabbix-client ~]$ Sync status From Master Gateway to slave zone " san-jose": [cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone san-jose 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) metadata sync no sync (zone is master) [cephuser@zabbix-server ~]$ Zone detail from master gateway : [cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ { "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", "name": "noida1", "endpoints": [ "http:\/\/vlno-ceph01:7480" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" } [cephuser@zabbix-server ~]$ Zone detail from slave gateway: [cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1b
Re: [ceph-users] Multisite Ceph setup sync issue
Have you commit your changes on slave gateway? First, run commit command on slave gateway and then try. -AmitG On Wed, 30 Jan 2019, 21:06 Krishna Verma, wrote: > Hi Casey, > > Thanks for your reply, however I tried with "--source-zone" option with > sync command but getting below error: > > Sync status From slave gateway to master zone "noida1" > > [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone > noida1 2>/dev/null > realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) > zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) >zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) > metadata sync failed to read sync status: (2) No such file or directory > data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) > failed to retrieve sync info: (5) Input/output > error > [cephuser@zabbix-client ~]$ > > Sync status From Master Gateway to slave zone " san-jose": > > [cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone > san-jose 2>/dev/null > realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) > zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) >zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) > metadata sync no sync (zone is master) > [cephuser@zabbix-server ~]$ > > Zone detail from master gateway : > > [cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null > { > "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", > "name": "noida", > "api_name": "noida", > "is_master": "true", > "endpoints": [ > "http:\/\/zabbix-server:7480" > ], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", > "zones": [ > { > "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", > "name": "noida1", > "endpoints": [ > "http:\/\/vlno-ceph01:7480" > ], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 0, > "read_only": "false" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [] > } > ], > "default_placement": "default-placement", > "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" > } > > [cephuser@zabbix-server ~]$ > > > Zone detail from slave gateway: > > [cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null > { > "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", > "name": "noida", > "api_name": "noida", > "is_master": "true", > "endpoints": [ > "http:\/\/zabbix-server:7480" > ], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", > "zones": [ > { > "id": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac", > "name": "san-jose", > "endpoints": [ > "http:\/\/zabbix-client:7480" > ], > "log_meta": "false", > "log_data": "true", > "bucket_index_max_shards": 0, > "read_only": "false" > }, > { > "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", > "name": "noida1", > "endpoints": [ > "http:\/\/vlno-ceph01:7480" > ], > "log_meta": "false", > "log_data": "true", > "bucket_index_max_shards": 0, > "read_only": "false" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [] > } > ], > "default_placement": "default-placement", > "realm_id"
Re: [ceph-users] Multisite Ceph setup sync issue
Hi Casey, Thanks for your reply, however I tried with "--source-zone" option with sync command but getting below error: Sync status From slave gateway to master zone "noida1" [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose) metadata sync failed to read sync status: (2) No such file or directory data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) failed to retrieve sync info: (5) Input/output error [cephuser@zabbix-client ~]$ Sync status From Master Gateway to slave zone " san-jose": [cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone san-jose 2>/dev/null realm 1102c891-d81c-480e-9487-c9f874287d13 (georep) zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida) zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1) metadata sync no sync (zone is master) [cephuser@zabbix-server ~]$ Zone detail from master gateway : [cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ { "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", "name": "noida1", "endpoints": [ "http:\/\/vlno-ceph01:7480" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" } [cephuser@zabbix-server ~]$ Zone detail from slave gateway: [cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null { "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "name": "noida", "api_name": "noida", "is_master": "true", "endpoints": [ "http:\/\/zabbix-server:7480" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ { "id": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac", "name": "san-jose", "endpoints": [ "http:\/\/zabbix-client:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false" }, { "id": "71931e0e-1be6-449f-af34-edb4166c4e4a", "name": "noida1", "endpoints": [ "http:\/\/vlno-ceph01:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1102c891-d81c-480e-9487-c9f874287d13" } [cephuser@zabbix-client ~] I need your expert advice. /Krishna -Original Message- From: Casey Bodley Sent: Wednesday, January 30, 2019 1:54 AM To: Krishna Verma Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Multisite Ceph setup sync issue EXTERNAL MAIL On Tue, Jan 29, 2019 at 12:24 PM Krishna Verma wrote: > > Hi Ceph Users, > > > > I need your to fix sync issue in multisite setup. > > > > I have 2 cluster in different datacenter that we want to use for > bidirectional data replication. By followed the documentation > https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ceph.com_docs_
Re: [ceph-users] Multisite Ceph setup sync issue
On Tue, Jan 29, 2019 at 12:24 PM Krishna Verma wrote: > > Hi Ceph Users, > > > > I need your to fix sync issue in multisite setup. > > > > I have 2 cluster in different datacenter that we want to use for > bidirectional data replication. By followed the documentation > http://docs.ceph.com/docs/master/radosgw/multisite/ I have setup the gateway > on each site but when I am checking the sync status its getting failed as > below: > > > > Admin node at master : > > [cephuser@vlno-ceph01 cluster]$ radosgw-admin data sync status > > ERROR: source zone not specified > > [cephuser@vlno-ceph01 cluster]$ radosgw-admin realm list > > { > > "default_info": "1102c891-d81c-480e-9487-c9f874287d13", > > "realms": [ > > "georep", > > "geodata" > > ] > > } > > > > [cephuser@vlno-ceph01 cluster]$ radosgw-admin zonegroup list > > read_default_id : 0 > > { > > "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223", > > "zonegroups": [ > > "noida" > > ] > > } > > > > [cephuser@vlno-ceph01 cluster]$ radosgw-admin zone list > > { > > "default_info": "71931e0e-1be6-449f-af34-edb4166c4e4a", > > "zones": [ > > "noida1" > > ] > > } > > > > [cephuser@vlno-ceph01 cluster]$ > > > > [cephuser@vlno-ceph01 cluster]$ cat ceph.conf > > [global] > > fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55 > > mon_initial_members = vlno-ceph01 > > mon_host = 172.23.16.67 > > auth_cluster_required = cephx > > auth_service_required = cephx > > auth_client_required = cephx > > # Your network address > > public network = 172.23.16.0/24 > > osd pool default size = 2 > > rgw_override_bucket_index_max_shards = 100 > > debug ms = 1 > > debug rgw = 20 > > [cephuser@vlno-ceph01 cluster]$ > > > > On Master Gateway : > > > > [cephuser@zabbix-server ~]$ cat /etc/ceph/ceph.conf > > [global] > > fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55 > > mon_initial_members = vlno-ceph01 > > mon_host = 172.23.16.67 > > auth_cluster_required = cephx > > auth_service_required = cephx > > auth_client_required = cephx > > # Your network address > > public network = 172.23.16.0/24 > > osd pool default size = 2 > > rgw_override_bucket_index_max_shards = 100 > > debug ms = 1 > > debug rgw = 20 > > [client.rgw.zabbix-server] > > host = zabbix-server > > rgw frontends = "civetweb port=7480" > > rgw_zone=noida1 > > [cephuser@zabbix-server ~]$ > > > > > > On Secondary site admin node. > > > > [cephuser@vlsj-kverma1 cluster]$ radosgw-admin realm list > > { > > "default_info": "1102c891-d81c-480e-9487-c9f874287d13", > > "realms": [ > > "georep" > > ] > > } > > > > [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zonegroup list > > read_default_id : 0 > > { > > "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223", > > "zonegroups": [ > > "noida", > > "default" > > ] > > } > > > > [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zone list > > { > > "default_info": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac", > > "zones": [ > > "san-jose" > > ] > > } > > > > [cephuser@vlsj-kverma1 cluster]$ > > > > > > [cephuser@vlsj-kverma1 cluster]$ cat ceph.conf > > [global] > > fsid = c626be3a-4536-48b9-8db8-470437052313 > > mon_initial_members = vlsj-kverma1 > > mon_host = 172.18.84.131 > > auth_cluster_required = cephx > > auth_service_required = cephx > > auth_client_required = cephx > > # Your network address > > public network = 172.18.84.0/24 > > osd pool default size = 2 > > rgw_override_bucket_index_max_shards = 100 > > debug ms = 1 > > debug rgw = 20 > > > > > > [cephuser@vlsj-kverma1 cluster]$ > > > > [cephuser@vlsj-kverma1 cluster]$ radosgw-admin data sync status > > 2019-01-28 10:33:12.163298 7f11c24c79c0 1 Cannot find zone > id=45c690a8-f39c-4b1d-9faf-e0e991ceaaac (name=san-jose), switching to local > zonegroup configuration > > ERROR: source zone not specified > > [cephuser@vlsj-kverma1 cluster]$ > > > > On Secondary site Gateway host: > > > > [cephuser@zabbix-client ceph]$ cat /etc/ceph/ceph.conf > > [global] > > fsid = c626be3a-4536-48b9-8db8-470437052313 > > mon_initial_members = vlsj-kverma1 > > mon_host = 172.18.84.131 > > auth_cluster_required = cephx > > auth_service_required = cephx > > auth_client_required = cephx > > # Your network address > > public network = 172.18.84.0/24 > > osd pool default size = 2 > > rgw_override_bucket_index_max_shards = 100 > > debug ms = 1 > > debug rgw = 20 > > [client.rgw.zabbix-client] > > host = zabbix-client > > rgw frontends = "civetweb port=7480" > > rgw_zone=san-jose > > > > [cephuser@zabbix-client ceph]$ > > > > > > > > Appreciate any help in the setup. > > > > /Krishna > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com The 'radosgw-admin data sync status' command requires a --source-zone argument, which is generally the zone name on the opposite cluster. But you're probably just looking for the
[ceph-users] Multisite Ceph setup sync issue
Hi Ceph Users, I need your to fix sync issue in multisite setup. I have 2 cluster in different datacenter that we want to use for bidirectional data replication. By followed the documentation http://docs.ceph.com/docs/master/radosgw/multisite/ I have setup the gateway on each site but when I am checking the sync status its getting failed as below: Admin node at master : [cephuser@vlno-ceph01 cluster]$ radosgw-admin data sync status ERROR: source zone not specified [cephuser@vlno-ceph01 cluster]$ radosgw-admin realm list { "default_info": "1102c891-d81c-480e-9487-c9f874287d13", "realms": [ "georep", "geodata" ] } [cephuser@vlno-ceph01 cluster]$ radosgw-admin zonegroup list read_default_id : 0 { "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "zonegroups": [ "noida" ] } [cephuser@vlno-ceph01 cluster]$ radosgw-admin zone list { "default_info": "71931e0e-1be6-449f-af34-edb4166c4e4a", "zones": [ "noida1" ] } [cephuser@vlno-ceph01 cluster]$ [cephuser@vlno-ceph01 cluster]$ cat ceph.conf [global] fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55 mon_initial_members = vlno-ceph01 mon_host = 172.23.16.67 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.23.16.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [cephuser@vlno-ceph01 cluster]$ On Master Gateway : [cephuser@zabbix-server ~]$ cat /etc/ceph/ceph.conf [global] fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55 mon_initial_members = vlno-ceph01 mon_host = 172.23.16.67 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.23.16.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [client.rgw.zabbix-server] host = zabbix-server rgw frontends = "civetweb port=7480" rgw_zone=noida1 [cephuser@zabbix-server ~]$ On Secondary site admin node. [cephuser@vlsj-kverma1 cluster]$ radosgw-admin realm list { "default_info": "1102c891-d81c-480e-9487-c9f874287d13", "realms": [ "georep" ] } [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zonegroup list read_default_id : 0 { "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223", "zonegroups": [ "noida", "default" ] } [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zone list { "default_info": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac", "zones": [ "san-jose" ] } [cephuser@vlsj-kverma1 cluster]$ [cephuser@vlsj-kverma1 cluster]$ cat ceph.conf [global] fsid = c626be3a-4536-48b9-8db8-470437052313 mon_initial_members = vlsj-kverma1 mon_host = 172.18.84.131 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.18.84.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [cephuser@vlsj-kverma1 cluster]$ [cephuser@vlsj-kverma1 cluster]$ radosgw-admin data sync status 2019-01-28 10:33:12.163298 7f11c24c79c0 1 Cannot find zone id=45c690a8-f39c-4b1d-9faf-e0e991ceaaac (name=san-jose), switching to local zonegroup configuration ERROR: source zone not specified [cephuser@vlsj-kverma1 cluster]$ On Secondary site Gateway host: [cephuser@zabbix-client ceph]$ cat /etc/ceph/ceph.conf [global] fsid = c626be3a-4536-48b9-8db8-470437052313 mon_initial_members = vlsj-kverma1 mon_host = 172.18.84.131 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # Your network address public network = 172.18.84.0/24 osd pool default size = 2 rgw_override_bucket_index_max_shards = 100 debug ms = 1 debug rgw = 20 [client.rgw.zabbix-client] host = zabbix-client rgw frontends = "civetweb port=7480" rgw_zone=san-jose [cephuser@zabbix-client ceph]$ Appreciate any help in the setup. /Krishna ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com