Re: [Gluster-users] Gluster Geo replication
We are revisiting geo-replication on some Centos7 systems running 3.8 so I am not sure this observation will help with something newer. But we were getting burned by the systems connect locally using ::1 as the localhost as opposed to 127.0.0.1. This was caused by the hosts file having an IPv6 localhost line. On 2023-11-03 06:13, Aravinda wrote: While creating the Geo-replication session it mounts the secondary Volume to see the available size. To mount the secondary volume in Primary, port 24007 and 49152-49664 of the secondary volume needs to be accessible from the Primary (Only in the node from where the Geo-rep create command is executed). This need to be changed to use SSH(bug). Alternatively use georep setup tool from https://github.com/aravindavk/gluster-georep-tools. This tool only uses Port 22 of SSH. Once the Geo-rep session is created, all the communication and the data transfer happens via SSH(Default port: 22). Aravinda Kadalu Technologies https://kadalu.tech On Tue, 31 Oct 2023 08:40:17 +0530 *dev devops * wrote --- Hi All, What are the ports needed to be opened for Gluster Geo replication ? We have a very closed setup, I could gather below info, does all of these ports need to be open on master and slave for inter communication or just 22 would work since it's using the rsync over ssh for actual data push ? *•**Port 22 (TCP):* Used by SSH for secure data communication in Geo-replication. *•**Port 24007 (TCP):* Used by the Gluster daemon (glusterd) for management and to intercommunicate with other glusterd instances. *•**Ports 24008 & 24009 (TCP):* Used for GlusterFS data and metadata operations. *•**Port 49152 to 49664 (TCP):* Used by GlusterFS for client connections. I see some monitoring happening on the tcp ports for slave volume, is this communication secure ? Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge:https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster Geo replication
While creating the Geo-replication session it mounts the secondary Volume to see the available size. To mount the secondary volume in Primary, port 24007 and 49152-49664 of the secondary volume needs to be accessible from the Primary (Only in the node from where the Geo-rep create command is executed). This need to be changed to use SSH(bug). Alternatively use georep setup tool from https://github.com/aravindavk/gluster-georep-tools. This tool only uses Port 22 of SSH. Once the Geo-rep session is created, all the communication and the data transfer happens via SSH(Default port: 22). Aravinda Kadalu Technologies https://kadalu.tech On Tue, 31 Oct 2023 08:40:17 +0530 dev devops wrote --- Hi All, What are the ports needed to be opened for Gluster Geo replication ? We have a very closed setup, I could gather below info, does all of these ports need to be open on master and slave for inter communication or just 22 would work since it's using the rsync over ssh for actual data push ? • Port 22 (TCP): Used by SSH for secure data communication in Geo-replication. • Port 24007 (TCP): Used by the Gluster daemon (glusterd) for management and to intercommunicate with other glusterd instances. • Ports 24008 & 24009 (TCP): Used for GlusterFS data and metadata operations. • Port 49152 to 49664 (TCP): Used by GlusterFS for client connections. I see some monitoring happening on the tcp ports for slave volume, is this communication secure ? Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list mailto:Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster Geo replication
Hi, You simply need to enable port 22 on the geo-replication slave side. This will allow the master node to establish an SSH connection with the slave server and transfer data securely over SSH. Thanks, Anant From: Gluster-users on behalf of dev devops Sent: 31 October 2023 3:10 AM To: gluster-users@gluster.org Subject: [Gluster-users] Gluster Geo replication EXTERNAL: Do not click links or open attachments if you do not recognize the sender. Hi All, What are the ports needed to be opened for Gluster Geo replication ? We have a very closed setup, I could gather below info, does all of these ports need to be open on master and slave for inter communication or just 22 would work since it's using the rsync over ssh for actual data push ? • Port 22 (TCP): Used by SSH for secure data communication in Geo-replication. • Port 24007 (TCP): Used by the Gluster daemon (glusterd) for management and to intercommunicate with other glusterd instances. • Ports 24008 & 24009 (TCP): Used for GlusterFS data and metadata operations. • Port 49152 to 49664 (TCP): Used by GlusterFS for client connections. I see some monitoring happening on the tcp ports for slave volume, is this communication secure ? DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster GEO replication fault after write over nfs-ganesha
CCIng sunn as well. On 28/03/19 4:05 PM, Soumya Koduri wrote: On 3/27/19 7:39 PM, Alexey Talikov wrote: I have two clusters with dispersed volumes (2+1) with GEO replication It works fine till I use glusterfs-fuse, but as even one file written over nfs-ganesha replication goes to Fault and recovers after I remove this file (sometimes after stop/start) I think nfs-hanesha writes file in some way that produces problem with replication I am not much familiar with geo-rep and not sure what/why exactly failed here. Request Kotresh (cc'ed) to take a look and provide his insights on the issue. Thanks, Soumya |OSError: [Errno 61] No data available: '.gfid/9c9514ce-a310-4a1c-a87b-a800a32a99f8' | but if I check over glusterfs mounted with aux-gfid-mount |getfattr -n trusted.glusterfs.pathinfo -e text /mnt/TEST/.gfid/9c9514ce-a310-4a1c-a87b-a800a32a99f8 getfattr: Removing leading '/' from absolute path names # file: mnt/TEST/.gfid/9c9514ce-a310-4a1c-a87b-a800a32a99f8 trusted.glusterfs.pathinfo="( ( ))" | File exists Details available here https://github.com/nfs-ganesha/nfs-ganesha/issues/408 ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster GEO replication fault after write over nfs-ganesha
On 3/27/19 7:39 PM, Alexey Talikov wrote: I have two clusters with dispersed volumes (2+1) with GEO replication It works fine till I use glusterfs-fuse, but as even one file written over nfs-ganesha replication goes to Fault and recovers after I remove this file (sometimes after stop/start) I think nfs-hanesha writes file in some way that produces problem with replication I am not much familiar with geo-rep and not sure what/why exactly failed here. Request Kotresh (cc'ed) to take a look and provide his insights on the issue. Thanks, Soumya |OSError: [Errno 61] No data available: '.gfid/9c9514ce-a310-4a1c-a87b-a800a32a99f8' | but if I check over glusterfs mounted with aux-gfid-mount |getfattr -n trusted.glusterfs.pathinfo -e text /mnt/TEST/.gfid/9c9514ce-a310-4a1c-a87b-a800a32a99f8 getfattr: Removing leading '/' from absolute path names # file: mnt/TEST/.gfid/9c9514ce-a310-4a1c-a87b-a800a32a99f8 trusted.glusterfs.pathinfo="( ( ))" | File exists Details available here https://github.com/nfs-ganesha/nfs-ganesha/issues/408 ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster geo-replication rsync error 3
Hi Christos, few month ago i had a similar problem but on ubuntu 16.04. At that time Kotresh gave me a hint : https://www.spinics.net/lists/gluster-users/msg33694.html gluster volume geo-replication :: config access_mount true this hint solved my problem on ubuntu 16.04. hope that helps... best regards Dietmar On 06.10.2018 10:58, Christos Tsalidis wrote: Hi all, I am testing a gluster geo-replication setup in glusterfs 3.12.14 version on CentOS Linux release 7.5.1804 and getting a faulty session due to rsync. It returns error 3. After I start the session, it goes from initializing, then to active and finally to faulty. Here is what I can see in logs. cat /var/log/glusterfs/geo-replication/mastervol/ssh%3A%2F%2Fgeoaccount%4010.0.2.13%3Agluster%3A%2F%2F127.0.0.1%3Aslavevol.log [2018-10-06 08:55:02.246958] I [monitor(monitor):280:monitor] Monitor: starting gsyncd worker brick=/bricks/brick-a1/brick slave_node=ssh://geoaccount@servere:gluster://localhost:slavevol [2018-10-06 08:55:02.503489] I [resource(/bricks/brick-a1/brick):1780:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-10-06 08:55:02.515492] I [changelogagent(/bricks/brick-a1/brick):73:__init__] ChangelogAgent: Agent listining... [2018-10-06 08:55:04.571449] I [resource(/bricks/brick-a1/brick):1787:connect_remote] SSH: SSH connection between master and slave established. duration=2.0676 [2018-10-06 08:55:04.571890] I [resource(/bricks/brick-a1/brick):1502:connect] GLUSTER: Mounting gluster volume locally... [2018-10-06 08:55:05.693440] I [resource(/bricks/brick-a1/brick):1515:connect] GLUSTER: Mounted gluster volume duration=1.1212 [2018-10-06 08:55:05.693741] I [gsyncd(/bricks/brick-a1/brick):799:main_i] : Closing feedback fd, waking up the monitor [2018-10-06 08:55:07.711970] I [master(/bricks/brick-a1/brick):1518:register] _GMaster: Working dir path=/var/lib/misc/glusterfsd/mastervol/ssh%3A%2F%2Fgeoaccount%4010.0.2.13%3Agluster%3A%2F%2F127.0.0.1%3Aslavevol/9517ac67e25c7491f03ba5e2506505bd [2018-10-06 08:55:07.712357] I [resource(/bricks/brick-a1/brick):1662:service_loop] GLUSTER: Register time time=1538816107 [2018-10-06 08:55:07.764151] I [master(/bricks/brick-a1/brick):490:mgmt_lock] _GMaster: Got lock Becoming ACTIVE brick=/bricks/brick-a1/brick [2018-10-06 08:55:07.768949] I [gsyncdstatus(/bricks/brick-a1/brick):276:set_active] GeorepStatus: Worker Status Change status=Active [2018-10-06 08:55:07.770529] I [gsyncdstatus(/bricks/brick-a1/brick):248:set_worker_crawl_status] GeorepStatus: Crawl Status Changestatus=History Crawl [2018-10-06 08:55:07.770975] I [master(/bricks/brick-a1/brick):1432:crawl] _GMaster: starting history crawl turns=1 stime=(1538745843, 0) entry_stime=None etime=1538816107 [2018-10-06 08:55:08.773402] I [master(/bricks/brick-a1/brick):1461:crawl] _GMaster: slave's time stime=(1538745843, 0) [2018-10-06 08:55:09.262964] I [master(/bricks/brick-a1/brick):1863:syncjob] Syncer: Sync Time Taken duration=0.0606 num_files=1job=2 return_code=3 [2018-10-06 08:55:09.263253] E [resource(/bricks/brick-a1/brick):210:errlog] Popen: command returned error cmd=rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-wVbxGU/05b8d7b5dab75575689c0e1a2ec33b3f.sock --compress geoaccount@servere:/proc/12335/cwd error=3 [2018-10-06 08:55:09.275593] I [syncdutils(/bricks/brick-a1/brick):271:finalize] : exiting. [2018-10-06 08:55:09.279442] I [repce(/bricks/brick-a1/brick):92:service_loop] RepceServer: terminating on reaching EOF. [2018-10-06 08:55:09.279936] I [syncdutils(/bricks/brick-a1/brick):271:finalize] : exiting. [2018-10-06 08:55:09.698153] I [monitor(monitor):363:monitor] Monitor: worker died in startup phase brick=/bricks/brick-a1/brick [2018-10-06 08:55:09.707330] I [gsyncdstatus(monitor):243:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-10-06 08:55:19.888017] I [monitor(monitor):280:monitor] Monitor: starting gsyncd worker brick=/bricks/brick-a1/brick slave_node=ssh://geoaccount@servere:gluster://localhost:slavevol [2018-10-06 08:55:20.140819] I [resource(/bricks/brick-a1/brick):1780:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-10-06 08:55:20.141815] I [changelogagent(/bricks/brick-a1/brick):73:__init__] ChangelogAgent: Agent listining... [2018-10-06 08:55:22.245625] I [resource(/bricks/brick-a1/brick):1787:connect_remote] SSH: SSH connection between master and slave established. duration=2.1046 [2018-10-06 08:55:22.246062] I [resource(/bricks/brick-a1/brick):1502:connect] GLUSTER: Mounting gluster volume locally... [2018-10-06 08:55:23.370100] I
Re: [Gluster-users] Gluster geo replication volume is faulty
On 09/29/2017 09:30 PM, rick sanchez wrote: I am trying to get up geo replication between two gluster volumes I have set up two replica 2 arbiter 1 volumes with 9 bricks [root@gfs1 ~]# gluster volume info Volume Name: gfsvol Type: Distributed-Replicate Volume ID: c2fb4365-480b-4d37-8c7d-c3046bca7306 Status: Started Snapshot Count: 0 Number of Bricks: 3 x (2 + 1) = 9 Transport-type: tcp Bricks: Brick1: gfs2:/gfs/brick1/gv0 Brick2: gfs3:/gfs/brick1/gv0 Brick3: gfs1:/gfs/arbiter/gv0 (arbiter) Brick4: gfs1:/gfs/brick1/gv0 Brick5: gfs3:/gfs/brick2/gv0 Brick6: gfs2:/gfs/arbiter/gv0 (arbiter) Brick7: gfs1:/gfs/brick2/gv0 Brick8: gfs2:/gfs/brick2/gv0 Brick9: gfs3:/gfs/arbiter/gv0 (arbiter) Options Reconfigured: nfs.disable: on transport.address-family: inet geo-replication.indexing: on geo-replication.ignore-pid-check: on changelog.changelog: on [root@gfs4 ~]# gluster volume info Volume Name: gfsvol_rep Type: Distributed-Replicate Volume ID: 42bfa062-ad0d-4242-a813-63389be1c404 Status: Started Snapshot Count: 0 Number of Bricks: 3 x (2 + 1) = 9 Transport-type: tcp Bricks: Brick1: gfs5:/gfs/brick1/gv0 Brick2: gfs6:/gfs/brick1/gv0 Brick3: gfs4:/gfs/arbiter/gv0 (arbiter) Brick4: gfs4:/gfs/brick1/gv0 Brick5: gfs6:/gfs/brick2/gv0 Brick6: gfs5:/gfs/arbiter/gv0 (arbiter) Brick7: gfs4:/gfs/brick2/gv0 Brick8: gfs5:/gfs/brick2/gv0 Brick9: gfs6:/gfs/arbiter/gv0 (arbiter) Options Reconfigured: nfs.disable: on transport.address-family: inet I set up passwordless ssh login from all the master servers to all the slave servers then created and started the geo replicated volume Passwordless SSH not required for all nodes, it is required from one of the master node to one of the slave node. (From the master node where you want to run create command to the slave node which will be specified in the Create command) Alternatively you can use a tool called "gluster-georep-setup", which doesn't require initial passwordless step. http://aravindavk.in/blog/gluster-georep-tools/ https://github.com/aravindavk/gluster-georep-tools I check the status and they switch between being active with history crawl and faulty with n/a every few second MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED - gfs1 gfsvol /gfs/arbiter/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs1 gfsvol /gfs/brick1/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep gfs6 Active History Crawl 2017-09-28 23:30:19 gfs1 gfsvol /gfs/brick2/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs3 gfsvol /gfs/brick1/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs3 gfsvol /gfs/brick2/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs3 gfsvol /gfs/arbiter/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs2 gfsvol /gfs/brick1/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs2 gfsvol /gfs/arbiter/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A gfs2 gfsvol /gfs/brick2/gv0 geo-rep-user geo-rep-user@gfs4::gfsvol_rep N/A Faulty N/A N/A Here is the output of the geo replication log file [root@gfs1 ~]# tail -n 100 $(gluster volume geo-replication gfsvol geo-rep-user@gfs4::gfsvol_rep config log-file) [2017-09-29 15:53:29.785386] I [master(/gfs/brick2/gv0):1860:syncjob] Syncer: Sync Time Takenduration=0.0357num_files=1job=3return_code=12 [2017-09-29 15:53:29.785615] E [resource(/gfs/brick2/gv0):208:errlog] Popen: command returned errorcmd=rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-fdyDHm/78cf8b204207154de59d7ac32eee737f.sock --compress geo-rep-user@gfs6:/proc/17554/cwderror=12 [2017-09-29 15:53:29.797259] I [syncdutils(/gfs/brick2/gv0):271:finalize] : exiting. [2017-09-29 15:53:29.799386] I [repce(/gfs/brick2/gv0):92:service_loop] RepceServer: terminating on reaching EOF. [2017-09-29 15:53:29.799570] I [syncdutils(/gfs/brick2/gv0):271:finalize] : exiting. [2017-09-29 15:53:30.105407] I [monitor(monitor):280:monitor] Monitor: starting gsyncd workerbrick=/gfs/brick1/gv0slave_node=ssh://geo-rep-user@gfs6:gluster://localhost:gfsvol_rep [2017-09-29
Re: [Gluster-users] Gluster geo-replication failure
Looks like SELinux issue. As a workaround use the other method mentioned
in documentation
gluster system:: gsec_create
regards
Aravinda
On 04/14/2017 12:36 AM, Tanner Bruce wrote:
I have a 4 node gluster cluster, and a separate 1 node cluster that I
want to setup geo-replication on.
I have my main volume on the 4 node cluster.
On the slave node, I've setup the geoaccount/geogroup and mountbroker
according to these steps:
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Geo%20Replication/
I am able to ssh from both the root user and ubuntu user to all of
gluster-{0,1,2,3}.grandukevpc as well as
gluster-replica.grandukevpc however when I call sudo
gluster-georep-sshkey generate I receive a python stack trace that
ends with the rather unintuitive error:
gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Commit failed
on gluster-2.grandukevpc. Error: Unable to end. Error :
Success\nCommit failed on gluster-3.grandukevpc. Error: Unable to end.
Error : Success\nCommit failed on gluster-1.grandukevpc. Error: Unable
to end. Error : Success\n', 'gluster system:: execute georep-sshkey.py
node-generate .')
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
