Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad, A few days back, you had some issue with the replicate volumes and you had a log like this ``[2020-06-16 07:19:27.418884] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol: xlator_api`` present in the glusterd.log file. So, I just want to clarify which steps you followed that led to such a log, as it was not reproducible by me. On running a few common gluster commands like create, start, stop, status. If there is any other configuration that you made in the volume or performed any other operation, then please let me know. Regards Nikhil Ladha Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad, Glad to hear that your problem is resolved. Thanks Strahil and Hubert for your suggestions. On Wed, Jun 17, 2020 at 12:29 PM ahemad shaik wrote: > Hi > > I tried starting and enabling the glusterfsd service suggested by Hubert > and Strahil, I see that works when one of the gluster volume is not > available and client still able to access the mount point. > > Thanks so much Strahil , Hubert and Karthik on your suggestion and for the > time. > > can you please help on making data consistent in all nodes when we have > some 5 hours of down time and one of the server . how to achieve data > consistency in all 3 nodes. > When the node/brick which was down comes back up, gluster self heal daemon (glustershd) will automatically do the syncing of the data to the down brick and make it consistent with the good copies. You can alternatively run the index heal command "gluster volume heal " to trigger the heal manually and you can see the entries needing heal and the progress of heal by running "gluster volume heal info". HTH, Karthik > > Any documentation on that end will be helpful. > > Thanks, > Ahemad > > On Wednesday, 17 June, 2020, 12:03:06 pm IST, Karthik Subrahmanya < > ksubr...@redhat.com> wrote: > > > Hi Ahemad, > > Sorry for a lot of back and forth on this. But we might need a few more > details to find the actual cause here. > What version of gluster you are running on server and client nodes? > Also provide the statedump [1] of the bricks and the client process when > the hang is seen. > > [1] https://docs.gluster.org/en/latest/Troubleshooting/statedump/ > > Regards, > Karthik > > On Wed, Jun 17, 2020 at 9:25 AM ahemad_sh...@yahoo.com < > ahemad_sh...@yahoo.com> wrote: > > I have a 3 replica gluster volume created in 3 nodes and when one node is > down due to some issue and the clients not able access volume. This was the > issue. I have fixed the server and it is back. There was downtime at > client. I just want to avoid the downtime since it is 3 replica. > > I am testing the high availability now by making one of the brick server > rebooting or shut down manually. I just want to make volume accessible > always by client. That is the reason we went for replica volume. > > So I just would like to know how to make the client volume high available > even some VM or node which is having gluster volume goes down unexpectedly > had down time of 10 hours. > > > > Glusterfsd service is used to stop which is disabled in my cluster and I > see one more service running gluserd. > > Will starting glusterfsd service in all 3 replica nodes will help in > achieving what I am trying. > > Hope I am clear. > > Thanks, > Ahemad > > > > Thanks, > Ahemad > > > > On Tue, Jun 16, 2020 at 23:12, Strahil Nikolov > wrote: > In my cluster , the service is enabled and running. > > What actually is your problem ? > When a gluster brick process dies unexpectedly - all fuse clients will be > waiting for the timeout . > The service glusterfsd is ensuring that during system shutdown , the > brick procesees will be shutdown in such way that all native clients won't > 'hang' and wait for the timeout, but will directly choose another brick. > > The same happens when you manually run the kill script - all gluster > processes shutdown and all clients are redirected to another brick. > > Keep in mind that fuse mounts will also be killed both by the script and > the glusterfsd service. > > Best Regards, > Strahil Nikolov > > На 16 юни 2020 г. 19:48:32 GMT+03:00, ahemad shaik > написа: > > Hi Strahil, > >I have the gluster setup on centos 7 cluster.I see glusterfsd service > >and it is in inactive state. > >systemctl status glusterfsd.service● glusterfsd.service - GlusterFS > >brick processes (stopping only) Loaded: loaded > >(/usr/lib/systemd/system/glusterfsd.service; disabled; vendor preset: > >disabled) Active: inactive (dead) > > > >so you mean starting this service in all the nodes where gluster > >volumes are created, will solve the issue ? > > > >Thanks,Ahemad > > > > > >On Tuesday, 16 June, 2020, 10:12:22 pm IST, Strahil Nikolov > > wrote: > > > > Hi ahemad, > > > >the script kills all gluster processes, so the clients won't wait > >for the timeout before switching to another node in the TSP. > > > >In CentOS/RHEL, there is a systemd service called > >'glusterfsd.service' that is taking care on shutdown to kill all > >processes, so clients won't hung. > > > >systemctl cat glusterfsd.service --no-pager > ># /usr/lib/systemd/system/glusterfsd.service > >[Unit] > >Description=GlusterFS brick processes (stopping only) > >After=network.target glusterd.service > > > >[Service] > >Type=oneshot > ># glusterd starts the glusterfsd processed on-demand > ># /bin/true will mark this service as started, RemainAfterExit keeps it > >active > >ExecStart=/bin/true > >RemainAfterExit=yes > ># if there are no glusterfsd processes, a stop/reload should not give > >an error > >ExecStop=/bin/sh -c
Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad, Sorry for a lot of back and forth on this. But we might need a few more details to find the actual cause here. What version of gluster you are running on server and client nodes? Also provide the statedump [1] of the bricks and the client process when the hang is seen. [1] https://docs.gluster.org/en/latest/Troubleshooting/statedump/ Regards, Karthik On Wed, Jun 17, 2020 at 9:25 AM ahemad_sh...@yahoo.com < ahemad_sh...@yahoo.com> wrote: > I have a 3 replica gluster volume created in 3 nodes and when one node is > down due to some issue and the clients not able access volume. This was the > issue. I have fixed the server and it is back. There was downtime at > client. I just want to avoid the downtime since it is 3 replica. > > I am testing the high availability now by making one of the brick server > rebooting or shut down manually. I just want to make volume accessible > always by client. That is the reason we went for replica volume. > > So I just would like to know how to make the client volume high available > even some VM or node which is having gluster volume goes down unexpectedly > had down time of 10 hours. > > > > Glusterfsd service is used to stop which is disabled in my cluster and I > see one more service running gluserd. > > Will starting glusterfsd service in all 3 replica nodes will help in > achieving what I am trying. > > Hope I am clear. > > Thanks, > Ahemad > > > > Thanks, > Ahemad > > > > On Tue, Jun 16, 2020 at 23:12, Strahil Nikolov > wrote: > In my cluster , the service is enabled and running. > > What actually is your problem ? > When a gluster brick process dies unexpectedly - all fuse clients will be > waiting for the timeout . > The service glusterfsd is ensuring that during system shutdown , the > brick procesees will be shutdown in such way that all native clients won't > 'hang' and wait for the timeout, but will directly choose another brick. > > The same happens when you manually run the kill script - all gluster > processes shutdown and all clients are redirected to another brick. > > Keep in mind that fuse mounts will also be killed both by the script and > the glusterfsd service. > > Best Regards, > Strahil Nikolov > > На 16 юни 2020 г. 19:48:32 GMT+03:00, ahemad shaik > написа: > > Hi Strahil, > >I have the gluster setup on centos 7 cluster.I see glusterfsd service > >and it is in inactive state. > >systemctl status glusterfsd.service● glusterfsd.service - GlusterFS > >brick processes (stopping only) Loaded: loaded > >(/usr/lib/systemd/system/glusterfsd.service; disabled; vendor preset: > >disabled) Active: inactive (dead) > > > >so you mean starting this service in all the nodes where gluster > >volumes are created, will solve the issue ? > > > >Thanks,Ahemad > > > > > >On Tuesday, 16 June, 2020, 10:12:22 pm IST, Strahil Nikolov > > wrote: > > > > Hi ahemad, > > > >the script kills all gluster processes, so the clients won't wait > >for the timeout before switching to another node in the TSP. > > > >In CentOS/RHEL, there is a systemd service called > >'glusterfsd.service' that is taking care on shutdown to kill all > >processes, so clients won't hung. > > > >systemctl cat glusterfsd.service --no-pager > ># /usr/lib/systemd/system/glusterfsd.service > >[Unit] > >Description=GlusterFS brick processes (stopping only) > >After=network.target glusterd.service > > > >[Service] > >Type=oneshot > ># glusterd starts the glusterfsd processed on-demand > ># /bin/true will mark this service as started, RemainAfterExit keeps it > >active > >ExecStart=/bin/true > >RemainAfterExit=yes > ># if there are no glusterfsd processes, a stop/reload should not give > >an error > >ExecStop=/bin/sh -c "/bin/killall --wait glusterfsd || /bin/true" > >ExecReload=/bin/sh -c "/bin/killall -HUP glusterfsd || /bin/true" > > > >[Install] > >WantedBy=multi-user.target > > > >Best Regards, > >Strahil Nikolov > > > >На 16 юни 2020 г. 18:41:59 GMT+03:00, ahemad shaik > > написа: > >> Hi, > >>I see there is a script file in below mentioned path in all nodes > >using > >>which gluster volume > >>created./usr/share/glusterfs/scripts/stop-all-gluster-processes.sh > >>I need to create a system service and when ever there is some server > >>down, we need to call this script or we need to have it run always it > >>will take care when some node is down to make sure that client will > >not > >>have any issues in accessing mount point ? > >>can you please share any documentation on how to use this.That will be > >>great help. > >>Thanks,Ahemad > >> > >> > >> > >> > >>On Tuesday, 16 June, 2020, 08:59:31 pm IST, Strahil Nikolov > >> wrote: > >> > >> Hi Ahemad, > >> > >>You can simplify it by creating a systemd service that will call > >>the script. > >> > >>It was already mentioned in a previous thread (with example), so > >>you can just use it. > >> > >>Best Regards, > >>Strahil Nikolov > >> > >>На 16 юни 2020 г. 16:02:07 GMT+03:00, Hu Bert > >>написа: >
Re: [Gluster-users] Issues with replicated gluster volume
In my cluster , the service is enabled and running. What actually is your problem ? When a gluster brick process dies unexpectedly - all fuse clients will be waiting for the timeout . The service glusterfsd is ensuring that during system shutdown , the brick procesees will be shutdown in such way that all native clients won't 'hang' and wait for the timeout, but will directly choose another brick. The same happens when you manually run the kill script - all gluster processes shutdown and all clients are redirected to another brick. Keep in mind that fuse mounts will also be killed both by the script and the glusterfsd service. Best Regards, Strahil Nikolov На 16 юни 2020 г. 19:48:32 GMT+03:00, ahemad shaik написа: > Hi Strahil, >I have the gluster setup on centos 7 cluster.I see glusterfsd service >and it is in inactive state. >systemctl status glusterfsd.service● glusterfsd.service - GlusterFS >brick processes (stopping only) Loaded: loaded >(/usr/lib/systemd/system/glusterfsd.service; disabled; vendor preset: >disabled) Active: inactive (dead) > >so you mean starting this service in all the nodes where gluster >volumes are created, will solve the issue ? > >Thanks,Ahemad > > >On Tuesday, 16 June, 2020, 10:12:22 pm IST, Strahil Nikolov > wrote: > > Hi ahemad, > >the script kills all gluster processes, so the clients won't wait >for the timeout before switching to another node in the TSP. > >In CentOS/RHEL, there is a systemd service called >'glusterfsd.service' that is taking care on shutdown to kill all >processes, so clients won't hung. > >systemctl cat glusterfsd.service --no-pager ># /usr/lib/systemd/system/glusterfsd.service >[Unit] >Description=GlusterFS brick processes (stopping only) >After=network.target glusterd.service > >[Service] >Type=oneshot ># glusterd starts the glusterfsd processed on-demand ># /bin/true will mark this service as started, RemainAfterExit keeps it >active >ExecStart=/bin/true >RemainAfterExit=yes ># if there are no glusterfsd processes, a stop/reload should not give >an error >ExecStop=/bin/sh -c "/bin/killall --wait glusterfsd || /bin/true" >ExecReload=/bin/sh -c "/bin/killall -HUP glusterfsd || /bin/true" > >[Install] >WantedBy=multi-user.target > >Best Regards, >Strahil Nikolov > >На 16 юни 2020 г. 18:41:59 GMT+03:00, ahemad shaik > написа: >> Hi, >>I see there is a script file in below mentioned path in all nodes >using >>which gluster volume >>created./usr/share/glusterfs/scripts/stop-all-gluster-processes.sh >>I need to create a system service and when ever there is some server >>down, we need to call this script or we need to have it run always it >>will take care when some node is down to make sure that client will >not >>have any issues in accessing mount point ? >>can you please share any documentation on how to use this.That will be >>great help. >>Thanks,Ahemad >> >> >> >> >>On Tuesday, 16 June, 2020, 08:59:31 pm IST, Strahil Nikolov >> wrote: >> >> Hi Ahemad, >> >>You can simplify it by creating a systemd service that will call >>the script. >> >>It was already mentioned in a previous thread (with example), so >>you can just use it. >> >>Best Regards, >>Strahil Nikolov >> >>На 16 юни 2020 г. 16:02:07 GMT+03:00, Hu Bert >>написа: >>>Hi, >>> >>>if you simply reboot or shutdown one of the gluster nodes, there >might >>>be a (short or medium) unavailability of the volume on the nodes. To >>>avoid this there's script: >>> >>>/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh (path may >>>be different depending on distribution) >>> >>>If i remember correctly: this notifies the clients that this node is >>>going to be unavailable (please correct me if the details are wrong). >>>If i do reboots of one gluster node, i always call this script and >>>never have seen unavailability issues on the clients. >>> >>> >>>Regards, >>>Hubert >>> >>>Am Mo., 15. Juni 2020 um 19:36 Uhr schrieb ahemad shaik >>>: Hi There, I have created 3 replica gluster volume with 3 bricks from 3 nodes. "gluster volume create glustervol replica 3 transport tcp >>node1:/data >>>node2:/data node3:/data force" mounted on client node using below command. "mount -t glusterfs node4:/glustervol /mnt/" when any of the node (either node1,node2 or node3) goes down, >>gluster >>>mount/volume (/mnt) not accessible at client (node4). purpose of replicated volume is high availability but not able to >>>achieve it. Is it a bug or i am missing anything. Any suggestions will be great help!!! kindly suggest. Thanks, Ahemad Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org
Re: [Gluster-users] Issues with replicated gluster volume
Hi ahemad, the script kills all gluster processes, so the clients won't wait for the timeout before switching to another node in the TSP. In CentOS/RHEL, there is a systemd service called 'glusterfsd.service' that is taking care on shutdown to kill all processes, so clients won't hung. systemctl cat glusterfsd.service --no-pager # /usr/lib/systemd/system/glusterfsd.service [Unit] Description=GlusterFS brick processes (stopping only) After=network.target glusterd.service [Service] Type=oneshot # glusterd starts the glusterfsd processed on-demand # /bin/true will mark this service as started, RemainAfterExit keeps it active ExecStart=/bin/true RemainAfterExit=yes # if there are no glusterfsd processes, a stop/reload should not give an error ExecStop=/bin/sh -c "/bin/killall --wait glusterfsd || /bin/true" ExecReload=/bin/sh -c "/bin/killall -HUP glusterfsd || /bin/true" [Install] WantedBy=multi-user.target Best Regards, Strahil Nikolov На 16 юни 2020 г. 18:41:59 GMT+03:00, ahemad shaik написа: > Hi, >I see there is a script file in below mentioned path in all nodes using >which gluster volume >created./usr/share/glusterfs/scripts/stop-all-gluster-processes.sh >I need to create a system service and when ever there is some server >down, we need to call this script or we need to have it run always it >will take care when some node is down to make sure that client will not >have any issues in accessing mount point ? >can you please share any documentation on how to use this.That will be >great help. >Thanks,Ahemad > > > > >On Tuesday, 16 June, 2020, 08:59:31 pm IST, Strahil Nikolov > wrote: > > Hi Ahemad, > >You can simplify it by creating a systemd service that will call >the script. > >It was already mentioned in a previous thread (with example), so >you can just use it. > >Best Regards, >Strahil Nikolov > >На 16 юни 2020 г. 16:02:07 GMT+03:00, Hu Bert >написа: >>Hi, >> >>if you simply reboot or shutdown one of the gluster nodes, there might >>be a (short or medium) unavailability of the volume on the nodes. To >>avoid this there's script: >> >>/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh (path may >>be different depending on distribution) >> >>If i remember correctly: this notifies the clients that this node is >>going to be unavailable (please correct me if the details are wrong). >>If i do reboots of one gluster node, i always call this script and >>never have seen unavailability issues on the clients. >> >> >>Regards, >>Hubert >> >>Am Mo., 15. Juni 2020 um 19:36 Uhr schrieb ahemad shaik >>: >>> >>> Hi There, >>> >>> I have created 3 replica gluster volume with 3 bricks from 3 nodes. >>> >>> "gluster volume create glustervol replica 3 transport tcp >node1:/data >>node2:/data node3:/data force" >>> >>> mounted on client node using below command. >>> >>> "mount -t glusterfs node4:/glustervol /mnt/" >>> >>> when any of the node (either node1,node2 or node3) goes down, >gluster >>mount/volume (/mnt) not accessible at client (node4). >>> >>> purpose of replicated volume is high availability but not able to >>achieve it. >>> >>> Is it a bug or i am missing anything. >>> >>> >>> Any suggestions will be great help!!! >>> >>> kindly suggest. >>> >>> Thanks, >>> Ahemad >>> >>> >>> >>> >>> >>> Community Meeting Calendar: >>> >>> Schedule - >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>> Bridge: https://bluejeans.com/441850968 >>> >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >>Community Meeting Calendar: >> >>Schedule - >>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>Bridge: https://bluejeans.com/441850968 >> >>Gluster-users mailing list >>Gluster-users@gluster.org >>https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Issues with replicated gluster volume
/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh is to stop the processes ? if one of the gluster node volume goes down with some issues, we didn't do reboot or shutdown, in those cases what we need to do. Will the script/systemd service created from script will be able to notify client that this node going down ? can someone please share documentation related to it. kidnly help. Thanks,AhemadOn Tuesday, 16 June, 2020, 09:11:59 pm IST, ahemad shaik wrote: Hi, I see there is a script file in below mentioned path in all nodes using which gluster volume created./usr/share/glusterfs/scripts/stop-all-gluster-processes.sh I need to create a system service and when ever there is some server down, we need to call this script or we need to have it run always it will take care when some node is down to make sure that client will not have any issues in accessing mount point ? can you please share any documentation on how to use this.That will be great help. Thanks,Ahemad On Tuesday, 16 June, 2020, 08:59:31 pm IST, Strahil Nikolov wrote: Hi Ahemad, You can simplify it by creating a systemd service that will call the script. It was already mentioned in a previous thread (with example), so you can just use it. Best Regards, Strahil Nikolov На 16 юни 2020 г. 16:02:07 GMT+03:00, Hu Bert написа: >Hi, > >if you simply reboot or shutdown one of the gluster nodes, there might >be a (short or medium) unavailability of the volume on the nodes. To >avoid this there's script: > >/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh (path may >be different depending on distribution) > >If i remember correctly: this notifies the clients that this node is >going to be unavailable (please correct me if the details are wrong). >If i do reboots of one gluster node, i always call this script and >never have seen unavailability issues on the clients. > > >Regards, >Hubert > >Am Mo., 15. Juni 2020 um 19:36 Uhr schrieb ahemad shaik >: >> >> Hi There, >> >> I have created 3 replica gluster volume with 3 bricks from 3 nodes. >> >> "gluster volume create glustervol replica 3 transport tcp node1:/data >node2:/data node3:/data force" >> >> mounted on client node using below command. >> >> "mount -t glusterfs node4:/glustervol /mnt/" >> >> when any of the node (either node1,node2 or node3) goes down, gluster >mount/volume (/mnt) not accessible at client (node4). >> >> purpose of replicated volume is high availability but not able to >achieve it. >> >> Is it a bug or i am missing anything. >> >> >> Any suggestions will be great help!!! >> >> kindly suggest. >> >> Thanks, >> Ahemad >> >> >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > >Community Meeting Calendar: > >Schedule - >Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >Bridge: https://bluejeans.com/441850968 > >Gluster-users mailing list >Gluster-users@gluster.org >https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad, You can simplify it by creating a systemd service that will call the script. It was already mentioned in a previous thread (with example), so you can just use it. Best Regards, Strahil Nikolov На 16 юни 2020 г. 16:02:07 GMT+03:00, Hu Bert написа: >Hi, > >if you simply reboot or shutdown one of the gluster nodes, there might >be a (short or medium) unavailability of the volume on the nodes. To >avoid this there's script: > >/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh (path may >be different depending on distribution) > >If i remember correctly: this notifies the clients that this node is >going to be unavailable (please correct me if the details are wrong). >If i do reboots of one gluster node, i always call this script and >never have seen unavailability issues on the clients. > > >Regards, >Hubert > >Am Mo., 15. Juni 2020 um 19:36 Uhr schrieb ahemad shaik >: >> >> Hi There, >> >> I have created 3 replica gluster volume with 3 bricks from 3 nodes. >> >> "gluster volume create glustervol replica 3 transport tcp node1:/data >node2:/data node3:/data force" >> >> mounted on client node using below command. >> >> "mount -t glusterfs node4:/glustervol/mnt/" >> >> when any of the node (either node1,node2 or node3) goes down, gluster >mount/volume (/mnt) not accessible at client (node4). >> >> purpose of replicated volume is high availability but not able to >achieve it. >> >> Is it a bug or i am missing anything. >> >> >> Any suggestions will be great help!!! >> >> kindly suggest. >> >> Thanks, >> Ahemad >> >> >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > >Community Meeting Calendar: > >Schedule - >Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >Bridge: https://bluejeans.com/441850968 > >Gluster-users mailing list >Gluster-users@gluster.org >https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Issues with replicated gluster volume
Hi, if you simply reboot or shutdown one of the gluster nodes, there might be a (short or medium) unavailability of the volume on the nodes. To avoid this there's script: /usr/share/glusterfs/scripts/stop-all-gluster-processes.sh (path may be different depending on distribution) If i remember correctly: this notifies the clients that this node is going to be unavailable (please correct me if the details are wrong). If i do reboots of one gluster node, i always call this script and never have seen unavailability issues on the clients. Regards, Hubert Am Mo., 15. Juni 2020 um 19:36 Uhr schrieb ahemad shaik : > > Hi There, > > I have created 3 replica gluster volume with 3 bricks from 3 nodes. > > "gluster volume create glustervol replica 3 transport tcp node1:/data > node2:/data node3:/data force" > > mounted on client node using below command. > > "mount -t glusterfs node4:/glustervol/mnt/" > > when any of the node (either node1,node2 or node3) goes down, gluster > mount/volume (/mnt) not accessible at client (node4). > > purpose of replicated volume is high availability but not able to achieve it. > > Is it a bug or i am missing anything. > > > Any suggestions will be great help!!! > > kindly suggest. > > Thanks, > Ahemad > > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Issues with replicated gluster volume
mount with -o backup-volfile-servers=: On Mon, Jun 15, 2020 at 1:36 PM ahemad shaik wrote: > Hi There, > > I have created 3 replica gluster volume with 3 bricks from 3 nodes. > > "gluster volume create glustervol replica 3 transport tcp node1:/data > node2:/data node3:/data force" > > mounted on client node using below command. > > "mount -t glusterfs node4:/glustervol/mnt/" > > when any of the node (either node1,node2 or node3) goes down, gluster > mount/volume (/mnt) not accessible at client (node4). > > purpose of replicated volume is high availability but not able to achieve > it. > > Is it a bug or i am missing anything. > > > Any suggestions will be great help!!! > > kindly suggest. > > Thanks, > Ahemad > > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad,
The logs don't seem to indicate anything specific.
Except this message in the glusterd logs, which I am not sure whether it
might cause any problems
[2020-06-16 07:19:27.418884] E [MSGID: 101097]
[xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
/usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol:
xlator_api
@Sanju Rakonde could you please let us know what this
error message means and whether this could be leading to this specific
problem?
@ahemad shaik replica 3 volumes will be available
when 1 of 3 bicks is down. Could you also paste the exact error that you
are getting while accessing or doing any operations on the mount? Did you
try rebooting any other node and facing the same problem?
To check whether there is any issue with the client connection can you give
the output of this command from node4?
cat /.meta/graphs/active/-client-*/private | egrep -i
'connected'
In your case the actual command might look like this
cat /mnt/.meta/graphs/active/glustervol-client-*/private | egrep -i
'connected'
Regards,
Karthik
On Tue, Jun 16, 2020 at 1:31 PM ahemad shaik wrote:
> Hi Karthik,
>
> Please find attached logs.
>
> kindly suggest on how to make the volume high available.
>
> Thanks,
> Ahemad
>
>
>
> On Tuesday, 16 June, 2020, 12:09:10 pm IST, Karthik Subrahmanya <
> ksubr...@redhat.com> wrote:
>
>
> Hi,
>
> Thanks for the clarification.
> In that case can you attach complete glusterd, bricks and mount logs from
> all the nodes when this happened.
> Also paste the output that you are seeing when you try to access or do
> operations on the mount point.
>
> Regards,
> Karthik
>
> On Tue, Jun 16, 2020 at 11:55 AM ahemad shaik
> wrote:
>
> Sorry, It was a typo.
>
> The command i exact command i have used is below.
>
> The volume is mounted on node4.
>
> ""mount -t glusterfs node1:/glustervol /mnt/ ""
>
>
> gluster Volume is created from node1,node2 and node3.
>
> ""gluster volume create glustervol replica 3 transport tcp node1:/data
> node2:/data node3:/data force""
>
> I have tried rebooting node3 to test high availability.
>
> I hope it is clear now.
>
> Please let me know if any questions.
>
> Thanks,
> Ahemad
>
>
>
> On Tuesday, 16 June, 2020, 11:45:48 am IST, Karthik Subrahmanya <
> ksubr...@redhat.com> wrote:
>
>
> Hi Ahemad,
>
> A quick question on the mount command that you have used
> "mount -t glusterfs node4:/glustervol/mnt/"
> Here you are specifying the hostname as node4 instead of node{1,2,3} which
> actually host the volume that you intend to mount. Is this a typo or did
> you paste the same command that you used for mounting?
> If it is the actual command that you have used, then node4 seems to have
> some old volume details which are not cleared properly and it is being used
> while mounting. According to the peer info that you provided, only node1, 2
> & 3 are part of the list, so node4 is unaware of the volume that you want
> to mount and this command is mounting a volume which is only visible to
> node4.
>
> Regards,
> Karthik
>
> On Tue, Jun 16, 2020 at 11:11 AM ahemad shaik
> wrote:
>
> Hi Karthik,
>
>
> Please provide the following info, I see there are errors unable to
> connect to port and warning that transport point end connected. Please find
> the complete logs below.
>
> kindly suggest.
>
> 1. gluster peer status
>
> gluster peer status
> Number of Peers: 2
>
> Hostname: node1
> Uuid: 0e679115-15ad-4a85-9d0a-9178471ef90
> State: Peer in Cluster (Connected)
>
> Hostname: node2
> Uuid: 785a7c5b-86d3-45b9-b371-7e66e7fa88e0
> State: Peer in Cluster (Connected)
>
>
> gluster pool list
> UUIDHostname
> State
> 0e679115-15ad-4a85-9d0a-9178471ef90 node1 Connected
> 785a7c5b-86d3-45b9-b371-7e66e7fa88e0node2
> Connected
> ec137af6-4845-4ebb-955a-fac1df9b7b6clocalhost(node3)
> Connected
>
> 2. gluster volume info glustervol
>
> Volume Name: glustervol
> Type: Replicate
> Volume ID: 5422bb27-1863-47d5-b216-61751a01b759
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: node1:/data
> Brick2: node2:/data
> Brick3: node3:/data
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
>
> 3. gluster volume status glustervol
>
> gluster volume status glustervol
> Status of volume: glustervol
> Gluster process TCP Port RDMA Port Online
> Pid
>
> --
> Brick node1:/data49152 0 Y
> 59739
> Brick node2:/data49153 0 Y
> 3498
> Brick node3:/data49152 0 Y
> 1880
> Self-heal Daemon on localhostN/A N/AY
> 1905
> Self-heal Daemon on node1N/A N/A
Re: [Gluster-users] Issues with replicated gluster volume
Sorry, It was a typo.
The command i exact command i have used is below.
The volume is mounted on node4.
""mount -t glusterfs node1:/glustervol /mnt/ ""
gluster Volume is created from node1,node2 and node3.
""gluster volume create glustervol replica 3 transport tcp node1:/data
node2:/data node3:/data force""
I have tried rebooting node3 to test high availability.
I hope it is clear now.
Please let me know if any questions.
Thanks,Ahemad
On Tuesday, 16 June, 2020, 11:45:48 am IST, Karthik Subrahmanya
wrote:
Hi Ahemad,
A quick question on the mount command that you have used"mount -t glusterfs
node4:/glustervol /mnt/"
Here you are specifying the hostname as node4 instead of node{1,2,3} which
actually host the volume that you intend to mount. Is this a typo or did you
paste the same command that you used for mounting?
If it is the actual command that you have used, then node4 seems to have some
old volume details which are not cleared properly and it is being used while
mounting. According to the peer info that you provided, only node1, 2 & 3 are
part of the list, so node4 is unaware of the volume that you want to mount and
this command is mounting a volume which is only visible to node4.
Regards,Karthik
On Tue, Jun 16, 2020 at 11:11 AM ahemad shaik wrote:
Hi Karthik,
Please provide the following info, I see there are errors unable to connect to
port and warning that transport point end connected. Please find the complete
logs below.
kindly suggest.
1. gluster peer status
gluster peer statusNumber of Peers: 2
Hostname: node1Uuid: 0e679115-15ad-4a85-9d0a-9178471ef90State: Peer in Cluster
(Connected)
Hostname: node2Uuid: 785a7c5b-86d3-45b9-b371-7e66e7fa88e0State: Peer in Cluster
(Connected)
gluster pool listUUID Hostname
State0e679115-15ad-4a85-9d0a-9178471ef90 node1
Connected785a7c5b-86d3-45b9-b371-7e66e7fa88e0 node2
Connectedec137af6-4845-4ebb-955a-fac1df9b7b6c localhost(node3)
Connected
2. gluster volume info glustervol
Volume Name: glustervolType: ReplicateVolume ID:
5422bb27-1863-47d5-b216-61751a01b759Status: StartedSnapshot Count: 0Number of
Bricks: 1 x 3 = 3Transport-type: tcpBricks:Brick1: node1:/dataBrick2:
node2:/dataBrick3: node3:/dataOptions
Reconfigured:performance.client-io-threads: offnfs.disable:
onstorage.fips-mode-rchecksum: ontransport.address-family: inet
3. gluster volume status glustervol
gluster volume status glustervolStatus of volume: glustervolGluster process
TCP Port RDMA Port Online
Pid--Brick
node1:/data 49152 0 Y 59739Brick
node2:/data 49153 0 Y 3498Brick
node3:/data 49152 0 Y
1880Self-heal Daemon on localhost N/A N/A Y
1905Self-heal Daemon on node1 N/A N/A Y
3519Self-heal Daemon on node2 N/A N/A Y
59760
Task Status of Volume
glustervol--There
are no active volume tasks
4. client log from node4 when you saw unavailability-
Below are the logs when i reboot server node3, we can see in logs that
"0-glustervol-client-2: disconnected from glustervol-client-2".
Please find the complete logs below from the reboot to until the server
available. I am testing high availability by just rebooting server. In real
case scenario chances are there that server may not available for some hours so
i just dont want to have the long down time.
[2020-06-16 05:14:25.256136] I [MSGID: 114046]
[client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-0: Connected
to glustervol-client-0, attached to remote volume '/data'.[2020-06-16
05:14:25.256179] I [MSGID: 108005]
[afr-common.c:5247:__afr_handle_child_up_event] 0-glustervol-replicate-0:
Subvolume 'glustervol-client-0' came back up; going online.[2020-06-16
05:14:25.257972] I [MSGID: 114046]
[client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-1: Connected
to glustervol-client-1, attached to remote volume '/data'.[2020-06-16
05:14:25.258014] I [MSGID: 108002] [afr-common.c:5609:afr_notify]
0-glustervol-replicate-0: Client-quorum is met[2020-06-16 05:14:25.260312] I
[MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk]
0-glustervol-client-2: Connected to glustervol-client-2, attached to remote
volume '/data'.[2020-06-16 05:14:25.261935] I [fuse-bridge.c:5145:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
7.23[2020-06-16 05:14:25.261957] I [fuse-bridge.c:5756:fuse_graph_sync] 0-fuse:
switched to graph 0[2020-06-16 05:16:59.729400] I [MSGID:
Re: [Gluster-users] Issues with replicated gluster volume
Hi Karthik, Please provide the following info, I see there are errors unable to connect to port and warning that transport point end connected. Please find the complete logs below. kindly suggest. 1. gluster peer status gluster peer statusNumber of Peers: 2 Hostname: node1Uuid: 0e679115-15ad-4a85-9d0a-9178471ef90State: Peer in Cluster (Connected) Hostname: node2Uuid: 785a7c5b-86d3-45b9-b371-7e66e7fa88e0State: Peer in Cluster (Connected) gluster pool listUUID Hostname State0e679115-15ad-4a85-9d0a-9178471ef90 node1 Connected785a7c5b-86d3-45b9-b371-7e66e7fa88e0 node2 Connectedec137af6-4845-4ebb-955a-fac1df9b7b6c localhost(node3) Connected 2. gluster volume info glustervol Volume Name: glustervolType: ReplicateVolume ID: 5422bb27-1863-47d5-b216-61751a01b759Status: StartedSnapshot Count: 0Number of Bricks: 1 x 3 = 3Transport-type: tcpBricks:Brick1: node1:/dataBrick2: node2:/dataBrick3: node3:/dataOptions Reconfigured:performance.client-io-threads: offnfs.disable: onstorage.fips-mode-rchecksum: ontransport.address-family: inet 3. gluster volume status glustervol gluster volume status glustervolStatus of volume: glustervolGluster process TCP Port RDMA Port Online Pid--Brick node1:/data 49152 0 Y 59739Brick node2:/data 49153 0 Y 3498Brick node3:/data 49152 0 Y 1880Self-heal Daemon on localhost N/A N/A Y 1905Self-heal Daemon on node1 N/A N/A Y 3519Self-heal Daemon on node2 N/A N/A Y 59760 Task Status of Volume glustervol--There are no active volume tasks 4. client log from node4 when you saw unavailability- Below are the logs when i reboot server node3, we can see in logs that "0-glustervol-client-2: disconnected from glustervol-client-2". Please find the complete logs below from the reboot to until the server available. I am testing high availability by just rebooting server. In real case scenario chances are there that server may not available for some hours so i just dont want to have the long down time. [2020-06-16 05:14:25.256136] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-0: Connected to glustervol-client-0, attached to remote volume '/data'.[2020-06-16 05:14:25.256179] I [MSGID: 108005] [afr-common.c:5247:__afr_handle_child_up_event] 0-glustervol-replicate-0: Subvolume 'glustervol-client-0' came back up; going online.[2020-06-16 05:14:25.257972] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-1: Connected to glustervol-client-1, attached to remote volume '/data'.[2020-06-16 05:14:25.258014] I [MSGID: 108002] [afr-common.c:5609:afr_notify] 0-glustervol-replicate-0: Client-quorum is met[2020-06-16 05:14:25.260312] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-2: Connected to glustervol-client-2, attached to remote volume '/data'.[2020-06-16 05:14:25.261935] I [fuse-bridge.c:5145:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23[2020-06-16 05:14:25.261957] I [fuse-bridge.c:5756:fuse_graph_sync] 0-fuse: switched to graph 0[2020-06-16 05:16:59.729400] I [MSGID: 114018] [client.c:2331:client_rpc_notify] 0-glustervol-client-2: disconnected from glustervol-client-2. Client process will keep trying to connect to glusterd until brick's port is available[2020-06-16 05:16:59.730053] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:08.175698 (xid=0xae)[2020-06-16 05:16:59.730089] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-glustervol-client-2: remote operation failed. Path: / (----0001) [Transport endpoint is not connected][2020-06-16 05:16:59.730336] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518]
Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad,
A quick question on the mount command that you have used
"mount -t glusterfs node4:/glustervol/mnt/"
Here you are specifying the hostname as node4 instead of node{1,2,3} which
actually host the volume that you intend to mount. Is this a typo or did
you paste the same command that you used for mounting?
If it is the actual command that you have used, then node4 seems to have
some old volume details which are not cleared properly and it is being used
while mounting. According to the peer info that you provided, only node1, 2
& 3 are part of the list, so node4 is unaware of the volume that you want
to mount and this command is mounting a volume which is only visible to
node4.
Regards,
Karthik
On Tue, Jun 16, 2020 at 11:11 AM ahemad shaik
wrote:
> Hi Karthik,
>
>
> Please provide the following info, I see there are errors unable to
> connect to port and warning that transport point end connected. Please find
> the complete logs below.
>
> kindly suggest.
>
> 1. gluster peer status
>
> gluster peer status
> Number of Peers: 2
>
> Hostname: node1
> Uuid: 0e679115-15ad-4a85-9d0a-9178471ef90
> State: Peer in Cluster (Connected)
>
> Hostname: node2
> Uuid: 785a7c5b-86d3-45b9-b371-7e66e7fa88e0
> State: Peer in Cluster (Connected)
>
>
> gluster pool list
> UUIDHostname
> State
> 0e679115-15ad-4a85-9d0a-9178471ef90 node1 Connected
> 785a7c5b-86d3-45b9-b371-7e66e7fa88e0node2
> Connected
> ec137af6-4845-4ebb-955a-fac1df9b7b6clocalhost(node3)
> Connected
>
> 2. gluster volume info glustervol
>
> Volume Name: glustervol
> Type: Replicate
> Volume ID: 5422bb27-1863-47d5-b216-61751a01b759
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: node1:/data
> Brick2: node2:/data
> Brick3: node3:/data
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
>
> 3. gluster volume status glustervol
>
> gluster volume status glustervol
> Status of volume: glustervol
> Gluster process TCP Port RDMA Port Online
> Pid
>
> --
> Brick node1:/data49152 0 Y
> 59739
> Brick node2:/data49153 0 Y
> 3498
> Brick node3:/data49152 0 Y
> 1880
> Self-heal Daemon on localhostN/A N/AY
> 1905
> Self-heal Daemon on node1N/A N/AY
> 3519
> Self-heal Daemon on node2N/A N/AY
> 59760
>
> Task Status of Volume glustervol
>
> --
> There are no active volume tasks
>
> 4. client log from node4 when you saw unavailability-
>
> Below are the logs when i reboot server node3, we can see in logs that
> "0-glustervol-client-2: disconnected from glustervol-client-2".
>
> Please find the complete logs below from the reboot to until the server
> available. I am testing high availability by just rebooting server. In real
> case scenario chances are there that server may not available for some
> hours so i just dont want to have the long down time.
>
>
> [2020-06-16 05:14:25.256136] I [MSGID: 114046]
> [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-0:
> Connected to glustervol-client-0, attached to remote volume '/data'.
> [2020-06-16 05:14:25.256179] I [MSGID: 108005]
> [afr-common.c:5247:__afr_handle_child_up_event] 0-glustervol-replicate-0:
> Subvolume 'glustervol-client-0' came back up; going online.
> [2020-06-16 05:14:25.257972] I [MSGID: 114046]
> [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-1:
> Connected to glustervol-client-1, attached to remote volume '/data'.
> [2020-06-16 05:14:25.258014] I [MSGID: 108002]
> [afr-common.c:5609:afr_notify] 0-glustervol-replicate-0: Client-quorum is
> met
> [2020-06-16 05:14:25.260312] I [MSGID: 114046]
> [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-2:
> Connected to glustervol-client-2, attached to remote volume '/data'.
> [2020-06-16 05:14:25.261935] I [fuse-bridge.c:5145:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
> 7.23
> [2020-06-16 05:14:25.261957] I [fuse-bridge.c:5756:fuse_graph_sync]
> 0-fuse: switched to graph 0
> [2020-06-16 05:16:59.729400] I [MSGID: 114018]
> [client.c:2331:client_rpc_notify] 0-glustervol-client-2: disconnected from
> glustervol-client-2. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2020-06-16 05:16:59.730053] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (-->
>
Re: [Gluster-users] Issues with replicated gluster volume
Hi Ahemad, Please provide the following info: 1. gluster peer status 2. gluster volume info glustervol 3. gluster volume status glustervol 4. client log from node4 when you saw unavailability Regards, Karthik On Mon, Jun 15, 2020 at 11:07 PM ahemad shaik wrote: > Hi There, > > I have created 3 replica gluster volume with 3 bricks from 3 nodes. > > "gluster volume create glustervol replica 3 transport tcp node1:/data > node2:/data node3:/data force" > > mounted on client node using below command. > > "mount -t glusterfs node4:/glustervol/mnt/" > > when any of the node (either node1,node2 or node3) goes down, gluster > mount/volume (/mnt) not accessible at client (node4). > > purpose of replicated volume is high availability but not able to achieve > it. > > Is it a bug or i am missing anything. > > > Any suggestions will be great help!!! > > kindly suggest. > > Thanks, > Ahemad > > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
