Re: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04)

[Aravinda Vishwanathapura Krishna Murthy]

On Wed, Oct 16, 2019 at 11:36 PM Strahil  wrote:

> By the  way,
>
> I have  been left  with the  impresssion that  data  is transferred via
> 'rsync' and not via FUSE.
> Am I wrong ?
>

Rsync syncs data from Master FUSE mount to Slave/Remote FUSE mount.


>
> Best Regards,
> Strahil NikolovOn Oct 16, 2019 19:59, Alexander Iliev <
> ailiev+glus...@mamul.org> wrote:
> >
> > Hi Aravinda,
> >
> > All volume brick on the slave volume are up and the volume seems
> functional.
> >
> > Your suggestion about trying to mount the slave volume on a master node
> > brings up my question about network connectivity again - the GlusterFS
> > documentation[1] says:
> >
> > > The server specified in the mount command is only used to fetch the
> > gluster configuration volfile describing the volume name. Subsequently,
> > the client will communicate directly with the servers mentioned in the
> > volfile (which might not even include the one used for mount).
> >
> > To me this means that the masternode from your example is expected to
> > have connectivity to the network where the slave volume runs, i.e. to
> > have network access to the slave nodes. In my geo-replication scenario
> > this is definitely not the case. The two cluster are running in two
> > completely different networks that are not interconnected.
> >
> > So my question is - how is the slave volume mount expected to happen if
> > the client host cannot access the GlusterFS nodes? Or is the
> > connectivity a requirement even for geo-replication?
> >
> > I'm not sure if I'm missing something, but any help will be highly
> > appreciated!
> >
> > Thanks!
> >
> > Links:
> > [1]
> >
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Setting%20Up%20Clients/
> > --
> > alexander iliev
> >
> > On 10/16/19 6:03 AM, Aravinda Vishwanathapura Krishna Murthy wrote:
> > > Hi Alexander,
> > >
> > > Please check the status of Volume. Looks like the Slave volume mount
> is
> > > failing because bricks are down or not reachable. If Volume status
> shows
> > > all bricks are up then try mounting the slave volume using mount
> command.
> > >
> > > ```
> > > masternode$ mkdir /mnt/vol
> > > masternode$ mount -t glusterfs : /mnt/vol
> > > ```
> > >
> > > On Fri, Oct 11, 2019 at 4:03 AM Alexander Iliev
> > > mailto:ailiev%2bglus...@mamul.org>> wrote:
> > >
> > > Hi all,
> > >
> > > I ended up reinstalling the nodes with CentOS 7.5 and GlusterFS 6.5
> > > (installed from the SIG.)
> > >
> > > Now when I try to create a replication session I get the following:
> > >
> > >   > # gluster volume geo-replication store1 ::store2
> create
> > > push-pem
> > >   > Unable to mount and fetch slave volume details. Please check
> the
> > > log:
> > > /var/log/glusterfs/geo-replication/gverify-slavemnt.log
> > >   > geo-replication command failed
> > >
> > > You can find the contents of gverify-slavemnt.log below, but the
> > > initial
> > > error seems to be:
> > >
> > >   > [2019-10-10 22:07:51.578519] E
> > > [fuse-bridge.c:5211:fuse_first_lookup]
> > > 0-fuse: first lookup on root failed (Transport endpoint is not
> > > connected)
> > >
> > > I only found
> > > [this](https://bugzilla.redhat.com/show_bug.cgi?id=1659824)
> > > bug report which doesn't seem to help. The reported issue is
> failure to
> > > mount a volume on a GlusterFS client, but in my case I need
> > > geo-replication which implies the client (geo-replication master)
> being
> > > on a different network.
> > >
> > > Any help will be appreciated.
> > >
> > > Thanks!
> > >
> > > gverify-slavemnt.log:
> > >
> > >   > [2019-10-10 22:07:40.571256] I [MSGID: 100030]
> > > [glusterfsd.c:2847:main] 0-glusterfs: Started running glusterfs
> version
> > > 6.5 (args: glusterfs --xlator-option=*dht.lookup-unhashed=off
> > > --volfile-server  --volfile-id store2 -l
> > > /var/log/glusterfs/geo-replication/gverify-slavemnt.log
> > > /tmp/gverify.sh.5nFlRh)
> > >   > [2019-10-10 22:07:40.575438] I [glusterfsd.c:2556:daemonize]
> > > 0-glusterfs: Pid of current running process is 6021
> > >   > [2019-10-10 22:07:40.584282] I [MSGID: 101190]
> > > [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started
> thread
> > > with index 0
> > >   > [2019-10-10 22:07:40.584299] I [MSGID: 101190]
> > >



-- 
regards
Aravinda VK


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04)

[Aravinda Vishwanathapura Krishna Murthy]

Got it.

Geo-replication uses slave nodes IP in the following cases,

- Verification during Session creation - It tries to mount the Slave volume
using the hostname/IP provided in Geo-rep create command. Try Geo-rep
create by specifying the external IP which is accessible from the master
node.
- Once Geo-replication is started, it gets the list of Slave nodes
IP/hostname from Slave volume info and connects to those IPs. But in this
case, those are internal IP addresses that are not accessible from Master
nodes. - We need to enhance Geo-replication to accept external IP and
internal IP map details so that for all connections it can use external IP.

On Wed, Oct 16, 2019 at 10:29 PM Alexander Iliev 
wrote:

> Hi Aravinda,
>
> All volume brick on the slave volume are up and the volume seems
> functional.
>
> Your suggestion about trying to mount the slave volume on a master node
> brings up my question about network connectivity again - the GlusterFS
> documentation[1] says:
>
>  > The server specified in the mount command is only used to fetch the
> gluster configuration volfile describing the volume name. Subsequently,
> the client will communicate directly with the servers mentioned in the
> volfile (which might not even include the one used for mount).
>
> To me this means that the masternode from your example is expected to
> have connectivity to the network where the slave volume runs, i.e. to
> have network access to the slave nodes. In my geo-replication scenario
> this is definitely not the case. The two cluster are running in two
> completely different networks that are not interconnected.
>
> So my question is - how is the slave volume mount expected to happen if
> the client host cannot access the GlusterFS nodes? Or is the
> connectivity a requirement even for geo-replication?
>
> I'm not sure if I'm missing something, but any help will be highly
> appreciated!
>
> Thanks!
>
> Links:
> [1]
>
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Setting%20Up%20Clients/
> --
> alexander iliev
>
> On 10/16/19 6:03 AM, Aravinda Vishwanathapura Krishna Murthy wrote:
> > Hi Alexander,
> >
> > Please check the status of Volume. Looks like the Slave volume mount is
> > failing because bricks are down or not reachable. If Volume status shows
> > all bricks are up then try mounting the slave volume using mount command.
> >
> > ```
> > masternode$ mkdir /mnt/vol
> > masternode$ mount -t glusterfs : /mnt/vol
> > ```
> >
> > On Fri, Oct 11, 2019 at 4:03 AM Alexander Iliev
> > mailto:ailiev%2bglus...@mamul.org>> wrote:
> >
> > Hi all,
> >
> > I ended up reinstalling the nodes with CentOS 7.5 and GlusterFS 6.5
> > (installed from the SIG.)
> >
> > Now when I try to create a replication session I get the following:
> >
> >   > # gluster volume geo-replication store1 ::store2
> create
> > push-pem
> >   > Unable to mount and fetch slave volume details. Please check the
> > log:
> > /var/log/glusterfs/geo-replication/gverify-slavemnt.log
> >   > geo-replication command failed
> >
> > You can find the contents of gverify-slavemnt.log below, but the
> > initial
> > error seems to be:
> >
> >   > [2019-10-10 22:07:51.578519] E
> > [fuse-bridge.c:5211:fuse_first_lookup]
> > 0-fuse: first lookup on root failed (Transport endpoint is not
> > connected)
> >
> > I only found
> > [this](https://bugzilla.redhat.com/show_bug.cgi?id=1659824)
> > bug report which doesn't seem to help. The reported issue is failure
> to
> > mount a volume on a GlusterFS client, but in my case I need
> > geo-replication which implies the client (geo-replication master)
> being
> > on a different network.
> >
> > Any help will be appreciated.
> >
> > Thanks!
> >
> > gverify-slavemnt.log:
> >
> >   > [2019-10-10 22:07:40.571256] I [MSGID: 100030]
> > [glusterfsd.c:2847:main] 0-glusterfs: Started running glusterfs
> version
> > 6.5 (args: glusterfs --xlator-option=*dht.lookup-unhashed=off
> > --volfile-server  --volfile-id store2 -l
> > /var/log/glusterfs/geo-replication/gverify-slavemnt.log
> > /tmp/gverify.sh.5nFlRh)
> >   > [2019-10-10 22:07:40.575438] I [glusterfsd.c:2556:daemonize]
> > 0-glusterfs: Pid of current running process is 6021
> >   > [2019-10-10 22:07:40.584282] I [MSGID: 101190]
> > [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started
> thread
> > with index 0
> >   > [2019-10-10 22:07:40.584299] I [MSGID: 101190]
> > [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started
> thread
> > with index 1
> >   > [2019-10-10 22:07:40.928094] I [MSGID: 114020]
> > [client.c:2393:notify]
> > 0-store2-client-0: parent translators are ready, attempting connect
> on
> > transport
> >   > [2019-10-10 22:07:40.931121] I [MSGID: 114020]
> > [client.c:2393:notify]
> > 0-store2-client-1: parent translators are ready, attempting 

Re: [Gluster-users] Network Disturbance while Geo Replication

[Aravinda Vishwanathapura Krishna Murthy]

On Wed, Oct 16, 2019 at 2:47 PM deepu srinivasan  wrote:

> Hi Users
> How will the GeoReplication resumes if there is a network Disturbance or
> Network failure between the two Data Centre? What will happen if a rsync
> session for a file fails? Will the rsync session restart for the fail again?
>
>
Hi Deepu,

In case of partial failures, Rsync returns error code 23 or 24. Geo-rep
handles these errors and retries a couple of times. If the sync fails even
after retries then it logs the failures.

Each worker will maintain an ssh connection to a slave node, the SSH
connection will fail in case of network failure or Slave volume mount
failure. Workers will fail/exit in case of ssh failure, Monitor will
restart the failed workers and workers choose different slave node if the
previously connected node is unavailable.

-- 
regards
Aravinda VK


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Single Point of failure in geo Replication

[Aravinda Vishwanathapura Krishna Murthy]

On Wed, Oct 16, 2019 at 11:08 PM deepu srinivasan 
wrote:

> Hi Users
> Is there a single point of failure in GeoReplication for gluster?
> My Case:
> I Use 3 nodes in both master and slave volume.
> Master volume : Node1,Node2,Node3
> Slave Volume : Node4,Node5,Node6
> I tried to recreate the scenario to test a single point of failure.
>
> Geo-Replication Status:
>
> *Master Node Slave Node Status *
> Node1   Node4  Active
> Node2   Node4  Passive
> Node3   Node4  Passive
>
> Step 1: Stoped the glusterd daemon in Node4.
> Result: There were only two-node statuses like the one below.
>
> *Master Node Slave Node Status *
> Node2   Node4  Passive
> Node3   Node4  Passive
>
>
> Will the GeoReplication session goes down if the primary slave is down?
>


Hi Deepu,

Geo-replication depends on a primary slave node to get the information
about other nodes which are part of Slave Volume.

Once the workers are started, it is not dependent on the primary slave
node. Will not fail if a primary goes down. But if any other node goes down
then the worker will try to connect to some other node, for which it tries
to run Volume status command on the slave node using the following command.

```
ssh -i   gluster volume status 
```

The above command will fail and Worker will not get the list of Slave nodes
to which it can connect to.

This is only a temporary failure until the primary node comes back online.
If the primary node is permanently down then run Geo-rep delete and Geo-rep
create command again with the new primary node. (Note: Geo-rep Delete and
Create will remember the last sync time and resume once it starts)

I will evaluate the possibility of caching a list of Slave nodes so that it
can be used as a backup primary node in case of failures. I will open
Github issue for the same.

Thanks for reporting the issue.

-- 
regards
Aravinda VK


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Client Handling of Elastic Clusters

[Timothy Orme]

I did explore CEPH a bit, and that might be an option as well, still doing 
exploration on gluster.   Hopefully no one hates you for making the suggestion 

I haven't tried NFS Ganesha yet.  I was under the impression it was maybe a 
little unstable yet, and found the docs a little limited for it.  If that 
solves the issue that might also be a good option.  I've heard others suggest 
performance is better for it than the FUSE client as well.

I don't know how other systems deal with it currently, but it seems like even 
just leveraging the volfile itself as a source for backups would work well.  
There are still likely issues where things could lapse, but that seems like an 
improvement at least.  I'll try and dig into what other's are using, though 
maybe they don't have this issue at all since they tend to use metadata servers?

Thanks,
Tim

From: Strahil Nikolov 
Sent: Wednesday, October 16, 2019 11:49 AM
To: gluster-users ; Timothy Orme 
Subject: [EXTERNAL] Re: Re: [Gluster-users] Client Handling of Elastic Clusters

Most probably current version never supported (maybe there was no such need 
until now) such elasticity and the only option is to use Highly-Available NFS 
Ganesha as the built-in NFS is deprecated.
What about scaling on the same system ? Nowadays , servers have a lot of 
hot-plug disk slots and you can keep the number of servers the same ... still 
the server bandwidth will be a limit at some point .

I'm not sure how other SDS deal with such elasticity . I guess many users in 
the list will hate me for saying this , but have you checked CEPH for your 
needs ?

Best Regards,
Strahil Nikolov

В сряда, 16 октомври 2019 г., 21:13:58 ч. Гринуич+3, Timothy Orme 
 написа:


Yes, this makes the issue less likely, but doesn't make it impossible for 
something that is fully elastic.

For instance, if I had instead just started with A,B,C and then scaled out and 
in twice, all volfile servers would have potentially be destroyed and replaced. 
 I think the problem is that the selection of volfile servers is determined at 
mounting, rather than updating as the cluster changes.  There are ways to 
greatly reduce this issue, such as adding more backup servers, but it's still a 
possibility.

I think more important then, for me at least, is to have the option of failing 
when no volfile servers are remaining as it can produce incomplete views of the 
data.

Thanks!
Tim

From: Strahil 
Sent: Tuesday, October 15, 2019 8:46 PM
To: Timothy Orme ; gluster-users 
Subject: [EXTERNAL] Re: [Gluster-users] Client Handling of Elastic Clusters


Hi Timothy,

Have you tried to mount on the client  via all servers :

mount -t glusterfs -o backup-volfile-servers=B:C:D:E:F A:/volume  /destination

Best Regards,
Strahil Nikolov

On Oct 15, 2019 22:05, Timothy Orme  wrote:
Hello,

I'm trying to setup an elastic gluster cluster and am running into a few odd 
edge cases that I'm unsure how to address.  I'll try and walk through the setup 
as best I can.

If I have a replica 3 distributed-replicated volume, with 2 replicated volumes 
to start:

MyVolume
   Replica 1
  serverA
  serverB
  serverC
   Replica 2
  serverD
  serverE
  serverF

And the client mounts the volume with serverA as the primary volfile server, 
and B & C as the backups.

Then, if I perform a scale down event, it selects the first replica volume as 
the one to remove.  So I end up with a configuration like:

MyVolume
   Replica 2
  serverD
  serverE
  serverF

Everything rebalances and works great.  However, at this point, the client has 
lost any connection with a volfile server.  It knows about D, E, and F, so my 
data is all fine, but it can no longer retrieve a volfile.  In the logs I see:

[2019-10-15 17:21:59.232819] I [glusterfsd-mgmt.c:2463:mgmt_rpc_notify] 
0-glusterfsd-mgmt: Exhausted all volfile servers

This becomes problematic when I try and scale back up, and add a replicated 
volume back in:

MyVolume
   Replica 2
  serverD
  serverE
  serverF
   Replica 3
  serverG
  serverH
  serverI

And then rebalance the volume.  Now, I have all my data present, but the client 
only knows about D,E,F, so when I run an `ls` on a directory, only about half 
of the files are returned, since the other half live on G,H,I which the client 
doesn't know about.  The data is still there, but it would require a re-mount 
at one of the new servers.

My question then, is there a way to have a more dynamic set of volfile servers? 
What would be great is if there was a way to tell the mount to fall back on the 
servers returned in the volfile itself in case the primary one goes away.

If there's not an easy way to do this, is there a flag on the mount helper that 
can cause the mount to die or error out in the event that it is unable to 
retrieve volfiles?  The problem now is that it sort of silently fails and 
returns incomplete file 

Re: [Gluster-users] Client Handling of Elastic Clusters

[Strahil Nikolov]

 Most probably current version never supported (maybe there was no such need 
until now) such elasticity and the only option is to use Highly-Available NFS 
Ganesha as the built-in NFS is deprecated.What about scaling on the same system 
? Nowadays , servers have a lot of hot-plug disk slots and you can keep the 
number of servers the same ... still the server bandwidth will be a limit at 
some point .
I'm not sure how other SDS deal with such elasticity . I guess many users in 
the list will hate me for saying this , but have you checked CEPH for your 
needs ?
Best Regards,Strahil Nikolov
В сряда, 16 октомври 2019 г., 21:13:58 ч. Гринуич+3, Timothy Orme 
 написа:  
 
 Yes, this makes the issue less likely, but doesn't make it impossible for 
something that is fully elastic.
For instance, if I had instead just started with A,B,C and then scaled out and 
in twice, all volfile servers would have potentially be destroyed and replaced. 
 I think the problem is that the selection of volfile servers is determined at 
mounting, rather than updating as the cluster changes.  There are ways to 
greatly reduce this issue, such as adding more backup servers, but it's still a 
possibility. 

I think more important then, for me at least, is to have the option of failing 
when no volfile servers are remaining as it can produce incomplete views of the 
data.
Thanks!Tim
From: Strahil 
Sent: Tuesday, October 15, 2019 8:46 PM
To: Timothy Orme ; gluster-users 
Subject: [EXTERNAL] Re: [Gluster-users] Client Handling of Elastic Clusters 
Hi Timothy,

Have you tried to mount on the client  via all servers :

mount -t glusterfs -o backup-volfile-servers=B:C:D:E:F A:/volume  /destination

Best Regards,
Strahil Nikolov
On Oct 15, 2019 22:05, Timothy Orme  wrote:

Hello,
I'm trying to setup an elastic gluster cluster and am running into a few odd 
edge cases that I'm unsure how to address.  I'll try and walk through the setup 
as best I can.
If I have a replica 3 distributed-replicated volume, with 2 replicated volumes 
to start:
MyVolume   Replica 1
  serverA  serverB  serverC   Replica 2  serverD  serverE   
   serverF

And the client mounts the volume with serverA as the primary volfile server, 
and B & C as the backups.
Then, if I perform a scale down event, it selects the first replica volume as 
the one to remove.  So I end up with a configuration like:
MyVolume
   Replica 2  serverD  serverE  serverF

Everything rebalances and works great.  However, at this point, the client has 
lost any connection with a volfile server.  It knows about D, E, and F, so my 
data is all fine, but it can no longer retrieve a volfile.  In the logs I see:

[2019-10-15 17:21:59.232819] I [glusterfsd-mgmt.c:2463:mgmt_rpc_notify] 
0-glusterfsd-mgmt: Exhausted all volfile servers

This becomes problematic when I try and scale back up, and add a replicated 
volume back in:

MyVolume
   Replica 2  serverD  serverE  serverF   Replica 3  serverG
  serverH  serverI
And then rebalance the volume.  Now, I have all my data present, but the client 
only knows about D,E,F, so when I run an `ls` on a directory, only about half 
of the files are returned, since the other half live on G,H,I which the client 
doesn't know about.  The data is still there, but it would require a re-mount 
at one of the new servers.
My question then, is there a way to have a more dynamic set of volfile servers? 
What would be great is if there was a way to tell the mount to fall back on the 
servers returned in the volfile itself in case the primary one goes away.
If there's not an easy way to do this, is there a flag on the mount helper that 
can cause the mount to die or error out in the event that it is unable to 
retrieve volfiles?  The problem now is that it sort of silently fails and 
returns incomplete file listings, which for my use cases can cause improper 
processing of that data.  I'd rather have it hard error than provide bad 
results silently obviously.
Hope that makes sense, if you need further clarity please let me know.  

Thanks,Tim



  

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Client Handling of Elastic Clusters

[Timothy Orme]

Yeah there are somewhat dirty ways to work around it, and I hadn't thought of 
this one.  Another option for us is to try and tag certain instances as volfile 
servers, and always prevent the autoscaler from removing them.  It would be 
nice though if this behavior could be added to gluster itself as in cloud 
environments we don't typically rely on having hostnames.  I think I'd also 
sleep better if it would error out if it couldn't retrieve that info.  Ill 
throw in the feature request.  Probably out of my element, but Ill also take a 
shot at adding in a PR for it as well.

Thanks!
Tim

From: Amar Tumballi 
Sent: Tuesday, October 15, 2019 8:51 PM
To: Timothy Orme 
Cc: gluster-users 
Subject: [EXTERNAL] Re: [Gluster-users] Client Handling of Elastic Clusters

Hi Timothy,

Thanks for this report. This seems to be a genuine issue. I don't think we have 
a solution for this issue for now, other than may be making sure we point 
'serverD' (or new server's IPs) as ServerA in /etc/hosts on that particular 
client as a hack.

Meantime, it would be great if you copy paste this in an issue 
(https://github.com/gluster/glusterfs/issues/new),
 it would be good to track this.

Regards,
Amar

On Wed, Oct 16, 2019 at 12:35 AM Timothy Orme 
mailto:to...@ancestry.com>> wrote:
Hello,

I'm trying to setup an elastic gluster cluster and am running into a few odd 
edge cases that I'm unsure how to address.  I'll try and walk through the setup 
as best I can.

If I have a replica 3 distributed-replicated volume, with 2 replicated volumes 
to start:

MyVolume
   Replica 1
  serverA
  serverB
  serverC
   Replica 2
  serverD
  serverE
  serverF

And the client mounts the volume with serverA as the primary volfile server, 
and B & C as the backups.

Then, if I perform a scale down event, it selects the first replica volume as 
the one to remove.  So I end up with a configuration like:

MyVolume
   Replica 2
  serverD
  serverE
  serverF

Everything rebalances and works great.  However, at this point, the client has 
lost any connection with a volfile server.  It knows about D, E, and F, so my 
data is all fine, but it can no longer retrieve a volfile.  In the logs I see:

[2019-10-15 17:21:59.232819] I [glusterfsd-mgmt.c:2463:mgmt_rpc_notify] 
0-glusterfsd-mgmt: Exhausted all volfile servers

This becomes problematic when I try and scale back up, and add a replicated 
volume back in:

MyVolume
   Replica 2
  serverD
  serverE
  serverF
   Replica 3
  serverG
  serverH
  serverI

And then rebalance the volume.  Now, I have all my data present, but the client 
only knows about D,E,F, so when I run an `ls` on a directory, only about half 
of the files are returned, since the other half live on G,H,I which the client 
doesn't know about.  The data is still there, but it would require a re-mount 
at one of the new servers.

My question then, is there a way to have a more dynamic set of volfile servers? 
What would be great is if there was a way to tell the mount to fall back on the 
servers returned in the volfile itself in case the primary one goes away.

If there's not an easy way to do this, is there a flag on the mount helper that 
can cause the mount to die or error out in the event that it is unable to 
retrieve volfiles?  The problem now is that it sort of silently fails and 
returns incomplete file listings, which for my use cases can cause improper 
processing of that data.  I'd rather have it hard error than provide bad 
results silently obviously.

Hope that makes sense, if you need further clarity please let me know.

Thanks,
Tim




Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: 
https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: 
https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org

Re: [Gluster-users] Client Handling of Elastic Clusters

[Timothy Orme]

Yes, this makes the issue less likely, but doesn't make it impossible for 
something that is fully elastic.

For instance, if I had instead just started with A,B,C and then scaled out and 
in twice, all volfile servers would have potentially be destroyed and replaced. 
 I think the problem is that the selection of volfile servers is determined at 
mounting, rather than updating as the cluster changes.  There are ways to 
greatly reduce this issue, such as adding more backup servers, but it's still a 
possibility.

I think more important then, for me at least, is to have the option of failing 
when no volfile servers are remaining as it can produce incomplete views of the 
data.

Thanks!
Tim

From: Strahil 
Sent: Tuesday, October 15, 2019 8:46 PM
To: Timothy Orme ; gluster-users 
Subject: [EXTERNAL] Re: [Gluster-users] Client Handling of Elastic Clusters


Hi Timothy,

Have you tried to mount on the client  via all servers :

mount -t glusterfs -o backup-volfile-servers=B:C:D:E:F A:/volume  /destination

Best Regards,
Strahil Nikolov

On Oct 15, 2019 22:05, Timothy Orme  wrote:
Hello,

I'm trying to setup an elastic gluster cluster and am running into a few odd 
edge cases that I'm unsure how to address.  I'll try and walk through the setup 
as best I can.

If I have a replica 3 distributed-replicated volume, with 2 replicated volumes 
to start:

MyVolume
   Replica 1
  serverA
  serverB
  serverC
   Replica 2
  serverD
  serverE
  serverF

And the client mounts the volume with serverA as the primary volfile server, 
and B & C as the backups.

Then, if I perform a scale down event, it selects the first replica volume as 
the one to remove.  So I end up with a configuration like:

MyVolume
   Replica 2
  serverD
  serverE
  serverF

Everything rebalances and works great.  However, at this point, the client has 
lost any connection with a volfile server.  It knows about D, E, and F, so my 
data is all fine, but it can no longer retrieve a volfile.  In the logs I see:

[2019-10-15 17:21:59.232819] I [glusterfsd-mgmt.c:2463:mgmt_rpc_notify] 
0-glusterfsd-mgmt: Exhausted all volfile servers

This becomes problematic when I try and scale back up, and add a replicated 
volume back in:

MyVolume
   Replica 2
  serverD
  serverE
  serverF
   Replica 3
  serverG
  serverH
  serverI

And then rebalance the volume.  Now, I have all my data present, but the client 
only knows about D,E,F, so when I run an `ls` on a directory, only about half 
of the files are returned, since the other half live on G,H,I which the client 
doesn't know about.  The data is still there, but it would require a re-mount 
at one of the new servers.

My question then, is there a way to have a more dynamic set of volfile servers? 
What would be great is if there was a way to tell the mount to fall back on the 
servers returned in the volfile itself in case the primary one goes away.

If there's not an easy way to do this, is there a flag on the mount helper that 
can cause the mount to die or error out in the event that it is unable to 
retrieve volfiles?  The problem now is that it sort of silently fails and 
returns incomplete file listings, which for my use cases can cause improper 
processing of that data.  I'd rather have it hard error than provide bad 
results silently obviously.

Hope that makes sense, if you need further clarity please let me know.

Thanks,
Tim




Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04)

[Strahil]

By the  way,

I have  been left  with the  impresssion that  data  is transferred via 'rsync' 
and not via FUSE.
Am I wrong ?

Best Regards,
Strahil NikolovOn Oct 16, 2019 19:59, Alexander Iliev 
 wrote:
>
> Hi Aravinda,
>
> All volume brick on the slave volume are up and the volume seems functional.
>
> Your suggestion about trying to mount the slave volume on a master node 
> brings up my question about network connectivity again - the GlusterFS 
> documentation[1] says:
>
> > The server specified in the mount command is only used to fetch the 
> gluster configuration volfile describing the volume name. Subsequently, 
> the client will communicate directly with the servers mentioned in the 
> volfile (which might not even include the one used for mount).
>
> To me this means that the masternode from your example is expected to 
> have connectivity to the network where the slave volume runs, i.e. to 
> have network access to the slave nodes. In my geo-replication scenario 
> this is definitely not the case. The two cluster are running in two 
> completely different networks that are not interconnected.
>
> So my question is - how is the slave volume mount expected to happen if 
> the client host cannot access the GlusterFS nodes? Or is the 
> connectivity a requirement even for geo-replication?
>
> I'm not sure if I'm missing something, but any help will be highly 
> appreciated!
>
> Thanks!
>
> Links:
> [1] 
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Setting%20Up%20Clients/
> --
> alexander iliev
>
> On 10/16/19 6:03 AM, Aravinda Vishwanathapura Krishna Murthy wrote:
> > Hi Alexander,
> > 
> > Please check the status of Volume. Looks like the Slave volume mount is 
> > failing because bricks are down or not reachable. If Volume status shows 
> > all bricks are up then try mounting the slave volume using mount command.
> > 
> > ```
> > masternode$ mkdir /mnt/vol
> > masternode$ mount -t glusterfs : /mnt/vol
> > ```
> > 
> > On Fri, Oct 11, 2019 at 4:03 AM Alexander Iliev 
> > mailto:ailiev%2bglus...@mamul.org>> wrote:
> > 
> > Hi all,
> > 
> > I ended up reinstalling the nodes with CentOS 7.5 and GlusterFS 6.5
> > (installed from the SIG.)
> > 
> > Now when I try to create a replication session I get the following:
> > 
> >   > # gluster volume geo-replication store1 ::store2 create
> > push-pem
> >   > Unable to mount and fetch slave volume details. Please check the
> > log:
> > /var/log/glusterfs/geo-replication/gverify-slavemnt.log
> >   > geo-replication command failed
> > 
> > You can find the contents of gverify-slavemnt.log below, but the
> > initial
> > error seems to be:
> > 
> >   > [2019-10-10 22:07:51.578519] E
> > [fuse-bridge.c:5211:fuse_first_lookup]
> > 0-fuse: first lookup on root failed (Transport endpoint is not
> > connected)
> > 
> > I only found
> > [this](https://bugzilla.redhat.com/show_bug.cgi?id=1659824)
> > bug report which doesn't seem to help. The reported issue is failure to
> > mount a volume on a GlusterFS client, but in my case I need
> > geo-replication which implies the client (geo-replication master) being
> > on a different network.
> > 
> > Any help will be appreciated.
> > 
> > Thanks!
> > 
> > gverify-slavemnt.log:
> > 
> >   > [2019-10-10 22:07:40.571256] I [MSGID: 100030]
> > [glusterfsd.c:2847:main] 0-glusterfs: Started running glusterfs version
> > 6.5 (args: glusterfs --xlator-option=*dht.lookup-unhashed=off
> > --volfile-server  --volfile-id store2 -l
> > /var/log/glusterfs/geo-replication/gverify-slavemnt.log
> > /tmp/gverify.sh.5nFlRh)
> >   > [2019-10-10 22:07:40.575438] I [glusterfsd.c:2556:daemonize]
> > 0-glusterfs: Pid of current running process is 6021
> >   > [2019-10-10 22:07:40.584282] I [MSGID: 101190]
> > [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread
> > with index 0
> >   > [2019-10-10 22:07:40.584299] I [MSGID: 101190]
> >


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04)

[Alexander Iliev]

Hi Aravinda,

All volume brick on the slave volume are up and the volume seems functional.

Your suggestion about trying to mount the slave volume on a master node 
brings up my question about network connectivity again - the GlusterFS 
documentation[1] says:


> The server specified in the mount command is only used to fetch the 
gluster configuration volfile describing the volume name. Subsequently, 
the client will communicate directly with the servers mentioned in the 
volfile (which might not even include the one used for mount).


To me this means that the masternode from your example is expected to 
have connectivity to the network where the slave volume runs, i.e. to 
have network access to the slave nodes. In my geo-replication scenario 
this is definitely not the case. The two cluster are running in two 
completely different networks that are not interconnected.


So my question is - how is the slave volume mount expected to happen if 
the client host cannot access the GlusterFS nodes? Or is the 
connectivity a requirement even for geo-replication?


I'm not sure if I'm missing something, but any help will be highly 
appreciated!


Thanks!

Links:
[1] 
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Setting%20Up%20Clients/

--
alexander iliev

On 10/16/19 6:03 AM, Aravinda Vishwanathapura Krishna Murthy wrote:

Hi Alexander,

Please check the status of Volume. Looks like the Slave volume mount is 
failing because bricks are down or not reachable. If Volume status shows 
all bricks are up then try mounting the slave volume using mount command.


```
masternode$ mkdir /mnt/vol
masternode$ mount -t glusterfs : /mnt/vol
```

On Fri, Oct 11, 2019 at 4:03 AM Alexander Iliev 
mailto:ailiev%2bglus...@mamul.org>> wrote:


Hi all,

I ended up reinstalling the nodes with CentOS 7.5 and GlusterFS 6.5
(installed from the SIG.)

Now when I try to create a replication session I get the following:

  > # gluster volume geo-replication store1 ::store2 create
push-pem
  > Unable to mount and fetch slave volume details. Please check the
log:
/var/log/glusterfs/geo-replication/gverify-slavemnt.log
  > geo-replication command failed

You can find the contents of gverify-slavemnt.log below, but the
initial
error seems to be:

  > [2019-10-10 22:07:51.578519] E
[fuse-bridge.c:5211:fuse_first_lookup]
0-fuse: first lookup on root failed (Transport endpoint is not
connected)

I only found
[this](https://bugzilla.redhat.com/show_bug.cgi?id=1659824)
bug report which doesn't seem to help. The reported issue is failure to
mount a volume on a GlusterFS client, but in my case I need
geo-replication which implies the client (geo-replication master) being
on a different network.

Any help will be appreciated.

Thanks!

gverify-slavemnt.log:

  > [2019-10-10 22:07:40.571256] I [MSGID: 100030]
[glusterfsd.c:2847:main] 0-glusterfs: Started running glusterfs version
6.5 (args: glusterfs --xlator-option=*dht.lookup-unhashed=off
--volfile-server  --volfile-id store2 -l
/var/log/glusterfs/geo-replication/gverify-slavemnt.log
/tmp/gverify.sh.5nFlRh)
  > [2019-10-10 22:07:40.575438] I [glusterfsd.c:2556:daemonize]
0-glusterfs: Pid of current running process is 6021
  > [2019-10-10 22:07:40.584282] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 0
  > [2019-10-10 22:07:40.584299] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
  > [2019-10-10 22:07:40.928094] I [MSGID: 114020]
[client.c:2393:notify]
0-store2-client-0: parent translators are ready, attempting connect on
transport
  > [2019-10-10 22:07:40.931121] I [MSGID: 114020]
[client.c:2393:notify]
0-store2-client-1: parent translators are ready, attempting connect on
transport
  > [2019-10-10 22:07:40.933976] I [MSGID: 114020]
[client.c:2393:notify]
0-store2-client-2: parent translators are ready, attempting connect on
transport
  > Final graph:
  >

+--+
  >   1: volume store2-client-0
  >   2:     type protocol/client
  >   3:     option ping-timeout 42
  >   4:     option remote-host 172.31.36.11
  >   5:     option remote-subvolume /data/gfs/store1/1/brick-store2
  >   6:     option transport-type socket
  >   7:     option transport.address-family inet
  >   8:     option transport.socket.ssl-enabled off
  >   9:     option transport.tcp-user-timeout 0
  >  10:     option transport.socket.keepalive-time 20
  >  11:     option transport.socket.keepalive-interval 2
  >  12:     option transport.socket.keepalive-count 9
  >  13:     option send-gids true
  >  14: end-volume
  >  15:
  >  16: volume 

[Gluster-users] Adding arbiter on a large existing replica 2 set

[Thorgeir Marthinussen]

Hi,

We have an old Gluster cluster setup, running a replica 2 across two 
datacenters, and currently on version 4.1.5

I need to add an arbiter to this setup, but I'm concerned about the performance 
impact of this on the volumes.

I recently set up a new cluster, for a different purpose, and decided to test 
adding an arbiter to the volume after adding in some data.
Had a volume with ~435,000 files totaling about 12TB.
Adding the arbiter initiated a heal-operation that took almost 3 hours.

The older cluster, one of the volumes is about 14TB, but ~45,5 million files.

Since arbiter is only concerned about metadata and checksums, I'm concerned 
about the fact that we have 100 times the amount of files, i.e. 100 times the 
amount of I/O operations to execute during healing, and possibly 100 times the 
time which would mean about 12,5 days.

Another "issue" is that the 'gluster volume heal  info summary' 
command seems to "count" all the files, so the command can take a very long 
time to complete.
The metrics-scraping script I created for us, with a timeout of 110seconds, 
fails to complete when a volume has over ~800-900 files unsynced (which happens 
regularily when taking one cluster-node down for patching).


Does anyone have any experience with adding arbiter afterwards, performance 
impact, time to heal, etc.
Also other ways to get the status on healing.

Any advice would be appreciated.


Best regards
--
THORGEIR MARTHINUSSEN
Senior Systems Consultant
BASEFARM


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport Endpoint Not Connected When Writing a Lot of Files

[DUCARROZ Birgit]

Thank you for your response!

it does not reply.
but all other servers neither reply to ping  24007
pinged this from each server to each server, no reply at all.
even with firewall disabled it does not reply.

root@diufnas22:/home/diuf-sysadmin# netstat -tulpe |grep 24007
tcp0  0 *:24007 *:* 
LISTEN  root   47716960141881/glusterd


Firewall:
8458,24007,24008,49150,49151,49152,49153,49154,49155,49156,49157,49158/tcp 
(v6) on bond1 ALLOW   Anywhere (v6)
8458,24007,24008,49150,49151,49152,49153,49154,49155,49156,49157,49158/tcp 
(v6) on bond0 ALLOW   Anywhere (v6)


Kind regards,
Birgit


On 16/10/19 05:59, Amar Tumballi wrote:
Went through the design. In my opinion, this makes sense, ie, as long as 
you can use better faster network as alternative to reach server, it is 
fine. And considering gluster servers are stateless, that shouldn't 
cause a problem.


Coming to the my suspicion of an issue of network in this particular 
issue comes from the log which mentions 'No route to host'  in the log. 
That particular log prints errno from RPC layer's connect() system call.


Snippet from `man 3 connect`

        EHOSTUNREACH
               The destination host cannot be reached (probably
because the  host  is  down  or  a  remote
               router cannot reach it).


It may be working from few servers, but on the ones you are getting this 
error, can you check if you can reach the server (specific IP), using 
'ping' or 'telnet  24007' ?


Regards,
Amar




On Tue, Oct 15, 2019 at 5:00 PM DUCARROZ Birgit 
mailto:birgit.ducar...@unifr.ch>> wrote:


Hi,

I send you this mail without sending it to the gluster mailing list
because of the pdf attachment (I do not want it to be published).

Do you think it might be because of the different IP address of
diufnas22 where the arbiter brick 3 is installed?

2 hosts communicate with internal 192.168.x.x Network whereas the 3rd
host diufnas22 is connected to the two other hosts via a switch, using
ip address 134.21.x.x. Both networks have a speed of 20gb (2 x 10gb
using lacp bond).

(The scheme shows this).

I would be able to remove the 192.168.x.x network, but my aim was to
speed up the network using an internal communication between bricks
1 and 2.

If this is really the problem, why does the installation mostly works
and crush down when there is heavy network usage with a lot of small
files?


Kind regards
-- 
Birgit Ducarroz

Unix Systems Administration
Department of Informatics
University of Fribourg Switzerland
mailto:birgit.ducar...@unifr.ch 
Phone: +41 (26) 300 8342
https://diuf.unifr.ch/people/ducarroz/
INTRANET / SECURITY NEWS: https://diuf-file.unifr.ch

On 14/10/19 12:47, Amar Tumballi wrote:
 > One of the host ( 134.21.57.122 ) is
not
 > reachable from your network. Also checking at the IP, it would have
 > gotten resolved to something else than expected. Can you check if
 > 'diufnas22' is properly resolved?
 >
 > -Amar
 >
 > On Mon, Oct 14, 2019 at 3:44 PM DUCARROZ Birgit
 > mailto:birgit.ducar...@unifr.ch>
>>
wrote:
 >
 >     Thank you.
 >     I checked the logs but the information was not clear to me.
 >
 >     I add the log of two different crashes. I will do an upgrade to
 >     glusterFS 6 in some weeks. Actually I cannot interrupt user
activity on
 >     these servers since we are in the middle of the uni-semester.
 >
 >     If these logfiles reveal something interesting to you, would
be nice to
 >     get a hint.
 >
 >
 >     ol-data-client-2. Client process will keep trying to connect to
 >     glusterd
 >     until brick's port is available
 >     [2019-09-16 19:05:34.028164] E
[rpc-clnt.c:348:saved_frames_unwind]
 >     (-->
 >   
  /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7ff167753ddb]

 >
 >     (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xc021)[0x7ff167523021]
 >     (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xc14e)[0x7ff16752314e]
 >     (-->
 >   
  /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8e)[0x7ff1675246be]

 >
 >     (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe268)[0x7ff167525268]
 >     ) 0-vol-data-client-2: forced unwinding frame
type(GlusterFS 4.x
 >     v1)
 >     op(FSTAT(25)) called at 2019-09-16 19:05:28.736873
(xid=0x113aecf)
 >     [2019-09-16 19:05:34.028206] W [MSGID: 114031]
 >     [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk]
0-vol-data-client-2:
 >     remote operation failed [Transport endpoint is not connected]
 >     [2019-09-16 

Re: [Gluster-users] Gluster slow reads

[Ashish Pandey]

Hi, 

I am keeping Raghvendra in loop and hope he can comment on the "Read being 
scheduled as slow fop". 

Other than that, I would request you to provide following information to debug 
this issue. 

1 - Profile information of the volume. You can find the steps here - 
https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/ 

2 - Gluster volume  status 

3 - Gluster volume heal  info 

4 - Check the size of each bricks to see if it has been filled. 

5 - gluster version of client and server. 

6 - Host OS version of client and server. 

--- 
Ashish 

- Original Message -

From: a...@telecompartners.ro 
To: gluster-users@gluster.org 
Sent: Thursday, October 10, 2019 8:08:24 PM 
Subject: [Gluster-users] Gluster slow reads 


Hi, 

I have a very small distributed dispersed volume with 4+2 and 2 servers 
with 3 bricks on each of them. The volume is mounted via fuse client on 
another Linux server. 

The volume worked well for a few months in this setup. However in the 
last few days I have a very slow read speed (gluster uploads to the 
gluster client via the mount point). By slow I mean 3-4 MB/sec on a 
gigabit link. I don't have small files stored on gluster, the smallest 
file is around 30-40 MB. The networking between the client and the 
bricks is fine (all of them are connected in the same switch, no errors, 
some iptraf tests directly between client and gluster servers are 
looking good). On the same client I have mounted another 2 gluster 
volumes from other servers, both of them are ok. 

My 'gluster volume info' details are the following : 

Volume Name: gluster4-vol 
Type: Disperse 
Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 1 x (4 + 2) = 6 
Transport-type: tcp 
Bricks: 
Brick1: gluster4:/export/sdb1/brick 
Brick2: gluster5:/export/sdb1/brick 
Brick3: gluster4:/export/sdc1/brick 
Brick4: gluster5:/export/sdc1/brick 
Brick5: gluster4:/export/sdd1/brick 
Brick6: gluster5:/export/sdd1/brick 
Options Reconfigured: 
network.ping-timeout: 60 
performance.client-io-threads: on 
performance.io-thread-count: 32 
cluster.readdir-optimize: on 
performance.cache-size: 1GB 
client.event-threads: 10 
server.event-threads: 10 
cluster.lookup-optimize: on 
server.allow-insecure: on 
storage.reserve: 0 
transport.address-family: inet 
nfs.disable: on 


In the gluster client debug logs I noticed the following lines : 

[2019-10-10 13:49:58.571879] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN 
scheduled as fast fop 
[2019-10-10 13:49:58.571964] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT 
scheduled as fast fop 
[2019-10-10 13:49:58.572058] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH 
scheduled as normal fop 
[2019-10-10 13:49:58.572728] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN 
scheduled as fast fop 
[2019-10-10 13:49:58.576275] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT 
scheduled as fast fop 
[2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk 
The message "D [MSGID: 0] [io-threads.c:356:iot_schedule] 
0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285 
times between [2019-10-10 13:49:58.357922] and [2019-10-10 
13:50:07.837047] 
[2019-10-10 13:50:07.837068] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH 
scheduled as normal fop 
[2019-10-10 13:50:07.837165] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ 
scheduled as slow fop 


What does it mean exactly "READ scheduled as slow fop" ? Can I schedule 
READs as normal or fast fop like the other operations ? 

I'm using this gluster volume only for reading so I don't care about 
writes right now. 


Thanks. 

 

Community Meeting Calendar: 

APAC Schedule - 
Every 2nd and 4th Tuesday at 11:30 AM IST 
Bridge: https://bluejeans.com/118564314 

NA/EMEA Schedule - 
Every 1st and 3rd Tuesday at 01:00 PM EDT 
Bridge: https://bluejeans.com/118564314 

Gluster-users mailing list 
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users