[Gluster-users] Brick Disconnect from Gluster Volume
Hi All, We are facing Brick Disconnect issue on our setup , please find more info about the issue below: *Gluster: 5.3-6 node distributed Cluster -No replication-2 Bricks Per Node-Ubuntu 18.04* *-Issue:* 1 of the Brick on 1 node keeps disconnecting , (after every n days) not very frequently , but observed more than 3-4 times Nothing much in glusterd or bricks logs *GlusterD Logs:* [2019-08-23 10:39:29.811950] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-08-23 10:39:29.815723] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-08-23 10:39:38.88] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 77 times between [2019-08-23 10:39:38.88] an d [2019-08-23 10:39:44.990156] ^^ Messages keeps pouring in the logs , this from time when we saw loss of 1 brick,nothing specail about the above logs , but this is what is there in logs from the time we saw loss of the brick *DataBrick Logs:* The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 4788 times between [2019-08-23 10:39:44.993667] ^^ Same here we see this type to logs only Would be helpful if we can get some pointers about the above issue. Thanks Ashayam ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] sharding in glusterfs
Hi Pranith, Thanks for you reply, it would be helpful if you can please help us with the following issues with respect to sharding. The gluster version we are using is *glusterfs 4.1.4 *on Ubuntu 18.04.1 LTS - *Shards-Creation Algo*: We were interested in understanding the way in which shards are distributed across bricks and nodes, is it Round-Robin or some other algo and can we change this mechanism using some config file. E.g. If we have 2 nodes with each nodes having 2 bricks , with a total of 4 (2*2) bricks how will the shards be distributed, will it be always even distribution?(Volume type in this case is plain) - *Sharding+Distributed-Volume*: Currently we are using plain volume with sharding enabled and we do not see even distribution of shards across bricks .Can we use sharding with distributed volume to achieve evenly and better distribution of shards? Would be helpful if you can suggest the most efficient way of using sharding , our goal is to have a evenly distributed file system(we have large files hence using sharding) and we are not concerned with replication as of now. - *Shard-Block-Size: *In case we change the * features.shard-block-size* value from X -> Y after lots of data has been populated , how does this affect the existing shards are they auto corrected as per the new size or do we need to run some commdands to get this done or is this even recommended to do the change? - *Rebalance-Shard*: As per the docs whenever we add new server/node to the existing gluster we need to run Rebalance command, we would like to know if there are any known issues for re-balancing with sharding enabled. We would highly appreciate if you can point us to the latest sharding docs, we tried to search but could not find better than this https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/ . Thanks Ashayam On Thu, Sep 20, 2018 at 7:47 PM Pranith Kumar Karampuri wrote: > > > On Wed, Sep 19, 2018 at 11:37 AM Ashayam Gupta < > ashayam.gu...@alpha-grep.com> wrote: > >> Please find our workload details as requested by you : >> >> * Only 1 write-mount point as of now >> * Read-Mount : Since we auto-scale our machines this can be as big as >> 300-400 machines during peak times >> * >" multiple concurrent reads means that Reads will not happen until the >> file is completely written to" Yes , in our current scenario we can ensure >> that indeed this is the case. >> >> But when you say it only supports single writer workload we would like to >> understand the following scenarios with respect to multiple writers and the >> current behaviour of glusterfs with sharding >> >>- Multiple Writer writes to different files >> >> When I say multiple writers, I mean multiple mounts. Since you were > saying earlier there is only one mount which does all writes, everything > should work as expected. > >> >>- Multiple Writer writes to same file >> - they write to same file but different shards of same file >> - they write to same file (no gurantee if they write to different >> shards) >> >> As long as the above happens from same mount, things should be fine. > Otherwise there could be problems. > > >> There might be some more cases which are known to you , would be helpful >> if you can describe us about those scenarios as well or may point us to the >> relevant documents. >> > Also it would be helpful if you can suggest the most stable version of >> glusterfs with sharding feature to use , since we would like to use this in >> production. >> > > It has been stable for a while, so use any of the latest maintained > releases like 3.12.x or 4.1.x > > As I was mentioning already, sharding is mainly tested with > VM/gluster-block workloads. So there could be some corner cases with single > writer workload which we never ran into for the VM/block workloads we test. > But you may run into them. Do let us know and we can take a look if you > find something out of the ordinary. What I would suggest is to use one of > the maintained releases and run the workloads you have for some time to > test things out, once you feel confident, you can put it in production. > > HTH > >> >> Thanks >> Ashayam Gupta >> >> On Tue, Sep 18, 2018 at 11:00 AM Pranith Kumar Karampuri < >> pkara...@redhat.com> wrote: >> >>> >>> >>> On Mon, Sep 17, 2018 at 4:14 AM Ashayam Gupta < >>> ashayam.gu...@alpha-grep.com> wrote: >>> >>>> Hi All, >>>> >>>> We are currently using glusterfs for storing large files with >>>> wr
Re: [Gluster-users] sharding in glusterfs
Please find our workload details as requested by you : * Only 1 write-mount point as of now * Read-Mount : Since we auto-scale our machines this can be as big as 300-400 machines during peak times * >" multiple concurrent reads means that Reads will not happen until the file is completely written to" Yes , in our current scenario we can ensure that indeed this is the case. But when you say it only supports single writer workload we would like to understand the following scenarios with respect to multiple writers and the current behaviour of glusterfs with sharding - Multiple Writer writes to different files - Multiple Writer writes to same file - they write to same file but different shards of same file - they write to same file (no gurantee if they write to different shards) There might be some more cases which are known to you , would be helpful if you can describe us about those scenarios as well or may point us to the relevant documents. Also it would be helpful if you can suggest the most stable version of glusterfs with sharding feature to use , since we would like to use this in production. Thanks Ashayam Gupta On Tue, Sep 18, 2018 at 11:00 AM Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > > > On Mon, Sep 17, 2018 at 4:14 AM Ashayam Gupta < > ashayam.gu...@alpha-grep.com> wrote: > >> Hi All, >> >> We are currently using glusterfs for storing large files with write-once >> and multiple concurrent reads, and were interested in understanding one of >> the features of glusterfs called sharding for our use case. >> >> So far from the talk given by the developer [ >> https://www.youtube.com/watch?v=aAlLy9k65Gw] and the git issue [ >> https://github.com/gluster/glusterfs/issues/290] , we know that it was >> developed for large VM images as use case and the second link does talk >> about a more general purpose usage , but we are not clear if there are some >> issues if used for non-VM image large files [which is the use case for us]. >> >> Therefore it would be helpful if we can have some pointers or more >> information about the more general use-case scenario for sharding and any >> shortcomings if any , in case we use it for our scenario which is non-VM >> large files with write-once and multiple concurrent reads.Also it would be >> very helpful if you can suggest the best approach/settings for our use case >> scenario. >> > > Sharding is developed for Big file usecases and at the moment only > supports single writer workload. I also added the maintainers for sharding > to the thread. May be giving a bit of detail about access pattern w.r.t. > number of mounts that are used for writing/reading would be helpful. I am > assuming write-once and multiple concurrent reads means that Reads will not > happen until the file is completely written to. Could you explain a bit > more about the workload? > > >> >> Thanks >> Ashayam Gupta >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Pranith > ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] sharding in glusterfs
Hi All, We are currently using glusterfs for storing large files with write-once and multiple concurrent reads, and were interested in understanding one of the features of glusterfs called sharding for our use case. So far from the talk given by the developer [ https://www.youtube.com/watch?v=aAlLy9k65Gw] and the git issue [ https://github.com/gluster/glusterfs/issues/290] , we know that it was developed for large VM images as use case and the second link does talk about a more general purpose usage , but we are not clear if there are some issues if used for non-VM image large files [which is the use case for us]. Therefore it would be helpful if we can have some pointers or more information about the more general use-case scenario for sharding and any shortcomings if any , in case we use it for our scenario which is non-VM large files with write-once and multiple concurrent reads.Also it would be very helpful if you can suggest the best approach/settings for our use case scenario. Thanks Ashayam Gupta ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
