[Gluster-users] State of Gluster project
Hello, I'm wondering what's the current and future plan for Gluster project overall, I see that the project is not as busy as it was before "at least this is what I'm seeing" Like there are fewer blogs about what the roadmap or future plans of the project, the deprecation of Glusterd2, even Red Hat Openshift storage switched to Ceph. As the community of this project, do you feel the same? Is the deprecation of Glusterd2 concerning? Do you feel that the project is slowing down somehow? Do you think Red Hat is abandoning the project or giving fewer resources to Gluster? -- Respectfully Mahdi Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Has anyone tried to pit Ceph against gluster? I'm curious what the ups and downs are. On Tue, Jun 16, 2020, 4:32 PM Strahil Nikolov wrote: > Hey Mahdi, > > For me it looks like Red Hat are focusing more on CEPH than on Gluster. > I hope the project remains active, cause it's very difficult to find a > Software-defined Storage as easy and as scalable as Gluster. > > Best Regards, > Strahil Nikolov > > На 17 юни 2020 г. 0:06:33 GMT+03:00, Mahdi Adnan написа: > >Hello, > > > > I'm wondering what's the current and future plan for Gluster project > >overall, I see that the project is not as busy as it was before "at > >least > >this is what I'm seeing" Like there are fewer blogs about what the > >roadmap > >or future plans of the project, the deprecation of Glusterd2, even Red > >Hat > >Openshift storage switched to Ceph. > >As the community of this project, do you feel the same? Is the > >deprecation > >of Glusterd2 concerning? Do you feel that the project is slowing down > >somehow? Do you think Red Hat is abandoning the project or giving fewer > >resources to Gluster? > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hey Mahdi, For me it looks like Red Hat are focusing more on CEPH than on Gluster. I hope the project remains active, cause it's very difficult to find a Software-defined Storage as easy and as scalable as Gluster. Best Regards, Strahil Nikolov На 17 юни 2020 г. 0:06:33 GMT+03:00, Mahdi Adnan написа: >Hello, > > I'm wondering what's the current and future plan for Gluster project >overall, I see that the project is not as busy as it was before "at >least >this is what I'm seeing" Like there are fewer blogs about what the >roadmap >or future plans of the project, the deprecation of Glusterd2, even Red >Hat >Openshift storage switched to Ceph. >As the community of this project, do you feel the same? Is the >deprecation >of Glusterd2 concerning? Do you feel that the project is slowing down >somehow? Do you think Red Hat is abandoning the project or giving fewer >resources to Gluster? Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hi Mahdi, On Wed, Jun 17, 2020 at 5:02 AM Strahil Nikolov wrote: > Hey Mahdi, > > For me it looks like Red Hat are focusing more on CEPH than on Gluster. > I hope the project remains active, cause it's very difficult to find a > Software-defined Storage as easy and as scalable as Gluster. > > Best Regards, > Strahil Nikolov > > На 17 юни 2020 г. 0:06:33 GMT+03:00, Mahdi Adnan написа: > >Hello, > > > > I'm wondering what's the current and future plan for Gluster project > >overall, I see that the project is not as busy as it was before "at > >least > >this is what I'm seeing" Like there are fewer blogs about what the > >roadmap > >or future plans of the project, the deprecation of Glusterd2, even Red > >Hat > >Openshift storage switched to Ceph. > I can't talk about Red Hat's focus or roadmap, but as far as Gluster Project is concerned, the project will be maintained, and improvements would be made. That is a promise from us at Kadalu.IO. Just that rate of activity may be lesser than earlier depending on how much other consumers of the project contribute for development! > >As the community of this project, do you feel the same? Is the > >deprecation > >of Glusterd2 concerning? Do you feel that the project is slowing down > >somehow? Do you think Red Hat is abandoning the project or giving fewer > >resources to Gluster? > A project's activities are directly related to people contributing, and there seems to be a low tide right now. About glusterd2 is concerned, the project's abandoning decision happened almost 18months or so back. So, not something new IMO. But, that shouldn't stop one from thinking of better alternatives to gluster's management layer. Aravinda, Co-Founder of https://kadalu.io, started a very nice project called moana (https://github.com/kadalu/moana) which I believe has a very good potential to make many of gluster's management challenges go away! Regards, Amar > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -- -- https://kadalu.io Container Storage made easy! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
17.06.2020 01:06, Mahdi Adnan пишет: Hello, I'm wondering what's the current and future plan for Gluster project overall, I see that the project is not as busy as it was before "at least this is what I'm seeing" Like there are fewer blogs about what the roadmap or future plans of the project, the deprecation of Glusterd2, even Red Hat Openshift storage switched to Ceph. As the community of this project, do you feel the same? Is the deprecation of Glusterd2 concerning? Do you feel that the project is slowing down somehow? Do you think Red Hat is abandoning the project or giving fewer resources to Gluster? Gluster2 was mistake, imho. It's deprecation means nothing. For me looks like gluster in now stable , this is why it is not as busy as before. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
The note from Mahdi raises some key aspects in which our community evaluates the project and we should consider those as important key indicators. The essence of what was put together at https://www.gluster.org/building-a-longer-term-focus-for-gluster/ continues to hold true and a summary of the work ongoing is provided at https://www.gluster.org/update-from-the-team/ I agree that this is a good time to build on the last update and provide better guidance on the improvements coming up. Gluster will continue to deliver on the promise of a simple and scalable software defined storage. What we need includes more focus on creating test paths and testing matrix which allows more complex scenarios to be tested. This helps us address the topics raised often during release conversations - performance for workloads (and there's already work underway for this). I'll take away the expressed need to be more open about what is happening with the project and be able to share it with the community. On Wed, 17 Jun 2020 at 09:52, Amar Tumballi wrote: > > Hi Mahdi, > > On Wed, Jun 17, 2020 at 5:02 AM Strahil Nikolov wrote: >> >> Hey Mahdi, >> >> For me it looks like Red Hat are focusing more on CEPH than on Gluster. >> I hope the project remains active, cause it's very difficult to find a >> Software-defined Storage as easy and as scalable as Gluster. >> >> Best Regards, >> Strahil Nikolov >> >> На 17 юни 2020 г. 0:06:33 GMT+03:00, Mahdi Adnan написа: >> >Hello, >> > >> > I'm wondering what's the current and future plan for Gluster project >> >overall, I see that the project is not as busy as it was before "at >> >least >> >this is what I'm seeing" Like there are fewer blogs about what the >> >roadmap >> >or future plans of the project, the deprecation of Glusterd2, even Red >> >Hat >> >Openshift storage switched to Ceph. > > > I can't talk about Red Hat's focus or roadmap, but as far as Gluster Project > is concerned, the project will be maintained, and improvements would be made. > That is a promise from us at Kadalu.IO. Just that rate of activity may be > lesser than earlier depending on how much other consumers of the project > contribute for development! > >> >> >As the community of this project, do you feel the same? Is the >> >deprecation >> >of Glusterd2 concerning? Do you feel that the project is slowing down >> >somehow? Do you think Red Hat is abandoning the project or giving fewer >> >resources to Gluster? > > > A project's activities are directly related to people contributing, and there > seems to be a low tide right now. > > About glusterd2 is concerned, the project's abandoning decision happened > almost 18months or so back. So, not something new IMO. But, that shouldn't > stop one from thinking of better alternatives to gluster's management layer. > Aravinda, Co-Founder of https://kadalu.io, started a very nice project called > moana (https://github.com/kadalu/moana) which I believe has a very good > potential to make many of gluster's management challenges go away! -- sankars...@kadalu.io | TZ: UTC+0530 | +91 99606 03294 kadalu.io : Making it easy to provision storage in k8s! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hi All, Talking about the gluster community contribution outside of Red Hat, you can count me in. I'm working as software engineer at iTernity GmbH, the company is committed to provide an archival solution. iTernity has been using Gluster backed storage solution mainly for archival use case. We have two dedicated developers to work on Gluster, David and Myself. We will be contributing/maintaining to the Gluster community mainly worm xlator, halo replication, snapshot feature, and probably crypto xlator in the future. Needless to say we will also be very happy to contribute to the other components of Gluster. So in general we are very happy involve in ways to make Gluster community more vibrant and active. Regards Rafi KC From: gluster-users-boun...@gluster.org On Behalf Of Aravinda VK Sent: 17 June 2020 11:09 AM To: Mahdi Adnan Cc: gluster-users Subject: Re: [Gluster-users] State of Gluster project Hi Mahdi, I am writing my views as an external contributor outside the Red Hat. Hopefully, someone from Red Hat will respond with their focus/roadmap on Gluster development. Recently Amar wrote about the focus of the GlusterFS core team(https://www.gluster.org/update-from-the-team/). He mentioned that the attention of the core team would be stabilizing the FS layer and enable the ecosystem to adapt to their environment quickly. >From Kadalu.io<http://Kadalu.io>(https://kadalu.io), we love GlusterFS and >focusing on creating an Opinionated Storage solution based on GlusterFS. >GlusterFS is so modular that it enables us to integrate natively with >Kubernetes APIs without the need of Gluster's management layer Glusterd. Other >projects we are focussing on now are Moana and Binnacle. >Moana(https://github.com/kadalu/moana) is an external control plane for >managing the GlusterFS storage layer. And >Binnacle(https://github.com/kadalu/binnacle) is a distributed test framework >with the focus on Tester's delight. I do see a lot of activity in our Slack channel, at present more than 200 members in the Gluster Slack channel, which is encouraging. Please do join if not already joined. https://join.slack.com/t/gluster/shared_invite/enQtODMwMDU5MTI0OTQ3LTNjMTA4NTJmMDY3OGM4YTA0ZDlhOGM5ZWYzNjRkYmQ0Mjg3NDUxODRjZGI4ZjQxNDczNTYxZWZjMmY5NDMyNzM In summary, I agree that GlusterFS needs a lot of improvement in Performance, Monitoring, Documentation, and other areas. I am very hopeful that we will get there soon. -- Regards Aravinda Vishwanathapura https://kadalu.io On 17-Jun-2020, at 2:36 AM, Mahdi Adnan mailto:ma...@sysmin.io>> wrote: Hello, I'm wondering what's the current and future plan for Gluster project overall, I see that the project is not as busy as it was before "at least this is what I'm seeing" Like there are fewer blogs about what the roadmap or future plans of the project, the deprecation of Glusterd2, even Red Hat Openshift storage switched to Ceph. As the community of this project, do you feel the same? Is the deprecation of Glusterd2 concerning? Do you feel that the project is slowing down somehow? Do you think Red Hat is abandoning the project or giving fewer resources to Gluster? -- Respectfully Mahdi Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org<mailto:Gluster-users@gluster.org> https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
We never ran tests with Ceph mostly due to time constraints in engineering. We also liked that, at least when I started as a novice, gluster seemed easier to set up. We use the solution in automated setup scripts for maintaining very large clusters. Simplicity in automated setup is critical here for us including automated installation of supercomputers in QE and near-automation at customer sites. We have been happy with our performance using gluster and gluster NFS for root filesystems when using squashfs object files for the NFS roots as opposed to expanded files (on a sharded volume). For writable NFS, we use XFS filesystem images on gluster NFS instead of expanded trees (in this case, not on sharded volume). We have systems running as large as 3072 nodes with 16 gluster servers (subvolumes of 3, distributed/replicate). We will have 5k nodes in production soon and will need to support 10k nodes in a year or so. So far we use CTDB for "ha-like" functionality as pacemaker is scary to us. We also have designed a second solution around gluster for high-availability head nodes (aka admin nodes). The old solution used two admin nodes, pacemaker, external shared storage, to host a VM that would start on the 2nd server if the first server died. As we know, 2-node ha is not optimal. We designed a new 3-server HA solution that eliminates the external shared storage (which was expensive) and instead uses gluster, sharded volume, and a qemu raw image hosted in the shared storage to host the virtual admin node. We use RAIDD10 4-disk per server for gluster use in this. We have been happy with the performance of this. It's only a little slower than the external shared filesystem solution (we tended to use GFS2 or OCFS or whatever it is called in the past solution). We did need to use pacemaker for this one as virtual machine availability isn't suitable for CTDB (or less natural anyway). One highlight of this solution is it allows a customer to put each of the 3 servers in a separate firewalled vault or room to keep the head alive even if there were a fire that destroyed one server. A key to our use of gluster and not suffering from poor performance in our root-filesystem-workloads is encapsulating filesystems in image files instead of using expanded trees of small files. So far we have relied on gluster NFS for the boot servers as Ganesha would crash. We haven't re-tried in several months though and owe debugging on that front. We have not had resources to put in to debugging Ganesha just yet. I sure hope Gluster stays healthy and active. It is good to have multiple solutions with various strengths out there. I like choice. Plus, choice lets us learn from each other. I hope project sponsors see that too. Erik > 17.06.2020 08:59, Artem Russakovskii пишет: > > It may be stable, but it still suffers from performance issues, which > > the team is working on. But nevertheless, I'm curious if maybe Ceph has > > those problem sorted by now. > > > Dunno, we run gluster on small clusters, kvm and gluster on the same hosts. > > There were plans to use ceph on dedicated server next year, but budget cut > because you don't want to buy our oil for $120 ;-) > > Anyway, in our tests ceph is faster, this is why we wanted to use it, but > not migrate from gluster. > > > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
> It is very hard to compare them because they are structurally very different. > For example, GlusterFS performance will depend *a lot* on the underlying file > system performance. Ceph eliminated that factor by using Bluestore. > Ceph is very well performing for VM storage, since it's block based and as > such optimized for that. I haven't tested CephFS a lot (I used it but only > for very small storage) so I cannot speak for its performance, but I am > guessing it's not ideal. For large amount of files thus GlusterFS is still a > good choice. Was your experience above based on using a sharded volume or a normal one? When we worked with virtual machine images, we followed the volume sharding advice. I don't have a comparison for Ceph handy. I was just curious. It worked so well for us (but maybe our storage is "too good") that we found it hard to imagine it could be improved much. This was a simple case though of a single VM, 3 gluster servers, a sharded volume, and a raw virtual machine image. Probably a simpler case than yours. Thank you for writing this and take care, Erik > > One *MAJOR* advantage of Ceph over GlusterFS is tooling. Ceph's > self-analytics, status reporting and problem fixing toolset is just so far > beyond GlusterFS that it's really hard for me to recommend GlusterFS for any > but the most experienced sysadmins. It does come with the type of > implementation Ceph has chosen that they have to have such good tooling > (because honestly, poking around in binary data structures really wouldn't be > practical for most users), but whenever I had a problem with Ceph the > solution was just a couple of command line commands (even if it meant to > remove a storage device, wipe it and add it back), where with GlusterFS it > means poking around in the .glusterfs directory, looking up inode numbers, > extended attributes etc. which is a real pain if you have a > multi-million-file filesystem to work on. And that's not even with sharding > or distributed volumes. > > Also, Ceph has been a lot more stable that GlusterFS for us. The amount of > hand-holding GlusterFS needs is crazy. With Ceph, there is this one bug (I > think in certain Linux kernel versions) where it sometimes reads only zeroes > from disk and complains about that and then you have to restart that OSD to > not run into problems, but that's one "swatch" process on each machine that > will do that automatically for us. I have run some Ceph clusters for several > years now and only once or twice I had to deal with problems. The several > GlusterFS clusters we operate constantly run into troubles. We now shut down > all GlusterFS clients before we reboot any GlusterFS node because it was near > impossible to reboot a single node without running into unrecoverable > troubles (heal entries that will not heal etc.). With Ceph we can achieve > 100% uptime, we regularly reboot our hosts one by one and some minutes later > the Ceph cluster is clean again. > > If others have more insights I'd be very happy to hear them. > > Stefan > > > - Original Message - > > Date: Tue, 16 Jun 2020 20:30:34 -0700 > > From: Artem Russakovskii > > To: Strahil Nikolov > > Cc: gluster-users > > Subject: Re: [Gluster-users] State of Gluster project > > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > > > Has anyone tried to pit Ceph against gluster? I'm curious what the ups and > > downs are. > > > > On Tue, Jun 16, 2020, 4:32 PM Strahil Nikolov wrote: > > > >> Hey Mahdi, > >> > >> For me it looks like Red Hat are focusing more on CEPH than on Gluster. > >> I hope the project remains active, cause it's very difficult to find a > >> Software-defined Storage as easy and as scalable as Gluster. > >> > >> Best Regards, > >> Strahil Nikolov > >> > >> ?? 17 ??? 2020 ?. 0:06:33 GMT+03:00, Mahdi Adnan ??: > >> >Hello, > >> > > >> > I'm wondering what's the current and future plan for Gluster project > >> >overall, I see that the project is not as busy as it was before "at > >> >least > >> >this is what I'm seeing" Like there are fewer blogs about what the > >> >roadmap > >> >or future plans of the project, the deprecation of Glusterd2, even Red > >> >Hat > >> >Openshift storage switched to Ceph. > >> >As the community of this project, do you feel the same? Is the > >> >deprecation > >> >of Glusterd2 concerning?
Re: [Gluster-users] State of Gluster project
Hm... Actually comparing CEPH to Gluster is like to compare apples to pears. I have played with both and I can say that CEPH is optimised for disk images - which gives benefit in Openstack and virtualizations, but CEPH is harder to learn and lacks some features of Gluster. For examlle Gluster has: 1. On Fedora-bases systems, you can use Virtual Data Optimizer (VDO) for data deduplication or data compression On Debian Based - ZOL is an option 2. Geo-replication - very useful for setting up DR 3. In replica and dispersed volumes , you can loose any node (sometimes more) without worrying about the type of the system - they have the same role and same purpose 4. Gluster clients are more diverse - it is nice that Windows(cifs), Apple(whatever they call it), Linux(fuse/nfs) and BSDs(nfs) can use the volume. 5. Easier setup with less nodes - Gluster is easy to setup and with sharding - is great even for virtualization. The client uses the algorithm to find the file without the necessity to query another server. Of course Gluster has it's own drawbacks - but the software is optimal for HPC and Archive storage.If you know the path to your file, Gluster will retrieve it quite fast. I have to mention that in the past years (since Dec 2017 to be more precise) , I had only 2 major issues - always during upgrades. When I compare another environment with enterprise-class storages (in cluster mode) , which also failed once after an update (3 years period of comparison) - the issue rate is not so big, but the peice for Gluster is not millions :) Best Regards, Strahil Nikolov На 17 юни 2020 г. 19:15:00 GMT+03:00, Erik Jacobson написа: >> It is very hard to compare them because they are structurally very >different. For example, GlusterFS performance will depend *a lot* on >the underlying file system performance. Ceph eliminated that factor by >using Bluestore. >> Ceph is very well performing for VM storage, since it's block based >and as such optimized for that. I haven't tested CephFS a lot (I used >it but only for very small storage) so I cannot speak for its >performance, but I am guessing it's not ideal. For large amount of >files thus GlusterFS is still a good choice. > > >Was your experience above based on using a sharded volume or a normal >one? When we worked with virtual machine images, we followed the volume >sharding advice. I don't have a comparison for Ceph handy. I was just >curious. It worked so well for us (but maybe our storage is "too good") >that we found it hard to imagine it could be improved much. This was a >simple case though of a single VM, 3 gluster servers, a sharded volume, >and a raw virtual machine image. Probably a simpler case than yours. > >Thank you for writing this and take care, > >Erik > >> >> One *MAJOR* advantage of Ceph over GlusterFS is tooling. Ceph's >self-analytics, status reporting and problem fixing toolset is just so >far beyond GlusterFS that it's really hard for me to recommend >GlusterFS for any but the most experienced sysadmins. It does come with >the type of implementation Ceph has chosen that they have to have such >good tooling (because honestly, poking around in binary data structures >really wouldn't be practical for most users), but whenever I had a >problem with Ceph the solution was just a couple of command line >commands (even if it meant to remove a storage device, wipe it and add >it back), where with GlusterFS it means poking around in the .glusterfs >directory, looking up inode numbers, extended attributes etc. which is >a real pain if you have a multi-million-file filesystem to work on. And >that's not even with sharding or distributed volumes. >> >> Also, Ceph has been a lot more stable that GlusterFS for us. The >amount of hand-holding GlusterFS needs is crazy. With Ceph, there is >this one bug (I think in certain Linux kernel versions) where it >sometimes reads only zeroes from disk and complains about that and then >you have to restart that OSD to not run into problems, but that's one >"swatch" process on each machine that will do that automatically for >us. I have run some Ceph clusters for several years now and only once >or twice I had to deal with problems. The several GlusterFS clusters we >operate constantly run into troubles. We now shut down all GlusterFS >clients before we reboot any GlusterFS node because it was near >impossible to reboot a single node without running into unrecoverable >troubles (heal entries that will not heal etc.). With Ceph we can >achieve 100% uptime, we regularly reboot our hosts one by one and some >minutes later the Ceph cluster is clean again. >> >> If others have more insights I'd be very happy to he
Re: [Gluster-users] State of Gluster project
On 6/17/20 6:19 AM, Dmitry Melekhov wrote: 17.06.2020 01:06, Mahdi Adnan пишет: Hello, I'm wondering what's the current and future plan for Gluster project overall, I see that the project is not as busy as it was before "at least this is what I'm seeing" Like there are fewer blogs about what the roadmap or future plans of the project, the deprecation of Glusterd2, even Red Hat Openshift storage switched to Ceph. As the community of this project, do you feel the same? Is the deprecation of Glusterd2 concerning? Do you feel that the project is slowing down somehow? Do you think Red Hat is abandoning the project or giving fewer resources to Gluster? Gluster2 was mistake, imho. It's deprecation means nothing. For me looks like gluster in now stable , this is why it is not as busy as before. Some parts of Gluster have been really stable for a very long time now. Which is good, IMHO. I want to be bored by storage, because valuable data is there. And new features obviously make situation less boring ;) bc they are... new (and buggy). Furthermore it should be remembered that Gluster had been driven for a long time by RedHat, that is a company, and used gluster community in a similar way to what they do with Fedora/RHEL. You want maximum stability, you pay for RH Gluster Storage (or whatever it is called now). You go with community, you have similar risks to runing Fedora in production. Having said that, I really appreciated the statement that core development will focus on stability first. Gluster community now is very different from what it was just 2 years ago. Several Gluster people left RH or moved to different projects. Conversely, new companies are now involved with Gluster. In a sense it may be a new start. Let's see as it turns out. Ivan Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
On Wed, 17 Jun 2020 00:06:33 +0300 Mahdi Adnan wrote: > [gluster going down ] I am following this project for quite some years now, probably longer than most of the people nowadays on the list. The project started with the brilliant idea of making a fs on top of classical fs's distributed over several hardware pieces without need to re-copy data for entering or leaving the gluster. (I thought) it started as a proof-of-concept fs on fuse with the intention to turn into kernel-space as soon as possible to get the performance that it should have for a fs. After five years of waiting (and using) I declared the project dead for our use (about five years ago) because it evolved more and more to bloatware. And I do think that Red Hat understood that finally (too), and what you mentioned is just the outcome of that. I really hate the way this project took, because for me it was visible from the very start that it is a dead-end. After all those years it is bloatware on fuse. And I feel very sorry for that brilliant idea that it once was. _FS IN USERSPACE IS SH*T_ - understand that. -- Regards, Stephan Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
18.06.2020 12:54, Stephan von Krawczynski пишет: _FS IN USERSPACE IS SH*T_ - understand that. we use qemu and it uses gfapi... :-) Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
On Thu, 18 Jun 2020 13:06:51 +0400 Dmitry Melekhov wrote: > 18.06.2020 12:54, Stephan von Krawczynski пишет: > > > > _FS IN USERSPACE IS SH*T_ - understand that. > > > > we use qemu and it uses gfapi... :-) And exactly this kind of "insight" is base of my critics. gfapi is _userspace_ on client (given, without fuse), but does not at all handle the basic glusterfs problem: the need to go through _userspace_ on _server_. Simply look at the docs here and understand where the work should have been done: https://www.humblec.com/libgfapi-interface-glusterfs/ On the server you have to go from kernel-space network to userspace glusterfs back to kernel-space underlying fs. So gfapi only eliminates one of two major problems. Comparing performance to NFS on ZFS shows the flaw. If it was implemented like it should you would have almost _no_ difference, because you would be able to split up the two network paths to gluster servers (for a setup with two) on different switches and network cards on client. So for reading it should be as fast, for writing you may calculate a (very) small loss. -- Regards, Stephan Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
18.06.2020 13:55, Stephan von Krawczynski пишет: On Thu, 18 Jun 2020 13:06:51 +0400 Dmitry Melekhov wrote: 18.06.2020 12:54, Stephan von Krawczynski пишет: _FS IN USERSPACE IS SH*T_ - understand that. we use qemu and it uses gfapi... :-) And exactly this kind of "insight" is base of my critics. gfapi is _userspace_ on client (given, without fuse), but does not at all handle the basic glusterfs problem: the need to go through _userspace_ on _server_. Sorry, I can't understand what is real problem here? Gluster is not fast, but I don't think problem is in client in userspace. I guess it is possible to write kernel module :-) If it was implemented like it should you would have almost _no_ difference, because you would be able to split up the two network paths to gluster servers (for a setup with two) on different switches and network cards on client. Never had problem in this on 10Gbit network, we can't saturate 10Gbit (well, really 20 ) using gluster. btw, gluster 4 was promised to have different networks for clients and inter-server replication ( healing) , but when glusterd2 arrived it has nothing about this. Anyway, we are quite happy with our current gluster setup, which allows us have just 3 servers and no shared storage in several places, not suitable for all our workloads though, so gluster is not universal tool. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
You're still here and still hurt about that? It was never intended to be in kernel. It was always intended to run in userspace. After all these years I thought you'd be over that by now. On June 18, 2020 1:54:18 AM PDT, Stephan von Krawczynski wrote: >On Wed, 17 Jun 2020 00:06:33 +0300 >Mahdi Adnan wrote: > >> [gluster going down ] > >I am following this project for quite some years now, probably longer >than >most of the people nowadays on the list. The project started with the >brilliant idea of making a fs on top of classical fs's distributed over >several hardware pieces without need to re-copy data for entering or >leaving >the gluster. (I thought) it started as a proof-of-concept fs on fuse >with the >intention to turn into kernel-space as soon as possible to get the >performance >that it should have for a fs. >After five years of waiting (and using) I declared the project dead for >our >use (about five years ago) because it evolved more and more to >bloatware. And >I do think that Red Hat understood that finally (too), and what you >mentioned >is just the outcome of that. >I really hate the way this project took, because for me it was visible >from >the very start that it is a dead-end. After all those years it is >bloatware on >fuse. And I feel very sorry for that brilliant idea that it once was. >_FS IN USERSPACE IS SH*T_ - understand that. > > >-- >Regards, >Stephan > > > > >Community Meeting Calendar: > >Schedule - >Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >Bridge: https://bluejeans.com/441850968 > >Gluster-users mailing list >Gluster-users@gluster.org >https://lists.gluster.org/mailman/listinfo/gluster-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
On Thu, 18 Jun 2020 07:40:36 -0700 Joe Julian wrote: > You're still here and still hurt about that? It was never intended to be in > kernel. It was always intended to run in userspace. After all these years I > thought you'd be over that by now. Top Poster ;-) And in fact, it's not true. The clear message to me once was: we are not able to make a kernel version. Which I understood as: we have not the knowledge to do that. Since that was quite some time before Red Hat stepped in there was still hope that some day someone capable may come ... Since 2009 when I entered the list there was not a single month where there were no complaints about gluster being slow. I wonder if you could accept after 11 years and the projects near death now that I was right from the very first day. -- Regards, Stephan Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
18.06.2020 20:41, Stephan von Krawczynski пишет: Since 2009 when I entered the list there was not a single month where there were no complaints about gluster being slow. It is slow not because client works in userspace, we have no problems with cpu load here, caused by context switching, yes, this also adds delay, but it is acceptable in our case. Really, I don't know why it is slow. And yes, we know how it looks when userspace applications reach limits- for instance, openvpn works in userspace too and is siglethreaded, so yes, we hit this limitation during this idiotic "self"isolation providing remote access for hundreds of users. I wonder if you could accept after 11 years and the projects near death Near death? Really? AFAIK ( I'm not one who runs gluster servers, I'm one who helps to fix them ), we have no gluster crashes for about a year now, this is not death, this is stable state :-) Oh, I already wrote this... Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
On Thu, 18 Jun 2020 13:27:19 -0400 Alvin Starr wrote: > > [me] > This is an amazingly unreasonable comment. > First off ALL distributed file systems are slower than non-distributed > file systems. Obviously you fail to understand my point: the design of glusterfs implies that it can be as fast as a non-distributed fs. If you have not understood this by now you should stay another 10 years on this list. As glusterfs should only read from a single node, and write concurrently to all nodes it must only be slower than a non-distributed fs if your network is not designed according to the needed paths. Glusterfs being slow is not by design but by implementation. > Second ALL network file systems are slower than local hardware. Uh, really no comment. > Kernel inclusion does not make for a radically faster implementation. > I have worked with Kernel included NFS and user space NFS > implementations and the performance differences have not been all that > amazingly radical. Probably you are not talking about linux based NFS experience. The last time we used userspace nfs (very long ago) on linux it was slow _and_ amazingly buggy. Many thanks to Neil that he made kernel nfs what it is today. > If your so convinced that a kernel included file system is the answer > you are free to implement a solution. Well, people were _paid_ for years now and came up with this mess. And you want me to implement it for free in what time? If you give me the bucks that were paid during the last decade you can be sure the solution is a lot better, easier to configure, and thought-through. > I am sure the project maintainers would love to have someone come along > and improve the code. Yes, we perfectly agree on that. -- Regards, Stephan Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
On 6/18/20 12:41 PM, Stephan von Krawczynski wrote Top Poster ;-) And in fact, it's not true. The clear message to me once was: we are not able to make a kernel version. Which I understood as: we have not the knowledge to do that. Since that was quite some time before Red Hat stepped in there was still hope that some day someone capable may come ... Since 2009 when I entered the list there was not a single month where there were no complaints about gluster being slow. I wonder if you could accept after 11 years and the projects near death now that I was right from the very first day. This is an amazingly unreasonable comment. First off ALL distributed file systems are slower than non-distributed file systems. Second ALL network file systems are slower than local hardware. Kernel inclusion does not make for a radically faster implementation. I have worked with Kernel included NFS and user space NFS implementations and the performance differences have not been all that amazingly radical. If your so convinced that a kernel included file system is the answer you are free to implement a solution. I am sure the project maintainers would love to have someone come along and improve the code. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
My concern for Glusterd2 deprecation is, it tried to implement and fix several things that we need in Gluster, and the promised features were not considered in Gluster afterward "better logging, Journal Based Replication". We're running both Ceph and Gluster, while both solutions are great in terms of features and performance, Gluster is a little bit behind in terms of stability and monitoring, We have multiple production clusters running Gluster and Red Hat Gluster, and both are somehow challenging "stability wise" when trying to update or replace a node or brick, administration operations are risky. Troubleshooting tools is something also has not improved from the past releases. The strength of Gluster, in my opinion, is the simplicity of creating distributed volumes that can be consumed by different clients, and this is why we chose Gluster back in 2016 as our main VMs Storage backend for VMWare and oVirt. Red Hat contribution to the project is important of course and when we see that major product like OCS is replacing Gluster with Ceph raises some eyebrows. The announcement of Kadalu a few months ago "or at least when we noticed it" was uncomfortable, as part of the core team of Gluster is now focusing on a new project, although it is depending on Gluster, Kadalu is focusing on persistence storage for containers, while Gluster use cases are not limited to containers. We suffered "and still" from performance issues with Gluster on use cases related to small files, but Gluster as a storage backend for Virtual Machines is really performant. Sadly I haven't contributed to the project, but I still want this project to evolve more and be used widely as I still see it as a good Distributed Storage option, with the hope of more focus on stability, tooling for easier troubleshooting and practical monitoring. -- Respectfully Mahdi Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
DISCLAIMER: I *really* appreciate this project and I thank all peoples involved. Il 2020-06-19 21:33 Mahdi Adnan ha scritto: The strength of Gluster, in my opinion, is the simplicity of creating distributed volumes that can be consumed by different clients, and this is why we chose Gluster back in 2016 as our main VMs Storage backend for VMWare and oVirt. I absolutely agree on that: it is very simple to create replicated and distributed storage with Gluster. This is a key Gluster feature: last time I checked Ceph, it basically required 6+ machines and a small team managing them. I read it is much easier now, but I doubt it is as easy as Gluster. This is the main reason why I periodically reconsider Gluster as a backend for hyperconverged virtual machines (coupled with the metadata server-less approach). Sadly, I arrive each time at the same conclusion: do not run Gluster in critical production environment without RedHat support and stable release, as Red Hat Gluster Storage is. Maybe I am overcautious, but I find too easy to end with split brain scenario, sometime even by simply rebooting a node at the wrong moment. The needing to enable sharding for efficient virtual disk healing scares me: if gluster fails, I need to reconstruct the disk images from each chunk with a tedious, time-and-space consuming process. Moreover, from a cursory view of Gluster's and other mailing lists, Qemu/KVM gfapi backend seems somewhat less stable than a FUSE gluster mount, and FUSE is not so fast either. Is my impression wrong? I really have the fealing that Gluster is good at a diametrically opposed scenario: scale-out NAS, for which it was conceived many moons ago. And RedHat backing on Ceph right now for block storage let me think Gluster issues with virtual disks were noticed by many. We suffered "and still" from performance issues with Gluster on use cases related to small files, but Gluster as a storage backend for Virtual Machines is really performant. I have mixed feelings about Gluster performance. In the past[1] I measured ~500 max IOPs for synchronous 4K writes per brick, which is a lot for a single disk, less so for a RAID array or SSD. Aggregated performance scaled linearly with increasing brick count, so maybe the solution is to simply go full-Gluster and ditch RAID, but replacing a disk in Gluster is much less convenient that doing the same with a RAID array (both hardware and software RAID) or ZFS. Note that I am not implying that Ceph is faster; rather, than a small Gluster setup with few brick can be slower than expected. I would love to ear other opinions and on-the-field experiences. Thanks. [1] https://lists.gluster.org/pipermail/gluster-users/2020-January/037607.html -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.da...@assyoma.it - i...@assyoma.it GPG public key ID: FF5F32A8 Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
На 20 юни 2020 г. 19:08:49 GMT+03:00, Gionatan Danti написа: >DISCLAIMER: I *really* appreciate this project and I thank all peoples >involved. > >Il 2020-06-19 21:33 Mahdi Adnan ha scritto: >> The strength of Gluster, in my opinion, is the simplicity of creating >> distributed volumes that can be consumed by different clients, and >> this is why we chose Gluster back in 2016 as our main VMs Storage >> backend for VMWare and oVirt. > >I absolutely agree on that: it is very simple to create replicated and >distributed storage with Gluster. This is a key Gluster feature: last >time I checked Ceph, it basically required 6+ machines and a small team > >managing them. I read it is much easier now, but I doubt it is as easy >as Gluster. > >This is the main reason why I periodically reconsider Gluster as a >backend for hyperconverged virtual machines (coupled with the metadata >server-less approach). Sadly, I arrive each time at the same >conclusion: >do not run Gluster in critical production environment without RedHat >support and stable release, as Red Hat Gluster Storage is. > >Maybe I am overcautious, but I find too easy to end with split brain >scenario, sometime even by simply rebooting a node at the wrong moment. > >The needing to enable sharding for efficient virtual disk healing >scares >me: if gluster fails, I need to reconstruct the disk images from each >chunk with a tedious, time-and-space consuming process. The efforts are far less than reconstructing the disk of a VM from CEPH. In gluster , just run a find on the brick searching for the name of the VM disk and you will find the VM_IMAGE.xyz (where xyz is just a number) and then concatenate the list into a single file. >Moreover, from a cursory view of Gluster's and other mailing lists, >Qemu/KVM gfapi backend seems somewhat less stable than a FUSE gluster >mount, and FUSE is not so fast either. Is my impression wrong? That's true, but you could also use NFS Ganesha, which is more performant than FUSE and also as reliable as it. >I really have the fealing that Gluster is good at a diametrically >opposed scenario: scale-out NAS, for which it was conceived many moons >ago. And RedHat backing on Ceph right now for block storage let me >think >Gluster issues with virtual disks were noticed by many. > >> We suffered "and still" from performance issues with Gluster on use >> cases related to small files, but Gluster as a storage backend for >> Virtual Machines is really performant. > >I have mixed feelings about Gluster performance. In the past[1] I >measured ~500 max IOPs for synchronous 4K writes per brick, which is a >lot for a single disk, less so for a RAID array or SSD. Aggregated >performance scaled linearly with increasing brick count, so maybe the >solution is to simply go full-Gluster and ditch RAID, but replacing a >disk in Gluster is much less convenient that doing the same with a RAID > >array (both hardware and software RAID) or ZFS. It's not so hard to do it - just use either 'reset-brick' or 'replace-brick' . >Note that I am not implying that Ceph is faster; rather, than a small >Gluster setup with few brick can be slower than expected. > >I would love to ear other opinions and on-the-field experiences. >Thanks. > >[1] >https://lists.gluster.org/pipermail/gluster-users/2020-January/037607.html Best Regards, Strahil Nikolov Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Il 2020-06-21 01:26 Strahil Nikolov ha scritto: The efforts are far less than reconstructing the disk of a VM from CEPH. In gluster , just run a find on the brick searching for the name of the VM disk and you will find the VM_IMAGE.xyz (where xyz is just a number) and then concatenate the list into a single file. Sure, but it is somewhat impractical with a 6 TB fileserver image and 500 users screaming for their files ;) And I fully expect to be the reconstruction much easier than Ceph but, from what I read, Ceph is less likely to broke in the first place. But I admit I never seriously run a Ceph cluster, so maybe it is more fragile than I expect. That's true, but you could also use NFS Ganesha, which is more performant than FUSE and also as reliable as it. From this very list I read about many users with various problems when using NFS Ganesha. Is that a wrong impression? It's not so hard to do it - just use either 'reset-brick' or 'replace-brick' . Sure - the command itself is simple enough. The point it that each reconstruction is quite more "riskier" than a simple RAID reconstruction. Do you run a full Gluster SDS, skipping RAID? How do you found this setup? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.da...@assyoma.it - i...@assyoma.it GPG public key ID: FF5F32A8 Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
I agree with this assessment for the most part. I'll just add that, during development of Gluster based solutions, we had internal use of Redhat Gluster. This was over a year and a half ago when we started. For my perhaps non-mainstream use cases, I found the latest versions of gluster 7 actually fixed several of my issues. Now, I did not try to work with RedHat when I hit problems as it was only "non-shipable support" - we could install it but not deliver it. Since it didn't work well for our strange use cases, we moved on to building our own Gluster instead instead of working to have customers buy the Red Hat one. (We also support sles12, sles15, rhel7, rhel8 - so having Red Hat's version of Gluster sort of wouldn't have worked out for us anyway). However, I also found that it is quite easy for my use case to hit new bugs. When we go from gluster72 to one of the newer ones, little things might happen (and did happen). I don't complain because I get free support from you and I do my best to fix them if I have time and access to a failing system. A tricky thing in my world is we will sell a cluster with 5,000 nodes to boot and my test cluster may have 3 nodes. I can get time up to 128 nodes on one test system. But I only get short-term access to bigger systems at the factory. So being able to change from one Gluster version to another is a real challenge for us because there simply is no way for us to test very often and, like is normal in HPC, problems only show at scale. hahaa :) :) This is also why we are still using Gluster NFS. We know we need to work with the community on fixing some Ganesha issues, but the amount of time we get on a large machine that exhibits the problem is short and we must prioritize. This is why I'm careful to never "blame Ganesha" but rather point out that we haven't had time to track the issues down with the Ganesha community. Meanwhile we hope we can keep building Gluster NFS :) When I next do a version-change of Gluster or try Ganesha again, it will be when I have sustained access to at least a 1024 node cluster to boot with 3 or 6 Gluster servers to really work out any issues. I consider this "a cost of doing business in the world I work in" but it is a real challenge indeed. I assume some challenges parallel Gluster developers "Works fine on my limited hardware or virtual machines". Erik > With every community project , you are in the position of a Betta Tester > - no matter Fedora, Gluster or CEPH. So far , I had issues with upstream > projects only diring and immediately after patching - but this is properly > mitigated with a reasonable patching strategy (patch test environment and > several months later patch prod with the same repos). > Enterprise Linux breaks (and alot) having 10-times more users and use > cases, so you cannot expect to start to use Gluster and assume that a > free peoject won't break at all. > Our part in this project is to help the devs to create a test case for our > workload , so regressions will be reduced to minimum. > > In the past 2 years, we got 2 major issues with VMware VSAN and 1 major > issue with a Enterprise Storage cluster (both solutions are quite > expensive) - so I always recommend proper testing of your software . > > > >> That's true, but you could also use NFS Ganesha, which is > >> more performant than FUSE and also as reliable as it. > > > >From this very list I read about many users with various problems when > >using NFS Ganesha. Is that a wrong impression? > > >From my observations, almost nobody is complaining about Ganesha in the > >mailing list -> 50% are having issues with geo replication,20% are > >having issues with small file performance and the rest have issues with very > >old version of gluster -> v5 or older. > > >> It's not so hard to do it - just use either 'reset-brick' or > >> 'replace-brick' . > > > >Sure - the command itself is simple enough. The point it that each > >reconstruction is quite more "riskier" than a simple RAID > >reconstruction. Do you run a full Gluster SDS, skipping RAID? How do > >you > >found this setup? > > I can't say that a replace-brick on a 'replica 3' volume is more riskier > than a rebuild of a raid, but I have noticed that nobody is following Red > Hat's guide to use either: > - a Raid6 of 12 Disks (2-3 TB big) > - a Raid10 of 12 Disks (2-3 TB big) > - JBOD disks in 'replica 3' mode (i'm not sure about the size RH > recommends, most probably 2-3 TB) > So far, I didn' have the opportunity to run on JBODs. > > > >Thanks. > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users Community Meeting Calendar: Schedule -
Re: [Gluster-users] State of Gluster project
На 21 юни 2020 г. 10:53:10 GMT+03:00, Gionatan Danti написа: >Il 2020-06-21 01:26 Strahil Nikolov ha scritto: >> The efforts are far less than reconstructing the disk of a VM from >> CEPH. In gluster , just run a find on the brick searching for the >> name of the VM disk and you will find the VM_IMAGE.xyz (where xyz is >> just a number) and then concatenate the list into a single file. > >Sure, but it is somewhat impractical with a 6 TB fileserver image and >500 users screaming for their files ;) >And I fully expect to be the reconstruction much easier than Ceph but, >from what I read, Ceph is less likely to broke in the first place. But >I >admit I never seriously run a Ceph cluster, so maybe it is more fragile > >than I expect. With every community project , you are in the position of a Betta Tester - no matter Fedora, Gluster or CEPH. So far , I had issues with upstream projects only diring and immediately after patching - but this is properly mitigated with a reasonable patching strategy (patch test environment and several months later patch prod with the same repos). Enterprise Linux breaks (and alot) having 10-times more users and use cases, so you cannot expect to start to use Gluster and assume that a free peoject won't break at all. Our part in this project is to help the devs to create a test case for our workload , so regressions will be reduced to minimum. In the past 2 years, we got 2 major issues with VMware VSAN and 1 major issue with a Enterprise Storage cluster (both solutions are quite expensive) - so I always recommend proper testing of your software . >> That's true, but you could also use NFS Ganesha, which is >> more performant than FUSE and also as reliable as it. > >From this very list I read about many users with various problems when >using NFS Ganesha. Is that a wrong impression? From my observations, almost nobody is complaining about Ganesha in the mailing list -> 50% are having issues with geo replication,20% are having issues with small file performance and the rest have issues with very old version of gluster -> v5 or older. >> It's not so hard to do it - just use either 'reset-brick' or >> 'replace-brick' . > >Sure - the command itself is simple enough. The point it that each >reconstruction is quite more "riskier" than a simple RAID >reconstruction. Do you run a full Gluster SDS, skipping RAID? How do >you >found this setup? I can't say that a replace-brick on a 'replica 3' volume is more riskier than a rebuild of a raid, but I have noticed that nobody is following Red Hat's guide to use either: - a Raid6 of 12 Disks (2-3 TB big) - a Raid10 of 12 Disks (2-3 TB big) - JBOD disks in 'replica 3' mode (i'm not sure about the size RH recommends, most probably 2-3 TB) So far, I didn' have the opportunity to run on JBODs. >Thanks. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Il 2020-06-21 14:20 Strahil Nikolov ha scritto: With every community project , you are in the position of a Betta Tester - no matter Fedora, Gluster or CEPH. So far , I had issues with upstream projects only diring and immediately after patching - but this is properly mitigated with a reasonable patching strategy (patch test environment and several months later patch prod with the same repos). Enterprise Linux breaks (and alot) having 10-times more users and use cases, so you cannot expect to start to use Gluster and assume that a free peoject won't break at all. Our part in this project is to help the devs to create a test case for our workload , so regressions will be reduced to minimum. Well, this is true, and both devs & community deserve a big thanks for all the work done. In the past 2 years, we got 2 major issues with VMware VSAN and 1 major issue with a Enterprise Storage cluster (both solutions are quite expensive) - so I always recommend proper testing of your software . Interesting, I am almost tempted to ask you what issue you had with vSAN, but this is not the right mailing list ;) From my observations, almost nobody is complaining about Ganesha in the mailing list -> 50% are having issues with geo replication,20% are having issues with small file performance and the rest have issues with very old version of gluster -> v5 or older. Mmm, I would swear to have read quite a few posts where the problem was solved by migrating away from NFS Ganesha. Still, for hyperconverged setup a problem remains: NFS on loopback/localhost is not 100% supported (or, at least, RH is not willing to declare it supportable/production ready [1]). A fuse mount would be the more natural way to access the underlying data. I can't say that a replace-brick on a 'replica 3' volume is more riskier than a rebuild of a raid, but I have noticed that nobody is following Red Hat's guide to use either: - a Raid6 of 12 Disks (2-3 TB big) - a Raid10 of 12 Disks (2-3 TB big) - JBOD disks in 'replica 3' mode (i'm not sure about the size RH recommends, most probably 2-3 TB) So far, I didn' have the opportunity to run on JBODs. For the RAID6/10 setup, I found no issues: simply replace the broken disk without involing Gluster at all. However, this also means facing the "iops wall" I described earlier for single-brick node. Going full-Guster with JBODs would be interesting from a performance standpoint, but this complicate eventual recovery from bad disks. Does someone use Gluster in JBOD mode? If so, can you share your experience? Thanks. [1] https://access.redhat.com/solutions/22231 (accound required) [2] https://bugzilla.redhat.com/show_bug.cgi?id=489889 (old, but I can not find anything newer) -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.da...@assyoma.it - i...@assyoma.it GPG public key ID: FF5F32A8 Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hello Gionatan, Using Gluster brick in a RAID configuration might be safer and require less work from Gluster admins but, it is a waste of disk space. Gluster bricks are replicated "assuming you're creating a distributed-replica volume" so when brick went down, it should be easy to recover it and should not affect the client's IO. We are using JBOD in all of our Gluster setups, overall, performance is good, and replacing a brick would work "most" of the time without issues. On Sun, Jun 21, 2020 at 8:43 PM Gionatan Danti wrote: > Il 2020-06-21 14:20 Strahil Nikolov ha scritto: > > With every community project , you are in the position of a Betta > > Tester - no matter Fedora, Gluster or CEPH. So far , I had > > issues with upstream projects only diring and immediately after > > patching - but this is properly mitigated with a reasonable > > patching strategy (patch test environment and several months later > > patch prod with the same repos). > > Enterprise Linux breaks (and alot) having 10-times more users and > > use cases, so you cannot expect to start to use Gluster and assume > > that a free peoject won't break at all. > > Our part in this project is to help the devs to create a test case for > > our workload , so regressions will be reduced to minimum. > > Well, this is true, and both devs & community deserve a big thanks for > all the work done. > > > In the past 2 years, we got 2 major issues with VMware VSAN and 1 > > major issue with a Enterprise Storage cluster (both solutions are > > quite expensive) - so I always recommend proper testing of your > > software . > > Interesting, I am almost tempted to ask you what issue you had with > vSAN, but this is not the right mailing list ;) > > > From my observations, almost nobody is complaining about Ganesha in > > the mailing list -> 50% are having issues with geo replication,20% > > are having issues with small file performance and the rest have > > issues with very old version of gluster -> v5 or older. > > Mmm, I would swear to have read quite a few posts where the problem was > solved by migrating away from NFS Ganesha. Still, for hyperconverged > setup a problem remains: NFS on loopback/localhost is not 100% supported > (or, at least, RH is not willing to declare it supportable/production > ready [1]). A fuse mount would be the more natural way to access the > underlying data. > > > I can't say that a replace-brick on a 'replica 3' volume is more > > riskier than a rebuild of a raid, but I have noticed that nobody is > > following Red Hat's guide to use either: > > - a Raid6 of 12 Disks (2-3 TB big) > > - a Raid10 of 12 Disks (2-3 TB big) > > - JBOD disks in 'replica 3' mode (i'm not sure about the size RH > > recommends, most probably 2-3 TB) > > So far, I didn' have the opportunity to run on JBODs. > > For the RAID6/10 setup, I found no issues: simply replace the broken > disk without involing Gluster at all. However, this also means facing > the "iops wall" I described earlier for single-brick node. Going > full-Guster with JBODs would be interesting from a performance > standpoint, but this complicate eventual recovery from bad disks. > > Does someone use Gluster in JBOD mode? If so, can you share your > experience? > Thanks. > > [1] https://access.redhat.com/solutions/22231 (accound required) > [2] https://bugzilla.redhat.com/show_bug.cgi?id=489889 (old, but I can > not find anything newer) > > -- > Danti Gionatan > Supporto Tecnico > Assyoma S.r.l. - www.assyoma.it [1] > email: g.da...@assyoma.it - i...@assyoma.it > GPG public key ID: FF5F32A8 > -- Respectfully Mahdi Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Il 2020-06-21 20:41 Mahdi Adnan ha scritto: Hello Gionatan, Using Gluster brick in a RAID configuration might be safer and require less work from Gluster admins but, it is a waste of disk space. Gluster bricks are replicated "assuming you're creating a distributed-replica volume" so when brick went down, it should be easy to recover it and should not affect the client's IO. We are using JBOD in all of our Gluster setups, overall, performance is good, and replacing a brick would work "most" of the time without issues. Hi Mahdi, thank you for reporting. I am interested in the "most of the time without isses" statement. Can you elaborate on what happened the few times when it did not work correctly? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.da...@assyoma.it - i...@assyoma.it GPG public key ID: FF5F32A8 Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Il 2020-06-22 06:58 Hu Bert ha scritto: Am So., 21. Juni 2020 um 19:43 Uhr schrieb Gionatan Danti : For the RAID6/10 setup, I found no issues: simply replace the broken disk without involing Gluster at all. However, this also means facing the "iops wall" I described earlier for single-brick node. Going full-Guster with JBODs would be interesting from a performance standpoint, but this complicate eventual recovery from bad disks. Does someone use Gluster in JBOD mode? If so, can you share your experience? Thanks. Hi, we once used gluster with disks in JBOD mode (3 servers, 4x10TB hdd each, 4 x 3 = 12), and to make it short: in our special case it wasn't that funny. Big HDDs, lots of small files, (highly) concurrent access through our application. It was running quite fine, until a disk failed. The reset-disk took ~30 (!) days, as you have gluster copying/restoring the data and the normal application read/write. After the first reset had finished, a couple of days later another disk died, and the fun started again :-) Maybe a bad use case. Hi Hubert, this is the exact scenario which scares me if/when using JBOD. Maybe for virtual machine disks (ie: big files) it would be faster, but still... Latest setup (for the high I/O part) looks like this: 3 servers, 10 disks with 10TB each -> 5 raid1, forming a distribute replicate with 5 bricks, 5 x 3 = 15. No disk has failed so far (fingers crossed), but if now a disk fails, gluster is still running with all bricks available, and after changing the failed, there's one raid resync running, affecting only 1/5 of the volume. In theory that should be better ;-) The regularly running raid checks are no problem so far, for 15 raid1 only 1 is running, none parallel. Ok, so you multiplied the number of bricks by using multiple RAID1 array. Good idea, it should be fine in this manner. disclaimer: JBOD may work better with SSDs/NVMes - untested ;-) Yeah, I think so! Regads. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.da...@assyoma.it - i...@assyoma.it GPG public key ID: FF5F32A8 Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Am So., 21. Juni 2020 um 19:43 Uhr schrieb Gionatan Danti : > For the RAID6/10 setup, I found no issues: simply replace the broken > disk without involing Gluster at all. However, this also means facing > the "iops wall" I described earlier for single-brick node. Going > full-Guster with JBODs would be interesting from a performance > standpoint, but this complicate eventual recovery from bad disks. > > Does someone use Gluster in JBOD mode? If so, can you share your > experience? > Thanks. Hi, we once used gluster with disks in JBOD mode (3 servers, 4x10TB hdd each, 4 x 3 = 12), and to make it short: in our special case it wasn't that funny. Big HDDs, lots of small files, (highly) concurrent access through our application. It was running quite fine, until a disk failed. The reset-disk took ~30 (!) days, as you have gluster copying/restoring the data and the normal application read/write. After the first reset had finished, a couple of days later another disk died, and the fun started again :-) Maybe a bad use case. With this experience, the next setup was: splitting data into 2 chunks (high I/O, low I/O), 3 servers with 2 raid10 (same type of disk), each raid used as a brick, resulting in replicate 3: 1 x 3 = 3. Changing a failed disk now results in a complete raid resync, but regarding I/O this is far better than using reset-disk with a HDD only. Only the regularly running raid check was a bit of a performance issue. Latest setup (for the high I/O part) looks like this: 3 servers, 10 disks with 10TB each -> 5 raid1, forming a distribute replicate with 5 bricks, 5 x 3 = 15. No disk has failed so far (fingers crossed), but if now a disk fails, gluster is still running with all bricks available, and after changing the failed, there's one raid resync running, affecting only 1/5 of the volume. In theory that should be better ;-) The regularly running raid checks are no problem so far, for 15 raid1 only 1 is running, none parallel. disclaimer: JBOD may work better with SSDs/NVMes - untested ;-) Best regards, Hubert Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hi Hubert, keep in mind RH recommends disks of size 2-3 TB, not 10. I guess that has changed the situation. For NVMe/SSD - raid controller is pointless , so JBOD makes most sense. Best Regards, Strahil Nikolov На 22 юни 2020 г. 7:58:56 GMT+03:00, Hu Bert написа: >Am So., 21. Juni 2020 um 19:43 Uhr schrieb Gionatan Danti >: > >> For the RAID6/10 setup, I found no issues: simply replace the broken >> disk without involing Gluster at all. However, this also means facing >> the "iops wall" I described earlier for single-brick node. Going >> full-Guster with JBODs would be interesting from a performance >> standpoint, but this complicate eventual recovery from bad disks. >> >> Does someone use Gluster in JBOD mode? If so, can you share your >> experience? >> Thanks. > >Hi, >we once used gluster with disks in JBOD mode (3 servers, 4x10TB hdd >each, 4 x 3 = 12), and to make it short: in our special case it wasn't >that funny. Big HDDs, lots of small files, (highly) concurrent access >through our application. It was running quite fine, until a disk >failed. The reset-disk took ~30 (!) days, as you have gluster >copying/restoring the data and the normal application read/write. >After the first reset had finished, a couple of days later another >disk died, and the fun started again :-) Maybe a bad use case. > >With this experience, the next setup was: splitting data into 2 chunks >(high I/O, low I/O), 3 servers with 2 raid10 (same type of disk), each >raid used as a brick, resulting in replicate 3: 1 x 3 = 3. Changing a >failed disk now results in a complete raid resync, but regarding I/O >this is far better than using reset-disk with a HDD only. Only the >regularly running raid check was a bit of a performance issue. > >Latest setup (for the high I/O part) looks like this: 3 servers, 10 >disks with 10TB each -> 5 raid1, forming a distribute replicate with 5 >bricks, 5 x 3 = 15. No disk has failed so far (fingers crossed), but >if now a disk fails, gluster is still running with all bricks >available, and after changing the failed, there's one raid resync >running, affecting only 1/5 of the volume. In theory that should be >better ;-) The regularly running raid checks are no problem so far, >for 15 raid1 only 1 is running, none parallel. > >disclaimer: JBOD may work better with SSDs/NVMes - untested ;-) > > >Best regards, >Hubert Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
We had a distributed replicated volume of 3 x 7 HDD, the volume was used for small files workload with heavy IO, we decided to replace the bricks with SSDs because of IO saturation to the disks, so we started by swapping the bricks one by one, and the fun started, some files lost its attributes and we had to manually fix the missing attributes by removing the file and its gfid and copy the file again to the volume. This issue affected 5 of the 21 bricks. On another volume, we had a disk failure and during the replace brick process, the mount point of one of the clients crashed. On Mon, Jun 22, 2020 at 10:55 AM Gionatan Danti wrote: > Il 2020-06-21 20:41 Mahdi Adnan ha scritto: > > Hello Gionatan, > > > > Using Gluster brick in a RAID configuration might be safer and > > require less work from Gluster admins but, it is a waste of disk > > space. > > Gluster bricks are replicated "assuming you're creating a > > distributed-replica volume" so when brick went down, it should be easy > > to recover it and should not affect the client's IO. > > We are using JBOD in all of our Gluster setups, overall, performance > > is good, and replacing a brick would work "most" of the time without > > issues. > > Hi Mahdi, > thank you for reporting. I am interested in the "most of the time > without isses" statement. Can you elaborate on what happened the few > times when it did not work correctly? > > Thanks. > > -- > Danti Gionatan > Supporto Tecnico > Assyoma S.r.l. - www.assyoma.it [1] > email: g.da...@assyoma.it - i...@assyoma.it > GPG public key ID: FF5F32A8 > -- Respectfully Mahdi Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
> For NVMe/SSD - raid controller is pointless , so JBOD makes most sense. I am game for an education lesson here. We're still using spinng drives with big RAID caches but we keep discussing SSD in the context of RAID. I have read for many real-world workloads, RAID0 makes no sense with modern SSDs. I get that part. But if your concern is reliability and reducing the need to mess with Gluster to recover from a drive failure, a RAID1 or or RADI10 (or some other with redundancy) would seem to at least make sense from that perspective. Was your answer a performance answer? Or am I missing something about RAIDs for redundancy and SSDs being a bad choice? Thanks again as always, Erik Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hey Erik, I actually ment that there is no point in using controllers with fast storage like SAS SSDs or NVMEs. They (the controllers) usually have 1-2 GB of RAM to buffer writes until the risc processor analyzes the requests and stacks them - thus JBOD (in 'replica 3' )makes much more sense for any kind of software defined storage (no matter Gluster, CEPH or Lustre). Of course, I could be wrong and I would be glad to read benchmark results on this topic. Best Regards, Strahil Nikolov На 22 юни 2020 г. 18:48:43 GMT+03:00, Erik Jacobson написа: >> For NVMe/SSD - raid controller is pointless , so JBOD makes most >sense. > >I am game for an education lesson here. We're still using spinng drives >with big RAID caches but we keep discussing SSD in the context of RAID. >I >have read for many real-world workloads, RAID0 makes no sense with >modern SSDs. I get that part. But if your concern is reliability and >reducing the need to mess with Gluster to recover from a drive failure, >a RAID1 or or RADI10 (or some other with redundancy) would seem to at >least make sense from that perspective. > >Was your answer a performance answer? Or am I missing something about >RAIDs for redundancy and SSDs being a bad choice? > >Thanks again as always, > >Erik Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
On Wed, 17 Jun 2020 00:06:33 +0300 Mahdi Adnan wrote: > [gluster going down ] I am following this project for quite some years now, probably longer than most of the people nowadays on the list. The project started with the brilliant idea of making a fs on top of classical fs's distributed over several hardware pieces without need to re-copy data for entering or leaving the gluster. (I thought) it started as a proof-of-concept fs on fuse with the intention to turn into kernel-space as soon as possible to get the performance that it should have for a fs. After five years of waiting (and using) I declared the project dead for our use (about five years ago) because it evolved more and more to bloatware. And I do think that Red Hat understood that finally (too), and what you mentioned is just the outcome of that. I really hate the way this project took, because for me it was visible from the very start that it is a dead-end. After all those years it is bloatware on fuse. And I feel very sorry for that brilliant idea that it once was. _FS IN USERSPACE IS SH*T_ - understand that. -- Regards, Stephan Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Hello Gionatan, Using Gluster brick in a RAID configuration might be safer and require less work from Gluster admins but, it is a waste of disk space. Gluster bricks are replicated "assuming you're creating a distributed-replica volume" so when brick went down, it should be easy to recover it and should not affect the client's IO. We are using JBOD in all of our Gluster setups, overall, performance is good, and replacing a brick would work "most" of the time without issues. On Sun, Jun 21, 2020 at 8:43 PM Gionatan Danti wrote: > Il 2020-06-21 14:20 Strahil Nikolov ha scritto: > > With every community project , you are in the position of a Betta > > Tester - no matter Fedora, Gluster or CEPH. So far , I had > > issues with upstream projects only diring and immediately after > > patching - but this is properly mitigated with a reasonable > > patching strategy (patch test environment and several months later > > patch prod with the same repos). > > Enterprise Linux breaks (and alot) having 10-times more users and > > use cases, so you cannot expect to start to use Gluster and assume > > that a free peoject won't break at all. > > Our part in this project is to help the devs to create a test case for > > our workload , so regressions will be reduced to minimum. > > Well, this is true, and both devs & community deserve a big thanks for > all the work done. > > > In the past 2 years, we got 2 major issues with VMware VSAN and 1 > > major issue with a Enterprise Storage cluster (both solutions are > > quite expensive) - so I always recommend proper testing of your > > software . > > Interesting, I am almost tempted to ask you what issue you had with > vSAN, but this is not the right mailing list ;) > > > From my observations, almost nobody is complaining about Ganesha in > > the mailing list -> 50% are having issues with geo replication,20% > > are having issues with small file performance and the rest have > > issues with very old version of gluster -> v5 or older. > > Mmm, I would swear to have read quite a few posts where the problem was > solved by migrating away from NFS Ganesha. Still, for hyperconverged > setup a problem remains: NFS on loopback/localhost is not 100% supported > (or, at least, RH is not willing to declare it supportable/production > ready [1]). A fuse mount would be the more natural way to access the > underlying data. > > > I can't say that a replace-brick on a 'replica 3' volume is more > > riskier than a rebuild of a raid, but I have noticed that nobody is > > following Red Hat's guide to use either: > > - a Raid6 of 12 Disks (2-3 TB big) > > - a Raid10 of 12 Disks (2-3 TB big) > > - JBOD disks in 'replica 3' mode (i'm not sure about the size RH > > recommends, most probably 2-3 TB) > > So far, I didn' have the opportunity to run on JBODs. > > For the RAID6/10 setup, I found no issues: simply replace the broken > disk without involing Gluster at all. However, this also means facing > the "iops wall" I described earlier for single-brick node. Going > full-Guster with JBODs would be interesting from a performance > standpoint, but this complicate eventual recovery from bad disks. > > Does someone use Gluster in JBOD mode? If so, can you share your > experience? > Thanks. > > [1] https://access.redhat.com/solutions/22231 (accound required) > [2] https://bugzilla.redhat.com/show_bug.cgi?id=489889 (old, but I can > not find anything newer) > > -- > Danti Gionatan > Supporto Tecnico > Assyoma S.r.l. - www.assyoma.it [1] > email: g.da...@assyoma.it - i...@assyoma.it > GPG public key ID: FF5F32A8 > -- [image: photograph] Mahdi Adnan IT Manager Information Technology EarthLink VoIP: 69 Cell: 07903316180 Website: www.earthlink.iq -- This message has been scanned for viruses and dangerous content and is believed to be clean. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] State of Gluster project
Been using Gluster since the 3.3.x days, been burned a few times and if it was not for the help of the community (one specific time saved big by Joe Julian), I would had not continued using it. My main use was originally as a self hosted engine via NFS and a file server for windows clients with Samba. After a couple of years, decided to get rid of virtualization so now exclusively used as a file server. I never dared turn on sharding because of the horror stories I saw in the old days for people who turned it on, then back off and then lost VMs to corruption. The 3.3.x to 3.7.x days were good. Burned somewhere in the post 3.7.x version. Mostly problems with samba VFS gluster and also gluster mounts for the file server. Since I stopped using VFS gluster in samba due to problems with specific files breaking and no longer working form the Windows side, I had to permanently go to using gluster mounts. So since the > 3.7.x and throughout all the 4.x versions I lived with memory leaks related to the gluster mounts, where I had to reboot servers once a month or even once every 15 days. Memory would start at around 11GB from the get go, and slowly go up till it would crash the server after eating up all 64GB of RAM in 15 to 30 days. After finally upgrading to 6.6.x I no longer have the memory leak. I have seen many improvements in speed but really did suffer with very slow performance for a long time. Nowadays, with 6.6.x and the sam gluster mounts, memory usage stays at around 4-5GB total even after 80 days of uptime! I can definitively echo the ease of use and setup that others have mentioned, but I really cannot vouch for the stability until now, with 6.6.x. I have not dared going back to using VFS gluster diue to the bad experience I had, so I guess I could try again and probably even get some performance gains, but why try to fix what ain't broken at the moment and make my life harder? I have tried other solutions and they are fast (significantly faster than gluster for as a file server), reliable and stable. Unfortunately, they are not free (well, the version with automatic failover isn't free), otherwise, I would gladly give up glusterfs for MooseFS in my file server use case. Diego On Tue, Jun 23, 2020 at 4:13 AM Mahdi Adnan wrote: > Hello Gionatan, > > Using Gluster brick in a RAID configuration might be safer and require > less work from Gluster admins but, it is a waste of disk space. > Gluster bricks are replicated "assuming you're creating a > distributed-replica volume" so when brick went down, it should be easy to > recover it and should not affect the client's IO. > We are using JBOD in all of our Gluster setups, overall, performance is > good, and replacing a brick would work "most" of the time without issues. > > On Sun, Jun 21, 2020 at 8:43 PM Gionatan Danti wrote: > >> Il 2020-06-21 14:20 Strahil Nikolov ha scritto: >> > With every community project , you are in the position of a Betta >> > Tester - no matter Fedora, Gluster or CEPH. So far , I had >> > issues with upstream projects only diring and immediately after >> > patching - but this is properly mitigated with a reasonable >> > patching strategy (patch test environment and several months later >> > patch prod with the same repos). >> > Enterprise Linux breaks (and alot) having 10-times more users and >> > use cases, so you cannot expect to start to use Gluster and assume >> > that a free peoject won't break at all. >> > Our part in this project is to help the devs to create a test case for >> > our workload , so regressions will be reduced to minimum. >> >> Well, this is true, and both devs & community deserve a big thanks for >> all the work done. >> >> > In the past 2 years, we got 2 major issues with VMware VSAN and 1 >> > major issue with a Enterprise Storage cluster (both solutions are >> > quite expensive) - so I always recommend proper testing of your >> > software . >> >> Interesting, I am almost tempted to ask you what issue you had with >> vSAN, but this is not the right mailing list ;) >> >> > From my observations, almost nobody is complaining about Ganesha in >> > the mailing list -> 50% are having issues with geo replication,20% >> > are having issues with small file performance and the rest have >> > issues with very old version of gluster -> v5 or older. >> >> Mmm, I would swear to have read quite a few posts where the problem was >> solved by migrating away from NFS Ganesha. Still, for hyperconverged >> setup a problem remains: NFS on loopback/localhost is not 100% supported >> (or, at least, RH is not willing to declare it supportable/production >> ready [1]). A fuse mount would be the more natural way to access the >> underlying data. >> >> > I can't say that a replace-brick on a 'replica 3' volume is more >> > riskier than a rebuild of a raid, but I have noticed that nobody is >> > following Red Hat's guide to use either: >> > - a Raid6 of 12 Disks (2
