Hi Rupert, If you try RMDA (or work on the older, now removed RMDA code in glusterfs that use(d) ibverbs, be aware that there was a glitch, at least as of glusterfs v3.12.7: https://lists.gluster.org/pipermail/gluster-users/2018-May/034135.html <https://lists.gluster.org/pipermail/gluster-users/2018-May/034135.html> A directory with tens of thousands of files did not show all directory entries.) I couldn't fix this back then and instead switched to ipoib. Although the measured per-file bandwith back then was better with RDMA than with ipoib by about 50%, I stayed with ipoib because in the aggreate bandwith (many clients reading many files simultaneously) it didn't make a difference. (Our bottlenecks are elsewhere, not in the network bandwidth between server and client.) I've used ipoib since then to run a glusterfs with more than 2 PB and haven't had serious problems so far.
best wishes, Stefan > Am 24.02.2021 um 10:59 schrieb Saju Mohammed Noohu <sajmo...@redhat.com>: > > Hi Rupert, > > Apologies for the inconvenience, even though this was discussed, somehow it > got missed in official release notes or documentation. > > Few old discussions are below: > https://github.com/gluster/glusterfs/issues/692 > <https://github.com/gluster/glusterfs/issues/692> > https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html > <https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html> > > Will surely document this as part of Gluster docs on an appropriate heading. > > As evident from the above discussions, we deprecated it since we had no means > to support it. If that changes, we'll be happy to work on re-enabling it, > maybe as a compile-time option. > > Thanks > Saju > > > > > On Wed, Feb 24, 2021 at 1:23 PM Schlick Rupert <rupert.schl...@ait.ac.at > <mailto:rupert.schl...@ait.ac.at>> wrote: > What surprised me is more that, if not looking into the developer channels, > there is no indication that the feature is gone till it crashes when you try > to use it. I did not think of Gluster as a developer-only tool, even in the > “community version”. > > > > Native RDMA support was the main reason for me to choose GlusterFS over > MooseFS – although it’s just a feature on “paper” and I do not really know > yet the real needs and bottlenecks of our new cluster. With both RDMA and > tiering gone, I based my decision on completely outdated information about > gluster features. > > > > @strahil: I hope to find some time to try your suggestion next week, thanks. > But even if it works, I will most probably not risk using something the > developers consider not fit for use anymore. There seem also to be issues > with mixed mode and multiple IP addresses anyway – our computation and > storage nodes are on a infiniband (RDMA and IP over Infiniband) island > network, but all are also connected to the outside via IP over Ethernet. The > login nodes, which would also need access to the cluster file system are in a > VM and only accessible via IP over Ethernet > > > > Rupert > > > > Von: Strahil Nikolov <hunter86...@yahoo.com <mailto:hunter86...@yahoo.com>> > Gesendet: Dienstag, 23. Februar 2021 11:20 > An: Schlick Rupert <rupert.schl...@ait.ac.at > <mailto:rupert.schl...@ait.ac.at>>; Saju Mohammed Noohu <sajmo...@redhat.com > <mailto:sajmo...@redhat.com>> > Cc: gluster-users@gluster.org <mailto:gluster-users@gluster.org> > Betreff: Re: [Gluster-users] RDMA support in gluster 9 > > > > As everything deprecated, it never get's completely removed from source. > > > > Can you try to 'make' from source and check if v9 by default has 'Infiniband > verbs' disabled ? > > If yes, then you can compile with enabled supportand try that. > > > > Best Regards, > > Strahil Nikolov > > On Tue, Feb 23, 2021 at 10:40, Schlick Rupert > > <rupert.schl...@ait.ac.at <mailto:rupert.schl...@ait.ac.at>> wrote: > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > <https://meet.google.com/cpu-eiue-hvk> > Gluster-users mailing list > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users > <https://lists.gluster.org/mailman/listinfo/gluster-users>________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
