Re: [Gluster-users] Self-heal never finishes

[Karthik Subrahmanya]

Hey,

I think [1] should help you.
If you can't find anything matching your situation or can't resolve it with
any of the methods listed there, please open an issue for this at [2], with
the following information.
- volume info, volume status, heal info and shd logs from node-1 & arbiter.
- Output of "getfattr -d -e hex -m. " of a few
entries from all the bricks which are listed in the heal info output.

[1]
https://uskarthik.blogspot.com/2020/02/entries-are-not-getting-healed-what-is_26.html
[2] https://github.com/gluster/glusterfs/issues

Regards,
Karthik

On Mon, Mar 1, 2021 at 8:12 AM Ben  wrote:

> I'm having a problem where once one of my volumes requires healing, it
> never finishes the process. I use a 3-node replica cluster (2 node +
> arbiter) as oVirt storage for virtual machines. I'm using Gluster version
> 8.3.
>
> When I patch my Gluster nodes, I try to keep the system online by
> rebooting them one at a time. However, I've found that once I reboot node
> 2, when it comes back up, self-heal will begin on both node 1 and the
> arbiter and never finish. I have let it run for weeks and still have
> entries in gluster volume heal  info. No heal entries are reported
> on the node that rebooted.
>
> I've set the volumes to the virt group (gluster volume set  group
> virt) per the RHEV documentation, and the gluster nodes don't seem to be
> overly busy. I'm hoping someone can point me in the right direction --
> since the volumes never heal, I'm basically running on one node. Let me
> know what additional info will be helpful for troubleshooting, and thank
> you in advance.
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [EXT] RDMA support in gluster 9

[Schlick Rupert]

@Stefan: Thank you. I‘ll stick to IPoIB for now, trusting that real bottlenecks 
will probably be elsewhere. At the moment on an almost empty node, gluster uses 
0.5-1 cores of 102.
I still hope that RDMA will be back sometime.

@Saju: Thanks for the clarifications. The main reason I “complained” publicly 
is that RDMA support 
(https://docs.gluster.org/en/latest/Administrator-Guide/RDMA-Transport/) in the 
official and current documentation has no indication in which versions it works 
or not. At least tiering is only mentioned as experimental in 3.7 and 
discontinued in release 6.

Rupert

Von: Stefan Solbrig 
Gesendet: Donnerstag, 25. Februar 2021 16:37
An: gluster-users@gluster.org
Cc: Schlick Rupert 
Betreff: Re: [EXT] [Gluster-users] RDMA support in gluster 9

Hi Rupert,

If you try RMDA (or work on the older, now removed RMDA code in glusterfs that 
use(d) ibverbs, be aware that there was a glitch, at least as of glusterfs 
v3.12.7:  
https://lists.gluster.org/pipermail/gluster-users/2018-May/034135.html   A 
directory with tens of thousands of files did not show all directory entries.)  
I couldn't fix this back then and instead switched to ipoib.  Although the 
measured per-file bandwith back then was better with RDMA than with ipoib by 
about 50%, I stayed with ipoib because in the aggreate bandwith (many clients 
reading many files simultaneously) it didn't make a difference.  (Our 
bottlenecks are elsewhere, not in the network bandwidth between server and 
client.) I've used ipoib since then to run a glusterfs with more than 2 PB and 
haven't had serious problems so far.

best wishes,
Stefan



Am 24.02.2021 um 10:59 schrieb Saju Mohammed Noohu 
mailto:sajmo...@redhat.com>>:

Hi Rupert,

Apologies for the inconvenience, even though this was discussed, somehow it got 
missed in official release notes or documentation.

Few old discussions are below:
https://github.com/gluster/glusterfs/issues/692
https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

Will surely document this as part of Gluster docs on an appropriate heading.

As evident from the above discussions, we deprecated it since we had no means 
to support it. If that changes, we'll be happy to work on re-enabling it, maybe 
as a compile-time option.

Thanks
Saju




On Wed, Feb 24, 2021 at 1:23 PM Schlick Rupert 
mailto:rupert.schl...@ait.ac.at>> wrote:
What surprised me is more that, if not looking into the developer channels, 
there is no indication that the feature is gone till it crashes when you try to 
use it. I did not think of Gluster as a developer-only tool, even in the 
“community version”.

Native RDMA support was the main reason for me to choose GlusterFS over MooseFS 
– although it’s just a feature on “paper” and I do not really know yet the real 
needs and bottlenecks of our new cluster. With both RDMA and tiering gone, I 
based my decision on completely outdated information about gluster features.

@strahil: I hope to find some time to try your suggestion next week, thanks. 
But even if it works, I will most probably not risk using something the 
developers consider not fit for use anymore. There seem also to be issues with 
mixed mode and multiple IP addresses anyway – our computation and storage nodes 
are on a infiniband (RDMA and IP over Infiniband) island network, but all are 
also connected to the outside via IP over Ethernet. The login nodes, which 
would also need access to the cluster file system are in a VM and only 
accessible via IP over Ethernet

Rupert

Von: Strahil Nikolov mailto:hunter86...@yahoo.com>>
Gesendet: Dienstag, 23. Februar 2021 11:20
An: Schlick Rupert mailto:rupert.schl...@ait.ac.at>>; 
Saju Mohammed Noohu mailto:sajmo...@redhat.com>>
Cc: gluster-users@gluster.org
Betreff: Re: [Gluster-users] RDMA support in gluster 9

As everything deprecated, it never get's completely removed from source.

Can you try to 'make' from source and check if v9 by default has 'Infiniband 
verbs'  disabled ?
If yes, then you can compile with enabled supportand try that.

Best Regards,
Strahil Nikolov
On Tue, Feb 23, 2021 at 10:40, Schlick Rupert
mailto:rupert.schl...@ait.ac.at>> wrote:




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org

[Gluster-users] [Gluster-devel] Announcing Gluster release 8.4

[Rinku Kothiya]

Hi,

The Gluster community is pleased to announce the release of Gluster8.4
(packages available at [1]).
Release notes for the release can be found at [2].

*Highlights of Release:*

- Healing data in 1MB chunks instead of 128KB for improving healing
performance

Users are highly encouraged to upgrade to newer releases of GlusterFS
available at [2].


Thanks,
Gluster community

References:

[1] Packages for 8.4:
https://download.gluster.org/pub/gluster/glusterfs/8/8.4/

[2] Release notes for 8.4:
https://docs.gluster.org/en/latest/release-notes/8.4/




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Self-heal never finishes

[Ben]

I'm having a problem where once one of my volumes requires healing, it
never finishes the process. I use a 3-node replica cluster (2 node +
arbiter) as oVirt storage for virtual machines. I'm using Gluster version
8.3.

When I patch my Gluster nodes, I try to keep the system online by rebooting
them one at a time. However, I've found that once I reboot node 2, when it
comes back up, self-heal will begin on both node 1 and the arbiter and
never finish. I have let it run for weeks and still have entries in gluster
volume heal  info. No heal entries are reported on the node that
rebooted.

I've set the volumes to the virt group (gluster volume set  group
virt) per the RHEV documentation, and the gluster nodes don't seem to be
overly busy. I'm hoping someone can point me in the right direction --
since the volumes never heal, I'm basically running on one node. Let me
know what additional info will be helpful for troubleshooting, and thank
you in advance.




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users