Re: [Gluster-users] Healing entries get healed but there are constantly new entries appearing

2020-02-10 Thread Karthik Subrahmanya 
Hi Ulrich,

Thank you for letting us know. Glad to hear that your system is back to
normal.

Regards,
Karthik

On Mon, Feb 10, 2020 at 9:51 PM Ulrich Pötter 
wrote:

> Hello Karthik,
>
> thank you very much. That was exactly the problem.
> Running the command (cat
> /.meta/graphs/active/-client-*/private | egrep -i
> 'connected') on the clients revealed that a few were not connected to all
> bricks.
> After restarting them, everything went back to normal.
>
> Regards,
> Ulrich
> Am 06.02.20 um 12:51 schrieb Karthik Subrahmanya:
>
> Hi Ulrich,
>
> From the problem statement, seems like the client(s) have lost connection
> with brick. Can you give the following information?
> - How many clients are there for this volume and which version they are in?
> - gluster volume info  & gluster volume status  outputs
> - Check whether all the clients are connected to all the bricks.
> If you are using the fuse clients give the output of the following from
> all the clients
> cat /.meta/graphs/active/-client-*/private | egrep
> -i 'connected'
> -If you are using non fuse clients generate the statedumps (
> https://docs.gluster.org/en/latest/Troubleshooting/statedump/) of each
> clients and give the output of
> grep -A 2 "xlator.protocol.client" /var/run/gluster/
> (If you have changed the statedump-path replace the path in the above
> command)
>
> Regards,
> Karthik
>
> On Thu, Feb 6, 2020 at 5:06 PM Ulrich Pötter 
> wrote:
>
>> Dear Gluster Users,
>>
>> we are running the following Gluster setup:
>> Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and one was
>> upgraded to Centos 7.7 with Gluster 6.7.
>>
>> Since the upgrade to gluster 6.7 on one of the servers, we encountered
>> the following issue:
>> New healing entries appear and get healed, but soon afterwards new
>> healing entries appear.
>> The abovementioned problem started after we upgraded the server.
>> The healing issues do not only appear on the upgraded server, but on all
>> three.
>>
>> This does not seem to be a split brain issue as the output of the
>> command "gluster volume head  info split-brain" is "number of
>> entries in split-brain: 0"
>>
>> Has anyone else observed such behavior with different Gluster versions
>> in one replica setup?
>>
>> We hesitate with updating the other nodes, as we do not know if this
>> standard Gluster behaviour or if there is more to this problem.
>>
>> Can you help us?
>>
>> Thanks in advance,
>> Ulrich
>>
>> 
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/441850968
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Healing entries get healed but there are constantly new entries appearing

2020-02-10 Thread Ulrich Pötter 

Hello Karthik,

thank you very much. That was exactly the problem.
Running the command (cat 
/.meta/graphs/active/-client-*/private | egrep -i 
'connected') on the clients revealed that a few were not connected to 
all bricks.

After restarting them, everything went back to normal.

Regards,
Ulrich

Am 06.02.20 um 12:51 schrieb Karthik Subrahmanya:

Hi Ulrich,

From the problem statement, seems like the client(s) have lost 
connection with brick. Can you give the following information?
- How many clients are there for this volume and which version they 
are in?
- gluster volume info  & gluster volume status  
outputs

- Check whether all the clients are connected to all the bricks.
If you are using the fuse clients give the output of the following 
from all the clients
cat /.meta/graphs/active/-client-*/private | 
egrep -i 'connected'
-If you are using non fuse clients generate the statedumps 
(https://docs.gluster.org/en/latest/Troubleshooting/statedump/) of 
each clients and give the output of

grep -A 2 "xlator.protocol.client" /var/run/gluster/
(If you have changed the statedump-path replace the path in the above 
command)


Regards,
Karthik

On Thu, Feb 6, 2020 at 5:06 PM Ulrich Pötter 
mailto:ulrich.poet...@menzel-it.net>> 
wrote:


Dear Gluster Users,

we are running the following Gluster setup:
Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and
one was
upgraded to Centos 7.7 with Gluster 6.7.

Since the upgrade to gluster 6.7 on one of the servers, we
encountered
the following issue:
New healing entries appear and get healed, but soon afterwards new
healing entries appear.
The abovementioned problem started after we upgraded the server.
The healing issues do not only appear on the upgraded server, but
on all
three.

This does not seem to be a split brain issue as the output of the
command "gluster volume head  info split-brain" is "number of
entries in split-brain: 0"

Has anyone else observed such behavior with different Gluster
versions
in one replica setup?

We hesitate with updating the other nodes, as we do not know if this
standard Gluster behaviour or if there is more to this problem.

Can you help us?

Thanks in advance,
Ulrich



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Healing entries get healed but there are constantly new entries appearing

2020-02-06 Thread Karthik Subrahmanya 
Hi Ulrich,

>From the problem statement, seems like the client(s) have lost connection
with brick. Can you give the following information?
- How many clients are there for this volume and which version they are in?
- gluster volume info  & gluster volume status  outputs
- Check whether all the clients are connected to all the bricks.
If you are using the fuse clients give the output of the following from all
the clients
cat /.meta/graphs/active/-client-*/private | egrep -i
'connected'
-If you are using non fuse clients generate the statedumps (
https://docs.gluster.org/en/latest/Troubleshooting/statedump/) of each
clients and give the output of
grep -A 2 "xlator.protocol.client" /var/run/gluster/
(If you have changed the statedump-path replace the path in the above
command)

Regards,
Karthik

On Thu, Feb 6, 2020 at 5:06 PM Ulrich Pötter 
wrote:

> Dear Gluster Users,
>
> we are running the following Gluster setup:
> Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and one was
> upgraded to Centos 7.7 with Gluster 6.7.
>
> Since the upgrade to gluster 6.7 on one of the servers, we encountered
> the following issue:
> New healing entries appear and get healed, but soon afterwards new
> healing entries appear.
> The abovementioned problem started after we upgraded the server.
> The healing issues do not only appear on the upgraded server, but on all
> three.
>
> This does not seem to be a split brain issue as the output of the
> command "gluster volume head  info split-brain" is "number of
> entries in split-brain: 0"
>
> Has anyone else observed such behavior with different Gluster versions
> in one replica setup?
>
> We hesitate with updating the other nodes, as we do not know if this
> standard Gluster behaviour or if there is more to this problem.
>
> Can you help us?
>
> Thanks in advance,
> Ulrich
>
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Healing entries get healed but there are constantly new entries appearing

2020-02-06 Thread Ulrich Pötter 

Dear Gluster Users,

we are running the following Gluster setup:
Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and one was 
upgraded to Centos 7.7 with Gluster 6.7.


Since the upgrade to gluster 6.7 on one of the servers, we encountered 
the following issue:
New healing entries appear and get healed, but soon afterwards new 
healing entries appear.

The abovementioned problem started after we upgraded the server.
The healing issues do not only appear on the upgraded server, but on all 
three.


This does not seem to be a split brain issue as the output of the 
command "gluster volume head  info split-brain" is "number of 
entries in split-brain: 0"


Has anyone else observed such behavior with different Gluster versions 
in one replica setup?


We hesitate with updating the other nodes, as we do not know if this 
standard Gluster behaviour or if there is more to this problem.


Can you help us?

Thanks in advance,
Ulrich



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users