Re: [Gluster-users] Brick count limit in a volume

[Serkan Çoban]

This is the command line output:
Total brick list is larger than a request. Can take (brick_count )
Usage: volume create  [stripe ] [replica ] 

I am testing if a big single volume will work for us. Now I am
continuing testing with three volumes each 13PB...
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Brick count limit in a volume

[Vijay Bellur]

Can you also please provide more detail on why those many bricks are needed
in a single volume?

Thanks,
Vijay

On Wed, Aug 23, 2017 at 12:43 AM, Atin Mukherjee 
wrote:

> An upstream bug would be ideal as github issue is mainly used for
> enhancements. In the mean time, could you point to the exact failure shown
> at the command line and the log entry from cli.log?
>
> On Wed, Aug 23, 2017 at 12:10 AM, Serkan Çoban 
> wrote:
>
>> Hi, I think this is the line limiting brick count:
>> https://github.com/gluster/glusterfs/blob/c136024613c697fec8
>> 7aaff3a070862b92c57977/cli/src/cli-cmd-parser.c#L84
>>
>> Can gluster-devs increase this limit? Should I open a github issue?
>>
>> On Mon, Aug 21, 2017 at 7:01 PM, Serkan Çoban 
>> wrote:
>> > Hi,
>> > Gluster version is 3.10.5. I am trying to create a 5500 brick volume,
>> > but getting an error stating that  bricks is the limit. Is this a
>> > known limit? Can I change this with an option?
>> >
>> > Thanks,
>> > Serkan
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Brick count limit in a volume

[Atin Mukherjee]

An upstream bug would be ideal as github issue is mainly used for
enhancements. In the mean time, could you point to the exact failure shown
at the command line and the log entry from cli.log?

On Wed, Aug 23, 2017 at 12:10 AM, Serkan Çoban 
wrote:

> Hi, I think this is the line limiting brick count:
> https://github.com/gluster/glusterfs/blob/c136024613c697fec87aaff3a07086
> 2b92c57977/cli/src/cli-cmd-parser.c#L84
>
> Can gluster-devs increase this limit? Should I open a github issue?
>
> On Mon, Aug 21, 2017 at 7:01 PM, Serkan Çoban 
> wrote:
> > Hi,
> > Gluster version is 3.10.5. I am trying to create a 5500 brick volume,
> > but getting an error stating that  bricks is the limit. Is this a
> > known limit? Can I change this with an option?
> >
> > Thanks,
> > Serkan
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Brick count limit in a volume

[Serkan Çoban]

Hi, I think this is the line limiting brick count:
https://github.com/gluster/glusterfs/blob/c136024613c697fec87aaff3a070862b92c57977/cli/src/cli-cmd-parser.c#L84

Can gluster-devs increase this limit? Should I open a github issue?

On Mon, Aug 21, 2017 at 7:01 PM, Serkan Çoban  wrote:
> Hi,
> Gluster version is 3.10.5. I am trying to create a 5500 brick volume,
> but getting an error stating that  bricks is the limit. Is this a
> known limit? Can I change this with an option?
>
> Thanks,
> Serkan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterd proccess hangs on reboot

[Serkan Çoban]

I reboot multiple times, also I destroyed the gluster configuration
and recreate multiple times. The behavior is same.

On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee  wrote:
> My guess is there is a corruption in vol list or peer list which has lead
> glusterd to get into a infinite loop of traversing a peer/volume list and
> CPU to hog up. Again this is a guess and I've not got a chance to take a
> detail look at the logs and the strace output.
>
> I believe if you get to reboot the node again the problem will disappear.
>
> On Tue, 22 Aug 2017 at 20:07, Serkan Çoban  wrote:
>>
>> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42 during
>> glusterd %100 cpu usage
>> Hope this helps...
>>
>> On Tue, Aug 22, 2017 at 2:41 PM, Serkan Çoban 
>> wrote:
>> > Hi there,
>> >
>> > I have a strange problem.
>> > Gluster version in 3.10.5, I am testing new servers. Gluster
>> > configuration is 16+4 EC, I have three volumes, each have 1600 bricks.
>> > I can successfully create the cluster and volumes without any
>> > problems. I write data to cluster from 100 clients for 12 hours again
>> > no problem. But when I try to reboot a node, glusterd process hangs on
>> > %100 CPU usage and seems to do nothing, no brick processes come
>> > online. You can find strace of glusterd process for 1 minutes here:
>> >
>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
>> >
>> > Here is the glusterd logs:
>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
>> >
>> >
>> > By the way, reboot of one server completes without problem if I reboot
>> > the servers before creating any volumes.
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
> --
> - Atin (atinm)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterd proccess hangs on reboot

[Atin Mukherjee]

My guess is there is a corruption in vol list or peer list which has lead
glusterd to get into a infinite loop of traversing a peer/volume list and
CPU to hog up. Again this is a guess and I've not got a chance to take a
detail look at the logs and the strace output.

I believe if you get to reboot the node again the problem will disappear.

On Tue, 22 Aug 2017 at 20:07, Serkan Çoban  wrote:

> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42 during
> glusterd %100 cpu usage
> Hope this helps...
>
> On Tue, Aug 22, 2017 at 2:41 PM, Serkan Çoban 
> wrote:
> > Hi there,
> >
> > I have a strange problem.
> > Gluster version in 3.10.5, I am testing new servers. Gluster
> > configuration is 16+4 EC, I have three volumes, each have 1600 bricks.
> > I can successfully create the cluster and volumes without any
> > problems. I write data to cluster from 100 clients for 12 hours again
> > no problem. But when I try to reboot a node, glusterd process hangs on
> > %100 CPU usage and seems to do nothing, no brick processes come
> > online. You can find strace of glusterd process for 1 minutes here:
> >
> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
> >
> > Here is the glusterd logs:
> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
> >
> >
> > By the way, reboot of one server completes without problem if I reboot
> > the servers before creating any volumes.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

-- 
- Atin (atinm)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterd proccess hangs on reboot

[Serkan Çoban]

As an addition perf top shows %80 libc-2.12.so __strcmp_sse42 during
glusterd %100 cpu usage
Hope this helps...

On Tue, Aug 22, 2017 at 2:41 PM, Serkan Çoban  wrote:
> Hi there,
>
> I have a strange problem.
> Gluster version in 3.10.5, I am testing new servers. Gluster
> configuration is 16+4 EC, I have three volumes, each have 1600 bricks.
> I can successfully create the cluster and volumes without any
> problems. I write data to cluster from 100 clients for 12 hours again
> no problem. But when I try to reboot a node, glusterd process hangs on
> %100 CPU usage and seems to do nothing, no brick processes come
> online. You can find strace of glusterd process for 1 minutes here:
>
> https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
>
> Here is the glusterd logs:
> https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
>
>
> By the way, reboot of one server completes without problem if I reboot
> the servers before creating any volumes.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] self-heal not working

[mabi]

Yes, I have indeed a small test cluster with 3 Raspberry Pis but unfortunately 
I have other issues with that one.

So I tried the "volume start force" which restarted glustershd on every nodes 
but nothing changed and running the heal does not do anything, there is still 
that one single file to be healed.

Finally I would like to use your last suggestion to increase client-log-level 
to DEBUG. In order to do that would the following command be correct?

gluster volume set myvolume diagnostics.client-log-level DEBUG

>  Original Message 
> Subject: Re: [Gluster-users] self-heal not working
> Local Time: August 22, 2017 11:51 AM
> UTC Time: August 22, 2017 9:51 AM
> From: ravishan...@redhat.com
> To: mabi 
> Ben Turner , Gluster Users 
>
> On 08/22/2017 02:30 PM, mabi wrote:
>
>> Thanks for the additional hints, I have the following 2 questions first:
>>
>> - In order to launch the index heal is the following command correct:
>> gluster volume heal myvolume
>
> Yes
>
>> - If I run a "volume start force" will it have any short disruptions on my 
>> clients which mount the volume through FUSE? If yes, how long? This is a 
>> production system that's why I am asking.
>
> No. You can actually create a test volume on  your personal linux box to try 
> these kinds of things without needing multiple machines. This is how we 
> develop and test our patches :)
> 'gluster volume create testvol replica 3 /home/mabi/bricks/brick{1..3} force` 
> and so on.
>
> HTH,
> Ravi
>
>>>  Original Message 
>>> Subject: Re: [Gluster-users] self-heal not working
>>> Local Time: August 22, 2017 6:26 AM
>>> UTC Time: August 22, 2017 4:26 AM
>>> From: ravishan...@redhat.com
>>> To: mabi [](mailto:m...@protonmail.ch), Ben Turner 
>>> [](mailto:btur...@redhat.com)
>>> Gluster Users 
>>> [](mailto:gluster-users@gluster.org)
>>>
>>> Explore the following:
>>>
>>> - Launch index heal and look at the glustershd logs of all bricks for 
>>> possible errors
>>>
>>> - See if the glustershd in each node is connected to all bricks.
>>>
>>> - If not try to restart shd by `volume start force`
>>>
>>> - Launch index heal again and try.
>>>
>>> - Try debugging the shd log by setting client-log-level to DEBUG 
>>> temporarily.
>>>
>>> On 08/22/2017 03:19 AM, mabi wrote:
>>>
 Sure, it doesn't look like a split brain based on the output:

 Brick node1.domain.tld:/data/myvolume/brick
 Status: Connected
 Number of entries in split-brain: 0

 Brick node2.domain.tld:/data/myvolume/brick
 Status: Connected
 Number of entries in split-brain: 0

 Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
 Status: Connected
 Number of entries in split-brain: 0

>  Original Message 
> Subject: Re: [Gluster-users] self-heal not working
> Local Time: August 21, 2017 11:35 PM
> UTC Time: August 21, 2017 9:35 PM
> From: btur...@redhat.com
> To: mabi [](mailto:m...@protonmail.ch)
> Gluster Users 
> [](mailto:gluster-users@gluster.org)
>
> Can you also provide:
>
> gluster v heal  info split-brain
>
> If it is split brain just delete the incorrect file from the brick and 
> run heal again. I haven"t tried this with arbiter but I assume the 
> process is the same.
>
> -b
>
> - Original Message -
>> From: "mabi" [](mailto:m...@protonmail.ch)
>> To: "Ben Turner" [](mailto:btur...@redhat.com)
>> Cc: "Gluster Users" 
>> [](mailto:gluster-users@gluster.org)
>> Sent: Monday, August 21, 2017 4:55:59 PM
>> Subject: Re: [Gluster-users] self-heal not working
>>
>> Hi Ben,
>>
>> So it is really a 0 kBytes file everywhere (all nodes including the 
>> arbiter
>> and from the client).
>> Here below you will find the output you requested. Hopefully that will 
>> help
>> to find out why this specific file is not healing... Let me know if you 
>> need
>> any more information. Btw node3 is my arbiter node.
>>
>> NODE1:
>>
>> STAT:
>> File:
>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
>> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
>> Device: 24h/36d Inode: 10033884 Links: 2
>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>> Access: 2017-08-14 17:04:55.530681000 +0200
>> Modify: 2017-08-14 17:11:46.407404779 +0200
>> Change: 2017-08-14 17:11:46.407404779 +0200
>> Birth: -
>>
>> GETFATTR:
>> trusted.afr.dirty=0sAQAA
>> trusted.bit-rot.version=0sAgBZhuknAAlJAg==
>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=
>>
>> NODE2:
>>
>> STAT:
>> File:
>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
>> Size: 0 Blocks: 38 IO Block: 131072 regular empty f

Re: [Gluster-users] Add brick to a disperse volume

[Nag Pavan Chilakam]

Hi Michel,
Do you want to increase the distribute count or do you want to increase the 
number of data bricks.
converting a 1x(3+1) to 1x(4+1), means increasing data brick count, which is 
not supported, yet. The distribute count here is 1 and it still remains same. 
Converting a 1x(3+1) to 2x(3+1) , means adding more at distribute layer. This 
is supported and can be done. However you would need additional 4 disks.
In short in your case, for the same volume, you cannot increase your storage 
space by adding one brick. You would need to add in multiples of 4 (ie 3+1)

thanks,
nag


- Original Message -
From: "Ashish Pandey" 
To: "Michel Lombart" 
Cc: gluster-users@gluster.org
Sent: Sunday, 20 August, 2017 6:27:00 PM
Subject: Re: [Gluster-users] Add brick to a disperse volume


Hi, 

Adding bricks to a disperse volume is very easy and same as replica volume. 
You just need to add bricks in the multiple of the number of bricks which you 
already have. 

So if you have disperse volume with n+k configuration, you need to add n+k more 
bricks. 
Example : 
If your disperse volume is 4+2, where 2 is the redundancy count, you need to 
provide 6 (or multiple of 6) bricks (4+2 = 6) when you want to expand or add 
bricks on your volume. 

I have following volume - 
[root@apandey apandey]# gluster v info 

Volume Name: vol 
Type: Disperse 
Volume ID: 4826eb6e-187d-4ccc-aa89-445338ec3620 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 1 x (4 + 2) = 6 
Transport-type: tcp 
Bricks: 
Brick1: apandey:/home/apandey/bricks/gluster/vol-1 
Brick2: apandey:/home/apandey/bricks/gluster/vol-2 
Brick3: apandey:/home/apandey/bricks/gluster/vol-3 
Brick4: apandey:/home/apandey/bricks/gluster/vol-4 
Brick5: apandey:/home/apandey/bricks/gluster/vol-5 
Brick6: apandey:/home/apandey/bricks/gluster/vol-6 
Options Reconfigured: 
transport.address-family: inet 
performance.readdir-ahead: on 
nfs.disable: on 
[root@apandey apandey]# 

Now, If I want to add bricks in this volume I have to provide bricks in the 
multiple of 6. It could be 6, 12,18. 

[root@apandey apandey]# gluster v add-brick vol 
apandey:/home/apandey/bricks/gluster/newbrick-1 
apandey:/home/apandey/bricks/gluster/newbrick-2 
apandey:/home/apandey/bricks/gluster/newbrick-3 
apandey:/home/apandey/bricks/gluster/newbrick-4 
apandey:/home/apandey/bricks/gluster/newbrick-5 
apandey:/home/apandey/bricks/gluster/newbrick-6 force 
volume add-brick: success 
[root@apandey apandey]# gluster v info 

Volume Name: vol 
Type: Distributed-Disperse 
Volume ID: 4826eb6e-187d-4ccc-aa89-445338ec3620 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 x (4 + 2) = 12 
Transport-type: tcp 
Bricks: 
Brick1: apandey:/home/apandey/bricks/gluster/vol-1 
Brick2: apandey:/home/apandey/bricks/gluster/vol-2 
Brick3: apandey:/home/apandey/bricks/gluster/vol-3 
Brick4: apandey:/home/apandey/bricks/gluster/vol-4 
Brick5: apandey:/home/apandey/bricks/gluster/vol-5 
Brick6: apandey:/home/apandey/bricks/gluster/vol-6 
Brick7: apandey:/home/apandey/bricks/gluster/newbrick-1 
Brick8: apandey:/home/apandey/bricks/gluster/newbrick-2 
Brick9: apandey:/home/apandey/bricks/gluster/newbrick-3 
Brick10: apandey:/home/apandey/bricks/gluster/newbrick-4 
Brick11: apandey:/home/apandey/bricks/gluster/newbrick-5 
Brick12: apandey:/home/apandey/bricks/gluster/newbrick-6 
Options Reconfigured: 
transport.address-family: inet 
performance.readdir-ahead: on 
nfs.disable: on 
[root@apandey apandey]# 

-- 
Ashish 





From: "Michel Lombart"  
To: gluster-users@gluster.org 
Sent: Saturday, August 19, 2017 11:50:31 PM 
Subject: [Gluster-users] Add brick to a disperse volume 



Hello, 



I’m using Gluster since 2 years but only with distributed volumes. 



I’m trying now to set dispersed volumes to have some redundancy. 



I had any problem to create a functional test volume with 4 bricks and 1 
redundancy ( Number of Bricks: 1 x (3 + 1) = 4 ). 



I had also any problem to replace a supposed faulty brick with another one. 



My problem is that I can not add a brick to increase the size of the volume as 
I do with distributed ones. I would have a volume of 5 bricks ( Number of 
Bricks: 1 x (4 + 1) = 5 ). 



The command « gluster volume add-brick test-volume 10.0.13.192:/data » return « 
volume add-brick: failed: Incorrect number of bricks supplied 1 with count 4 ». 



The documentation and Google was helpless. 



How can I do ? 



Thank for your help. 



Michel. 









___ 
Gluster-users mailing list 
Gluster-users@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users 


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance testing with sysbench...

[Krist van Besien]

Hi all,

I'm doing some performance test...

If I test a simple sequential write using dd I get a thorughput of about
550 Mb/s. When I do a sequential write test using sysbench this drops to
about 200. Is this due to the way sysbench tests? Or has in this case the
performance of sysbench itself become the bottleneck?

Krist


-- 
Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
--

Krist van Besien

senior architect, RHCE, RHCSA Open Stack

Red Hat Red Hat Switzerland S.A. 

kr...@redhat.comM: +41-79-5936260

TRIED. TESTED. TRUSTED. 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Glusterd proccess hangs on reboot

[Serkan Çoban]

Hi there,

I have a strange problem.
Gluster version in 3.10.5, I am testing new servers. Gluster
configuration is 16+4 EC, I have three volumes, each have 1600 bricks.
I can successfully create the cluster and volumes without any
problems. I write data to cluster from 100 clients for 12 hours again
no problem. But when I try to reboot a node, glusterd process hangs on
%100 CPU usage and seems to do nothing, no brick processes come
online. You can find strace of glusterd process for 1 minutes here:

https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0

Here is the glusterd logs:
https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0


By the way, reboot of one server completes without problem if I reboot
the servers before creating any volumes.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] self-heal not working

[Ravishankar N]

On 08/22/2017 02:30 PM, mabi wrote:

Thanks for the additional hints, I have the following 2 questions first:

- In order to launch the index heal is the following command correct:
gluster volume heal myvolume


Yes
- If I run a "volume start force" will it have any short disruptions 
on my clients which mount the volume through FUSE? If yes, how long? 
This is a production system that's why I am asking.



No. You can actually create a test volume on  your personal linux box to 
try these kinds of things without needing multiple machines. This is how 
we develop and test our patches :)
'gluster volume create testvol replica 3 /home/mabi/bricks/brick{1..3} 
force` and so on.


HTH,
Ravi




 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 22, 2017 6:26 AM
UTC Time: August 22, 2017 4:26 AM
From: ravishan...@redhat.com
To: mabi , Ben Turner 
Gluster Users 


Explore the following:

- Launch index heal and look at the glustershd logs of all bricks for 
possible errors


- See if the glustershd in each node is connected to all bricks.

- If not try to restart shd by `volume start force`

- Launch index heal again and try.

- Try debugging the shd log by setting client-log-level to DEBUG 
temporarily.



On 08/22/2017 03:19 AM, mabi wrote:

Sure, it doesn't look like a split brain based on the output:

Brick node1.domain.tld:/data/myvolume/brick
Status: Connected
Number of entries in split-brain: 0

Brick node2.domain.tld:/data/myvolume/brick
Status: Connected
Number of entries in split-brain: 0

Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
Status: Connected
Number of entries in split-brain: 0





 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 21, 2017 11:35 PM
UTC Time: August 21, 2017 9:35 PM
From: btur...@redhat.com
To: mabi 
Gluster Users 

Can you also provide:

gluster v heal  info split-brain

If it is split brain just delete the incorrect file from the brick 
and run heal again. I haven"t tried this with arbiter but I assume 
the process is the same.


-b

- Original Message -
> From: "mabi" 
> To: "Ben Turner" 
> Cc: "Gluster Users" 
> Sent: Monday, August 21, 2017 4:55:59 PM
> Subject: Re: [Gluster-users] self-heal not working
>
> Hi Ben,
>
> So it is really a 0 kBytes file everywhere (all nodes including 
the arbiter

> and from the client).
> Here below you will find the output you requested. Hopefully that 
will help
> to find out why this specific file is not healing... Let me know 
if you need

> any more information. Btw node3 is my arbiter node.
>
> NODE1:
>
> STAT:
> File:
> 
‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’

> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
> Device: 24h/36d Inode: 10033884 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> Access: 2017-08-14 17:04:55.530681000 +0200
> Modify: 2017-08-14 17:11:46.407404779 +0200
> Change: 2017-08-14 17:11:46.407404779 +0200
> Birth: -
>
> GETFATTR:
> trusted.afr.dirty=0sAQAA
> trusted.bit-rot.version=0sAgBZhuknAAlJAg==
> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
> 
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=

>
> NODE2:
>
> STAT:
> File:
> 
‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’

> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
> Device: 26h/38d Inode: 10031330 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> Access: 2017-08-14 17:04:55.530681000 +0200
> Modify: 2017-08-14 17:11:46.403704181 +0200
> Change: 2017-08-14 17:11:46.403704181 +0200
> Birth: -
>
> GETFATTR:
> trusted.afr.dirty=0sAQAA
> trusted.bit-rot.version=0sAgBZhu6wAA8Hpw==
> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
> 
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=

>
> NODE3:
> STAT:
> File:
> 
/srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png

> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: ca11h/51729d Inode: 405208959 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> Access: 2017-08-14 17:04:55.530681000 +0200
> Modify: 2017-08-14 17:04:55.530681000 +0200
> Change: 2017-08-14 17:11:46.604380051 +0200
> Birth: -
>
> GETFATTR:
> trusted.afr.dirty=0sAQAA
> trusted.bit-rot.version=0sAgBZe6ejAAKPAg==
> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
> 
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=

>
> CLIENT GLUSTER MOUNT:
> STAT:
> File:
> 
"/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"

> Size: 0 Blocks: 0 IO Block: 131072 regular empty file
> Device: 1eh/30d Inode: 11897049013408443114 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> Access: 2017-08-14 17:04:55.530681000 +0200
> Modify: 2017-08-14 17:11:46.4074047

Re: [Gluster-users] self-heal not working

[mabi]

Thanks for the additional hints, I have the following 2 questions first:

- In order to launch the index heal is the following command correct:
gluster volume heal myvolume

- If I run a "volume start force" will it have any short disruptions on my 
clients which mount the volume through FUSE? If yes, how long? This is a 
production system that's why I am asking.

>  Original Message 
> Subject: Re: [Gluster-users] self-heal not working
> Local Time: August 22, 2017 6:26 AM
> UTC Time: August 22, 2017 4:26 AM
> From: ravishan...@redhat.com
> To: mabi , Ben Turner 
> Gluster Users 
>
> Explore the following:
>
> - Launch index heal and look at the glustershd logs of all bricks for 
> possible errors
>
> - See if the glustershd in each node is connected to all bricks.
>
> - If not try to restart shd by `volume start force`
>
> - Launch index heal again and try.
>
> - Try debugging the shd log by setting client-log-level to DEBUG temporarily.
>
> On 08/22/2017 03:19 AM, mabi wrote:
>
>> Sure, it doesn't look like a split brain based on the output:
>>
>> Brick node1.domain.tld:/data/myvolume/brick
>> Status: Connected
>> Number of entries in split-brain: 0
>>
>> Brick node2.domain.tld:/data/myvolume/brick
>> Status: Connected
>> Number of entries in split-brain: 0
>>
>> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
>> Status: Connected
>> Number of entries in split-brain: 0
>>
>>>  Original Message 
>>> Subject: Re: [Gluster-users] self-heal not working
>>> Local Time: August 21, 2017 11:35 PM
>>> UTC Time: August 21, 2017 9:35 PM
>>> From: btur...@redhat.com
>>> To: mabi [](mailto:m...@protonmail.ch)
>>> Gluster Users 
>>> [](mailto:gluster-users@gluster.org)
>>>
>>> Can you also provide:
>>>
>>> gluster v heal  info split-brain
>>>
>>> If it is split brain just delete the incorrect file from the brick and run 
>>> heal again. I haven"t tried this with arbiter but I assume the process is 
>>> the same.
>>>
>>> -b
>>>
>>> - Original Message -
 From: "mabi" [](mailto:m...@protonmail.ch)
 To: "Ben Turner" [](mailto:btur...@redhat.com)
 Cc: "Gluster Users" 
 [](mailto:gluster-users@gluster.org)
 Sent: Monday, August 21, 2017 4:55:59 PM
 Subject: Re: [Gluster-users] self-heal not working

 Hi Ben,

 So it is really a 0 kBytes file everywhere (all nodes including the arbiter
 and from the client).
 Here below you will find the output you requested. Hopefully that will help
 to find out why this specific file is not healing... Let me know if you 
 need
 any more information. Btw node3 is my arbiter node.

 NODE1:

 STAT:
 File:
 ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
 Size: 0 Blocks: 38 IO Block: 131072 regular empty file
 Device: 24h/36d Inode: 10033884 Links: 2
 Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
 Access: 2017-08-14 17:04:55.530681000 +0200
 Modify: 2017-08-14 17:11:46.407404779 +0200
 Change: 2017-08-14 17:11:46.407404779 +0200
 Birth: -

 GETFATTR:
 trusted.afr.dirty=0sAQAA
 trusted.bit-rot.version=0sAgBZhuknAAlJAg==
 trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
 trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=

 NODE2:

 STAT:
 File:
 ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
 Size: 0 Blocks: 38 IO Block: 131072 regular empty file
 Device: 26h/38d Inode: 10031330 Links: 2
 Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
 Access: 2017-08-14 17:04:55.530681000 +0200
 Modify: 2017-08-14 17:11:46.403704181 +0200
 Change: 2017-08-14 17:11:46.403704181 +0200
 Birth: -

 GETFATTR:
 trusted.afr.dirty=0sAQAA
 trusted.bit-rot.version=0sAgBZhu6wAA8Hpw==
 trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
 trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=

 NODE3:
 STAT:
 File:
 /srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
 Size: 0 Blocks: 0 IO Block: 4096 regular empty file
 Device: ca11h/51729d Inode: 405208959 Links: 2
 Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
 Access: 2017-08-14 17:04:55.530681000 +0200
 Modify: 2017-08-14 17:04:55.530681000 +0200
 Change: 2017-08-14 17:11:46.604380051 +0200
 Birth: -

 GETFATTR:
 trusted.afr.dirty=0sAQAA
 trusted.bit-rot.version=0sAgBZe6ejAAKPAg==
 trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
 trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=

 CLIENT GLUSTER MOUNT:
 STAT:
 File:
 "/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"
 Size: 0 Blocks: 0 IO Block: 131072 regular empty file
 Device: 1eh/30d In