Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[Krutika Dhananjay]

Tried this.

With me, only 'fake2' gets healed after i bring the 'empty' brick back up
and it stops there unless I do a 'heal-full'.

Is that what you're seeing as well?

-Krutika

On Wed, Aug 31, 2016 at 4:43 AM, David Gossage 
wrote:

> Same issue brought up glusterd on problem node heal count still stuck at
> 6330.
>
> Ran gluster v heal GUSTER1 full
>
> glustershd on problem node shows a sweep starting and finishing in
> seconds.  Other 2 nodes show no activity in log.  They should start a sweep
> too shouldn't they?
>
> Tried starting from scratch
>
> kill -15 brickpid
> rm -Rf /brick
> mkdir -p /brick
> mkdir mkdir /gsmount/fake2
> setfattr -n "user.some-name" -v "some-value" /gsmount/fake2
>
> Heals visible dirs instantly then stops.
>
> gluster v heal GLUSTER1 full
>
> see sweep star on problem node and end almost instantly.  no files added t
> heal list no files healed no more logging
>
> [2016-08-30 23:11:31.544331] I [MSGID: 108026]
> [afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
> starting full sweep on subvol GLUSTER1-client-1
> [2016-08-30 23:11:33.776235] I [MSGID: 108026]
> [afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
> finished full sweep on subvol GLUSTER1-client-1
>
> same results no matter which node you run command on.  Still stuck with
> 6330 files showing needing healed out of 19k.  still showing in logs no
> heals are occuring.
>
> Is their a way to forcibly reset any prior heal data?  Could it be stuck
> on some past failed heal start?
>
>
>
>
> *David Gossage*
> *Carousel Checks Inc. | System Administrator*
> *Office* 708.613.2284
>
> On Tue, Aug 30, 2016 at 10:03 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Tue, Aug 30, 2016 at 10:02 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> updated test server to 3.8.3
>>>
>>> Brick1: 192.168.71.10:/gluster2/brick1/1
>>> Brick2: 192.168.71.11:/gluster2/brick2/1
>>> Brick3: 192.168.71.12:/gluster2/brick3/1
>>> Options Reconfigured:
>>> cluster.granular-entry-heal: on
>>> performance.readdir-ahead: on
>>> performance.read-ahead: off
>>> nfs.disable: on
>>> nfs.addr-namelookup: off
>>> nfs.enable-ino32: off
>>> cluster.background-self-heal-count: 16
>>> cluster.self-heal-window-size: 1024
>>> performance.quick-read: off
>>> performance.io-cache: off
>>> performance.stat-prefetch: off
>>> cluster.eager-lock: enable
>>> network.remote-dio: on
>>> cluster.quorum-type: auto
>>> cluster.server-quorum-type: server
>>> storage.owner-gid: 36
>>> storage.owner-uid: 36
>>> server.allow-insecure: on
>>> features.shard: on
>>> features.shard-block-size: 64MB
>>> performance.strict-o-direct: off
>>> cluster.locking-scheme: granular
>>>
>>> kill -15 brickpid
>>> rm -Rf /gluster2/brick3
>>> mkdir -p /gluster2/brick3/1
>>> mkdir mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard
>>> /fake2
>>> setfattr -n "user.some-name" -v "some-value"
>>> /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake2
>>> gluster v start glustershard force
>>>
>>> at this point brick process starts and all visible files including new
>>> dir are made on brick
>>> handful of shards are in heal statistics still but no .shard directory
>>> created and no increase in shard count
>>>
>>> gluster v heal glustershard
>>>
>>> At this point still no increase in count or dir made no additional
>>> activity in logs for healing generated.  waited few minutes tailing logs to
>>> check if anything kicked in.
>>>
>>> gluster v heal glustershard full
>>>
>>> gluster shards added to list and heal commences.  logs show full sweep
>>> starting on all 3 nodes.  though this time it only shows as finishing on
>>> one which looks to be the one that had brick deleted.
>>>
>>> [2016-08-30 14:45:33.098589] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
>>> starting full sweep on subvol glustershard-client-0
>>> [2016-08-30 14:45:33.099492] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
>>> starting full sweep on subvol glustershard-client-1
>>> [2016-08-30 14:45:33.100093] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
>>> starting full sweep on subvol glustershard-client-2
>>> [2016-08-30 14:52:29.760213] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
>>> finished full sweep on subvol glustershard-client-2
>>>
>>
>> Just realized its still healing so that may be why sweep on 2 other
>> bricks haven't replied as finished.
>>
>>>
>>>
>>> my hope is that later tonight a full heal will work on production.  Is
>>> it possible self-heal daemon can get stale or stop listening but still show
>>> as active?  Would stopping and starting self-heal daemon from gluster cli
>>> before doing these heals be helpful?
>>>
>>>
>>> On Tue, Aug 30, 2016 at 9:29 AM, David Gossage <
>>> 

[Gluster-users] group write permissions not being respected

[Pat Haley]

Hi

We have just migrated our data to a new file server (more space, old 
server was showing its age). We have a volume for collaborative use, 
based on group membership.  In our new server, the group write 
permissions are not being respected (e.g.  the owner of a directory can 
still write to that directory but any other member of the associated 
group cannot, even though the directory clearly has group write 
permissions set).  This is occurring regardless of how many groups the 
user is a member of (i.e. users that are members of fewer then 16 groups 
are still affected).


the relevant fstab line from the server looks like
localhost:/data-volume /gdataglusterfs   defaults 0 0

and for a client:
mseas-data2:/gdata   /gdata  nfs defaults0 0

Any help would be greatly appreciated.

Thanks

--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:  pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] incorrect usage value on a directory

[Manikandan Selvaganesh]

Hi Sergei,

Apologies for the delay. I am extremely sorry, I was struck on something
important
It's great that you figured out the solution.

Whenever you set a dirty flag as mentioned in the previous thread, the
quota values will be recalcualted.
Yep, as you mentioned there are lot of changes that has gone in from 3.7.
We have
introduced Inode-quota feature in 3.7, then we have implemented the Quota
versioning
in 3.7.5 and then enhance quota enable/disable feature in 3.7.12. So a lot
of code changes
has been done.

In case would you like to know more, you can refer our specs[1].

[1] https://github.com/gluster/glusterfs-specs

On Tue, Aug 30, 2016 at 9:27 PM, Sergei Gerasenko  wrote:

> The problem must have started because of an upgrade to 3.7.12 from an
> older version. Not sure exactly how.
>
> On Aug 30, 2016, at 10:44 AM, Sergei Gerasenko  wrote:
>
> It seems that it did the trick. The usage is being recalculated. I’m glad
> to be posting a solution to the original problem on this thread. It’s so
> frequent that threads contain only incomplete or partially complete
> solutions.
>
> Thanks,
>   Sergei
>
> On Aug 29, 2016, at 3:41 PM, Sergei Gerasenko 
> wrote:
>
> I found an informative thread on a similar problem:
>
> http://www.spinics.net/lists/gluster-devel/msg18400.html
>
> According to the thread, it seems that the solution is to disable the
> quota, which will clear the relevant xattrs and then re-enable the quota
> which should force a recalc. I will try this tomorrow.
>
> On Thu, Aug 11, 2016 at 9:31 AM, Sergei Gerasenko 
> wrote:
>
>> Hi Selvaganesh,
>>
>> Thanks so much for your help. I didn’t have that option on probably
>> because I originally had a lower version of cluster and then upgraded. I
>> turned the option on just now.
>>
>> The usage is still off. Should I wait a certain time?
>>
>> Thanks,
>>   Sergei
>>
>> On Aug 9, 2016, at 7:26 AM, Manikandan Selvaganesh 
>> wrote:
>>
>> Hi Sergei,
>>
>> When quota is enabled, quota-deem-statfs should be set to ON(By default
>> with the recent versions). But apparently
>> from your 'gluster v info' output, it is like quota-deem-statfs is not
>> on.
>>
>> Could you please check and confirm the same on
>> /var/lib/glusterd/vols//info. If you do not find an option
>> 'features.quota-deem-statfs=on', then this feature is turned off. Did
>> you turn off this one? You could turn it on by doing this
>> 'gluster volume set  quota-deem-statfs on'.
>>
>> To know more about this feature, please refer here[1]
>>
>> [1] https://gluster.readthedocs.io/en/latest/Administrator%
>> 20Guide/Directory%20Quota/
>>
>>
>> On Tue, Aug 9, 2016 at 5:43 PM, Sergei Gerasenko 
>> wrote:
>>
>>> Hi ,
>>>
>>> The gluster version is 3.7.12. Here’s the output of `gluster info`:
>>>
>>> Volume Name: ftp_volume
>>> Type: Distributed-Replicate
>>> Volume ID: SOME_VOLUME_ID
>>> Status: Started
>>> Number of Bricks: 3 x 2 = 6
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: host03:/data/ftp_gluster_brick
>>> Brick2: host04:/data/ftp_gluster_brick
>>> Brick3: host05:/data/ftp_gluster_brick
>>> Brick4: host06:/data/ftp_gluster_brick
>>> Brick5: host07:/data/ftp_gluster_brick
>>> Brick6: host08:/data/ftp_gluster_brick
>>> Options Reconfigured:
>>> features.quota: on
>>>
>>> Thanks for the reply!! I thought nobody would reply at this point :)
>>>
>>> Sergei
>>>
>>> On Aug 9, 2016, at 6:03 AM, Manikandan Selvaganesh 
>>> wrote:
>>>
>>> Hi,
>>>
>>> Sorry, I missed the mail. May I know which version of gluster you are
>>> using and please paste the output of
>>> gluster v info?
>>>
>>> On Sat, Aug 6, 2016 at 8:19 AM, Sergei Gerasenko 
>>> wrote:
>>>
 Hi,

 I'm playing with quotas and the quota list command on one of the
 directories claims it uses 3T, whereas the du command says only 512G is
 used.

 Anything I can do to force a re-calc, re-crawl, etc?

 Thanks,
  Sergei

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users

>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manikandan Selvaganesh.
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Manikandan Selvaganesh.
>>
>>
>>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>



-- 
Regards,
Manikandan Selvaganesh.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] incorrect usage value on a directory

[Sergei Gerasenko]

The problem must have started because of an upgrade to 3.7.12 from an older 
version. Not sure exactly how.

> On Aug 30, 2016, at 10:44 AM, Sergei Gerasenko  wrote:
> 
> It seems that it did the trick. The usage is being recalculated. I’m glad to 
> be posting a solution to the original problem on this thread. It’s so 
> frequent that threads contain only incomplete or partially complete solutions.
> 
> Thanks,
>   Sergei
> 
>> On Aug 29, 2016, at 3:41 PM, Sergei Gerasenko > > wrote:
>> 
>> I found an informative thread on a similar problem:
>> 
>> http://www.spinics.net/lists/gluster-devel/msg18400.html 
>> 
>> 
>> According to the thread, it seems that the solution is to disable the quota, 
>> which will clear the relevant xattrs and then re-enable the quota which 
>> should force a recalc. I will try this tomorrow. 
>> 
>> On Thu, Aug 11, 2016 at 9:31 AM, Sergei Gerasenko > > wrote:
>> Hi Selvaganesh,
>> 
>> Thanks so much for your help. I didn’t have that option on probably because 
>> I originally had a lower version of cluster and then upgraded. I turned the 
>> option on just now.
>> 
>> The usage is still off. Should I wait a certain time?
>> 
>> Thanks,
>>   Sergei
>> 
>>> On Aug 9, 2016, at 7:26 AM, Manikandan Selvaganesh >> > wrote:
>>> 
>>> Hi Sergei,
>>> 
>>> When quota is enabled, quota-deem-statfs should be set to ON(By default 
>>> with the recent versions). But apparently 
>>> from your 'gluster v info' output, it is like quota-deem-statfs is not on. 
>>> 
>>> Could you please check and confirm the same on 
>>> /var/lib/glusterd/vols//info. If you do not find an option 
>>> 'features.quota-deem-statfs=on', then this feature is turned off. Did you 
>>> turn off this one? You could turn it on by doing this
>>> 'gluster volume set  quota-deem-statfs on'.
>>> 
>>> To know more about this feature, please refer here[1] 
>>> 
>>> [1] 
>>> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Directory%20Quota/
>>>  
>>> 
>>>  
>>> 
>>> 
>>> On Tue, Aug 9, 2016 at 5:43 PM, Sergei Gerasenko >> > wrote:
>>> Hi ,
>>> 
>>> The gluster version is 3.7.12. Here’s the output of `gluster info`:
>>> 
>>> Volume Name: ftp_volume
>>> Type: Distributed-Replicate
>>> Volume ID: SOME_VOLUME_ID
>>> Status: Started
>>> Number of Bricks: 3 x 2 = 6
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: host03:/data/ftp_gluster_brick
>>> Brick2: host04:/data/ftp_gluster_brick
>>> Brick3: host05:/data/ftp_gluster_brick
>>> Brick4: host06:/data/ftp_gluster_brick
>>> Brick5: host07:/data/ftp_gluster_brick
>>> Brick6: host08:/data/ftp_gluster_brick
>>> Options Reconfigured:
>>> features.quota: on
>>> 
>>> Thanks for the reply!! I thought nobody would reply at this point :)
>>> 
>>> Sergei
>>> 
 On Aug 9, 2016, at 6:03 AM, Manikandan Selvaganesh > wrote:
 
 Hi,
 
 Sorry, I missed the mail. May I know which version of gluster you are 
 using and please paste the output of
 gluster v info?
 
 On Sat, Aug 6, 2016 at 8:19 AM, Sergei Gerasenko > wrote:
 Hi,
 
 I'm playing with quotas and the quota list command on one of the 
 directories claims it uses 3T, whereas the du command says only 512G is 
 used.
 
 Anything I can do to force a re-calc, re-crawl, etc?
 
 Thanks,
  Sergei
 
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org 
 http://www.gluster.org/mailman/listinfo/gluster-users 
 
 
 
 
 -- 
 Regards,
 Manikandan Selvaganesh.
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Regards,
>>> Manikandan Selvaganesh.
>> 
>> 
> 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] incorrect usage value on a directory

[Sergei Gerasenko]

It seems that it did the trick. The usage is being recalculated. I’m glad to be 
posting a solution to the original problem on this thread. It’s so frequent 
that threads contain only incomplete or partially complete solutions.

Thanks,
  Sergei

> On Aug 29, 2016, at 3:41 PM, Sergei Gerasenko  wrote:
> 
> I found an informative thread on a similar problem:
> 
> http://www.spinics.net/lists/gluster-devel/msg18400.html 
> 
> 
> According to the thread, it seems that the solution is to disable the quota, 
> which will clear the relevant xattrs and then re-enable the quota which 
> should force a recalc. I will try this tomorrow. 
> 
> On Thu, Aug 11, 2016 at 9:31 AM, Sergei Gerasenko  > wrote:
> Hi Selvaganesh,
> 
> Thanks so much for your help. I didn’t have that option on probably because I 
> originally had a lower version of cluster and then upgraded. I turned the 
> option on just now.
> 
> The usage is still off. Should I wait a certain time?
> 
> Thanks,
>   Sergei
> 
>> On Aug 9, 2016, at 7:26 AM, Manikandan Selvaganesh > > wrote:
>> 
>> Hi Sergei,
>> 
>> When quota is enabled, quota-deem-statfs should be set to ON(By default with 
>> the recent versions). But apparently 
>> from your 'gluster v info' output, it is like quota-deem-statfs is not on. 
>> 
>> Could you please check and confirm the same on 
>> /var/lib/glusterd/vols//info. If you do not find an option 
>> 'features.quota-deem-statfs=on', then this feature is turned off. Did you 
>> turn off this one? You could turn it on by doing this
>> 'gluster volume set  quota-deem-statfs on'.
>> 
>> To know more about this feature, please refer here[1] 
>> 
>> [1] 
>> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Directory%20Quota/
>>  
>> 
>>  
>> 
>> 
>> On Tue, Aug 9, 2016 at 5:43 PM, Sergei Gerasenko > > wrote:
>> Hi ,
>> 
>> The gluster version is 3.7.12. Here’s the output of `gluster info`:
>> 
>> Volume Name: ftp_volume
>> Type: Distributed-Replicate
>> Volume ID: SOME_VOLUME_ID
>> Status: Started
>> Number of Bricks: 3 x 2 = 6
>> Transport-type: tcp
>> Bricks:
>> Brick1: host03:/data/ftp_gluster_brick
>> Brick2: host04:/data/ftp_gluster_brick
>> Brick3: host05:/data/ftp_gluster_brick
>> Brick4: host06:/data/ftp_gluster_brick
>> Brick5: host07:/data/ftp_gluster_brick
>> Brick6: host08:/data/ftp_gluster_brick
>> Options Reconfigured:
>> features.quota: on
>> 
>> Thanks for the reply!! I thought nobody would reply at this point :)
>> 
>> Sergei
>> 
>>> On Aug 9, 2016, at 6:03 AM, Manikandan Selvaganesh >> > wrote:
>>> 
>>> Hi,
>>> 
>>> Sorry, I missed the mail. May I know which version of gluster you are using 
>>> and please paste the output of
>>> gluster v info?
>>> 
>>> On Sat, Aug 6, 2016 at 8:19 AM, Sergei Gerasenko >> > wrote:
>>> Hi,
>>> 
>>> I'm playing with quotas and the quota list command on one of the 
>>> directories claims it uses 3T, whereas the du command says only 512G is 
>>> used.
>>> 
>>> Anything I can do to force a re-calc, re-crawl, etc?
>>> 
>>> Thanks,
>>>  Sergei
>>> 
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org 
>>> http://www.gluster.org/mailman/listinfo/gluster-users 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Regards,
>>> Manikandan Selvaganesh.
>> 
>> 
>> 
>> 
>> -- 
>> Regards,
>> Manikandan Selvaganesh.
> 
> 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Terrible performance when a gluster node is down/rebooting

[David Cowe]

Hello,

We have a 5 node gluster setup with the following configuration;


[root@gb0015nasslow01 Test]# gluster vol info all

Volume Name: data
Type: Tier
Volume ID: 702daa3d-b3fa-4e66-bcec-a13f7ec1d47d
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: glusternasbackup01:/bricks/hotbrick/brick
Brick2: glusternasfast02:/bricks/brick4/brick
Brick3: glusternasfast01:/bricks/brick3/brick
Cold Tier:
Cold Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick4: glusternasslow01:/bricks/brick1/brick
Brick5: glusternasslow02:/bricks/brick2/brick
Brick6: glusternasbackup01:/bricks/coldbrick/brick
Options Reconfigured:
performance.write-behind-window-size: 100mb
performance.cache-size: 20GB
storage.batch-fsync-delay-usec: 0
server.allow-insecure: on
performance.stat-prefetch: off
performance.readdir-ahead: on
features.ctr-enabled: on
cluster.tier-mode: cache


When we shutdown/reboot/pull the power from 1 of the nodes, the other
servers are very laggy when even doing an ls within the glusterfs mounted
filesystem (/glusterdata).

df -h
Filesystem   Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  4.7G   46G  10% /
devtmpfs  48G 0   48G   0% /dev
tmpfs 48G 0   48G   0% /dev/shm
tmpfs 48G   66M   48G   1% /run
tmpfs 48G 0   48G   0% /sys/fs/cgroup
/dev/sdb1110T  221M  110T   1% /bricks/brick1
/dev/sda2497M  201M  297M  41% /boot
/dev/sda1200M  9.5M  191M   5% /boot/efi
/dev/mapper/centos-home  225G   33M  224G   1% /home
localhost:data   109T  464G  108T   1% /glusterdata
tmpfs9.5G 0  9.5G   0% /run/user/0

During the time the node is down or coming back up, we not only get the lag
described above on the server but file reads and writes to the gluster
volume pretty much stall.




Can anyone give me some much needed advice as surely this should not be the
case?




*packages installed;*

yum list gluster*

Installed Packages
glusterfs.x86_643.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-api.x86_643.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-cli.x86_643.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-client-xlators.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-fuse.x86_64   3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-libs.x86_64   3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-server.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64


yum list samba*

Installed Packages
samba.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client-libs.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common.noarch
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-libs.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-tools.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-libs.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-vfs-glusterfs.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-clients.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-modules.x86_64
 4.2.10-7.el7_2
@centos7-updates-x86_64


yum list ctdb*

Installed Packages
ctdb.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64


Background - We use CTDB for VIP's and heartbeat and we present our gluster
volume via samba to our Windows clients.

Regards,
David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

On Tue, Aug 30, 2016 at 10:02 AM, David Gossage  wrote:

> updated test server to 3.8.3
>
> Brick1: 192.168.71.10:/gluster2/brick1/1
> Brick2: 192.168.71.11:/gluster2/brick2/1
> Brick3: 192.168.71.12:/gluster2/brick3/1
> Options Reconfigured:
> cluster.granular-entry-heal: on
> performance.readdir-ahead: on
> performance.read-ahead: off
> nfs.disable: on
> nfs.addr-namelookup: off
> nfs.enable-ino32: off
> cluster.background-self-heal-count: 16
> cluster.self-heal-window-size: 1024
> performance.quick-read: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: on
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> storage.owner-gid: 36
> storage.owner-uid: 36
> server.allow-insecure: on
> features.shard: on
> features.shard-block-size: 64MB
> performance.strict-o-direct: off
> cluster.locking-scheme: granular
>
> kill -15 brickpid
> rm -Rf /gluster2/brick3
> mkdir -p /gluster2/brick3/1
> mkdir mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10\:_
> glustershard/fake2
> setfattr -n "user.some-name" -v "some-value" /rhev/data-center/mnt/
> glusterSD/192.168.71.10\:_glustershard/fake2
> gluster v start glustershard force
>
> at this point brick process starts and all visible files including new dir
> are made on brick
> handful of shards are in heal statistics still but no .shard directory
> created and no increase in shard count
>
> gluster v heal glustershard
>
> At this point still no increase in count or dir made no additional
> activity in logs for healing generated.  waited few minutes tailing logs to
> check if anything kicked in.
>
> gluster v heal glustershard full
>
> gluster shards added to list and heal commences.  logs show full sweep
> starting on all 3 nodes.  though this time it only shows as finishing on
> one which looks to be the one that had brick deleted.
>
> [2016-08-30 14:45:33.098589] I [MSGID: 108026]
> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
> starting full sweep on subvol glustershard-client-0
> [2016-08-30 14:45:33.099492] I [MSGID: 108026]
> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
> starting full sweep on subvol glustershard-client-1
> [2016-08-30 14:45:33.100093] I [MSGID: 108026]
> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
> starting full sweep on subvol glustershard-client-2
> [2016-08-30 14:52:29.760213] I [MSGID: 108026]
> [afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
> finished full sweep on subvol glustershard-client-2
>

Just realized its still healing so that may be why sweep on 2 other bricks
haven't replied as finished.

>
>
> my hope is that later tonight a full heal will work on production.  Is it
> possible self-heal daemon can get stale or stop listening but still show as
> active?  Would stopping and starting self-heal daemon from gluster cli
> before doing these heals be helpful?
>
>
> On Tue, Aug 30, 2016 at 9:29 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Tue, Aug 30, 2016 at 8:52 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> On Tue, Aug 30, 2016 at 8:01 AM, Krutika Dhananjay 
>>> wrote:
>>>


 On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay  wrote:

>
>
> On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay <
>> kdhan...@redhat.com> wrote:
>>
>>> Could you also share the glustershd logs?
>>>
>>
>> I'll get them when I get to work sure
>>
>
>>
>>>
>>> I tried the same steps that you mentioned multiple times, but heal
>>> is running to completion without any issues.
>>>
>>> It must be said that 'heal full' traverses the files and directories
>>> in a depth-first order and does heals also in the same order. But if it
>>> gets interrupted in the middle (say because self-heal-daemon was either
>>> intentionally or unintentionally brought offline and then brought back 
>>> up),
>>> self-heal will only pick up the entries that are so far marked as
>>> new-entries that need heal which it will find in indices/xattrop 
>>> directory.
>>> What this means is that those files and directories that were not 
>>> visited
>>> during the crawl, will remain untouched and unhealed in this second
>>> iteration of heal, unless you execute a 'heal-full' again.
>>>
>>
>> So should it start healing shards as it crawls or not until after it
>> crawls the entire .shard directory?  At the pace it was going that could 
>> be
>> a week with one node appearing in the cluster but with no shard files if
>> anything tries to access a file on that node.  From my experience other 
>> day
>> telling it to heal full again did 

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

updated test server to 3.8.3

Brick1: 192.168.71.10:/gluster2/brick1/1
Brick2: 192.168.71.11:/gluster2/brick2/1
Brick3: 192.168.71.12:/gluster2/brick3/1
Options Reconfigured:
cluster.granular-entry-heal: on
performance.readdir-ahead: on
performance.read-ahead: off
nfs.disable: on
nfs.addr-namelookup: off
nfs.enable-ino32: off
cluster.background-self-heal-count: 16
cluster.self-heal-window-size: 1024
performance.quick-read: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: on
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on
features.shard: on
features.shard-block-size: 64MB
performance.strict-o-direct: off
cluster.locking-scheme: granular

kill -15 brickpid
rm -Rf /gluster2/brick3
mkdir -p /gluster2/brick3/1
mkdir mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10
\:_glustershard/fake2
setfattr -n "user.some-name" -v "some-value"
/rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake2
gluster v start glustershard force

at this point brick process starts and all visible files including new dir
are made on brick
handful of shards are in heal statistics still but no .shard directory
created and no increase in shard count

gluster v heal glustershard

At this point still no increase in count or dir made no additional activity
in logs for healing generated.  waited few minutes tailing logs to check if
anything kicked in.

gluster v heal glustershard full

gluster shards added to list and heal commences.  logs show full sweep
starting on all 3 nodes.  though this time it only shows as finishing on
one which looks to be the one that had brick deleted.

[2016-08-30 14:45:33.098589] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
starting full sweep on subvol glustershard-client-0
[2016-08-30 14:45:33.099492] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
starting full sweep on subvol glustershard-client-1
[2016-08-30 14:45:33.100093] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
starting full sweep on subvol glustershard-client-2
[2016-08-30 14:52:29.760213] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
finished full sweep on subvol glustershard-client-2


my hope is that later tonight a full heal will work on production.  Is it
possible self-heal daemon can get stale or stop listening but still show as
active?  Would stopping and starting self-heal daemon from gluster cli
before doing these heals be helpful?


On Tue, Aug 30, 2016 at 9:29 AM, David Gossage 
wrote:

> On Tue, Aug 30, 2016 at 8:52 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Tue, Aug 30, 2016 at 8:01 AM, Krutika Dhananjay 
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay 
>>> wrote:
>>>


 On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
 dgoss...@carouselchecks.com> wrote:

> On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay <
> kdhan...@redhat.com> wrote:
>
>> Could you also share the glustershd logs?
>>
>
> I'll get them when I get to work sure
>

>
>>
>> I tried the same steps that you mentioned multiple times, but heal is
>> running to completion without any issues.
>>
>> It must be said that 'heal full' traverses the files and directories
>> in a depth-first order and does heals also in the same order. But if it
>> gets interrupted in the middle (say because self-heal-daemon was either
>> intentionally or unintentionally brought offline and then brought back 
>> up),
>> self-heal will only pick up the entries that are so far marked as
>> new-entries that need heal which it will find in indices/xattrop 
>> directory.
>> What this means is that those files and directories that were not visited
>> during the crawl, will remain untouched and unhealed in this second
>> iteration of heal, unless you execute a 'heal-full' again.
>>
>
> So should it start healing shards as it crawls or not until after it
> crawls the entire .shard directory?  At the pace it was going that could 
> be
> a week with one node appearing in the cluster but with no shard files if
> anything tries to access a file on that node.  From my experience other 
> day
> telling it to heal full again did nothing regardless of node used.
>

>>> Crawl is started from '/' of the volume. Whenever self-heal detects
>>> during the crawl that a file or directory is present in some brick(s) and
>>> absent in others, it creates the file on the bricks where it is absent and
>>> marks the fact that the file or directory might need data/entry and
>>> metadata heal too (this also means that an index 

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

On Tue, Aug 30, 2016 at 8:52 AM, David Gossage 
wrote:

> On Tue, Aug 30, 2016 at 8:01 AM, Krutika Dhananjay 
> wrote:
>
>>
>>
>> On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay 
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
 On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay  wrote:

> Could you also share the glustershd logs?
>

 I'll get them when I get to work sure

>>>

>
> I tried the same steps that you mentioned multiple times, but heal is
> running to completion without any issues.
>
> It must be said that 'heal full' traverses the files and directories
> in a depth-first order and does heals also in the same order. But if it
> gets interrupted in the middle (say because self-heal-daemon was either
> intentionally or unintentionally brought offline and then brought back 
> up),
> self-heal will only pick up the entries that are so far marked as
> new-entries that need heal which it will find in indices/xattrop 
> directory.
> What this means is that those files and directories that were not visited
> during the crawl, will remain untouched and unhealed in this second
> iteration of heal, unless you execute a 'heal-full' again.
>

 So should it start healing shards as it crawls or not until after it
 crawls the entire .shard directory?  At the pace it was going that could be
 a week with one node appearing in the cluster but with no shard files if
 anything tries to access a file on that node.  From my experience other day
 telling it to heal full again did nothing regardless of node used.

>>>
>> Crawl is started from '/' of the volume. Whenever self-heal detects
>> during the crawl that a file or directory is present in some brick(s) and
>> absent in others, it creates the file on the bricks where it is absent and
>> marks the fact that the file or directory might need data/entry and
>> metadata heal too (this also means that an index is created under
>> .glusterfs/indices/xattrop of the src bricks). And the data/entry and
>> metadata heal are picked up and done in
>>
> the background with the help of these indices.
>>
>
> Looking at my 3rd node as example i find nearly an exact same number of
> files in xattrop dir as reported by heal count at time I brought down node2
> to try and alleviate read io errors that seemed to occur from what I was
> guessing as attempts to use the node with no shards for reads.
>
> Also attached are the glustershd logs from the 3 nodes, along with the
> test node i tried yesterday with same results.
>

Looking at my own logs I notice that a full sweep was only ever recorded in
glustershd.log on 2nd node with missing directory.  I believe I should have
found a sweep begun on every node correct?

On my test dev when it did work I do see that

[2016-08-30 13:56:25.22] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
starting full sweep on subvol glustershard-client-0
[2016-08-30 13:56:25.223522] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
starting full sweep on subvol glustershard-client-1
[2016-08-30 13:56:25.224616] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
starting full sweep on subvol glustershard-client-2
[2016-08-30 14:18:48.333740] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
finished full sweep on subvol glustershard-client-2
[2016-08-30 14:18:48.356008] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
finished full sweep on subvol glustershard-client-1
[2016-08-30 14:18:49.637811] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
finished full sweep on subvol glustershard-client-0

While when looking at past few days of the 3 prod nodes i only found that
on my 2nd node
[2016-08-27 01:26:42.638772] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0: starting
full sweep on subvol GLUSTER1-client-1
[2016-08-27 11:37:01.732366] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0: finished
full sweep on subvol GLUSTER1-client-1
[2016-08-27 12:58:34.597228] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0: starting
full sweep on subvol GLUSTER1-client-1
[2016-08-27 12:59:28.041173] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0: finished
full sweep on subvol GLUSTER1-client-1
[2016-08-27 20:03:42.560188] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0: starting
full sweep on subvol GLUSTER1-client-1
[2016-08-27 20:03:44.278274] I [MSGID: 108026]

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

On Tue, Aug 30, 2016 at 8:52 AM, David Gossage 
wrote:

> On Tue, Aug 30, 2016 at 8:01 AM, Krutika Dhananjay 
> wrote:
>
>>
>>
>> On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay 
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
 On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay  wrote:

> Could you also share the glustershd logs?
>

 I'll get them when I get to work sure

>>>

>
> I tried the same steps that you mentioned multiple times, but heal is
> running to completion without any issues.
>
> It must be said that 'heal full' traverses the files and directories
> in a depth-first order and does heals also in the same order. But if it
> gets interrupted in the middle (say because self-heal-daemon was either
> intentionally or unintentionally brought offline and then brought back 
> up),
> self-heal will only pick up the entries that are so far marked as
> new-entries that need heal which it will find in indices/xattrop 
> directory.
> What this means is that those files and directories that were not visited
> during the crawl, will remain untouched and unhealed in this second
> iteration of heal, unless you execute a 'heal-full' again.
>

 So should it start healing shards as it crawls or not until after it
 crawls the entire .shard directory?  At the pace it was going that could be
 a week with one node appearing in the cluster but with no shard files if
 anything tries to access a file on that node.  From my experience other day
 telling it to heal full again did nothing regardless of node used.

>>>
>> Crawl is started from '/' of the volume. Whenever self-heal detects
>> during the crawl that a file or directory is present in some brick(s) and
>> absent in others, it creates the file on the bricks where it is absent and
>> marks the fact that the file or directory might need data/entry and
>> metadata heal too (this also means that an index is created under
>> .glusterfs/indices/xattrop of the src bricks). And the data/entry and
>> metadata heal are picked up and done in
>>
> the background with the help of these indices.
>>
>
> Looking at my 3rd node as example i find nearly an exact same number of
> files in xattrop dir as reported by heal count at time I brought down node2
> to try and alleviate read io errors that seemed to occur from what I was
> guessing as attempts to use the node with no shards for reads.
>
> Also attached are the glustershd logs from the 3 nodes, along with the
> test node i tried yesterday with same results.
>

Is it possible you just need to spam the heal full command?  Wait for a
certain amount of time for it to time out?

The test server that I did yesterday that stopped at listing 33 shards then
healing none of them stlll had 33 shards in list this morning.  I issued
another heal full and it jumped up and found the missing shards.

On the one hand its reassuring that if I just spam the command enough
eventually it will heal.  It's also disconcerting that if I spam the
command enough times the heal will start.

I can't test if same behavior would occur on live node as I expect if it
did kick in heals I'd have 12 hours of high load during copy again
perhaps.  But I can test if it happens after last shift.  Though I lost
track of how many times I tried restarting heal full over Saturday and
Sunday when it looked to be doing nothing from all heal tracking commands
documented.


>>

> My suspicion is that this is what happened on your setup. Could you
> confirm if that was the case?
>

 Brick was brought online with force start then a full heal launched.
 Hours later after it became evident that it was not adding new files to
 heal I did try restarting self-heal daemon and relaunching full heal again.
 But this was after the heal had basically already failed to work as
 intended.

>>>
>>> OK. How did you figure it was not adding any new files? I need to know
>>> what places you were monitoring to come to this conclusion.
>>>
>>> -Krutika
>>>
>>>


> As for those logs, I did manager to do something that caused these
> warning messages you shared earlier to appear in my client and server 
> logs.
> Although these logs are annoying and a bit scary too, they didn't do
> any harm to the data in my volume. Why they appear just after a brick is
> replaced and under no other circumstances is something I'm still
> investigating.
>
> But for future, it would be good to follow the steps Anuradha gave as
> that would allow self-heal to at least detect that it has some repairing 
> to
> do whenever it is restarted whether intentionally or otherwise.
>

 I followed those steps as 

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

On Tue, Aug 30, 2016 at 7:50 AM, Krutika Dhananjay 
wrote:

>
>
> On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay 
>> wrote:
>>
>>> Could you also share the glustershd logs?
>>>
>>
>> I'll get them when I get to work sure.
>>
>>
>>>
>>> I tried the same steps that you mentioned multiple times, but heal is
>>> running to completion without any issues.
>>>
>>> It must be said that 'heal full' traverses the files and directories in
>>> a depth-first order and does heals also in the same order. But if it gets
>>> interrupted in the middle (say because self-heal-daemon was either
>>> intentionally or unintentionally brought offline and then brought back up),
>>> self-heal will only pick up the entries that are so far marked as
>>> new-entries that need heal which it will find in indices/xattrop directory.
>>> What this means is that those files and directories that were not visited
>>> during the crawl, will remain untouched and unhealed in this second
>>> iteration of heal, unless you execute a 'heal-full' again.
>>>
>>
>> So should it start healing shards as it crawls or not until after it
>> crawls the entire .shard directory?  At the pace it was going that could be
>> a week with one node appearing in the cluster but with no shard files if
>> anything tries to access a file on that node.  From my experience other day
>> telling it to heal full again did nothing regardless of node used.
>>
>>
>>> My suspicion is that this is what happened on your setup. Could you
>>> confirm if that was the case?
>>>
>>
>> Brick was brought online with force start then a full heal launched.
>> Hours later after it became evident that it was not adding new files to
>> heal I did try restarting self-heal daemon and relaunching full heal again.
>> But this was after the heal had basically already failed to work as
>> intended.
>>
>
> OK. How did you figure it was not adding any new files? I need to know
> what places you were monitoring to come to this conclusion.
>

As far as I know their are only 2 ways to monitor the heal process in
anyway.


gluster volume heal <> info
This lists the actual files needing healed.

gluster volume heal <> statistics heal-count
This lists just number of files in previous command

Both commands when run came to same conclusions about number of files
needing healed

If their is another way to tell if a heal is in operation or progressing I
have not found any documentation.




>
> -Krutika
>
>
>>
>>
>>> As for those logs, I did manager to do something that caused these
>>> warning messages you shared earlier to appear in my client and server logs.
>>> Although these logs are annoying and a bit scary too, they didn't do any
>>> harm to the data in my volume. Why they appear just after a brick is
>>> replaced and under no other circumstances is something I'm still
>>> investigating.
>>>
>>> But for future, it would be good to follow the steps Anuradha gave as
>>> that would allow self-heal to at least detect that it has some repairing to
>>> do whenever it is restarted whether intentionally or otherwise.
>>>
>>
>> I followed those steps as described on my test box and ended up with
>> exact same outcome of adding shards at an agonizing slow pace and no
>> creation of .shard directory or heals on shard directory.  Directories
>> visible from mount healed quickly.  This was with one VM so it has only 800
>> shards as well.  After hours at work it had added a total of 33 shards to
>> be healed.  I sent those logs yesterday as well though not the glustershd.
>>
>> Does replace-brick command copy files in same manner?  For these purposes
>> I am contemplating just skipping the heal route.
>>
>>
>>> -Krutika
>>>
>>> On Tue, Aug 30, 2016 at 2:22 AM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
 attached brick and client logs from test machine where same behavior
 occurred not sure if anything new is there.  its still on 3.8.2

 Number of Bricks: 1 x 3 = 3
 Transport-type: tcp
 Bricks:
 Brick1: 192.168.71.10:/gluster2/brick1/1
 Brick2: 192.168.71.11:/gluster2/brick2/1
 Brick3: 192.168.71.12:/gluster2/brick3/1
 Options Reconfigured:
 cluster.locking-scheme: granular
 performance.strict-o-direct: off
 features.shard-block-size: 64MB
 features.shard: on
 server.allow-insecure: on
 storage.owner-uid: 36
 storage.owner-gid: 36
 cluster.server-quorum-type: server
 cluster.quorum-type: auto
 network.remote-dio: on
 cluster.eager-lock: enable
 performance.stat-prefetch: off
 performance.io-cache: off
 performance.quick-read: off
 cluster.self-heal-window-size: 1024
 cluster.background-self-heal-count: 16
 nfs.enable-ino32: off
 nfs.addr-namelookup: off
 nfs.disable: on
 performance.read-ahead: off
 performance.readdir-ahead: on
 

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[Krutika Dhananjay]

On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay 
wrote:

>
>
> On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay 
>> wrote:
>>
>>> Could you also share the glustershd logs?
>>>
>>
>> I'll get them when I get to work sure.
>>
>>
>>>
>>> I tried the same steps that you mentioned multiple times, but heal is
>>> running to completion without any issues.
>>>
>>> It must be said that 'heal full' traverses the files and directories in
>>> a depth-first order and does heals also in the same order. But if it gets
>>> interrupted in the middle (say because self-heal-daemon was either
>>> intentionally or unintentionally brought offline and then brought back up),
>>> self-heal will only pick up the entries that are so far marked as
>>> new-entries that need heal which it will find in indices/xattrop directory.
>>> What this means is that those files and directories that were not visited
>>> during the crawl, will remain untouched and unhealed in this second
>>> iteration of heal, unless you execute a 'heal-full' again.
>>>
>>
>> So should it start healing shards as it crawls or not until after it
>> crawls the entire .shard directory?  At the pace it was going that could be
>> a week with one node appearing in the cluster but with no shard files if
>> anything tries to access a file on that node.  From my experience other day
>> telling it to heal full again did nothing regardless of node used.
>>
>
Crawl is started from '/' of the volume. Whenever self-heal detects during
the crawl that a file or directory is present in some brick(s) and absent
in others, it creates the file on the bricks where it is absent and marks
the fact that the file or directory might need data/entry and metadata heal
too (this also means that an index is created under
.glusterfs/indices/xattrop of the src bricks). And the data/entry and
metadata heal are picked up and done in the background with the help of
these indices.


>>
>>> My suspicion is that this is what happened on your setup. Could you
>>> confirm if that was the case?
>>>
>>
>> Brick was brought online with force start then a full heal launched.
>> Hours later after it became evident that it was not adding new files to
>> heal I did try restarting self-heal daemon and relaunching full heal again.
>> But this was after the heal had basically already failed to work as
>> intended.
>>
>
> OK. How did you figure it was not adding any new files? I need to know
> what places you were monitoring to come to this conclusion.
>
> -Krutika
>
>
>>
>>
>>> As for those logs, I did manager to do something that caused these
>>> warning messages you shared earlier to appear in my client and server logs.
>>> Although these logs are annoying and a bit scary too, they didn't do any
>>> harm to the data in my volume. Why they appear just after a brick is
>>> replaced and under no other circumstances is something I'm still
>>> investigating.
>>>
>>> But for future, it would be good to follow the steps Anuradha gave as
>>> that would allow self-heal to at least detect that it has some repairing to
>>> do whenever it is restarted whether intentionally or otherwise.
>>>
>>
>> I followed those steps as described on my test box and ended up with
>> exact same outcome of adding shards at an agonizing slow pace and no
>> creation of .shard directory or heals on shard directory.  Directories
>> visible from mount healed quickly.  This was with one VM so it has only 800
>> shards as well.  After hours at work it had added a total of 33 shards to
>> be healed.  I sent those logs yesterday as well though not the glustershd.
>>
>> Does replace-brick command copy files in same manner?  For these purposes
>> I am contemplating just skipping the heal route.
>>
>>
>>> -Krutika
>>>
>>> On Tue, Aug 30, 2016 at 2:22 AM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
 attached brick and client logs from test machine where same behavior
 occurred not sure if anything new is there.  its still on 3.8.2

 Number of Bricks: 1 x 3 = 3
 Transport-type: tcp
 Bricks:
 Brick1: 192.168.71.10:/gluster2/brick1/1
 Brick2: 192.168.71.11:/gluster2/brick2/1
 Brick3: 192.168.71.12:/gluster2/brick3/1
 Options Reconfigured:
 cluster.locking-scheme: granular
 performance.strict-o-direct: off
 features.shard-block-size: 64MB
 features.shard: on
 server.allow-insecure: on
 storage.owner-uid: 36
 storage.owner-gid: 36
 cluster.server-quorum-type: server
 cluster.quorum-type: auto
 network.remote-dio: on
 cluster.eager-lock: enable
 performance.stat-prefetch: off
 performance.io-cache: off
 performance.quick-read: off
 cluster.self-heal-window-size: 1024
 cluster.background-self-heal-count: 16
 nfs.enable-ino32: off
 nfs.addr-namelookup: off
 nfs.disable: on
 

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[Krutika Dhananjay]

On Tue, Aug 30, 2016 at 6:07 PM, David Gossage 
wrote:

> On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay 
> wrote:
>
>> Could you also share the glustershd logs?
>>
>
> I'll get them when I get to work sure.
>
>
>>
>> I tried the same steps that you mentioned multiple times, but heal is
>> running to completion without any issues.
>>
>> It must be said that 'heal full' traverses the files and directories in a
>> depth-first order and does heals also in the same order. But if it gets
>> interrupted in the middle (say because self-heal-daemon was either
>> intentionally or unintentionally brought offline and then brought back up),
>> self-heal will only pick up the entries that are so far marked as
>> new-entries that need heal which it will find in indices/xattrop directory.
>> What this means is that those files and directories that were not visited
>> during the crawl, will remain untouched and unhealed in this second
>> iteration of heal, unless you execute a 'heal-full' again.
>>
>
> So should it start healing shards as it crawls or not until after it
> crawls the entire .shard directory?  At the pace it was going that could be
> a week with one node appearing in the cluster but with no shard files if
> anything tries to access a file on that node.  From my experience other day
> telling it to heal full again did nothing regardless of node used.
>
>
>> My suspicion is that this is what happened on your setup. Could you
>> confirm if that was the case?
>>
>
> Brick was brought online with force start then a full heal launched.
> Hours later after it became evident that it was not adding new files to
> heal I did try restarting self-heal daemon and relaunching full heal again.
> But this was after the heal had basically already failed to work as
> intended.
>

OK. How did you figure it was not adding any new files? I need to know what
places you were monitoring to come to this conclusion.

-Krutika


>
>
>> As for those logs, I did manager to do something that caused these
>> warning messages you shared earlier to appear in my client and server logs.
>> Although these logs are annoying and a bit scary too, they didn't do any
>> harm to the data in my volume. Why they appear just after a brick is
>> replaced and under no other circumstances is something I'm still
>> investigating.
>>
>> But for future, it would be good to follow the steps Anuradha gave as
>> that would allow self-heal to at least detect that it has some repairing to
>> do whenever it is restarted whether intentionally or otherwise.
>>
>
> I followed those steps as described on my test box and ended up with exact
> same outcome of adding shards at an agonizing slow pace and no creation of
> .shard directory or heals on shard directory.  Directories visible from
> mount healed quickly.  This was with one VM so it has only 800 shards as
> well.  After hours at work it had added a total of 33 shards to be healed.
> I sent those logs yesterday as well though not the glustershd.
>
> Does replace-brick command copy files in same manner?  For these purposes
> I am contemplating just skipping the heal route.
>
>
>> -Krutika
>>
>> On Tue, Aug 30, 2016 at 2:22 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> attached brick and client logs from test machine where same behavior
>>> occurred not sure if anything new is there.  its still on 3.8.2
>>>
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 192.168.71.10:/gluster2/brick1/1
>>> Brick2: 192.168.71.11:/gluster2/brick2/1
>>> Brick3: 192.168.71.12:/gluster2/brick3/1
>>> Options Reconfigured:
>>> cluster.locking-scheme: granular
>>> performance.strict-o-direct: off
>>> features.shard-block-size: 64MB
>>> features.shard: on
>>> server.allow-insecure: on
>>> storage.owner-uid: 36
>>> storage.owner-gid: 36
>>> cluster.server-quorum-type: server
>>> cluster.quorum-type: auto
>>> network.remote-dio: on
>>> cluster.eager-lock: enable
>>> performance.stat-prefetch: off
>>> performance.io-cache: off
>>> performance.quick-read: off
>>> cluster.self-heal-window-size: 1024
>>> cluster.background-self-heal-count: 16
>>> nfs.enable-ino32: off
>>> nfs.addr-namelookup: off
>>> nfs.disable: on
>>> performance.read-ahead: off
>>> performance.readdir-ahead: on
>>> cluster.granular-entry-heal: on
>>>
>>>
>>>
>>> On Mon, Aug 29, 2016 at 2:20 PM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
 On Mon, Aug 29, 2016 at 7:01 AM, Anuradha Talur 
 wrote:

>
>
> - Original Message -
> > From: "David Gossage" 
> > To: "Anuradha Talur" 
> > Cc: "gluster-users@gluster.org List" ,
> "Krutika Dhananjay" 
> > Sent: Monday, August 29, 2016 5:12:42 PM
> > Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
> >
> > On Mon, Aug 29, 2016 

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay 
wrote:

> Could you also share the glustershd logs?
>

I'll get them when I get to work sure.


>
> I tried the same steps that you mentioned multiple times, but heal is
> running to completion without any issues.
>
> It must be said that 'heal full' traverses the files and directories in a
> depth-first order and does heals also in the same order. But if it gets
> interrupted in the middle (say because self-heal-daemon was either
> intentionally or unintentionally brought offline and then brought back up),
> self-heal will only pick up the entries that are so far marked as
> new-entries that need heal which it will find in indices/xattrop directory.
> What this means is that those files and directories that were not visited
> during the crawl, will remain untouched and unhealed in this second
> iteration of heal, unless you execute a 'heal-full' again.
>

So should it start healing shards as it crawls or not until after it crawls
the entire .shard directory?  At the pace it was going that could be a week
with one node appearing in the cluster but with no shard files if anything
tries to access a file on that node.  From my experience other day telling
it to heal full again did nothing regardless of node used.


> My suspicion is that this is what happened on your setup. Could you
> confirm if that was the case?
>

Brick was brought online with force start then a full heal launched.  Hours
later after it became evident that it was not adding new files to heal I
did try restarting self-heal daemon and relaunching full heal again. But
this was after the heal had basically already failed to work as intended.


> As for those logs, I did manager to do something that caused these warning
> messages you shared earlier to appear in my client and server logs.
> Although these logs are annoying and a bit scary too, they didn't do any
> harm to the data in my volume. Why they appear just after a brick is
> replaced and under no other circumstances is something I'm still
> investigating.
>
> But for future, it would be good to follow the steps Anuradha gave as that
> would allow self-heal to at least detect that it has some repairing to do
> whenever it is restarted whether intentionally or otherwise.
>

I followed those steps as described on my test box and ended up with exact
same outcome of adding shards at an agonizing slow pace and no creation of
.shard directory or heals on shard directory.  Directories visible from
mount healed quickly.  This was with one VM so it has only 800 shards as
well.  After hours at work it had added a total of 33 shards to be healed.
I sent those logs yesterday as well though not the glustershd.

Does replace-brick command copy files in same manner?  For these purposes I
am contemplating just skipping the heal route.


> -Krutika
>
> On Tue, Aug 30, 2016 at 2:22 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> attached brick and client logs from test machine where same behavior
>> occurred not sure if anything new is there.  its still on 3.8.2
>>
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: 192.168.71.10:/gluster2/brick1/1
>> Brick2: 192.168.71.11:/gluster2/brick2/1
>> Brick3: 192.168.71.12:/gluster2/brick3/1
>> Options Reconfigured:
>> cluster.locking-scheme: granular
>> performance.strict-o-direct: off
>> features.shard-block-size: 64MB
>> features.shard: on
>> server.allow-insecure: on
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> cluster.server-quorum-type: server
>> cluster.quorum-type: auto
>> network.remote-dio: on
>> cluster.eager-lock: enable
>> performance.stat-prefetch: off
>> performance.io-cache: off
>> performance.quick-read: off
>> cluster.self-heal-window-size: 1024
>> cluster.background-self-heal-count: 16
>> nfs.enable-ino32: off
>> nfs.addr-namelookup: off
>> nfs.disable: on
>> performance.read-ahead: off
>> performance.readdir-ahead: on
>> cluster.granular-entry-heal: on
>>
>>
>>
>> On Mon, Aug 29, 2016 at 2:20 PM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> On Mon, Aug 29, 2016 at 7:01 AM, Anuradha Talur 
>>> wrote:
>>>


 - Original Message -
 > From: "David Gossage" 
 > To: "Anuradha Talur" 
 > Cc: "gluster-users@gluster.org List" ,
 "Krutika Dhananjay" 
 > Sent: Monday, August 29, 2016 5:12:42 PM
 > Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
 >
 > On Mon, Aug 29, 2016 at 5:39 AM, Anuradha Talur 
 wrote:
 >
 > > Response inline.
 > >
 > > - Original Message -
 > > > From: "Krutika Dhananjay" 
 > > > To: "David Gossage" 
 > > > Cc: "gluster-users@gluster.org List" 
 > > > Sent: Monday, August 29, 

[Gluster-users] Switching bricks

[Kevin Lemonnier]

Hi,

I'm about to bump a 1x3 (replicated) volume up to 2x3, but I just realised the 
3 new servers
are physically in the same datacenter. Is there a safe way to switch a brick 
from the first
replica set with one from the second replica set ?

The only way I see how would be to go down to replica 2, removing a brick from 
the first replica,
then add 2 of the new servers as disperese (at that point the volume would be 
2x2),
then go up to replica 3 adding the third one plus the one I removed earlier.
That should work, right ? There is no other "better" way of doing it ?

Thanks,
-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[Krutika Dhananjay]

Could you also share the glustershd logs?

I tried the same steps that you mentioned multiple times, but heal is
running to completion without any issues.

It must be said that 'heal full' traverses the files and directories in a
depth-first order and does heals also in the same order. But if it gets
interrupted in the middle (say because self-heal-daemon was either
intentionally or unintentionally brought offline and then brought back up),
self-heal will only pick up the entries that are so far marked as
new-entries that need heal which it will find in indices/xattrop directory.
What this means is that those files and directories that were not visited
during the crawl, will remain untouched and unhealed in this second
iteration of heal, unless you execute a 'heal-full' again.

My suspicion is that this is what happened on your setup. Could you confirm
if that was the case?

As for those logs, I did manager to do something that caused these warning
messages you shared earlier to appear in my client and server logs.
Although these logs are annoying and a bit scary too, they didn't do any
harm to the data in my volume. Why they appear just after a brick is
replaced and under no other circumstances is something I'm still
investigating.

But for future, it would be good to follow the steps Anuradha gave as that
would allow self-heal to at least detect that it has some repairing to do
whenever it is restarted whether intentionally or otherwise.

-Krutika

On Tue, Aug 30, 2016 at 2:22 AM, David Gossage 
wrote:

> attached brick and client logs from test machine where same behavior
> occurred not sure if anything new is there.  its still on 3.8.2
>
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.71.10:/gluster2/brick1/1
> Brick2: 192.168.71.11:/gluster2/brick2/1
> Brick3: 192.168.71.12:/gluster2/brick3/1
> Options Reconfigured:
> cluster.locking-scheme: granular
> performance.strict-o-direct: off
> features.shard-block-size: 64MB
> features.shard: on
> server.allow-insecure: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> network.remote-dio: on
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.quick-read: off
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> performance.read-ahead: off
> performance.readdir-ahead: on
> cluster.granular-entry-heal: on
>
>
>
> On Mon, Aug 29, 2016 at 2:20 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Mon, Aug 29, 2016 at 7:01 AM, Anuradha Talur 
>> wrote:
>>
>>>
>>>
>>> - Original Message -
>>> > From: "David Gossage" 
>>> > To: "Anuradha Talur" 
>>> > Cc: "gluster-users@gluster.org List" ,
>>> "Krutika Dhananjay" 
>>> > Sent: Monday, August 29, 2016 5:12:42 PM
>>> > Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
>>> >
>>> > On Mon, Aug 29, 2016 at 5:39 AM, Anuradha Talur 
>>> wrote:
>>> >
>>> > > Response inline.
>>> > >
>>> > > - Original Message -
>>> > > > From: "Krutika Dhananjay" 
>>> > > > To: "David Gossage" 
>>> > > > Cc: "gluster-users@gluster.org List" 
>>> > > > Sent: Monday, August 29, 2016 3:55:04 PM
>>> > > > Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow
>>> > > >
>>> > > > Could you attach both client and brick logs? Meanwhile I will try
>>> these
>>> > > steps
>>> > > > out on my machines and see if it is easily recreatable.
>>> > > >
>>> > > > -Krutika
>>> > > >
>>> > > > On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <
>>> > > dgoss...@carouselchecks.com
>>> > > > > wrote:
>>> > > >
>>> > > >
>>> > > >
>>> > > > Centos 7 Gluster 3.8.3
>>> > > >
>>> > > > Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
>>> > > > Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
>>> > > > Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
>>> > > > Options Reconfigured:
>>> > > > cluster.data-self-heal-algorithm: full
>>> > > > cluster.self-heal-daemon: on
>>> > > > cluster.locking-scheme: granular
>>> > > > features.shard-block-size: 64MB
>>> > > > features.shard: on
>>> > > > performance.readdir-ahead: on
>>> > > > storage.owner-uid: 36
>>> > > > storage.owner-gid: 36
>>> > > > performance.quick-read: off
>>> > > > performance.read-ahead: off
>>> > > > performance.io-cache: off
>>> > > > performance.stat-prefetch: on
>>> > > > cluster.eager-lock: enable
>>> > > > network.remote-dio: enable
>>> > > > cluster.quorum-type: auto
>>> > > > cluster.server-quorum-type: server
>>> > > > server.allow-insecure: on
>>> > > > cluster.self-heal-window-size: 1024
>>> > > > cluster.background-self-heal-count: 16
>>> > > > performance.strict-write-ordering: off

[Gluster-users] Traige meeting

[Ankit Raj]

Hi Gluster team,

The weekly Gluster bug triage is about to take place in 50 min.

Meeting details:
- location: #gluster-meeting on Freenode IRC
( https://webchat.freenode.net/?channels=gluster-meeting )
- date: every Tuesday
- time: 12:00 UTC
(in your terminal, run: date -d "12:00 UTC")
- agenda: https://public.pad.fsfe.org/p/gluster-bug-triage

Currently the following items are listed:
* Roll Call
* Status of last weeks action items
* Group Triage
* Open Floor

Appreciate your participation

Regards,
Ankit Raj
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

[David Gossage]

On Mon, Aug 29, 2016 at 11:25 PM, Darrell Budic 
wrote:

> I noticed that my new brick (replacement disk) did not have a .shard
> directory created on the brick, if that helps.
>
> I removed the affected brick from the volume and then wiped the disk, did
> an add-brick, and everything healed right up. I didn’t try and set any
> attrs or anything else, just removed and added the brick as new.
>

I was considering just using the replace brick command instead.  The use of
heal was basically I was hoping it worked so I could keep brick directory
scheme same as before.  But if the tradeoff is just having brick
directories with slightly different paths, but copy of files succeeding I'd
rather have the node back up,

>
> On Aug 29, 2016, at 9:49 AM, Darrell Budic  wrote:
>
> Just to let you know I’m seeing the same issue under 3.7.14 on CentOS 7.
> Some content was healed correctly, now all the shards are queued up in a
> heal list, but nothing is healing. Got similar brick errors logged to the
> ones David was getting on the brick that isn’t healing:
>
> [2016-08-29 03:31:40.436110] E [MSGID: 115050]
> [server-rpc-fops.c:179:server_lookup_cbk] 0-gv0-rep-server: 1613822:
> LOOKUP (null) 
> (----/0f61bf63-8ef1-4e53-8bc3-6d46590c4fb1.29)
> ==> (Invalid argument) [Invalid argument]
> [2016-08-29 03:31:43.005013] E [MSGID: 115050]
> [server-rpc-fops.c:179:server_lookup_cbk] 0-gv0-rep-server: 1616802:
> LOOKUP (null) 
> (----/0f61bf63-8ef1-4e53-8bc3-6d46590c4fb1.40)
> ==> (Invalid argument) [Invalid argument]
>
> This was after replacing the drive the brick was on and trying to get it
> back into the system by setting the volume's fattr on the brick dir. I’ll
> try the suggested method here on it it shortly.
>
>   -Darrell
>
>
> On Aug 29, 2016, at 7:25 AM, Krutika Dhananjay 
> wrote:
>
> Got it. Thanks.
>
> I tried the same test and shd crashed with SIGABRT (well, that's because I
> compiled from src with -DDEBUG).
> In any case, this error would prevent full heal from proceeding further.
> I'm debugging the crash now. Will let you know when I have the RC.
>
> -Krutika
>
> On Mon, Aug 29, 2016 at 5:47 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>>
>> On Mon, Aug 29, 2016 at 7:14 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> On Mon, Aug 29, 2016 at 5:25 AM, Krutika Dhananjay 
>>> wrote:
>>>
 Could you attach both client and brick logs? Meanwhile I will try these
 steps out on my machines and see if it is easily recreatable.


>>> Hoping 7z files are accepted by mail server.
>>>
>>
>> looks like zip file awaiting approval due to size
>>
>>>
>>> -Krutika

 On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <
 dgoss...@carouselchecks.com> wrote:

> Centos 7 Gluster 3.8.3
>
> Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
> Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
> Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
> Options Reconfigured:
> cluster.data-self-heal-algorithm: full
> cluster.self-heal-daemon: on
> cluster.locking-scheme: granular
> features.shard-block-size: 64MB
> features.shard: on
> performance.readdir-ahead: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: on
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> server.allow-insecure: on
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> performance.strict-write-ordering: off
> nfs.disable: on
> nfs.addr-namelookup: off
> nfs.enable-ino32: off
> cluster.granular-entry-heal: on
>
> Friday did rolling upgrade from 3.8.3->3.8.3 no issues.
> Following steps detailed in previous recommendations began proces of
> replacing and healngbricks one node at a time.
>
> 1) kill pid of brick
> 2) reconfigure brick from raid6 to raid10
> 3) recreate directory of brick
> 4) gluster volume start <> force
> 5) gluster volume heal <> full
>
> 1st node worked as expected took 12 hours to heal 1TB data.  Load was
> little heavy but nothing shocking.
>
> About an hour after node 1 finished I began same process on node2.
> Heal proces kicked in as before and the files in directories visible from
> mount and .glusterfs healed in short time.  Then it began crawl of .shard
> adding those files to heal count at which point the entire proces ground 
> to
> a halt basically.  After 48 hours out of 19k shards it has added 5900 to
> heal list.  Load on all 3 machnes is negligible.   It was suggested to
> change this value to full cluster.data-self-heal-algorithm