Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?

[Serkan Çoban]

If I were you I follow the following steps. Stop the rebalance and fix
the cluster health first.
Bring up the down server, replace server4:brick4 with a new disk,
format and be sure it is started, then start a full heal.
Without all bricks up full heal will not start. The you can continue
with rebalance.


On Fri, Feb 2, 2018 at 1:27 PM, Alessandro Ipe  wrote:
> Hi,
>
>
> I simplified the config in my first email, but I actually have 2x4 servers in 
> replicate-distribute with each 4 bricks for 6 of them and 2 bricks  for the 
> remaining 2. Full healing will just take ages... for a just single brick to 
> resync !
>
>> gluster v status home
> volume status home
> Status of volume: home
> Gluster process TCP Port  RDMA Port  Online  Pid
> --
> Brick server1:/data/glusterfs/home/brick1  49157 0  Y   5003
> Brick server1:/data/glusterfs/home/brick2  49153 0  Y   5023
> Brick server1:/data/glusterfs/home/brick3  49154 0  Y   5004
> Brick server1:/data/glusterfs/home/brick4  49155 0  Y   5011
> Brick server3:/data/glusterfs/home/brick1  49152 0  Y   5422
> Brick server4:/data/glusterfs/home/brick1  49152 0  Y   5019
> Brick server3:/data/glusterfs/home/brick2  49153 0  Y   5429
> Brick server4:/data/glusterfs/home/brick2  49153 0  Y   5033
> Brick server3:/data/glusterfs/home/brick3  49154 0  Y   5437
> Brick server4:/data/glusterfs/home/brick3  49154 0  Y   5026
> Brick server3:/data/glusterfs/home/brick4  49155 0  Y   5444
> Brick server4:/data/glusterfs/home/brick4  N/A   N/AN   N/A
> Brick server5:/data/glusterfs/home/brick1  49152 0  Y   5275
> Brick server6:/data/glusterfs/home/brick1  49152 0  Y   5786
> Brick server5:/data/glusterfs/home/brick2  49153 0  Y   5276
> Brick server6:/data/glusterfs/home/brick2  49153 0  Y   5792
> Brick server5:/data/glusterfs/home/brick3  49154 0  Y   5282
> Brick server6:/data/glusterfs/home/brick3  49154 0  Y   5794
> Brick server5:/data/glusterfs/home/brick4  49155 0  Y   5293
> Brick server6:/data/glusterfs/home/brick4  49155 0  Y   5806
> Brick server7:/data/glusterfs/home/brick1  49156 0  Y   22339
> Brick server8:/data/glusterfs/home/brick1  49153 0  Y   17992
> Brick server7:/data/glusterfs/home/brick2  49157 0  Y   22347
> Brick server8:/data/glusterfs/home/brick2  49154 0  Y   18546
> NFS Server on localhost 2049  0  Y   683
> Self-heal Daemon on localhost   N/A   N/AY   693
> NFS Server on server8  2049  0  Y   18553
> Self-heal Daemon on server8N/A   N/AY   18566
> NFS Server on server5  2049  0  Y   23115
> Self-heal Daemon on server5N/A   N/AY   23121
> NFS Server on server7  2049  0  Y   4201
> Self-heal Daemon on server7N/A   N/AY   4210
> NFS Server on server3  2049  0  Y   5460
> Self-heal Daemon on server3N/A   N/AY   5469
> NFS Server on server6  2049  0  Y   22709
> Self-heal Daemon on server6N/A   N/AY   22718
> NFS Server on server4  2049  0  Y   6044
> Self-heal Daemon on server4N/A   N/AY   6243
>
> server 2 is currently powered off as we are waiting a replacement RAID 
> controller, as well as for
> server4:/data/glusterfs/home/brick4
>
> And as I said, there is a rebalance in progress
>> gluster rebalance home status
> Node Rebalanced-files  size   
> scanned  failures   skipped   status  run time in h:m:s
>-  ---   ---   
> ---   ---   ---  
> --
>localhost4208323.3GB   
> 1568065  1359303734  in progress   16:49:31
> server53569823.8GB   
> 1027934 0240748  in progress   16:49:23
> server43509623.4GB
> 899491 0229064  in progress   16:49:18
> server32703118.0GB
> 701759  

Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?

[Alessandro Ipe]

Hi,


I simplified the config in my first email, but I actually have 2x4 servers in 
replicate-distribute with each 4 bricks for 6 of them and 2 bricks  for the 
remaining 2. Full healing will just take ages... for a just single brick to 
resync !

> gluster v status home
volume status home
Status of volume: home
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick server1:/data/glusterfs/home/brick1  49157 0  Y   5003 
Brick server1:/data/glusterfs/home/brick2  49153 0  Y   5023 
Brick server1:/data/glusterfs/home/brick3  49154 0  Y   5004 
Brick server1:/data/glusterfs/home/brick4  49155 0  Y   5011 
Brick server3:/data/glusterfs/home/brick1  49152 0  Y   5422 
Brick server4:/data/glusterfs/home/brick1  49152 0  Y   5019 
Brick server3:/data/glusterfs/home/brick2  49153 0  Y   5429 
Brick server4:/data/glusterfs/home/brick2  49153 0  Y   5033 
Brick server3:/data/glusterfs/home/brick3  49154 0  Y   5437 
Brick server4:/data/glusterfs/home/brick3  49154 0  Y   5026 
Brick server3:/data/glusterfs/home/brick4  49155 0  Y   5444 
Brick server4:/data/glusterfs/home/brick4  N/A   N/AN   N/A  
Brick server5:/data/glusterfs/home/brick1  49152 0  Y   5275 
Brick server6:/data/glusterfs/home/brick1  49152 0  Y   5786 
Brick server5:/data/glusterfs/home/brick2  49153 0  Y   5276 
Brick server6:/data/glusterfs/home/brick2  49153 0  Y   5792 
Brick server5:/data/glusterfs/home/brick3  49154 0  Y   5282 
Brick server6:/data/glusterfs/home/brick3  49154 0  Y   5794 
Brick server5:/data/glusterfs/home/brick4  49155 0  Y   5293 
Brick server6:/data/glusterfs/home/brick4  49155 0  Y   5806 
Brick server7:/data/glusterfs/home/brick1  49156 0  Y   22339
Brick server8:/data/glusterfs/home/brick1  49153 0  Y   17992
Brick server7:/data/glusterfs/home/brick2  49157 0  Y   22347
Brick server8:/data/glusterfs/home/brick2  49154 0  Y   18546
NFS Server on localhost 2049  0  Y   683  
Self-heal Daemon on localhost   N/A   N/AY   693  
NFS Server on server8  2049  0  Y   18553
Self-heal Daemon on server8N/A   N/AY   18566
NFS Server on server5  2049  0  Y   23115
Self-heal Daemon on server5N/A   N/AY   23121
NFS Server on server7  2049  0  Y   4201 
Self-heal Daemon on server7N/A   N/AY   4210 
NFS Server on server3  2049  0  Y   5460 
Self-heal Daemon on server3N/A   N/AY   5469 
NFS Server on server6  2049  0  Y   22709
Self-heal Daemon on server6N/A   N/AY   22718
NFS Server on server4  2049  0  Y   6044 
Self-heal Daemon on server4N/A   N/AY   6243 

server 2 is currently powered off as we are waiting a replacement RAID 
controller, as well as for 
server4:/data/glusterfs/home/brick4

And as I said, there is a rebalance in progress
> gluster rebalance home status
Node Rebalanced-files  size   
scanned  failures   skipped   status  run time in h:m:s
   -  ---   ---   
---   ---   ---  --
   localhost4208323.3GB   
1568065  1359303734  in progress   16:49:31
server53569823.8GB   
1027934 0240748  in progress   16:49:23
server43509623.4GB
899491 0229064  in progress   16:49:18
server32703118.0GB
701759 8182592  in progress   16:49:27
server800Bytes
327602 0   805  in progress   16:49:18
server63567223.9GB   
1028469 0240810  in progress   16:49:17
server71   45Bytes  
  53 0 0

Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?

[Alessandro Ipe]

Hi,


Thanks. However "gluster v heal volname full" returned the following error 
message
Commit failed on server4. Please check log file for details.

I have checked the log files in /var/log/glusterfs on server4 (by grepping 
heal), but did not get any match. What should I be looking for and in which 
log file, please ?

Note that there is currently a rebalance process running on the volume.


Many thanks,


A. 


On Thursday, 1 February 2018 17:32:19 CET Serkan Çoban wrote:
> You do not need to reset brick if brick path does not change. Replace
> the brick format and mount, then gluster v start volname force.
> To start self heal just run gluster v heal volname full.
> 
> On Thu, Feb 1, 2018 at 6:39 PM, Alessandro Ipe  
wrote:
> > Hi,
> > 
> > 
> > My volume home is configured in replicate mode (version 3.12.4) with the
> > bricks server1:/data/gluster/brick1
> > server2:/data/gluster/brick1
> > 
> > server2:/data/gluster/brick1 was corrupted, so I killed gluster daemon for
> > that brick on server2, umounted it, reformated it, remounted it and did a> 
> >> gluster volume reset-brick home server2:/data/gluster/brick1
> >> server2:/data/gluster/brick1 commit force> 
> > I was expecting that the self-heal daemon would start copying data from
> > server1:/data/gluster/brick1 (about 7.4 TB) to the empty
> > server2:/data/gluster/brick1, which it only did for directories, but not
> > for files.
> > 
> > For the moment, I launched on the fuse mount point
> > 
> >> find . | xargs stat
> > 
> > but crawling the whole volume (100 TB) to trigger self-healing of a single
> > brick of 7.4 TB is unefficient.
> > 
> > Is there any trick to only self-heal a single brick, either by setting
> > some attributes to its top directory, for example ?
> > 
> > 
> > Many thanks,
> > 
> > 
> > Alessandro
> > 
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users


-- 

 Dr. Ir. Alessandro Ipe   
 Department of Observations Tel. +32 2 373 06 31
 Remote Sensing from Space
 Royal Meteorological Institute  
 Avenue Circulaire 3Email:  
 B-1180 BrusselsBelgium alessandro@meteo.be 
 Web: http://gerb.oma.be   



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?

[Serkan Çoban]

You do not need to reset brick if brick path does not change. Replace
the brick format and mount, then gluster v start volname force.
To start self heal just run gluster v heal volname full.

On Thu, Feb 1, 2018 at 6:39 PM, Alessandro Ipe  wrote:
> Hi,
>
>
> My volume home is configured in replicate mode (version 3.12.4) with the 
> bricks
> server1:/data/gluster/brick1
> server2:/data/gluster/brick1
>
> server2:/data/gluster/brick1 was corrupted, so I killed gluster daemon for 
> that brick on server2, umounted it, reformated it, remounted it and did a
>> gluster volume reset-brick home server2:/data/gluster/brick1 
>> server2:/data/gluster/brick1 commit force
>
> I was expecting that the self-heal daemon would start copying data from 
> server1:/data/gluster/brick1
> (about 7.4 TB) to the empty server2:/data/gluster/brick1, which it only did 
> for directories, but not for files.
>
> For the moment, I launched on the fuse mount point
>> find . | xargs stat
> but crawling the whole volume (100 TB) to trigger self-healing of a single 
> brick of 7.4 TB is unefficient.
>
> Is there any trick to only self-heal a single brick, either by setting some 
> attributes to its top directory, for example ?
>
>
> Many thanks,
>
>
> Alessandro
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?

[Alessandro Ipe]

Hi,


My volume home is configured in replicate mode (version 3.12.4) with the bricks
server1:/data/gluster/brick1
server2:/data/gluster/brick1

server2:/data/gluster/brick1 was corrupted, so I killed gluster daemon for that 
brick on server2, umounted it, reformated it, remounted it and did a
> gluster volume reset-brick home server2:/data/gluster/brick1 
> server2:/data/gluster/brick1 commit force

I was expecting that the self-heal daemon would start copying data from 
server1:/data/gluster/brick1 
(about 7.4 TB) to the empty server2:/data/gluster/brick1, which it only did for 
directories, but not for files. 

For the moment, I launched on the fuse mount point
> find . | xargs stat
but crawling the whole volume (100 TB) to trigger self-healing of a single 
brick of 7.4 TB is unefficient.

Is there any trick to only self-heal a single brick, either by setting some 
attributes to its top directory, for example ?


Many thanks,


Alessandro


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users