Re: [Gluster-users] Rebalancing after adding larger bricks

2016-10-14 Thread Nithya Balachandran 
On 11 October 2016 at 22:32, Jackie Tung  wrote:

> Joe,
>
> Thanks for that, that was educational.  Gluster docs claim that since 3.7,
> DHT hash ranges are weighted based on brick sizes by default:
>
> $ gluster volume get  Option  Value
>
> --  -
>
> cluster.weighted-rebalance  on
>
>
> When running rebalance with force, I see this in the rebalance log:
>
> ...
> [2016-10-11 16:38:37.655144] I [MSGID: 109045]
> [dht-selfheal.c:1751:dht_fix_layout_of_directory] 0-cronut-dht: subvolume
> 10 (cronut-replicate-10): *5721127* chunks
> [2016-10-11 16:38:37.655154] I [MSGID: 109045]
> [dht-selfheal.c:1751:dht_fix_layout_of_directory] 0-cronut-dht: subvolume
> 11 (cronut-replicate-11): *7628846* chunks
> …
>
> subvolume >=11 are 8TB, subvolume <= 10 is are 6TB.
>
> Do you think it is possible to even out usage on all bricks by % utilized
> now?  This would be the case if gluster rebalanced simply by what the
> scaled DHT says, including all required data migrations?
>
>
Can you please send the following:

1. The rebalance logs (/var/log/gluster/-rebalance.log) from each
node
2. The output of the following for the root of each brick:
  getfattr -e hex -m . -d 
3. gluster volume info
4. The version of glusterfs that you are running.
5. gluster volume rebalance  status

Are the file sizes more or less the same or are there large variations in
them?


Thanks,
Nithya


> It would be preferable for us to avoid having to depend on
> cluster.min-free-disk to manage overflow later on - as this introduces one
> extra read of the link followed by the actual IOP.
>
> Thanks,
> Jackie
>
> On Oct 10, 2016, at 11:13 AM, Joe Julian  wrote:
>
> I've written an example of how gluster's dht works on my blog at
> https://joejulian.name/blog/dht-misses-are-expensive/ which might make it
> clear why the end result is not what you expected.
>
> By setting cluster.min-free-disk (defaults to 10%) you can, at least,
> ensure that your new bricks are utilized as needed to prevent over filling
> your smaller bricks.
> On 10/10/2016 10:13 AM, Jackie Tung wrote:
>
> Hi,
>
> We have a 2 node, distributed replicated setup (11 bricks on each node).
> Each of these bricks are 6TB in size.
>
> node_A:/brick1 replicates node_B:/brick1
> node_A:/brick2 replicates node_B:/brick2
> node_A:/brick3 replicates node_B:/brick3
> …
> …
> node_A:/brick11 replicates node_B:/brick11
>
> We recently added 5 more bricks to make it 16 bricks on each node in
> total.  Each of these new bricks are 8TB in size.
>
> We completed a full rebalance operation (status says “completed”).
>
> However the end result is somewhat unexpected:
> */dev/sdl1 7.3T 2.2T 5.2T 29%*
> */dev/sdk1 7.3T 2.0T 5.3T 28%*
> */dev/sdj1 7.3T 2.0T 5.3T 28%*
> */dev/sdn1 7.3T 2.2T 5.2T 30%*
> */dev/sdp1 7.3T 2.2T 5.2T 30%*
> /dev/sdc1 5.5T 2.3T 3.2T 42%
> /dev/sdf1 5.5T 2.3T 3.2T 43%
> /dev/sdo1 5.5T 2.3T 3.2T 42%
> /dev/sda1 5.5T 2.3T 3.2T 43%
> /dev/sdi1 5.5T 2.3T 3.2T 42%
> /dev/sdh1 5.5T 2.3T 3.2T 43%
> /dev/sde1 5.5T 2.3T 3.2T 42%
> /dev/sdb1 5.5T 2.3T 3.2T 42%
> /dev/sdm1 5.5T 2.3T 3.2T 42%
> /dev/sdg1 5.5T 2.3T 3.2T 42%
> /dev/sdd1 5.5T 2.3T 3.2T 42%
>
> The df output in *bold* are the new 8TB drives.
> Was I wrong to expect the % usage to be roughly equal?  Is there some
> parameter I need to tweak to make rebalance account for disk sizes properly?
>
> I’m using Gluster 3.8 on Ubuntu.
>
> Thanks,
> Jackie
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalancing after adding larger bricks

2016-10-11 Thread Jackie Tung 
Joe,

Thanks for that, that was educational.  Gluster docs claim that since 3.7, DHT 
hash ranges are weighted based on brick sizes by default:

$ gluster volume get =11 are 8TB, subvolume <= 10 is are 6TB.

Do you think it is possible to even out usage on all bricks by % utilized now?  
This would be the case if gluster rebalanced simply by what the scaled DHT 
says, including all required data migrations?

It would be preferable for us to avoid having to depend on 
cluster.min-free-disk to manage overflow later on - as this introduces one 
extra read of the link followed by the actual IOP.

Thanks,
Jackie

> On Oct 10, 2016, at 11:13 AM, Joe Julian  wrote:
> 
> I've written an example of how gluster's dht works on my blog at 
> https://joejulian.name/blog/dht-misses-are-expensive/ 
>  which might make it 
> clear why the end result is not what you expected.
> 
> By setting cluster.min-free-disk (defaults to 10%) you can, at least, ensure 
> that your new bricks are utilized as needed to prevent over filling your 
> smaller bricks.
> On 10/10/2016 10:13 AM, Jackie Tung wrote:
>> Hi,
>> 
>> We have a 2 node, distributed replicated setup (11 bricks on each node).  
>> Each of these bricks are 6TB in size.
>> 
>> node_A:/brick1 replicates node_B:/brick1
>> node_A:/brick2 replicates node_B:/brick2
>> node_A:/brick3 replicates node_B:/brick3
>> …
>> …
>> node_A:/brick11 replicates node_B:/brick11
>> 
>> We recently added 5 more bricks to make it 16 bricks on each node in total.  
>> Each of these new bricks are 8TB in size.
>> 
>> We completed a full rebalance operation (status says “completed”).
>> 
>> However the end result is somewhat unexpected:
>> /dev/sdl1 7.3T 2.2T 5.2T 29%
>> /dev/sdk1 7.3T 2.0T 5.3T 28%
>> /dev/sdj1 7.3T 2.0T 5.3T 28%
>> /dev/sdn1 7.3T 2.2T 5.2T 30%
>> /dev/sdp1 7.3T 2.2T 5.2T 30%
>> /dev/sdc1 5.5T 2.3T 3.2T 42%
>> /dev/sdf1 5.5T 2.3T 3.2T 43%
>> /dev/sdo1 5.5T 2.3T 3.2T 42%
>> /dev/sda1 5.5T 2.3T 3.2T 43%
>> /dev/sdi1 5.5T 2.3T 3.2T 42%
>> /dev/sdh1 5.5T 2.3T 3.2T 43%
>> /dev/sde1 5.5T 2.3T 3.2T 42%
>> /dev/sdb1 5.5T 2.3T 3.2T 42%
>> /dev/sdm1 5.5T 2.3T 3.2T 42%
>> /dev/sdg1 5.5T 2.3T 3.2T 42%
>> /dev/sdd1 5.5T 2.3T 3.2T 42%
>> 
>> The df output in bold are the new 8TB drives.
>> Was I wrong to expect the % usage to be roughly equal?  Is there some 
>> parameter I need to tweak to make rebalance account for disk sizes properly?
>> 
>> I’m using Gluster 3.8 on Ubuntu.
>> 
>> Thanks,
>> Jackie
>> 
>> The information in this email is confidential and may be legally privileged. 
>> It is intended solely for the addressee. Access to this email by anyone else 
>> is unauthorized. If you are not the intended recipient, any disclosure, 
>> copying, distribution or any action taken or omitted to be taken in reliance 
>> on it, is prohibited and may be unlawful.
>> 
>> 
>> 
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://www.gluster.org/mailman/listinfo/gluster-users 
>> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users


-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalancing after adding larger bricks

2016-10-10 Thread Joe Julian 
I've written an example of how gluster's dht works on my blog at 
https://joejulian.name/blog/dht-misses-are-expensive/ which might make 
it clear why the end result is not what you expected.


By setting cluster.min-free-disk (defaults to 10%) you can, at least, 
ensure that your new bricks are utilized as needed to prevent over 
filling your smaller bricks.


On 10/10/2016 10:13 AM, Jackie Tung wrote:

Hi,

We have a 2 node, distributed replicated setup (11 bricks on each 
node).  Each of these bricks are 6TB in size.


node_A:/brick1 replicates node_B:/brick1
node_A:/brick2 replicates node_B:/brick2
node_A:/brick3 replicates node_B:/brick3
…
…
node_A:/brick11 replicates node_B:/brick11

We recently added 5 more bricks to make it 16 bricks on each node in 
total.  Each of these new bricks are 8TB in size.


We completed a full rebalance operation (status says “completed”).

However the end result is somewhat unexpected:
*/dev/sdl1 7.3T 2.2T 5.2T 29%*
*/dev/sdk1 7.3T 2.0T 5.3T 28%*
*/dev/sdj1 7.3T 2.0T 5.3T 28%*
*/dev/sdn1 7.3T 2.2T 5.2T 30%*
*/dev/sdp1 7.3T 2.2T 5.2T 30%*
/dev/sdc1 5.5T 2.3T 3.2T 42%
/dev/sdf1 5.5T 2.3T 3.2T 43%
/dev/sdo1 5.5T 2.3T 3.2T 42%
/dev/sda1 5.5T 2.3T 3.2T 43%
/dev/sdi1 5.5T 2.3T 3.2T 42%
/dev/sdh1 5.5T 2.3T 3.2T 43%
/dev/sde1 5.5T 2.3T 3.2T 42%
/dev/sdb1 5.5T 2.3T 3.2T 42%
/dev/sdm1 5.5T 2.3T 3.2T 42%
/dev/sdg1 5.5T 2.3T 3.2T 42%
/dev/sdd1 5.5T 2.3T 3.2T 42%

The df output in *bold* are the new 8TB drives.
Was I wrong to expect the % usage to be roughly equal?  Is there some 
parameter I need to tweak to make rebalance account for disk sizes 
properly?


I’m using Gluster 3.8 on Ubuntu.

Thanks,
Jackie

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this 
email by anyone else is unauthorized. If you are not the intended 
recipient, any disclosure, copying, distribution or any action taken 
or omitted to be taken in reliance on it, is prohibited and may be 
unlawful.




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Rebalancing after adding larger bricks

2016-10-10 Thread Jackie Tung 
Hi,

We have a 2 node, distributed replicated setup (11 bricks on each node).  Each 
of these bricks are 6TB in size.

node_A:/brick1 replicates node_B:/brick1
node_A:/brick2 replicates node_B:/brick2
node_A:/brick3 replicates node_B:/brick3
…
…
node_A:/brick11 replicates node_B:/brick11

We recently added 5 more bricks to make it 16 bricks on each node in total.  
Each of these new bricks are 8TB in size.

We completed a full rebalance operation (status says “completed”).

However the end result is somewhat unexpected:
/dev/sdl1 7.3T 2.2T 5.2T 29%
/dev/sdk1 7.3T 2.0T 5.3T 28%
/dev/sdj1 7.3T 2.0T 5.3T 28%
/dev/sdn1 7.3T 2.2T 5.2T 30%
/dev/sdp1 7.3T 2.2T 5.2T 30%
/dev/sdc1 5.5T 2.3T 3.2T 42%
/dev/sdf1 5.5T 2.3T 3.2T 43%
/dev/sdo1 5.5T 2.3T 3.2T 42%
/dev/sda1 5.5T 2.3T 3.2T 43%
/dev/sdi1 5.5T 2.3T 3.2T 42%
/dev/sdh1 5.5T 2.3T 3.2T 43%
/dev/sde1 5.5T 2.3T 3.2T 42%
/dev/sdb1 5.5T 2.3T 3.2T 42%
/dev/sdm1 5.5T 2.3T 3.2T 42%
/dev/sdg1 5.5T 2.3T 3.2T 42%
/dev/sdd1 5.5T 2.3T 3.2T 42%

The df output in bold are the new 8TB drives.
Was I wrong to expect the % usage to be roughly equal?  Is there some parameter 
I need to tweak to make rebalance account for disk sizes properly?

I’m using Gluster 3.8 on Ubuntu.

Thanks,
Jackie
-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users