Re: [Gluster-users] Speed up heal performance

[Pranith Kumar Karampuri]

On 10/14/2015 10:43 PM, Mohamed Pakkeer wrote:

Hi Pranith,

Will this patch improve the heal performance on distributed disperse 
volume?. Currently we are getting 10MB/s heal performance on 10G 
backed network. SHD daemon takes 5 days to complete the heal operation 
for single 4TB( 3.5 TB data) disk failure.
I will try to see if the patch for replication can be generalized for 
disperse volume as well. Keep watching the mailing list for updates :-)


Pranith


Regards,
Backer

On Wed, Oct 14, 2015 at 9:08 PM, Ben Turner <mailto:btur...@redhat.com>> wrote:


- Original Message -
> From: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> To: "Ben Turner" mailto:btur...@redhat.com>>, "Humble Devassy Chirammal"
mailto:humble.deva...@gmail.com>>,
"Atin Mukherjee"
> mailto:atin.mukherje...@gmail.com>>
> Cc: "gluster-users" mailto:gluster-users@gluster.org>>
> Sent: Wednesday, October 14, 2015 1:39:14 AM
> Subject: Re: [Gluster-users] Speed up heal performance
>
>
>
> On 10/13/2015 07:11 PM, Ben Turner wrote:
> > - Original Message -
> >> From: "Humble Devassy Chirammal" mailto:humble.deva...@gmail.com>>
> >> To: "Atin Mukherjee" mailto:atin.mukherje...@gmail.com>>
> >> Cc: "Ben Turner" mailto:btur...@redhat.com>>, "gluster-users"
> >> mailto:gluster-users@gluster.org>>
> >> Sent: Tuesday, October 13, 2015 6:14:46 AM
> >> Subject: Re: [Gluster-users] Speed up heal performance
> >>
> >>> Good news is we already have a WIP patch
review.glusterd.org/10851 <http://review.glusterd.org/10851> to
> >> introduce multi threaded shd. Credits to Richard/Shreyas from
facebook for
> >> this. IIRC, we also have a BZ for the same
> >> Isnt it the same bugzilla (
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737)
mentioned in the
> >> commit log?
> > @Lindsay - No need for a BZ, the above BZ should suffice.
> >
> > @Anyone - In the commit I see:
> >
> >  { .key= "cluster.shd-max-threads",
> >.voltype= "cluster/replicate",
> >.option = "shd-max-threads",
> >.op_version = 1,
> >.flags  = OPT_FLAG_CLIENT_OPT
> >  },
> >  { .key= "cluster.shd-thread-batch-size",
> >.voltype= "cluster/replicate",
> >.option = "shd-thread-batch-size",
> >.op_version = 1,
> >.flags  = OPT_FLAG_CLIENT_OPT
> >  },
> >
> > So we can tune max threads and thread batch size?  I
understand max
> > threads, but what is batch size?  In my testing on 10G NICs
with a backend
> > that will service 10G throughput I see about 1.5 GB per minute
of SH
> > throughput.  To Lindsay's other point, will this patch improve SH
> > throughput?  My systems can write at 1.5 GB / Sec and NICs can
to 1.2 GB /
> > sec but I only see ~1.5 GB per _minute_ of SH throughput.  If
we can not
> > only make SH multi threaded, but improve the performance of a
single
> > thread that would be awesome.  Super bonus points if we can
have some sort
> > of tunible that can limit the bandwidth each thread can
consume.  It would
> > be great to be able to crank things up when the systems aren't
busy and
> > slow things down when load increases.
> This patch is not merged because I thought we needed throttling
feature
> to go in before we can merge this for better control of the
self-heal
> speed. We are doing that for 3.8. So expect to see both of these
for 3.8.

Great news!  You da man Pranith, next time I am on your side of
the world beers are on me :)

-b

>
> Pranith
> >
> > -b
    > >
    > >
> >> --Humble
> >>
> >>
> >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> >> mailto:atin.mukherje...@gmail.com>>
> >> wrote:
> >>
> >>> -Atin
> >>> Sent from one plus one
> >>> On Oct 13, 2015 3:16 AM, "Ben Turner" mailto:btur...@redhat.com>> wrote:
> >>>> - Original Message -
> >>>>> From: "

Re: [Gluster-users] Speed up heal performance

[Mohamed Pakkeer]

Hi Pranith,

Will this patch improve the heal performance on distributed disperse
volume?. Currently we are getting 10MB/s heal performance on 10G backed
network. SHD daemon takes 5 days to complete the heal operation for single
4TB( 3.5 TB data) disk failure.

Regards,
Backer

On Wed, Oct 14, 2015 at 9:08 PM, Ben Turner  wrote:

> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Ben Turner" , "Humble Devassy Chirammal" <
> humble.deva...@gmail.com>, "Atin Mukherjee"
> > 
> > Cc: "gluster-users" 
> > Sent: Wednesday, October 14, 2015 1:39:14 AM
> > Subject: Re: [Gluster-users] Speed up heal performance
> >
> >
> >
> > On 10/13/2015 07:11 PM, Ben Turner wrote:
> > > - Original Message -
> > >> From: "Humble Devassy Chirammal" 
> > >> To: "Atin Mukherjee" 
> > >> Cc: "Ben Turner" , "gluster-users"
> > >> 
> > >> Sent: Tuesday, October 13, 2015 6:14:46 AM
> > >> Subject: Re: [Gluster-users] Speed up heal performance
> > >>
> > >>> Good news is we already have a WIP patch review.glusterd.org/10851
> to
> > >> introduce multi threaded shd. Credits to Richard/Shreyas from
> facebook for
> > >> this. IIRC, we also have a BZ for the same
> > >> Isnt it the same bugzilla (
> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> > >> commit log?
> > > @Lindsay - No need for a BZ, the above BZ should suffice.
> > >
> > > @Anyone - In the commit I see:
> > >
> > >  { .key= "cluster.shd-max-threads",
> > >.voltype= "cluster/replicate",
> > >.option = "shd-max-threads",
> > >.op_version = 1,
> > >.flags  = OPT_FLAG_CLIENT_OPT
> > >  },
> > >  { .key= "cluster.shd-thread-batch-size",
> > >.voltype= "cluster/replicate",
> > >.option = "shd-thread-batch-size",
> > >.op_version = 1,
> > >.flags  = OPT_FLAG_CLIENT_OPT
> > >  },
> > >
> > > So we can tune max threads and thread batch size?  I understand max
> > > threads, but what is batch size?  In my testing on 10G NICs with a
> backend
> > > that will service 10G throughput I see about 1.5 GB per minute of SH
> > > throughput.  To Lindsay's other point, will this patch improve SH
> > > throughput?  My systems can write at 1.5 GB / Sec and NICs can to 1.2
> GB /
> > > sec but I only see ~1.5 GB per _minute_ of SH throughput.  If we can
> not
> > > only make SH multi threaded, but improve the performance of a single
> > > thread that would be awesome.  Super bonus points if we can have some
> sort
> > > of tunible that can limit the bandwidth each thread can consume.  It
> would
> > > be great to be able to crank things up when the systems aren't busy and
> > > slow things down when load increases.
> > This patch is not merged because I thought we needed throttling feature
> > to go in before we can merge this for better control of the self-heal
> > speed. We are doing that for 3.8. So expect to see both of these for 3.8.
>
> Great news!  You da man Pranith, next time I am on your side of the world
> beers are on me :)
>
> -b
>
> >
> > Pranith
> > >
> > > -b
> > >
> > >
> > >> --Humble
> > >>
> > >>
> > >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> > >> 
> > >> wrote:
> > >>
> > >>> -Atin
> > >>> Sent from one plus one
> > >>> On Oct 13, 2015 3:16 AM, "Ben Turner"  wrote:
> > >>>> - Original Message -
> > >>>>> From: "Lindsay Mathieson" 
> > >>>>> To: "gluster-users" 
> > >>>>> Sent: Friday, October 9, 2015 9:18:11 AM
> > >>>>> Subject: [Gluster-users] Speed up heal performance
> > >>>>>
> > >>>>> Is there any way to max out heal performance? My cluster is unused
> > >>> overnight,
> > >>>>> and lightly used at lunchtimes, it would be handy to speed up a
> heal.
> > >>>>>
> > >>>>> The only tuneable I found was clust

Re: [Gluster-users] Speed up heal performance

[Ben Turner]

- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Ben Turner" , "Humble Devassy Chirammal" 
> , "Atin Mukherjee"
> 
> Cc: "gluster-users" 
> Sent: Wednesday, October 14, 2015 1:39:14 AM
> Subject: Re: [Gluster-users] Speed up heal performance
> 
> 
> 
> On 10/13/2015 07:11 PM, Ben Turner wrote:
> > - Original Message -
> >> From: "Humble Devassy Chirammal" 
> >> To: "Atin Mukherjee" 
> >> Cc: "Ben Turner" , "gluster-users"
> >> 
> >> Sent: Tuesday, October 13, 2015 6:14:46 AM
> >> Subject: Re: [Gluster-users] Speed up heal performance
> >>
> >>> Good news is we already have a WIP patch review.glusterd.org/10851 to
> >> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
> >> this. IIRC, we also have a BZ for the same
> >> Isnt it the same bugzilla (
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> >> commit log?
> > @Lindsay - No need for a BZ, the above BZ should suffice.
> >
> > @Anyone - In the commit I see:
> >
> >  { .key= "cluster.shd-max-threads",
> >.voltype= "cluster/replicate",
> >.option = "shd-max-threads",
> >.op_version = 1,
> >.flags  = OPT_FLAG_CLIENT_OPT
> >  },
> >  { .key= "cluster.shd-thread-batch-size",
> >.voltype= "cluster/replicate",
> >.option = "shd-thread-batch-size",
> >.op_version = 1,
> >.flags  = OPT_FLAG_CLIENT_OPT
> >  },
> >
> > So we can tune max threads and thread batch size?  I understand max
> > threads, but what is batch size?  In my testing on 10G NICs with a backend
> > that will service 10G throughput I see about 1.5 GB per minute of SH
> > throughput.  To Lindsay's other point, will this patch improve SH
> > throughput?  My systems can write at 1.5 GB / Sec and NICs can to 1.2 GB /
> > sec but I only see ~1.5 GB per _minute_ of SH throughput.  If we can not
> > only make SH multi threaded, but improve the performance of a single
> > thread that would be awesome.  Super bonus points if we can have some sort
> > of tunible that can limit the bandwidth each thread can consume.  It would
> > be great to be able to crank things up when the systems aren't busy and
> > slow things down when load increases.
> This patch is not merged because I thought we needed throttling feature
> to go in before we can merge this for better control of the self-heal
> speed. We are doing that for 3.8. So expect to see both of these for 3.8.

Great news!  You da man Pranith, next time I am on your side of the world beers 
are on me :)

-b

> 
> Pranith
> >
> > -b
> >
> >
> >> --Humble
> >>
> >>
> >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> >> 
> >> wrote:
> >>
> >>> -Atin
> >>> Sent from one plus one
> >>> On Oct 13, 2015 3:16 AM, "Ben Turner"  wrote:
> >>>> - Original Message -
> >>>>> From: "Lindsay Mathieson" 
> >>>>> To: "gluster-users" 
> >>>>> Sent: Friday, October 9, 2015 9:18:11 AM
> >>>>> Subject: [Gluster-users] Speed up heal performance
> >>>>>
> >>>>> Is there any way to max out heal performance? My cluster is unused
> >>> overnight,
> >>>>> and lightly used at lunchtimes, it would be handy to speed up a heal.
> >>>>>
> >>>>> The only tuneable I found was cluster.self-heal-window-size, which
> >>> doesn't
> >>>>> seem to make much difference.
> >>>> I don't know of any way to speed this up, maybe someone else could chime
> >>> in here that knows the heal daemon better than me.  Maybe you could open
> >>> an
> >>> RFE on this?  In my testing I only see 2 files getting healed at a time
> >>> per
> >>> replica pair.  I would like to see this be multi threaded(if its not
> >>> already) with the ability to tune it to control resource usage(similar to
> >>> what we did in the rebalance refactoring done recently).  If you let me
> >>> know the BZ # I'll add my data + suggestions, I have been testing this
> >>> pretty extensively

Re: [Gluster-users] Speed up heal performance

[Pranith Kumar Karampuri]

On 10/13/2015 07:11 PM, Ben Turner wrote:

- Original Message -

From: "Humble Devassy Chirammal" 
To: "Atin Mukherjee" 
Cc: "Ben Turner" , "gluster-users" 

Sent: Tuesday, October 13, 2015 6:14:46 AM
Subject: Re: [Gluster-users] Speed up heal performance


Good news is we already have a WIP patch review.glusterd.org/10851 to

introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
this. IIRC, we also have a BZ for the same
Isnt it the same bugzilla (
https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
commit log?

@Lindsay - No need for a BZ, the above BZ should suffice.

@Anyone - In the commit I see:

 { .key= "cluster.shd-max-threads",
   .voltype= "cluster/replicate",
   .option = "shd-max-threads",
   .op_version = 1,
   .flags  = OPT_FLAG_CLIENT_OPT
 },
 { .key= "cluster.shd-thread-batch-size",
   .voltype= "cluster/replicate",
   .option = "shd-thread-batch-size",
   .op_version = 1,
   .flags  = OPT_FLAG_CLIENT_OPT
 },

So we can tune max threads and thread batch size?  I understand max threads, 
but what is batch size?  In my testing on 10G NICs with a backend that will 
service 10G throughput I see about 1.5 GB per minute of SH throughput.  To 
Lindsay's other point, will this patch improve SH throughput?  My systems can 
write at 1.5 GB / Sec and NICs can to 1.2 GB / sec but I only see ~1.5 GB per 
_minute_ of SH throughput.  If we can not only make SH multi threaded, but 
improve the performance of a single thread that would be awesome.  Super bonus 
points if we can have some sort of tunible that can limit the bandwidth each 
thread can consume.  It would be great to be able to crank things up when the 
systems aren't busy and slow things down when load increases.
This patch is not merged because I thought we needed throttling feature 
to go in before we can merge this for better control of the self-heal 
speed. We are doing that for 3.8. So expect to see both of these for 3.8.


Pranith


-b



--Humble


On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee 
wrote:


-Atin
Sent from one plus one
On Oct 13, 2015 3:16 AM, "Ben Turner"  wrote:

- Original Message -

From: "Lindsay Mathieson" 
To: "gluster-users" 
Sent: Friday, October 9, 2015 9:18:11 AM
Subject: [Gluster-users] Speed up heal performance

Is there any way to max out heal performance? My cluster is unused

overnight,

and lightly used at lunchtimes, it would be handy to speed up a heal.

The only tuneable I found was cluster.self-heal-window-size, which

doesn't

seem to make much difference.

I don't know of any way to speed this up, maybe someone else could chime

in here that knows the heal daemon better than me.  Maybe you could open an
RFE on this?  In my testing I only see 2 files getting healed at a time per
replica pair.  I would like to see this be multi threaded(if its not
already) with the ability to tune it to control resource usage(similar to
what we did in the rebalance refactoring done recently).  If you let me
know the BZ # I'll add my data + suggestions, I have been testing this
pretty extensively in recent weeks and good data + some ideas on how to
speed things up.
Good news is we already have a WIP patch review.glusterd.org/10851 to
introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
this. IIRC, we also have a BZ for the same but the patch is in rfc as of
now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct
me otherwise.

-b


thanks,
--
Lindsay

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Ben Turner]

- Original Message -
> From: "Humble Devassy Chirammal" 
> To: "Atin Mukherjee" 
> Cc: "Ben Turner" , "gluster-users" 
> 
> Sent: Tuesday, October 13, 2015 6:14:46 AM
> Subject: Re: [Gluster-users] Speed up heal performance
> 
> >Good news is we already have a WIP patch review.glusterd.org/10851 to
> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
> this. IIRC, we also have a BZ for the same
> >
> Isnt it the same bugzilla (
> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> commit log?

@Lindsay - No need for a BZ, the above BZ should suffice.

@Anyone - In the commit I see:

{ .key= "cluster.shd-max-threads",
  .voltype= "cluster/replicate",
  .option = "shd-max-threads",
  .op_version = 1,
  .flags  = OPT_FLAG_CLIENT_OPT
},
{ .key= "cluster.shd-thread-batch-size",
  .voltype= "cluster/replicate",
  .option = "shd-thread-batch-size",
  .op_version = 1,
  .flags  = OPT_FLAG_CLIENT_OPT
},

So we can tune max threads and thread batch size?  I understand max threads, 
but what is batch size?  In my testing on 10G NICs with a backend that will 
service 10G throughput I see about 1.5 GB per minute of SH throughput.  To 
Lindsay's other point, will this patch improve SH throughput?  My systems can 
write at 1.5 GB / Sec and NICs can to 1.2 GB / sec but I only see ~1.5 GB per 
_minute_ of SH throughput.  If we can not only make SH multi threaded, but 
improve the performance of a single thread that would be awesome.  Super bonus 
points if we can have some sort of tunible that can limit the bandwidth each 
thread can consume.  It would be great to be able to crank things up when the 
systems aren't busy and slow things down when load increases.

-b


> 
> --Humble
> 
> 
> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee 
> wrote:
> 
> > -Atin
> > Sent from one plus one
> > On Oct 13, 2015 3:16 AM, "Ben Turner"  wrote:
> > >
> > > - Original Message -
> > > > From: "Lindsay Mathieson" 
> > > > To: "gluster-users" 
> > > > Sent: Friday, October 9, 2015 9:18:11 AM
> > > > Subject: [Gluster-users] Speed up heal performance
> > > >
> > > > Is there any way to max out heal performance? My cluster is unused
> > overnight,
> > > > and lightly used at lunchtimes, it would be handy to speed up a heal.
> > > >
> > > > The only tuneable I found was cluster.self-heal-window-size, which
> > doesn't
> > > > seem to make much difference.
> > >
> > > I don't know of any way to speed this up, maybe someone else could chime
> > in here that knows the heal daemon better than me.  Maybe you could open an
> > RFE on this?  In my testing I only see 2 files getting healed at a time per
> > replica pair.  I would like to see this be multi threaded(if its not
> > already) with the ability to tune it to control resource usage(similar to
> > what we did in the rebalance refactoring done recently).  If you let me
> > know the BZ # I'll add my data + suggestions, I have been testing this
> > pretty extensively in recent weeks and good data + some ideas on how to
> > speed things up.
> > Good news is we already have a WIP patch review.glusterd.org/10851 to
> > introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
> > this. IIRC, we also have a BZ for the same but the patch is in rfc as of
> > now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct
> > me otherwise.
> > >
> > > -b
> > >
> > > > thanks,
> > > > --
> > > > Lindsay
> > > >
> > > > ___
> > > > Gluster-users mailing list
> > > > Gluster-users@gluster.org
> > > > http://www.gluster.org/mailman/listinfo/gluster-users
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Lindsay Mathieson]

On 13 October 2015 at 20:14, Humble Devassy Chirammal <
humble.deva...@gmail.com> wrote:

> >Good news is we already have a WIP patch review.glusterd.org/10851 to
> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
> this. IIRC, we also have a BZ for the same
> >
> Isnt it the same bugzilla (
> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> commit log?


I'm confused, the bug report states that its already in 3.6? what settings
would control the self heal threads?


-- 
Lindsay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Atin Mukherjee]

On 10/13/2015 03:44 PM, Humble Devassy Chirammal wrote:
>>Good news is we already have a WIP patch review.glusterd.org/10851
> <http://review.glusterd.org/10851> to introduce multi threaded shd.
> Credits to Richard/Shreyas from facebook for this. IIRC, we also have a
> BZ for the same
>>
> Isnt it the same bugzilla
> (https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> commit log? 
It is, just that I was too lazy to scroll down for the bug as topic
would have been also ideally be the same (bug-XXX)
> 
> --Humble
> 
> 
> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> mailto:atin.mukherje...@gmail.com>> wrote:
> 
> -Atin
> Sent from one plus one
> On Oct 13, 2015 3:16 AM, "Ben Turner"  <mailto:btur...@redhat.com>> wrote:
> >
> > - Original Message -
> > > From: "Lindsay Mathieson"  <mailto:lindsay.mathie...@gmail.com>>
>     > > To: "gluster-users"  <mailto:gluster-users@gluster.org>>
> > > Sent: Friday, October 9, 2015 9:18:11 AM
> > > Subject: [Gluster-users] Speed up heal performance
> > >
> > > Is there any way to max out heal performance? My cluster is unused 
> overnight,
> > > and lightly used at lunchtimes, it would be handy to speed up a heal.
> > >
> > > The only tuneable I found was cluster.self-heal-window-size, which 
> doesn't
> > > seem to make much difference.
> >
> > I don't know of any way to speed this up, maybe someone else could 
> chime in here that knows the heal daemon better than me.  Maybe you could 
> open an RFE on this?  In my testing I only see 2 files getting healed at a 
> time per replica pair.  I would like to see this be multi threaded(if its not 
> already) with the ability to tune it to control resource usage(similar to 
> what we did in the rebalance refactoring done recently).  If you let me know 
> the BZ # I'll add my data + suggestions, I have been testing this pretty 
> extensively in recent weeks and good data + some ideas on how to speed things 
> up.
> Good news is we already have a WIP patch review.glusterd.org/10851
> <http://review.glusterd.org/10851> to introduce multi threaded shd.
> Credits to Richard/Shreyas from facebook for this. IIRC, we also
> have a BZ for the same but the patch is in rfc as of now. AFAIK,
> this is a candidate to land in 3.8 as well, Vijay can correct me
> otherwise.
> >
> > -b
> >
> > > thanks,
> > > --
> > > Lindsay
> > >
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
> > http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Humble Devassy Chirammal]

>Good news is we already have a WIP patch review.glusterd.org/10851 to
introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
this. IIRC, we also have a BZ for the same
>
Isnt it the same bugzilla (
https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
commit log?

--Humble


On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee 
wrote:

> -Atin
> Sent from one plus one
> On Oct 13, 2015 3:16 AM, "Ben Turner"  wrote:
> >
> > - Original Message -
> > > From: "Lindsay Mathieson" 
> > > To: "gluster-users" 
> > > Sent: Friday, October 9, 2015 9:18:11 AM
> > > Subject: [Gluster-users] Speed up heal performance
> > >
> > > Is there any way to max out heal performance? My cluster is unused
> overnight,
> > > and lightly used at lunchtimes, it would be handy to speed up a heal.
> > >
> > > The only tuneable I found was cluster.self-heal-window-size, which
> doesn't
> > > seem to make much difference.
> >
> > I don't know of any way to speed this up, maybe someone else could chime
> in here that knows the heal daemon better than me.  Maybe you could open an
> RFE on this?  In my testing I only see 2 files getting healed at a time per
> replica pair.  I would like to see this be multi threaded(if its not
> already) with the ability to tune it to control resource usage(similar to
> what we did in the rebalance refactoring done recently).  If you let me
> know the BZ # I'll add my data + suggestions, I have been testing this
> pretty extensively in recent weeks and good data + some ideas on how to
> speed things up.
> Good news is we already have a WIP patch review.glusterd.org/10851 to
> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
> this. IIRC, we also have a BZ for the same but the patch is in rfc as of
> now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct
> me otherwise.
> >
> > -b
> >
> > > thanks,
> > > --
> > > Lindsay
> > >
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Vijay Bellur]

On Tuesday 13 October 2015 09:03 AM, Atin Mukherjee wrote:



On 10/13/2015 08:13 AM, Lindsay Mathieson wrote:


On 13 October 2015 at 11:56, Atin Mukherjee mailto:atin.mukherje...@gmail.com>> wrote:

 Good news is we already have a WIP patch review.glusterd.org/10851
  to introduce multi threaded shd.


Thanks Atin, no need for the RFE then?

https://bugzilla.redhat.com/show_bug.cgi?id=1221737 - as I mentioned
earlier it was already opened.


I notice the patch mainly discusses improvingthe performanceof healing
many small files, will it help with large VM files as well?

AFAIK, it would help large files as well, but you would see a
significant improvement specially for small files IMO. AFR team can
correct me otherwise.



[1] has details about the improvements being planned for both data and 
entry self-heals. While data self-healing improvements mostly help large 
files, granular entry self-heal can help when you have a large number of 
files in a single directory (typically seen with small files).


multi-threaded self-heal daemon should help both small & large files by 
parallelizing the number of heals.


Regards,
Vijay


[1] 
http://review.gluster.org/#/c/12257/1/in_progress/afr-self-heal-improvements.md



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Atin Mukherjee]

On 10/13/2015 08:13 AM, Lindsay Mathieson wrote:
> 
> On 13 October 2015 at 11:56, Atin Mukherjee  > wrote:
> 
> Good news is we already have a WIP patch review.glusterd.org/10851
>  to introduce multi threaded shd.
> 
> 
> Thanks Atin, no need for the RFE then?
https://bugzilla.redhat.com/show_bug.cgi?id=1221737 - as I mentioned
earlier it was already opened.
> 
> I notice the patch mainly discusses improvingthe performanceof healing
> many small files, will it help with large VM files as well?
AFAIK, it would help large files as well, but you would see a
significant improvement specially for small files IMO. AFR team can
correct me otherwise.

> 
> 
> -- 
> Lindsay
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Lindsay Mathieson]

On 13 October 2015 at 11:56, Atin Mukherjee 
wrote:

> Good news is we already have a WIP patch review.glusterd.org/10851 to
> introduce multi threaded shd.
>

Thanks Atin, no need for the RFE then?

I notice the patch mainly discusses improvingthe performanceof healing many
small files, will it help with large VM files as well?


-- 
Lindsay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Atin Mukherjee]

-Atin
Sent from one plus one
On Oct 13, 2015 3:16 AM, "Ben Turner"  wrote:
>
> - Original Message -
> > From: "Lindsay Mathieson" 
> > To: "gluster-users" 
> > Sent: Friday, October 9, 2015 9:18:11 AM
> > Subject: [Gluster-users] Speed up heal performance
> >
> > Is there any way to max out heal performance? My cluster is unused
overnight,
> > and lightly used at lunchtimes, it would be handy to speed up a heal.
> >
> > The only tuneable I found was cluster.self-heal-window-size, which
doesn't
> > seem to make much difference.
>
> I don't know of any way to speed this up, maybe someone else could chime
in here that knows the heal daemon better than me.  Maybe you could open an
RFE on this?  In my testing I only see 2 files getting healed at a time per
replica pair.  I would like to see this be multi threaded(if its not
already) with the ability to tune it to control resource usage(similar to
what we did in the rebalance refactoring done recently).  If you let me
know the BZ # I'll add my data + suggestions, I have been testing this
pretty extensively in recent weeks and good data + some ideas on how to
speed things up.
Good news is we already have a WIP patch review.glusterd.org/10851 to
introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
this. IIRC, we also have a BZ for the same but the patch is in rfc as of
now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct
me otherwise.
>
> -b
>
> > thanks,
> > --
> > Lindsay
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed up heal performance

[Ben Turner]

- Original Message -
> From: "Lindsay Mathieson" 
> To: "gluster-users" 
> Sent: Friday, October 9, 2015 9:18:11 AM
> Subject: [Gluster-users] Speed up heal performance
> 
> Is there any way to max out heal performance? My cluster is unused overnight,
> and lightly used at lunchtimes, it would be handy to speed up a heal.
> 
> The only tuneable I found was cluster.self-heal-window-size, which doesn't
> seem to make much difference.

I don't know of any way to speed this up, maybe someone else could chime in 
here that knows the heal daemon better than me.  Maybe you could open an RFE on 
this?  In my testing I only see 2 files getting healed at a time per replica 
pair.  I would like to see this be multi threaded(if its not already) with the 
ability to tune it to control resource usage(similar to what we did in the 
rebalance refactoring done recently).  If you let me know the BZ # I'll add my 
data + suggestions, I have been testing this pretty extensively in recent weeks 
and good data + some ideas on how to speed things up.

-b
 
> thanks,
> --
> Lindsay
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Speed up heal performance

[Lindsay Mathieson]

Is there any way to max out heal performance? My cluster is unused
overnight, and lightly used at lunchtimes, it would be handy to speed up a
heal.

The only tuneable I found was cluster.self-heal-window-size, which doesn't
seem to make much difference.

thanks,
-- 
Lindsay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users