On Mon, Jun 27, 2016 at 2:38 PM, Manoj Pillai <mpil...@redhat.com> wrote:
> > > ----- Original Message ----- > > From: "Raghavendra Gowdappa" <rgowd...@redhat.com> > > To: "Pranith Kumar Karampuri" <pkara...@redhat.com> > > Cc: "Gluster Devel" <gluster-devel@gluster.org> > > Sent: Monday, June 27, 2016 12:48:49 PM > > Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing > > > > > > > > ----- Original Message ----- > > > From: "Pranith Kumar Karampuri" <pkara...@redhat.com> > > > To: "Xavier Hernandez" <xhernan...@datalab.es> > > > Cc: "Gluster Devel" <gluster-devel@gluster.org> > > > Sent: Monday, June 27, 2016 12:42:35 PM > > > Subject: Re: [Gluster-devel] performance issues Manoj found in EC > testing > > > > > > > > > > > > On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez < > xhernan...@datalab.es > > > > > > > wrote: > > > > > > > > > Hi Manoj, > > > > > > I always enable client-io-threads option for disperse volumes. It > improves > > > performance sensibly, most probably because of the problem you have > > > detected. > > > > > > I don't see any other way to solve that problem. > > > > > > I agree. Updated the bug with same info. > > > > > > > > > > > > I think it would be a lot better to have a true thread pool (and maybe > an > > > I/O > > > thread pool shared by fuse, client and server xlators) in libglusterfs > > > instead of the io-threads xlator. This would allow each xlator to > decide > > > when and what should be parallelized in a more intelligent way, since > > > basing > > > the decision solely on the fop type seems too simplistic to me. > > > > > > In the specific case of EC, there are a lot of operations to perform > for a > > > single high level fop, and not all of them require the same priority. > Also > > > some of them could be executed in parallel instead of sequentially. > > > > > > I think it is high time we actually schedule(for which release) to get > this > > > in gluster. May be you should send out a doc where we can work out > details? > > > I will be happy to explore options to integrate io-threads, > syncop/barrier > > > with this infra based on the design may be. > > > > +1. I can volunteer too. > > Thanks, folks! As a quick update, throughput on a single client test jumped > from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is > now more in line with what is expected for this workload based on > back-of-the-envelope calculations. > > Are there any reservations about recommending client-io-threads=on as > "default" tuning, until the enhancement discussed above becomes reality? > The only thing I can think of is possible races we may have to address after enabling this option. So I would let it bake on master for a while with this as default may be? > -- Manoj > > > > > > > > > > > > > > > Xavi > > > > > > > > > On 25/06/16 19:42, Manoj Pillai wrote: > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com > > > > To: "Xavier Hernandez" < xhernan...@datalab.es > > > > Cc: "Manoj Pillai" < mpil...@redhat.com >, "Gluster Devel" < > > > gluster-devel@gluster.org > > > > Sent: Thursday, June 23, 2016 8:50:44 PM > > > Subject: performance issues Manoj found in EC testing > > > > > > hi Xavi, > > > Meet Manoj from performance team Redhat. He has been testing EC > > > performance in his stretch clusters. He found some interesting things > we > > > would like to share with you. > > > > > > 1) When we perform multiple streams of big file writes(12 parallel dds > I > > > think) he found one thread to be always hot (99%CPU always). He was > asking > > > me if fuse_reader thread does any extra processing in EC compared to > > > replicate. Initially I thought it would just lock and epoll threads > will > > > perform the encoding but later realized that once we have the lock and > > > version details, next writes on the file would be encoded in the same > > > thread that comes to EC. write-behind could play a role and make the > writes > > > come to EC in an epoll thread but we saw consistently there was just > one > > > thread that is hot. Not multiple threads. We will be able to confirm > this > > > in tomorrow's testing. > > > > > > 2) This is one more thing Raghavendra G found, that our current > > > implementation of epoll doesn't let other epoll threads pick messages > from > > > a socket while one thread is processing one message from that socket. > In > > > EC's case that can be encoding of the write/decoding read. This will > not > > > let replies of operations on different files to be processed in > parallel. > > > He thinks this can be fixed for 3.9. > > > > > > Manoj will be raising a bug to gather all his findings. I just wanted > to > > > introduce him and let you know the interesting things he is finding > before > > > you see the bug :-). > > > -- > > > Pranith > > > > > > Thanks, Pranith :). > > > > > > Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953 > > > > > > Comparing EC and replica-2 runs, the hot thread is seen in both cases, > so > > > I have not opened this as an EC bug. But initial impression is that > > > performance impact for EC is particularly bad (details in the bug). > > > > > > -- Manoj > > > > > > > > > > > > > > > -- > > > Pranith > > > > > > _______________________________________________ > > > Gluster-devel mailing list > > > Gluster-devel@gluster.org > > > http://www.gluster.org/mailman/listinfo/gluster-devel > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-devel > > > -- Pranith
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel