Re: [Nfs-ganesha-devel] dispatch queues

2017-03-10 Thread William Allen Simpson 
On 3/9/17 10:22 AM, Matt Benjamin wrote:
>> From: "William Allen Simpson" 
>> After a somewhat loud discussion with Matt, we've agreed on a
>> different approach.  This will also be useful for fully async IO
>> that is planned for V2.6.
>
> Um, I don't think this statement represents either of the two internal 
> meetings we had accurately, but ok.
>
I think it does.  Please be more specific.  Or not, because these
offhand comments without substance are confusing to everybody on
this very public mailing list.


>> The sequential IO reports were from specific .
>
> For posterity, the feedback I've seen regarding sequential i/o was provided 
> by upstream folks on our regular concall. nobody uses the term "customer" on 
> this list, for obvious reasons.
>
The mailing list is the best place for these bug report discussions.
Also, IRC is totally worthless for recording discussions.

Because I've got an overlapping meeting on the first half hour, and
another on the second half hour, I don't attend many of the concalls
these days unless I've got something specific to report.

This "new" (since Thanksgiving 2015 anyway) time slot has never been
good for me.  I'd much prefer that it be moved to Wednesdays.  We don't
seem to get new dev releases until late Friday anyway.

I've always treated NFS as an IETF activity and follow IETF rules.
Unlike IEEE or CTIA (where it is against the rules), in IETF we always
talk about implementation, deployment, and customers quite specifically,
and confirm reported problems.  And success in fixing them.


> Well, as I think we've all agreed, nothing like this is going into 2.5.  
> Anything that DOES make it to the nfs-ganesha upstream is going to need to be 
> well motivated, well measured, and matured.
>
I thought you wanted my performance patches now.  Doing nothing at
this time is much easier

This particular patch was written for the second or third time back
2015, so it feels "mature" to me.  But then we are finally getting my
removal of the gsh_rpc.h [R|S]LOCK in this week, and I've got variants
dated Aug 3, 2015, and Sep 2, 2015, in my old-patches folder.

So, baby steps.  At least that gets rid of 3 layers of locks, so it
should be very helpful.

Since I've known about the queuing problem since the first time that
I'd looked at the code 3'ish years ago, and folks have customers who
are complaining, I'd thought this was an opportune time.

For RDMA, I simply bypassed it.

It is my considered opinion that fixing the rampant alloc/free, queuing,
and thread switches will do much more for TCP performance than async IO
that merely gives us a few percentage improvement at best.


>> To really do a good job, we need some kind of FSAL feedback API.
>> I'm going to ask the Gluster folks for some help on designing it, so
>> that we have a good use case and testing infrastructure.  But we'll
>> post the design iterations here in the same fashion as an IETF
>> Working Group, so that maybe we can get other FSAL feedback, too.
>>
>> Is anybody specifically interested in helping design the API?
>
> As the proposer of this idea, I'm interested in seeing experimental 
> prototypes that help us establish and refine something that works.  Let's 
> post running code, and then write specs.
>
API first to inform the design, then prototypes, then running code.

Oh, and for the record, you proposed that you would write up an API
over the Thanksgiving holidays, then the Christmas holidays.

Not seeing it, now I'm soliciting help from everybody.


> That said, upstream participation is welcome.
>
We're upstream here.


--
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] dispatch queues

2017-03-09 Thread Matt Benjamin 
Hi Bill,

- Original Message -
> From: "William Allen Simpson" <william.allen.simp...@gmail.com>
> To: "NFS Ganesha Developers" <nfs-ganesha-devel@lists.sourceforge.net>
> Sent: Thursday, March 9, 2017 2:44:29 AM
> Subject: Re: [Nfs-ganesha-devel] dispatch queues
> 
> On 3/8/17 5:34 AM, William Allen Simpson wrote:
> > Ganesha currently has 2 phases of dispatch queuing: one for input and
> > decoding, then another for executing/encoding output.  (I've fixed the
> > third queue for later sending the output, where the thread should stay
> > hot as long as there's more to process.)
> >
> > On Monday, Matt told me we were having problems with sequential IO.
> > Wishing that somebody had mentioned this to me sooner.
> >
> > [...]
> >
> After a somewhat loud discussion with Matt, we've agreed on a
> different approach.  This will also be useful for fully async IO
> that is planned for V2.6.

Um, I don't think this statement represents either of the two internal meetings 
we had accurately, but ok.

> 
> The sequential IO reports were from specific .

For posterity, the feedback I've seen regarding sequential i/o was provided by 
upstream folks on our regular concall. nobody uses the term "customer" on this 
list, for obvious reasons.

> 
> I'm going to code something more like Weighted Fair Queuing that
> I've mentioned on this list back in June 2015.  The only weight is
> that we want any initial TCP connection to be handled as rapidly as
> possible to get the standard TCP slow start moving.
> 
> Are there other priorities that we should handle?
> 
> Otherwise, we really need a more even handed approach across large
> numbers of clients, keeping each client's requests in strict order,
> even though some of them could be "faster" than others.  The fair
> queuing should also help prevent traffic spikes.
> 
> I think I can have something coded by next week.  I'd already done
> some preliminary work in 2015.  But the time constraint means this
> will be pretty bare bones for V2.5.

Well, as I think we've all agreed, nothing like this is going into 2.5.  
Anything that DOES make it to the nfs-ganesha upstream is going to need to be 
well motivated, well measured, and matured.

> 
> To really do a good job, we need some kind of FSAL feedback API.
> I'm going to ask the Gluster folks for some help on designing it, so
> that we have a good use case and testing infrastructure.  But we'll
> post the design iterations here in the same fashion as an IETF
> Working Group, so that maybe we can get other FSAL feedback, too.
> 
> Is anybody specifically interested in helping design the API?

As the proposer of this idea, I'm interested in seeing experimental prototypes 
that help us establish and refine something that works.  Let's post running 
code, and then write specs.

That said, upstream participation is welcome.

Matt

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309

--
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] dispatch queues

2017-03-09 Thread Daniel Gryniewicz 
On 03/09/2017 02:44 AM, William Allen Simpson wrote:
> On 3/8/17 5:34 AM, William Allen Simpson wrote:
>> Ganesha currently has 2 phases of dispatch queuing: one for input and
>> decoding, then another for executing/encoding output.  (I've fixed the
>> third queue for later sending the output, where the thread should stay
>> hot as long as there's more to process.)
>>
>> On Monday, Matt told me we were having problems with sequential IO.
>> Wishing that somebody had mentioned this to me sooner.
>>
>> [...]
>>
> After a somewhat loud discussion with Matt, we've agreed on a
> different approach.  This will also be useful for fully async IO
> that is planned for V2.6.
>
> The sequential IO reports were from specific customers.  Since I'm
> not in direct contact with customers I'm not hearing about it.  But
> we should be able to get field confirmation about whether the new
> approach is working better
>
> I'm going to code something more like Weighted Fair Queuing that
> I've mentioned on this list back in June 2015.  The only weight is
> that we want any initial TCP connection to be handled as rapidly as
> possible to get the standard TCP slow start moving.
>
> Are there other priorities that we should handle?

How does WFQ get us closer to Async I/O?  I'm not seeing the connection, 
so could you elaborate on this?

> Otherwise, we really need a more even handed approach across large
> numbers of clients, keeping each client's requests in strict order,
> even though some of them could be "faster" than others.  The fair
> queuing should also help prevent traffic spikes.
>
> I think I can have something coded by next week.  I'd already done
> some preliminary work in 2015.  But the time constraint means this
> will be pretty bare bones for V2.5.

I disagree with this.  Nothing as invasive as queuing changes should go 
into 2.5.  2.5 should get the dirent chunking and bug fixes only at this 
point.

Daniel

--
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] dispatch queues

2017-03-08 Thread William Allen Simpson 
On 3/8/17 5:34 AM, William Allen Simpson wrote:
> Ganesha currently has 2 phases of dispatch queuing: one for input and
> decoding, then another for executing/encoding output.  (I've fixed the
> third queue for later sending the output, where the thread should stay
> hot as long as there's more to process.)
>
> On Monday, Matt told me we were having problems with sequential IO.
> Wishing that somebody had mentioned this to me sooner.
>
> [...]
>
After a somewhat loud discussion with Matt, we've agreed on a
different approach.  This will also be useful for fully async IO
that is planned for V2.6.

The sequential IO reports were from specific customers.  Since I'm
not in direct contact with customers I'm not hearing about it.  But
we should be able to get field confirmation about whether the new
approach is working better

I'm going to code something more like Weighted Fair Queuing that
I've mentioned on this list back in June 2015.  The only weight is
that we want any initial TCP connection to be handled as rapidly as
possible to get the standard TCP slow start moving.

Are there other priorities that we should handle?

Otherwise, we really need a more even handed approach across large
numbers of clients, keeping each client's requests in strict order,
even though some of them could be "faster" than others.  The fair
queuing should also help prevent traffic spikes.

I think I can have something coded by next week.  I'd already done
some preliminary work in 2015.  But the time constraint means this
will be pretty bare bones for V2.5.

To really do a good job, we need some kind of FSAL feedback API.
I'm going to ask the Gluster folks for some help on designing it, so
that we have a good use case and testing infrastructure.  But we'll
post the design iterations here in the same fashion as an IETF
Working Group, so that maybe we can get other FSAL feedback, too.

Is anybody specifically interested in helping design the API?


--
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel