[EMAIL PROTECTED] wrote on Mon, 11 Dec 2006 14:34 -0600:
> This is some really nice analysis Pete. One thing we might consider
> for reducing context switches would be to re-use the coalescing ideas
> we've tried to apply for meta data syncs to context switches as
> well. If we've got a lot of operations completing in a particular
> module (trove for example), we can just signal once instead of for
> each operation. The sync coalescing code already does this, but you
> would only see any benefits from that with a bunch of clients.
Maybe that would be effective for a loaded server, but that won't
help single-op latency much.
> The locking and context switches are inherent to our design of server
> module separation and queueing framework. I think it would be hard
> to replace the context switches without doing some serious redesign.
Agreed.
> We might be able to eliminate the trove thread though. It doesn't do
> anything but move items from the trove completion queue to the job
> completion queue. Since that thread waits on a condition variable
> (and gets signalled by trove), and then signals the job completion
> condition variable, we're essentially doing a double context switch
> when we only need one. Instead we could change the trove apis to
> take a callback and user ptr, and have the callback add the completed
> job to the completion queue directly. The bits of flow that use job
> callbacks with trove would have to be changed too, but I think the
> flows would benefit from the bmi callback being called directly from
> trove as well. Does this seem reasonable?
This is a good idea. The job- or flow-supplied callback function
can do whatever locking it needs to do protect its own resources.
Before the callback is invoked you can make sure the completion is
out of any trove lists. If you are interested in doing this part,
I'll be happy to do some more timing tests.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
