On 08/ 3/12 07:45 PM, Sanjay Nadkarni wrote:
On 8/2/12 3:20 PM, Shawn Walker wrote:
On 08/02/12 11:08, Mike Gerdts wrote:
On 08/02/12 12:51, Rob Thurlow wrote:
On 8/1/12 4:53 PM, Mike Gerdts wrote:
Is it known and/or expected that nfs:/// repos perform *much* worse
than
http:// repos?
Certainly; small I/Os and high latency kill NFS. On AK builds,
we get seriously spanked by using remote repos for image creation,
taking a build from 15 minutes to 3 hours unless the cache is warm.
If the necessary files can get staged into cache in a more efficient
way, something better might happen.
Agreed. What is it about the way that pkg accesses file repos that makes
it so much slower than http repos?
Each method seems to look to see if various files are there and get some
attributes (stat(2) or http HEAD) and need to transfer the files that
are not already present locally. Last time I checked, http and nfs
packets go across the wire at about the same speed. I think that NFS
packets actually carry less protocol baggage with the requests and
returned data, so they should go a tiny bit faster only because they
should be a tiny bit smaller.
This leads me to think that pkg is doing a lot of unnecessary file
system operations such as multiple stat calls on the same file or
calling stat before trying to open a file. If file-based repos are
intended to just be a convenience for developers, perhaps this isn't an
issue. If file-based repos are intended to be used with NFS in global
enterprises, this performance disparity deserves attention.
It could be there are some unnecessary stats, but what's there is not
intentionally naive; some performance work has already been done
specifically to reduce the total number of file operations during pkg
execution.
Some of the access patterns are also intentional tradeoffs in
robustness vs. performance. With that said, these areas can be
revisited if something specific can be identified as being
problematic, or potentially change access patterns more generally.
There's also always the potential for bugs.
As I said before, NFS performance is generally significantly better
than HTTP performance (especially for large transfers) as the client
does significantly less I/O when using repositories via NFS.
I wonder if this is another manifestation of an issue that I was
tracking a few years ago with ON builds over NFS. Specifically, when NFS
v4 is enabled ON build times increased significantly (more than
doubled). The overhead was associated with stat'ing files. I had also
noted back then that the issue was specific Solaris 11. When a NetApp
box was used, the degradation was minimal. pkg'ing could be the victim
of the same effect.
Hi Sanjay,
What I remember from that period was that NFSv4 was slower because
of the protocol issue that serializes opens; that the parallel
build hits that bottleneck hard. (That protocol limitation is
fixed in a later version, which we're implementing.) I don't
recall anything special slowing down stat()s, unless it was just
a bug in the implementation. But I was not hip deep in that issue,
either.
Rob T
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
