I really don't have time today to respond to every one-liner
throw-away comment here, so I'll try to stick to the most cogent.
On 2/20/18 8:33 AM, Matt Benjamin wrote:
On Tue, Feb 20, 2018 at 8:12 AM, William Allen Simpson
<william.allen.simp...@gmail.com> wrote:
On 2/18/18 2:47 PM, Matt Benjamin wrote:
On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson
But the planned 2.7 improvements are mostly throughput related, not IOPS.
Not at all, though I am trying to ensure that we get async FSAL ops
in. There are people working on IOPs too.
async FSAL ops are not likely to further improve IOPS.
As I've pointed out many times in the past, async only allows
the same number of threads to handle more concurrent operations.
But it's actually slower. It basically doubles the number of
system calls. System calls are one of the reasons that Ganesha is
slower than knfsd. It also ruins CPU cache coherency.
If we're CPU bound or network bandwidth constrained, it won't help.
Not everything being worked on is a direct answer to this problem.
Anything that isn't about protocol correctness or performance
should be lower priority.
Async FSAL ops could -- in a few cases where network or disk spinning
dominate the thread timing -- improve thread utilization. But it
will never improve IOPS.
The effort that I've been waiting for *3* years -- add io vectors to
the FSAL interface, so that we can zero-copy buffers -- is more
likely to improve throughput.
Which, as you've noted, also is happening.
Finally. Again, I've been asking for it now 3 years. For V2.7, this is
what I anticipate will result in the most performance increase.
Moreover, going through the code and removing locking other such
bottlenecks is more likely to improve IOPS.
No one is disputing this. It is not a new discovery, however.
And yet we seem to have to keep reminding folks. I remember that
Malahal did a survey of lock timing/congestion a couple of years ago.
Perhaps he's willing to run it again?
If Ganesha is adding 6 ms to every read operation, we have a serious
problem, and need to profile immediately!
That's kind of what our team is doing. I look forward to your work
with rpc-ping-mt.
Well, you were able to send me a copy after 6 pm on a Friday, so I'm
now taking a look at it. Hopefully I'll have something by Friday.
It came up in a meeting on Friday, that's why I sent it Friday. I got
busy in meetings and issues, that's why after 6 pm.
But I really wish you'd posted it 3 years ago. It doesn't really test
IOPS, other than whatever bandwidth limit is imposed by the interface,
but it does test the client call interface.
It measures minimal request latency all the way to nfs-ganesha's, or
the Linux kernel's, rpcnull handler--the "top" of the request handling
stack, in the given client/server/network configuration. Scaled up,
it can report the max such calls/s, which is a kind of best-possible
value for iops, taking FSAL ops to have 0 latency.
As I've mentioned elsewhere, this should be entirely dominated by the
link speed and protocol. We should see UDP as slowest, TCP in the
middle, and RDMA as fastest.
OTOH, the "max such calls/s" would be reported by using XDR_RAW, which
is currently not working.
It was posted to this list by Tigran, iirc, in 2011 or 2012.
In which case, I really wish Tigran had put it in the tests folder,
so we'd have been continuously using and updating it. The gtests
are a great improvement. Now all we need to do is run them every
week for each new -dev release to track improvements/regressions.
I didn't know about this old test, because my archives only go back
to late 2014. I wasn't involved before then.
Moreover, I didn't appreciate being criticized for not running an
old test that wasn't in the tree and wasn't maintained.
We've been using Jeff Layton's delegation callback work to test, and
this test would have been better and easier.
But a unit test is not what we need. I wrote "profile". We need to
know where the CPU bottlenecks are in Ganesha itself.
You also wrote unit test.
Looking back over this thread, I don't see those words.
I wrote "profile".
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
