On 02/20/2015 03:07 AM, James Page wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi All
The Ubuntu Kernel team have spent the last few weeks investigating the
apparent performance disparity between RHEL 7 and Ubuntu 14.04; we've
focussed efforts in a few ways (see below).
All testing has been done using the latest Firefly release.
1) Base network latency
Jay Vosburgh looked at the base network latencies between RHEL 7 and
Ubuntu 14.04; under default install, RHEL actually had slightly worse
latency than Ubuntu due to the default enablement of a firewall;
disabling this brought latency back inline between the two distributions:
OS rtt min/avg/max/mdev
Ubuntu 14.04 (3.13) 0.013/0.016/0.018/0.005 ms
RHEL7 (3.10) 0.010/0.018/0.025/0.005 ms
...base network latency is pretty much the same.
This testing was performed on a matched pair of Dell Poweredge R610's,
configured with a single 4 core CPU and 8G of RAM.
2) Latency and performance in Ceph using Rados bench
Colin King spent a number of days testing and analysing results using
rados bench against a single node ceph deployment, configured with a
single memory backed OSD, to see if we could reproduce the disparities
reported.
He ran 120 second OSD benchmarks on RHEL 7 as well as Ubuntu 14.04 LTS
with a selection of kernels including 3.10 vanilla, 3.13.0-44 (release
kernel), 3.16.0-30 (utopic HWE kernel), 3.18.0-12 (vivid HWE kernel)
and 3.19-rc6 with 1, 16 and 128 client threads. The data collected is
available at [0].
Each round of tests consisted of 15 runs, from which we computed
average latency, latency deviation and latency distribution:
120 second x 1 thread
Results all seem to cluster around 0.04->0.05ms, with RHEL 7 averaging
at 0.044 and recent Ubuntu kernels at 0.036-0.037ms. The older 3.10
kernel in RHEL 7 does have some slightly higher average latency.
120 second x 16 threads
Results all seem to cluster around 0.6-0.7ms. 3.19.0-rc6 had a couple
of 1.4ms outliers which pushed it out to be worse than RHEL 7. On the
whole Ubuntu 3.10-3.18 kernels are better than RHEL 7 by ~0.1ms. RHEL
shows a far higher standard deviation, due to the bimodal latency
distribution, which from the casual observer may appear to be more
"jittery".
120 second x 128 threads
Later kernels show up to have less standard deviation than RHEL 7, so
that shows perhaps less jitter in the stats than RHEL 7's 3.10 kernel.
With this many threads pounding the test, we get a wider spread of
latencies and it is hard to tell any kind of latency distribution
patterns with just 15 rounds because of the large amount of latency
jitter. All systems show a latency of ~ 5ms. Taking into
consideration the amount of jitter, we think these results do not make
much sense unless we repeat these tests with say 100 samples.
3) Conclusion
We’ve have not been able to show any major anomalies in Ceph on Ubuntu
compared to RHEL 7 when using memstore. Our current hypothesis is that
one needs to run the OSD bench stressor many times to get a fair capture
of system latency stats. The reason for this is:
* Latencies are very low with memstore, so any small jitter in
scheduling etc will show up as a large distortion (as shown by the large
standard deviations in the samples).
* When memstore is heavily utilized, memory pressure causes the system
to page heavily and so we are subject to the nature of perhaps delays on
paging that cause some latency jitters. Latency differences may be just
down to where a random page is in memory or in swap, and with memstore
these may cause the large perturbations we see when running just a
single test.
* We needed to make *many* tens of measurements to get a typical idea of
average latency and the latency distributions. Don't trust the results
from just one test
* We ran the tests with a pool configured to 100 pgs and 100 pgps [1].
One can get different results with different placement group configs.
I've CC'ed both Colin and Jay on this mail - so if anyone has any
specific questions about the testing they can chime in with responses.
Hi James,
Good testing! Other than the NUMA questions in the other posts, is it
possible the disparity we saw could be related to the network
hardware/driver? Our nodes just are using 1GbE Intel I350s. I'm not
sure what Stephen's uses, but given that he's at Intel I suspect it
would also be an Intel NIC. Any chance you could post the specs of the
boxes you guys tested on so we could try to look for other differences?
Also, was there any additional tuning done on either of the boxes (other
than disabling RHEL's firewall)?
Mark
Regards
James
[0] http://kernel.ubuntu.com/~cking/.ceph/ceph-benchmarks.ods
[1] http://ceph.com/docs/master/rados/configuration/pool-pg-config-ref/
- --
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBCAAGBQJU5vlrAAoJEL/srsug59jDMvAQAIhSR4GFTXNc4RLpHtLT6h/X
K5uyauKZGtL+wqtPKRfsXqbbUw9I5AZDifQuOEJ0APccLIPbgqxEN3d2uht/qygH
G8q2Ax+M8OyZz07yqTitnD4JV3RmL8wNHUveWPLV0gs2TzBBYwP1ywExbRPed3PY
cfDrszgkQszA/JwT5W5YNf1LZc+5VpOEFrTiLIaRzUDoxg7mm6Hwr3XT8OFjZhjm
LSenKREHtrKKWoBh+OKTvuCUnHzEemK+CiwwRbNQ8l7xbp71wLyS08NpSB5C1y70
7uft+kP6XOGE9AKLvsdEL1PIXHfeKNonBEN5mO6nsXIW+MQzou01zHgDtne7AxDA
5OebQJfJtArmKt78WHuVg7h8gPcIRTRSW43LqJiADnIHL8fnZxj2v5yDiUQj7isw
nYWXEJ3rR7mlVgydN34KQ7gpVWmGjhrVb8N01+zYOMAaTBnekldHdueEAXR07eU0
PXiP9aOZiAxbEnDiJmreehjCuNFTagQqNeECRIHssSacfQXPxVljaImvuSfrxf8i
myQLzftiObINTIHSN4TVDKMyveYrU2hILCKfYuxnSJh29j35wsRSeftjntOEyHai
RDnrLD3fCPk4h3hCY6l60nqu9MQfbgdSB/FItvhiBGYqXvGb4+wuBeU9RT9SwG8N
XPih7nLNvqDNw38IkkDN
=qcvG
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
