Re: Memstore performance improvements v0.90 vs v0.87

2015-02-21 Thread Alexandre DERUMIER
quot;James Page" , "ceph-devel" , "Stephen L Blinick" , "Jay Vosburgh" , "Colin Ian King" , "Patricia Gaughen" , "Leann Ogasawara" Envoyé: Samedi 21 Février 2015 09:10:29 Objet: Re: Memstore performance improvements v0.90 vs v0.87

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-21 Thread Alexandre DERUMIER
e" , "James Page" , "ceph-devel" , "Stephen L Blinick" , "Jay Vosburgh" , "Colin Ian King" , "Patricia Gaughen" , "Leann Ogasawara" Envoyé: Vendredi 20 Février 2015 17:12:46 Objet: Re: Memstore performance improvem

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Stefan Priebe
;Stephen L Blinick" , "Jay Vosburgh" , "Colin Ian King" , "Patricia Gaughen" , "Leann Ogasawara" Envoyé: Vendredi 20 Février 2015 16:38:02 Objet: Re: Memstore performance improvements v0.90 vs v0.87 I think paying attention to NUMA is good advice. One of

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Mark Nelson
On 02/20/2015 10:03 AM, Alexandre DERUMIER wrote: http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/ It's possible that this could be having an effect on the results. Isn't auto numa balancing enabled by default since kernel 3.8 ? No idea, I'm behind the tim

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Alexandre DERUMIER
oc/sys/kernel/numa_balancing - Mail original - De: "Mark Nelson" À: "Blair Bethwaite" , "James Page" Cc: "ceph-devel" , "Stephen L Blinick" , "Jay Vosburgh" , "Colin Ian King" , "Patricia Gaughen" , "Leann Ogas

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread James Page
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Mark On 20/02/15 15:51, Mark Nelson wrote: >> I've CC'ed both Colin and Jay on this mail - so if anyone has >> any specific questions about the testing they can chime in with >> responses. > Good testing! Other than the NUMA questions in the oth

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Mark Nelson
On 02/20/2015 03:07 AM, James Page wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi All The Ubuntu Kernel team have spent the last few weeks investigating the apparent performance disparity between RHEL 7 and Ubuntu 14.04; we've focussed efforts in a few ways (see below). All testin

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Mark Nelson
I think paying attention to NUMA is good advice. One of the things that apparently changed in RHEL7 is that they are now doing automatic NUMA tuning: http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/ It's possible that this could be having an effect on the re

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Haomai Wang
Actually, I'm concerned about the correctness of benchmark using MemStore. AFAR it may cause lots of memory frag and cause performance degraded hugely. Maybe set "filestore_blackhole=true" is more precious? On Fri, Feb 20, 2015 at 5:49 PM, Blair Bethwaite wrote: > Hi James, > > Interesting resul

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Blair Bethwaite
Hi James, Interesting results, but did you do any tests with a NUMA system? IIUC the original report was from a dual socket setup, and that'd presumably be the standard setup for most folks (both OSD server and client side). Cheers, On 20 February 2015 at 20:07, James Page wrote: > -BEGIN P

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread James Page
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi All The Ubuntu Kernel team have spent the last few weeks investigating the apparent performance disparity between RHEL 7 and Ubuntu 14.04; we've focussed efforts in a few ways (see below). All testing has been done using the latest Firefly relea

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-29 Thread James Page
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Mark On 28/01/15 21:51, Mark Nelson wrote: > Per Sage's suggestion in the perf meeting this morning I dumped > sysctl -a on both systems and wrote a little script to compare an > arbitrary number of sysctl output files. It only lists settings >

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-28 Thread Mark Nelson
27, 2015 2:03 PM To: Blinick, Stephen L; Ceph Development Subject: Re: Memstore performance improvements v0.90 vs v0.87 Hi Stephen, Took a little longer than I wanted it to, but I finally got some results looking at RHEL7 and Ubuntu 14.04 in our test lab. This is with a recent master pull. Tests are

RE: Memstore performance improvements v0.90 vs v0.87

2015-01-27 Thread Blinick, Stephen L
-Original Message- From: Mark Nelson [mailto:mark.nel...@inktank.com] Sent: Tuesday, January 27, 2015 2:03 PM To: Blinick, Stephen L; Ceph Development Subject: Re: Memstore performance improvements v0.90 vs v0.87 Hi Stephen, Took a little longer than I wanted it to, but I finally got s

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-27 Thread Mark Nelson
ent Subject: Re: Memstore performance improvements v0.90 vs v0.87 On 01/14/2015 04:32 PM, Blinick, Stephen L wrote: I went back and grabbed 87 and built it on RHEL7 as well, and performance is also similar (much better). I've also run it on a few systems (Dual socket 10-core E5v2, Dual sock

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-19 Thread Andreas Bluemle
> > > cephx_sign_messages = false > > > cephx_require_signatures = false > > > ms_dispatch_throttle_bytes = 0 > > > throttler_perf_counter = false > > > > > > [osd] > > > osd_client_message_size_cap = 0 > > > osd_client_mess

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-15 Thread Sage Weil
gt; osd_client_message_cap = 0 > > osd_enable_op_tracker = false > > > > Also, run more clients (in your case rados bench) and see if it is > > scaling or not (it should, till it saturates your cpu). > > > > But, your observation on RHEL7 vs UBUNTU 14.04 LTS i

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-15 Thread Sage Weil
false > > ms_dispatch_throttle_bytes = 0 > > throttler_perf_counter = false > > > > [osd] > > osd_client_message_size_cap = 0 > > osd_client_message_cap = 0 > > osd_enable_op_tracker = false > > > > Also, run more clients (in your case rados b

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-15 Thread Andreas Bluemle
egards > Somnath > -----Original Message----- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Blinick, > Stephen L Sent: Wednesday, January 14, 2015 2:32 PM To: Ceph > Development Subject: RE: Memstore performance improvements v0.90

RE: Memstore performance improvements v0.90 vs v0.87

2015-01-14 Thread Blinick, Stephen L
Ceph Development Subject: Re: Memstore performance improvements v0.90 vs v0.87 On 01/14/2015 04:32 PM, Blinick, Stephen L wrote: > I went back and grabbed 87 and built it on RHEL7 as well, and performance is > also similar (much better). I've also run it on a few systems (Dual socket

RE: Memstore performance improvements v0.90 vs v0.87

2015-01-14 Thread Blinick, Stephen L
2:32 PM To: Ceph Development Subject: RE: Memstore performance improvements v0.90 vs v0.87 I went back and grabbed 87 and built it on RHEL7 as well, and performance is also similar (much better). I've also run it on a few systems (Dual socket 10-core E5v2, Dual socket 6-core E5v3). So, it'

RE: Memstore performance improvements v0.90 vs v0.87

2015-01-14 Thread Somnath Roy
: RE: Memstore performance improvements v0.90 vs v0.87 I went back and grabbed 87 and built it on RHEL7 as well, and performance is also similar (much better). I've also run it on a few systems (Dual socket 10-core E5v2, Dual socket 6-core E5v3). So, it's related to my switch to RHEL7,

Re: Memstore performance improvements v0.90 vs v0.87

2015-01-14 Thread Mark Nelson
kernel.org] On Behalf Of Blinick, Stephen L Sent: Wednesday, January 14, 2015 12:06 AM To: Ceph Development Subject: Memstore performance improvements v0.90 vs v0.87 In the process of moving to a new cluster (RHEL7 based) I grabbed v0.90, compiled RPM's and re-ran the simple local-node memstor

RE: Memstore performance improvements v0.90 vs v0.87

2015-01-14 Thread Blinick, Stephen L
and v0.87. Will post when I get more data. Thanks, Stephen -Original Message- From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Blinick, Stephen L Sent: Wednesday, January 14, 2015 12:06 AM To: Ceph Development Subject: Memstore performance im

Memstore performance improvements v0.90 vs v0.87

2015-01-13 Thread Blinick, Stephen L
In the process of moving to a new cluster (RHEL7 based) I grabbed v0.90, compiled RPM's and re-ran the simple local-node memstore test I've run on .80 - .87. It's a single Memstore OSD and a single Rados Bench client locally on the same node. Increasing queue depth and measuring latency /IOPS.