Re: [ceph-users] Scaling radosgw module

[hadi golestani]

I've found nothing related in Apache logs,
I believe it's something related to Radosgw,
Anyone else tested the same thing on owned Radosgw?

Regards



On Mon, Oct 28, 2013 at 11:52 PM, Mark Nelson wrote:

> I'm not really an apache expert, but you could try looking at the apache
> and rgw logs and see if you can trace where the 2nd request is hanging up.
>  Also, just to be sure, both clients can download data independently, just
> not together?
>
> Mark
>
>
> On 10/28/2013 02:54 PM, hadi golestani wrote:
>
>> I'm running Ubuntu 12 on all my nodes and I've just installed every
>> package with default configs like what is mentioned in quick installtion
>> guide of Ceph
>> Anyone else experiancing the same issue?
>>
>> Regards
>>
>>
>>
>> On Mon, Oct 28, 2013 at 11:09 PM, Mark Nelson > > wrote:
>>
>> Strange!  I'm not sure I've actually ever seen two concurrent
>> downloads fail to work properly.  Is there anything unusual about
>> the setup?
>>
>> Mark
>>
>>
>> On 10/28/2013 01:58 PM, hadi golestani wrote:
>>
>> My test is so simple,
>> On a cluster with 3 MON, 4 OSD, 1 RGW I can't download a big
>> file from
>> two different clients concurrently,
>> One of them will wait till the other finish downloading it.
>>
>> Regards
>>
>>
>>
>> On Mon, Oct 28, 2013 at 10:19 PM, Mark Nelson
>> > > >
>> >
>> >>
>> wrote:
>>
>>  Hi Hadi,
>>
>>  Can you tell me a bit about the tests you are doing and
>> seeing poor
>>  performance on?
>>
>>  Mark
>>
>>
>>  On 10/28/2013 01:32 PM, hadi golestani wrote:
>>
>>  Any answer to this question? I'm hitting almost the
>> same issue
>>  with radosgw,
>>  Read performance is not fine with radosgw
>>
>>  Regards
>>
>>  __**_
>>
>>  ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> > >
>> > >
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph._**
>> ___com 
>> 
>> 
>> >
>>
>> 
>> 
>> 
>> 
>> >>
>>
>>
>>  __**_
>>
>>  ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> > >
>> > >
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph._**
>> ___com 
>> 
>> 
>> >
>>  > _com 
>> 
>> 
>> >>
>>
>>
>>
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

I'm not really an apache expert, but you could try looking at the apache 
and rgw logs and see if you can trace where the 2nd request is hanging 
up.  Also, just to be sure, both clients can download data 
independently, just not together?


Mark

On 10/28/2013 02:54 PM, hadi golestani wrote:

I'm running Ubuntu 12 on all my nodes and I've just installed every
package with default configs like what is mentioned in quick installtion
guide of Ceph
Anyone else experiancing the same issue?

Regards



On Mon, Oct 28, 2013 at 11:09 PM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote:

Strange!  I'm not sure I've actually ever seen two concurrent
downloads fail to work properly.  Is there anything unusual about
the setup?

Mark


On 10/28/2013 01:58 PM, hadi golestani wrote:

My test is so simple,
On a cluster with 3 MON, 4 OSD, 1 RGW I can't download a big
file from
two different clients concurrently,
One of them will wait till the other finish downloading it.

Regards



On Mon, Oct 28, 2013 at 10:19 PM, Mark Nelson
mailto:mark.nel...@inktank.com>
>> wrote:

 Hi Hadi,

 Can you tell me a bit about the tests you are doing and
seeing poor
 performance on?

 Mark


 On 10/28/2013 01:32 PM, hadi golestani wrote:

 Any answer to this question? I'm hitting almost the
same issue
 with radosgw,
 Read performance is not fine with radosgw

 Regards

 ___
 ceph-users mailing list
ceph-users@lists.ceph.com 
>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


>


 ___
 ceph-users mailing list
ceph-users@lists.ceph.com 
>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 >






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[hadi golestani]

I'm running Ubuntu 12 on all my nodes and I've just installed every package
with default configs like what is mentioned in quick installtion guide of
Ceph
Anyone else experiancing the same issue?

Regards



On Mon, Oct 28, 2013 at 11:09 PM, Mark Nelson wrote:

> Strange!  I'm not sure I've actually ever seen two concurrent downloads
> fail to work properly.  Is there anything unusual about the setup?
>
> Mark
>
>
> On 10/28/2013 01:58 PM, hadi golestani wrote:
>
>> My test is so simple,
>> On a cluster with 3 MON, 4 OSD, 1 RGW I can't download a big file from
>> two different clients concurrently,
>> One of them will wait till the other finish downloading it.
>>
>> Regards
>>
>>
>>
>> On Mon, Oct 28, 2013 at 10:19 PM, Mark Nelson > > wrote:
>>
>> Hi Hadi,
>>
>> Can you tell me a bit about the tests you are doing and seeing poor
>> performance on?
>>
>> Mark
>>
>>
>> On 10/28/2013 01:32 PM, hadi golestani wrote:
>>
>> Any answer to this question? I'm hitting almost the same issue
>> with radosgw,
>> Read performance is not fine with radosgw
>>
>> Regards
>>
>> __**___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> > >
>> 
>> http://lists.ceph.com/__**listinfo.cgi/ceph-users-ceph._**_com
>> 
>> 
>> >
>>
>>
>> __**___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> > >
>> 
>> http://lists.ceph.com/__**listinfo.cgi/ceph-users-ceph._**_com
>> 
>> 
>> >
>>
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

Strange!  I'm not sure I've actually ever seen two concurrent downloads 
fail to work properly.  Is there anything unusual about the setup?


Mark

On 10/28/2013 01:58 PM, hadi golestani wrote:

My test is so simple,
On a cluster with 3 MON, 4 OSD, 1 RGW I can't download a big file from
two different clients concurrently,
One of them will wait till the other finish downloading it.

Regards



On Mon, Oct 28, 2013 at 10:19 PM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote:

Hi Hadi,

Can you tell me a bit about the tests you are doing and seeing poor
performance on?

Mark


On 10/28/2013 01:32 PM, hadi golestani wrote:

Any answer to this question? I'm hitting almost the same issue
with radosgw,
Read performance is not fine with radosgw

Regards

_
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com



_
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[Yehuda Sadeh]

Sounds like an issue with your apache config. How did you install your
apache? What distribution are you running on? Are you using it as
mpm-worker? Do you have non-default radosgw settings?

Yehuda

On Mon, Oct 28, 2013 at 11:58 AM, hadi golestani
 wrote:
> My test is so simple,
> On a cluster with 3 MON, 4 OSD, 1 RGW I can't download a big file from two
> different clients concurrently,
> One of them will wait till the other finish downloading it.
>
> Regards
>
>
>
> On Mon, Oct 28, 2013 at 10:19 PM, Mark Nelson 
> wrote:
>>
>> Hi Hadi,
>>
>> Can you tell me a bit about the tests you are doing and seeing poor
>> performance on?
>>
>> Mark
>>
>>
>> On 10/28/2013 01:32 PM, hadi golestani wrote:
>>>
>>> Any answer to this question? I'm hitting almost the same issue with
>>> radosgw,
>>> Read performance is not fine with radosgw
>>>
>>> Regards
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[hadi golestani]

My test is so simple,
On a cluster with 3 MON, 4 OSD, 1 RGW I can't download a big file from two
different clients concurrently,
One of them will wait till the other finish downloading it.

Regards



On Mon, Oct 28, 2013 at 10:19 PM, Mark Nelson wrote:

> Hi Hadi,
>
> Can you tell me a bit about the tests you are doing and seeing poor
> performance on?
>
> Mark
>
>
> On 10/28/2013 01:32 PM, hadi golestani wrote:
>
>> Any answer to this question? I'm hitting almost the same issue with
>> radosgw,
>> Read performance is not fine with radosgw
>>
>> Regards
>>
>> __**_
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>>
>>
> __**_
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

Hi Hadi,

Can you tell me a bit about the tests you are doing and seeing poor 
performance on?


Mark

On 10/28/2013 01:32 PM, hadi golestani wrote:

Any answer to this question? I'm hitting almost the same issue with radosgw,
Read performance is not fine with radosgw

Regards

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[hadi golestani]

Any answer to this question? I'm hitting almost the same issue with radosgw,
Read performance is not fine with radosgw

Regards

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

Likely on the radosgw side you are going to see the top consumers be 
malloc/free/memcpy/memcmp.  If you have kernel 3.9 or newer compiled 
with libunwind, you might get better callgraphs in perf which could be 
helpful.


Mark

On 09/27/2013 01:56 PM, Somnath Roy wrote:

Yes, I understand that..
I tried with thread pool size of 300 (default 100, I believe). I am in process 
of running perf on radosgw as well as on osds for profiling.
BTW,  let me know if any particular ceph component  you want me to focus.

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Friday, September 27, 2013 11:50 AM
To: Somnath Roy
Cc: Yehuda Sadeh; ceph-users@lists.ceph.com; Anirban Ray; 
ceph-de...@vger.kernel.org
Subject: Re: [ceph-users] Scaling radosgw module

Hi Somnath,

With SSDs, you almost certainly are going to be running into bottlenecks on the 
RGW side... Maybe even fastcgi or apache depending on the machine and how 
things are configured.  Unfortunately this is probably one of the more complex 
performance optimization scenarios in the Ceph world and is going to require 
figuring out exactly where things are slowing down.

I don't remember if you've done this already, but you could try increasing the 
number of radosgw threads and try to throw more concurrency at the problem, but 
other than that it's probably going to come down to profiling, and lots of it. 
:)

Mark

On 09/26/2013 07:04 PM, Somnath Roy wrote:

Hi Yehuda,
With my 3 node cluster (30 OSDs in total, all in ssds), I am getting avg of 
~3000 Gets/s from a single swift-bench client hitting single radosgw instance. 
Put is ~1000/s. BTW, I am not able to generate very big load yet and as the 
server has ~140G RAM, all the GET requests are served from memory , no disk 
utilization here.

Thanks & Regards
Somnath

-Original Message-
From: Yehuda Sadeh [mailto:yeh...@inktank.com]
Sent: Thursday, September 26, 2013 4:48 PM
To: Somnath Roy
Cc: Mark Nelson; ceph-users@lists.ceph.com; Anirban Ray;
ceph-de...@vger.kernel.org
Subject: Re: [ceph-users] Scaling radosgw module

You specify the relative performance, but what the actual numbers that you're 
seeing? How many GETs per second, and how many PUTs per second do you see?

On Thu, Sep 26, 2013 at 4:00 PM, Somnath Roy  wrote:

Mark,
One more thing, all my test is with rgw cache enabled , disabling the cache the 
performance is around 3x slower.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 26, 2013 3:59 PM
To: Mark Nelson
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban
Ray
Subject: RE: [ceph-users] Scaling radosgw module

Nope...With one client hitting the radaosgw , the daemon cpu usage is going up 
till 400-450% i.e taking in avg 4 core..In one client scenario, the server node 
(having radosgw + osds) cpu usage is ~80% idle and out of the 20% usage bulk is 
consumed by radosgw.

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Thursday, September 26, 2013 3:50 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban
Ray
Subject: Re: [ceph-users] Scaling radosgw module

Ah, that's very good to know!

And RGW CPU usage you said was low?

Mark

On 09/26/2013 05:40 PM, Somnath Roy wrote:

Mark,
I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
throughput is linearly scaling. But, as an individual radosgw performance is 
very low we need to put lots of radosgw/apache server combination to get very 
high throughput. I guess that will be a problem.
I will try to do some profiling and share the data.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Thursday, September 26, 2013 3:33 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban
Ray
Subject: Re: [ceph-users] Scaling radosgw module

It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
seeing if having two copies of the benchmark going at the same time on two 
separate RGW servers increases aggregate throughput.

Also, it may be worth tracking down latencies with messenger
debugging enabled, but I'm afraid I'm pretty bogged down right now
and probably wouldn't be able to look at it for a while. :(

Mark

On 09/26/2013 05:15 PM, Somnath Roy wrote:

Hi Mark,
FYI, I tried with wip-6286-dumpling release and the results are the same for 
me. The radosgw throughput is around ~6x slower than the single rados bench 
output!
 Any other suggestion ?

Thanks & Regards
Somnath
-Original Message-
From: Somnath Roy
Sent

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Yes, I understand that..
I tried with thread pool size of 300 (default 100, I believe). I am in process 
of running perf on radosgw as well as on osds for profiling.
BTW,  let me know if any particular ceph component  you want me to focus.

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com] 
Sent: Friday, September 27, 2013 11:50 AM
To: Somnath Roy
Cc: Yehuda Sadeh; ceph-users@lists.ceph.com; Anirban Ray; 
ceph-de...@vger.kernel.org
Subject: Re: [ceph-users] Scaling radosgw module

Hi Somnath,

With SSDs, you almost certainly are going to be running into bottlenecks on the 
RGW side... Maybe even fastcgi or apache depending on the machine and how 
things are configured.  Unfortunately this is probably one of the more complex 
performance optimization scenarios in the Ceph world and is going to require 
figuring out exactly where things are slowing down.

I don't remember if you've done this already, but you could try increasing the 
number of radosgw threads and try to throw more concurrency at the problem, but 
other than that it's probably going to come down to profiling, and lots of it. 
:)

Mark

On 09/26/2013 07:04 PM, Somnath Roy wrote:
> Hi Yehuda,
> With my 3 node cluster (30 OSDs in total, all in ssds), I am getting avg of 
> ~3000 Gets/s from a single swift-bench client hitting single radosgw 
> instance. Put is ~1000/s. BTW, I am not able to generate very big load yet 
> and as the server has ~140G RAM, all the GET requests are served from memory 
> , no disk utilization here.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: Yehuda Sadeh [mailto:yeh...@inktank.com]
> Sent: Thursday, September 26, 2013 4:48 PM
> To: Somnath Roy
> Cc: Mark Nelson; ceph-users@lists.ceph.com; Anirban Ray; 
> ceph-de...@vger.kernel.org
> Subject: Re: [ceph-users] Scaling radosgw module
>
> You specify the relative performance, but what the actual numbers that you're 
> seeing? How many GETs per second, and how many PUTs per second do you see?
>
> On Thu, Sep 26, 2013 at 4:00 PM, Somnath Roy  wrote:
>> Mark,
>> One more thing, all my test is with rgw cache enabled , disabling the cache 
>> the performance is around 3x slower.
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-
>> From: ceph-devel-ow...@vger.kernel.org 
>> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Somnath Roy
>> Sent: Thursday, September 26, 2013 3:59 PM
>> To: Mark Nelson
>> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban 
>> Ray
>> Subject: RE: [ceph-users] Scaling radosgw module
>>
>> Nope...With one client hitting the radaosgw , the daemon cpu usage is going 
>> up till 400-450% i.e taking in avg 4 core..In one client scenario, the 
>> server node (having radosgw + osds) cpu usage is ~80% idle and out of the 
>> 20% usage bulk is consumed by radosgw.
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-----
>> From: Mark Nelson [mailto:mark.nel...@inktank.com]
>> Sent: Thursday, September 26, 2013 3:50 PM
>> To: Somnath Roy
>> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban 
>> Ray
>> Subject: Re: [ceph-users] Scaling radosgw module
>>
>> Ah, that's very good to know!
>>
>> And RGW CPU usage you said was low?
>>
>> Mark
>>
>> On 09/26/2013 05:40 PM, Somnath Roy wrote:
>>> Mark,
>>> I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
>>> swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
>>> throughput is linearly scaling. But, as an individual radosgw performance 
>>> is very low we need to put lots of radosgw/apache server combination to get 
>>> very high throughput. I guess that will be a problem.
>>> I will try to do some profiling and share the data.
>>>
>>> Thanks & Regards
>>> Somnath
>>>
>>> -Original Message-
>>> From: ceph-devel-ow...@vger.kernel.org 
>>> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
>>> Sent: Thursday, September 26, 2013 3:33 PM
>>> To: Somnath Roy
>>> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban 
>>> Ray
>>> Subject: Re: [ceph-users] Scaling radosgw module
>>>
>>> It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
>>> seeing if having two copies of the benchmark going at the same time on two 
>>> separate RGW servers increases aggregate throughput.
>>>
>>> Also, it may be worth t

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

Hi Somnath,

With SSDs, you almost certainly are going to be running into bottlenecks 
on the RGW side... Maybe even fastcgi or apache depending on the machine 
and how things are configured.  Unfortunately this is probably one of 
the more complex performance optimization scenarios in the Ceph world 
and is going to require figuring out exactly where things are slowing down.


I don't remember if you've done this already, but you could try 
increasing the number of radosgw threads and try to throw more 
concurrency at the problem, but other than that it's probably going to 
come down to profiling, and lots of it. :)


Mark

On 09/26/2013 07:04 PM, Somnath Roy wrote:

Hi Yehuda,
With my 3 node cluster (30 OSDs in total, all in ssds), I am getting avg of 
~3000 Gets/s from a single swift-bench client hitting single radosgw instance. 
Put is ~1000/s. BTW, I am not able to generate very big load yet and as the 
server has ~140G RAM, all the GET requests are served from memory , no disk 
utilization here.

Thanks & Regards
Somnath

-Original Message-
From: Yehuda Sadeh [mailto:yeh...@inktank.com]
Sent: Thursday, September 26, 2013 4:48 PM
To: Somnath Roy
Cc: Mark Nelson; ceph-users@lists.ceph.com; Anirban Ray; 
ceph-de...@vger.kernel.org
Subject: Re: [ceph-users] Scaling radosgw module

You specify the relative performance, but what the actual numbers that you're 
seeing? How many GETs per second, and how many PUTs per second do you see?

On Thu, Sep 26, 2013 at 4:00 PM, Somnath Roy  wrote:

Mark,
One more thing, all my test is with rgw cache enabled , disabling the cache the 
performance is around 3x slower.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 26, 2013 3:59 PM
To: Mark Nelson
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: RE: [ceph-users] Scaling radosgw module

Nope...With one client hitting the radaosgw , the daemon cpu usage is going up 
till 400-450% i.e taking in avg 4 core..In one client scenario, the server node 
(having radosgw + osds) cpu usage is ~80% idle and out of the 20% usage bulk is 
consumed by radosgw.

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Thursday, September 26, 2013 3:50 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: Re: [ceph-users] Scaling radosgw module

Ah, that's very good to know!

And RGW CPU usage you said was low?

Mark

On 09/26/2013 05:40 PM, Somnath Roy wrote:

Mark,
I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
throughput is linearly scaling. But, as an individual radosgw performance is 
very low we need to put lots of radosgw/apache server combination to get very 
high throughput. I guess that will be a problem.
I will try to do some profiling and share the data.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Thursday, September 26, 2013 3:33 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban
Ray
Subject: Re: [ceph-users] Scaling radosgw module

It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
seeing if having two copies of the benchmark going at the same time on two 
separate RGW servers increases aggregate throughput.

Also, it may be worth tracking down latencies with messenger
debugging enabled, but I'm afraid I'm pretty bogged down right now
and probably wouldn't be able to look at it for a while. :(

Mark

On 09/26/2013 05:15 PM, Somnath Roy wrote:

Hi Mark,
FYI, I tried with wip-6286-dumpling release and the results are the same for 
me. The radosgw throughput is around ~6x slower than the single rados bench 
output!
Any other suggestion ?

Thanks & Regards
Somnath
-Original Message-
From: Somnath Roy
Sent: Friday, September 20, 2013 4:08 PM
To: 'Mark Nelson'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Scaling radosgw module

Hi Mark,
It's a test cluster and I will try with the new release.
As I mentioned in the mail, I think number of rados client instance is the 
limitation. Could you please let me know how many rados client instance the 
radosgw daemon is instantiating ? Is it configurable somehow ?

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Friday, September 20, 2013 4:02 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

On 09/20/2013 05:49 PM, Somnath Roy wrote:

Hi Mark,
Thanks for your quick response.
I tried adding the 'num_container = 100&#x

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Hi Yehuda,
With my 3 node cluster (30 OSDs in total, all in ssds), I am getting avg of 
~3000 Gets/s from a single swift-bench client hitting single radosgw instance. 
Put is ~1000/s. BTW, I am not able to generate very big load yet and as the 
server has ~140G RAM, all the GET requests are served from memory , no disk 
utilization here.

Thanks & Regards
Somnath

-Original Message-
From: Yehuda Sadeh [mailto:yeh...@inktank.com] 
Sent: Thursday, September 26, 2013 4:48 PM
To: Somnath Roy
Cc: Mark Nelson; ceph-users@lists.ceph.com; Anirban Ray; 
ceph-de...@vger.kernel.org
Subject: Re: [ceph-users] Scaling radosgw module

You specify the relative performance, but what the actual numbers that you're 
seeing? How many GETs per second, and how many PUTs per second do you see?

On Thu, Sep 26, 2013 at 4:00 PM, Somnath Roy  wrote:
> Mark,
> One more thing, all my test is with rgw cache enabled , disabling the cache 
> the performance is around 3x slower.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org 
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Thursday, September 26, 2013 3:59 PM
> To: Mark Nelson
> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
> Subject: RE: [ceph-users] Scaling radosgw module
>
> Nope...With one client hitting the radaosgw , the daemon cpu usage is going 
> up till 400-450% i.e taking in avg 4 core..In one client scenario, the server 
> node (having radosgw + osds) cpu usage is ~80% idle and out of the 20% usage 
> bulk is consumed by radosgw.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: Mark Nelson [mailto:mark.nel...@inktank.com]
> Sent: Thursday, September 26, 2013 3:50 PM
> To: Somnath Roy
> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
> Subject: Re: [ceph-users] Scaling radosgw module
>
> Ah, that's very good to know!
>
> And RGW CPU usage you said was low?
>
> Mark
>
> On 09/26/2013 05:40 PM, Somnath Roy wrote:
>> Mark,
>> I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
>> swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
>> throughput is linearly scaling. But, as an individual radosgw performance is 
>> very low we need to put lots of radosgw/apache server combination to get 
>> very high throughput. I guess that will be a problem.
>> I will try to do some profiling and share the data.
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-
>> From: ceph-devel-ow...@vger.kernel.org 
>> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
>> Sent: Thursday, September 26, 2013 3:33 PM
>> To: Somnath Roy
>> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban 
>> Ray
>> Subject: Re: [ceph-users] Scaling radosgw module
>>
>> It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
>> seeing if having two copies of the benchmark going at the same time on two 
>> separate RGW servers increases aggregate throughput.
>>
>> Also, it may be worth tracking down latencies with messenger 
>> debugging enabled, but I'm afraid I'm pretty bogged down right now 
>> and probably wouldn't be able to look at it for a while. :(
>>
>> Mark
>>
>> On 09/26/2013 05:15 PM, Somnath Roy wrote:
>>> Hi Mark,
>>> FYI, I tried with wip-6286-dumpling release and the results are the same 
>>> for me. The radosgw throughput is around ~6x slower than the single rados 
>>> bench output!
>>>Any other suggestion ?
>>>
>>> Thanks & Regards
>>> Somnath
>>> -Original Message-
>>> From: Somnath Roy
>>> Sent: Friday, September 20, 2013 4:08 PM
>>> To: 'Mark Nelson'
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: RE: [ceph-users] Scaling radosgw module
>>>
>>> Hi Mark,
>>> It's a test cluster and I will try with the new release.
>>> As I mentioned in the mail, I think number of rados client instance is the 
>>> limitation. Could you please let me know how many rados client instance the 
>>> radosgw daemon is instantiating ? Is it configurable somehow ?
>>>
>>> Thanks & Regards
>>> Somnath
>>>
>>> -Original Message-
>>> From: Mark Nelson [mailto:mark.nel...@inktank.com]
>>> Sent: Friday, September 20, 2013 4:02 PM
>>> To: Somnath Roy
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] Scaling rad

Re: [ceph-users] Scaling radosgw module

[Yehuda Sadeh]

You specify the relative performance, but what the actual numbers that
you're seeing? How many GETs per second, and how many PUTs per second
do you see?

On Thu, Sep 26, 2013 at 4:00 PM, Somnath Roy  wrote:
> Mark,
> One more thing, all my test is with rgw cache enabled , disabling the cache 
> the performance is around 3x slower.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org 
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Thursday, September 26, 2013 3:59 PM
> To: Mark Nelson
> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
> Subject: RE: [ceph-users] Scaling radosgw module
>
> Nope...With one client hitting the radaosgw , the daemon cpu usage is going 
> up till 400-450% i.e taking in avg 4 core..In one client scenario, the server 
> node (having radosgw + osds) cpu usage is ~80% idle and out of the 20% usage 
> bulk is consumed by radosgw.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: Mark Nelson [mailto:mark.nel...@inktank.com]
> Sent: Thursday, September 26, 2013 3:50 PM
> To: Somnath Roy
> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
> Subject: Re: [ceph-users] Scaling radosgw module
>
> Ah, that's very good to know!
>
> And RGW CPU usage you said was low?
>
> Mark
>
> On 09/26/2013 05:40 PM, Somnath Roy wrote:
>> Mark,
>> I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
>> swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
>> throughput is linearly scaling. But, as an individual radosgw performance is 
>> very low we need to put lots of radosgw/apache server combination to get 
>> very high throughput. I guess that will be a problem.
>> I will try to do some profiling and share the data.
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-
>> From: ceph-devel-ow...@vger.kernel.org
>> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
>> Sent: Thursday, September 26, 2013 3:33 PM
>> To: Somnath Roy
>> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
>> Subject: Re: [ceph-users] Scaling radosgw module
>>
>> It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
>> seeing if having two copies of the benchmark going at the same time on two 
>> separate RGW servers increases aggregate throughput.
>>
>> Also, it may be worth tracking down latencies with messenger debugging
>> enabled, but I'm afraid I'm pretty bogged down right now and probably
>> wouldn't be able to look at it for a while. :(
>>
>> Mark
>>
>> On 09/26/2013 05:15 PM, Somnath Roy wrote:
>>> Hi Mark,
>>> FYI, I tried with wip-6286-dumpling release and the results are the same 
>>> for me. The radosgw throughput is around ~6x slower than the single rados 
>>> bench output!
>>>Any other suggestion ?
>>>
>>> Thanks & Regards
>>> Somnath
>>> -Original Message-
>>> From: Somnath Roy
>>> Sent: Friday, September 20, 2013 4:08 PM
>>> To: 'Mark Nelson'
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: RE: [ceph-users] Scaling radosgw module
>>>
>>> Hi Mark,
>>> It's a test cluster and I will try with the new release.
>>> As I mentioned in the mail, I think number of rados client instance is the 
>>> limitation. Could you please let me know how many rados client instance the 
>>> radosgw daemon is instantiating ? Is it configurable somehow ?
>>>
>>> Thanks & Regards
>>> Somnath
>>>
>>> -Original Message-
>>> From: Mark Nelson [mailto:mark.nel...@inktank.com]
>>> Sent: Friday, September 20, 2013 4:02 PM
>>> To: Somnath Roy
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] Scaling radosgw module
>>>
>>> On 09/20/2013 05:49 PM, Somnath Roy wrote:
>>>> Hi Mark,
>>>> Thanks for your quick response.
>>>> I tried adding the 'num_container = 100' in the job file and found that 
>>>> the performance actually decreasing with that option. I am getting around 
>>>> 1K less iops after putting this. Another observation is that in order to 
>>>> get back the earlier iops I need to restart the radosgw service. Just 
>>>> removing the num_container option from the job file and running 
>>>> swift-bench again is not he

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Mark,
One more thing, all my test is with rgw cache enabled , disabling the cache the 
performance is around 3x slower.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org 
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 26, 2013 3:59 PM
To: Mark Nelson
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: RE: [ceph-users] Scaling radosgw module

Nope...With one client hitting the radaosgw , the daemon cpu usage is going up 
till 400-450% i.e taking in avg 4 core..In one client scenario, the server node 
(having radosgw + osds) cpu usage is ~80% idle and out of the 20% usage bulk is 
consumed by radosgw.

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Thursday, September 26, 2013 3:50 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: Re: [ceph-users] Scaling radosgw module

Ah, that's very good to know!

And RGW CPU usage you said was low?

Mark

On 09/26/2013 05:40 PM, Somnath Roy wrote:
> Mark,
> I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
> swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
> throughput is linearly scaling. But, as an individual radosgw performance is 
> very low we need to put lots of radosgw/apache server combination to get very 
> high throughput. I guess that will be a problem.
> I will try to do some profiling and share the data.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org 
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
> Sent: Thursday, September 26, 2013 3:33 PM
> To: Somnath Roy
> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
> Subject: Re: [ceph-users] Scaling radosgw module
>
> It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
> seeing if having two copies of the benchmark going at the same time on two 
> separate RGW servers increases aggregate throughput.
>
> Also, it may be worth tracking down latencies with messenger debugging 
> enabled, but I'm afraid I'm pretty bogged down right now and probably 
> wouldn't be able to look at it for a while. :(
>
> Mark
>
> On 09/26/2013 05:15 PM, Somnath Roy wrote:
>> Hi Mark,
>> FYI, I tried with wip-6286-dumpling release and the results are the same for 
>> me. The radosgw throughput is around ~6x slower than the single rados bench 
>> output!
>>Any other suggestion ?
>>
>> Thanks & Regards
>> Somnath
>> -Original Message-
>> From: Somnath Roy
>> Sent: Friday, September 20, 2013 4:08 PM
>> To: 'Mark Nelson'
>> Cc: ceph-users@lists.ceph.com
>> Subject: RE: [ceph-users] Scaling radosgw module
>>
>> Hi Mark,
>> It's a test cluster and I will try with the new release.
>> As I mentioned in the mail, I think number of rados client instance is the 
>> limitation. Could you please let me know how many rados client instance the 
>> radosgw daemon is instantiating ? Is it configurable somehow ?
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-
>> From: Mark Nelson [mailto:mark.nel...@inktank.com]
>> Sent: Friday, September 20, 2013 4:02 PM
>> To: Somnath Roy
>> Cc: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Scaling radosgw module
>>
>> On 09/20/2013 05:49 PM, Somnath Roy wrote:
>>> Hi Mark,
>>> Thanks for your quick response.
>>> I tried adding the 'num_container = 100' in the job file and found that the 
>>> performance actually decreasing with that option. I am getting around 1K 
>>> less iops after putting this. Another observation is that in order to get 
>>> back the earlier iops I need to restart the radosgw service. Just removing 
>>> the num_container option from the job file and running swift-bench again is 
>>> not helping. It seems something radosgw service is caching here.
>>
>> Interesting, that means you aren't being limited by a single container index 
>> only residing on 1 OSD.  Eventually that might be a limitation, but not here 
>> apparently.
>>
>>>
>>> Regarding object size, I have tried with larger object size as well but 
>>> iops are much lower in those cases.
>>
>> Yeah, the larger the object size the lower the iops, but potentially the 
>> higher the MB/s throughput.
>>
>>>
>>> Regarding moving it to the ceph wip branch, can I just upgrade from 
>>>

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Nope...With one client hitting the radaosgw , the daemon cpu usage is going up 
till 400-450% i.e taking in avg 4 core..In one client scenario, the server node 
(having radosgw + osds) cpu usage is ~80% idle and out of the 20% usage bulk is 
consumed by radosgw.

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com] 
Sent: Thursday, September 26, 2013 3:50 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: Re: [ceph-users] Scaling radosgw module

Ah, that's very good to know!

And RGW CPU usage you said was low?

Mark

On 09/26/2013 05:40 PM, Somnath Roy wrote:
> Mark,
> I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
> swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
> throughput is linearly scaling. But, as an individual radosgw performance is 
> very low we need to put lots of radosgw/apache server combination to get very 
> high throughput. I guess that will be a problem.
> I will try to do some profiling and share the data.
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org 
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
> Sent: Thursday, September 26, 2013 3:33 PM
> To: Somnath Roy
> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
> Subject: Re: [ceph-users] Scaling radosgw module
>
> It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
> seeing if having two copies of the benchmark going at the same time on two 
> separate RGW servers increases aggregate throughput.
>
> Also, it may be worth tracking down latencies with messenger debugging 
> enabled, but I'm afraid I'm pretty bogged down right now and probably 
> wouldn't be able to look at it for a while. :(
>
> Mark
>
> On 09/26/2013 05:15 PM, Somnath Roy wrote:
>> Hi Mark,
>> FYI, I tried with wip-6286-dumpling release and the results are the same for 
>> me. The radosgw throughput is around ~6x slower than the single rados bench 
>> output!
>>Any other suggestion ?
>>
>> Thanks & Regards
>> Somnath
>> -----Original Message-
>> From: Somnath Roy
>> Sent: Friday, September 20, 2013 4:08 PM
>> To: 'Mark Nelson'
>> Cc: ceph-users@lists.ceph.com
>> Subject: RE: [ceph-users] Scaling radosgw module
>>
>> Hi Mark,
>> It's a test cluster and I will try with the new release.
>> As I mentioned in the mail, I think number of rados client instance is the 
>> limitation. Could you please let me know how many rados client instance the 
>> radosgw daemon is instantiating ? Is it configurable somehow ?
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-
>> From: Mark Nelson [mailto:mark.nel...@inktank.com]
>> Sent: Friday, September 20, 2013 4:02 PM
>> To: Somnath Roy
>> Cc: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Scaling radosgw module
>>
>> On 09/20/2013 05:49 PM, Somnath Roy wrote:
>>> Hi Mark,
>>> Thanks for your quick response.
>>> I tried adding the 'num_container = 100' in the job file and found that the 
>>> performance actually decreasing with that option. I am getting around 1K 
>>> less iops after putting this. Another observation is that in order to get 
>>> back the earlier iops I need to restart the radosgw service. Just removing 
>>> the num_container option from the job file and running swift-bench again is 
>>> not helping. It seems something radosgw service is caching here.
>>
>> Interesting, that means you aren't being limited by a single container index 
>> only residing on 1 OSD.  Eventually that might be a limitation, but not here 
>> apparently.
>>
>>>
>>> Regarding object size, I have tried with larger object size as well but 
>>> iops are much lower in those cases.
>>
>> Yeah, the larger the object size the lower the iops, but potentially the 
>> higher the MB/s throughput.
>>
>>>
>>> Regarding moving it to the ceph wip branch, can I just upgrade from 
>>> dumpling ?
>>
>> Yes, it's actually just dumpling with a minor code change, however given 
>> that it's development code I would not recommend doing this if the cluster 
>> is in production.
>>
>>>
>>> Thanks & Regards
>>> Somnath
>>>
>>> -Original Message-
>>> From: ceph-users-boun...@lists.ceph.com 
>>> [mailto:ceph-users-boun...@lists.ceph.c

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

Ah, that's very good to know!

And RGW CPU usage you said was low?

Mark

On 09/26/2013 05:40 PM, Somnath Roy wrote:

Mark,
I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
throughput is linearly scaling. But, as an individual radosgw performance is 
very low we need to put lots of radosgw/apache server combination to get very 
high throughput. I guess that will be a problem.
I will try to do some profiling and share the data.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org 
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Thursday, September 26, 2013 3:33 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: Re: [ceph-users] Scaling radosgw module

It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
seeing if having two copies of the benchmark going at the same time on two 
separate RGW servers increases aggregate throughput.

Also, it may be worth tracking down latencies with messenger debugging enabled, 
but I'm afraid I'm pretty bogged down right now and probably wouldn't be able 
to look at it for a while. :(

Mark

On 09/26/2013 05:15 PM, Somnath Roy wrote:

Hi Mark,
FYI, I tried with wip-6286-dumpling release and the results are the same for 
me. The radosgw throughput is around ~6x slower than the single rados bench 
output!
   Any other suggestion ?

Thanks & Regards
Somnath
-Original Message-
From: Somnath Roy
Sent: Friday, September 20, 2013 4:08 PM
To: 'Mark Nelson'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Scaling radosgw module

Hi Mark,
It's a test cluster and I will try with the new release.
As I mentioned in the mail, I think number of rados client instance is the 
limitation. Could you please let me know how many rados client instance the 
radosgw daemon is instantiating ? Is it configurable somehow ?

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Friday, September 20, 2013 4:02 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

On 09/20/2013 05:49 PM, Somnath Roy wrote:

Hi Mark,
Thanks for your quick response.
I tried adding the 'num_container = 100' in the job file and found that the 
performance actually decreasing with that option. I am getting around 1K less 
iops after putting this. Another observation is that in order to get back the 
earlier iops I need to restart the radosgw service. Just removing the 
num_container option from the job file and running swift-bench again is not 
helping. It seems something radosgw service is caching here.


Interesting, that means you aren't being limited by a single container index 
only residing on 1 OSD.  Eventually that might be a limitation, but not here 
apparently.



Regarding object size, I have tried with larger object size as well but iops 
are much lower in those cases.


Yeah, the larger the object size the lower the iops, but potentially the higher 
the MB/s throughput.



Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ?


Yes, it's actually just dumpling with a minor code change, however given that 
it's development code I would not recommend doing this if the cluster is in 
production.



Thanks & Regards
Somnath

-Original Message-
From: ceph-users-boun...@lists.ceph.com
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
Sent: Friday, September 20, 2013 3:03 PM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

Hi,

A couple of things that might be worth trying:

use multiple containers in swift-bench.  Newer versions should
support this.  Also, if this is a test cluster, you may want to try
the ceph
wip-6286 branch as we have a rather major performance improvement in it when 
dealing with small objects.

Beyond that, we are currently investigating performance slowdowns due to OSD 
directory splitting behavior that can crop up with many (millions) of objects.  
This we think has potentially been hitting a couple of folks that have very 
large object collections.

Thanks,
Mark

On 09/20/2013 04:57 PM, Somnath Roy wrote:

Hi,
I am running Ceph on a 3 node cluster and each of my server node is running 10 
OSDs, one for each disk. I have one admin node and all the nodes are connected 
with 2 X 10G network. One network is for cluster and other one configured as 
public network.

All the OSD journals are on SSDs.

I started with rados bench command to benchmark the read performance of this 
Cluster on a large pool (~10K PGs) and found that each rados client has a 
limitation. Each client can only drive up to a certain mark. Each server  node 
cpu utilization shows it is  around 85-90% idle and the admin no

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Mark,
I did set up 3 radosgw servers in 3 server nodes and the tested with 3 
swift-bench client hitting 3 radosgw in the same time. I saw the aggregated 
throughput is linearly scaling. But, as an individual radosgw performance is 
very low we need to put lots of radosgw/apache server combination to get very 
high throughput. I guess that will be a problem.
I will try to do some profiling and share the data.

Thanks & Regards
Somnath

-Original Message-
From: ceph-devel-ow...@vger.kernel.org 
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Thursday, September 26, 2013 3:33 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org; Anirban Ray
Subject: Re: [ceph-users] Scaling radosgw module

It's kind of annoying, but it may be worth setting up a 2nd RGW server and 
seeing if having two copies of the benchmark going at the same time on two 
separate RGW servers increases aggregate throughput.

Also, it may be worth tracking down latencies with messenger debugging enabled, 
but I'm afraid I'm pretty bogged down right now and probably wouldn't be able 
to look at it for a while. :(

Mark

On 09/26/2013 05:15 PM, Somnath Roy wrote:
> Hi Mark,
> FYI, I tried with wip-6286-dumpling release and the results are the same for 
> me. The radosgw throughput is around ~6x slower than the single rados bench 
> output!
>   Any other suggestion ?
>
> Thanks & Regards
> Somnath
> -Original Message-
> From: Somnath Roy
> Sent: Friday, September 20, 2013 4:08 PM
> To: 'Mark Nelson'
> Cc: ceph-users@lists.ceph.com
> Subject: RE: [ceph-users] Scaling radosgw module
>
> Hi Mark,
> It's a test cluster and I will try with the new release.
> As I mentioned in the mail, I think number of rados client instance is the 
> limitation. Could you please let me know how many rados client instance the 
> radosgw daemon is instantiating ? Is it configurable somehow ?
>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: Mark Nelson [mailto:mark.nel...@inktank.com]
> Sent: Friday, September 20, 2013 4:02 PM
> To: Somnath Roy
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Scaling radosgw module
>
> On 09/20/2013 05:49 PM, Somnath Roy wrote:
>> Hi Mark,
>> Thanks for your quick response.
>> I tried adding the 'num_container = 100' in the job file and found that the 
>> performance actually decreasing with that option. I am getting around 1K 
>> less iops after putting this. Another observation is that in order to get 
>> back the earlier iops I need to restart the radosgw service. Just removing 
>> the num_container option from the job file and running swift-bench again is 
>> not helping. It seems something radosgw service is caching here.
>
> Interesting, that means you aren't being limited by a single container index 
> only residing on 1 OSD.  Eventually that might be a limitation, but not here 
> apparently.
>
>>
>> Regarding object size, I have tried with larger object size as well but iops 
>> are much lower in those cases.
>
> Yeah, the larger the object size the lower the iops, but potentially the 
> higher the MB/s throughput.
>
>>
>> Regarding moving it to the ceph wip branch, can I just upgrade from dumpling 
>> ?
>
> Yes, it's actually just dumpling with a minor code change, however given that 
> it's development code I would not recommend doing this if the cluster is in 
> production.
>
>>
>> Thanks & Regards
>> Somnath
>>
>> -Original Message-
>> From: ceph-users-boun...@lists.ceph.com 
>> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
>> Sent: Friday, September 20, 2013 3:03 PM
>> To: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Scaling radosgw module
>>
>> Hi,
>>
>> A couple of things that might be worth trying:
>>
>> use multiple containers in swift-bench.  Newer versions should 
>> support this.  Also, if this is a test cluster, you may want to try 
>> the ceph
>> wip-6286 branch as we have a rather major performance improvement in it when 
>> dealing with small objects.
>>
>> Beyond that, we are currently investigating performance slowdowns due to OSD 
>> directory splitting behavior that can crop up with many (millions) of 
>> objects.  This we think has potentially been hitting a couple of folks that 
>> have very large object collections.
>>
>> Thanks,
>> Mark
>>
>> On 09/20/2013 04:57 PM, Somnath Roy wrote:
>>> Hi,
>>> I am running Ceph on a 3 node cluster and each of my server node is running 
>>&

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

It's kind of annoying, but it may be worth setting up a 2nd RGW server 
and seeing if having two copies of the benchmark going at the same time 
on two separate RGW servers increases aggregate throughput.


Also, it may be worth tracking down latencies with messenger debugging 
enabled, but I'm afraid I'm pretty bogged down right now and probably 
wouldn't be able to look at it for a while. :(


Mark

On 09/26/2013 05:15 PM, Somnath Roy wrote:

Hi Mark,
FYI, I tried with wip-6286-dumpling release and the results are the same for 
me. The radosgw throughput is around ~6x slower than the single rados bench 
output!
  Any other suggestion ?

Thanks & Regards
Somnath
-Original Message-
From: Somnath Roy
Sent: Friday, September 20, 2013 4:08 PM
To: 'Mark Nelson'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Scaling radosgw module

Hi Mark,
It's a test cluster and I will try with the new release.
As I mentioned in the mail, I think number of rados client instance is the 
limitation. Could you please let me know how many rados client instance the 
radosgw daemon is instantiating ? Is it configurable somehow ?

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Friday, September 20, 2013 4:02 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

On 09/20/2013 05:49 PM, Somnath Roy wrote:

Hi Mark,
Thanks for your quick response.
I tried adding the 'num_container = 100' in the job file and found that the 
performance actually decreasing with that option. I am getting around 1K less 
iops after putting this. Another observation is that in order to get back the 
earlier iops I need to restart the radosgw service. Just removing the 
num_container option from the job file and running swift-bench again is not 
helping. It seems something radosgw service is caching here.


Interesting, that means you aren't being limited by a single container index 
only residing on 1 OSD.  Eventually that might be a limitation, but not here 
apparently.



Regarding object size, I have tried with larger object size as well but iops 
are much lower in those cases.


Yeah, the larger the object size the lower the iops, but potentially the higher 
the MB/s throughput.



Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ?


Yes, it's actually just dumpling with a minor code change, however given that 
it's development code I would not recommend doing this if the cluster is in 
production.



Thanks & Regards
Somnath

-Original Message-
From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
Sent: Friday, September 20, 2013 3:03 PM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

Hi,

A couple of things that might be worth trying:

use multiple containers in swift-bench.  Newer versions should support this.  
Also, if this is a test cluster, you may want to try the ceph
wip-6286 branch as we have a rather major performance improvement in it when 
dealing with small objects.

Beyond that, we are currently investigating performance slowdowns due to OSD 
directory splitting behavior that can crop up with many (millions) of objects.  
This we think has potentially been hitting a couple of folks that have very 
large object collections.

Thanks,
Mark

On 09/20/2013 04:57 PM, Somnath Roy wrote:

Hi,
I am running Ceph on a 3 node cluster and each of my server node is running 10 
OSDs, one for each disk. I have one admin node and all the nodes are connected 
with 2 X 10G network. One network is for cluster and other one configured as 
public network.

All the OSD journals are on SSDs.

I started with rados bench command to benchmark the read performance of this 
Cluster on a large pool (~10K PGs) and found that each rados client has a 
limitation. Each client can only drive up to a certain mark. Each server  node 
cpu utilization shows it is  around 85-90% idle and the admin node (from where 
rados client is running) is around ~80-85% idle. I am trying with 4K object 
size.

I started running more clients on the admin node and the performance is scaling 
till it hits the client cpu limit. Server still has the cpu of 30-35% idle.

Now, I am behind radosgw and in one of the server node I installed the required 
modules like apache, fastcgi, radosgw etc.  I configured swift bench and 
started benchmarking. Here is my swift-bench job script.

[bench]
auth = http:///auth
user = somroy:swift
key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
concurrency = 64
object_size = 4096
num_objects = 1000
num_gets = 20
delete = yes
auth_version = 1.0


First of all,  the read performance I am getting with one radosgw is more than 
5x slower than what I am getting with one rbd client or one rados bench client. 
Is this expected ? Here is my ceph.conf ra

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Hi Mark, 
FYI, I tried with wip-6286-dumpling release and the results are the same for 
me. The radosgw throughput is around ~6x slower than the single rados bench 
output!
 Any other suggestion ?

Thanks & Regards
Somnath
-Original Message-
From: Somnath Roy 
Sent: Friday, September 20, 2013 4:08 PM
To: 'Mark Nelson'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Scaling radosgw module

Hi Mark,
It's a test cluster and I will try with the new release. 
As I mentioned in the mail, I think number of rados client instance is the 
limitation. Could you please let me know how many rados client instance the 
radosgw daemon is instantiating ? Is it configurable somehow ?

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Friday, September 20, 2013 4:02 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

On 09/20/2013 05:49 PM, Somnath Roy wrote:
> Hi Mark,
> Thanks for your quick response.
> I tried adding the 'num_container = 100' in the job file and found that the 
> performance actually decreasing with that option. I am getting around 1K less 
> iops after putting this. Another observation is that in order to get back the 
> earlier iops I need to restart the radosgw service. Just removing the 
> num_container option from the job file and running swift-bench again is not 
> helping. It seems something radosgw service is caching here.

Interesting, that means you aren't being limited by a single container index 
only residing on 1 OSD.  Eventually that might be a limitation, but not here 
apparently.

>
> Regarding object size, I have tried with larger object size as well but iops 
> are much lower in those cases.

Yeah, the larger the object size the lower the iops, but potentially the higher 
the MB/s throughput.

>
> Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ?

Yes, it's actually just dumpling with a minor code change, however given that 
it's development code I would not recommend doing this if the cluster is in 
production.

>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-users-boun...@lists.ceph.com 
> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
> Sent: Friday, September 20, 2013 3:03 PM
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Scaling radosgw module
>
> Hi,
>
> A couple of things that might be worth trying:
>
> use multiple containers in swift-bench.  Newer versions should support this.  
> Also, if this is a test cluster, you may want to try the ceph
> wip-6286 branch as we have a rather major performance improvement in it when 
> dealing with small objects.
>
> Beyond that, we are currently investigating performance slowdowns due to OSD 
> directory splitting behavior that can crop up with many (millions) of 
> objects.  This we think has potentially been hitting a couple of folks that 
> have very large object collections.
>
> Thanks,
> Mark
>
> On 09/20/2013 04:57 PM, Somnath Roy wrote:
>> Hi,
>> I am running Ceph on a 3 node cluster and each of my server node is running 
>> 10 OSDs, one for each disk. I have one admin node and all the nodes are 
>> connected with 2 X 10G network. One network is for cluster and other one 
>> configured as public network.
>>
>> All the OSD journals are on SSDs.
>>
>> I started with rados bench command to benchmark the read performance of this 
>> Cluster on a large pool (~10K PGs) and found that each rados client has a 
>> limitation. Each client can only drive up to a certain mark. Each server  
>> node cpu utilization shows it is  around 85-90% idle and the admin node 
>> (from where rados client is running) is around ~80-85% idle. I am trying 
>> with 4K object size.
>>
>> I started running more clients on the admin node and the performance is 
>> scaling till it hits the client cpu limit. Server still has the cpu of 
>> 30-35% idle.
>>
>> Now, I am behind radosgw and in one of the server node I installed the 
>> required modules like apache, fastcgi, radosgw etc.  I configured swift 
>> bench and started benchmarking. Here is my swift-bench job script.
>>
>> [bench]
>> auth = http:///auth
>> user = somroy:swift
>> key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
>> concurrency = 64
>> object_size = 4096
>> num_objects = 1000
>> num_gets = 20
>> delete = yes
>> auth_version = 1.0
>>
>>
>> First of all,  the read performance I am getting with one radosgw is more 
>> than 5x slower than what I am getting with one rbd client or one rados bench 

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Hi Mark,
It's a test cluster and I will try with the new release. 
As I mentioned in the mail, I think number of rados client instance is the 
limitation. Could you please let me know how many rados client instance the 
radosgw daemon is instantiating ? Is it configurable somehow ?

Thanks & Regards
Somnath

-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com] 
Sent: Friday, September 20, 2013 4:02 PM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

On 09/20/2013 05:49 PM, Somnath Roy wrote:
> Hi Mark,
> Thanks for your quick response.
> I tried adding the 'num_container = 100' in the job file and found that the 
> performance actually decreasing with that option. I am getting around 1K less 
> iops after putting this. Another observation is that in order to get back the 
> earlier iops I need to restart the radosgw service. Just removing the 
> num_container option from the job file and running swift-bench again is not 
> helping. It seems something radosgw service is caching here.

Interesting, that means you aren't being limited by a single container index 
only residing on 1 OSD.  Eventually that might be a limitation, but not here 
apparently.

>
> Regarding object size, I have tried with larger object size as well but iops 
> are much lower in those cases.

Yeah, the larger the object size the lower the iops, but potentially the 
higher the MB/s throughput.

>
> Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ?

Yes, it's actually just dumpling with a minor code change, however given 
that it's development code I would not recommend doing this if the 
cluster is in production.

>
> Thanks & Regards
> Somnath
>
> -Original Message-
> From: ceph-users-boun...@lists.ceph.com 
> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
> Sent: Friday, September 20, 2013 3:03 PM
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Scaling radosgw module
>
> Hi,
>
> A couple of things that might be worth trying:
>
> use multiple containers in swift-bench.  Newer versions should support this.  
> Also, if this is a test cluster, you may want to try the ceph
> wip-6286 branch as we have a rather major performance improvement in it when 
> dealing with small objects.
>
> Beyond that, we are currently investigating performance slowdowns due to OSD 
> directory splitting behavior that can crop up with many (millions) of 
> objects.  This we think has potentially been hitting a couple of folks that 
> have very large object collections.
>
> Thanks,
> Mark
>
> On 09/20/2013 04:57 PM, Somnath Roy wrote:
>> Hi,
>> I am running Ceph on a 3 node cluster and each of my server node is running 
>> 10 OSDs, one for each disk. I have one admin node and all the nodes are 
>> connected with 2 X 10G network. One network is for cluster and other one 
>> configured as public network.
>>
>> All the OSD journals are on SSDs.
>>
>> I started with rados bench command to benchmark the read performance of this 
>> Cluster on a large pool (~10K PGs) and found that each rados client has a 
>> limitation. Each client can only drive up to a certain mark. Each server  
>> node cpu utilization shows it is  around 85-90% idle and the admin node 
>> (from where rados client is running) is around ~80-85% idle. I am trying 
>> with 4K object size.
>>
>> I started running more clients on the admin node and the performance is 
>> scaling till it hits the client cpu limit. Server still has the cpu of 
>> 30-35% idle.
>>
>> Now, I am behind radosgw and in one of the server node I installed the 
>> required modules like apache, fastcgi, radosgw etc.  I configured swift 
>> bench and started benchmarking. Here is my swift-bench job script.
>>
>> [bench]
>> auth = http:///auth
>> user = somroy:swift
>> key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
>> concurrency = 64
>> object_size = 4096
>> num_objects = 1000
>> num_gets = 20
>> delete = yes
>> auth_version = 1.0
>>
>>
>> First of all,  the read performance I am getting with one radosgw is more 
>> than 5x slower than what I am getting with one rbd client or one rados bench 
>> client. Is this expected ? Here is my ceph.conf radosgw config option.
>>
>> [client.radosgw.gateway]
>> host = emsserver1
>> keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path =
>> /tmp/radosgw.sock log_file = /var/log/ceph/radosgw.log rgw_dns_name =
>>  rgw_ops_log_rados = false debug_rgw = 0 rgw_thread_pool_size =
>> 300
>>
>> The server node 

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

On 09/20/2013 05:49 PM, Somnath Roy wrote:

Hi Mark,
Thanks for your quick response.
I tried adding the 'num_container = 100' in the job file and found that the 
performance actually decreasing with that option. I am getting around 1K less 
iops after putting this. Another observation is that in order to get back the 
earlier iops I need to restart the radosgw service. Just removing the 
num_container option from the job file and running swift-bench again is not 
helping. It seems something radosgw service is caching here.


Interesting, that means you aren't being limited by a single container 
index only residing on 1 OSD.  Eventually that might be a limitation, 
but not here apparently.




Regarding object size, I have tried with larger object size as well but iops 
are much lower in those cases.


Yeah, the larger the object size the lower the iops, but potentially the 
higher the MB/s throughput.




Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ?


Yes, it's actually just dumpling with a minor code change, however given 
that it's development code I would not recommend doing this if the 
cluster is in production.




Thanks & Regards
Somnath

-Original Message-
From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
Sent: Friday, September 20, 2013 3:03 PM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

Hi,

A couple of things that might be worth trying:

use multiple containers in swift-bench.  Newer versions should support this.  
Also, if this is a test cluster, you may want to try the ceph
wip-6286 branch as we have a rather major performance improvement in it when 
dealing with small objects.

Beyond that, we are currently investigating performance slowdowns due to OSD 
directory splitting behavior that can crop up with many (millions) of objects.  
This we think has potentially been hitting a couple of folks that have very 
large object collections.

Thanks,
Mark

On 09/20/2013 04:57 PM, Somnath Roy wrote:

Hi,
I am running Ceph on a 3 node cluster and each of my server node is running 10 
OSDs, one for each disk. I have one admin node and all the nodes are connected 
with 2 X 10G network. One network is for cluster and other one configured as 
public network.

All the OSD journals are on SSDs.

I started with rados bench command to benchmark the read performance of this 
Cluster on a large pool (~10K PGs) and found that each rados client has a 
limitation. Each client can only drive up to a certain mark. Each server  node 
cpu utilization shows it is  around 85-90% idle and the admin node (from where 
rados client is running) is around ~80-85% idle. I am trying with 4K object 
size.

I started running more clients on the admin node and the performance is scaling 
till it hits the client cpu limit. Server still has the cpu of 30-35% idle.

Now, I am behind radosgw and in one of the server node I installed the required 
modules like apache, fastcgi, radosgw etc.  I configured swift bench and 
started benchmarking. Here is my swift-bench job script.

[bench]
auth = http:///auth
user = somroy:swift
key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
concurrency = 64
object_size = 4096
num_objects = 1000
num_gets = 20
delete = yes
auth_version = 1.0


First of all,  the read performance I am getting with one radosgw is more than 
5x slower than what I am getting with one rbd client or one rados bench client. 
Is this expected ? Here is my ceph.conf radosgw config option.

[client.radosgw.gateway]
host = emsserver1
keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path =
/tmp/radosgw.sock log_file = /var/log/ceph/radosgw.log rgw_dns_name =
 rgw_ops_log_rados = false debug_rgw = 0 rgw_thread_pool_size =
300

The server node (where radosgw is also present) avg cpu utilization is very low 
(~75-80% idle). Out of the ~20% consumption, I saw radosgw is consuming bulk of 
the cpu in the node and ceph-osds are not much. The other two server node is 
~95% idle ; 10 ceph-osds are consuming this of total 5% of cpu !!

So, clearly, I am not able to generate much load on the cluster.
So, I tried to run multiple swift-bench instances with the same job , all 
hitting the single radosgw instance. I saw no improvement on the performance, 
each instance iops is almost now = (single instance iop/number of swift-bench 
instance). The aggregated iops is remaining almost same as of single instance.

This means we are hitting the single client instance limit here too.
My question is, for all the requests radosgw is opening only single client 
connection to the object store ?
If so, is there any configuration like 'noshare' option in case of rbd that 
Josh pointed out in my earlier mail ?

If not, how a single radosgw instance will scale ?

Appreciate, if anybody can help me on this.

Thanks & Regards
Somnath



PLE

Re: [ceph-users] Scaling radosgw module

[Somnath Roy]

Hi Mark,
Thanks for your quick response.
I tried adding the 'num_container = 100' in the job file and found that the 
performance actually decreasing with that option. I am getting around 1K less 
iops after putting this. Another observation is that in order to get back the 
earlier iops I need to restart the radosgw service. Just removing the 
num_container option from the job file and running swift-bench again is not 
helping. It seems something radosgw service is caching here.

Regarding object size, I have tried with larger object size as well but iops 
are much lower in those cases.

Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ?

Thanks & Regards
Somnath

-Original Message-
From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson
Sent: Friday, September 20, 2013 3:03 PM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Scaling radosgw module

Hi,

A couple of things that might be worth trying:

use multiple containers in swift-bench.  Newer versions should support this.  
Also, if this is a test cluster, you may want to try the ceph
wip-6286 branch as we have a rather major performance improvement in it when 
dealing with small objects.

Beyond that, we are currently investigating performance slowdowns due to OSD 
directory splitting behavior that can crop up with many (millions) of objects.  
This we think has potentially been hitting a couple of folks that have very 
large object collections.

Thanks,
Mark

On 09/20/2013 04:57 PM, Somnath Roy wrote:
> Hi,
> I am running Ceph on a 3 node cluster and each of my server node is running 
> 10 OSDs, one for each disk. I have one admin node and all the nodes are 
> connected with 2 X 10G network. One network is for cluster and other one 
> configured as public network.
>
> All the OSD journals are on SSDs.
>
> I started with rados bench command to benchmark the read performance of this 
> Cluster on a large pool (~10K PGs) and found that each rados client has a 
> limitation. Each client can only drive up to a certain mark. Each server  
> node cpu utilization shows it is  around 85-90% idle and the admin node (from 
> where rados client is running) is around ~80-85% idle. I am trying with 4K 
> object size.
>
> I started running more clients on the admin node and the performance is 
> scaling till it hits the client cpu limit. Server still has the cpu of 30-35% 
> idle.
>
> Now, I am behind radosgw and in one of the server node I installed the 
> required modules like apache, fastcgi, radosgw etc.  I configured swift bench 
> and started benchmarking. Here is my swift-bench job script.
>
> [bench]
> auth = http:///auth
> user = somroy:swift
> key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
> concurrency = 64
> object_size = 4096
> num_objects = 1000
> num_gets = 20
> delete = yes
> auth_version = 1.0
>
>
> First of all,  the read performance I am getting with one radosgw is more 
> than 5x slower than what I am getting with one rbd client or one rados bench 
> client. Is this expected ? Here is my ceph.conf radosgw config option.
>
> [client.radosgw.gateway]
> host = emsserver1
> keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path = 
> /tmp/radosgw.sock log_file = /var/log/ceph/radosgw.log rgw_dns_name = 
>  rgw_ops_log_rados = false debug_rgw = 0 rgw_thread_pool_size = 
> 300
>
> The server node (where radosgw is also present) avg cpu utilization is very 
> low (~75-80% idle). Out of the ~20% consumption, I saw radosgw is consuming 
> bulk of the cpu in the node and ceph-osds are not much. The other two server 
> node is ~95% idle ; 10 ceph-osds are consuming this of total 5% of cpu !!
>
> So, clearly, I am not able to generate much load on the cluster.
> So, I tried to run multiple swift-bench instances with the same job , all 
> hitting the single radosgw instance. I saw no improvement on the performance, 
> each instance iops is almost now = (single instance iop/number of swift-bench 
> instance). The aggregated iops is remaining almost same as of single instance.
>
> This means we are hitting the single client instance limit here too.
> My question is, for all the requests radosgw is opening only single client 
> connection to the object store ?
> If so, is there any configuration like 'noshare' option in case of rbd that 
> Josh pointed out in my earlier mail ?
>
> If not, how a single radosgw instance will scale ?
>
> Appreciate, if anybody can help me on this.
>
> Thanks & Regards
> Somnath
>
> 
>
> PLEASE NOTE: The information contained in this electronic mail message is 
> intended only for the use of the designated recipient(s) named above. If the 
> reade

Re: [ceph-users] Scaling radosgw module

[Mark Nelson]

Hi,

A couple of things that might be worth trying:

use multiple containers in swift-bench.  Newer versions should support 
this.  Also, if this is a test cluster, you may want to try the ceph 
wip-6286 branch as we have a rather major performance improvement in it 
when dealing with small objects.


Beyond that, we are currently investigating performance slowdowns due to 
OSD directory splitting behavior that can crop up with many (millions) 
of objects.  This we think has potentially been hitting a couple of 
folks that have very large object collections.


Thanks,
Mark

On 09/20/2013 04:57 PM, Somnath Roy wrote:

Hi,
I am running Ceph on a 3 node cluster and each of my server node is running 10 
OSDs, one for each disk. I have one admin node and all the nodes are connected 
with 2 X 10G network. One network is for cluster and other one configured as 
public network.

All the OSD journals are on SSDs.

I started with rados bench command to benchmark the read performance of this 
Cluster on a large pool (~10K PGs) and found that each rados client has a 
limitation. Each client can only drive up to a certain mark. Each server  node 
cpu utilization shows it is  around 85-90% idle and the admin node (from where 
rados client is running) is around ~80-85% idle. I am trying with 4K object 
size.

I started running more clients on the admin node and the performance is scaling 
till it hits the client cpu limit. Server still has the cpu of 30-35% idle.

Now, I am behind radosgw and in one of the server node I installed the required 
modules like apache, fastcgi, radosgw etc.  I configured swift bench and 
started benchmarking. Here is my swift-bench job script.

[bench]
auth = http:///auth
user = somroy:swift
key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
concurrency = 64
object_size = 4096
num_objects = 1000
num_gets = 20
delete = yes
auth_version = 1.0


First of all,  the read performance I am getting with one radosgw is more than 
5x slower than what I am getting with one rbd client or one rados bench client. 
Is this expected ? Here is my ceph.conf radosgw config option.

[client.radosgw.gateway]
host = emsserver1
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
rgw_dns_name = 
rgw_ops_log_rados = false
debug_rgw = 0
rgw_thread_pool_size = 300

The server node (where radosgw is also present) avg cpu utilization is very low 
(~75-80% idle). Out of the ~20% consumption, I saw radosgw is consuming bulk of 
the cpu in the node and ceph-osds are not much. The other two server node is 
~95% idle ; 10 ceph-osds are consuming this of total 5% of cpu !!

So, clearly, I am not able to generate much load on the cluster.
So, I tried to run multiple swift-bench instances with the same job , all 
hitting the single radosgw instance. I saw no improvement on the performance, 
each instance iops is almost now = (single instance iop/number of swift-bench 
instance). The aggregated iops is remaining almost same as of single instance.

This means we are hitting the single client instance limit here too.
My question is, for all the requests radosgw is opening only single client 
connection to the object store ?
If so, is there any configuration like 'noshare' option in case of rbd that 
Josh pointed out in my earlier mail ?

If not, how a single radosgw instance will scale ?

Appreciate, if anybody can help me on this.

Thanks & Regards
Somnath



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Scaling radosgw module

[Somnath Roy]

Hi,
I am running Ceph on a 3 node cluster and each of my server node is running 10 
OSDs, one for each disk. I have one admin node and all the nodes are connected 
with 2 X 10G network. One network is for cluster and other one configured as 
public network.

All the OSD journals are on SSDs.

I started with rados bench command to benchmark the read performance of this 
Cluster on a large pool (~10K PGs) and found that each rados client has a 
limitation. Each client can only drive up to a certain mark. Each server  node 
cpu utilization shows it is  around 85-90% idle and the admin node (from where 
rados client is running) is around ~80-85% idle. I am trying with 4K object 
size.

I started running more clients on the admin node and the performance is scaling 
till it hits the client cpu limit. Server still has the cpu of 30-35% idle.

Now, I am behind radosgw and in one of the server node I installed the required 
modules like apache, fastcgi, radosgw etc.  I configured swift bench and 
started benchmarking. Here is my swift-bench job script.

[bench]
auth = http:///auth
user = somroy:swift
key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt
concurrency = 64
object_size = 4096
num_objects = 1000
num_gets = 20
delete = yes
auth_version = 1.0


First of all,  the read performance I am getting with one radosgw is more than 
5x slower than what I am getting with one rbd client or one rados bench client. 
Is this expected ? Here is my ceph.conf radosgw config option.

[client.radosgw.gateway]
host = emsserver1
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
rgw_dns_name = 
rgw_ops_log_rados = false
debug_rgw = 0
rgw_thread_pool_size = 300

The server node (where radosgw is also present) avg cpu utilization is very low 
(~75-80% idle). Out of the ~20% consumption, I saw radosgw is consuming bulk of 
the cpu in the node and ceph-osds are not much. The other two server node is 
~95% idle ; 10 ceph-osds are consuming this of total 5% of cpu !!

So, clearly, I am not able to generate much load on the cluster.
So, I tried to run multiple swift-bench instances with the same job , all 
hitting the single radosgw instance. I saw no improvement on the performance, 
each instance iops is almost now = (single instance iop/number of swift-bench 
instance). The aggregated iops is remaining almost same as of single instance.

This means we are hitting the single client instance limit here too.
My question is, for all the requests radosgw is opening only single client 
connection to the object store ?
If so, is there any configuration like 'noshare' option in case of rbd that 
Josh pointed out in my earlier mail ?

If not, how a single radosgw instance will scale ?

Appreciate, if anybody can help me on this.

Thanks & Regards
Somnath



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com