Re: [ceph-users] performance decrease after continuous run

[RDS]

I have seen this and some of our big customers have also seen this. I was using 
8TB HDDs and when running small tests using a fresh HDD setup, these tests 
resulted in very good performance. I then loaded the ceph cluster so each of 
the 8TB HDD used 4TB and reran the same tests. performance was cut in 1/2. This 
is using the default settings on how ceph creates the dirs and sub-directories 
on each OSD. You can flatten out this dir structure so the structure is more 
wide than deep and performance is improved. Check out the 
filestore_merge_threshold and filestore_split_multiple settings.
Rick
> On Jul 20, 2016, at 3:19 PM, Kane Kim  wrote:
> 
> Hello,
> 
> I was running cosbench for some time and noticed sharp consistent performance 
> decrease at some point.
> 
> Image is here: http://take.ms/rorPw 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] performance decrease after continuous run

[K K]

Hello.



Maybe deep-scrub starting at this time? Четверг, 21 июля 2016, 11:10 +05:00 от 
Christian Balzer :
>
>
>Hello,
>
>On Wed, 20 Jul 2016 12:19:07 -0700 Kane Kim wrote:
>
>> Hello,
>> 
>> I was running cosbench for some time and noticed sharp consistent
>> performance decrease at some point.
>> 
>> Image is here:  http://take.ms/rorPw
>
>Unfortunately that statement and image is rather pointless without about a
>metric ton of relevant data.
>
>Start with describing your setup (HW and SW) in intricate detail.
>
>Any changes to default configuration parameters. 
>
>Then see if you can determine with tools like atop if there is a
>bottleneck with your setup when it hits that drop, on OSDs nodes and on
>your RGW (since you mention cosbench). 
>
>Christian
>-- 
>Christian BalzerNetwork/Systems Engineer 
>ch...@gol.com Global OnLine Japan/Rakuten Communications
>http://www.gol.com/
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] performance decrease after continuous run

[Christian Balzer]

Hello,

On Wed, 20 Jul 2016 12:19:07 -0700 Kane Kim wrote:

> Hello,
> 
> I was running cosbench for some time and noticed sharp consistent
> performance decrease at some point.
> 
> Image is here: http://take.ms/rorPw

Unfortunately that statement and image is rather pointless without about a
metric ton of relevant data.

Start with describing your setup (HW and SW) in intricate detail.

Any changes to default configuration parameters. 

Then see if you can determine with tools like atop if there is a
bottleneck with your setup when it hits that drop, on OSDs nodes and on
your RGW (since you mention cosbench). 

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] performance decrease after continuous run

[Kane Kim]

Hello,

I was running cosbench for some time and noticed sharp consistent
performance decrease at some point.

Image is here: http://take.ms/rorPw
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com