Re: [ceph-users] luminous vs jewel rbd performance

[Rafael Lopez]

Hey Linh...have not but if it makes any difference we are still using
filestore.

On 16 Nov. 2017 12:31, "Linh Vu" <v...@unimelb.edu.au> wrote:

> Noticed that you're on 12.2.0 Raf. 12.2.1 fixed a lot of performance
> issues from 12.2.0 for us on Luminous/Bluestore. Have you tried upgrading
> to it?
> --
> *From:* ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of
> Rafael Lopez <rafael.lo...@monash.edu>
> *Sent:* Thursday, 16 November 2017 11:59:14 AM
> *To:* Mark Nelson
> *Cc:* ceph-users
> *Subject:* Re: [ceph-users] luminous vs jewel rbd performance
>
> Hi Mark,
>
> Sorry for the late reply... I have been away on vacation/openstack summit
> etc for over a month now and looking at this again.
>
> Yeah the snippet was a bit misleading. The fio file contains small block
> jobs as well as big block jobs:
>
> [write-rbd1-4m-depth1]
> rbdname=rbd-tester-fio
> bs=4m
> iodepth=1
> rw=write
> stonewall
> [write-rbd2-4m-depth16]
> rbdname=rbd-tester-fio-2
> bs=4m
> iodepth=16
> rw=write
> stonewall
>
> [read-rbd1-4m-depth1]
> rbdname=rbd-tester-fio
> bs=4m
> iodepth=1
> rw=read
> stonewall
> [read-rbd2-4m-depth16]
> rbdname=rbd-tester-fio-2
> bs=4m
> iodepth=16
> rw=read
> stonewall
>
> The performance hit is more noticeable on bigblock, I think up to 10x
> slower on some runs but as a percentage it seems to affect a small block
> workload too. I understand that runs will vary... I wish I had more runs
> from before upgrading to luminous but I only have that single set of
> results. Regardless, I cannot come close to that single set of results
> since upgrading to luminous.
> I understand the caching stuff you mentioned, however we have not changed
> any of that config since the upgrade and the fio job is exactly the same.
> So if I do many runs on luminous throughout the course of a day, including
> when we think the cluster is least busy, we should be able to come pretty
> close to the jewel result on at least one of the runs or is my thinking
> flawed?
>
> Sage mentioned at openstack that there was a perf regression with librbd
> which will be fixed in 12.2.2 are you aware of this? If so can you send
> me the link to the bug?
>
> Cheers,
> Raf
>
>
> On 22 September 2017 at 00:31, Mark Nelson <mnel...@redhat.com> wrote:
>
> Hi Rafael,
>
> In the original email you mentioned 4M block size, seq read, but here it
> looks like you are doing 4k writes?  Can you clarify?  If you are doing 4k
> direct sequential writes with iodepth=1 and are also using librbd cache,
> please make sure that librbd is set to writeback mode in both cases.  RBD
> by default will not kick into WB mode until it sees a flush request, and
> the librbd engine in fio doesn't issue one before a test is started.  It
> can be pretty easy to end up in a situation where writeback cache is active
> on some tests but not others if you aren't careful.  IE If one of your
> tests was done after a flush and the other was not, you'd likely see a
> dramatic difference in performance during this test.
>
> You can avoid this by telling librbd to always use WB mode (at least when
> benchmarking):
>
> rbd cache writethrough until flush = false
>
> Mark
>
>
> On 09/20/2017 01:51 AM, Rafael Lopez wrote:
>
> Hi Alexandre,
>
> Yeah we are using filestore for the moment with luminous. With regards
> to client, I tried both jewel and luminous librbd versions against the
> luminous cluster - similar results.
>
> I am running fio on a physical machine with fio rbd engine. This is a
> snippet of the fio config for the runs (the complete jobfile adds
> variations of read/write/block size/iodepth).
>
> [global]
> ioengine=rbd
> clientname=cinder-volume
> pool=rbd-bronze
> invalidate=1
> ramp_time=5
> runtime=30
> time_based
> direct=1
>
> [write-rbd1-4k-depth1]
> rbdname=rbd-tester-fio
> bs=4k
> iodepth=1
> rw=write
> stonewall
>
> [write-rbd2-4k-depth16]
> rbdname=rbd-tester-fio-2
> bs=4k
> iodepth=16
> rw=write
> stonewall
>
> Raf
>
> On 20 September 2017 at 16:43, Alexandre DERUMIER <aderum...@odiso.com
> <mailto:aderum...@odiso.com>> wrote:
>
> Hi
>
> so, you use also filestore on luminous ?
>
> do you have also upgraded librbd on client ? (are you benching
> inside a qemu machine ? or directly with fio-rbd ?)
>
>
>
> (I'm going to do a lot of benchmarks in coming week, I'll post
> results on mailing soon.)
>
>
>
> - Mail original -
> De: "Rafael Lopez" <rafael.lo...@monash.edu
> <mailto

Re: [ceph-users] luminous vs jewel rbd performance

[Linh Vu]

Noticed that you're on 12.2.0 Raf. 12.2.1 fixed a lot of performance issues 
from 12.2.0 for us on Luminous/Bluestore. Have you tried upgrading to it?


From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of Rafael Lopez 
<rafael.lo...@monash.edu>
Sent: Thursday, 16 November 2017 11:59:14 AM
To: Mark Nelson
Cc: ceph-users
Subject: Re: [ceph-users] luminous vs jewel rbd performance

Hi Mark,

Sorry for the late reply... I have been away on vacation/openstack summit etc 
for over a month now and looking at this again.

Yeah the snippet was a bit misleading. The fio file contains small block jobs 
as well as big block jobs:

[write-rbd1-4m-depth1]
rbdname=rbd-tester-fio
bs=4m
iodepth=1
rw=write
stonewall
[write-rbd2-4m-depth16]
rbdname=rbd-tester-fio-2
bs=4m
iodepth=16
rw=write
stonewall

[read-rbd1-4m-depth1]
rbdname=rbd-tester-fio
bs=4m
iodepth=1
rw=read
stonewall
[read-rbd2-4m-depth16]
rbdname=rbd-tester-fio-2
bs=4m
iodepth=16
rw=read
stonewall

The performance hit is more noticeable on bigblock, I think up to 10x slower on 
some runs but as a percentage it seems to affect a small block workload too. I 
understand that runs will vary... I wish I had more runs from before upgrading 
to luminous but I only have that single set of results. Regardless, I cannot 
come close to that single set of results since upgrading to luminous.
I understand the caching stuff you mentioned, however we have not changed any 
of that config since the upgrade and the fio job is exactly the same. So if I 
do many runs on luminous throughout the course of a day, including when we 
think the cluster is least busy, we should be able to come pretty close to the 
jewel result on at least one of the runs or is my thinking flawed?

Sage mentioned at openstack that there was a perf regression with librbd which 
will be fixed in 12.2.2 are you aware of this? If so can you send me the 
link to the bug?

Cheers,
Raf


On 22 September 2017 at 00:31, Mark Nelson 
<mnel...@redhat.com<mailto:mnel...@redhat.com>> wrote:
Hi Rafael,

In the original email you mentioned 4M block size, seq read, but here it looks 
like you are doing 4k writes?  Can you clarify?  If you are doing 4k direct 
sequential writes with iodepth=1 and are also using librbd cache, please make 
sure that librbd is set to writeback mode in both cases.  RBD by default will 
not kick into WB mode until it sees a flush request, and the librbd engine in 
fio doesn't issue one before a test is started.  It can be pretty easy to end 
up in a situation where writeback cache is active on some tests but not others 
if you aren't careful.  IE If one of your tests was done after a flush and the 
other was not, you'd likely see a dramatic difference in performance during 
this test.

You can avoid this by telling librbd to always use WB mode (at least when 
benchmarking):

rbd cache writethrough until flush = false

Mark


On 09/20/2017 01:51 AM, Rafael Lopez wrote:
Hi Alexandre,

Yeah we are using filestore for the moment with luminous. With regards
to client, I tried both jewel and luminous librbd versions against the
luminous cluster - similar results.

I am running fio on a physical machine with fio rbd engine. This is a
snippet of the fio config for the runs (the complete jobfile adds
variations of read/write/block size/iodepth).

[global]
ioengine=rbd
clientname=cinder-volume
pool=rbd-bronze
invalidate=1
ramp_time=5
runtime=30
time_based
direct=1

[write-rbd1-4k-depth1]
rbdname=rbd-tester-fio
bs=4k
iodepth=1
rw=write
stonewall

[write-rbd2-4k-depth16]
rbdname=rbd-tester-fio-2
bs=4k
iodepth=16
rw=write
stonewall

Raf

On 20 September 2017 at 16:43, Alexandre DERUMIER 
<aderum...@odiso.com<mailto:aderum...@odiso.com>
<mailto:aderum...@odiso.com<mailto:aderum...@odiso.com>>> wrote:

Hi

so, you use also filestore on luminous ?

do you have also upgraded librbd on client ? (are you benching
inside a qemu machine ? or directly with fio-rbd ?)



(I'm going to do a lot of benchmarks in coming week, I'll post
results on mailing soon.)



- Mail original -
De: "Rafael Lopez" <rafael.lo...@monash.edu<mailto:rafael.lo...@monash.edu>
<mailto:rafael.lo...@monash.edu<mailto:rafael.lo...@monash.edu>>>
À: "ceph-users" <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
<mailto:ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>>>

Envoyé: Mercredi 20 Septembre 2017 08:17:23
Objet: [ceph-users] luminous vs jewel rbd performance

hey guys.
wondering if anyone else has done some solid benchmarking of jewel
vs luminous, in particular on the same cluster that has been
upgraded (same cluster, client and config).

we have recently upgraded a cluster from 10.2.9 to 12.2.0, and
unfortunately i only captured results from a single fio (librb

Re: [ceph-users] luminous vs jewel rbd performance

[Rafael Lopez]

Hi Mark,

Sorry for the late reply... I have been away on vacation/openstack summit
etc for over a month now and looking at this again.

Yeah the snippet was a bit misleading. The fio file contains small block
jobs as well as big block jobs:

[write-rbd1-4m-depth1]
rbdname=rbd-tester-fio
bs=4m
iodepth=1
rw=write
stonewall
[write-rbd2-4m-depth16]
rbdname=rbd-tester-fio-2
bs=4m
iodepth=16
rw=write
stonewall

[read-rbd1-4m-depth1]
rbdname=rbd-tester-fio
bs=4m
iodepth=1
rw=read
stonewall
[read-rbd2-4m-depth16]
rbdname=rbd-tester-fio-2
bs=4m
iodepth=16
rw=read
stonewall

The performance hit is more noticeable on bigblock, I think up to 10x
slower on some runs but as a percentage it seems to affect a small block
workload too. I understand that runs will vary... I wish I had more runs
from before upgrading to luminous but I only have that single set of
results. Regardless, I cannot come close to that single set of results
since upgrading to luminous.
I understand the caching stuff you mentioned, however we have not changed
any of that config since the upgrade and the fio job is exactly the same.
So if I do many runs on luminous throughout the course of a day, including
when we think the cluster is least busy, we should be able to come pretty
close to the jewel result on at least one of the runs or is my thinking
flawed?

Sage mentioned at openstack that there was a perf regression with librbd
which will be fixed in 12.2.2 are you aware of this? If so can you send
me the link to the bug?

Cheers,
Raf


On 22 September 2017 at 00:31, Mark Nelson  wrote:

> Hi Rafael,
>
> In the original email you mentioned 4M block size, seq read, but here it
> looks like you are doing 4k writes?  Can you clarify?  If you are doing 4k
> direct sequential writes with iodepth=1 and are also using librbd cache,
> please make sure that librbd is set to writeback mode in both cases.  RBD
> by default will not kick into WB mode until it sees a flush request, and
> the librbd engine in fio doesn't issue one before a test is started.  It
> can be pretty easy to end up in a situation where writeback cache is active
> on some tests but not others if you aren't careful.  IE If one of your
> tests was done after a flush and the other was not, you'd likely see a
> dramatic difference in performance during this test.
>
> You can avoid this by telling librbd to always use WB mode (at least when
> benchmarking):
>
> rbd cache writethrough until flush = false
>
> Mark
>
>
> On 09/20/2017 01:51 AM, Rafael Lopez wrote:
>
>> Hi Alexandre,
>>
>> Yeah we are using filestore for the moment with luminous. With regards
>> to client, I tried both jewel and luminous librbd versions against the
>> luminous cluster - similar results.
>>
>> I am running fio on a physical machine with fio rbd engine. This is a
>> snippet of the fio config for the runs (the complete jobfile adds
>> variations of read/write/block size/iodepth).
>>
>> [global]
>> ioengine=rbd
>> clientname=cinder-volume
>> pool=rbd-bronze
>> invalidate=1
>> ramp_time=5
>> runtime=30
>> time_based
>> direct=1
>>
>> [write-rbd1-4k-depth1]
>> rbdname=rbd-tester-fio
>> bs=4k
>> iodepth=1
>> rw=write
>> stonewall
>>
>> [write-rbd2-4k-depth16]
>> rbdname=rbd-tester-fio-2
>> bs=4k
>> iodepth=16
>> rw=write
>> stonewall
>>
>> Raf
>>
>> On 20 September 2017 at 16:43, Alexandre DERUMIER > > wrote:
>>
>> Hi
>>
>> so, you use also filestore on luminous ?
>>
>> do you have also upgraded librbd on client ? (are you benching
>> inside a qemu machine ? or directly with fio-rbd ?)
>>
>>
>>
>> (I'm going to do a lot of benchmarks in coming week, I'll post
>> results on mailing soon.)
>>
>>
>>
>> - Mail original -
>> De: "Rafael Lopez" > >
>> À: "ceph-users" > >
>>
>> Envoyé: Mercredi 20 Septembre 2017 08:17:23
>> Objet: [ceph-users] luminous vs jewel rbd performance
>>
>> hey guys.
>> wondering if anyone else has done some solid benchmarking of jewel
>> vs luminous, in particular on the same cluster that has been
>> upgraded (same cluster, client and config).
>>
>> we have recently upgraded a cluster from 10.2.9 to 12.2.0, and
>> unfortunately i only captured results from a single fio (librbd) run
>> with a few jobs in it before upgrading. i have run the same fio
>> jobfile many times at different times of the day since upgrading,
>> and been unable to produce a close match to the pre-upgrade (jewel)
>> run from the same client. one particular job is significantly slower
>> (4M block size, iodepth=1, seq read), up to 10x in one run.
>>
>> i realise i havent supplied much detail and it could be dozens of
>> things, but i just wanted to see if anyone else had done more
>> quantitative benchmarking or had similar 

Re: [ceph-users] luminous vs jewel rbd performance

[Mark Nelson]

Hi Rafael,

In the original email you mentioned 4M block size, seq read, but here it 
looks like you are doing 4k writes?  Can you clarify?  If you are doing 
4k direct sequential writes with iodepth=1 and are also using librbd 
cache, please make sure that librbd is set to writeback mode in both 
cases.  RBD by default will not kick into WB mode until it sees a flush 
request, and the librbd engine in fio doesn't issue one before a test is 
started.  It can be pretty easy to end up in a situation where writeback 
cache is active on some tests but not others if you aren't careful.  IE 
If one of your tests was done after a flush and the other was not, you'd 
likely see a dramatic difference in performance during this test.


You can avoid this by telling librbd to always use WB mode (at least 
when benchmarking):


rbd cache writethrough until flush = false

Mark

On 09/20/2017 01:51 AM, Rafael Lopez wrote:

Hi Alexandre,

Yeah we are using filestore for the moment with luminous. With regards
to client, I tried both jewel and luminous librbd versions against the
luminous cluster - similar results.

I am running fio on a physical machine with fio rbd engine. This is a
snippet of the fio config for the runs (the complete jobfile adds
variations of read/write/block size/iodepth).

[global]
ioengine=rbd
clientname=cinder-volume
pool=rbd-bronze
invalidate=1
ramp_time=5
runtime=30
time_based
direct=1

[write-rbd1-4k-depth1]
rbdname=rbd-tester-fio
bs=4k
iodepth=1
rw=write
stonewall

[write-rbd2-4k-depth16]
rbdname=rbd-tester-fio-2
bs=4k
iodepth=16
rw=write
stonewall

Raf

On 20 September 2017 at 16:43, Alexandre DERUMIER > wrote:

Hi

so, you use also filestore on luminous ?

do you have also upgraded librbd on client ? (are you benching
inside a qemu machine ? or directly with fio-rbd ?)



(I'm going to do a lot of benchmarks in coming week, I'll post
results on mailing soon.)



- Mail original -
De: "Rafael Lopez" >
À: "ceph-users" >
Envoyé: Mercredi 20 Septembre 2017 08:17:23
Objet: [ceph-users] luminous vs jewel rbd performance

hey guys.
wondering if anyone else has done some solid benchmarking of jewel
vs luminous, in particular on the same cluster that has been
upgraded (same cluster, client and config).

we have recently upgraded a cluster from 10.2.9 to 12.2.0, and
unfortunately i only captured results from a single fio (librbd) run
with a few jobs in it before upgrading. i have run the same fio
jobfile many times at different times of the day since upgrading,
and been unable to produce a close match to the pre-upgrade (jewel)
run from the same client. one particular job is significantly slower
(4M block size, iodepth=1, seq read), up to 10x in one run.

i realise i havent supplied much detail and it could be dozens of
things, but i just wanted to see if anyone else had done more
quantitative benchmarking or had similar experiences. keep in mind
all we changed was daemons were restarted to use luminous code,
everything else exactly the same. granted it is possible that
some/all osds had some runtime config injected that differs from
now, but i'm fairly confident this is not the case as they were
recently restarted (on jewel code) after OS upgrades.

cheers,
Raf

___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





--
*Rafael Lopez*
Research Devops Engineer
Monash University eResearch Centre

T: +61 3 9905 9118 
M: +61 (0)427682670 
E: rafael.lo...@monash.edu 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] luminous vs jewel rbd performance

[Alexandre DERUMIER]

ok, thanks.

I'll try to do same bench in coming week, I'll you in touch with results.


- Mail original -
De: "Rafael Lopez" <rafael.lo...@monash.edu>
À: "aderumier" <aderum...@odiso.com>
Cc: "ceph-users" <ceph-users@lists.ceph.com>
Envoyé: Mercredi 20 Septembre 2017 08:51:22
Objet: Re: [ceph-users] luminous vs jewel rbd performance

Hi Alexandre, 
Yeah we are using filestore for the moment with luminous. With regards to 
client, I tried both jewel and luminous librbd versions against the luminous 
cluster - similar results. 

I am running fio on a physical machine with fio rbd engine. This is a snippet 
of the fio config for the runs (the complete jobfile adds variations of 
read/write/block size/iodepth). 

[global] 
ioengine=rbd 
clientname=cinder-volume 
pool=rbd-bronze 
invalidate=1 
ramp_time=5 
runtime=30 
time_based 
direct=1 

[write-rbd1-4k-depth1] 
rbdname=rbd-tester-fio 
bs=4k 
iodepth=1 
rw=write 
stonewall 

[write-rbd2-4k-depth16] 
rbdname=rbd-tester-fio-2 
bs=4k 
iodepth=16 
rw=write 
stonewall 

Raf 

On 20 September 2017 at 16:43, Alexandre DERUMIER < [ 
mailto:aderum...@odiso.com | aderum...@odiso.com ] > wrote: 


Hi 

so, you use also filestore on luminous ? 

do you have also upgraded librbd on client ? (are you benching inside a qemu 
machine ? or directly with fio-rbd ?) 



(I'm going to do a lot of benchmarks in coming week, I'll post results on 
mailing soon.) 



- Mail original - 
De: "Rafael Lopez" < [ mailto:rafael.lo...@monash.edu | rafael.lo...@monash.edu 
] > 
À: "ceph-users" < [ mailto:ceph-users@lists.ceph.com | 
ceph-users@lists.ceph.com ] > 
Envoyé: Mercredi 20 Septembre 2017 08:17:23 
Objet: [ceph-users] luminous vs jewel rbd performance 

hey guys. 
wondering if anyone else has done some solid benchmarking of jewel vs luminous, 
in particular on the same cluster that has been upgraded (same cluster, client 
and config). 

we have recently upgraded a cluster from 10.2.9 to 12.2.0, and unfortunately i 
only captured results from a single fio (librbd) run with a few jobs in it 
before upgrading. i have run the same fio jobfile many times at different times 
of the day since upgrading, and been unable to produce a close match to the 
pre-upgrade (jewel) run from the same client. one particular job is 
significantly slower (4M block size, iodepth=1, seq read), up to 10x in one 
run. 

i realise i havent supplied much detail and it could be dozens of things, but i 
just wanted to see if anyone else had done more quantitative benchmarking or 
had similar experiences. keep in mind all we changed was daemons were restarted 
to use luminous code, everything else exactly the same. granted it is possible 
that some/all osds had some runtime config injected that differs from now, but 
i'm fairly confident this is not the case as they were recently restarted (on 
jewel code) after OS upgrades. 

cheers, 
Raf 

___ 
ceph-users mailing list 
[ mailto:ceph-users@lists.ceph.com | ceph-users@lists.ceph.com ] 
[ http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com | 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ] 







-- 
Rafael Lopez 
Research Devops Engineer 
Monash University eResearch Centre 

T: [ tel:%2B61%203%209905%209118 | +61 3 9905 9118 ] 
M: [ tel:%2B61%204%2027682%20670 | +61 (0)427682670 ] 
E: [ mailto:rafael.lo...@monash.edu | rafael.lo...@monash.edu ] 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] luminous vs jewel rbd performance

[Rafael Lopez]

Hi Alexandre,

Yeah we are using filestore for the moment with luminous. With regards to
client, I tried both jewel and luminous librbd versions against the
luminous cluster - similar results.

I am running fio on a physical machine with fio rbd engine. This is a
snippet of the fio config for the runs (the complete jobfile adds
variations of read/write/block size/iodepth).

[global]
ioengine=rbd
clientname=cinder-volume
pool=rbd-bronze
invalidate=1
ramp_time=5
runtime=30
time_based
direct=1

[write-rbd1-4k-depth1]
rbdname=rbd-tester-fio
bs=4k
iodepth=1
rw=write
stonewall

[write-rbd2-4k-depth16]
rbdname=rbd-tester-fio-2
bs=4k
iodepth=16
rw=write
stonewall

Raf

On 20 September 2017 at 16:43, Alexandre DERUMIER 
wrote:

> Hi
>
> so, you use also filestore on luminous ?
>
> do you have also upgraded librbd on client ? (are you benching inside a
> qemu machine ? or directly with fio-rbd ?)
>
>
>
> (I'm going to do a lot of benchmarks in coming week, I'll post results on
> mailing soon.)
>
>
>
> - Mail original -
> De: "Rafael Lopez" 
> À: "ceph-users" 
> Envoyé: Mercredi 20 Septembre 2017 08:17:23
> Objet: [ceph-users] luminous vs jewel rbd performance
>
> hey guys.
> wondering if anyone else has done some solid benchmarking of jewel vs
> luminous, in particular on the same cluster that has been upgraded (same
> cluster, client and config).
>
> we have recently upgraded a cluster from 10.2.9 to 12.2.0, and
> unfortunately i only captured results from a single fio (librbd) run with a
> few jobs in it before upgrading. i have run the same fio jobfile many times
> at different times of the day since upgrading, and been unable to produce a
> close match to the pre-upgrade (jewel) run from the same client. one
> particular job is significantly slower (4M block size, iodepth=1, seq
> read), up to 10x in one run.
>
> i realise i havent supplied much detail and it could be dozens of things,
> but i just wanted to see if anyone else had done more quantitative
> benchmarking or had similar experiences. keep in mind all we changed was
> daemons were restarted to use luminous code, everything else exactly the
> same. granted it is possible that some/all osds had some runtime config
> injected that differs from now, but i'm fairly confident this is not the
> case as they were recently restarted (on jewel code) after OS upgrades.
>
> cheers,
> Raf
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
*Rafael Lopez*
Research Devops Engineer
Monash University eResearch Centre

T: +61 3 9905 9118
M: +61 (0)427682670 <%2B61%204%2027682%20670>
E: rafael.lo...@monash.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] luminous vs jewel rbd performance

[Alexandre DERUMIER]

Hi

so, you use also filestore on luminous ?

do you have also upgraded librbd on client ? (are you benching inside a qemu 
machine ? or directly with fio-rbd ?)



(I'm going to do a lot of benchmarks in coming week, I'll post results on 
mailing soon.)



- Mail original -
De: "Rafael Lopez" 
À: "ceph-users" 
Envoyé: Mercredi 20 Septembre 2017 08:17:23
Objet: [ceph-users] luminous vs jewel rbd performance

hey guys. 
wondering if anyone else has done some solid benchmarking of jewel vs luminous, 
in particular on the same cluster that has been upgraded (same cluster, client 
and config). 

we have recently upgraded a cluster from 10.2.9 to 12.2.0, and unfortunately i 
only captured results from a single fio (librbd) run with a few jobs in it 
before upgrading. i have run the same fio jobfile many times at different times 
of the day since upgrading, and been unable to produce a close match to the 
pre-upgrade (jewel) run from the same client. one particular job is 
significantly slower (4M block size, iodepth=1, seq read), up to 10x in one 
run. 

i realise i havent supplied much detail and it could be dozens of things, but i 
just wanted to see if anyone else had done more quantitative benchmarking or 
had similar experiences. keep in mind all we changed was daemons were restarted 
to use luminous code, everything else exactly the same. granted it is possible 
that some/all osds had some runtime config injected that differs from now, but 
i'm fairly confident this is not the case as they were recently restarted (on 
jewel code) after OS upgrades. 

cheers, 
Raf 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com