Re: [ceph-users] Performance issues on Luminous
This drives are running as osd, not as journal.
I think I can't understand is, why the performance of using rados bench
with 1 thread is 3 times slower? Ceph osd bench shows good results.
In my opinion it could be a 20% less speed, because of software overhead.
I read the blog post
(http://ceph.com/geen-categorie/quick-analysis-of-the-ceph-io-layer/)
and it will be good to have an explanation about it.
@Mark, Could you tell us (community) is it normal behaviour of these
tests? What is the difference?
BR,
Rafał Wądołowski
On 05.01.2018 19:29, Christian Wuerdig wrote:
You should do your reference test with dd with oflag=direct,dsync
direct will only bypass the cache while dsync will fsync on every
block which is much closer to reality of what ceph is doing afaik
On Thu, Jan 4, 2018 at 9:54 PM, Rafał Wądołowski
wrote:
Hi folks,
I am currently benchmarking my cluster for an performance issue and I have
no idea, what is going on. I am using these devices in qemu.
Ceph version 12.2.2
Infrastructure:
3 x Ceph-mon
11 x Ceph-osd
Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
96GB RAM
2x E5-2650 v4
4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
I had tested it with rados bench:
# rados bench -p rbdbench 30 write -t 1
Total time run: 30.055677
Total writes made: 1199
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 159.571
Stddev Bandwidth: 6.83601
Max bandwidth (MB/sec): 168
Min bandwidth (MB/sec): 140
Average IOPS: 39
Stddev IOPS:1
Max IOPS: 42
Min IOPS: 35
Average Latency(s): 0.0250656
Stddev Latency(s): 0.00321545
Max latency(s): 0.0471699
Min latency(s): 0.0206325
# ceph tell osd.0 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"bytes_per_sec": 414199397
}
Testing osd directly
# dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
When I do dd inside vm (bs=4M wih direct), I have result like in rados
bench.
I think that the speed should be arround ~400MB/s.
Is there any new parameters for rbd in luminous? Maybe I forgot about some
performance tricks? If more information needed feel free to ask.
--
BR,
Rafal Wadolowski
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance issues on Luminous
You should do your reference test with dd with oflag=direct,dsync
direct will only bypass the cache while dsync will fsync on every
block which is much closer to reality of what ceph is doing afaik
On Thu, Jan 4, 2018 at 9:54 PM, Rafał Wądołowski
wrote:
> Hi folks,
>
> I am currently benchmarking my cluster for an performance issue and I have
> no idea, what is going on. I am using these devices in qemu.
>
> Ceph version 12.2.2
>
> Infrastructure:
>
> 3 x Ceph-mon
>
> 11 x Ceph-osd
>
> Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
>
> 96GB RAM
>
> 2x E5-2650 v4
>
> 4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
>
>
> I had tested it with rados bench:
>
> # rados bench -p rbdbench 30 write -t 1
>
> Total time run: 30.055677
> Total writes made: 1199
> Write size: 4194304
> Object size:4194304
> Bandwidth (MB/sec): 159.571
> Stddev Bandwidth: 6.83601
> Max bandwidth (MB/sec): 168
> Min bandwidth (MB/sec): 140
> Average IOPS: 39
> Stddev IOPS:1
> Max IOPS: 42
> Min IOPS: 35
> Average Latency(s): 0.0250656
> Stddev Latency(s): 0.00321545
> Max latency(s): 0.0471699
> Min latency(s): 0.0206325
>
> # ceph tell osd.0 bench
> {
> "bytes_written": 1073741824,
> "blocksize": 4194304,
> "bytes_per_sec": 414199397
> }
>
> Testing osd directly
>
> # dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
> 100+0 records in
> 100+0 records out
> 419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
>
> When I do dd inside vm (bs=4M wih direct), I have result like in rados
> bench.
>
> I think that the speed should be arround ~400MB/s.
>
> Is there any new parameters for rbd in luminous? Maybe I forgot about some
> performance tricks? If more information needed feel free to ask.
>
> --
> BR,
> Rafal Wadolowski
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance issues on Luminous
Do not use consumer SSD for OSD. Especially for journal disk.
If you use consumer SSD, please consider add some dedicated SSD Enterprise
for journal disk. Ratio should be 1:2 or 1:4 (1 SSD Enterprise with 4 SSD
Consumer).
Best Regards,
On Fri, Jan 5, 2018 at 3:20 PM, Marc Roos wrote:
>
>
> Maybe because of this 850 evo / 850 pro listed here as 1.9MB/s 1.5MB/s
>
> http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-
> test-if-your-ssd-is-suitable-as-a-journal-device/
>
>
>
>
> -Original Message-
> From: Rafał Wądołowski [mailto:rwadolow...@cloudferro.com]
> Sent: donderdag 4 januari 2018 16:56
> To: c...@elchaka.de; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Performance issues on Luminous
>
> I have size of 2.
>
> We know about this risk and we accept it, but we still don't know why
> performance so so bad.
>
>
> Cheers,
>
> Rafał Wądołowski
>
>
> On 04.01.2018 16:51, c...@elchaka.de wrote:
>
>
> I assume you have size of 3 then divide your expected 400 with 3
> and you are not far Away from what you get...
>
> In Addition you should Never use Consumer grade ssds for ceph as
> they will be reach the DWPD very soon...
>
>
> Am 4. Januar 2018 09:54:55 MEZ schrieb "Rafał Wądołowski"
> <mailto:rwadolow...@cloudferro.com> :
>
> Hi folks,
>
> I am currently benchmarking my cluster for an performance
> issue and I
> have no idea, what is going on. I am using these devices in
> qemu.
>
> Ceph version 12.2.2
>
> Infrastructure:
>
> 3 x Ceph-mon
>
> 11 x Ceph-osd
>
> Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
>
> 96GB RAM
>
> 2x E5-2650 v4
>
> 4x10G Network (2 seperate bounds for cluster and public)
> with
> MTU 9000
>
>
> I had tested it with rados bench:
>
> # rados bench -p rbdbench 30 write -t 1
>
> Total time run: 30.055677
> Total writes made: 1199
> Write size: 4194304
> Object size:4194304
> Bandwidth (MB/sec): 159.571
> Stddev Bandwidth: 6.83601
> Max bandwidth (MB/sec): 168
> Min bandwidth (MB/sec): 140
> Average IOPS: 39
> Stddev IOPS:1
> Max IOPS: 42
> Min IOPS: 35
> Average Latency(s): 0.0250656
> Stddev Latency(s): 0.00321545
> Max latency(s): 0.0471699
> Min latency(s): 0.0206325
>
> # ceph tell osd.0 bench
> {
> "bytes_written": 1073741824,
> "blocksize": 4194304,
> "bytes_per_sec": 414199397
> }
>
> Testing osd directly
>
> # dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
> 100+0 records in
> 100+0 records out
> 419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417
> MB/s
>
> When I do dd inside vm (bs=4M wih direct), I have result
> like
> in rados
> bench.
>
> I think that the speed should be arround ~400MB/s.
>
> Is there any new parameters for rbd in luminous? Maybe I
> forgot about
> some performance tricks? If more information needed feel
> free
> to ask.
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
==
Nghia Than
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Performance issues on Luminous
Maybe because of this 850 evo / 850 pro listed here as 1.9MB/s 1.5MB/s
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
-Original Message-
From: Rafał Wądołowski [mailto:rwadolow...@cloudferro.com]
Sent: donderdag 4 januari 2018 16:56
To: c...@elchaka.de; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Performance issues on Luminous
I have size of 2.
We know about this risk and we accept it, but we still don't know why
performance so so bad.
Cheers,
Rafał Wądołowski
On 04.01.2018 16:51, c...@elchaka.de wrote:
I assume you have size of 3 then divide your expected 400 with 3
and you are not far Away from what you get...
In Addition you should Never use Consumer grade ssds for ceph as
they will be reach the DWPD very soon...
Am 4. Januar 2018 09:54:55 MEZ schrieb "Rafał Wądołowski"
<mailto:rwadolow...@cloudferro.com> :
Hi folks,
I am currently benchmarking my cluster for an performance
issue and I
have no idea, what is going on. I am using these devices in
qemu.
Ceph version 12.2.2
Infrastructure:
3 x Ceph-mon
11 x Ceph-osd
Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
96GB RAM
2x E5-2650 v4
4x10G Network (2 seperate bounds for cluster and public) with
MTU 9000
I had tested it with rados bench:
# rados bench -p rbdbench 30 write -t 1
Total time run: 30.055677
Total writes made: 1199
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 159.571
Stddev Bandwidth: 6.83601
Max bandwidth (MB/sec): 168
Min bandwidth (MB/sec): 140
Average IOPS: 39
Stddev IOPS:1
Max IOPS: 42
Min IOPS: 35
Average Latency(s): 0.0250656
Stddev Latency(s): 0.00321545
Max latency(s): 0.0471699
Min latency(s): 0.0206325
# ceph tell osd.0 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"bytes_per_sec": 414199397
}
Testing osd directly
# dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
When I do dd inside vm (bs=4M wih direct), I have result like
in rados
bench.
I think that the speed should be arround ~400MB/s.
Is there any new parameters for rbd in luminous? Maybe I
forgot about
some performance tricks? If more information needed feel free
to ask.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance issues on Luminous
They are configured with bluestore.
The network, cpu and disk are doing nothing. I was observing with atop,
iostat, top.
Similiar hardware configuration I have on jewel (with filestore), and
there are performing good.
Cheers,
Rafał Wądołowski
On 04.01.2018 17:05, Luis Periquito wrote:
you never said if it was bluestore or filestore?
Can you look in the server to see which component is being stressed
(network, cpu, disk)? Utilities like atop are very handy for this.
Regarding those specific SSDs they are particularly bad when running
some time without trimming - performance nosedives by at least an
order of magnitude. If you really want to go with that risk look at
least to the PROs. And some workloads will always be slow on them.
You never say what's your target environment: do you value
IOPS/latency? Those CPUs won't be great, and I've read a few things
recommending to avoid NUMA (2 CPUs in there). And (higher) frequency
is more important than # of cores to have a high IOPS cluster.
On Thu, Jan 4, 2018 at 3:56 PM, Rafał Wądołowski
wrote:
I have size of 2.
We know about this risk and we accept it, but we still don't know why
performance so so bad.
Cheers,
Rafał Wądołowski
On 04.01.2018 16:51, c...@elchaka.de wrote:
I assume you have size of 3 then divide your expected 400 with 3 and you are
not far Away from what you get...
In Addition you should Never use Consumer grade ssds for ceph as they will
be reach the DWPD very soon...
Am 4. Januar 2018 09:54:55 MEZ schrieb "Rafał Wądołowski"
:
Hi folks,
I am currently benchmarking my cluster for an performance issue and I
have no idea, what is going on. I am using these devices in qemu.
Ceph version 12.2.2
Infrastructure:
3 x Ceph-mon
11 x Ceph-osd
Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
96GB RAM
2x E5-2650 v4
4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
I had tested it with rados bench:
# rados bench -p rbdbench 30 write -t 1
Total time run: 30.055677
Total writes made: 1199
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 159.571
Stddev Bandwidth: 6.83601
Max bandwidth (MB/sec): 168
Min bandwidth (MB/sec): 140
Average IOPS: 39
Stddev IOPS:1
Max IOPS: 42
Min IOPS: 35
Average Latency(s): 0.0250656
Stddev Latency(s): 0.00321545
Max latency(s): 0.0471699
Min latency(s): 0.0206325
# ceph tell osd.0 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"bytes_per_sec": 414199397
}
Testing osd directly
# dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
When I do dd inside vm (bs=4M wih direct), I have result like in rados
bench.
I think that the speed should be arround ~400MB/s.
Is there any new parameters for rbd in luminous? Maybe I forgot about
some performance tricks? If more information needed feel free to ask.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance issues on Luminous
you never said if it was bluestore or filestore?
Can you look in the server to see which component is being stressed
(network, cpu, disk)? Utilities like atop are very handy for this.
Regarding those specific SSDs they are particularly bad when running
some time without trimming - performance nosedives by at least an
order of magnitude. If you really want to go with that risk look at
least to the PROs. And some workloads will always be slow on them.
You never say what's your target environment: do you value
IOPS/latency? Those CPUs won't be great, and I've read a few things
recommending to avoid NUMA (2 CPUs in there). And (higher) frequency
is more important than # of cores to have a high IOPS cluster.
On Thu, Jan 4, 2018 at 3:56 PM, Rafał Wądołowski
wrote:
> I have size of 2.
>
> We know about this risk and we accept it, but we still don't know why
> performance so so bad.
>
> Cheers,
>
> Rafał Wądołowski
>
> On 04.01.2018 16:51, c...@elchaka.de wrote:
>
> I assume you have size of 3 then divide your expected 400 with 3 and you are
> not far Away from what you get...
>
> In Addition you should Never use Consumer grade ssds for ceph as they will
> be reach the DWPD very soon...
>
> Am 4. Januar 2018 09:54:55 MEZ schrieb "Rafał Wądołowski"
> :
>>
>> Hi folks,
>>
>> I am currently benchmarking my cluster for an performance issue and I
>> have no idea, what is going on. I am using these devices in qemu.
>>
>> Ceph version 12.2.2
>>
>> Infrastructure:
>>
>> 3 x Ceph-mon
>>
>> 11 x Ceph-osd
>>
>> Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
>>
>> 96GB RAM
>>
>> 2x E5-2650 v4
>>
>> 4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
>>
>>
>> I had tested it with rados bench:
>>
>> # rados bench -p rbdbench 30 write -t 1
>>
>> Total time run: 30.055677
>> Total writes made: 1199
>> Write size: 4194304
>> Object size:4194304
>> Bandwidth (MB/sec): 159.571
>> Stddev Bandwidth: 6.83601
>> Max bandwidth (MB/sec): 168
>> Min bandwidth (MB/sec): 140
>> Average IOPS: 39
>> Stddev IOPS:1
>> Max IOPS: 42
>> Min IOPS: 35
>> Average Latency(s): 0.0250656
>> Stddev Latency(s): 0.00321545
>> Max latency(s): 0.0471699
>> Min latency(s): 0.0206325
>>
>> # ceph tell osd.0 bench
>> {
>> "bytes_written": 1073741824,
>> "blocksize": 4194304,
>> "bytes_per_sec": 414199397
>> }
>>
>> Testing osd directly
>>
>> # dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
>> 100+0 records in
>> 100+0 records out
>> 419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
>>
>> When I do dd inside vm (bs=4M wih direct), I have result like in rados
>> bench.
>>
>> I think that the speed should be arround ~400MB/s.
>>
>> Is there any new parameters for rbd in luminous? Maybe I forgot about
>> some performance tricks? If more information needed feel free to ask.
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance issues on Luminous
I have size of 2.
We know about this risk and we accept it, but we still don't know why
performance so so bad.
Cheers,
Rafał Wądołowski
On 04.01.2018 16:51, c...@elchaka.de wrote:
I assume you have size of 3 then divide your expected 400 with 3 and
you are not far Away from what you get...
In Addition you should Never use Consumer grade ssds for ceph as they
will be reach the DWPD very soon...
Am 4. Januar 2018 09:54:55 MEZ schrieb "Rafał Wądołowski"
:
Hi folks,
I am currently benchmarking my cluster for an performance issue and I
have no idea, what is going on. I am using these devices in qemu.
Ceph version 12.2.2
Infrastructure:
3 x Ceph-mon
11 x Ceph-osd
Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
96GB RAM
2x E5-2650 v4
4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
I had tested it with rados bench:
# rados bench -p rbdbench 30 write -t 1
Total time run: 30.055677
Total writes made: 1199
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 159.571
Stddev Bandwidth: 6.83601
Max bandwidth (MB/sec): 168
Min bandwidth (MB/sec): 140
Average IOPS: 39
Stddev IOPS: 1
Max IOPS: 42
Min IOPS: 35
Average Latency(s): 0.0250656
Stddev Latency(s): 0.00321545
Max latency(s): 0.0471699
Min latency(s): 0.0206325
# ceph tell osd.0 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"bytes_per_sec": 414199397
}
Testing osd directly
# dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
When I do dd inside vm (bs=4M wih direct), I have result like in rados
bench.
I think that the speed should be arround ~400MB/s.
Is there any new parameters for rbd in luminous? Maybe I forgot about
some performance tricks? If more information needed feel free to ask.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance issues on Luminous
I assume you have size of 3 then divide your expected 400 with 3 and you are
not far Away from what you get...
In Addition you should Never use Consumer grade ssds for ceph as they will be
reach the DWPD very soon...
Am 4. Januar 2018 09:54:55 MEZ schrieb "Rafał Wądołowski"
:
>Hi folks,
>
>I am currently benchmarking my cluster for an performance issue and I
>have no idea, what is going on. I am using these devices in qemu.
>
>Ceph version 12.2.2
>
>Infrastructure:
>
>3 x Ceph-mon
>
>11 x Ceph-osd
>
>Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
>
>96GB RAM
>
>2x E5-2650 v4
>
>4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
>
>
>I had tested it with rados bench:
>
># rados bench -p rbdbench 30 write -t 1
>
>Total time run: 30.055677
>Total writes made: 1199
>Write size: 4194304
>Object size: 4194304
>Bandwidth (MB/sec): 159.571
>Stddev Bandwidth: 6.83601
>Max bandwidth (MB/sec): 168
>Min bandwidth (MB/sec): 140
>Average IOPS: 39
>Stddev IOPS: 1
>Max IOPS: 42
>Min IOPS: 35
>Average Latency(s): 0.0250656
>Stddev Latency(s): 0.00321545
>Max latency(s): 0.0471699
>Min latency(s): 0.0206325
>
># ceph tell osd.0 bench
>{
> "bytes_written": 1073741824,
> "blocksize": 4194304,
> "bytes_per_sec": 414199397
>}
>
>Testing osd directly
>
># dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
>100+0 records in
>100+0 records out
>419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
>
>When I do dd inside vm (bs=4M wih direct), I have result like in rados
>bench.
>
>I think that the speed should be arround ~400MB/s.
>
>Is there any new parameters for rbd in luminous? Maybe I forgot about
>some performance tricks? If more information needed feel free to ask.
>
>--
>BR,
>Rafal Wadolowski
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Performance issues on Luminous
Hi folks,
I am currently benchmarking my cluster for an performance issue and I
have no idea, what is going on. I am using these devices in qemu.
Ceph version 12.2.2
Infrastructure:
3 x Ceph-mon
11 x Ceph-osd
Ceph-osd has 22x1TB Samsung SSD 850 EVO 1TB
96GB RAM
2x E5-2650 v4
4x10G Network (2 seperate bounds for cluster and public) with MTU 9000
I had tested it with rados bench:
# rados bench -p rbdbench 30 write -t 1
Total time run: 30.055677
Total writes made: 1199
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 159.571
Stddev Bandwidth: 6.83601
Max bandwidth (MB/sec): 168
Min bandwidth (MB/sec): 140
Average IOPS: 39
Stddev IOPS: 1
Max IOPS: 42
Min IOPS: 35
Average Latency(s): 0.0250656
Stddev Latency(s): 0.00321545
Max latency(s): 0.0471699
Min latency(s): 0.0206325
# ceph tell osd.0 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"bytes_per_sec": 414199397
}
Testing osd directly
# dd if=/dev/zero of=/dev/sdc bs=4M oflag=direct count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 1.0066 s, 417 MB/s
When I do dd inside vm (bs=4M wih direct), I have result like in rados
bench.
I think that the speed should be arround ~400MB/s.
Is there any new parameters for rbd in luminous? Maybe I forgot about
some performance tricks? If more information needed feel free to ask.
--
BR,
Rafal Wadolowski
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
