[ceph-users] anybody looking for ceph jobs?

[Ken Peng]

Is there anybody looking for a job related to dev/ops on ceph?
If so we (a NASDAQ listed company) can provide one. please PM me for
details.

Thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mail Test

[Ken Peng]

You are welcome. But please don't send test message to a public list. :)

2016-07-12 11:07 GMT+08:00 xiongnuwang :

> I have joined。
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] librados and multithreading

[Ken Peng]

Hi,

We had experienced the similar error, when writing to RBD block with
multi-threads using fio, some OSD got error and down.
Did we talk about the same stuff?

2016-06-11 0:37 GMT+08:00 Юрий Соколов :

> Good day, all.
>
> I found this issue: https://github.com/ceph/ceph/pull/5991
>
> Did this issue affected librados ?
> Were it safe to use single rados_ioctx_t from multiple threads before this
> fix?
>
> --
> With regards,
> Sokolov Yura aka funny_falcon
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] stuck in rbd client accessing pool

[Ken Peng]

Hi,

I think you can specify the pool name in client settings.
for example, in our environment,

# rbd ls
rbd: error opening pool rbd: (2) No such file or directory

# rbd ls -p block
f7470c3f-e051-4f3d-86ff-52e8ba78ac4a
022e9944-122c-4ad0-b652-9e52ba32e2c0

Here -p pool_name was specified. that works.



2016-06-07 0:01 GMT+08:00 strony zhang <strony.zh...@yahoo.com>:

> Hi Ken,
>
> Thanks for your reply.
> The ceph cluster runs well.
>
> :~$ sudo ceph -s
> cluster 285441d6-c059-405d-9762-86bd91f279d0
>  health HEALTH_OK
>  monmap e1: 1 mons at {strony-pc=10.132.138.233:6789/0}
> election epoch 9, quorum 0 strony-pc
>  osdmap e200: 2 osds: 2 up, 2 in
> flags sortbitwise
>   pgmap v225126: 256 pgs, 1 pools, 345 bytes data, 10 objects
> 10326 MB used, 477 GB / 488 GB avail
>  256 active+clean
>   client io 0 B/s rd, 193 op/s rd, 0 op/s wr
>
> $ ceph osd lspools
> 6 rbd,
>
> I previously deleted some pools. So, the latest ID for the pool, 'rbd', is
> 6. I guess the client probably tries accessing the first pool by default
> and then got stuck. So, how can I change the pool ID into '0'?
>
> Thanks,
> Strony
>
>
> On Monday, June 6, 2016 1:46 AM, Ken Peng <k...@dnsbed.com> wrote:
>
>
> hello,
>
> Does ceph cluster work right? run ceph -s and ceph -w for watching more
> details.
>
> 2016-06-06 16:17 GMT+08:00 strony zhang <strony.zh...@yahoo.com>:
>
> Hi,
>
> I am a new learner in ceph. Now I install an All-in-one ceph on the host
> A. Then I tried accessing the ceph from another host B with librados and
> librbd installed.
>
> From host B: I run python to access the ceph on host A.
> >>> import rados
> >>> cluster1 = rados.Rados(conffile='/etc/ceph/ceph.conf')
> >>> cluster1.connect()
> >>> print cluster1.get_fsid()
> 285441d6-c059-405d-9762-86bd91f279d0
> >>>
> >>> import rbd
> >>> rbd_inst = rbd.RBD()
> >>> ioctx = cluster1.open_ioctx('rbd')
> >>> rbd_inst.list(ioctx)
>  stuck in here; it never exits until the python program is killed
> manually.
>
> But in Host A, I don't find any error info.
> zq@zq-ubuntu:~$ rbd list -l
> NAME  SIZE PARENT FMT PROT LOCK
> z1   1024M  2
> z2   1024M  2
> z3   1024M  2
>
> The ceph.conf and ceph.client.admin.keyring in host B are the same to
> those in host A. Any comments are appreciated.
>
> Thanks,
> Strony
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] stuck in rbd client accessing pool

[Ken Peng]

hello,

Does ceph cluster work right? run ceph -s and ceph -w for watching more
details.

2016-06-06 16:17 GMT+08:00 strony zhang :

> Hi,
>
> I am a new learner in ceph. Now I install an All-in-one ceph on the host
> A. Then I tried accessing the ceph from another host B with librados and
> librbd installed.
>
> From host B: I run python to access the ceph on host A.
> >>> import rados
> >>> cluster1 = rados.Rados(conffile='/etc/ceph/ceph.conf')
> >>> cluster1.connect()
> >>> print cluster1.get_fsid()
> 285441d6-c059-405d-9762-86bd91f279d0
> >>>
> >>> import rbd
> >>> rbd_inst = rbd.RBD()
> >>> ioctx = cluster1.open_ioctx('rbd')
> >>> rbd_inst.list(ioctx)
>  stuck in here; it never exits until the python program is killed
> manually.
>
> But in Host A, I don't find any error info.
> zq@zq-ubuntu:~$ rbd list -l
> NAME  SIZE PARENT FMT PROT LOCK
> z1   1024M  2
> z2   1024M  2
> z3   1024M  2
>
> The ceph.conf and ceph.client.admin.keyring in host B are the same to
> those in host A. Any comments are appreciated.
>
> Thanks,
> Strony
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] seqwrite gets good performance but random rw gets worse

[Ken Peng]

You are right. anyway our sysbench result for random R/W gets so worse,
sysbench by default sets up file-fsync-freq=100.
Do you guys have any idea for debug and tuning ceph cluster for better
random IO performance?

Thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] seqwrite gets good performance but random rw gets worse

[Ken Peng]

Hi again,

when setup file-fsync-freq=1 (fsync for each time writing) and
file-fsync-freq=0 (never fsync by sysbench), the result gets huge
difference.
(one is 382.94Kb/sec, another is 25.921Mb/sec).
How do you think of it? thanks.


file-fsync-freq=1,
# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0 --file-fsync-freq=1 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  4309 Read, 2873 Write, 367707 Other = 374889 Total
Read 67.328Mb  Written 44.891Mb  Total transferred 112.22Mb  (382.94Kb/sec)
   23.93 Requests/sec executed

Test execution summary:
total time:  300.0782s
total number of events:  7182
total time taken by event execution: 2.3207
per-request statistics:
 min:  0.01ms
 avg:  0.32ms
 max: 80.17ms
 approx.  95 percentile:   1.48ms

Threads fairness:
events (avg/stddev):   7182./0.00
execution time (avg/stddev):   2.3207/0.00



file-fsync-freq=0,

# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0 --file-fsync-freq=0 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  298613 Read, 199075 Write, 0 Other = 497688 Total
Read 4.5565Gb  Written 3.0376Gb  Total transferred 7.5941Gb  (25.921Mb/sec)
 1658.93 Requests/sec executed

Test execution summary:
total time:  300.0049s
total number of events:  497688
total time taken by event execution: 299.7026
per-request statistics:
 min:  0.00ms
 avg:  0.60ms
 max:   2211.13ms
 approx.  95 percentile:   1.21ms

Threads fairness:
events (avg/stddev):   497688./0.00
execution time (avg/stddev):   299.7026/0.00


2016-05-25 15:01 GMT+08:00 Ken Peng <k...@dnsbed.com>:

> Hello,
>
> We have a cluster with 20+ hosts and 200+ OSDs, each 4T SATA disk for an
> OSD, no SSD cache.
> OS is Ubuntu 16.04 LTS, ceph version 10.2.0
> Both data network and cluster network are 10Gbps.
> We run ceph as block storage service only (rbd client within VM).
>
> For testing within a VM with sysbench tool, we see that the seqwrite has a
> relatively good performance, it can reach 170.37Mb/sec, but random
> read/write always gets bad result, it can be only 474.63Kb/sec (shown as
> below).
>
> Can you help give the idea why the random IO is so worse? Thanks.
>
> This is what sysbench outputs,
>
> # sysbench --test=fileio --file-total-size=5G prepare
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> 128 files, 40960Kb each, 5120Mb total
> Creating files for the test...
>
>
> # sysbench --test=fileio --file-total-size=5G --file-test-mode=seqwr
> --init-rng=on --max-time=300 --max-requests=0 run
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 1
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 40Mb each
> 5Gb total file size
> Block size 16Kb
> Periodic FSYNC enabled, calling fsync() each 100 requests.
> Calling fsync() at the end of test, Enabled.
> Using synchronous I/O mode
> Doing sequential write (creation) test
> Threads started!
> Done.
>
> Operations performed:  0 Read, 327680 Write, 128 Other = 327808 Total
> Read 0b  Written 5Gb  Total transferred 5Gb  (170.37Mb/sec)
> 10903.42 Requests/sec executed
>
> Test execution summary:
> total time:  30.0530s
> total number of events: 

Re: [ceph-users] seqwrite gets good performance but random rw gets worse

[Ken Peng]

Hi,

After comparison we found there is nothing much difference between format 1
and format 2.
format 1 is even worse for randrw.

format 1 result:

# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0  run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!

Time limit exceeded, exiting...
Done.

Operations performed:  6240 Read, 4160 Write, 13301 Other = 23701 Total
Read 97.5Mb  Written 65Mb  Total transferred 162.5Mb  (554.64Kb/sec)
   34.67 Requests/sec executed

Test execution summary:
total time:  300.0118s
total number of events:  10400
total time taken by event execution: 8.3638
per-request statistics:
 min:  0.01ms
 avg:  0.80ms
 max:181.63ms
 approx.  95 percentile:   1.72ms

Threads fairness:
events (avg/stddev):   10400./0.00
execution time (avg/stddev):   8.3638/0.00



format 2 result:

# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0  run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  7260 Read, 4840 Write, 15363 Other = 27463 Total
Read 113.44Mb  Written 75.625Mb  Total transferred 189.06Mb  (645.15Kb/sec)
   40.32 Requests/sec executed

Test execution summary:
total time:  300.0843s
total number of events:  12100
total time taken by event execution: 9.8130
per-request statistics:
 min:  0.01ms
 avg:  0.81ms
 max:209.24ms
 approx.  95 percentile:   1.64ms

Threads fairness:
events (avg/stddev):   12100./0.00
execution time (avg/stddev):   9.8130/0.00


2016-05-25 15:31 GMT+08:00 Adrian Saul <adrian.s...@tpgtelecom.com.au>:

>
>
> Are you using image-format 2 RBD images?
>
>
>
> We found a major performance hit using format 2 images under 10.2.0 today
> in some testing.  When we switched to using format 1 images we literally
> got 10x random write IOPS performance (1600 IOPs up to 3 IOPS for the
> same test).
>
>
>
>
>
>
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Ken Peng
> *Sent:* Wednesday, 25 May 2016 5:02 PM
> *To:* ceph-users@lists.ceph.com
> *Subject:* [ceph-users] seqwrite gets good performance but random rw gets
> worse
>
>
>
> Hello,
>
> We have a cluster with 20+ hosts and 200+ OSDs, each 4T SATA disk for an
> OSD, no SSD cache.
>
> OS is Ubuntu 16.04 LTS, ceph version 10.2.0
>
> Both data network and cluster network are 10Gbps.
>
> We run ceph as block storage service only (rbd client within VM).
>
> For testing within a VM with sysbench tool, we see that the seqwrite has a
> relatively good performance, it can reach 170.37Mb/sec, but random
> read/write always gets bad result, it can be only 474.63Kb/sec (shown as
> below).
>
> Can you help give the idea why the random IO is so worse? Thanks.
>
> This is what sysbench outputs,
>
> # sysbench --test=fileio --file-total-size=5G prepare
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> 128 files, 40960Kb each, 5120Mb total
> Creating files for the test...
>
>
> # sysbench --test=fileio --file-total-size=5G --file-test-mode=seqwr
> --init-rng=on --max-time=300 --max-requests=0 run
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 1
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 40Mb each
> 5Gb total file size
> Block s

Re: [ceph-users] seqwrite gets good performance but random rw gets worse

[Ken Peng]

yes we run format 2 only.
We will run a data disk with format 1 for a testing for comparsion.
I will tell you the results. thanks.

2016-05-25 15:31 GMT+08:00 Adrian Saul <adrian.s...@tpgtelecom.com.au>:

>
>
> Are you using image-format 2 RBD images?
>
>
>
> We found a major performance hit using format 2 images under 10.2.0 today
> in some testing.  When we switched to using format 1 images we literally
> got 10x random write IOPS performance (1600 IOPs up to 3 IOPS for the
> same test).
>
>
>
>
>
>
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Ken Peng
> *Sent:* Wednesday, 25 May 2016 5:02 PM
> *To:* ceph-users@lists.ceph.com
> *Subject:* [ceph-users] seqwrite gets good performance but random rw gets
> worse
>
>
>
> Hello,
>
> We have a cluster with 20+ hosts and 200+ OSDs, each 4T SATA disk for an
> OSD, no SSD cache.
>
> OS is Ubuntu 16.04 LTS, ceph version 10.2.0
>
> Both data network and cluster network are 10Gbps.
>
> We run ceph as block storage service only (rbd client within VM).
>
> For testing within a VM with sysbench tool, we see that the seqwrite has a
> relatively good performance, it can reach 170.37Mb/sec, but random
> read/write always gets bad result, it can be only 474.63Kb/sec (shown as
> below).
>
> Can you help give the idea why the random IO is so worse? Thanks.
>
> This is what sysbench outputs,
>
> # sysbench --test=fileio --file-total-size=5G prepare
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> 128 files, 40960Kb each, 5120Mb total
> Creating files for the test...
>
>
> # sysbench --test=fileio --file-total-size=5G --file-test-mode=seqwr
> --init-rng=on --max-time=300 --max-requests=0 run
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 1
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 40Mb each
> 5Gb total file size
> Block size 16Kb
> Periodic FSYNC enabled, calling fsync() each 100 requests.
> Calling fsync() at the end of test, Enabled.
> Using synchronous I/O mode
> Doing sequential write (creation) test
> Threads started!
> Done.
>
> Operations performed:  0 Read, 327680 Write, 128 Other = 327808 Total
> Read 0b  Written 5Gb  Total transferred 5Gb  (170.37Mb/sec)
> 10903.42 Requests/sec executed
>
> Test execution summary:
> total time:  30.0530s
> total number of events:  327680
> total time taken by event execution: 28.5936
> per-request statistics:
>  min:  0.01ms
>  avg:  0.09ms
>  max:192.84ms
>  approx.  95 percentile:   0.03ms
>
> Threads fairness:
> events (avg/stddev):   327680./0.00
> execution time (avg/stddev):   28.5936/0.00
>
>
>
> # sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
> --init-rng=on --max-time=300 --max-requests=0 run
> sysbench 0.4.12:  multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 1
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 40Mb each
> 5Gb total file size
> Block size 16Kb
> Number of random requests for random IO: 0
> Read/Write ratio for combined random IO test: 1.50
> Periodic FSYNC enabled, calling fsync() each 100 requests.
> Calling fsync() at the end of test, Enabled.
> Using synchronous I/O mode
> Doing random r/w test
> Threads started!
>
> Time limit exceeded, exiting...
> Done.
>
> Operations performed:  5340 Read, 3560 Write, 11269 Other = 20169 Total
> Read 83.438Mb  Written 55.625Mb  Total transferred 139.06Mb  (474.63Kb/sec)
>29.66 Requests/sec executed
>
> Test execution summary:
> total time:  300.0216s
> total number of events:  8900
> total time taken by event execution: 6.4774
> per-request statistics:
>  min:  0.01ms
>  avg:  0.73ms
>  max: 90.18ms
>  approx.  95 percentile:   1.60ms
>
> Threads fairness:
> events (avg/stddev):   8900./0.00
> execution time (avg/stddev):   6.4774/0.00
> Confidentiality: This email and any attachments are confidential and may
> be subject to copyright, legal or some other professional privilege. They
> are intended solely for the attention

[ceph-users] seqwrite gets good performance but random rw gets worse

[Ken Peng]

Hello,

We have a cluster with 20+ hosts and 200+ OSDs, each 4T SATA disk for an
OSD, no SSD cache.
OS is Ubuntu 16.04 LTS, ceph version 10.2.0
Both data network and cluster network are 10Gbps.
We run ceph as block storage service only (rbd client within VM).

For testing within a VM with sysbench tool, we see that the seqwrite has a
relatively good performance, it can reach 170.37Mb/sec, but random
read/write always gets bad result, it can be only 474.63Kb/sec (shown as
below).

Can you help give the idea why the random IO is so worse? Thanks.

This is what sysbench outputs,

# sysbench --test=fileio --file-total-size=5G prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

128 files, 40960Kb each, 5120Mb total
Creating files for the test...


# sysbench --test=fileio --file-total-size=5G --file-test-mode=seqwr
--init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential write (creation) test
Threads started!
Done.

Operations performed:  0 Read, 327680 Write, 128 Other = 327808 Total
Read 0b  Written 5Gb  Total transferred 5Gb  (170.37Mb/sec)
10903.42 Requests/sec executed

Test execution summary:
total time:  30.0530s
total number of events:  327680
total time taken by event execution: 28.5936
per-request statistics:
 min:  0.01ms
 avg:  0.09ms
 max:192.84ms
 approx.  95 percentile:   0.03ms

Threads fairness:
events (avg/stddev):   327680./0.00
execution time (avg/stddev):   28.5936/0.00



# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw
--init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!

Time limit exceeded, exiting...
Done.

Operations performed:  5340 Read, 3560 Write, 11269 Other = 20169 Total
Read 83.438Mb  Written 55.625Mb  Total transferred 139.06Mb  (474.63Kb/sec)
   29.66 Requests/sec executed

Test execution summary:
total time:  300.0216s
total number of events:  8900
total time taken by event execution: 6.4774
per-request statistics:
 min:  0.01ms
 avg:  0.73ms
 max: 90.18ms
 approx.  95 percentile:   1.60ms

Threads fairness:
events (avg/stddev):   8900./0.00
execution time (avg/stddev):   6.4774/0.00
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] dd testing from within the VM

[Ken Peng]

Oliver,

Thanks for the info.
We then run sysbench for random IO testing, the result is even worse 
(757 KB/s).

each object has 3 replicas.
Both networks are 10Gbps, I don't think there are issues with network.
Maybe lacking of SSD cache, and miscorrect configure to the cluster are 
the reason.




Extra file open flags: 0
128 files, 360Mb each
45Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!

Time limit exceeded, exiting...
Done.

Operations performed:  8520 Read, 5680 Write, 18056 Other = 32256 Total
Read 133.12Mb  Written 88.75Mb  Total transferred 221.88Mb  (757.33Kb/sec)
   47.33 Requests/sec executed

Test execution summary:
total time:  300.0012s
total number of events:  14200
total time taken by event execution: 21.6865
per-request statistics:
 min:  0.02ms
 avg:  1.53ms
 max:   1325.73ms
 approx.  95 percentile:   1.92ms

Threads fairness:
events (avg/stddev):   14200./0.00
execution time (avg/stddev):   21.6865/0.00




On 2016/5/19 星期四 18:24, Oliver Dzombic wrote:

Hi Ken,

dd is ok, but you should consider the fact that dd is a squence of writing.

So if you have random writes in your later productive usage, then this
test is basically only good to meassure the maximum squential write
performance in idle state.

And 250 MB for 200 HDD's is quiet evil bad as a performance for a
sequential write.

Sequential write of a 7200 RPM SATA HDD should be around 70-100 MB,
maybe more.

So if you have 200 of them, idle, and writing a sequence, and resulting
in 250 MB/s. That does not look good to me.

So eighter your network is not good, or your settings are not good. Or
you have too high replica number or something like that.

At least for me, 200x HDDs and each HDD deliver 1,2 MB/s writing speed
performance.

I assume that your 4 GB won't be spread over all 200 HDDs. But still,
the result does not look like good performance.

FIO is a nice test with different settings.

---

The effect of conv=fdatasync will be only as big, as the RAM memory of
your test client will be.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] dd testing from within the VM

[Ken Peng]

Hi,

Our VM has been using ceph as block storage for both system and data 
patition.


This is what dd shows,

# dd if=/dev/zero of=test.file bs=4k count=1024k
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 16.7969 s, 256 MB/s

When dd again with fdatasync argument,the result is similar.

# dd if=/dev/zero of=test.file bs=4k count=1024k conv=fdatasync
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 17.6878 s, 243 MB/s


My questions include,

1. for a cluster which has more than 200 disks as OSD storage (SATA 
only), both the cluster and data network are 10Gbps, does the 
performance from within the VM behave well as the results above?


2. is "dd" suitable for testing a block storage within the VM?

3. why "fdatasync" influences nothing on the testing?

Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com