How are you mounting CephFS? It may be that the cache settings are just set
very badly for a 10G pipe. Plus rados bench is a very parallel large-IO
benchmark and many benchmarks you might dump into a filesystem are
definitely not.
-Greg
On Thu, Aug 30, 2018 at 7:54 AM Peter Eisch <peter.ei...@virginpulse.com>
wrote:
> Hi,
>
> I have a cluster serving cephfs and it works. It’s just slow. Client is
> using the kernel driver. I can ‘rados bench’ writes to the cephfs_data pool
> at wire speeds (9580Mb/s on a 10G link) but when I copy data into cephfs it
> is rare to get above 100Mb/s. Large file writes may start fast (2Gb/s) but
> within a minute slows. In the dashboard at the OSDs I get lots of triangles
> (it doesn't stream) which seems to be lots of starts and stops. By contrast
> the graphs show constant flow when using 'rados bench.'
>
> I feel like I'm missing something obvious. What can I do to help diagnose
> this better or resolve the issue?
>
> Errata:
> Version: 12.2.7 (on everything)
> mon: 3 daemons, quorum cephmon-s01,cephmon-s03,cephmon-s02
> mgr: cephmon-s02(active), standbys: cephmon-s01, cephmon-s03
> mds: cephfs1-1/1/1 up {0=cephmon-s02=up:active}, 2 up:standby
> osd: 70 osds: 70 up, 70 in
> rgw: 3 daemons active
>
> rados bench summary:
> Total time run: 600.043733
> Total writes made: 167725
> Write size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 1118.09
> Stddev Bandwidth: 7.23868
> Max bandwidth (MB/sec): 1140
> Min bandwidth (MB/sec): 1084
> Average IOPS: 279
> Stddev IOPS: 1
> Max IOPS: 285
> Min IOPS: 271
> Average Latency(s): 0.057239
> Stddev Latency(s): 0.0354817
> Max latency(s): 0.367037
> Min latency(s): 0.0120791
>
> peter
>
>
>
> Peter Eisch
> [image: Facebook] <https://www.facebook.com/VirginPulse>
> [image: LinkedIn] <https://www.linkedin.com/company/virgin-pulse>
> [image: Twitter] <https://twitter.com/virginpulse>
> *virginpulse.com* <https://www.virginpulse.com/>
> | *globalchallenge.virginpulse.com*
> <https://globalchallenge.virginpulse.com/>
>
> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
> Switzerland | United Kingdom | USA
> Confidentiality Notice: The information contained in this e-mail,
> including any attachment(s), is intended solely for use by the designated
> recipient(s). Unauthorized use, dissemination, distribution, or
> reproduction of this message by anyone other than the intended
> recipient(s), or a person designated as responsible for delivering such
> messages to the intended recipient, is strictly prohibited and may be
> unlawful. This e-mail may contain proprietary, confidential or privileged
> information. Any views or opinions expressed are solely those of the author
> and do not necessarily represent those of Virgin Pulse, Inc. If you have
> received this message in error, or are not the named recipient(s), please
> immediately notify the sender and delete this e-mail message.
> v2.10
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
