You're doing 4K direct IOs on a distributed storage system and then comparing it to what the local device does with 1GB blocks? :)
Try feeding Ceph with some larger IOs and check how it does. -Greg On Fri, May 18, 2018 at 1:22 PM Rhugga Harper <rhu...@gmail.com> wrote: > > We're evaluating persistent block providers for Kubernetes and looking at > ceph at the moment. > > We aren't seeing performance anywhere near what we expect. > > I have a 50-node proof of concept cluster with 40 nodes available for > storage and configured with rook/ceph. Each has 10GB nics and 8 x 1TB > SSD's. (only 3 drives on each node have been allocated to ceph use) > > We are testing with replicated pools of size 1 and 3. I've been doing fio > tests in parallel (pod setup to run fio) and it seems to average aggregate > bandwidth around 150 MB/sec. > > I'm running the fio tests as follows: > > direct=1, fsync=8|16|32|64, readwrite=write, blocksize=4k, numjobs=4|8, > size=10G > > Regardless of doing 1 stream or 40, the aggregate bandwidth as reported by > "ceph -s" is ~150MB/sec: > > I'm creating my pool with pg_num/pgp_num=1024|2048|4096 > > A baseline dd (100GB file using blocksize 1G) on these SSD's shows them > capable of 1.6 GB/s. > > I can't seem to find any limitations or bottlenecks on the nodes or the > network. > > Anyone have any idea where else I can look? > > I'm new to ceph and it just seems like this should be pushing more I/O. > I've dug thru a lot of performance tuning sites and have m implemented most > of the suggestions. > > > # ceph -s > cluster: > id: 949a8caf-9a9b-4f09-8711-1d5158a65bd8 > health: HEALTH_OK > > services: > mon: 7 daemons, quorum > rook-ceph-mon1,rook-ceph-mon3,rook-ceph-mon0,rook-ceph-mon5,rook-ceph-mon4,rook-ceph-mon2,rook-ceph-mon6 > mgr: rook-ceph-mgr0(active) > osd: 123 osds: 123 up, 123 in > > data: > pools: 1 pools, 2048 pgs > objects: 134k objects, 508 GB > usage: 1240 GB used, 110 TB / 112 TB avail > pgs: 2048 active+clean > > io: > client: 138 MB/s wr, 0 op/s rd, 71163 op/s wr > > Thanks for any help, > CC > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
