On 2019-07-15T19:08:26, " Drobyshevskiy, Vladimir " <v...@itgorod.ru> wrote:
Hi Vladimir, is this a replicated or EC pool? > The cluster itself: > nautilus > 6 nodes, 7 SSD with 2 OSDs per SSD (14 OSDs in overall). You mean 14 OSDs per node, right? > Each node: 2x Intel Xeon E5-2665 v1 (governor = performance, powersaving > disabled), 64GB RAM, Samsung SM863 1.92TB SSD, QDR Infiniband. I assume that's the cluster backend. How are the clients connected? > I've tried to make an RAID0 with mdraid and 2 virtual drives but haven't > noticed any difference. Your problem isn't bandwidth - it's the commit latency for the small IO. In your enviroment, that's primarily going to be governed by network (and possibly ceph-osd CPU) latency. That doesn't show up as high utilization anywhere, because it's mainly waiting. Most networking is terrifyingly slow compared to the latency of a local flash storage device. And with Ceph, you've got to add at least two roundtrips to every IO (client - primary OSD, primary OSD - replicas, probably more, and if you us EC with ec_overwrites, definitely more roundtrips). Regards, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden) _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com