Andrei, Yes, I see it has lot of potential and I believe fixing the performance bottlenecks inside XIO messenger it should go further. We are working on it and will keep community posted..
Thanks & Regards Somnath From: Andrei Mikhailovsky [mailto:[email protected]] Sent: Wednesday, April 08, 2015 2:22 AM To: Andrey Korolyov Cc: [email protected]; ceph-devel; Somnath Roy Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers Hi, Am I the only person noticing disappointing results from the preliminary RDMA testing, or am I reading the numbers wrong? Yes, it's true that on a very small cluster you do see a great improvement in rdma, but in real life rdma is used in large infrastructure projects, not on a few servers with a handful of osds. In fact, from what i've seen from the slides, the rdma implementation scales horribly to the point that it becomes slower the more osds you through at it. From my limited knowledge, i have expected a much higher performance gains with rdma, taking into account that you should have much lower latency and overhead and lower cpu utilisation when using this transport in comparison with tcp. Are we likely to see a great deal of improvement with ceph and rdma in a near future? Is there a roadmap for having a stable and reliable rdma protocol support? Thanks Andrei ________________________________ From: "Andrey Korolyov" <[email protected]<mailto:[email protected]>> To: "Somnath Roy" <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>, "ceph-devel" <[email protected]<mailto:[email protected]>> Sent: Wednesday, 8 April, 2015 9:28:12 AM Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <[email protected]<mailto:[email protected]>> wrote: > > Hi, > Please find the preliminary performance numbers of TCP Vs RDMA (XIO) > implementation (on top of SSDs) in the following link. > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma > > The attachment didn't go through it seems, so, I had to use slideshare. > > Mark, > If we have time, I can present it in tomorrow's performance meeting. > > Thanks & Regards > Somnath > Those numbers are really impressive (for small numbers at least)! What are TCP settings you using?For example, difference can be lowered on scale due to less intensive per-connection acceleration on CUBIC on a larger number of nodes, though I do not believe that it was a main reason for an observed TCP catchup on a relatively flat workload such as fio generates. _______________________________________________ ceph-users mailing list [email protected]<mailto:[email protected]> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
