[ceph-users] Re: RDMA

2019-10-17 Thread Stig Telfer
Hi All - I did some investigation into Ceph RDMA as part of a performance analysis project working with Ceph over Omnipath and NVME. I wrote up some of the analysis here: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html

[ceph-users] Re: RDMA

2019-10-15 Thread Paul Emmerich
That's apply/commit latency (the exact same since BlueStore btw, no point in tracking both). It should not contain any network component. Since the path you are optimizing is inter-OSD communication: check out subop latency, that's the one where this should show up. Paul -- Paul Emmerich Look

[ceph-users] Re: RDMA

2019-10-15 Thread vitalif
I don't see any changes here... There is graph here. It was pure Nautilus before 10-05 and Nautilus+RDMA after. https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD Link expires on Nov 1. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an

[ceph-users] Re: RDMA

2019-10-15 Thread Max Krasilnikov
День добрий! Tue, Oct 15, 2019 at 02:29:58PM +0300, vitalif wrote: > Wow, does it really work? > > And why is it not supported by RBD? I hadn't dive into sources, but it stated in docs. > > Can you show us the latency graphs before and after and tell the I/O pattern > to which the latency

[ceph-users] Re: RDMA

2019-10-15 Thread Wido den Hollander
On 10/15/19 1:29 PM, Vitaliy Filippov wrote: > Wow, does it really work? > > And why is it not supported by RBD? > > Can you show us the latency graphs before and after and tell the I/O > pattern to which the latency applies? Previous common knowledge was that > RDMA almost doesn't affect laten

[ceph-users] Re: RDMA

2019-10-15 Thread Vitaliy Filippov
Wow, does it really work? And why is it not supported by RBD? Can you show us the latency graphs before and after and tell the I/O pattern to which the latency applies? Previous common knowledge was that RDMA almost doesn't affect latency with Ceph, because most of the latency is in Ceph i

[ceph-users] Re: RDMA

2019-10-15 Thread Max Krasilnikov
Hello! Mon, Oct 14, 2019 at 07:28:07AM -, gabryel.mason-williams wrote: > Hello, > > I was wondering what user experience was with using Ceph over RDMA? > - How you set it up? We had used RoCE Lag with Mellanox ConnectX-4 Lx. > - Documentation used to set it up? Generally, Mellano