Re: [ceph-users] Infiniband backend OSD communication
Ok, so ipoib is required... ><> nathan stratton On Mon, Jan 6, 2020 at 4:45 AM Wei Zhao wrote: > From my understanding, the basic idea is that ceph exchange rdma > information(qp,gid and so) through ip address on rdma device, and then > communicate with each other throng rdma. But in my tests, there > seemed to be some issues in that codes. > > On Fri, Jan 3, 2020 at 2:24 AM Nathan Stratton > wrote: > > > > I am working on upgrading my current ethernet only ceph cluster to a > combined ethernet frontend and infiniband backend. From my research I > understand that I set: > > > > ms_cluster_type = async+rdma > > ms_async_rdma_device_name = mlx4_0 > > > > What I don't understand is how does ceph know how to reach each OSD over > RDMA? Do I have to run IPoIB on top of infiniband and use that for OSD > addresses? > > > > Is there a way to use infiniband on backend without IPoIB and just use > rdma verbs? > > > > ><> > > nathan stratton > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Infiniband backend OSD communication
>From my understanding, the basic idea is that ceph exchange rdma information(qp,gid and so) through ip address on rdma device, and then communicate with each other throng rdma. But in my tests, there seemed to be some issues in that codes. On Fri, Jan 3, 2020 at 2:24 AM Nathan Stratton wrote: > > I am working on upgrading my current ethernet only ceph cluster to a combined > ethernet frontend and infiniband backend. From my research I understand that > I set: > > ms_cluster_type = async+rdma > ms_async_rdma_device_name = mlx4_0 > > What I don't understand is how does ceph know how to reach each OSD over > RDMA? Do I have to run IPoIB on top of infiniband and use that for OSD > addresses? > > Is there a way to use infiniband on backend without IPoIB and just use rdma > verbs? > > ><> > nathan stratton > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Infiniband backend OSD communication
I am working on upgrading my current ethernet only ceph cluster to a combined ethernet frontend and infiniband backend. From my research I understand that I set: ms_cluster_type = async+rdma ms_async_rdma_device_name = mlx4_0 What I don't understand is how does ceph know how to reach each OSD over RDMA? Do I have to run IPoIB on top of infiniband and use that for OSD addresses? Is there a way to use infiniband on backend without IPoIB and just use rdma verbs? ><> nathan stratton ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
