Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Hi Roman, We recently discussed your tests and a simple idea came to my mind - can you repeat your tests targeting latency instead of max throughput? I mean just use iodepth=1. What the latency is and on what hardware? Well, I am playing with ceph rdma implementation quite a while and it h

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

I was informed today that the CEPH environment I’ve been working on is no longer available. Unfortunately this happened before I could try any of your suggestions, Roman. Thank you for all the attention and advice. -- Michael Green > On Dec 20, 2018, at 08:21, Roman Penyaev wrote: > >> On

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Thanks for posting this Roman. -Original Message- From: Roman Penyaev [mailto:rpeny...@suse.de] Sent: 20 December 2018 14:21 To: Marc Roos Cc: green; mgebai; ceph-users Subject: Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host On 2018-12-19 22:01, Marc Roos

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Thanks, Roman. My RDMA is working correctly, I'm pretty sure of that for two reasons. (1) E8 Storage agent running on all OSDs uses RDMA to communicate with our E8 Storage controller and it's working correctly at the moment. The volumes are available and IO can be done at full line rate and ex

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

00 = 11.452 sockperf: ---> percentile 25.00 = 11.188 sockperf: ---> observation =8.995 -Original Message- From: Michael Green [mailto:gr...@e8storage.com] Sent: 19 December 2018 21:00 To: Roman Penyaev; Mohamad Gebai Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] RDMA/RoCE

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Thanks for the insights Mohammad and Roman. Interesting read. My interest in RDMA is purely from testing perspective. Still I would be interested if somebody who has RDMA enabled and running, to share their ceph.conf. My RDMA related entries are taken from Mellanox blog here https://communit

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Last I heard (read) was that the RDMA implementation is somewhat experimental. Search for "troubleshooting ceph rdma performance" on this mailing list for more info. (Adding Roman in CC who has been working on this recently.) Mohamad On 12/18/18 11:42 AM, Michael Green wrote: > I don't know.  >

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

I don't know. Ceph documentation on Mimic doesn't appear to go into too much details on RDMA in general, but still it's mentioned in the Ceph docs here and there. Some examples: Change log - http://docs.ceph.com/docs/master/releases/mimic/ Asy

Re: [ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Is RDMA officially supported? I'm asking because I recently tried to use DPDK and it seems it's broken... i.e the code is there, but does not compile until I fix cmake scripts, and after fixing the build OSDs just get segfaults and die after processing something like 40-50 incoming packets. May

[ceph-users] RDMA/RoCE enablement failed with (113) No route to host

Hello collective wisdom, ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) here. I have a working cluster here consisting of 3 monitor hosts, 64 OSD processes across 4 osd hosts, plus 2 MDSs, plus 2 MGRs. All of that is consumed by 10 client nodes. Every host in t

[ceph-users] RDMA and ceph-mgr

Hello! I have Ubuntu 16.04.05 and ceph 13.2.1. On fresh system install mon and mgr only for one node mtn7. Work ok. systemctl stop ceph-mon@mtn7   systemctl stop ceph-mgr@mtn7 Next, I changed the settings for RDMA for /lib/systemd/system/ceph-mon@.service remove PrivateDevice=yes add PrivateD

Re: [ceph-users] RDMA question for ceph

Ok, Thank you very much . I will try to caontack them and update the problem. And in the meantime , I will try to debug it by just seting up one mon and one osd. Thanks again. On Mon, Jul 23, 2018 at 3:49 PM John Hearns wrote: > Will, looking at the logs which you sent, the connection canno

Re: [ceph-users] RDMA question for ceph

Hi John: Thanks for your reply. Yes, the following is the detail . ibdev2netdev mlx4_0 port 1 ==> ib0 (Down) mlx4_0 port 2 ==> ib1 (Up) sh show-gids.sh DEV PORTINDEX GID IPv4 VER DEV --- - --- -

[ceph-users] RDMA question for ceph

Hi all: Has anyone successfully set up ceph with rdma over IB ? By following the instructions: (https://community.mellanox.com/docs/DOC-2721) (https://community.mellanox.com/docs/DOC-2693) (http://hwchiu.com/2017-05-03-ceph-with-rdma.html) I'm trying to configure CEPH with RDMA feature

Re: [ceph-users] RDMA support in Ceph

On 6/28/18, 12:11 AM, "kefu chai" wrote: > What is the state of the RDMA code in the Ceph Luminous and later releases? in Ceph, the RDMA support has been constantly worked on. xio messenger support was added 4 years ago, but i don't think it's maintained anymore. and async

Re: [ceph-users] RDMA support in Ceph

On Wed, Jun 27, 2018 at 1:21 AM Kamble, Nitin A wrote: > > I tried enabling the RDMA support in Ceph Luminous release following this [1] > guide. > > I used the released Luminous bits, and not the Mellanox branches mentioned in > the guide. > > > > I could see some RDMA traffic in the perf count

[ceph-users] RDMA support in Ceph

I tried enabling the RDMA support in Ceph Luminous release following this [1] guide. I used the released Luminous bits, and not the Mellanox branches mentioned in the guide. I could see some RDMA traffic in the perf counters, but the ceph daemons were still complaining that they are not able to

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

Hi Haomai, can you please guide me to a running cluster with RDMA ? regards Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing +49 171 4802507 Am 28.09.2017 um 04:21 schrieb Haomai Wang: > previously we have a infiniband cluster, recently we deploy a roce > cluster. th

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

previously we have a infiniband cluster, recently we deploy a roce cluster. they are both test purpose for users. On Wed, Sep 27, 2017 at 11:38 PM, Gerhard W. Recher wrote: > Haomai, > > I looked at your presentation, so i guess you already have a running > cluster with RDMA & mellanox > (https:/

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

Yep ROcE i followed up all recommendations in mellanox papers ... */etc/security/limits.conf* * soft memlock unlimited * hard memlock unlimited root soft memlock unlimited root hard memlock unlimited also set properties on daemons (chapter 11) in https://community.mellanox.com/docs/DOC-27

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

Haomai, I looked at your presentation, so i guess you already have a running cluster with RDMA & mellanox (https://www.youtube.com/watch?v=Qb2SUWLdDCw) Is nobody out there having a running cluster with RDMA ? any help is appreciated ! Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

ah ok but as i stated before : ceph.conf is a cluster wide file on proxmox! so if i specify [global] //Set local GID for ROCEv2 interface used for CEPH //The GID corresponding to IPv4 or IPv6 networks //should be taken from show_gids command output //This parameter should be uniquely set

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

https://community.mellanox.com/docs/DOC-2415 On Wed, Sep 27, 2017 at 10:01 PM, Gerhard W. Recher wrote: > How to set local gid option ? > > I have no glue :) > > Gerhard W. Recher > > net4sec UG (haftungsbeschränkt) > Leitenweg 6 > 86929 Penzing > > +49 171 4802507 > Am 27.09.2017 um 15:59 schrie

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

How to set local gid option ? I have no glue :) Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing +49 171 4802507 Am 27.09.2017 um 15:59 schrieb Haomai Wang: > do you set local gid option? > > On Wed, Sep 27, 2017 at 9:52 PM, Gerhard W. Recher > wrote: >> Yep ROcE ...

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

do you set local gid option? On Wed, Sep 27, 2017 at 9:52 PM, Gerhard W. Recher wrote: > Yep ROcE > > i followed up all recommendations in mellanox papers ... > > */etc/security/limits.conf* > > * soft memlock unlimited > * hard memlock unlimited > root soft memlock unlimited > root hard mem

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

Yep ROcE i followed up all recommendations in mellanox papers ... */etc/security/limits.conf* * soft memlock unlimited * hard memlock unlimited root soft memlock unlimited root hard memlock unlimited also set properties on daemons (chapter 11) in https://community.mellanox.com/docs/DOC-27

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

Haomai,  ibstat CA 'mlx4_0'     CA type: MT4103     Number of ports: 2     Firmware version: 2.40.7000     Hardware version: 0     Node GUID: 0x248a070300e26070     System image GUID: 0x248a070300e26070     Port 1:     State: Active     Physical

Re: [ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

On Wed, Sep 27, 2017 at 8:33 PM, Gerhard W. Recher wrote: > Hi Folks! > > I'm totally stuck > > rdma is running on my nics, rping udaddy etc will give positive results. > > cluster consist of: > proxmox-ve: 5.0-23 (running kernel: 4.10.17-3-pve) > pve-manager: 5.0-32 (running version: 5.0-32/2560e

[ceph-users] RDMA with mellanox connect x3pro on debian stretch and proxmox v5.0 kernel 4.10.17-3

Hi Folks! I'm totally stuck rdma is running on my nics, rping udaddy etc will give positive results. cluster consist of: proxmox-ve: 5.0-23 (running kernel: 4.10.17-3-pve) pve-manager: 5.0-32 (running version: 5.0-32/2560e073) system(4 nodes): Supermicro 2028U-TN24R4T+ 2 port Mellanox connect

Re: [ceph-users] RDMA/Infiniband status

Hello, What I took from the longish thread on the OFED ML was that certain things (and more than you'd think) with IPoIB happen in multicast, not ALL of them. For the record, my bog standard QDR, IPoIB clusters can do anywhere from 14 to 21Gb/s with iperf3 and about 20-30% less with NPtcp (netp

Re: [ceph-users] RDMA/Infiniband status

Infiniband uses multicast internally. It's not something you have a choice with. You won't see it on the local interface any more than you'd see individual drives of a raid 5. I believe it's one of the reasons the connection setup speeds are kept under the requisite 1.2usec limits etc. On Jun 10

Re: [ceph-users] RDMA/Infiniband status

On 10/06/16 02:33, Christian Balzer wrote: > > > This thread brings back memories of this one: > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/008792.html > > According to Robert IPoIB still uses IB multicast under the hood even when > from an IP perspective traffic would be uni

Re: [ceph-users] RDMA/Infiniband status

Hello, On Thu, 9 Jun 2016 20:28:41 +0200 Daniel Swarbrick wrote: > On 09/06/16 17:01, Gandalf Corvotempesta wrote: > > Il 09 giu 2016 15:41, "Adam Tygart" > > ha scritto: > >> > >> If you're > >> using pure DDR, you may need to tune the broadcast group in your > >> subnet

Re: [ceph-users] RDMA/Infiniband status

On 09/06/16 17:01, Gandalf Corvotempesta wrote: > Il 09 giu 2016 15:41, "Adam Tygart" > ha scritto: >> >> If you're >> using pure DDR, you may need to tune the broadcast group in your >> subnet manager to set the speed to DDR. > > Do you know how to set this with opensm? > I

Re: [ceph-users] RDMA/Infiniband status

I believe this is what you want: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Configuring_the_Subnet_Manager.html -- Adam On Thu, Jun 9, 2016 at 10:01 AM, Gandalf Corvotempesta wrote: > Il 09 giu 2016 15:41, "Adam Tygart" ha scritto: >> >> I

Re: [ceph-users] RDMA/Infiniband status

Il 09 giu 2016 15:41, "Adam Tygart" ha scritto: > > If you're > using pure DDR, you may need to tune the broadcast group in your > subnet manager to set the speed to DDR. Do you know how to set this with opensm? I would like to bring up my test cluster again next days

Re: [ceph-users] RDMA/Infiniband status

IPoIB is done with broadcast packets on the Infiniband fabric. Most switches and opensm (by default) setup a broadcast group at the lowest IB speed (SDR), to support all possible IB connections. If you're using pure DDR, you may need to tune the broadcast group in your subnet manager to set the spe

Re: [ceph-users] RDMA/Infiniband status

2016-06-09 10:18 GMT+02:00 Christian Balzer : > IPoIB is about half the speed of your IB layer, yes. Ok, so it's normal. I've seen benchmarks on net stating that IPoIB on DDR should reach about 16-17Gb/s I'll plan to move to QDR > And bandwidth is (usually) not the biggest issue, latency is. I'v

Re: [ceph-users] RDMA/Infiniband status

On Thu, 9 Jun 2016 10:00:33 +0200 Gandalf Corvotempesta wrote: > Last time i've used Ceph (about 2014) RDMA/Infiniband support was just > a proof of concept > and I was using IPoIB with low performance (about 8-10GB/s on a > Infiniband DDR 20Gb/s) > IPoIB is about half the speed of your IB layer,

[ceph-users] RDMA/Infiniband status

Last time i've used Ceph (about 2014) RDMA/Infiniband support was just a proof of concept and I was using IPoIB with low performance (about 8-10GB/s on a Infiniband DDR 20Gb/s) This was 2 years ago. Any news about this? Is RDMA/Infiniband supported like with GlusterFS?

Re: [ceph-users] RDMA

the user land preloader library like sdp isn't enough? Is the kernel version needed just for librbd? Il giorno 18/apr/2013 23:48, "Mark Nelson" ha scritto: > On 04/18/2013 04:46 PM, Gandalf Corvotempesta wrote: > >> 2013/4/18 Mark Nelson : >> >>> SDP is deprecated: >>> >>> http://comments.gmane.o

Re: [ceph-users] RDMA

On Thu, 18 Apr 2013, Gandalf Corvotempesta wrote: > 2013/4/18 Sage Weil : > > I'm no expert, but I've heard SDP is not likely to be supported/maintained > > by anyone in the long-term. (Please, anyone, correct me if that is not > > true!) That said, one user has tested it successfully (with kerne

Re: [ceph-users] RDMA

On 04/18/2013 04:46 PM, Gandalf Corvotempesta wrote: 2013/4/18 Mark Nelson : SDP is deprecated: http://comments.gmane.org/gmane.network.openfabrics.enterprise/5371 rsockets is the future I think. I don't know rsockets. Any plans about support for this or are they "transparent" like SDP? I

Re: [ceph-users] RDMA

2013/4/18 Sage Weil : > I'm no expert, but I've heard SDP is not likely to be supported/maintained > by anyone in the long-term. (Please, anyone, correct me if that is not > true!) That said, one user has tested it successfully (with kernel and > userland ceph) and it does seem to work.. Do you

Re: [ceph-users] RDMA

2013/4/18 Mark Nelson : > SDP is deprecated: > > http://comments.gmane.org/gmane.network.openfabrics.enterprise/5371 > > rsockets is the future I think. I don't know rsockets. Any plans about support for this or are they "transparent" like SDP? ___ ceph-

Re: [ceph-users] RDMA

On 04/18/2013 04:15 PM, Gandalf Corvotempesta wrote: 2013/4/18 Mark Nelson : 10GbE is fully supported and widely used with Ceph while IB is a bit more complicated with fewer users. Having said that, IPoIB seems to work just fine, and there is potential in the future for even better performance.

Re: [ceph-users] RDMA

2013/4/18 Mark Nelson : > 10GbE is fully supported and widely used with Ceph while IB is a bit more > complicated with fewer users. Having said that, IPoIB seems to work just > fine, and there is potential in the future for even better performance. > Which one is right for you probably depends on

Re: [ceph-users] RDMA

On 04/18/2013 03:40 PM, Gandalf Corvotempesta wrote: Hi, will RDMA be supported in the shortterm? I'm planning an infrastructure and I don't know if starting with IB QDR or 10GbE. Depends on your definition of RDMA, supported, and short term. ;) We like the idea of using rsockets as it would b

[ceph-users] RDMA

Hi, will RDMA be supported in the shortterm? I'm planning an infrastructure and I don't know if starting with IB QDR or 10GbE. IB is much cheaper than 10GbE and with RDMA should be 4x faster, but with IPoIB as workaround I've read that is very very heavy on CPU and very slow (15gbit more or less)