On 09/01/17 16:27, Andreas Kurzac wrote:
Hi Jiffin,
i raised bug 1411281.
If you could provide test-rpms i would be very happy to test them in
our environment.
Sorry If I missed in previous mail, which nfs-ganesha (2.3 or 2.4)
version are u using ?
--
Jiffin
In the meantime i will switch to tcp,rdma and continue working on our
setup, we can then switch back to pure rdma any time for testing.
Thanks for your help!
Regards,
Andreas
*Von:*Jiffin Tony Thottan [mailto:jthot...@redhat.com]
*Gesendet:* Montag, 9. Januar 2017 06:02
*An:* Andreas Kurzac <akur...@kinetik.de>; gluster-users@gluster.org
*Betreff:* Re: [Gluster-users] Ganesha with Gluster transport RDMA
does not work
Hi Andreas,
By checking the code IMO currently this is limitation with in
FSAL_GLUSTER. It tries to
establish connection with glusterfs servers only using "tcp". It is
easy to fix as well.
You can raise a bug in
https://bugzilla.redhat.com/enter_bug.cgi?product=nfs-ganesha
under FSAL_GLUSTER. I don't have any hardware to test the fix. I can
either help you in
writing up fix for the issue or provide a test rpms with the fix .
Also thanks for trying out nfs-ganesha with rdma and finding about
this issue.
For the time being , if possible you can try with tcp,rdma volume to
solve the problem.
Regards,
Jiffin
On 06/01/17 22:56, Andreas Kurzac wrote:
Dear All,
i have a glusterfs pool with 3 servers with Centos7.3, Glusterfs
3.8.5, network is Infiniband.
Pacemaker/Corosync and Ganesha-NFS is installed and all seems to
be OK, no error logged.
I created a replica 3 volume with transport rdma (without tcp!).
When i mount this volume via glusterfs and do some IO, no errors
are logged and everything seems to go pretty well.
When i mount the volume via nfs and do some IO, nfs freezes
immediatly and following logs are written to
ganesha-gfapi.log:
2017-01-05 23:23:53.536526] W [MSGID: 103004]
[rdma.c:452:gf_rdma_register_arena] 0-rdma: allocation of mr failed
[2017-01-05 23:23:53.541519] W [MSGID: 103004]
[rdma.c:1463:__gf_rdma_create_read_chunks_from_vector]
0-rpc-transport/rdma: memory registration failed
(peer:10.40.1.1:49152) [Keine Berechtigung]
[2017-01-05 23:23:53.541547] W [MSGID: 103029]
[rdma.c:1558:__gf_rdma_create_read_chunks] 0-rpc-transport/rdma:
cannot create read chunks from vector entry->prog_payload
[2017-01-05 23:23:53.541553] W [MSGID: 103033]
[rdma.c:2063:__gf_rdma_ioq_churn_request] 0-rpc-transport/rdma:
creation of read chunks failed
[2017-01-05 23:23:53.541557] W [MSGID: 103040]
[rdma.c:2775:__gf_rdma_ioq_churn_entry] 0-rpc-transport/rdma:
failed to process request ioq entry to peer(10.40.1.1:49152)
[2017-01-05 23:23:53.541562] W [MSGID: 103040]
[rdma.c:2859:gf_rdma_writev] 0-vmstor1-client-0: processing ioq
entry destined to (10.40.1.1:49152) failed
[2017-01-05 23:23:53.541569] W [MSGID: 103037]
[rdma.c:3016:gf_rdma_submit_request] 0-rpc-transport/rdma: sending
request to peer (10.40.1.1:49152) failed
[…]
Some additional info:
Firewall is disabled, SELinux is disabled.
Different hardware with Centos 7.1 and the Mellanox OFED 3.4
packages instead of the Centos Infiniband packages lead to the
same results.
Just to mention: I am not trying to do NFS over RDMA, the Ganesha
FSAL is just configured to "glusterfs".
I hope someone could help me, i am running out of ideas…
Kind regards,
Andreas
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
