Sorry the mail daemon just batched me the rest of this conversation and
I see this is already done. please ignore.
-Mic
Mickey Mazarick wrote:
I had some difficulty getting OFED 1.3 working on kernel 2.6.27 about
6 months back. It took some patching but I did find that you needed
to have the srq enabled for it to work. The ibv_srq_pingpong test app
was a good test for weather it would work with gluster of not.
I also had to upgrade the firmware on the mellanox cards I have to
enable srq (send recieve que)
-Mic
Nathan Stratton wrote:
Hate to post again, but anyone have any ideas on this?
-Nathan
On Fri, 18 Sep 2009, Nathan Stratton wrote:
Has anyone been able to get Infiniband working with 2.6.31 kernel
and fuse 2.8.0? My config works fine on my Centos 2.6.18 box, so I
know that is ok.
Infiniband looks good:
[r...@xen1 src]# lsmod |grep ib
ib_ucm 13752 0
ib_uverbs 32256 2 rdma_ucm,ib_ucm
ib_ipoib 68880 0
ib_mthca 123700 0
[r...@xen1 src]# ibv_devices
device node GUID
------ ----------------
mthca0 0005ad00000327e8
Gluster looks like it starts OK, but I can't touch the mount and
after a while it times out. Debug logs:
[2009-09-18 19:36:17] D [glusterfsd.c:354:_get_specfp] glusterfs:
loading volume file /usr/local/etc/glusterfs/glusterfs.vol
================================================================================
Version : glusterfs 2.0.6 built on Sep 18 2009 09:54:43
TLA Revision : v2.0.6
Starting Time: 2009-09-18 19:36:17
Command line : glusterfs -L DEBUG -l /var/log/glusterfs.log
--disable-direct-io-mode /share
PID : 8303
System name : Linux
Nodename : xen1.hou.blinkmind.com
Kernel Release : 2.6.31
Hardware Identifier: x86_64
Given volfile:
+------------------------------------------------------------------------------+
1: volume brick0
2: type protocol/client
3: option transport-type ib-verbs/client
4: option remote-host 172.16.0.200
5: option remote-port 6997
6: option transport.address-family inet/inet6
7: option remote-subvolume brick
8: end-volume
9:
10: volume mirror0
11: type protocol/client
12: option transport-type ib-verbs/client
13: option remote-host 172.16.0.201
14: option remote-port 6997
15: option transport.address-family inet/inet6
16: option remote-subvolume brick
17: end-volume
18:
19: volume brick1
20: type protocol/client
21: option transport-type ib-verbs/client
22: option remote-host 172.16.0.202
23: option remote-port 6997
24: option transport.address-family inet/inet6
25: option remote-subvolume brick
26: end-volume
27:
28: volume mirror1
29: type protocol/client
30: option transport-type ib-verbs/client
31: option remote-host 172.16.0.203
32: option remote-port 6997
33: option transport.address-family inet/inet6
34: option remote-subvolume brick
35: end-volume
36:
37: volume brick2
38: type protocol/client
39: option transport-type ib-verbs/client
40: option remote-host 172.16.0.204
41: option remote-port 6997
42: option transport.address-family inet/inet6
43: option remote-subvolume brick
44: end-volume
45:
46: volume mirror2
47: type protocol/client
48: option transport-type ib-verbs/client
49: option remote-host 172.16.0.205
50: option remote-port 6997
51: option transport.address-family inet/inet6
52: option remote-subvolume brick
53: end-volume
54:
55: volume block0
56: type cluster/replicate
57: subvolumes brick0 mirror0
58: end-volume
59:
60: volume block1
61: type cluster/replicate
62: subvolumes brick1 mirror1
63: end-volume
64:
65: volume block2
66: type cluster/replicate
67: subvolumes brick2 mirror2
68: end-volume
69:
70: volume unify
71: type cluster/distribute
72: subvolumes block0 block1 block2
73: end-volume
74:
+------------------------------------------------------------------------------+
[2009-09-18 19:36:17] D [glusterfsd.c:1205:main] glusterfs: running
in pid 8303
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick0:
defaulting frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick0:
defaulting ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
brick0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
brick0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror0:
defaulting frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror0:
defaulting ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
mirror0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
mirror0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick1:
defaulting frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick1:
defaulting ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
brick1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
brick1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror1:
defaulting frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror1:
defaulting ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
mirror1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
mirror1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick2:
defaulting frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick2:
defaulting ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
brick2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
brick2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror2:
defaulting frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror2:
defaulting ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
mirror2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport:
attempt to load file
/usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate]
mirror2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] N [glusterfsd.c:1224:main] glusterfs:
Successfully started
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got
GF_EVENT_CHILD_UP
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] brick0: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] brick0: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] mirror0: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] mirror0: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] brick1: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] brick1: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] mirror1: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] mirror1: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] brick2: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] brick2: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] mirror2: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2:
bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17.
frame-timeout = 1800
[2009-09-18 20:06:18] D
[client-protocol.c:5491:client_setvolume_cbk] mirror2: setvolume
failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] D [dht-common.c:820:dht_lookup] unify: no
subvolume in layout for path=/, checking on all the subvols to see
if it is a directory
[2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify:
lookup of / on block0 returned error (Transport endpoint is not
connected)
[2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify:
lookup of / on block1 returned error (Transport endpoint is not
connected)
[2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify:
lookup of / on block2 returned error (Transport endpoint is not
connected)
[2009-09-18 20:06:18] D [fuse-bridge.c:2385:fuse_root_lookup_cbk]
fuse: first lookup on root failed.
[2009-09-18 20:06:18] W [fuse-bridge.c:1841:fuse_statfs_cbk]
glusterfs-fuse: 2: ERR => -1 (Transport endpoint is not connected)
<>
Nathan Stratton CTO, BlinkMind, Inc.
nathan at robotics.net nathan at blinkmind.com
http://www.robotics.net http://www.blinkmind.com
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
