You have a host, fpsgluster, that's a peer. It's assigned a uuid.
You peer probe a different hostname, fpsglusterib, that reports the
same uuid - hence, already a peer.
You could remove it and re-add it with the new hostname unless you've
created volumes using the non-ib hostname. If you created any volumes
using fpsgluster, that hostname is now locked in. If you try to change
the hostname for that peer by probing it from another server, it should
fail because there's a volume using that hostname. It can't /add/ the
new hostname because that uuid is already in use.
If you want to brute-force it there's two ways of making it happen. Both
involve downtime.
1. delete the volume(s)
delete the server(s) from the peer group (gluster peer detach ...)
re-add the server(s) (gluster peer probe ...ib) remove the extended
attributes that mark a brick as having been part of a volume (
http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/
)
re-create the volume(s)
2. stop your volume(s) (gluster volume stop ...)
kill glusterd on all servers
search and replace all instances of the hostname under
/var/lib/glusterd on each server (find -type f /var/lib/glusterd |
xargs sed -i 's/fpsgluster/fpsglusterib/g') (This example presumes
that sed expression is safe. You'll have to ensure that for yourself.)
start glusterd on all servers
start your volume(s) (gluster volume start ...)
Method 1 is the "supported" method.
On 02/25/2013 03:07 PM, Tony Saenz wrote:
[root@fpsgluster ~]# gluster volume create testvault replica 2 transport
rdma,tcp fpsglusterib:/mnt/testbrick1 fpsgluster2ib:/mnt/testbrick1
fpsglusterib:/mnt/testbrick2 fpsgluster2ib:/mnt/testbrick2
fpsglusterib:/mnt/testbrick3 fpsgluster2ib:/mnt/testbrick3
Host fpsgluster2ib not a friend
Not a friend error. I'm at a loss.
On Feb 25, 2013, at 2:40 PM, harry mangalam <harry.manga...@uci.edu>
wrote:
That looks OK (but your 2 MTUs are mismatched - should fix that).
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 <- 1st
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 <- 2nd
^^^^^
IBDEV=ibX
modprobe ib_umad
modprobe ib_ipoib
echo connected > /sys/class/net/${IBDEV}/mode
echo 65520 > /sys/class/net/${IBDEV}/mtu
how did you set up the peering? By name? by IP#?
(I assume pinging by hostname also works both ways?)
If you can't get the peers to ack, then what do the logs say on failure to:
peer probe <host>
or create the volume
gluster volume create <volname> host1ib:/gl_part host2ib:/gl_part
hjm
On Monday, February 25, 2013 09:50:02 PM Tony Saenz wrote:
Trying to first get this working with IPoIB
[root@fpsgluster ~]# ibhosts
Ca : 0x00117500007937b2 ports 1 "fpsgluster2 qib0"
Ca : 0x0011750000792af2 ports 1 "fpsgluster qib0"
I'm able to ping the other box from Infiniband to Infiniband card
Ifconfig uses the ioctl access method to get the full address information,
which limits hardware addresses to 8 bytes. Because Infiniband address has
20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
obsolete! For replacement check ip.
ib0 Link encap:InfiniBand HWaddr
80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
addr:10.0.4.35 Bcast:10.0.4.255 Mask:255.255.255.0 inet6 addr:
fe80::211:7500:79:2af2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:1567 errors:0 dropped:0 overruns:0 frame:0
TX packets:587 errors:0 dropped:24 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:342622 (334.5 KiB) TX bytes:96554 (94.2 KiB)
[root@fpsgluster2 ~]# ifconfig ib0
Ifconfig uses the ioctl access method to get the full address information,
which limits hardware addresses to 8 bytes. Because Infiniband address has
20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
obsolete! For replacement check ip.
ib0 Link encap:InfiniBand HWaddr
80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
addr:10.0.4.34 Bcast:10.0.4.255 Mask:255.255.255.0 inet6 addr:
fe80::211:7500:79:37b2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:599 errors:0 dropped:0 overruns:0 frame:0
TX packets:1558 errors:0 dropped:8 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:95180 (92.9 KiB) TX bytes:346728 (338.6 KiB)
[root@fpsgluster ~]# ping -I ib0 10.0.4.34
PING 10.0.4.34 (10.0.4.34) from 10.0.4.35 ib0: 56(84) bytes of data.
64 bytes from 10.0.4.34: icmp_seq=1 ttl=64 time=12.6 ms
64 bytes from 10.0.4.34: icmp_seq=2 ttl=64 time=0.184 ms
/etc/hosts looks correct
[root@fpsgluster2 ~]# cat /etc/hosts | grep ib
10.0.4.35 fpsglusterib
10.0.4.34 fpsgluster2ib
[root@fpsgluster ~]# cat /etc/hosts| grep ib
10.0.4.35 fpsglusterib
10.0.4.34 fpsgluster2ib
I haven't created the new volume yet as I can't get the peer probe to work
off the Infiniband card. It's only seeing the NIC cards I currently have it
hooked in to.
On Feb 25, 2013, at 11:57 AM, harry mangalam <harry.manga...@uci.edu>
wrote:
It /might be/probably is/ DNS-related.
Are you trying to do this with RDMA or IPoIB?
If IPoIB, are ALL your /etc/hosts files in sync (IB names separate and
distinct from the ethernet interfaces) and responsive to the appropriate
interfaces?
Do the IB interfaces show up as distinct (and connected) on an 'ifconfig
-a' and 'ibstat' dump?
Do all the peers show up on an 'ibhosts' query?
What is the output of:
gluster volume status <your_volume>
and
gluster volume status <your_volume> detail
hjm
On Monday, February 25, 2013 07:46:00 PM Tony Saenz wrote:
It shows this but it's still going through my NIC cards and not the
Infiniband. (Checked the traffic on the cards themselves)
[root@fpsgluster ~]# gluster peer status
Number of Peers: 1
Hostname: fpsgluster2
Uuid: 9b7e7c2d-f05b-4cc8-b55a-571e383328d0
State: Peer in Cluster (Connected)
On Feb 25, 2013, at 10:51 AM, Torbjørn Thorsen <torbj...@trollweb.no>
wrote:
Your error message seems to indicate that the peer is already in the
storage pool ?
What is the output of "gluster peer status" ?
On Mon, Feb 25, 2013 at 7:28 PM, Tony Saenz <t...@filmsolutions.com>
wrote:
Any help please? The regular NICs are fine which is what it currently
sees but I'd like to move them over to the Infiniband cards.>>
On Feb 22, 2013, at 1:50 PM, Anthony Saenz <t...@filmsolutions.com>
wrote:
Hey,
I was wondering if I could get a bit of help.. I installed a new
Infiniband card into my servers but I'm unable to get it to come up as
a peer. Is there something I'm missing?
[root@fpsgluster testvault]# gluster peer probe fpsgluster2ib
Probe on host fpsgluster2ib port 0 already in peer list
[root@fpsgluster testvault]# yum list installed | grep gluster
glusterfs.x86_64 3.3.1-1.el6
installed
glusterfs-devel.x86_64 3.3.1-1.el6
installed
glusterfs-fuse.x86_64 3.3.1-1.el6
installed
glusterfs-geo-replication.x86_64 3.3.1-1.el6
installed
glusterfs-rdma.x86_64 3.3.1-1.el6
installed
glusterfs-server.x86_64 3.3.1-1.el6
installed
Thanks.
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
Vennlig hilsen
Torbjørn Thorsen
Utvikler / driftstekniker
Trollweb Solutions AS
- Professional Magento Partner
www.trollweb.no
Telefon dagtid: +47 51215300
Telefon kveld/helg: For kunder med Serviceavtale
Besøksadresse: Luramyrveien 40, 4313 Sandnes
Postadresse: Maurholen 57, 4316 Sandnes
Husk at alle våre standard-vilkår alltid er gjeldende
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
"Something must be done. [X] is something. Therefore, we must do it."
Bruce Schneier, on American response to just about anything.
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
"Something must be done. [X] is something. Therefore, we must do it."
Bruce Schneier, on American response to just about anything.
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
