Re: [Gluster-users] infiniband replicated distributed setup.--- network setup question...

[Matthew Temple]

Just got back to this after long delay.I have a slightly different
question now.   I have a working distributed/replicated setup with 4 bricks.
They are called:gf2ib-1, gf2ib-2,  gf2ib-1r, gf2ib-2r.

Running the "gluster peer status"  command from gf2ib-1 I get:

Number of Peers: 3

Hostname: gf2ib-2
Uuid: ed599401-25e2-41bb-8e41-b00247a3f3a0
State: Peer in Cluster (Connected)

Hostname: gf2ib-1r
Uuid: 8ea05f85-b416-4047-9e65-9a8185534bb6
State: Peer in Cluster (Connected)

Hostname: gf2ib-2r
Uuid: 34c58361-b3ef-4201-998b-09ff5c73558e
State: Peer in Cluster (Connected)

That's what I expect.   But from the other three peers, I get a variant of
this (the IP address of the first):

[root@gf2-2 rcadmin]# gluster peer status
Number of Peers: 3

Hostname: 172.24.215.202
Uuid: f5edea20-9467-48ed-b4f1-dc566a9b6d02
State: Peer in Cluster (Connected)

Hostname: gf2ib-1r
Uuid: 8ea05f85-b416-4047-9e65-9a8185534bb6
State: Peer in Cluster (Connected)

Hostname: gf2ib-2r
Uuid: 34c58361-b3ef-4201-998b-09ff5c73558e
State: Peer in Cluster (Connected)

Sure enough, when I look at /var/lib/glusterd/peers/* I see something like
this:

uuid=f5edea20-9467-48ed-b4f1-dc566a9b6d02
state=3
hostname1=172.24.215.202

Maybe it doesn't matter, but can I stop the volume and change that hostname
line?   The machine I'm looking at
does't have trouble with it's DNS entry:

[root@gf2-2r rcadmin]# nslookup 172.24.215.202
Server: 155.52.45.100
Address: 155.52.45.100#53

Non-authoritative answer:
202.215.24.172.in-addr.arpa name = gf2ib-1.dfci.harvard.edu.

The "volume info gf2" command gives the *names* of the 4 bricks, too.
Can anyone explain?

Matt Temple

--
Matt Temple
Director, Research Computing
Dana-Farber Cancer Institute.


On Wed, Dec 19, 2012 at 1:30 AM, Shawn Heisey  wrote:

> On 12/18/2012 7:33 PM, Matthew Temple wrote:
>
>> From any of those nodes, peer status is wrong:
>> For instance, if I ssh over to 2-ib,
>> and ask for a peer status, it shows the peers to to be
>>
>> 1-ib-r
>> 2-ib-r
>> and *155.52.48.1* (the last is the ethernet side of, not the IB side)
>>
>> I guess my question is this:   How do I get the first node to use the ib
>> side when
>> connecting to the other nodes. (both of these networks are public.)   Do
>> I have to
>> do something interesting with routes?
>>
>
> Just probe the first server by name from any of the other three.  That
> will fix the peers.
>
> Thanks,
> Shawn
>
> __**_
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.**org/mailman/listinfo/gluster-**users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] infiniband replicated distributed setup.--- network setup question...

[Shawn Heisey]

On 12/18/2012 7:33 PM, Matthew Temple wrote:

From any of those nodes, peer status is wrong:
For instance, if I ssh over to 2-ib,
and ask for a peer status, it shows the peers to to be

1-ib-r
2-ib-r
and *155.52.48.1* (the last is the ethernet side of, not the IB side)

I guess my question is this:   How do I get the first node to use the 
ib side when
connecting to the other nodes. (both of these networks are public.)   
Do I have to

do something interesting with routes?


Just probe the first server by name from any of the other three.  That 
will fix the peers.


Thanks,
Shawn

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] infiniband replicated distributed setup.--- network setup question...

[Matthew Temple]

We have two networks and I have 4 servers I'm trying to turn into a
distributed/replicated Gluster storage cluster.

Each of the servers has two addresses, one ethernet, one IB.
The externet addresses are 155.42.48.1-4 and
the Infiniband addresses are 172.24.215.1-4

I login to the first server ("1") which is 155.52.48.1.

Let's say the servers are named 1,2,1R and 2R (on the ethernet side), but
 1-IB, 2-IB, 1-IB-R and 2-IB-R on he Infiniband side.
The Volume is to be mounted by other servers in the IB network by native
Gluster clients.

so, I login to "1" over the ethernet network and do a peer probe.
All looks good from "1" with peer probe and peer status.

I probe the peers like this, successfully

gluster peer probe 2-ib
gluster peer probe 1-ib-r
gluster peer prove 2-ib-r

Peer status looks right.
I issue the following command which fails, of course:

gluster volume create gf2 replica 2 transport tcp,rdma \
1-ib:/mnt/d0-0 1-ib-r:/mnt/d0-0 2-ib:/mnt/d0-0 2-ib-r:/mnt/d0-0

>From any of those nodes, peer status is wrong:
For instance, if I ssh over to 2-ib,
and ask for a peer status, it shows the peers to to be

1-ib-r
2-ib-r
and *155.52.48.1* (the last is the ethernet side of, not the IB side)

I guess my question is this:   How do I get the first node to use the ib
side when
connecting to the other nodes. (both of these networks are public.)   Do I
have to
do something interesting with routes?

I need to fix this.   Don't most people with IB also have an internet side
to things?
How do their networks and any local routing info get set up?
Am I ,missing something really obvious?

Matt


--
Matt Temple
Director, Research Computing
Dana-Farber Cancer Institute.

We have two
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users