On Sat, Sep 19, 2015 at 8:21 PM, Tom Eastep <[email protected]> wrote:
Attached is an inlineable action that accepts two parameters:
>
> - The number of bricks in the cluster :Default is 2
> - Enable Infiniband port 24008 (0 or 1) :Default is 0 (Don't open 24008)
>
> Add
>
> GlusterFS inline # Handles Gluster FS
>
> to /etc/shorewall/actions and place the attached file in /etc/shorewall.
>
> This will be a standard Shorewall action in 5.0.0 Beta 2.
>
Your action file works but now that I've played a bit more seriously with
GlusterFS, it proves a bit more complicated than anticipated concerning
firewall configuration :-) It's not _that_ hard but not ultra easy either.
Side note : A third parameter to "action.GlusterFS" would be advisable to
accept or not NFS. NFS can be disabled (enabled by default) on GlusterFS
and I do so because the native kernel NFSv4 server I currently use is fast
and very reliable. I don't use the GlusterFS NFS implementation nor do I
wish to open it to Gluster clients (which may differ from NFS clients on
the network).
The changes in "action.GlusterFS" are the following if you need/want to
integrate them in the upcoming release of Shorewall:
Change:
DEFAULTS 2,0
To:
DEFAULTS 2,0,1
Change:
my ( $bricks, $ib ) = get_action_params( 2 );
To:
my ( $bricks, $ib, $nfs ) = get_action_params( 3 );
Add:
fatal_error "Invalid value for NFS ( $nfs )" unless $nfs =~ /^[01]$/;
Surround:
ACCEPT - - udp 111,2049
ACCEPT - - tcp 38465:38467
With:
?if @{3}
...
?endif
End of side note.
So, here's the thing about GlusterFS :
Indeed, it needs several ports but not on the host itself! You you create a
volume consisting of 3 bricks, you'll have indeed some ports open starting
from 49151 but not every host and one port only in that list. Let's see an
example to undestand:
Shell# gluster volume status
Status of volume: TestVol
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick host1:/mnt/Data 49155 Y
14274
Brick host2:/mnt/Data 49156 Y
32082
Brick host3:/mnt/Data 49152 Y
30025
Self-heal Daemon on localhost N/A Y
32087
Self-heal Daemon on host2 N/A Y
30030
Self-heal Daemon on host3 N/A Y
14514
.
Task Status of Volume TestVol
------------------------------------------------------------------------------
There are no active volume tasks
Which means that Shorewall (on host1) should only handle tcp port 49155
(along with 24007 or 24007+24008 when using InfiniBand) because the other
ports are opened on the other bricks!
I've searched everywhere within /var/lib/glusterd/* but there's no
(computer-parsing-friendly) file there with all needed information (host,
ip and port) for the bricks.
The only mildly useful (missing port number) information about the bricks I
could find lies in "/var/lib/glusterd/peers/".
Shell# gluster peer status
gluster peer status
Number of Peers: 2
Hostname: 10.10.10.2
Uuid: 20954555-6982-48b2-904e-0ab4bf6f71cb
State: Peer in Cluster (Connected)
Other names:
host2
Hostname: host3
Uuid: 3bd03980-55a1-4da0-a2d6-c8e6f7e59eb6
State: Peer in Cluster (Connected)
Other names:
host3
In "/var/lib/glusterd/peers/", every file is named after the UUID above and
contains pure ASCII "key=value" items but port number unfortunately.
Shell# ls -1 /var/lib/glusterd/peers/
20954555-6982-48b2-904e-0ab4bf6f71cb
3bd03980-55a1-4da0-a2d6-c8e6f7e59eb6
Shell# cat
/var/lib/glusterd/peers/20954555-6982-48b2-904e-0ab4bf6f71cb
uuid=20954555-6982-48b2-904e-0ab4bf6f71cb
state=3
hostname1=10.10.10.2
hostname2=host2
Shell# cat /var/lib/glusterd/peers/3bd03980-55a1-4da0-a2d6-c8e6f7e59eb6
uuid=3bd03980-55a1-4da0-a2d6-c8e6f7e59eb6
state=3
hostname1=10.10.10.3
hostname2=host3
I don't yet understand why in "gluster peer status" host2 is identified by
IP address (Hostname: 10.10.10.2) and host3 is identified by hostname
(Hostname: host3) while the UUID files contain both info (IP and hostname).
So, as you can see, opening all the ports on the firewall is not needed but
there's a way to dynamically determine how to build the rules to allow the
network traffic to correctly flow between the bricks.
Do you want to give a shot at that challenge? :-)
--
ObNox
------------------------------------------------------------------------------
_______________________________________________
Shorewall-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/shorewall-users
