Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Raghavendra Talur]

On Tue, Jun 6, 2017 at 10:13 AM, Jiffin Thottan  wrote:
>
>
> - Original Message -
> From: "hvjunk" 
> To: "Adam Ru" 
> Cc: gluster-users@gluster.org
> Sent: Monday, June 5, 2017 9:29:03 PM
> Subject: Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after 
> reboot
>
> Sorry, got sidetracked with invoicing etc.
>
> https://bitbucket.org/dismyne/gluster-ansibles/src/6df23803df43/ansible/files/?at=master
>
> The .service files are the stuff going into SystemD, and they call the 
> test-mounts.sh scripts.
> The playbook installing higher up in the directory
>
>
> I have submitted patch based on Hendrik scripts/systemd service file 
> https://review.gluster.org/#/c/17339/
>
> If everything works out, it can be included in next stable releases of 
> gluster(3.8.13 and 3.10.3)

This would be 3.10.4.

>
> On 05 Jun 2017, at 17:45 , Adam Ru < ad.ruc...@gmail.com > wrote:
>
> Hi hvjunk,
>
> could you please tell me have you had time to check my previous post?
>
> Could you please send me mentioned link to your Gluster Ansible scripts?
>
> Thank you,
>
> Adam
>
>
> On Sun, May 28, 2017 at 2:47 PM, Adam Ru < ad.ruc...@gmail.com > wrote:
>
>
> Hi hvjunk (Hi Hendrik),
>
> "centos-release-gluster" installs "centos-gluster310". I assume it
> picks the latest version and install it.
>
> Would you be so kind and send me a link to your script & systemd
> service / Ansible scripts? I cannot find a way how to list your posts
> on lists.gluster.org (I assume it's not possible to list posts of a
> specific user). Or if you cannot find it could you please tell me when
> did you post it? I'll try to find it.
>
> Meantime I wrote something very simple but I assume your scripting
> skills are better.
>
> Thank you.
>
> Kind regards.
>
> Adam
>
> --
>
> sudo sh -c 'cat > /root/gluster-run-ganesha << EOF
> #!/bin/bash
>
> while true; do
> echo "Wait"
> sleep 30
> if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
> ]]; then
> echo "Start Ganesha"
> systemctl start nfs-ganesha.service
> exit \$?
> else
> echo "Not mounted"
> fi
> done
> EOF'
>
> sudo chmod +x /root/gluster-run-ganesha
>
> sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF
> [Unit]
> Description=Start nfs-ganesha when Gluster shared storage is mounted
>
> [Service]
> Type=oneshot
> ExecStart=/root/gluster-run-ganesha
>
> [Install]
> WantedBy=multi-user.target
> EOF'
>
> sudo systemctl enable custom-gluster-ganesha.service
>
> --
>
>
> On Mon, May 15, 2017 at 12:27 PM, hvjunk < hvj...@gmail.com > wrote:
>
>
>
>
>
> On 15 May 2017, at 12:56 PM, Soumya Koduri < skod...@redhat.com > wrote:
>
>
>
> On 05/12/2017 06:27 PM, Adam Ru wrote:
>
>
> Hi Soumya,
>
> Thank you very much for last response – very useful.
>
> I apologize for delay, I had to find time for another testing.
>
> I updated instructions that I provided in previous e-mail. *** means
> that the step was added.
>
> Instructions:
> - Clean installation of CentOS 7.3 with all updates, 3x node,
> resolvable IPs and VIPs
> - Stopped firewalld (just for testing)
> - *** SELinux in permissive mode (I had to, will explain bellow)
> - Install “centos-release-gluster" to get "centos-gluster310" repo
>
> should I also install the centos-gluster310, or will that be automagically 
> chosen by the centos-release-gluster?
>
>
>
>
>
> and install following (nothing else):
> --- glusterfs-server
> --- glusterfs-ganesha
> - Passwordless SSH between all nodes
> (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
> - systemctl enable and start glusterd
> - gluster peer probe 
> - gluster volume set all cluster.enable-shared-storage enable
>
> After this step, I’ll advise (given my experience in doing this by Ansible) 
> to make sure that the shared filesystem have propagated to all the nodes, as 
> well as the needed entries made in fstab… safety check, and I’ll also load my 
> systemd service and helper script to assist in cluster cold-bootstrapping.
>
>
>
>
>
> - systemctl enable and start pcsd.service
> - systemctl enable pacemaker.service (cannot be started at this moment)
> - Set password for hacluster user on all nodes
> - pcs cluster auth-u hacluster -p blabla
> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
> sure if n

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Jiffin Thottan]

- Original Message -
From: "hvjunk" 
To: "Adam Ru" 
Cc: gluster-users@gluster.org
Sent: Monday, June 5, 2017 9:29:03 PM
Subject: Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after 
reboot

Sorry, got sidetracked with invoicing etc. 

https://bitbucket.org/dismyne/gluster-ansibles/src/6df23803df43/ansible/files/?at=master
 

The .service files are the stuff going into SystemD, and they call the 
test-mounts.sh scripts. 
The playbook installing higher up in the directory 


I have submitted patch based on Hendrik scripts/systemd service file 
https://review.gluster.org/#/c/17339/ 

If everything works out, it can be included in next stable releases of 
gluster(3.8.13 and 3.10.3)

On 05 Jun 2017, at 17:45 , Adam Ru < ad.ruc...@gmail.com > wrote: 

Hi hvjunk, 

could you please tell me have you had time to check my previous post? 

Could you please send me mentioned link to your Gluster Ansible scripts? 

Thank you, 

Adam 


On Sun, May 28, 2017 at 2:47 PM, Adam Ru < ad.ruc...@gmail.com > wrote: 


Hi hvjunk (Hi Hendrik), 

"centos-release-gluster" installs "centos-gluster310". I assume it 
picks the latest version and install it. 

Would you be so kind and send me a link to your script & systemd 
service / Ansible scripts? I cannot find a way how to list your posts 
on lists.gluster.org (I assume it's not possible to list posts of a 
specific user). Or if you cannot find it could you please tell me when 
did you post it? I'll try to find it. 

Meantime I wrote something very simple but I assume your scripting 
skills are better. 

Thank you. 

Kind regards. 

Adam 

-- 

sudo sh -c 'cat > /root/gluster-run-ganesha << EOF 
#!/bin/bash 

while true; do 
echo "Wait" 
sleep 30 
if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf 
]]; then 
echo "Start Ganesha" 
systemctl start nfs-ganesha.service 
exit \$? 
else 
echo "Not mounted" 
fi 
done 
EOF' 

sudo chmod +x /root/gluster-run-ganesha 

sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF 
[Unit] 
Description=Start nfs-ganesha when Gluster shared storage is mounted 

[Service] 
Type=oneshot 
ExecStart=/root/gluster-run-ganesha 

[Install] 
WantedBy=multi-user.target 
EOF' 

sudo systemctl enable custom-gluster-ganesha.service 

-- 


On Mon, May 15, 2017 at 12:27 PM, hvjunk < hvj...@gmail.com > wrote: 





On 15 May 2017, at 12:56 PM, Soumya Koduri < skod...@redhat.com > wrote: 



On 05/12/2017 06:27 PM, Adam Ru wrote: 


Hi Soumya, 

Thank you very much for last response – very useful. 

I apologize for delay, I had to find time for another testing. 

I updated instructions that I provided in previous e-mail. *** means 
that the step was added. 

Instructions: 
- Clean installation of CentOS 7.3 with all updates, 3x node, 
resolvable IPs and VIPs 
- Stopped firewalld (just for testing) 
- *** SELinux in permissive mode (I had to, will explain bellow) 
- Install “centos-release-gluster" to get "centos-gluster310" repo 

should I also install the centos-gluster310, or will that be automagically 
chosen by the centos-release-gluster? 





and install following (nothing else): 
--- glusterfs-server 
--- glusterfs-ganesha 
- Passwordless SSH between all nodes 
(/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes) 
- systemctl enable and start glusterd 
- gluster peer probe  
- gluster volume set all cluster.enable-shared-storage enable 

After this step, I’ll advise (given my experience in doing this by Ansible) to 
make sure that the shared filesystem have propagated to all the nodes, as well 
as the needed entries made in fstab… safety check, and I’ll also load my 
systemd service and helper script to assist in cluster cold-bootstrapping. 





- systemctl enable and start pcsd.service 
- systemctl enable pacemaker.service (cannot be started at this moment) 
- Set password for hacluster user on all nodes 
- pcs cluster auth-u hacluster -p blabla 
- mkdir /var/run/gluster/shared_storage/nfs-ganesha/ 
- touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not 
sure if needed) 
- vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and 
insert configuration 
- Try list files on other nodes: ls 
/var/run/gluster/shared_storage/nfs-ganesha/ 
- gluster nfs-ganesha enable 
- *** systemctl enable pacemaker.service (again, since pacemaker was 
disabled at this point) 
- *** Check owner of "state", "statd", "sm" and "sm.bak" in 
/var/lib/nfs/ (I had to: chown rpcuser:rpcuser 
/var/lib/nfs/statd/state) 
- Check on other nodes that nfs-ganesha.service is running and "pcs 
status" shows started resources 
- gluster volume create mynewshare replica 3 transport tcp 
node1:/ node2:/ node3:/ 
- gluster volume start mynewshare 

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[hvjunk]

Sorry, got sidetracked with invoicing etc.

https://bitbucket.org/dismyne/gluster-ansibles/src/6df23803df43/ansible/files/?at=master
 


The .service files are the stuff going into SystemD, and they call the 
test-mounts.sh scripts.
The playbook installing higher up in the directory

> On 05 Jun 2017, at 17:45 , Adam Ru  wrote:
> 
> Hi hvjunk,
> 
> could you please tell me have you had time to check my previous post?
> 
> Could you please send me mentioned link to your Gluster Ansible scripts?
> 
> Thank you,
> 
> Adam
> 
> 
> On Sun, May 28, 2017 at 2:47 PM, Adam Ru  wrote:
>> Hi hvjunk (Hi Hendrik),
>> 
>> "centos-release-gluster" installs "centos-gluster310". I assume it
>> picks the latest version and install it.
>> 
>> Would you be so kind and send me a link to your script & systemd
>> service / Ansible scripts? I cannot find a way how to list your posts
>> on lists.gluster.org (I assume it's not possible to list posts of a
>> specific user). Or if you cannot find it could you please tell me when
>> did you post it? I'll try to find it.
>> 
>> Meantime I wrote something very simple but I assume your scripting
>> skills are better.
>> 
>> Thank you.
>> 
>> Kind regards.
>> 
>> Adam
>> 
>> --
>> 
>> sudo sh -c 'cat > /root/gluster-run-ganesha << EOF
>> #!/bin/bash
>> 
>> while true; do
>>echo "Wait"
>>sleep 30
>>if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
>> ]]; then
>>echo "Start Ganesha"
>>systemctl start nfs-ganesha.service
>>exit \$?
>>else
>>echo "Not mounted"
>>fi
>> done
>> EOF'
>> 
>> sudo chmod +x /root/gluster-run-ganesha
>> 
>> sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF
>> [Unit]
>> Description=Start nfs-ganesha when Gluster shared storage is mounted
>> 
>> [Service]
>> Type=oneshot
>> ExecStart=/root/gluster-run-ganesha
>> 
>> [Install]
>> WantedBy=multi-user.target
>> EOF'
>> 
>> sudo systemctl enable custom-gluster-ganesha.service
>> 
>> --
>> 
>> 
>> On Mon, May 15, 2017 at 12:27 PM, hvjunk  wrote:
>>> 
 On 15 May 2017, at 12:56 PM, Soumya Koduri  wrote:
 
 
 
 On 05/12/2017 06:27 PM, Adam Ru wrote:
> Hi Soumya,
> 
> Thank you very much for last response – very useful.
> 
> I apologize for delay, I had to find time for another testing.
> 
> I updated instructions that I provided in previous e-mail. *** means
> that the step was added.
> 
> Instructions:
> - Clean installation of CentOS 7.3 with all updates, 3x node,
> resolvable IPs and VIPs
> - Stopped firewalld (just for testing)
> - *** SELinux in permissive mode (I had to, will explain bellow)
> - Install “centos-release-gluster" to get "centos-gluster310" repo
>>> 
>>> should I also install the centos-gluster310, or will that be automagically 
>>> chosen by the centos-release-gluster?
>>> 
> and install following (nothing else):
> --- glusterfs-server
> --- glusterfs-ganesha
> - Passwordless SSH between all nodes
> (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
> - systemctl enable and start glusterd
> - gluster peer probe 
> - gluster volume set all cluster.enable-shared-storage enable
>>> 
>>> After this step, I’ll advise (given my experience in doing this by Ansible) 
>>> to make sure that the shared filesystem have propagated to all the nodes, 
>>> as well as the needed entries made in fstab… safety check, and I’ll also 
>>> load my systemd service and helper script to assist in cluster 
>>> cold-bootstrapping.
>>> 
> - systemctl enable and start pcsd.service
> - systemctl enable pacemaker.service (cannot be started at this moment)
> - Set password for hacluster user on all nodes
> - pcs cluster auth-u hacluster -p blabla
> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
> sure if needed)
> - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
> insert configuration
> - Try list files on other nodes: ls
> /var/run/gluster/shared_storage/nfs-ganesha/
> - gluster nfs-ganesha enable
> - *** systemctl enable pacemaker.service (again, since pacemaker was
> disabled at this point)
> - *** Check owner of "state", "statd", "sm" and "sm.bak" in
> /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
> /var/lib/nfs/statd/state)
> - Check on other nodes that nfs-ganesha.service is running and "pcs
> status" shows started resources
> - gluster volume create mynewshare replica 3 transport tcp
> node1:/ node2:/ node3:/
> - gluster volume start mynewshare
> - gluster vol set mynewshare ganesha.enable on
> 
> At this moment, this is status of important (I think) services:
> 
> -- corosync.service di

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

Hi hvjunk,

could you please tell me have you had time to check my previous post?

Could you please send me mentioned link to your Gluster Ansible scripts?

Thank you,

Adam


On Sun, May 28, 2017 at 2:47 PM, Adam Ru  wrote:
> Hi hvjunk (Hi Hendrik),
>
> "centos-release-gluster" installs "centos-gluster310". I assume it
> picks the latest version and install it.
>
> Would you be so kind and send me a link to your script & systemd
> service / Ansible scripts? I cannot find a way how to list your posts
> on lists.gluster.org (I assume it's not possible to list posts of a
> specific user). Or if you cannot find it could you please tell me when
> did you post it? I'll try to find it.
>
> Meantime I wrote something very simple but I assume your scripting
> skills are better.
>
> Thank you.
>
> Kind regards.
>
> Adam
>
> --
>
> sudo sh -c 'cat > /root/gluster-run-ganesha << EOF
> #!/bin/bash
>
> while true; do
> echo "Wait"
> sleep 30
> if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
> ]]; then
> echo "Start Ganesha"
> systemctl start nfs-ganesha.service
> exit \$?
> else
> echo "Not mounted"
> fi
> done
> EOF'
>
> sudo chmod +x /root/gluster-run-ganesha
>
> sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF
> [Unit]
> Description=Start nfs-ganesha when Gluster shared storage is mounted
>
> [Service]
> Type=oneshot
> ExecStart=/root/gluster-run-ganesha
>
> [Install]
> WantedBy=multi-user.target
> EOF'
>
> sudo systemctl enable custom-gluster-ganesha.service
>
> --
>
>
> On Mon, May 15, 2017 at 12:27 PM, hvjunk  wrote:
>>
>>> On 15 May 2017, at 12:56 PM, Soumya Koduri  wrote:
>>>
>>>
>>>
>>> On 05/12/2017 06:27 PM, Adam Ru wrote:
 Hi Soumya,

 Thank you very much for last response – very useful.

 I apologize for delay, I had to find time for another testing.

 I updated instructions that I provided in previous e-mail. *** means
 that the step was added.

 Instructions:
 - Clean installation of CentOS 7.3 with all updates, 3x node,
 resolvable IPs and VIPs
 - Stopped firewalld (just for testing)
 - *** SELinux in permissive mode (I had to, will explain bellow)
 - Install “centos-release-gluster" to get "centos-gluster310" repo
>>
>> should I also install the centos-gluster310, or will that be automagically 
>> chosen by the centos-release-gluster?
>>
 and install following (nothing else):
 --- glusterfs-server
 --- glusterfs-ganesha
 - Passwordless SSH between all nodes
 (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
 - systemctl enable and start glusterd
 - gluster peer probe 
 - gluster volume set all cluster.enable-shared-storage enable
>>
>> After this step, I’ll advise (given my experience in doing this by Ansible) 
>> to make sure that the shared filesystem have propagated to all the nodes, as 
>> well as the needed entries made in fstab… safety check, and I’ll also load 
>> my systemd service and helper script to assist in cluster cold-bootstrapping.
>>
 - systemctl enable and start pcsd.service
 - systemctl enable pacemaker.service (cannot be started at this moment)
 - Set password for hacluster user on all nodes
 - pcs cluster auth-u hacluster -p blabla
 - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
 - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
 sure if needed)
 - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
 insert configuration
 - Try list files on other nodes: ls
 /var/run/gluster/shared_storage/nfs-ganesha/
 - gluster nfs-ganesha enable
 - *** systemctl enable pacemaker.service (again, since pacemaker was
 disabled at this point)
 - *** Check owner of "state", "statd", "sm" and "sm.bak" in
 /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
 /var/lib/nfs/statd/state)
 - Check on other nodes that nfs-ganesha.service is running and "pcs
 status" shows started resources
 - gluster volume create mynewshare replica 3 transport tcp
 node1:/ node2:/ node3:/
 - gluster volume start mynewshare
 - gluster vol set mynewshare ganesha.enable on

 At this moment, this is status of important (I think) services:

 -- corosync.service disabled
 -- corosync-notifyd.service disabled
 -- glusterd.service enabled
 -- glusterfsd.service   disabled
 -- pacemaker.serviceenabled
 -- pcsd.service enabled
 -- nfs-ganesha.service  disabled
 -- nfs-ganesha-config.service   static
 -- nfs-ganesha-lock.service static

 -- corosync.service active (running)
 -- corosync-notifyd.service inactive (dead)
 -- glusterd.service active (running)
 -- glusterfsd.service   inactive (dead)
 -- pace

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

;>>>  --- glusterfs-ganesha
>>>>>  - Passwordless SSH between all nodes (/var/lib/glusterd/nfs/secret.pem
>>>>> and secret.pem.pub on all nodes)
>>>>>  - systemctl enable and start glusterd
>>>>>  - gluster peer probe 
>>>>>  - gluster volume set all cluster.enable-shared-storage enable
>>>>>  - systemctl enable and start pcsd.service
>>>>>  - systemctl enable pacemaker.service (cannot be started at this moment)
>>>>>  - Set password for hacluster user on all nodes
>>>>>  - pcs cluster auth-u hacluster -p blabla
>>>>>  - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
>>>>>  - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
>>>>> sure if needed)
>>>>>  - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
>>>>> insert configuration
>>>>>  - Try list files on other nodes: ls
>>>>> /var/run/gluster/shared_storage/nfs-ganesha/
>>>>>  - gluster nfs-ganesha enable
>>>>>  - Check on other nodes that nfs-ganesha.service is running and "pcs
>>>>> status" shows started resources
>>>>>  - gluster volume create mynewshare replica 3 transport tcp node1:/
>>>>> node2:/ node3:/
>>>>>  - gluster volume start mynewshare
>>>>>  - gluster vol set mynewshare ganesha.enable on
>>>>>
>>>>> After these steps, all VIPs are pingable and I can mount
>>>>> node1:/mynewshare
>>>>>
>>>>> Funny thing is that pacemaker.service is disabled again (something
>>>>> disabled it). This is status of important (I think) services:
>>>>
>>>>
>>>>
>>>> yeah. We too had observed this recently. We guess probably pcs cluster
>>>> setup
>>>> command first destroys existing cluster (if any) which may be disabling
>>>> pacemaker too.
>>>>
>>>>>
>>>>> systemctl list-units --all
>>>>> # corosync.service loadedactive   running
>>>>> # glusterd.service loadedactive   running
>>>>> # nfs-config.service   loadedinactive dead
>>>>> # nfs-ganesha-config.service   loadedinactive dead
>>>>> # nfs-ganesha-lock.service loadedactive   running
>>>>> # nfs-ganesha.service  loadedactive   running
>>>>> # nfs-idmapd.service   loadedinactive dead
>>>>> # nfs-mountd.service   loadedinactive dead
>>>>> # nfs-server.service   loadedinactive dead
>>>>> # nfs-utils.serviceloadedinactive dead
>>>>> # pacemaker.serviceloadedactive   running
>>>>> # pcsd.service loadedactive   running
>>>>>
>>>>> systemctl list-unit-files --all
>>>>> # corosync-notifyd.servicedisabled
>>>>> # corosync.servicedisabled
>>>>> # glusterd.serviceenabled
>>>>> # glusterfsd.service  disabled
>>>>> # nfs-blkmap.service  disabled
>>>>> # nfs-config.service  static
>>>>> # nfs-ganesha-config.service  static
>>>>> # nfs-ganesha-lock.servicestatic
>>>>> # nfs-ganesha.service disabled
>>>>> # nfs-idmap.service   static
>>>>> # nfs-idmapd.service  static
>>>>> # nfs-lock.servicestatic
>>>>> # nfs-mountd.service  static
>>>>> # nfs-rquotad.service     disabled
>>>>> # nfs-secure-server.service   static
>>>>> # nfs-secure.service  static
>>>>> # nfs-server.service  disabled
>>>>> # nfs-utils.service   static
>>>>> # nfs.service disabled
>>>>> # nfslock.service static
>>>>> # pacemaker.service   disabled
>>>>> # pcsd.serviceenabled
>>>>>
>>>>> I enabled pacemaker again on all nodes and restart all nodes one by one.
>>>>>
>>>>> After reboot all VIPs are gone and I can see that nfs-ganesha.service
>>>>> isn’t running. When I start it on at least two nodes then VIPs are
>>>>> pingable again and I can mount NFS again. But there is still some issue
>>>&g

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

Hi hvjunk (Hi Hendrik),

"centos-release-gluster" installs "centos-gluster310". I assume it
picks the latest version and install it.

Would you be so kind and send me a link to your script & systemd
service / Ansible scripts? I cannot find a way how to list your posts
on lists.gluster.org (I assume it's not possible to list posts of a
specific user). Or if you cannot find it could you please tell me when
did you post it? I'll try to find it.

Meantime I wrote something very simple but I assume your scripting
skills are better.

Thank you.

Kind regards.

Adam

--

sudo sh -c 'cat > /root/gluster-run-ganesha << EOF
#!/bin/bash

while true; do
echo "Wait"
sleep 30
if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
]]; then
echo "Start Ganesha"
systemctl start nfs-ganesha.service
exit \$?
else
echo "Not mounted"
fi
done
EOF'

sudo chmod +x /root/gluster-run-ganesha

sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF
[Unit]
Description=Start nfs-ganesha when Gluster shared storage is mounted

[Service]
Type=oneshot
ExecStart=/root/gluster-run-ganesha

[Install]
WantedBy=multi-user.target
EOF'

sudo systemctl enable custom-gluster-ganesha.service

--


On Mon, May 15, 2017 at 12:27 PM, hvjunk  wrote:
>
>> On 15 May 2017, at 12:56 PM, Soumya Koduri  wrote:
>>
>>
>>
>> On 05/12/2017 06:27 PM, Adam Ru wrote:
>>> Hi Soumya,
>>>
>>> Thank you very much for last response – very useful.
>>>
>>> I apologize for delay, I had to find time for another testing.
>>>
>>> I updated instructions that I provided in previous e-mail. *** means
>>> that the step was added.
>>>
>>> Instructions:
>>> - Clean installation of CentOS 7.3 with all updates, 3x node,
>>> resolvable IPs and VIPs
>>> - Stopped firewalld (just for testing)
>>> - *** SELinux in permissive mode (I had to, will explain bellow)
>>> - Install “centos-release-gluster" to get "centos-gluster310" repo
>
> should I also install the centos-gluster310, or will that be automagically 
> chosen by the centos-release-gluster?
>
>>> and install following (nothing else):
>>> --- glusterfs-server
>>> --- glusterfs-ganesha
>>> - Passwordless SSH between all nodes
>>> (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
>>> - systemctl enable and start glusterd
>>> - gluster peer probe 
>>> - gluster volume set all cluster.enable-shared-storage enable
>
> After this step, I’ll advise (given my experience in doing this by Ansible) 
> to make sure that the shared filesystem have propagated to all the nodes, as 
> well as the needed entries made in fstab… safety check, and I’ll also load my 
> systemd service and helper script to assist in cluster cold-bootstrapping.
>
>>> - systemctl enable and start pcsd.service
>>> - systemctl enable pacemaker.service (cannot be started at this moment)
>>> - Set password for hacluster user on all nodes
>>> - pcs cluster auth-u hacluster -p blabla
>>> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
>>> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
>>> sure if needed)
>>> - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
>>> insert configuration
>>> - Try list files on other nodes: ls
>>> /var/run/gluster/shared_storage/nfs-ganesha/
>>> - gluster nfs-ganesha enable
>>> - *** systemctl enable pacemaker.service (again, since pacemaker was
>>> disabled at this point)
>>> - *** Check owner of "state", "statd", "sm" and "sm.bak" in
>>> /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
>>> /var/lib/nfs/statd/state)
>>> - Check on other nodes that nfs-ganesha.service is running and "pcs
>>> status" shows started resources
>>> - gluster volume create mynewshare replica 3 transport tcp
>>> node1:/ node2:/ node3:/
>>> - gluster volume start mynewshare
>>> - gluster vol set mynewshare ganesha.enable on
>>>
>>> At this moment, this is status of important (I think) services:
>>>
>>> -- corosync.service disabled
>>> -- corosync-notifyd.service disabled
>>> -- glusterd.service enabled
>>> -- glusterfsd.service   disabled
>>> -- pacemaker.serviceenabled
>>> -- pcsd.service enabled
>>> -- nfs-ganesha.service  disabled
>>> -- nfs-ganesha-config.service   static
>>> -- nfs-ganesha-lock.service static
>>>
>>> -- corosync.service active (running)
>>> -- corosync-notifyd.service inactive (dead)
>>> -- glusterd.service active (running)
>>> -- glusterfsd.service   inactive (dead)
>>> -- pacemaker.serviceactive (running)
>>> -- pcsd.service active (running)
>>> -- nfs-ganesha.service  active (running)
>>> -- nfs-ganesha-config.service   inactive (dead)
>>> -- nfs-ganesha-lock.service active (running)
>>>
>>> May I ask you a few questions please?
>>>
>>> 1. Could you please confirm that services above has correct status/state?
>>
>> Looks good to the b

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

d first destroys existing cluster (if any) which may be disabling
>>> pacemaker too.
>>>
>>>>
>>>> systemctl list-units --all
>>>> # corosync.service loadedactive   running
>>>> # glusterd.service loadedactive   running
>>>> # nfs-config.service   loadedinactive dead
>>>> # nfs-ganesha-config.service   loadedinactive dead
>>>> # nfs-ganesha-lock.service loadedactive   running
>>>> # nfs-ganesha.service  loadedactive   running
>>>> # nfs-idmapd.service   loadedinactive dead
>>>> # nfs-mountd.service   loadedinactive dead
>>>> # nfs-server.service   loadedinactive dead
>>>> # nfs-utils.serviceloadedinactive dead
>>>> # pacemaker.serviceloadedactive   running
>>>> # pcsd.service loadedactive   running
>>>>
>>>> systemctl list-unit-files --all
>>>> # corosync-notifyd.servicedisabled
>>>> # corosync.servicedisabled
>>>> # glusterd.serviceenabled
>>>> # glusterfsd.service  disabled
>>>> # nfs-blkmap.service  disabled
>>>> # nfs-config.service  static
>>>> # nfs-ganesha-config.service  static
>>>> # nfs-ganesha-lock.servicestatic
>>>> # nfs-ganesha.service disabled
>>>> # nfs-idmap.service   static
>>>> # nfs-idmapd.service  static
>>>> # nfs-lock.servicestatic
>>>> # nfs-mountd.service  static
>>>> # nfs-rquotad.service disabled
>>>> # nfs-secure-server.service   static
>>>> # nfs-secure.service  static
>>>> # nfs-server.service  disabled
>>>> # nfs-utils.service   static
>>>> # nfs.service disabled
>>>> # nfslock.service static
>>>> # pacemaker.service   disabled
>>>> # pcsd.serviceenabled
>>>>
>>>> I enabled pacemaker again on all nodes and restart all nodes one by one.
>>>>
>>>> After reboot all VIPs are gone and I can see that nfs-ganesha.service
>>>> isn’t running. When I start it on at least two nodes then VIPs are
>>>> pingable again and I can mount NFS again. But there is still some issue
>>>> in the setup because when I check nfs-ganesha-lock.service I get:
>>>>
>>>> systemctl -l status nfs-ganesha-lock.service
>>>> ● nfs-ganesha-lock.service - NFS status monitor for NFSv2/3 locking.
>>>>Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha-lock.service;
>>>> static; vendor preset: disabled)
>>>>Active: failed (Result: exit-code) since Fri 2017-05-05 13:43:37 UTC;
>>>> 31min ago
>>>>   Process: 6203 ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS
>>>> (code=exited, status=1/FAILURE)
>>>>
>>>> May 05 13:43:37 node0.localdomain systemd[1]: Starting NFS status
>>>> monitor for NFSv2/3 locking
>>>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Version 1.3.0
>>>> starting
>>>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Flags: TI-RPC
>>>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
>>>> directory sm: Permission denied
>>>
>>>
>>>
>>> Okay this issue was fixed and the fix should be present in 3.10 too -
>>>https://review.gluster.org/#/c/16433/
>>>
>>> Please check '/var/log/messages' for statd related errors and cross-check
>>> permissions of that directory. You could manually chown owner:group of
>>> /var/lib/nfs/statd/sm directory for now and then restart nfs-ganesha*
>>> services.
>>>
>>> Thanks,
>>> Soumya
>>>
>>>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
>>>> /var/lib/nfs/statd/state: Permission denied
>>>> May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service:
>>>> control process exited, code=exited status=1
>>>> May 05 13:43:37 node0.localdomain systemd[1]: Failed to start NFS status
>>>> monitor for NFSv2/3 locking..
>>>> May 05 13:43:37 node0.localdomain systemd[1]: Unit
>>>> nfs-ganesha-lock.service entered failed state.
>>>> May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[hvjunk]

> On 15 May 2017, at 12:56 PM, Soumya Koduri  wrote:
> 
> 
> 
> On 05/12/2017 06:27 PM, Adam Ru wrote:
>> Hi Soumya,
>> 
>> Thank you very much for last response – very useful.
>> 
>> I apologize for delay, I had to find time for another testing.
>> 
>> I updated instructions that I provided in previous e-mail. *** means
>> that the step was added.
>> 
>> Instructions:
>> - Clean installation of CentOS 7.3 with all updates, 3x node,
>> resolvable IPs and VIPs
>> - Stopped firewalld (just for testing)
>> - *** SELinux in permissive mode (I had to, will explain bellow)
>> - Install “centos-release-gluster" to get "centos-gluster310" repo

should I also install the centos-gluster310, or will that be automagically 
chosen by the centos-release-gluster?

>> and install following (nothing else):
>> --- glusterfs-server
>> --- glusterfs-ganesha
>> - Passwordless SSH between all nodes
>> (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
>> - systemctl enable and start glusterd
>> - gluster peer probe 
>> - gluster volume set all cluster.enable-shared-storage enable

After this step, I’ll advise (given my experience in doing this by Ansible) to 
make sure that the shared filesystem have propagated to all the nodes, as well 
as the needed entries made in fstab… safety check, and I’ll also load my 
systemd service and helper script to assist in cluster cold-bootstrapping.

>> - systemctl enable and start pcsd.service
>> - systemctl enable pacemaker.service (cannot be started at this moment)
>> - Set password for hacluster user on all nodes
>> - pcs cluster auth-u hacluster -p blabla
>> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
>> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
>> sure if needed)
>> - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
>> insert configuration
>> - Try list files on other nodes: ls
>> /var/run/gluster/shared_storage/nfs-ganesha/
>> - gluster nfs-ganesha enable
>> - *** systemctl enable pacemaker.service (again, since pacemaker was
>> disabled at this point)
>> - *** Check owner of "state", "statd", "sm" and "sm.bak" in
>> /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
>> /var/lib/nfs/statd/state)
>> - Check on other nodes that nfs-ganesha.service is running and "pcs
>> status" shows started resources
>> - gluster volume create mynewshare replica 3 transport tcp
>> node1:/ node2:/ node3:/
>> - gluster volume start mynewshare
>> - gluster vol set mynewshare ganesha.enable on
>> 
>> At this moment, this is status of important (I think) services:
>> 
>> -- corosync.service disabled
>> -- corosync-notifyd.service disabled
>> -- glusterd.service enabled
>> -- glusterfsd.service   disabled
>> -- pacemaker.serviceenabled
>> -- pcsd.service enabled
>> -- nfs-ganesha.service  disabled
>> -- nfs-ganesha-config.service   static
>> -- nfs-ganesha-lock.service static
>> 
>> -- corosync.service active (running)
>> -- corosync-notifyd.service inactive (dead)
>> -- glusterd.service active (running)
>> -- glusterfsd.service   inactive (dead)
>> -- pacemaker.serviceactive (running)
>> -- pcsd.service active (running)
>> -- nfs-ganesha.service  active (running)
>> -- nfs-ganesha-config.service   inactive (dead)
>> -- nfs-ganesha-lock.service active (running)
>> 
>> May I ask you a few questions please?
>> 
>> 1. Could you please confirm that services above has correct status/state?
> 
> Looks good to the best of my knowledge.
> 
>> 
>> 2. When I restart a node then nfs-ganesha is not running. Of course I
>> cannot enable it since it needs to be enabled after shared storage is
>> mounted. What is best practice to start it automatically so I don’t
>> have to worry about restarting node? Should I create a script that
>> will check whether shared storage was mounted and then start
>> nfs-ganesha? How do you do this in production?
> 
> That's right.. We have plans to address this in near future (probably by 
> having a new .service which mounts shared_storage before starting 
> nfs-ganesha). But until then ..yes having a custom defined script to do so is 
> the only way to automate it.

Refer to my previous posting that has a script & systemd service that address 
this problematic bootstrapping issue w.r.t. locally mounted gluster 
directories, which the shared directory is.
That could be used (with my permission) as a basis to help fix this issue…

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Soumya Koduri]

notifyd.servicedisabled
# corosync.servicedisabled
# glusterd.serviceenabled
# glusterfsd.service  disabled
# nfs-blkmap.service  disabled
# nfs-config.service  static
# nfs-ganesha-config.service  static
# nfs-ganesha-lock.servicestatic
# nfs-ganesha.service disabled
# nfs-idmap.service   static
# nfs-idmapd.service  static
# nfs-lock.servicestatic
# nfs-mountd.service  static
# nfs-rquotad.service disabled
# nfs-secure-server.service   static
# nfs-secure.service  static
# nfs-server.service  disabled
# nfs-utils.service   static
# nfs.service disabled
# nfslock.service static
# pacemaker.service   disabled
# pcsd.serviceenabled

I enabled pacemaker again on all nodes and restart all nodes one by one.

After reboot all VIPs are gone and I can see that nfs-ganesha.service
isn’t running. When I start it on at least two nodes then VIPs are
pingable again and I can mount NFS again. But there is still some issue
in the setup because when I check nfs-ganesha-lock.service I get:

systemctl -l status nfs-ganesha-lock.service
● nfs-ganesha-lock.service - NFS status monitor for NFSv2/3 locking.
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha-lock.service;
static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2017-05-05 13:43:37 UTC;
31min ago
  Process: 6203 ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS
(code=exited, status=1/FAILURE)

May 05 13:43:37 node0.localdomain systemd[1]: Starting NFS status
monitor for NFSv2/3 locking
May 05 13:43:37 node0.localdomain rpc.statd[6205]: Version 1.3.0 starting
May 05 13:43:37 node0.localdomain rpc.statd[6205]: Flags: TI-RPC
May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
directory sm: Permission denied



Okay this issue was fixed and the fix should be present in 3.10 too -
   https://review.gluster.org/#/c/16433/

Please check '/var/log/messages' for statd related errors and cross-check
permissions of that directory. You could manually chown owner:group of
/var/lib/nfs/statd/sm directory for now and then restart nfs-ganesha*
services.

Thanks,
Soumya


May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
/var/lib/nfs/statd/state: Permission denied
May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service:
control process exited, code=exited status=1
May 05 13:43:37 node0.localdomain systemd[1]: Failed to start NFS status
monitor for NFSv2/3 locking..
May 05 13:43:37 node0.localdomain systemd[1]: Unit
nfs-ganesha-lock.service entered failed state.
May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service
failed.

Thank you,

Kind regards,

Adam

On Wed, May 3, 2017 at 10:32 AM, Mahdi Adnan mailto:mahdi.ad...@outlook.com>> wrote:

Hi,


Same here, when i reboot the node i have to manually execute "pcs
cluster start gluster01" and pcsd already enabled and started.

Gluster 3.8.11

Centos 7.3 latest

Installed using CentOS Storage SIG repository



--

Respectfully*
**Mahdi A. Mahdi*



*From:* gluster-users-boun...@gluster.org
<mailto:gluster-users-boun...@gluster.org>
mailto:gluster-users-boun...@gluster.org>> on behalf of Adam Ru
mailto:ad.ruc...@gmail.com>>
    *Sent:* Wednesday, May 3, 2017 12:09:58 PM
    *To:* Soumya Koduri
*Cc:* gluster-users@gluster.org <mailto:gluster-users@gluster.org>
*Subject:* Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is

down after reboot

Hi Soumya,

thank you very much for your reply.

I enabled pcsd during setup and after reboot during troubleshooting
I manually started it and checked resources (pcs status). They were
not running. I didn’t find what was wrong but I’m going to try it
again.

I’ve thoroughly checked

http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

<http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/>
and I can confirm that I followed all steps with one exception. I
installed following RPMs:
glusterfs-server
glusterfs-fuse
glusterfs-cli
glusterfs-ganesha
nfs-ganesha-xfs

and the guide referenced above specifies:
glusterfs-server
glusterfs-api
glusterfs-ganesha

glusterfs-api is a dependency of one of RPMs that I installed so
this is not a problem. But I cannot find any mention to install
nfs-ganesha-xfs.

I’ll try to setup the whole environment again without installing
nfs-ganesha-xfs (I assume glusterfs-ganesha has all required
binaries).

Again, thank you for you time to answer my previous message.

Kind regards,
Adam

On Tue, May 2, 2017 at 8:49 AM, 

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

t-files --all
>> # corosync-notifyd.servicedisabled
>> # corosync.servicedisabled
>> # glusterd.serviceenabled
>> # glusterfsd.service  disabled
>> # nfs-blkmap.service  disabled
>> # nfs-config.service  static
>> # nfs-ganesha-config.service  static
>> # nfs-ganesha-lock.servicestatic
>> # nfs-ganesha.service disabled
>> # nfs-idmap.service   static
>> # nfs-idmapd.service  static
>> # nfs-lock.servicestatic
>> # nfs-mountd.service  static
>> # nfs-rquotad.service disabled
>> # nfs-secure-server.service   static
>> # nfs-secure.service  static
>> # nfs-server.service  disabled
>> # nfs-utils.service   static
>> # nfs.service disabled
>> # nfslock.service static
>> # pacemaker.service   disabled
>> # pcsd.serviceenabled
>>
>> I enabled pacemaker again on all nodes and restart all nodes one by one.
>>
>> After reboot all VIPs are gone and I can see that nfs-ganesha.service
>> isn’t running. When I start it on at least two nodes then VIPs are
>> pingable again and I can mount NFS again. But there is still some issue
>> in the setup because when I check nfs-ganesha-lock.service I get:
>>
>> systemctl -l status nfs-ganesha-lock.service
>> ● nfs-ganesha-lock.service - NFS status monitor for NFSv2/3 locking.
>>Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha-lock.service;
>> static; vendor preset: disabled)
>>Active: failed (Result: exit-code) since Fri 2017-05-05 13:43:37 UTC;
>> 31min ago
>>   Process: 6203 ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS
>> (code=exited, status=1/FAILURE)
>>
>> May 05 13:43:37 node0.localdomain systemd[1]: Starting NFS status
>> monitor for NFSv2/3 locking
>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Version 1.3.0 starting
>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Flags: TI-RPC
>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
>> directory sm: Permission denied
>
>
> Okay this issue was fixed and the fix should be present in 3.10 too -
>https://review.gluster.org/#/c/16433/
>
> Please check '/var/log/messages' for statd related errors and cross-check
> permissions of that directory. You could manually chown owner:group of
> /var/lib/nfs/statd/sm directory for now and then restart nfs-ganesha*
> services.
>
> Thanks,
> Soumya
>
>> May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
>> /var/lib/nfs/statd/state: Permission denied
>> May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service:
>> control process exited, code=exited status=1
>> May 05 13:43:37 node0.localdomain systemd[1]: Failed to start NFS status
>> monitor for NFSv2/3 locking..
>> May 05 13:43:37 node0.localdomain systemd[1]: Unit
>> nfs-ganesha-lock.service entered failed state.
>> May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service
>> failed.
>>
>> Thank you,
>>
>> Kind regards,
>>
>> Adam
>>
>> On Wed, May 3, 2017 at 10:32 AM, Mahdi Adnan > <mailto:mahdi.ad...@outlook.com>> wrote:
>>
>> Hi,
>>
>>
>> Same here, when i reboot the node i have to manually execute "pcs
>> cluster start gluster01" and pcsd already enabled and started.
>>
>> Gluster 3.8.11
>>
>> Centos 7.3 latest
>>
>> Installed using CentOS Storage SIG repository
>>
>>
>>
>> --
>>
>> Respectfully*
>> **Mahdi A. Mahdi*
>>
>>
>> 
>> *From:* gluster-users-boun...@gluster.org
>> <mailto:gluster-users-boun...@gluster.org>
>> > <mailto:gluster-users-boun...@gluster.org>> on behalf of Adam Ru
>> mailto:ad.ruc...@gmail.com>>
>> *Sent:* Wednesday, May 3, 2017 12:09:58 PM
>> *To:* Soumya Koduri
>> *Cc:* gluster-users@gluster.org <mailto:gluster-users@gluster.org>
>> *Subject:* Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is
>>
>> down after reboot
>>
>> Hi Soumya,
>>
>> thank you very much for your reply.
>>
>> I enabled pcsd during setup and after reboot during troubleshooting
>> I manually started it and checked resources (pcs status). They were
>> not running. I didn’t find what was w

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Soumya Koduri]

umya


May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open
/var/lib/nfs/statd/state: Permission denied
May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service:
control process exited, code=exited status=1
May 05 13:43:37 node0.localdomain systemd[1]: Failed to start NFS status
monitor for NFSv2/3 locking..
May 05 13:43:37 node0.localdomain systemd[1]: Unit
nfs-ganesha-lock.service entered failed state.
May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service
failed.

Thank you,

Kind regards,

Adam

On Wed, May 3, 2017 at 10:32 AM, Mahdi Adnan mailto:mahdi.ad...@outlook.com>> wrote:

Hi,


Same here, when i reboot the node i have to manually execute "pcs
cluster start gluster01" and pcsd already enabled and started.

Gluster 3.8.11

Centos 7.3 latest

Installed using CentOS Storage SIG repository



--

Respectfully*
**Mahdi A. Mahdi*


*From:* gluster-users-boun...@gluster.org
<mailto:gluster-users-boun...@gluster.org>
mailto:gluster-users-boun...@gluster.org>> on behalf of Adam Ru
mailto:ad.ruc...@gmail.com>>
*Sent:* Wednesday, May 3, 2017 12:09:58 PM
*To:* Soumya Koduri
*Cc:* gluster-users@gluster.org <mailto:gluster-users@gluster.org>
    *Subject:* Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is
down after reboot

Hi Soumya,

thank you very much for your reply.

I enabled pcsd during setup and after reboot during troubleshooting
I manually started it and checked resources (pcs status). They were
not running. I didn’t find what was wrong but I’m going to try it again.

I’ve thoroughly checked

http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

<http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/>
and I can confirm that I followed all steps with one exception. I
installed following RPMs:
glusterfs-server
glusterfs-fuse
glusterfs-cli
glusterfs-ganesha
nfs-ganesha-xfs

and the guide referenced above specifies:
glusterfs-server
glusterfs-api
glusterfs-ganesha

glusterfs-api is a dependency of one of RPMs that I installed so
this is not a problem. But I cannot find any mention to install
nfs-ganesha-xfs.

I’ll try to setup the whole environment again without installing
nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).

Again, thank you for you time to answer my previous message.

Kind regards,
Adam

On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri mailto:skod...@redhat.com>> wrote:

Hi,

On 05/02/2017 01:34 AM, Rudolf wrote:

Hi Gluster users,

First, I'd like to thank you all for this amazing
open-source! Thank you!

I'm working on home project – three servers with Gluster and
NFS-Ganesha. My goal is to create HA NFS share with three
copies of each
file on each server.

My systems are CentOS 7.3 Minimal install with the latest
updates and
the most current RPMs from "centos-gluster310" repository.

I followed this tutorial:

http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/

<http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/>
(second half that describes multi-node HA setup)

with a few exceptions:

1. All RPMs are from "centos-gluster310" repo that is
installed by "yum
-y install centos-release-gluster"
2. I have three nodes (not four) with "replica 3" volume.
3. I created empty ganesha.conf and not empty ganesha-ha.conf in
"/var/run/gluster/shared_storage/nfs-ganesha/" (referenced
blog post is
outdated, this is now requirement)
4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this
isn't needed
anymore.


Please refer to

http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

<http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/>

It is being updated with latest changes happened wrt setup.

When I finish configuration, all is good.
nfs-ganesha.service is active
and running and from client I can ping all three VIPs and I
can mount
NFS. Copied files are replicated to all nodes.

But when I restart nodes (one by one, w

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

 AM, Mahdi Adnan 
wrote:

> Hi,
>
>
> Same here, when i reboot the node i have to manually execute "pcs cluster
> start gluster01" and pcsd already enabled and started.
>
> Gluster 3.8.11
>
> Centos 7.3 latest
>
> Installed using CentOS Storage SIG repository
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* gluster-users-boun...@gluster.org  gluster.org> on behalf of Adam Ru 
> *Sent:* Wednesday, May 3, 2017 12:09:58 PM
> *To:* Soumya Koduri
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down
> after reboot
>
> Hi Soumya,
>
> thank you very much for your reply.
>
> I enabled pcsd during setup and after reboot during troubleshooting I
> manually started it and checked resources (pcs status). They were not
> running. I didn’t find what was wrong but I’m going to try it again.
>
> I’ve thoroughly checked
> http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%
> 20GlusterFS%20Integration/
> and I can confirm that I followed all steps with one exception. I
> installed following RPMs:
> glusterfs-server
> glusterfs-fuse
> glusterfs-cli
> glusterfs-ganesha
> nfs-ganesha-xfs
>
> and the guide referenced above specifies:
> glusterfs-server
> glusterfs-api
> glusterfs-ganesha
>
> glusterfs-api is a dependency of one of RPMs that I installed so this is
> not a problem. But I cannot find any mention to install nfs-ganesha-xfs.
>
> I’ll try to setup the whole environment again without installing
> nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).
>
> Again, thank you for you time to answer my previous message.
>
> Kind regards,
> Adam
>
> On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri  wrote:
>
>> Hi,
>>
>> On 05/02/2017 01:34 AM, Rudolf wrote:
>>
>>> Hi Gluster users,
>>>
>>> First, I'd like to thank you all for this amazing open-source! Thank you!
>>>
>>> I'm working on home project – three servers with Gluster and
>>> NFS-Ganesha. My goal is to create HA NFS share with three copies of each
>>> file on each server.
>>>
>>> My systems are CentOS 7.3 Minimal install with the latest updates and
>>> the most current RPMs from "centos-gluster310" repository.
>>>
>>> I followed this tutorial:
>>> http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-
>>> nfs-ganesha-and-glusterfs-one-step-at-a-time/
>>> (second half that describes multi-node HA setup)
>>>
>>> with a few exceptions:
>>>
>>> 1. All RPMs are from "centos-gluster310" repo that is installed by "yum
>>> -y install centos-release-gluster"
>>> 2. I have three nodes (not four) with "replica 3" volume.
>>> 3. I created empty ganesha.conf and not empty ganesha-ha.conf in
>>> "/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog post is
>>> outdated, this is now requirement)
>>> 4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed
>>> anymore.
>>>
>>>
>> Please refer to http://gluster.readthedocs.io/
>> en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
>>
>> It is being updated with latest changes happened wrt setup.
>>
>> When I finish configuration, all is good. nfs-ganesha.service is active
>>> and running and from client I can ping all three VIPs and I can mount
>>> NFS. Copied files are replicated to all nodes.
>>>
>>> But when I restart nodes (one by one, with 5 min. delay between) then I
>>> cannot ping or mount (I assume that all VIPs are down). So my setup
>>> definitely isn't HA.
>>>
>>> I found that:
>>> # pcs status
>>> Error: cluster is not currently running on this node
>>>
>>
>> This means pcsd service is not up. Did you enable (systemctl enable pcsd)
>> pcsd service so that is comes up post reboot automatically. If not please
>> start it manually.
>>
>>
>>> and nfs-ganesha.service is in inactive state. Btw. I didn't enable
>>> "systemctl enable nfs-ganesha" since I assume that this is something
>>> that Gluster does.
>>>
>>
>> Please check /var/log/ganesha.log for any errors/warnings.
>>
>> We recommend not to enable nfs-ganesha.service (by default), as the
>> shared storage (where the ganesha.conf file resides now) should be up and
>> running before nfs-ganesha

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Mahdi Adnan]

Hi,


Same here, when i reboot the node i have to manually execute "pcs cluster start 
gluster01" and pcsd already enabled and started.

Gluster 3.8.11

Centos 7.3 latest

Installed using CentOS Storage SIG repository


--

Respectfully
Mahdi A. Mahdi


From: gluster-users-boun...@gluster.org  on 
behalf of Adam Ru 
Sent: Wednesday, May 3, 2017 12:09:58 PM
To: Soumya Koduri
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after 
reboot

Hi Soumya,

thank you very much for your reply.

I enabled pcsd during setup and after reboot during troubleshooting I manually 
started it and checked resources (pcs status). They were not running. I didn’t 
find what was wrong but I’m going to try it again.

I’ve thoroughly checked
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
and I can confirm that I followed all steps with one exception. I installed 
following RPMs:
glusterfs-server
glusterfs-fuse
glusterfs-cli
glusterfs-ganesha
nfs-ganesha-xfs

and the guide referenced above specifies:
glusterfs-server
glusterfs-api
glusterfs-ganesha

glusterfs-api is a dependency of one of RPMs that I installed so this is not a 
problem. But I cannot find any mention to install nfs-ganesha-xfs.

I’ll try to setup the whole environment again without installing 
nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).

Again, thank you for you time to answer my previous message.

Kind regards,
Adam

On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri 
mailto:skod...@redhat.com>> wrote:
Hi,

On 05/02/2017 01:34 AM, Rudolf wrote:
Hi Gluster users,

First, I'd like to thank you all for this amazing open-source! Thank you!

I'm working on home project – three servers with Gluster and
NFS-Ganesha. My goal is to create HA NFS share with three copies of each
file on each server.

My systems are CentOS 7.3 Minimal install with the latest updates and
the most current RPMs from "centos-gluster310" repository.

I followed this tutorial:
http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/
(second half that describes multi-node HA setup)

with a few exceptions:

1. All RPMs are from "centos-gluster310" repo that is installed by "yum
-y install centos-release-gluster"
2. I have three nodes (not four) with "replica 3" volume.
3. I created empty ganesha.conf and not empty ganesha-ha.conf in
"/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog post is
outdated, this is now requirement)
4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed
anymore.


Please refer to 
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

It is being updated with latest changes happened wrt setup.

When I finish configuration, all is good. nfs-ganesha.service is active
and running and from client I can ping all three VIPs and I can mount
NFS. Copied files are replicated to all nodes.

But when I restart nodes (one by one, with 5 min. delay between) then I
cannot ping or mount (I assume that all VIPs are down). So my setup
definitely isn't HA.

I found that:
# pcs status
Error: cluster is not currently running on this node

This means pcsd service is not up. Did you enable (systemctl enable pcsd) pcsd 
service so that is comes up post reboot automatically. If not please start it 
manually.


and nfs-ganesha.service is in inactive state. Btw. I didn't enable
"systemctl enable nfs-ganesha" since I assume that this is something
that Gluster does.

Please check /var/log/ganesha.log for any errors/warnings.

We recommend not to enable nfs-ganesha.service (by default), as the shared 
storage (where the ganesha.conf file resides now) should be up and running 
before nfs-ganesha gets started.
So if enabled by default it could happen that shared_storage mount point is not 
yet up and it resulted in nfs-ganesha service failure. If you would like to 
address this, you could have a cron job which keeps checking the mount point 
health and then start nfs-ganesha service.

Thanks,
Soumya


I assume that my issue is that I followed instructions in blog post from
2015/10 that are outdated. Unfortunately I cannot find anything better –
I spent whole day by googling.

Would you be so kind and check the instructions in blog post and let me
know what steps are wrong / outdated? Or please do you have more current
instructions for Gluster+Ganesha setup?

Thank you.

Kind regards,
Adam



___
Gluster-users mailing list
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users




--
Adam
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Soumya Koduri]

On 05/03/2017 03:02 PM, Mahdi Adnan wrote:

Hi,


Same here, when i reboot the node i have to manually execute "pcs
cluster start gluster01" and pcsd already enabled and started.

Gluster 3.8.11

Centos 7.3 latest

Installed using CentOS Storage SIG repository




Could you please check too if the pacemaker service is also in started 
state post reboot of the node.


Thanks,
Soumya



--

Respectfully*
**Mahdi A. Mahdi*


*From:* gluster-users-boun...@gluster.org
 on behalf of Adam Ru

*Sent:* Wednesday, May 3, 2017 12:09:58 PM
*To:* Soumya Koduri
*Cc:* gluster-users@gluster.org
*Subject:* Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down
after reboot

Hi Soumya,

thank you very much for your reply.

I enabled pcsd during setup and after reboot during troubleshooting I
manually started it and checked resources (pcs status). They were not
running. I didn’t find what was wrong but I’m going to try it again.

I’ve thoroughly checked
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
and I can confirm that I followed all steps with one exception. I
installed following RPMs:
glusterfs-server
glusterfs-fuse
glusterfs-cli
glusterfs-ganesha
nfs-ganesha-xfs

and the guide referenced above specifies:
glusterfs-server
glusterfs-api
glusterfs-ganesha

glusterfs-api is a dependency of one of RPMs that I installed so this is
not a problem. But I cannot find any mention to install nfs-ganesha-xfs.

I’ll try to setup the whole environment again without installing
nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).

Again, thank you for you time to answer my previous message.

Kind regards,
Adam

On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri mailto:skod...@redhat.com>> wrote:

Hi,

On 05/02/2017 01:34 AM, Rudolf wrote:

Hi Gluster users,

First, I'd like to thank you all for this amazing open-source!
Thank you!

I'm working on home project – three servers with Gluster and
NFS-Ganesha. My goal is to create HA NFS share with three copies
of each
file on each server.

My systems are CentOS 7.3 Minimal install with the latest
updates and
the most current RPMs from "centos-gluster310" repository.

I followed this tutorial:

http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/

<http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/>
(second half that describes multi-node HA setup)

with a few exceptions:

1. All RPMs are from "centos-gluster310" repo that is installed
by "yum
-y install centos-release-gluster"
2. I have three nodes (not four) with "replica 3" volume.
3. I created empty ganesha.conf and not empty ganesha-ha.conf in
"/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog
post is
outdated, this is now requirement)
4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't
needed
anymore.


Please refer to

http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

<http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/>

It is being updated with latest changes happened wrt setup.

When I finish configuration, all is good. nfs-ganesha.service is
active
and running and from client I can ping all three VIPs and I can
mount
NFS. Copied files are replicated to all nodes.

But when I restart nodes (one by one, with 5 min. delay between)
then I
cannot ping or mount (I assume that all VIPs are down). So my setup
definitely isn't HA.

I found that:
# pcs status
Error: cluster is not currently running on this node


This means pcsd service is not up. Did you enable (systemctl enable
pcsd) pcsd service so that is comes up post reboot automatically. If
not please start it manually.


and nfs-ganesha.service is in inactive state. Btw. I didn't enable
"systemctl enable nfs-ganesha" since I assume that this is something
that Gluster does.


Please check /var/log/ganesha.log for any errors/warnings.

We recommend not to enable nfs-ganesha.service (by default), as the
shared storage (where the ganesha.conf file resides now) should be
up and running before nfs-ganesha gets started.
So if enabled by default it could happen that shared_storage mount
point is not yet up and it resulted in nfs-ganesha service failure.
If you would like to address this, you could

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Soumya Koduri]

On 05/03/2017 02:39 PM, Adam Ru wrote:

Hi Soumya,

thank you very much for your reply.

I enabled pcsd during setup and after reboot during troubleshooting I
manually started it and checked resources (pcs status). They were not
running. I didn’t find what was wrong but I’m going to try it again.


Oh..forgot to mention did you have 'pacemaker' service enabled too (if 
not needs to be started)


Thanks,
Soumya



I’ve thoroughly checked
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
and I can confirm that I followed all steps with one exception. I
installed following RPMs:
glusterfs-server
glusterfs-fuse
glusterfs-cli
glusterfs-ganesha
nfs-ganesha-xfs

and the guide referenced above specifies:
glusterfs-server
glusterfs-api
glusterfs-ganesha

glusterfs-api is a dependency of one of RPMs that I installed so this is
not a problem. But I cannot find any mention to install nfs-ganesha-xfs.

I’ll try to setup the whole environment again without installing
nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).

Again, thank you for you time to answer my previous message.

Kind regards,
Adam

On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri mailto:skod...@redhat.com>> wrote:

Hi,

On 05/02/2017 01:34 AM, Rudolf wrote:

Hi Gluster users,

First, I'd like to thank you all for this amazing open-source!
Thank you!

I'm working on home project – three servers with Gluster and
NFS-Ganesha. My goal is to create HA NFS share with three copies
of each
file on each server.

My systems are CentOS 7.3 Minimal install with the latest
updates and
the most current RPMs from "centos-gluster310" repository.

I followed this tutorial:

http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/


(second half that describes multi-node HA setup)

with a few exceptions:

1. All RPMs are from "centos-gluster310" repo that is installed
by "yum
-y install centos-release-gluster"
2. I have three nodes (not four) with "replica 3" volume.
3. I created empty ganesha.conf and not empty ganesha-ha.conf in
"/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog
post is
outdated, this is now requirement)
4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't
needed
anymore.


Please refer to

http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/



It is being updated with latest changes happened wrt setup.

When I finish configuration, all is good. nfs-ganesha.service is
active
and running and from client I can ping all three VIPs and I can
mount
NFS. Copied files are replicated to all nodes.

But when I restart nodes (one by one, with 5 min. delay between)
then I
cannot ping or mount (I assume that all VIPs are down). So my setup
definitely isn't HA.

I found that:
# pcs status
Error: cluster is not currently running on this node


This means pcsd service is not up. Did you enable (systemctl enable
pcsd) pcsd service so that is comes up post reboot automatically. If
not please start it manually.


and nfs-ganesha.service is in inactive state. Btw. I didn't enable
"systemctl enable nfs-ganesha" since I assume that this is something
that Gluster does.


Please check /var/log/ganesha.log for any errors/warnings.

We recommend not to enable nfs-ganesha.service (by default), as the
shared storage (where the ganesha.conf file resides now) should be
up and running before nfs-ganesha gets started.
So if enabled by default it could happen that shared_storage mount
point is not yet up and it resulted in nfs-ganesha service failure.
If you would like to address this, you could have a cron job which
keeps checking the mount point health and then start nfs-ganesha
service.

Thanks,
Soumya


I assume that my issue is that I followed instructions in blog
post from
2015/10 that are outdated. Unfortunately I cannot find anything
better –
I spent whole day by googling.

Would you be so kind and check the instructions in blog post and
let me
know what steps are wrong / outdated? Or please do you have more
current
instructions for Gluster+Ganesha setup?

Thank you.

Kind regards,
Adam



___
Gluster-users mailing lis

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

Hi Soumya,

thank you very much for your reply.

I enabled pcsd during setup and after reboot during troubleshooting I
manually started it and checked resources (pcs status). They were not
running. I didn’t find what was wrong but I’m going to try it again.

I’ve thoroughly checked
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
and I can confirm that I followed all steps with one exception. I installed
following RPMs:
glusterfs-server
glusterfs-fuse
glusterfs-cli
glusterfs-ganesha
nfs-ganesha-xfs

and the guide referenced above specifies:
glusterfs-server
glusterfs-api
glusterfs-ganesha

glusterfs-api is a dependency of one of RPMs that I installed so this is
not a problem. But I cannot find any mention to install nfs-ganesha-xfs.

I’ll try to setup the whole environment again without installing
nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).

Again, thank you for you time to answer my previous message.

Kind regards,
Adam

On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri  wrote:

> Hi,
>
> On 05/02/2017 01:34 AM, Rudolf wrote:
>
>> Hi Gluster users,
>>
>> First, I'd like to thank you all for this amazing open-source! Thank you!
>>
>> I'm working on home project – three servers with Gluster and
>> NFS-Ganesha. My goal is to create HA NFS share with three copies of each
>> file on each server.
>>
>> My systems are CentOS 7.3 Minimal install with the latest updates and
>> the most current RPMs from "centos-gluster310" repository.
>>
>> I followed this tutorial:
>> http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-
>> nfs-ganesha-and-glusterfs-one-step-at-a-time/
>> (second half that describes multi-node HA setup)
>>
>> with a few exceptions:
>>
>> 1. All RPMs are from "centos-gluster310" repo that is installed by "yum
>> -y install centos-release-gluster"
>> 2. I have three nodes (not four) with "replica 3" volume.
>> 3. I created empty ganesha.conf and not empty ganesha-ha.conf in
>> "/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog post is
>> outdated, this is now requirement)
>> 4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed
>> anymore.
>>
>>
> Please refer to http://gluster.readthedocs.io/
> en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/
>
> It is being updated with latest changes happened wrt setup.
>
> When I finish configuration, all is good. nfs-ganesha.service is active
>> and running and from client I can ping all three VIPs and I can mount
>> NFS. Copied files are replicated to all nodes.
>>
>> But when I restart nodes (one by one, with 5 min. delay between) then I
>> cannot ping or mount (I assume that all VIPs are down). So my setup
>> definitely isn't HA.
>>
>> I found that:
>> # pcs status
>> Error: cluster is not currently running on this node
>>
>
> This means pcsd service is not up. Did you enable (systemctl enable pcsd)
> pcsd service so that is comes up post reboot automatically. If not please
> start it manually.
>
>
>> and nfs-ganesha.service is in inactive state. Btw. I didn't enable
>> "systemctl enable nfs-ganesha" since I assume that this is something
>> that Gluster does.
>>
>
> Please check /var/log/ganesha.log for any errors/warnings.
>
> We recommend not to enable nfs-ganesha.service (by default), as the shared
> storage (where the ganesha.conf file resides now) should be up and running
> before nfs-ganesha gets started.
> So if enabled by default it could happen that shared_storage mount point
> is not yet up and it resulted in nfs-ganesha service failure. If you would
> like to address this, you could have a cron job which keeps checking the
> mount point health and then start nfs-ganesha service.
>
> Thanks,
> Soumya
>
>
>> I assume that my issue is that I followed instructions in blog post from
>> 2015/10 that are outdated. Unfortunately I cannot find anything better –
>> I spent whole day by googling.
>>
>> Would you be so kind and check the instructions in blog post and let me
>> know what steps are wrong / outdated? Or please do you have more current
>> instructions for Gluster+Ganesha setup?
>>
>> Thank you.
>>
>> Kind regards,
>> Adam
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>


-- 
Adam
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Soumya Koduri]

Hi,

On 05/02/2017 01:34 AM, Rudolf wrote:

Hi Gluster users,

First, I'd like to thank you all for this amazing open-source! Thank you!

I'm working on home project – three servers with Gluster and
NFS-Ganesha. My goal is to create HA NFS share with three copies of each
file on each server.

My systems are CentOS 7.3 Minimal install with the latest updates and
the most current RPMs from "centos-gluster310" repository.

I followed this tutorial:
http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/
(second half that describes multi-node HA setup)

with a few exceptions:

1. All RPMs are from "centos-gluster310" repo that is installed by "yum
-y install centos-release-gluster"
2. I have three nodes (not four) with "replica 3" volume.
3. I created empty ganesha.conf and not empty ganesha-ha.conf in
"/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog post is
outdated, this is now requirement)
4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed
anymore.



Please refer to 
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/


It is being updated with latest changes happened wrt setup.


When I finish configuration, all is good. nfs-ganesha.service is active
and running and from client I can ping all three VIPs and I can mount
NFS. Copied files are replicated to all nodes.

But when I restart nodes (one by one, with 5 min. delay between) then I
cannot ping or mount (I assume that all VIPs are down). So my setup
definitely isn't HA.

I found that:
# pcs status
Error: cluster is not currently running on this node


This means pcsd service is not up. Did you enable (systemctl enable 
pcsd) pcsd service so that is comes up post reboot automatically. If not 
please start it manually.




and nfs-ganesha.service is in inactive state. Btw. I didn't enable
"systemctl enable nfs-ganesha" since I assume that this is something
that Gluster does.


Please check /var/log/ganesha.log for any errors/warnings.

We recommend not to enable nfs-ganesha.service (by default), as the 
shared storage (where the ganesha.conf file resides now) should be up 
and running before nfs-ganesha gets started.
So if enabled by default it could happen that shared_storage mount point 
is not yet up and it resulted in nfs-ganesha service failure. If you 
would like to address this, you could have a cron job which keeps 
checking the mount point health and then start nfs-ganesha service.


Thanks,
Soumya



I assume that my issue is that I followed instructions in blog post from
2015/10 that are outdated. Unfortunately I cannot find anything better –
I spent whole day by googling.

Would you be so kind and check the instructions in blog post and let me
know what steps are wrong / outdated? Or please do you have more current
instructions for Gluster+Ganesha setup?

Thank you.

Kind regards,
Adam



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users