Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Yes dig and ping works fine. I used first the short hostname gfs1 and then I also tried gfs1.intra.domain.com. That did not change anything. Currently for testing I only have a single node setup so my gluster peer status output would be empty. Now I had a closed look at the brick logfile and it looks like my network is not read at the time glusterfsd kicks in. I can see the following entry: [2015-03-03 13:45:48.647981] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com happening at exactly 13:45:48 but on my Cisco Switch I can see the following log entry: Mar 3 13:45:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to up So as you see my switch's port is only ready 3 seconds later at: 13:45:51, which is too late for glusterfsd. Strangely enough I use spanning-tree portfast on all my ports and as such the switch ports are ready as soon as possible. I have the feeling here that there is an issue with the /etc/init.d/glusterfs-server script on Debian wheezy from the glusterfs-server package. Can anyone confirm and maybe fix this for the future? On Tuesday, March 3, 2015 12:57 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Yes, so I added a sleep 5 in the init script right before the startup of the daemon here: start-stop-daemon --start --quiet --oknodo --pidfile $PIDFILE --startas $DAEMON -- -p $PIDFILE $GLUSTERD_OPTS and yes it works. So this is a workaround but IMHO this should get fixed into the Debian package. Anyone from Gluster reading this? I've cc'ed the init script's authors just in case. On Tuesday, March 3, 2015 3:53 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Probably a stupid workaround but what if you just add a sleep n line into the init script? Glusterfs is working fine if it was launched manually, did I understand that right? It's only the automatic startup at boot which causes the lookup failure? On Tue, Mar 3, 2015 at 2:54 PM, ML mail mlnos...@yahoo.com wrote: Thanks for the tip but Debian wheezy does not use systemd at all, it's still old sysV style init scripts. On Tuesday, March 3, 2015 2:43 PM, Jeremy Young jrm16...@gmail.com wrote: I know that you mention the script in init.d but want to ask to make sure that you're not using systemd. Are you still using SysVinit with wheezy? I've had this exact problem with RHEL 7, and creating the file /etc/systemd/system/glusterd.service.d/restart.conf, containing the information below, resolved my issue. [Service] Restart=on-failure StartLimitInterval=5s StartLimitBurst=5 On Tue, Mar 3, 2015 at 7:04 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: So that would sound like the boot order is messed up. Sounds a bit like this: http://joejulian.name/blog/glusterfs-volumes-not-mounting-in-debian-squeeze-at-boot-time/ As I said before, I also run glusterfs 3.5.2 on Debian Wheezy and I don't have issues after a reboot. But unfortunately I don't remember if I had to manually adapt something like the boot order of the init scripts). You can try and make your gluster scripts run at the very end of the boot process and see if that helps. On Tue, Mar 3, 2015 at 1:56 PM, ML mail mlnos...@yahoo.com wrote: Yes dig and ping works fine. I used first the short hostname gfs1 and then I also tried gfs1.intra.domain.com. That did not change anything. Currently for testing I only have a single node setup so my gluster peer status output would be empty. Now I had a closed look at the brick logfile and it looks like my network is not read at the time glusterfsd kicks in. I can see the following entry: [2015-03-03 13:45:48.647981] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com happening at exactly 13:45:48 but on my Cisco Switch I can see the following log entry: Mar 3 13:45:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to up So as you see my switch's port is only ready 3 seconds later at: 13:45:51, which is too late for glusterfsd. Strangely enough I use spanning-tree portfast on all my ports and as such the switch ports are ready as soon as possible. I have the feeling here that there is an issue with the /etc/init.d/glusterfs-server script on Debian wheezy from the glusterfs-server package. Can anyone confirm and maybe fix this for the future? On Tuesday, March 3, 2015 12:57 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in
[Gluster-users] REMINDER: Gluster Community Bug Triage meeting today at 12:00 UTC
Hi all, This meeting is scheduled for anyone that is interested in learning more about, or assisting with the Bug Triage. Meeting details: - location: #gluster-meeting on Freenode IRC ( https://webchat.freenode.net/?channels=gluster-meeting ) - date: every Tuesday - time: 12:00 UTC, 13:00 CET (in your terminal, run: date -d 12:00 UTC) - agenda: https://public.pad.fsfe.org/p/gluster-bug-triage Currently the following items are listed: * Roll Call * Status of last weeks action items * Group Triage * Open Floor The last two topics have space for additions. If you have a suitable bug or topic to discuss, please add it to the agenda. Thanks, Niels pgpXSpp5_4enA.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers, ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
So that would sound like the boot order is messed up. Sounds a bit like this: http://joejulian.name/blog/glusterfs-volumes-not-mounting-in-debian-squeeze-at-boot-time/ As I said before, I also run glusterfs 3.5.2 on Debian Wheezy and I don't have issues after a reboot. But unfortunately I don't remember if I had to manually adapt something like the boot order of the init scripts). You can try and make your gluster scripts run at the very end of the boot process and see if that helps. On Tue, Mar 3, 2015 at 1:56 PM, ML mail mlnos...@yahoo.com wrote: Yes dig and ping works fine. I used first the short hostname gfs1 and then I also tried gfs1.intra.domain.com. That did not change anything. Currently for testing I only have a single node setup so my gluster peer status output would be empty. Now I had a closed look at the brick logfile and it looks like my network is not read at the time glusterfsd kicks in. I can see the following entry: [2015-03-03 13:45:48.647981] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com happening at exactly 13:45:48 but on my Cisco Switch I can see the following log entry: Mar 3 13:45:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to up So as you see my switch's port is only ready 3 seconds later at: 13:45:51, which is too late for glusterfsd. Strangely enough I use spanning-tree portfast on all my ports and as such the switch ports are ready as soon as possible. I have the feeling here that there is an issue with the /etc/init.d/glusterfs-server script on Debian wheezy from the glusterfs-server package. Can anyone confirm and maybe fix this for the future? On Tuesday, March 3, 2015 12:57 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
I tried Claudio's advice by renaming /etc/rc2.d/S12glusterfs-server to /etc/rc2.d/S99glusterfs-server but it did not change anything. It looks like that even at the S99 stage the network is just not ready by something like 1 second (based on the log files of gluster and my switch). Any other ideas? On Tuesday, March 3, 2015 2:05 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: So that would sound like the boot order is messed up. Sounds a bit like this: http://joejulian.name/blog/glusterfs-volumes-not-mounting-in-debian-squeeze-at-boot-time/ As I said before, I also run glusterfs 3.5.2 on Debian Wheezy and I don't have issues after a reboot. But unfortunately I don't remember if I had to manually adapt something like the boot order of the init scripts). You can try and make your gluster scripts run at the very end of the boot process and see if that helps. On Tue, Mar 3, 2015 at 1:56 PM, ML mail mlnos...@yahoo.com wrote: Yes dig and ping works fine. I used first the short hostname gfs1 and then I also tried gfs1.intra.domain.com. That did not change anything. Currently for testing I only have a single node setup so my gluster peer status output would be empty. Now I had a closed look at the brick logfile and it looks like my network is not read at the time glusterfsd kicks in. I can see the following entry: [2015-03-03 13:45:48.647981] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com happening at exactly 13:45:48 but on my Cisco Switch I can see the following log entry: Mar 3 13:45:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to up So as you see my switch's port is only ready 3 seconds later at: 13:45:51, which is too late for glusterfsd. Strangely enough I use spanning-tree portfast on all my ports and as such the switch ports are ready as soon as possible. I have the feeling here that there is an issue with the /etc/init.d/glusterfs-server script on Debian wheezy from the glusterfs-server package. Can anyone confirm and maybe fix this for the future? On Tuesday, March 3, 2015 12:57 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org
[Gluster-users] GSoC-2015 organization application rejected
Hi everyone. I'm disappointed to announce that our application to be a mentoring organization for GSoC-2015 has been rejected. Google has accepted 137 organizations [1] this year (down from 190 last year) and couldn't find place to accommodate us. The main reason behind the rejection was the presentation of our ideas list [2]. Though we had a good number of ideas, Google expected them to be more fleshed out and followed guidelines given at [3]. This some thing we must do and doing this will help new contributors. This was our communities first ever attempt, so we were bound to have some missteps. We'll prepare better for the next time and try again. In the meantime, anyone who was hoping to contribute via GSoC, but would still like to contribute to the community anyway, take a look at our ideas list [2]. If you'd like to work on something, get in touch with the listed mentor or with development mailing list. I want to thank the community members who volunteered to be mentors, and who came up with the ideas. I want to thank Spot, for helping me get our application submitted in a rush. Thanks everyone. Let's try again next year. - Kaushal [1]: https://www.google-melange.com/gsoc/org/list/public/google/gsoc2015 [2]: http://www.gluster.org/community/documentation/index.php/Projects [3]: http://en.flossmanuals.net/GSoCMentoring/making-your-ideas-page/ ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Isssue with glusterFS on docker
On Mon, Feb 23, 2015 at 11:43 AM, Manju Nath manjubs.m...@gmail.com wrote: Hi, We have installed glusterFS 3.5.1 on a docker container. Container is installed with RHEL7. We have executed following commands, but after mount if i check the mounted partition we are getting Input/output error. Can any one please help us what is the reason for this? gluster peer probe GlusterFSgeo gluster peer probe GlusterFSprimary gluster peer probe GlusterFSsecondary gluster volume create gv-afr replica 3 transport tcp GlusterFSprimary:/DGdata/ReplicatedStore GlusterFSsecondary:/DGdata/ReplicatedStore GlusterFSgeo:/DGdata/ReplicatedStore/ force /usr/sbin/gluster volume start gv-afr mount -t glusterfs $LOCAL_IP_ADDRESS:gv-afr /mnt/glusterfs/ [root@BP kodiak]# ls /mnt/glusterfs/cdr ls: cannot access /mnt/glusterfs/cdr: Input/output error [root@BP kodiak]# I have been able to run gluster in docker before. There might be many reasons why you get IO error. Most likely is that you do not have a dev/fuse file and mount failed. Looking at the mount log should give more info. Here is the command to get the dev fuse file. mknod /dev/fuse c 10 229 NOTE: You have to do that in a running container in privileged mode. Here is my github repo having dockerfile and helper scripts. https://github.com/raghavendra-talur/Gluster-in-Docker Thanks, Raghavendra Talur -- Regards Manjunath ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] [volume options] auth.allow fails to resolve hostname
Hi, [environment] 1. gluster version: 3.5.3 2. os: redhat6.5 3. volume info shown below: Volume Name: gv0 Type: Replicate Volume ID: 37742c27-9f6b-4a38-821f-ea68d1ec8950 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: dmf-ha-1-glusterfs:/export/vdb1/brick Brick2: dmf-ha-2-glusterfs:/export/vdb1/brick Options Reconfigured: cluster.server-quorum-type: server performance.cache-refresh-timeout: 60 performance.io-thread-count: 64 performance.cache-size: 8GB cluster.eager-lock: ON auth.allow: dmf-ha-1-glusterfs,dmf-ha-2-glusterfs nfs.disable: ON cluster.server-quorum-ratio: 51% [issue] When value of auth.allow is set to hostname of the client, one node mount succeeds, one node mount fails. If auth.allow value is set to the IP address of the client, both nodes mount succeed. The error log list below: [2015-03-04 04:52:38.580635] I [client-handshake.c:1677:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-03-04 04:52:38.580956] W [client-handshake.c:1371:client_setvolume_cbk] 0-gv0-client-1: failed to set the volume (Permission denied) [2015-03-04 04:52:38.580983] W [client-handshake.c:1397:client_setvolume_cbk] 0-gv0-client-1: failed to get 'process-uuid' from reply dict [2015-03-04 04:52:38.580993] E [client-handshake.c:1403:client_setvolume_cbk] 0-gv0-client-1: SETVOLUME on remote-host failed: Authentication failed [2015-03-04 04:52:38.581001] I [client-handshake.c:1489:client_setvolume_cbk] 0-gv0-client-1: sending AUTH_FAILED event [2015-03-04 04:52:38.581014] E [fuse-bridge.c:5081:notify] 0-fuse: Server authenication failed. Shutting down. [2015-03-04 04:52:38.581039] I [fuse-bridge.c:5514:fini] 0-fuse: Unmounting '/dmfcontents'. [2015-03-04 04:52:38.595944] I [client-handshake.c:1677:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-03-04 04:52:38.596366] W [glusterfsd.c:1095:cleanup_and_exit] (--/lib64/libc.so.6(clone+0x6d) [0x7f81fab6586d] (--/lib64/libpthread.so.0(+0x79d1) [0x7f81fb1f89d1] (--glusterfs(glusterfs_sigwaiter+0xd5) [0x4053e5]))) 0-: received signum (15), shutting down [2015-03-04 04:52:38.596667] W [client-handshake.c:1371:client_setvolume_cbk] 0-gv0-client-0: failed to set the volume (Permission denied) I also find similar problem described in https://bugzilla.redhat.com/show_bug.cgi?id=915153 . I'm confused why only one node fails . I think if hostname resolving problem, two nodes should fail. Any tips about debugging further or getting this fixed up would be appreciated. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Thanks for the tip but Debian wheezy does not use systemd at all, it's still old sysV style init scripts. On Tuesday, March 3, 2015 2:43 PM, Jeremy Young jrm16...@gmail.com wrote: I know that you mention the script in init.d but want to ask to make sure that you're not using systemd. Are you still using SysVinit with wheezy? I've had this exact problem with RHEL 7, and creating the file /etc/systemd/system/glusterd.service.d/restart.conf, containing the information below, resolved my issue. [Service] Restart=on-failure StartLimitInterval=5s StartLimitBurst=5 On Tue, Mar 3, 2015 at 7:04 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: So that would sound like the boot order is messed up. Sounds a bit like this: http://joejulian.name/blog/glusterfs-volumes-not-mounting-in-debian-squeeze-at-boot-time/ As I said before, I also run glusterfs 3.5.2 on Debian Wheezy and I don't have issues after a reboot. But unfortunately I don't remember if I had to manually adapt something like the boot order of the init scripts). You can try and make your gluster scripts run at the very end of the boot process and see if that helps. On Tue, Mar 3, 2015 at 1:56 PM, ML mail mlnos...@yahoo.com wrote: Yes dig and ping works fine. I used first the short hostname gfs1 and then I also tried gfs1.intra.domain.com. That did not change anything. Currently for testing I only have a single node setup so my gluster peer status output would be empty. Now I had a closed look at the brick logfile and it looks like my network is not read at the time glusterfsd kicks in. I can see the following entry: [2015-03-03 13:45:48.647981] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com happening at exactly 13:45:48 but on my Cisco Switch I can see the following log entry: Mar 3 13:45:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to up So as you see my switch's port is only ready 3 seconds later at: 13:45:51, which is too late for glusterfsd. Strangely enough I use spanning-tree portfast on all my ports and as such the switch ports are ready as soon as possible. I have the feeling here that there is an issue with the /etc/init.d/glusterfs-server script on Debian wheezy from the glusterfs-server package. Can anyone confirm and maybe fix this for the future? On Tuesday, March 3, 2015 12:57 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
While probing new nodes have you used mixed flavour of fqdn like short names, long names? Can you please paste peer status output here? ~Atin On 03/03/2015 05:08 PM, ML mail wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured correctly. I am using Debian Wheezy with GlusterFS 3.5.2. Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -- ~Atin ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] /etc/hosts entry requires for gluster servers?
Probably a stupid workaround but what if you just add a sleep n line into the init script? Glusterfs is working fine if it was launched manually, did I understand that right? It's only the automatic startup at boot which causes the lookup failure? On Tue, Mar 3, 2015 at 2:54 PM, ML mail mlnos...@yahoo.com wrote: Thanks for the tip but Debian wheezy does not use systemd at all, it's still old sysV style init scripts. On Tuesday, March 3, 2015 2:43 PM, Jeremy Young jrm16...@gmail.com wrote: I know that you mention the script in init.d but want to ask to make sure that you're not using systemd. Are you still using SysVinit with wheezy? I've had this exact problem with RHEL 7, and creating the file /etc/systemd/system/glusterd.service.d/restart.conf, containing the information below, resolved my issue. [Service] Restart=on-failure StartLimitInterval=5s StartLimitBurst=5 On Tue, Mar 3, 2015 at 7:04 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: So that would sound like the boot order is messed up. Sounds a bit like this: http://joejulian.name/blog/glusterfs-volumes-not-mounting-in-debian-squeeze-at-boot-time/ As I said before, I also run glusterfs 3.5.2 on Debian Wheezy and I don't have issues after a reboot. But unfortunately I don't remember if I had to manually adapt something like the boot order of the init scripts). You can try and make your gluster scripts run at the very end of the boot process and see if that helps. On Tue, Mar 3, 2015 at 1:56 PM, ML mail mlnos...@yahoo.com wrote: Yes dig and ping works fine. I used first the short hostname gfs1 and then I also tried gfs1.intra.domain.com. That did not change anything. Currently for testing I only have a single node setup so my gluster peer status output would be empty. Now I had a closed look at the brick logfile and it looks like my network is not read at the time glusterfsd kicks in. I can see the following entry: [2015-03-03 13:45:48.647981] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com happening at exactly 13:45:48 but on my Cisco Switch I can see the following log entry: Mar 3 13:45:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to up So as you see my switch's port is only ready 3 seconds later at: 13:45:51, which is too late for glusterfsd. Strangely enough I use spanning-tree portfast on all my ports and as such the switch ports are ready as soon as possible. I have the feeling here that there is an issue with the /etc/init.d/glusterfs-server script on Debian wheezy from the glusterfs-server package. Can anyone confirm and maybe fix this for the future? On Tuesday, March 3, 2015 12:57 PM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Can you resolve the other gluster peers with dig? Are you able to ping the other peers, too? On Tue, Mar 3, 2015 at 12:38 PM, ML mail mlnos...@yahoo.com wrote: Well the weird thing is that my DNS resolver servers are configured correctly and working fine. Here below is the exact error message from the brick log file: [2015-03-03 11:34:21.731639] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-03-03 11:34:21.731654] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1.intra.domain.com [2015-03-03 11:34:21.731707] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1.intra.domain.com (Success) [2015-03-03 11:34:21.731724] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers I checked in my /etc/hosts file and I have the following entry: 127.0.1.1 gfs1.intra.domain.com gfs1a This is Debian's default and did not touch the hosts file. I also tried to remove this 127.0.1.1 but nothing changed. On Tuesday, March 3, 2015 10:33 AM, Claudio Kuenzler c...@claudiokuenzler.com wrote: Hi ML, Here's what I have noted down in my personal documentation when I installed GlusterFS the first time in 2013 (also on Debian Wheezy with 3.5.2): All cluster nodes MUST resolve each other through DNS (preferred) or /etc/hosts. An entry in /etc/hosts is probably even more safe because you don't depend on external DNS resolvers. cheers,ck On Tue, Mar 3, 2015 at 8:43 AM, ML mail mlnos...@yahoo.com wrote: Hello, Is it required to have the GlusterFS servers in /etc/hosts for the gluster servers themselves? I read many tutorials where people always add an entry in their /etc/hosts file. I am asking because my issue is that my volumes, or more precisely glusterfsd, are not starting at system boot. The brick log shows that the hostname of the server could not be resolved but I have an entry in my DNS server and my /etc/resolv.conf is configured
