Public bug reported:
Consider that we have a fstab entry that uses both volfile-server and
backupvolfile-server in a Ubuntu 14.04.2 LTS Server:
mygluster:/mydir /var/mydir glusterfs
defaults,nobootwait,nofail,_netdev,backupvolfile-server=mygluster-bak 0
0
If the host mygluster is accessible at boot time, the mount works with
success. However, if mygluster is offline (because a DNS error, for
example) and mygluster-bak is online, the mount fails at boot time.
The bug only occurs at boot time. After the boot, if we run 'mount
/var/mydir', the mount will work using the mygluster-bak server as
expected.
## How to reproduce
Put the following entry in your fstab:
non-existent:/mydir /var/mydir glusterfs
defaults,nobootwait,nofail,_netdev,backupvolfile-server=mygluster-bak 0
0
Mount the system and check that the mount worked with success:
$ mount /var/mydir; mount | grep mydir; umount
non-existent:/mydir on /var/mydir type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)
Now, reboot your system some times and check that sometimes the mount
has failed. At this moment, run 'mount /var/mydir' and successfully
mount the filesystem.
$ mount | grep mydir
$ mount /var/mydir; mount | grep mydir
non-existent:/mydir on /var/mydir type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)
### Logs
The boot.log, dmesg, mountall and the gluster logfile (var-lib-glance-
images.log, at my specific case) will be attached (each one in a
separate comment because of
https://bugs.launchpad.net/launchpad/+bug/82652).
However, the only log that really helps is the gluster logfile, with
entries like:
[glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running
/usr/sbin/glusterfs version 3.4.2 (/usr/sbin/glusterfs --volfile-id=/mydir
--volfile-server=non-existent /var/mydir)
[name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution
failed on host non-existent
[fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/var/mydir'.
[glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running
/usr/sbin/glusterfs version 3.4.2 (/usr/sbin/glusterfs --volfile-id=/mydir
--volfile-server=mygluster-bak /var/mydir)
[fuse-bridge.c:5016:init] 0-fuse: Mountpoint /var/mydir seems to have a
stale mount, run 'umount /var/mydir' and try again.
[xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed,
review your volfile again
## Log analyses and debugging
The "Mountpoint /var/mydir seems to have a stale mount, run 'umount
/var/mydir' and try again" log helps a lot.
I've changed the /sbin/mount.glusterfs script to increase verbosity and
discovered some more useful info:
- At first, mount.glusterfs runs: /usr/sbin/glusterfs --volfile-id=/mydir
--volfile-server=non-existent /var/mydir
- After, it runs 'stat -c %i /var/mydir' to test if the inode is one (mount
successful at this mount point) or another number. In a normal mount try
(running 'mount /var/mydir' after the boot), this step returns a large number
like 4198417. However, during the boot, it returns no output and prints the
following error to stderr: **stat: cannot stat ‘/var/mydir’: Transport endpoint
is not connected**;
- In a second moment, mount.glusterfs runs: /usr/sbin/glusterfs
--volfile-id=/mydir --volfile-server=mygluster-bak /var/mydir;
- Again, it runs 'stat -c %i /var/mydir' and got the error **Transport endpoint
is not connected**;
- At the end, mount.glusterfs prints "Mount failed. Please check the log file
for more details.", runs "umount /var/mydir" and exits with status 1.
So, I have done some tests to got more info about the "Transport
endpoint is not connected" error and I discovered that it occurs for a
very shot time after a mount error. It's possible to got this error at
any time. The following command will sometimes reproduce this error
(it's sporadic):
$ /usr/sbin/glusterfs --volfile-id=/mydir --volfile-server=non-existent
/var/mydir; stat -c %i /var/mydir
4198417
$ /usr/sbin/glusterfs --volfile-id=/mydir --volfile-server=non-existent
/var/mydir; stat -c %i /var/mydir
stat: cannot stat ‘/var/mydir’: Transport endpoint is not connected
To get more debug info yet, I've modified the script mount.glusterfs
again to run 'fuser -m /var/mydir' after the first 'stat' / "Transport
endpoint is not connected" to get any PIDs using the filesystem and
got:
$ fuser -m /var/mydir:
1 371 1280 1758 2287 2503
$ ps -ww -up 1 371 1280 1758 2287 2503:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 16.8 0.0 34156 3532 ? Ss 16:36 0:04 /sbin/init
root 371 0.1 0.0 20132 976 ? S 16:36 0:00
@sbin/plymouthd --mode=boot --attach-to-session
--pid-file=/run/initramfs/plymouth.pid
statd 1280 0.0 0.0 21540 1396 ? Ss 16:36 0:00 rpc.statd
-L
syslog 1758 0.0 0.0 255840 1216 ? Ssl 16:36 0:00 rsyslogd
Unfortunately, I do not get the output of PIDs 2287 and 2503.
I don't know if the second mount error is related to these PIDs returned
from 'fuser' or if it's normal, and if it's related to the "Transport
endpoint is not connected" error, but it can be a start point.
## Possible solutions and workaround
I studied about the "seems to have a stale mount, run 'umount ...' and
try again" message and discovered this commit at the upstream code:
https://github.com/gluster/glusterfs/commit/08041c.
The commit message contains the message "Also, mount.glusterfs script unmounts
mount-point on mount failure to
prevent hung mounts". This message is about the umount line in mount.glusterfs:
https://github.com/gluster/glusterfs/commit/08041c#diff-7829823331339149cb845ff035efff54R165.
I do not know if running umount (as implemented after the last mount
error and suggested in the gluster log) solves any "Transport endpoint
not connected error" or only other specific mount hang, but a possible
solution is adding this line before the second mount (after the first
failure):
--- mount.glusterfs.orig 2015-06-12 01:02:18.943119823 -0300
+++ mount.glusterfs 2015-06-12 01:24:52.824311071 -0300
@@ -226,6 +226,7 @@
if [ $inode -ne 1 ]; then
err=1;
if [ -n "$cmd_line1" ]; then
+ umount $mount_point > /dev/null 2>&1;
cmd_line1=$(echo "$cmd_line1 $mount_point");
$cmd_line1;
err=0;
After this "patch", the mount point using the backupvolfile-server (that
failed most at the time) worked at most times, however it still failing
sometimes. The solution the always solved the mount was:
--- mount.glusterfs.orig 2015-06-12 01:02:18.943119823 -0300
+++ mount.glusterfs 2015-06-12 01:28:07.610199716 -0300
@@ -226,6 +226,7 @@
if [ $inode -ne 1 ]; then
err=1;
if [ -n "$cmd_line1" ]; then
+ sleep 0.1;
cmd_line1=$(echo "$cmd_line1 $mount_point");
$cmd_line1;
err=0;
I tested the last patch using many reboots (more then 60) and in all of
them the mount worked.
### Why do not backport some fix from the upstream
Since commit https://github.com/gluster/glusterfs/commit/b610f1,
upstream doesn't use two mounts in mount.glusterfs anymore (it uses
another solution). So, although this commit solves the problem, it makes
no sense to use the upstream solution because it changes the gluster
client behavior and should not be used in a bugfix update.
** Affects: glusterfs (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1464494
Title:
Gluster mount using volfile-bak fails on boot
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/glusterfs/+bug/1464494/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
