Re: Stale NFS file handle when using aufs on top of btrfs?

2015-01-28 Thread sfjro
Dan Kegel:
> root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar#
> mkdir -p dir1/dir2
> root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar#
> rm -rf dir1
> root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# 
> ls
> ls: cannot open directory .: Stale file handle

Hmm, I don't understand why I cannot reproduce...


> I may be able to try building the Ubuntu kernel, but I'm a bit slammed
> now, not sure when I can try it.
> But if your patch works it'd save us some effort.

Ok, when you can, please try the attached patch.


> Here's the other info you requested:

Ok, thanx.
I understand your branches and options. You don't have compress=zlib. Oh
but you have noplink.

Anyway I will try ubuntu kernel by myself, although it will take long
time...


J. R. Okajima



a.patch.bz2
Description: BZip2 compressed data
--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/

Re: Stale NFS file handle when using aufs on top of btrfs?

2015-01-28 Thread Dan Kegel
The short repro script reproduces the problem here:

root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# df -T .
Filesystem Type 1K-blocks Used Available Use% Mounted on
none   aufs 209711104 20796608 186780800  11%
/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique
root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique#
mkdir -p dir1/dir2
root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique#
mkdir foobar
root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# cd foobar
root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar#
mkdir -p dir1/dir2
root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar#
rm -rf dir1
root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# ls
ls: cannot open directory .: Stale file handle

This breaks a lot of tools, e.g.

root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar#
find /sys/module/aufs -ls
find: cannot get current directory: No such file or directory

I may be able to try building the Ubuntu kernel, but I'm a bit slammed
now, not sure when I can try it.
But if your patch works it'd save us some effort.

Here's the other info you requested:


# uname -a
Linux ubu14-bb-01 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

http://packages.ubuntu.com/trusty/kernel/linux-image-3.13.0-35-generic

# dmesg | grep -i aufs
[   10.099732] aufs 3.13-20140303
[   11.104168] aufs test_add:293:lxc-start[1516]: uid/gid/perm
0/0/0755, 1001/1001/0755
[   11.131591] aufs test_add:293:lxc-start[1517]: uid/gid/perm
0/0/0755, 1001/1001/0755
[  483.881535] aufs au_new_inode:432:bash[3127]: Warning: Un-notified
UDBA or repeatedly renamed dir, b0, btrfs, foobar, hi16400847, i1227.

(hey, looky, I got the same warning re UDBA!)

# ls -l /proc/config.gz
ls: cannot access /proc/config.gz: No such file or directory


root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar#
cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4076992k,nr_inodes=1019248,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=817672k,mode=755 0 0
/dev/sda1 / btrfs rw,relatime,space_cache 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,noexec,noatime,size=524288k 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/dm-0 /home btrfs rw,relatime,space_cache 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
systemd /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd
0 0
none /var/lib/lxc/ubu14-bb-01-ubu1404-temp-g-speak-unique aufs
rw,relatime,si=30d00c8dd5327da5,noplink 0 0
none /var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique aufs
rw,relatime,si=30d00c8f5e792da5,noplink 0 0
none /var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/ephemeralbind
tmpfs rw,relatime 0 0
none /var/lib/lxc/ubu14-bb-01-ubu1404-temp-g-speak-unique/ephemeralbind
tmpfs rw,relatime 0 0

# more /sys/module/aufs/*

/sys/module/aufs/coresize
202783

/sys/module/aufs/initsize
0

/sys/module/aufs/initstate
live

/sys/module/aufs/refcnt
431

/sys/module/aufs/srcversion
F8A3BAB17A4595B4AB06091

/sys/module/aufs/taint

/sys/module/aufs/uevent: Permission denied

/sys/module/aufs/version
3.13-20140303

/sys/module/aufs/parameters/brs
1


# ls /sys/fs/aufs/*
/sys/fs/aufs/config

/sys/fs/aufs/si_30d00c8dd51b3da5:
br0  br1  brid0  brid1  xi_path

/sys/fs/aufs/si_30d00c8dd5327da5:
br0  br1  brid0  brid1  xi_path

/sys/fs/aufs/si_30d00c8f5e00cda5:
br0  br1  brid0  brid1  xi_path

/sys/fs/aufs/si_30d00c8f5e792da5:
br0  br1  brid0  brid1  xi_path

root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique#
more /sys/fs/aufs/si_30d00c8f5e792da5/*
::
/sys/fs/aufs/si_30d00c8f5e792da5/br0
::
/home/data/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/overlay=rw
::
/sys/fs/aufs/si_30d00c8f5e792da5/br1
::
/var/lib/lxc/ubu14-bb-01-ubu1204=ro
::
/sys/fs/aufs/si_30d00c8f5e792da5/brid0
::
64
::
/sys/fs/aufs/si_30d00c8f5e792da5/brid1
::
65
::
/sys/fs/aufs/si_30d00c8f5e792da5/xi_path
::
/tmp/.aufs.xino

--
Dive into the World of Parallel Programming. The

Re: Stale NFS file handle when using aufs on top of btrfs?

2015-01-28 Thread sfjro
Michael Johnson - MJ:
> All branches of you aufs mount are btrfs and I do see the 'Warning:
> Un-notified UDBA or repeatedly renamed dir' messages in my logs if I have
> things using inotify.  I use udba=3Dreval.  In fact, here is the full set o=

That is strange.
The message is unrelated to udba option.
Anyway I could find the root cause of the message. Btrfs has a strange
link count of a dir. Even if the dir exists and has a child dir, its
link count is zero. But not always. I don't know the condition when
btrfs sets zero and when maintains correctly.

I created a patch to exclude testing i_nlink for btrfs and succeeded
stop the message. But I am not sure whether the patch can solve the
ESTALE problem you two see, because aufs checks i_nlink in several
places and I don't know where is the trigger of the problem.


J. R. Okajima



a.patch.bz2
Description: BZip2 compressed data
--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/

Re: Stale NFS file handle when using aufs on top of btrfs?

2015-01-28 Thread sfjro

Michael Johnson - MJ:
> I think this provides all the information from my system that you asked
> for.  I am running stock ubuntu 14.10.  I have been seeing this problem as
> long as I can recall (all the way back to Ubuntu 10.04?).  It's never
> really been a major issue for me, so I've not brought it up.

Ok, thanx.


> /dev/sde /mnt/vol0 btrfs rw,noatime,compress=3Dzlib,space_cache 0 0
> /dev/sdc /mnt/vol1 btrfs rw,noatime,compress=3Dzlib,space_cache 0 0
:::

Dan Kegel,
Are you using "compress=zlib" optin too?
If so, can you test aufs without this option? Because it is major diff
from my test env. I've add the option into my test script. But it was
not enabled until I tried "-o remount,compress=zlib" again. That is
strange. After enabling compress=zlib, the behaviour doesn't change
(aufs3.14). I don't get ESTALE.


J. R. Okajima

--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/