Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > Sorry it's been a bit since I reported back, but I got a chance to apply > your patch and I have mixed results to report. Basically, it looks like > the patch may resolve the issues, but not entirely. ::: > Ultimately, at this time I cannot reproduce the problem, so I guess we call > it fixed? Thanks for testing. On my side, I tried removing a bigger dir and could succeed to see the similar call trace. The bigger dir means - mkdir -p dir1/dir2/dir3 - cp -pr /another/dir dir1/ - /another/dir has lots of children, regular files, dirs and sub dirs. - and then rm -fr dir1 caused a bug and produced the same call trace. I am not sure this is the same problem to yours, but this new patch will fix, at least, the problem on my side. Would you try it again? - revert the previous patch the aufs source files should become same to ubuntu original. - apply this new patch - and test again to reproduce the problem. Thank you J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Sorry it's been a bit since I reported back, but I got a chance to apply your patch and I have mixed results to report. Basically, it looks like the patch may resolve the issues, but not entirely. After building the new kernel, I was still able to reproduce the problem initially in a sub directory, but not on the top level. When it was happening, this is what showed up in the logs: Feb  8 15:09:57 aufs kernel: [  998.810169] [ cut here ] Feb  8 15:09:57 aufs kernel: [  998.810177] WARNING: CPU: 0 PID: 2076 at /build/buildd/linux-3.16.0/fs/inode.c:282 drop_nlink+0x41/0x50() Feb  8 15:09:57 aufs kernel: [  998.810178] Modules linked in: aufs serio_raw snd_hda_codec_generic kvm_amd kvm qxl crct10dif_pclmul ttm drm_kms_helper crc32_pclmul ghash_clmulni_intel aesni_intel drm aes_x86_64 lrw ppdev snd_hda_intel snd_hda_controller i2c_piix4 snd_hda_codec gf128mul glue_helper snd_hwdep snd_pcm snd_timer ablk_helper cryptd snd parport_pc soundcore pvpanic parport mac_hid btrfs xor raid6_pq psmouse floppy pata_acpi Feb  8 15:09:57 aufs kernel: [  998.810200] CPU: 0 PID: 2076 Comm: rm Tainted: G     W   3.16.0-29-generic #39-Ubuntu Feb  8 15:09:57 aufs kernel: [  998.810202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014 Feb  8 15:09:57 aufs kernel: [  998.810204]  0009 88003c7f3cc8 8178218a Feb  8 15:09:57 aufs kernel: [  998.810206]  88003c7f3d00 8106fedd 88003bc1a658 88003bc1a658 Feb  8 15:09:57 aufs kernel: [  998.810208]  0001 88003cbcede0 880039d91c00 88003c7f3d10 Feb  8 15:09:57 aufs kernel: [  998.810210] Call Trace: Feb  8 15:09:57 aufs kernel: [  998.810216]  [] dump_stack+0x45/0x56 Feb  8 15:09:57 aufs kernel: [  998.810219]  [] warn_slowpath_common+0x7d/0xa0 Feb  8 15:09:57 aufs kernel: [  998.810221]  [] warn_slowpath_null+0x1a/0x20 Feb  8 15:09:57 aufs kernel: [  998.810224]  [] drop_nlink+0x41/0x50 Feb  8 15:09:57 aufs kernel: [  998.810231]  [] au_whtmp_rmdir+0x19d/0x1b0 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810236]  [] ? au_whtmp_ren+0x6e/0xe0 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810241]  [] aufs_rmdir+0x2b7/0x430 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810244]  [] ? prepend.constprop.25+0x30/0x30 Feb  8 15:09:57 aufs kernel: [  998.810246]  [] vfs_rmdir+0xa7/0x100 Feb  8 15:09:57 aufs kernel: [  998.810248]  [] do_rmdir+0x1d9/0x1f0 Feb  8 15:09:57 aufs kernel: [  998.810250]  [] ? fput+0xe/0x10 Feb  8 15:09:57 aufs kernel: [  998.810253]  [] ? task_work_run+0xbc/0xf0 Feb  8 15:09:57 aufs kernel: [  998.810256]  [] ? do_notify_resume+0x97/0xb0 Feb  8 15:09:57 aufs kernel: [  998.810258]  [] SyS_unlinkat+0x25/0x40 Feb  8 15:09:57 aufs kernel: [  998.810261]  [] system_call_fastpath+0x1a/0x1f Feb  8 15:09:57 aufs kernel: [  998.810263] ---[ end trace 7e60ac9f449b2a85 ]--- Once I saw that it was still happening, I attempted to clean things up and present a more complete example of how I was getting this to happen. I unmounted the aufs filesystem and removed the contents of the underlying filesystems (including the .wh* bits) and remounted. Once I saw what the aufs mountpoint was correctly reflecting that everything was removed, I tried to reproduce the error again, and I have not been able to. I can only assume that something with the old .wh* files was causing some sort of issue. Ultimately, at this time I cannot reproduce the problem, so I guess we call it fixed? On Wed, Feb 4, 2015 at 5:41 AM, <[1]sf...@users.sourceforge.net> wrote: Michael Johnson - MJ: > I've made it through the initial build and the problem does in fact still > occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would > expect, the problem still occurs.  Interestingly, I cannot reproduce 100% > of the time in the way previously described. Here is what does work to > reproduce for me 100% of the time currently: > > mkdir -p a/b; cd .; rm -rf a; ls; Still I cannot reproduce... But I will try more. > For the most part I did not get anything in my kernel debug logs, but after > hammering on it a bit I did get the following in my logs which appears to > be related:     ::: > Feb 3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at > /build/buildd/linux-3.16.0/fs > /inode.c:282 drop_nlink+0x41/0x50() It is one of good news. It is caused by a strange link count of a dir on btrfs. At least this problem will be solved by the patch I sent prev
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > I've made it through the initial build and the problem does in fact still > occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would > expect, the problem still occurs. Interestingly, I cannot reproduce 100% > of the time in the way previously described. Here is what does work to > reproduce for me 100% of the time currently: > > mkdir -p a/b; cd .; rm -rf a; ls; Still I cannot reproduce... But I will try more. > For the most part I did not get anything in my kernel debug logs, but after > hammering on it a bit I did get the following in my logs which appears to > be related: ::: > Feb 3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at > /build/buildd/linux-3.16.0/fs > /inode.c:282 drop_nlink+0x41/0x50() It is one of good news. It is caused by a strange link count of a dir on btrfs. At least this problem will be solved by the patch I sent previously. But we may meet another problem later. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs.  Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; The end result is always a stale file handle. If you 'cd .' following this set of commands the error clears, unless you are in the top level of the aufs filesystem. If you are in the top level, you must umount/mount the aufs file system to recover. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related: Feb  3 23:54:38 aufs kernel: [ 3577.351197] [ cut here ] Feb  3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at /build/buildd/linux-3.16.0/fs /inode.c:282 drop_nlink+0x41/0x50() Feb  3 23:54:38 aufs kernel: [ 3577.351208] Modules linked in: aufs snd_hda_codec_generic kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller snd_hda_codec qxl snd_hwdep snd_pcm ttm drm_kms_helper snd_timer serio_raw drm snd soundcore parport_pc pvpanic parport i2c_piix4 mac_hid btrfs xor raid6_pq psmouse floppy pata_acpi Feb  3 23:54:38 aufs kernel: [ 3577.351233] CPU: 0 PID: 3725 Comm: rm Tainted: G     W   3.16.0-23-generic #31-Ubuntu Feb  3 23:54:38 aufs kernel: [ 3577.351234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014 Feb  3 23:54:38 aufs kernel: [ 3577.351236]  0009 88003a07fcc8 8177fcbc Feb  3 23:54:38 aufs kernel: [ 3577.351238]  88003a07fd00 8106fd8d 88003a742058 88003a742058 Feb  3 23:54:38 aufs kernel: [ 3577.351240]  88003b3ac240 88003b92b100 88003a07fd10 Feb  3 23:54:38 aufs kernel: [ 3577.351251] Call Trace: Feb  3 23:54:38 aufs kernel: [ 3577.351257]  [] dump_stack+0x45/0x56 Feb  3 23:54:38 aufs kernel: [ 3577.351261]  [] warn_slowpath_common+0x7d/0xa0 Feb  3 23:54:38 aufs kernel: [ 3577.351263]  [] warn_slowpath_null+0x1a/0x20 Feb  3 23:54:38 aufs kernel: [ 3577.351265]  [] drop_nlink+0x41/0x50 Feb  3 23:54:38 aufs kernel: [ 3577.351273]  [] au_whtmp_rmdir+0x19d/0x1b0 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351278]  [] ? au_whtmp_ren+0x6e/0xe0 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351283]  [] aufs_rmdir+0x2b7/0x430 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351285]  [] ? prepend.constprop.25+0x30/0x30 Feb  3 23:54:38 aufs kernel: [ 3577.351288]  [] vfs_rmdir+0xa7/0x100 Feb  3 23:54:38 aufs kernel: [ 3577.351290]  [] do_rmdir+0x1d9/0x1f0 Feb  3 23:54:38 aufs kernel: [ 3577.351293]  [] ? fput+0xe/0x10 Feb  3 23:54:38 aufs kernel: [ 3577.351295]  [] ? task_work_run+0xbc/0xf0 Feb  3 23:54:38 aufs kernel: [ 3577.351299]  [] ? do_notify_resume+0x97/0xb0 Feb  3 23:54:38 aufs kernel: [ 3577.351301]  [] SyS_unlinkat+0x25/0x40 Feb  3 23:54:38 aufs kernel: [ 3577.351304]  [] system_call_fastpath+0x1a/0x1f Feb  3 23:54:38 aufs kernel: [ 3577.351391] ---[ end trace d717630435aab60d ]--- I could not get this to occur on a regular basis, but it popped up at some point. Hopefully that is of some help. Here are all the aufs config options for my current build: CONFIG_AUFS_FS=m CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y # CONFIG_AUFS_HNOTIFY is not set CONFIG_AUFS_EXPORT=y CONFIG_AUFS_INO_T_64=y # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SP_IATTR is not set # CONFIG_AUFS_SHWH is not set # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set # CONFIG_AUFS_BR_HFSPLUS is not set CONFIG_AUFS_BDEV_LOOP=y CONFIG_AUFS_DEBUG=y CONFIG_AUFS_MAGIC_SYSRQ=y I have not yet applied the patch, but I will give it a try tomorrow to see if it makes a difference in behavior or log messages. On Tue, Feb 3, 2015 at 8:06 PM, <[1]sf...@users.sourceforge.net> wrote: MJ, Dan, I appriciate your efforts. Dan Kegel: > That failed with "no config found", so I repeated it after doing > $ cp -vi /boot/config-`uname -r` .config > It then asked two questions; I said yes to X86_16BIT and no to > IPMI_SI_PROBE_DEFAULTS.     :::  (and about the uppercase in aufsD) Sorry about that. I
Re: Stale NFS file handle when using aufs on top of btrfs?
MJ, Dan, I appriciate your efforts. Dan Kegel: > That failed with "no config found", so I repeated it after doing > $ cp -vi /boot/config-`uname -r` .config > It then asked two questions; I said yes to X86_16BIT and no to > IPMI_SI_PROBE_DEFAULTS. ::: (and about the uppercase in aufsD) Sorry about that. I myself didn't use make-kpg to build ubuntu trusty kernel. I did all manually. - cd ubuntu-trusty.git - cp "MJ's config-3.13.0-35-generic" .config - make menuconfig edit CONFIG_LOCALVERSION="aufsD" - make - make headers_install - make modules_install - cp -p arch/x86/boot/bzImage /boot/vmlinuz-brabra - cp -p System.map /boot/System.map-brabra - edit /boot/grub/grub.cfg and add the entry Essentially these steps and make-kpg should be equivalent, and the purpose is to build/re-build and install the kernel. If you or any ubuntu user have a better way to do it, don't hesitate to stop using make-kpg. Thanks again J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I just tested an I get the error even when the aufs branches are reside on the same btrfs file system, so that doesn't seem to make any difference. Now to get aufs rebuilt. On Tue, Feb 3, 2015 at 8:05 AM, Dan Kegel <[1]d...@kegel.com> wrote: On Mon, Feb 2, 2015 at 8:56 PM, <[2]sf...@users.sourceforge.net> wrote: > [steps to build kernel] Thanks for posting the steps. I'm following them now. > - get all source files of ubuntu trusty and build as it is >  $ git clone git://[3]kernel.ubuntu.com/ubuntu/ubuntu-trusty.git >  $ cd ubuntu-trusty >  $ fakeroot make-kpkg --uc --us --config silentoldconfig configure That failed with "no config found", so I repeated it after doing $ cp -vi /boot/config-`uname -r` .config It then asked two questions; I said yes to X86_16BIT and no to IPMI_SI_PROBE_DEFAULTS. >  $ fakeroot make-kpkg --uc --us --append_to_version -aufsD --revision `date +%y%m%d%H%M` kernel_image >  (lots of time and space are necessary) That failed quickly, complaining that -aufsD had an uppercase char, so I fixed by lowercasing the aufsD: $ fakeroot make-kpkg --uc --us --append_to_version -aufsd --revision `date +%y%m%d%H%M` kernel_image I'll follow the rest of the steps after the build finishes. > - install the built kernel pkg >  $ dpkg -i ../linux-image-deb > - reboot > - make sure you can receive "kern.debug" logs >  $ logger -p kern.debug testing >  check the kernel log > - reproduce the problem, in order to make sure >  check the kernel log > > - enable debugging >  $ cd ubuntu-trusty >  $ make menuconfig >   enable CONFIG_AUFS_DEBUG >   (menu) "Ubuntu Supplied Third-Party Device Drivers" >      --> "Aufs (Advanced multi layered unification filesystem) support" >        --> "Debug aufs" > - re-run "make-kpkg ... kernel_image" and re-install the pkg > - reboot > - reproduce the problem >  check the kernel log > > - apply the patch in >  [4]http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.h tml > - re-run "make-kpkg ... kernel_image" and re-install the pkg > - reboot > - reproduce the problem >  check the kernel log > > >> If it would be helpful, I can figure out have to give you ssh access to >> this VM if you want to provide me with an SSH public to add. > > Can I reboot the system and check the log anytime via ssh? > If so, I will send my public key privately. > > > J. R. Okajima -- Michael Johnson - MJ References 1. mailto:d...@kegel.com 2. mailto:sf...@users.sourceforge.net 3. http://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git 4. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
On Mon, Feb 2, 2015 at 8:56 PM, wrote: > [steps to build kernel] Thanks for posting the steps. I'm following them now. > - get all source files of ubuntu trusty and build as it is > $ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git > $ cd ubuntu-trusty > $ fakeroot make-kpkg --uc --us --config silentoldconfig configure That failed with "no config found", so I repeated it after doing $ cp -vi /boot/config-`uname -r` .config It then asked two questions; I said yes to X86_16BIT and no to IPMI_SI_PROBE_DEFAULTS. > $ fakeroot make-kpkg --uc --us --append_to_version -aufsD --revision `date > +%y%m%d%H%M` kernel_image > (lots of time and space are necessary) That failed quickly, complaining that -aufsD had an uppercase char, so I fixed by lowercasing the aufsD: $ fakeroot make-kpkg --uc --us --append_to_version -aufsd --revision `date +%y%m%d%H%M` kernel_image I'll follow the rest of the steps after the build finishes. > - install the built kernel pkg > $ dpkg -i ../linux-image-deb > - reboot > - make sure you can receive "kern.debug" logs > $ logger -p kern.debug testing > check the kernel log > - reproduce the problem, in order to make sure > check the kernel log > > - enable debugging > $ cd ubuntu-trusty > $ make menuconfig > enable CONFIG_AUFS_DEBUG > (menu) "Ubuntu Supplied Third-Party Device Drivers" >--> "Aufs (Advanced multi layered unification filesystem) support" >--> "Debug aufs" > - re-run "make-kpkg ... kernel_image" and re-install the pkg > - reboot > - reproduce the problem > check the kernel log > > - apply the patch in > http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html > - re-run "make-kpkg ... kernel_image" and re-install the pkg > - reboot > - reproduce the problem > check the kernel log > > >> If it would be helpful, I can figure out have to give you ssh access to >> this VM if you want to provide me with an SSH public to add. > > Can I reboot the system and check the log anytime via ssh? > If so, I will send my public key privately. > > > J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I'll try to get to this today/tonight. I see another difference between my test case and yours which might be altering the result. In my case, both branches are on physically separate file systems rather than directories on the same fs. I'll check and see if this makes a difference in the result for me. On Feb 2, 2015 8:56 PM, <[1]sf...@users.sourceforge.net> wrote: Michael Johnson - MJ: > I think I know why you were unable to recreate the problem. It appears to > only occur in attempt the test case from a directory contained within the > aufs mount point, and not the base of the mount itself. In fact, if the > directory you are in exists in all branches, the problem does not occur. I > have verified that this problem does not occur when the underlying > filesystes are all ext4, but it does occur when they are all btrfs. Hmm... Still I cannot reproduce the problem. Linux jrofr 3.13.11.6aufsD #8 SMP Mon Feb 2 13:56:32 JST 2015 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro=rw none /u [  51.057477] aufs 3.13-20140303 + sudo mkdir -p /u/dir1/dir2/dir3 + cd /u/dir1 + sudo rm -fr dir2 + ls (no error) > I can try building a new aufs module, but I will need some directions to > get that started. - get all source files of ubuntu trusty and build as it is  $ git clone git://[2]kernel.ubuntu.com/ubuntu/ubuntu-trusty.git  $ cd ubuntu-trusty  $ fakeroot make-kpkg --uc --us --config silentoldconfig configure  $ fakeroot make-kpkg --uc --us --append_to_version -aufsD --revision `date +%y%m%d%H%M` kernel_image  (lots of time and space are necessary) - install the built kernel pkg  $ dpkg -i ../linux-image-deb - reboot - make sure you can receive "kern.debug" logs  $ logger -p kern.debug testing  check the kernel log - reproduce the problem, in order to make sure  check the kernel log - enable debugging  $ cd ubuntu-trusty  $ make menuconfig   enable CONFIG_AUFS_DEBUG   (menu) "Ubuntu Supplied Third-Party Device Drivers"       --> "Aufs (Advanced multi layered unification filesystem) support"         --> "Debug aufs" - re-run "make-kpkg ... kernel_image" and re-install the pkg - reboot - reproduce the problem  check the kernel log - apply the patch in  [3]http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.h tml - re-run "make-kpkg ... kernel_image" and re-install the pkg - reboot - reproduce the problem  check the kernel log > If it would be helpful, I can figure out have to give you ssh access to > this VM if you want to provide me with an SSH public to add. Can I reboot the system and check the log anytime via ssh? If so, I will send my public key privately. J. R. Okajima References 1. mailto:sf...@users.sourceforge.net 2. http://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git 3. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > I think I know why you were unable to recreate the problem. It appears to > only occur in attempt the test case from a directory contained within the > aufs mount point, and not the base of the mount itself. In fact, if the > directory you are in exists in all branches, the problem does not occur. I > have verified that this problem does not occur when the underlying > filesystes are all ext4, but it does occur when they are all btrfs. Hmm... Still I cannot reproduce the problem. Linux jrofr 3.13.11.6aufsD #8 SMP Mon Feb 2 13:56:32 JST 2015 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro=rw none /u [ 51.057477] aufs 3.13-20140303 + sudo mkdir -p /u/dir1/dir2/dir3 + cd /u/dir1 + sudo rm -fr dir2 + ls (no error) > I can try building a new aufs module, but I will need some directions to > get that started. - get all source files of ubuntu trusty and build as it is $ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git $ cd ubuntu-trusty $ fakeroot make-kpkg --uc --us --config silentoldconfig configure $ fakeroot make-kpkg --uc --us --append_to_version -aufsD --revision `date +%y%m%d%H%M` kernel_image (lots of time and space are necessary) - install the built kernel pkg $ dpkg -i ../linux-image-deb - reboot - make sure you can receive "kern.debug" logs $ logger -p kern.debug testing check the kernel log - reproduce the problem, in order to make sure check the kernel log - enable debugging $ cd ubuntu-trusty $ make menuconfig enable CONFIG_AUFS_DEBUG (menu) "Ubuntu Supplied Third-Party Device Drivers" --> "Aufs (Advanced multi layered unification filesystem) support" --> "Debug aufs" - re-run "make-kpkg ... kernel_image" and re-install the pkg - reboot - reproduce the problem check the kernel log - apply the patch in http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html - re-run "make-kpkg ... kernel_image" and re-install the pkg - reboot - reproduce the problem check the kernel log > If it would be helpful, I can figure out have to give you ssh access to > this VM if you want to provide me with an SSH public to add. Can I reboot the system and check the log anytime via ssh? If so, I will send my public key privately. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I think I know why you were unable to recreate the problem. It appears to only occur in attempt the test case from a directory contained within the aufs mount point, and not the base of the mount itself. In fact, if the directory you are in exists in all branches, the problem does not occur. I have verified that this problem does not occur when the underlying filesystes are all ext4, but it does occur when they are all btrfs. So to reproduce the problem, I do this: mount -t aufs -o br:/mnt/vol0=rw:/mnt/vol1=rw /z mkdir -p /z/foo/bar/baz cd /z/foo rm -rf bar ls The ls results in 'ls: cannot open directory .: Stale file handle' This is using a fresh install of Ubuntu 14.10 in a VM. I can try building a new aufs module, but I will need some directions to get that started. If it would be helpful, I can figure out have to give you ssh access to this VM if you want to provide me with an SSH public to add. On Mon, Feb 2, 2015 at 7:57 AM, <[1]sf...@users.sourceforge.net> wrote: Michael Johnson - MJ: > I will see if I can reproduce in a vm and generate very specific > instructions to reproduce. I think I will have time to do this later today. Thank you very much. Other testers are also welcome. The first debug patch I will ask will be the one in [2]http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.h tml J. R. Okajima -- Michael Johnson - MJ References 1. mailto:sf...@users.sourceforge.net 2. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > I will see if I can reproduce in a vm and generate very specific > instructions to reproduce. I think I will have time to do this later today. Thank you very much. Other testers are also welcome. The first debug patch I will ask will be the one in http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I will see if I can reproduce in a vm and generate very specific instructions to reproduce. I think I will have time to do this later today. On Feb 1, 2015 10:37 PM, <[1]sf...@users.sourceforge.net> wrote: [2]sf...@users.sourceforge.net: > I have tried but could not reproduce the problem. > - got full trusty kernel source, built it with the default configuration >  (as MJ posted) --> no disk space > - disabled unnecessary drivers, built again --> succeeded > - rebooted with the new kernel, tried, but could not reproduce the >  problem > > Now I am preparing a parition to install full ubuntu trusty by > debootstrap. I am going to make btrfs as its root partition. Do I need > some options for btrfs? Is it ok to mkfs.btrfs without any options > simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs  # mkdir rw ro u  # sudo mount -t aufs -o br=/rw:/ro none /u  # cd /u  # sudo touch fileA  # sudo mkdir -p dir1/dir2  # sudo rm -fr dir1  # ls  and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module  which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by  "ls -l $(modinfo -F filename aufs)" and see the timestamp. - make sure that you can receive the "kern.debug" logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just  before rm, run "echo 1 > /sys/module/aufs/parameters/debug". and just  after ls, run "echo 0 > /sys/module/aufs/parameters/debug".  Setting 1 enables aufs debugging feature and produces many kernel  debug log. To stop the debug log, set 0 to  /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima References 1. mailto:sf...@users.sourceforge.net 2. mailto:sf...@users.sourceforge.net -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: > I have tried but could not reproduce the problem. > - got full trusty kernel source, built it with the default configuration > (as MJ posted) --> no disk space > - disabled unnecessary drivers, built again --> succeeded > - rebooted with the new kernel, tried, but could not reproduce the > problem > > Now I am preparing a parition to install full ubuntu trusty by > debootstrap. I am going to make btrfs as its root partition. Do I need > some options for btrfs? Is it ok to mkfs.btrfs without any options > simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs # mkdir rw ro u # sudo mount -t aufs -o br=/rw:/ro none /u # cd /u # sudo touch fileA # sudo mkdir -p dir1/dir2 # sudo rm -fr dir1 # ls and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by "ls -l $(modinfo -F filename aufs)" and see the timestamp. - make sure that you can receive the "kern.debug" logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just before rm, run "echo 1 > /sys/module/aufs/parameters/debug". and just after ls, run "echo 0 > /sys/module/aufs/parameters/debug". Setting 1 enables aufs debugging feature and produces many kernel debug log. To stop the debug log, set 0 to /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: > Anyway I will try ubuntu kernel by myself, although it will take long > time... I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration (as MJ posted) --> no disk space - disabled unnecessary drivers, built again --> succeeded - rebooted with the new kernel, tried, but could not reproduce the problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Dan Kegel: > root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# > mkdir -p dir1/dir2 > root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# > rm -rf dir1 > root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# > ls > ls: cannot open directory .: Stale file handle Hmm, I don't understand why I cannot reproduce... > I may be able to try building the Ubuntu kernel, but I'm a bit slammed > now, not sure when I can try it. > But if your patch works it'd save us some effort. Ok, when you can, please try the attached patch. > Here's the other info you requested: Ok, thanx. I understand your branches and options. You don't have compress=zlib. Oh but you have noplink. Anyway I will try ubuntu kernel by myself, although it will take long time... J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
The short repro script reproduces the problem here: root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# df -T . Filesystem Type 1K-blocks Used Available Use% Mounted on none aufs 209711104 20796608 186780800 11% /var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# mkdir -p dir1/dir2 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# mkdir foobar root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# cd foobar root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# mkdir -p dir1/dir2 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# rm -rf dir1 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# ls ls: cannot open directory .: Stale file handle This breaks a lot of tools, e.g. root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# find /sys/module/aufs -ls find: cannot get current directory: No such file or directory I may be able to try building the Ubuntu kernel, but I'm a bit slammed now, not sure when I can try it. But if your patch works it'd save us some effort. Here's the other info you requested: # uname -a Linux ubu14-bb-01 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux http://packages.ubuntu.com/trusty/kernel/linux-image-3.13.0-35-generic # dmesg | grep -i aufs [ 10.099732] aufs 3.13-20140303 [ 11.104168] aufs test_add:293:lxc-start[1516]: uid/gid/perm 0/0/0755, 1001/1001/0755 [ 11.131591] aufs test_add:293:lxc-start[1517]: uid/gid/perm 0/0/0755, 1001/1001/0755 [ 483.881535] aufs au_new_inode:432:bash[3127]: Warning: Un-notified UDBA or repeatedly renamed dir, b0, btrfs, foobar, hi16400847, i1227. (hey, looky, I got the same warning re UDBA!) # ls -l /proc/config.gz ls: cannot access /proc/config.gz: No such file or directory root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=4076992k,nr_inodes=1019248,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=817672k,mode=755 0 0 /dev/sda1 / btrfs rw,relatime,space_cache 0 0 none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 tmpfs /tmp tmpfs rw,nosuid,noexec,noatime,size=524288k 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 none /sys/fs/pstore pstore rw,relatime 0 0 /dev/dm-0 /home btrfs rw,relatime,space_cache 0 0 rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd 0 0 none /var/lib/lxc/ubu14-bb-01-ubu1404-temp-g-speak-unique aufs rw,relatime,si=30d00c8dd5327da5,noplink 0 0 none /var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique aufs rw,relatime,si=30d00c8f5e792da5,noplink 0 0 none /var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/ephemeralbind tmpfs rw,relatime 0 0 none /var/lib/lxc/ubu14-bb-01-ubu1404-temp-g-speak-unique/ephemeralbind tmpfs rw,relatime 0 0 # more /sys/module/aufs/* /sys/module/aufs/coresize 202783 /sys/module/aufs/initsize 0 /sys/module/aufs/initstate live /sys/module/aufs/refcnt 431 /sys/module/aufs/srcversion F8A3BAB17A4595B4AB06091 /sys/module/aufs/taint /sys/module/aufs/uevent: Permission denied /sys/module/aufs/version 3.13-20140303 /sys/module/aufs/parameters/brs 1 # ls /sys/fs/aufs/* /sys/fs/aufs/config /sys/fs/aufs/si_30d00c8dd51b3da5: br0 br1 brid0 brid1 xi_path /sys/fs/aufs/si_30d00c8dd5327da5: br0 br1 brid0 brid1 xi_path /sys/fs/aufs/si_30d00c8f5e00cda5: br0 br1 brid0 brid1 xi_path /sys/fs/aufs/si_30d00c8f5e792da5: br0 br1 brid0 brid1 xi_path root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique# more /sys/fs/aufs/si_30d00c8f5e792da5/* :: /sys/fs/aufs/si_30d00c8f5e792da5/br0 :: /home/data/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/overlay=rw :: /sys/fs/aufs/si_30d00c8f5e792da5/br1 :: /var/lib/lxc/ubu14-bb-01-ubu1204=ro :: /sys/fs/aufs/si_30d00c8f5e792da5/brid0 :: 64 :: /sys/fs/aufs/si_30d00c8f5e792da5/brid1 :: 65 :: /sys/fs/aufs/si_30d00c8f5e792da5/xi_path :: /tmp/.aufs.xino -- Dive into the World of Parallel Programming. The
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > All branches of you aufs mount are btrfs and I do see the 'Warning: > Un-notified UDBA or repeatedly renamed dir' messages in my logs if I have > things using inotify. I use udba=3Dreval. In fact, here is the full set o= That is strange. The message is unrelated to udba option. Anyway I could find the root cause of the message. Btrfs has a strange link count of a dir. Even if the dir exists and has a child dir, its link count is zero. But not always. I don't know the condition when btrfs sets zero and when maintains correctly. I created a patch to exclude testing i_nlink for btrfs and succeeded stop the message. But I am not sure whether the patch can solve the ESTALE problem you two see, because aufs checks i_nlink in several places and I don't know where is the trigger of the problem. J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > I think this provides all the information from my system that you asked > for. I am running stock ubuntu 14.10. I have been seeing this problem as > long as I can recall (all the way back to Ubuntu 10.04?). It's never > really been a major issue for me, so I've not brought it up. Ok, thanx. > /dev/sde /mnt/vol0 btrfs rw,noatime,compress=3Dzlib,space_cache 0 0 > /dev/sdc /mnt/vol1 btrfs rw,noatime,compress=3Dzlib,space_cache 0 0 ::: Dan Kegel, Are you using "compress=zlib" optin too? If so, can you test aufs without this option? Because it is major diff from my test env. I've add the option into my test script. But it was not enabled until I tried "-o remount,compress=zlib" again. That is strange. After enabling compress=zlib, the behaviour doesn't change (aufs3.14). I don't get ESTALE. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: > Michael Johnson - MJ: > > $ cd /aufsdir; > > $ mkdir -p dir1/dir2; > > $ rm -rf dir1 > > $ ls > > ls: cannot open directory .: Stale file handle > > At least, these steps succeeded on my test machine. I've tested the same steps on - plain linux-3.13 - aufs3.13-20140303 (as Dan's ubuntu kernel) - plus my local debug patch. The result is unchanged, succeeded. I am afraid some external things may be related, for example kernel config, aufs-util in userspace, options for btrfs, etc... It will be important as a first step to make the differences clear and reproduce the problem on my side. Otherwise, I have to ask you to repeat inserting a debug print, re-build and test. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
> According to git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git, it uses > aufs3.13-20140303. As always, ubuntu uses old version enough to make it > very hard for me suppport. But I am trying, as always, to support and > help users. Ah, I might be confused, sorry. If ubuntu-trusty.git is release on Apr 2014, then aufs3.13-20140303 is not bad. It should be new at that time. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Dan Kegel: > I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer Is this your Ubuntu 14.04? 2d22fc7 2014-10-09 UBUNTU: Ubuntu-3.13.0-38.65 According to git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git, it uses aufs3.13-20140303. As always, ubuntu uses old version enough to make it very hard for me suppport. But I am trying, as always, to support and help users. Anyway I use MJ's short steps to test this problem because it looks like same to yours. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > $ cd /aufsdir; > $ mkdir -p dir1/dir2; > $ rm -rf dir1 > $ ls > ls: cannot open directory .: Stale file handle At least, these steps succeeded on my test machine. $ mkdir -p dir1/dir2 $ rm -ir dir1 rm: descend into directory `dir1'? y rm: remove directory `dir1/dir2'? y aufs au_new_inode:423:rm[4413]: Warning: Un-notified UDBA or repeatedly renamed dir, b0, btrfs, dir1, hi274, i11. rm: remove directory `dir1'? y $ ls b_dst empty f_src mv501_a/ p_src| sleep* ::: (shows everything) - latest aufs3.14 - u = rw + ro /dev/ram1 /run/shm/ro ext2 ro,relatime,errors=continue,user_xattr,acl 0 0 /dev/ram0 /run/shm/rw btrfs rw,relatime,space_cache 0 0 none /run/shm/u aufs rw,relatime,si=a8fd9d47e41c7918 0 0 An interesting thing is, a warning about UDBA was produced. Which means that by removing dir2, the something about dir1 was changed unexpectedly and aufs detects it (and produced a warning). But I am not sure this is related to your problem. Anyway it will be a good help for me to investigate the problem if you post these info. - /proc/mounts (instead of the output of mount(8)) - /sys/module/aufs/* - /sys/fs/aufs/* (if you have them) - /debug/aufs/* (if you have them) - linux kernel version if your kernel is not plain, for example modified by distributor, the url where i can download its source is necessary too. - aufs version which was printed at loading the module or booting the system, instead of the date you downloaded. - configuration (define/undefine CONFIG_AUFS_xxx) - kernel configuration or /proc/config.gz (if you have it) By the way, I have posted about some unusual behaviours of btrfs. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg02430.html Again I begin thinking btrfs is not usable still as aufs branch. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Hello Dan and Michael, Dan Kegel: > Thanks for the short repro script. I will try Michael's to reproduce the problem on my test env. > I believe this is bug #1: http://sourceforge.net/p/aufs/bugs/1/ It is doubtful since the versions, options, branch fs-types are all different from yours. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
... although if it's bug #1, I'm not sure why it just showed up for me with btrfs and not ext4... On Tue, Jan 27, 2015 at 4:03 PM, Dan Kegel wrote: > Thanks for the short repro script. > This prevents debian packaging tools from working; seems like it'd be > good to fix it in aufs rather than adding a workaround into dpkg and > friends. > > I believe this is bug #1: http://sourceforge.net/p/aufs/bugs/1/ > - Dan > > On Tue, Jan 27, 2015 at 3:47 PM, Michael Johnson - MJ wrote: >> I see very similar behavior (and always have with AUFS) and it is easily >> reproducible. Here is how to reproduce 100% of the time: >> >> $ cd /aufsdir; >> $ mkdir -p dir1/dir2; >> $ rm -rf dir1 >> $ ls >> ls: cannot open directory .: Stale file handle >> >> To mke the error go away: >> >> $ cd . >> >> And now an ls works flawlessly. >> >> Basically, any time you have a directory with one or more files or >> directories inside of it, and then you recursively remove it while the >> working directory is the parent, subsequent attempts to access files inside >> of the parent will fail with the 'Stale file handle' error. >> >> For the most part this does not impact me becuase most things are not >> working of files/directories that are a immediate descent of the current >> working directory. But it would be nice if this problem did not occur. >> >> Hope this helps to at least understand the problem better and possible >> provide you with a workaround. >> >> On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel wrote: >>> >>> I've been using aufs on top of ext4 reliably for some time. >>> >>> Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my >>> builds are failing in rm -rf with: >>> >>> /bin/rm: cannot remove >>> `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': >>> Directory not empty >>> internal error: failed to remove unpacked directory of oobleck >>> ... >>> /bin/rm: fts_read failed: Stale NFS file handle >>> >>> Has anybody else seen this? >>> >>> (A superficially similar problem was mentioned five years ago: >>> http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 >>> Likewise, there are superficially similar posts from docker users, >>> but they may be from user error.) >>> >>> I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer >>> supports 3.13 per >>> http://aufs.sourceforge.net/ >>> Not sure how easy it'd be for me to run vanilla kernels on this box. >>> My best move >>> is probably to drop back to ext4 for now. >>> >>> >>> -- >>> Dive into the World of Parallel Programming. The Go Parallel Website, >>> sponsored by Intel and developed in partnership with Slashdot Media, is >>> your >>> hub for all things parallel software development, from weekly thought >>> leadership blogs to news, videos, case studies, tutorials and more. Take a >>> look and join the conversation now. http://goparallel.sourceforge.net/ >> >> >> >> >> -- >> Michael Johnson - MJ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Thanks for the short repro script. This prevents debian packaging tools from working; seems like it'd be good to fix it in aufs rather than adding a workaround into dpkg and friends. I believe this is bug #1: http://sourceforge.net/p/aufs/bugs/1/ - Dan On Tue, Jan 27, 2015 at 3:47 PM, Michael Johnson - MJ wrote: > I see very similar behavior (and always have with AUFS) and it is easily > reproducible. Here is how to reproduce 100% of the time: > > $ cd /aufsdir; > $ mkdir -p dir1/dir2; > $ rm -rf dir1 > $ ls > ls: cannot open directory .: Stale file handle > > To mke the error go away: > > $ cd . > > And now an ls works flawlessly. > > Basically, any time you have a directory with one or more files or > directories inside of it, and then you recursively remove it while the > working directory is the parent, subsequent attempts to access files inside > of the parent will fail with the 'Stale file handle' error. > > For the most part this does not impact me becuase most things are not > working of files/directories that are a immediate descent of the current > working directory. But it would be nice if this problem did not occur. > > Hope this helps to at least understand the problem better and possible > provide you with a workaround. > > On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel wrote: >> >> I've been using aufs on top of ext4 reliably for some time. >> >> Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my >> builds are failing in rm -rf with: >> >> /bin/rm: cannot remove >> `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': >> Directory not empty >> internal error: failed to remove unpacked directory of oobleck >> ... >> /bin/rm: fts_read failed: Stale NFS file handle >> >> Has anybody else seen this? >> >> (A superficially similar problem was mentioned five years ago: >> http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 >> Likewise, there are superficially similar posts from docker users, >> but they may be from user error.) >> >> I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer >> supports 3.13 per >> http://aufs.sourceforge.net/ >> Not sure how easy it'd be for me to run vanilla kernels on this box. >> My best move >> is probably to drop back to ext4 for now. >> >> >> -- >> Dive into the World of Parallel Programming. The Go Parallel Website, >> sponsored by Intel and developed in partnership with Slashdot Media, is >> your >> hub for all things parallel software development, from weekly thought >> leadership blogs to news, videos, case studies, tutorials and more. Take a >> look and join the conversation now. http://goparallel.sourceforge.net/ > > > > > -- > Michael Johnson - MJ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I see very similar behavior (and always have with AUFS) and it is easily reproducible. Here is how to reproduce 100% of the time: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle To mke the error go away: $ cd . And now an ls works flawlessly. Basically, any time you have a directory with one or more files or directories inside of it, and then you recursively remove it while the working directory is the parent, subsequent attempts to access files inside of the parent will fail with the 'Stale file handle' error. For the most part this does not impact me becuase most things are not working of files/directories that are a immediate descent of the current working directory. But it would be nice if this problem did not occur. Hope this helps to at least understand the problem better and possible provide you with a workaround. On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel <[1]d...@kegel.com> wrote: I've been using aufs on top of ext4 reliably for some time. Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my builds are failing in rm -rf with: /bin/rm: cannot remove `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': Directory not empty internal error: failed to remove unpacked directory of oobleck ... /bin/rm: fts_read failed: Stale NFS file handle Has anybody else seen this? (A superficially similar problem was mentioned five years ago: [2]http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 Likewise, there are superficially similar posts from docker users, but they may be from user error.) I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer supports 3.13 per [3]http://aufs.sourceforge.net/ Not sure how easy it'd be for me to run vanilla kernels on this box. My best move is probably to drop back to ext4 for now. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. [4]http://goparallel.sourceforge.net/ -- Michael Johnson - MJ References 1. mailto:d...@kegel.com 2. http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 3. http://aufs.sourceforge.net/ 4. http://goparallel.sourceforge.net/ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Stale NFS file handle when using aufs on top of btrfs?
I've been using aufs on top of ext4 reliably for some time. Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my builds are failing in rm -rf with: /bin/rm: cannot remove `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': Directory not empty internal error: failed to remove unpacked directory of oobleck ... /bin/rm: fts_read failed: Stale NFS file handle Has anybody else seen this? (A superficially similar problem was mentioned five years ago: http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 Likewise, there are superficially similar posts from docker users, but they may be from user error.) I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer supports 3.13 per http://aufs.sourceforge.net/ Not sure how easy it'd be for me to run vanilla kernels on this box. My best move is probably to drop back to ext4 for now. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
