Re: remount,del maybe doesn't free resources
Tomas M: > I didn't test the patch you sent me last time yet because I'm packing > some stuff and I'm leaving to holidays for 10 days. So no need to hurry, > take your time :) Hi Tomas, When you return from your vacation, please test the latest aufs (last Monday release). Junjiro Okajima - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
Tomas M: > Do you think all the problems may be gone if I disable inotify support > in kernel somehow? (if that is even possible). I think I don't need it > at all, and for Slax it's more important to support removing of aufs > branches then using inotify. It is up to KDE or other applications who use inotify. If you disable inotify in kernle config, then those app will behave differently. I don't know it is preferable or not. One of the directory which I found inotify was set, was /usr/share/services/kresources/knotes by gam_server. This directory exists only 008-office.lzm, so you will not be able to delete the branch. Additionally lsof will not show you such directory. Junjiro Okajima - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
> Probably I could reproduce the problem, but my environemnt is far from > yours. I put your iso image on my nfs server and used pxe-boot client. > On the nfs client, mount several squashfs images via NFS, and mount aufs. > > The component(s) in KDE seems to issue inotify which refers inode > (increments its reference count). Current aufs checks only the dentry > (filename) is alive or not at deleting branch. The inode which is set > inotify can survive without dentry. So the br_del() succeeded while it > should fail with EBUSY. > > I will try fixing this bug and include in next Monday release. > But it may not be finished in a few days. > Please wait one or two weeks. I think I can understand what you mean, it seems reasonable. Do you think all the problems may be gone if I disable inotify support in kernel somehow? (if that is even possible). I think I don't need it at all, and for Slax it's more important to support removing of aufs branches then using inotify. I didn't test the patch you sent me last time yet because I'm packing some stuff and I'm leaving to holidays for 10 days. So no need to hurry, take your time :) Thank you very much. - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
Tomas M: > Hm it's hard :) My problem is that I usually can't reproduce it in a > normal Linux environment, I'm still finding these things in Slax, which > is a chrooted OS and makes things hard to debug. > > Nevertheless, you can download Slax from here > http://www.slax.org/dl/slax6broken.iso > - burn it to CD, boot it, choose the third option from boot menu (start > Slax with KDE directly). > - while in KDE, run konsole and execute: > > $ mount -n -o remount,del:/mnt/live/memory/images/008-office.lzm aufs / > $ umount /mnt/live/memory/images/008-office.lzm > $ dmesg Probably I could reproduce the problem, but my environemnt is far from yours. I put your iso image on my nfs server and used pxe-boot client. On the nfs client, mount several squashfs images via NFS, and mount aufs. The component(s) in KDE seems to issue inotify which refers inode (increments its reference count). Current aufs checks only the dentry (filename) is alive or not at deleting branch. The inode which is set inotify can survive without dentry. So the br_del() succeeded while it should fail with EBUSY. I will try fixing this bug and include in next Monday release. But it may not be finished in a few days. Please wait one or two weeks. Junjiro Okajima - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
Tomas M: > This was a different problem. > Please consider our conversation about /dev/initctl resolved. > > I am working with two problems which are not related together: I am afraid you are confusing my two patches. :-) > One is the problem of unmounting union (which probably doesn't work > because init process is started from it). (please consider this resolved) The last debug patch is for this problem, which include Dbg(...) line. It just reports dentries in use. > Second problem is I'm trying to remove branches while the system is > running KDE, and here I get the problem with busy inodes on loop. The last patch which you are going to try is for this problem. Junjiro Okajima - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
> You wrote that lsof reported /dev/initctl was opened. > Are your sure that the device file is NOT inside aufs? This was a different problem. Please consider our conversation about /dev/initctl resolved. I am working with two problems which are not related together: One is the problem of unmounting union (which probably doesn't work because init process is started from it). (please consider this resolved) Second problem is I'm trying to remove branches while the system is running KDE, and here I get the problem with busy inodes on loop. Both problems would be easier to fix if there is a way how to see if aufs has any resources used from a given branch, nevertheless please don't confuse them together :) I will try the patch, thank you very much. Tomas M - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
Tomas M:
> The 'umount /squashfs' part sometimes causes the error message
> "VFS: Busy inodes after unmount loop* ... etc", so I _think_ the loop
:::
About this problem, please test this patch.
> Hm it's hard :) My problem is that I usually can't reproduce it in a
> normal Linux environment, I'm still finding these things in Slax, which
> is a chrooted OS and makes things hard to debug.
>
> Nevertheless, you can download Slax from here
> http://www.slax.org/dl/slax6broken.iso
I will try after your uploading completes.
> >> I tried the patch you sent me, but it does nothing, or I can't see any
> >> output in dmesg after remount,del, I'm sorry.
> >
> > And EBUSY error returned?
> > Then it means,
> > - one or more files were opened on that branch.
>
> This is possible but i'm unable to say, I don't know.
> I hope the feature to see this info through /sys
> will be implemented soon :)
Basically, /sys/fs/aufs//opened_files will be equivalent to lsof.
And then EBUSY is not depends upon only the opened files.
You wrote that lsof reported /dev/initctl was opened.
Are your sure that the device file is NOT inside aufs?
If a process (init or something) invoked before you over-mount /dev, the
process still refers the device file.
Junjiro Okajima
--
diff -x CVS -x RCS -x .pdiff -rup ../../aufs.anon/aufs/fs/aufs/opts.c
./fs/aufs/opts.c
--- ../../aufs.anon/aufs/fs/aufs/opts.c 2007-06-18 10:51:10.0 +0900
+++ ./fs/aufs/opts.c2007-06-19 15:24:33.679360152 +0900
@@ -764,7 +764,7 @@ static int au_do_opt_simple(struct super
* minus: error
*/
static int au_do_opt_br(struct super_block *sb, struct opt *opt, int remount,
- int *do_refresh)
+ int *do_refresh, int *deleted)
{
int err;
@@ -790,7 +790,7 @@ static int au_do_opt_br(struct super_blo
case Opt_idel:
err = br_del(sb, &opt->del, remount);
if (!err)
- *do_refresh = err = 1;
+ *deleted = *do_refresh = err = 1;
break;
case Opt_mod:
@@ -906,7 +906,7 @@ static int verify_opts(struct super_bloc
int au_do_opts_mount(struct super_block *sb, struct opts *opts)
{
- int err, do_refresh;
+ int err, do_refresh, deleted;
struct inode *dir;
struct opt *opt;
unsigned int flags, given;
@@ -935,10 +935,11 @@ int au_do_opts_mount(struct super_block
au_flag_clr(sb, AuFlag_XINO | AuFlag_DLGT);
udba_set(sb, AuFlag_UDBA_REVAL);
- do_refresh = 0;
+ deleted = do_refresh = 0;
opt = opts->opt;
while (err >= 0 && opt->type != Opt_tail)
- err = au_do_opt_br(sb, opt++, /*remount*/0, &do_refresh);
+ err = au_do_opt_br(sb, opt++, /*remount*/0, &do_refresh,
+ &deleted);
if (err > 0)
err = 0;
else if (unlikely(err < 0))
@@ -996,7 +997,7 @@ int au_do_opts_mount(struct super_block
}
int au_do_opts_remount(struct super_block *sb, struct opts *opts,
- int *do_refresh, unsigned int *given)
+ int *do_refresh, unsigned int *given, int *deleted)
{
int err, rerr;
struct inode *dir;
@@ -1012,7 +1013,7 @@ int au_do_opts_remount(struct super_bloc
//AuDebugOn(au_flag_test(sb, AuFlag_UDBA_INOTIFY));
err = 0;
- *do_refresh = 0;
+ *deleted = *do_refresh = 0;
*given = 0;
dlgt = au_flag_test(sb, AuFlag_DLGT);
opt_xino = NULL;
@@ -1025,7 +1026,8 @@ int au_do_opts_remount(struct super_bloc
au_flag_clr(sb, AuFlag_DLGT);
if (!err)
- err = au_do_opt_br(sb, opt, /*remount*/1, do_refresh);
+ err = au_do_opt_br(sb, opt, /*remount*/1, do_refresh,
+ deleted);
if (!err)
err = au_do_opt_xino(sb, opt, /*remount*/1, &opt_xino);
diff -x CVS -x RCS -x .pdiff -rup ../../aufs.anon/aufs/fs/aufs/opts.h
./fs/aufs/opts.h
--- ../../aufs.anon/aufs/fs/aufs/opts.h 2007-06-18 10:51:10.0 +0900
+++ ./fs/aufs/opts.h2007-06-19 15:38:47.967488616 +0900
@@ -90,7 +90,7 @@ void au_free_opts(struct opts *opts);
int au_parse_opts(struct super_block *sb, char *str, struct opts *opts);
int au_do_opts_mount(struct super_block *sb, struct opts *opts);
int au_do_opts_remount(struct super_block *sb, struct opts *opts,
- int *do_refresh, unsigned int *given);
+ int *do_refresh, unsigned int *given, int *deleted);
#endif /* __KERNEL__ */
#endif /* __AUFS_OPTS_H__ */
diff -x CVS -x RCS -x .pdiff -rup ../../aufs.anon/aufs/fs/aufs/super.c
./fs/aufs/super.c
--- ../../aufs.anon/aufs/fs/aufs/super.c2007-06-18 10:53:21.0
+0900
+++ ./fs/aufs/super.c 2007-06-19 15:40:42.17
Re: remount,del maybe doesn't free resources
> What you did is, > - mount -o remount,del:/squashfs /aufs > - umount /squashfs > - some operation > - aufs crashes yes, exactly. The 'umount /squashfs' part sometimes causes the error message "VFS: Busy inodes after unmount loop* ... etc", so I _think_ the loop just dies because of aufs and then aufs is confused :) It's just my guess. It seems to me the problem is only with a branch where some files were opened in the past. There is one important note to say: if the branch is busy, the remount,del does nothing of course. Remount shows '/ is busy' and the system can work correctly. BUT, if the branch is NOT busy (or maybe it IS busy but aufs doesn't know) then it can be deleted without any error and THEN unmounting the branch's loop-source fails in VFS. > Please tell me how to reproduce this problem. > I want to reproduce your environment on my pc. Hm it's hard :) My problem is that I usually can't reproduce it in a normal Linux environment, I'm still finding these things in Slax, which is a chrooted OS and makes things hard to debug. Nevertheless, you can download Slax from here http://www.slax.org/dl/slax6broken.iso - burn it to CD, boot it, choose the third option from boot menu (start Slax with KDE directly). - while in KDE, run konsole and execute: $ mount -n -o remount,del:/mnt/live/memory/images/008-office.lzm aufs / $ umount /mnt/live/memory/images/008-office.lzm $ dmesg Interesting thing is, that if I try to remove '009-devel.lzm' instead (using the same commands), I never get the error. For this tests I use noplink mount option to create the aufs union, but it's the same without the noplink. Another note, maybe important, maybe not - I'm using Vanila Kernel with squashfs patch and loop patch. Since the error message (VFS:...) contains a reference to the loop device, I'm mentioning the fact I'm using a patch http://ftp.linux.cz/pub/linux/slax/source/slax/kernel/2.6.21.2/src-core/patches/loop/loop.c.diff But I believe it is not the reason for our problem, as this patch should be included in 2.6.22 or 2.6.23 mainline kernel so it should be safe. >> I tried the patch you sent me, but it does nothing, or I can't see any >> output in dmesg after remount,del, I'm sorry. > > And EBUSY error returned? > Then it means, > - one or more files were opened on that branch. This is possible but i'm unable to say, I don't know. I hope the feature to see this info through /sys will be implemented soon :) > Or can you receive the kernel debug log? > Try, > # echo 8 > /proc/sys/kernel/printk > (/etc/syslog.conf) > kern.debug /var/log/debug.log > # restart your_syslog_daemon I can do this, still the same, no info either in dmesg or on konsole. Thank you very much for taking care about this! Tomas M - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
Tomas M: > I modified initrd in order to mount the entire aufs using 'noplink' > mount option from the beginning. It doesn't help, there is still the > same problem. What you did is, - mount -o remount,del:/squashfs /aufs - umount /squashfs - some operation - aufs crashes Please tell me how to reproduce this problem. I want to reproduce your environment on my pc. > I tried the patch you sent me, but it does nothing, or I can't see any > output in dmesg after remount,del, I'm sorry. And EBUSY error returned? Then it means, - one or more files were opened on that branch. or - there is only one branch left in aufs. Or can you receive the kernel debug log? Try, # echo 8 > /proc/sys/kernel/printk (/etc/syslog.conf) kern.debug /var/log/debug.log # restart your_syslog_daemon Junjiro Okajima - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
> Basically aufs frees all the resources associated with the removed > branch. If you didn't install /sbin/mount.aufs (and other scripts) and > you didn't specify 'noplink' mount option, then unflushed > pseudo-linked inode may still alive. I modified initrd in order to mount the entire aufs using 'noplink' mount option from the beginning. It doesn't help, there is still the same problem. I didn't test to install aufs scripts, but I belive they are only needed for the plink feature, which was not used with noplink mount option, so the scripts are not needed. I tried the patch you sent me, but it does nothing, or I can't see any output in dmesg after remount,del, I'm sorry. Thank you Tomas M - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Re: remount,del maybe doesn't free resources
Hello, Tomas M: > But then, if I try to umount the branch (because the branch was a > loop-mounted filesystem), sometimes I can see in the dmesg: > > VFS: Busy inodes after unmount of loop3, Self=destruct in 5 seconds. > Have a nice day... ::: > So my question is, does aufs completely free all the resources > associated with the branch after 'remount,del' is issued? Or does it > still use something? Basically aufs frees all the resources associated with the removed branch. If you didn't install /sbin/mount.aufs (and other scripts) and you didn't specify 'noplink' mount option, then unflushed pseudo-linked inode may still alive. If you know how to reproduce this problem, try first one of these, - install aufs scripts to /sbin - specify noplink Anyway aufs has something to fix. Junjiro Okajima - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
