Re: [Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
I have finally discovered what's going on here--at least, somewhat. I discovered that, when I encounter this bug, the Adblock Plus Firefox extension is in some kind of loop constantly rewriting one of its filter ini files. If I constantly $(ls -l) in the directory, I can see the same file constantly being rewritten, with its size going to zero, then up to the proper size, and back down to zero, then back up... It's the same filename being rewritten over and over again, and the file is about 1.5 MB, so that explains the slow but steady apparent loss of gigs of space. Since the space is recovered on reboot (or perhaps on logout when the eCryptfs volume is closed), I'm guessing it's some kind of bug related to truncating files and it not releasing the space when the file is truncated, so the rewriting eventually uses all available space. I can imagine a similar situation happening with BitTorrent disk I/O patterns. I haven't tried to reproduce the bug manually, outside of Firefox, but perhaps this info will help point you in the right direction. I'm not completely sure if the ABP extension is still exhibiting this behavior--perhaps an update to it has fixed its looping bug. Incidentally, in the few instances of my using torrents in eCryptfs, I haven't had any problems. Thanks for your work on this bug, Tyler. On Thu, Sep 13, 2012 at 6:26 PM, Tyler Hicks wrote: > Oddly enough, I've never been able to reproduce this bug until today > while I was testing a fix for another bug. I've written a fix an built a > set of amd64 test kernels, one for Precise and one for Quantal. If > you're interested in giving one of the test kernels a try, you can find > them here: > > http://people.canonical.com/~tyhicks/ecryptfs/fixes/ > > ** Changed in: ecryptfs >Status: Confirmed => In Progress > > ** Description changed: > > + NOTE: A test case for this bug has been created at > + tests/kernel/lp-561129.sh (revno 731) in the upstream ecryptfs-utils > + project. > + > 1 - Try to download some Ubuntu DVD versions with bit torrent (so it > reserves space). > 2 - Fill your disk to the maximum and leave something like 5GB. > 3 - Wait > 4 - Disk space will go to 0 (it doesn't make any sense since space has > already been reserved) > 5 - Ctrl+Alt+F1 > > There we can see some messages about ecryptfs : > ecryptfs_write_lower: octets_written [-28]; expected [4096] > ecryptfs_encrypt_page: Error Attempting to write lower page; rc = [-22] > [...] > > I don't really know if it's really ecryptfs but it doesn't make any > sense that disk space just get used up like this when only my firefox > browser and bit torrent are open and not trying to write anything else. > > While your disk space is at 0, it's slow as hell and it will swap quite > much (even though your ram isn't full). > > ** Summary changed: > > - eCryptfs sucks up all disk space with Oneiric kernel > + Existing eCryptfs inodes are not evicted when they're the target of a > rename()/mv > > ** Description changed: > > NOTE: A test case for this bug has been created at > tests/kernel/lp-561129.sh (revno 731) in the upstream ecryptfs-utils > project. > + > + This bug is the result of existing eCryptfs inodes not being evicted > + when they are the target of a rename() syscall. The existing inodes are > + left around, meaning that the lower inodes are also left around, until > + the eCryptfs filesystem in unmounted. This means that disk space is not > + properly freed when mv'ing a file on top of another file. > + > + Here's the original bug report: > + --- > > 1 - Try to download some Ubuntu DVD versions with bit torrent (so it > reserves space). > 2 - Fill your disk to the maximum and leave something like 5GB. > 3 - Wait > 4 - Disk space will go to 0 (it doesn't make any sense since space has > already been reserved) > 5 - Ctrl+Alt+F1 > > There we can see some messages about ecryptfs : > ecryptfs_write_lower: octets_written [-28]; expected [4096] > ecryptfs_encrypt_page: Error Attempting to write lower page; rc = [-22] > [...] > > I don't really know if it's really ecryptfs but it doesn't make any > sense that disk space just get used up like this when only my firefox > browser and bit torrent are open and not trying to write anything else. > > While your disk space is at 0, it's slow as hell and it will swap quite > much (even though your ram isn't full). > > ** Changed in: ecryptfs-utils (Ubuntu) >Status: Confirmed => Invalid > > ** Also affects: linux (Ubuntu) >Importance: Undecided >Status: New > > ** Changed in: linux (Ubuntu) >Status: New => Triaged > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/561129 > > Title: > Existing eCryptfs inodes are not evicted when they're the target of a > rename()/mv > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ecryptfs/+bu
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
Oddly enough, I've never been able to reproduce this bug until today while I was testing a fix for another bug. I've written a fix an built a set of amd64 test kernels, one for Precise and one for Quantal. If you're interested in giving one of the test kernels a try, you can find them here: http://people.canonical.com/~tyhicks/ecryptfs/fixes/ ** Changed in: ecryptfs Status: Confirmed => In Progress ** Description changed: + NOTE: A test case for this bug has been created at + tests/kernel/lp-561129.sh (revno 731) in the upstream ecryptfs-utils + project. + 1 - Try to download some Ubuntu DVD versions with bit torrent (so it reserves space). 2 - Fill your disk to the maximum and leave something like 5GB. 3 - Wait 4 - Disk space will go to 0 (it doesn't make any sense since space has already been reserved) 5 - Ctrl+Alt+F1 There we can see some messages about ecryptfs : ecryptfs_write_lower: octets_written [-28]; expected [4096] ecryptfs_encrypt_page: Error Attempting to write lower page; rc = [-22] [...] I don't really know if it's really ecryptfs but it doesn't make any sense that disk space just get used up like this when only my firefox browser and bit torrent are open and not trying to write anything else. While your disk space is at 0, it's slow as hell and it will swap quite much (even though your ram isn't full). ** Summary changed: - eCryptfs sucks up all disk space with Oneiric kernel + Existing eCryptfs inodes are not evicted when they're the target of a rename()/mv ** Description changed: NOTE: A test case for this bug has been created at tests/kernel/lp-561129.sh (revno 731) in the upstream ecryptfs-utils project. + + This bug is the result of existing eCryptfs inodes not being evicted + when they are the target of a rename() syscall. The existing inodes are + left around, meaning that the lower inodes are also left around, until + the eCryptfs filesystem in unmounted. This means that disk space is not + properly freed when mv'ing a file on top of another file. + + Here's the original bug report: + --- 1 - Try to download some Ubuntu DVD versions with bit torrent (so it reserves space). 2 - Fill your disk to the maximum and leave something like 5GB. 3 - Wait 4 - Disk space will go to 0 (it doesn't make any sense since space has already been reserved) 5 - Ctrl+Alt+F1 There we can see some messages about ecryptfs : ecryptfs_write_lower: octets_written [-28]; expected [4096] ecryptfs_encrypt_page: Error Attempting to write lower page; rc = [-22] [...] I don't really know if it's really ecryptfs but it doesn't make any sense that disk space just get used up like this when only my firefox browser and bit torrent are open and not trying to write anything else. While your disk space is at 0, it's slow as hell and it will swap quite much (even though your ram isn't full). ** Changed in: ecryptfs-utils (Ubuntu) Status: Confirmed => Invalid ** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: Existing eCryptfs inodes are not evicted when they're the target of a rename()/mv To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
I can confirm in an old, well-used virtual machine. After Firefox use, free space decreases from ~2.7GB to 1 GB, then no space at all within another few minutes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
I'm seeing this on latest precise as well. However, I'm not using Firefox, I'm using chrome. Not sure if I'm having the same issue, but it seems related. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
Thanks for chiming in on this, Chun-Yu--I didn't think I could really be the only one seeing it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
Update: Here's an example of this happening (as root, after logging in as my user, running firefox, and logging out): # lsof +L1 # df -k /home/cshei Filesystem 1K-blocks Used Available Use% Mounted on /home/cshei/.Private 26172888 19455440 5405296 79% /home/cshei # umount /home/cshei # df -k /home/cshei Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 26172888 19333016 5527720 78% / Also, if I move ~/.mozilla to a separate (unencrypted) filesystem and symlink it, it seems to slow/stop(?) the bleeding. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
I'm now seeing the same thing on a Gentoo system running the vanilla 3.3.0 kernel and ext4. I tried rebooting into 3.2.11, and still see the same problem. I've been running this setup for quite a while, so somewhere along the line, something broke recently. As soon as I start Firefox 11, my free space constantly decreases until it hits 0, makeing eCryptfs unusable. Logging out and killing all my user's processes doesn't help, and looking for deleted files that were still open with "lsof +L1" showed nothing. As soon as I unmounted my encrypted home directory, though, the amount free space returned to normal. I was also able to reproduce this on an eCryptfs on ZFS filesystem, so it's not restricted to ext4. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
Yes, free disk space on the ext4 root partition /dev/sda5 in which /home/username/.ecryptfs resides. Memory usage is not an issue here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
Tyler, I saw your work on other eCryptfs bugs, so I added you to this one. I see that other bugs like #842647 are assigned to eCryptfs and ecryptfs-utils, so I'm switching it back. ** Package changed: linux (Ubuntu) => ecryptfs ** Also affects: ecryptfs-utils (Ubuntu) Importance: Undecided Status: New ** Changed in: ecryptfs-utils (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ecryptfs/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 561129] Re: eCryptfs sucks up all disk space with Oneiric kernel
I think this bug belongs to the kernel and not the userland package. ** Tags added: kernel ** Project changed: ecryptfs => linux (Ubuntu) ** No longer affects: ecryptfs-utils (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/561129 Title: eCryptfs sucks up all disk space with Oneiric kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/561129/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
