Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
2009/7/17 JoseStefan : > I've also been using the Karmic kernels on Jaunty (and the new nvidia > drivers) as suggested by martinm1000. Unfortunately, it seems to require > also updating the graphics drivers, in my case nvidia. Because I'm using Nvidia, too, and read about some problems, I went for 2.6.29 and it worked, I have 3D. > I understand package policy would make it difficult for kernel 2.6.29 or > newer to make it into jaunty. But isn't that what "jaunty-backports" is > for? Using mainline kernels or getting karmic packages is not exactly a > 1 click installation, and in fact could break your system. A backport on > the other hand can be enabled using the GUI. And could provide an easier > fix for those who need it. I actually didn't even enable jaunty-proposed or jaunty-backport, I wanted just a normal failsafe ubuntu. It took so much time to figure out it's actually ext4 causing the troubles! There should be an update even for users who got scared with the sentence "if you enable 'proposed' or 'backport', your system maybe not stable anymore!". Your "pin" sounds promising, I will try this. But with care, as it's currently running! :) Peter -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Colin Sindle wrote: > After an upgrade to 2.6.30-020630-generic #020630 from the Ubuntu Kernel-ppa > mainline, (to solve unrelated HP laptop sound issues), I have not > experienced any more "freezes" temporary, or otherwise. I have as well now manually switched to the 2.6.30-020630 kernel and the freezes are gone... Best regards, Stephan -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Apologies, this is a qualitative post --- but now that people are talking about different processors, I'll contribute some fluffy info. That said, I experienced many "freezes" per day on my Core Solo laptop when doing "dangerous" operations (svn update, rsync, rm, etc.). Then I swapped to a Core 2 Duo, and when doing these same operations, I got about the same number "freezes", only now they recovered faultlessly (so far...) after second or two. After an upgrade to 2.6.30-020630-generic #020630 from the Ubuntu Kernel-ppa mainline, (to solve unrelated HP laptop sound issues), I have not experienced any more "freezes" temporary, or otherwise. c. 2009/7/16 Jared Heath > It happened very frequently on my Dual Core i86 based system (never got > more than 5 single rm commands off without a hang before I went to the > higher kernel) so it certanly can happen on multi-core systems often. > > Your theory on race conditions is interesting though--it certainly > exhibits the behavior of a race that goes infinite and does not get > caught. > > -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Mon, 13 Jul 2009, Franz Dietzmann wrote: > I just read through all the comments (I hope), and did not find this > mentioned, so I thought it might be helpful.. > > I had the problem for a long time, but didn't bother too much. Now it got > annoying and after some searching I installed mainline 2.6.30 to see if it > would work. > As has been mentioned here before it does, but unfortunatly my UMTS didn't > work anymore, so I just deleted my Trash and went back to .28 > After logging in I found I had 10GB more space on my Home-Partition (the > Trash only had ~1GB in it) The partition is only 40 GB total, so that's a > lot. I checked if something was missing, but didn't find anything, which was > strange. > > I ran baobab just out of curiosity and there I found 5GB in > ~/.local/share/Trash/expunged/ > On closer inspection these were all files I supposedly deleted a long time > ago, when the freeze appeared afterwards. I have no idea how they got there, > I'm just a user...but maybe that info can point someone into the right > direction. I'm sure that this is just one of the many ways to trigger this ext4 thing, still, interested me even if not the cause of the bug. http://ubuntuforums.org/showthread.php?t=1196171&page=2 Found this thread which seems to be same issue. Appears that this is related to permission/ownership - so presumably you deleted read-only files. I can imagine that might happen if, for example, the files were copied off a CD and had default read-only permissions. I'm suprised nautilus doesn't handle this more gracefully. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Roland Dreier wrote: > > Luke Maurer wrote: > > I've been able to reproduce it 100% reliably (rm a single file => crashy > > crashy) just in the Jaunty > > LiveCD environment. AFAIK, the most exotic filesystem hackery it uses is > > that union filesystem, > > though I was crashing on deleting something on a separate, non-unionized > > volume. > > Can you give a recipe for how you're able to reproduce it with the > Jaunty Live CD? Um, the "rm" command? :-) Seriously, I boot it up, mount an ext4 volume on a garden-variety disk partition, and try to delete a file. It hangs (evidently *after* deleting the file). Every time. (I'm pretty sure I tried an ext4 image mounted over loopback as well, to no avail.) I realize it's a decidedly extreme case of the bug, but besides the ease of reproduction, the symptoms are identical to those reported here. - Luke -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sun, 2009-07-05 at 23:32 +, Theodore Ts'o wrote: > We might have a break in this bug. For those people who can reliably > reproduce the problem, are you using ecryptfs, possibly extensively? Not at all here. But I rolled back to using ext4dev with the Intrepid kernel and now have leapfrogged over the Jaunty kernel and am using the Karmic kernel on Jaunty userspace. I just don't have the time to continually recover from the Jaunty crashes. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Mon, Jul 06, 2009 at 02:57:32AM -, Feistybird wrote: > Hi Ted, > > > even though at this point we've got a huge number of Fedora 11 users > using ext4 w/o any problems.) > > Fedora 11 uses linux kernel 2.6.29.4, but Ubuntu 9.04 uses kernel 2.6.28 > by default. > > I don't have any ext4 soft-lock-up problems using Ubuntu mainline kernel > 2.6.29 and 2.6.30 netiher Yes, and some people have reported that mainline 2.6.28 or 2.6.28.* also seems to be without problems. I just haven't had the time to try to go through the Ubuntu Jaunty ext4 patch backports, which is where I assume the problem might be. However, the problem Roland reported could very well lead to a deadlock in the truncate path, which is very much related to the problems are reported. The only way the problem Roland's potential deadlock could be related to this is if there is a bug in the Ubuntu ext4 backport such that the truncate code thinks it's only doing a partial truncate in the case of a delete. But we it seems pretty clear that there was a bug in the Ubuntu backports of various ext4 patches, so the one-line patch which I suggested might actually make things better. - Ted -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Theodore Ts'o wrote: > We might have a break in this bug. For those people who can reliably > reproduce the problem, are you using ecryptfs, possibly extensively? > Roland Drier has reported a potential lockdep report that might explain > why some of us have had extreme problems reproducing the problem; namely > we might not be using ecryptfs. See: > > http://lkml.org/lkml/2009/7/4/93 > > If so, there is a sample patch which **might*** fix this problem. See: > > http://lkml.org/lkml/2009/7/5/79 > > Also, if people could try building their kernel with > CONFIG_PROVE_LOCKING, that would be interesting to see if we get a > lockdep report. > > If so, it might be that this bug has been around all along, but it's > something about Ubuntu patches that makes it 100,000 times more likely > to trigger. (In practice, it looks like it should only trigger on a > truncate to a size which is not a multiple of the filesystem blocksize, > not on an unlink --- but maybe there was a bug in the Ubuntu backports > that made this possible to trigger on an unlink. This is only a theory, > but it's the first lockdep report I've gotten that could at least > potentially be related to this problem that to date, only Ubuntu users > have been complaining about, even though at this point we've got a huge > number of Fedora 11 users using ext4 w/o any problems.) > > I've been using Ecryptfs only on ~/Private. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQEcBAEBCAAGBQJKUXBIAAoJEIltSrFpUGteit8IAI1XZ0Co/2bSnErSa5qid68J VEey/fPv8RRxvWd0QvZBRR9c9K+isjAS24+2A3V+veOh1aBd/C9dGOZ8fD4i5ZqZ 1i9f70ZPQnKxc9o1ER+qZ7X0EGiDXOVjFoHlV5GY8/OaaZ6/0R9ILOp2X+LvrY3s ZZgFDjdKMO4058nI11y/Z00fuLwOlk8i2wr4a0ofkRCEkMWCfOtYOD270unEcJNx CIi88ma0IwMelkjdoN+IqPqOSUbhAW/qCSJOBY5uuYlMXtdeXFFC/gMLdb5q34lk vXbbb38744g4THJC03g2qIuDMPAw+QGl3+kMX+DIVs+oP83Je25kHIGiW2oj1v8= =HMof -END PGP SIGNATURE- -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sun, 5 Jul 2009, Theodore Ts'o wrote: > We might have a break in this bug. For those people who can reliably > reproduce the problem, are you using ecryptfs, possibly extensively? > Roland Drier has reported a potential lockdep report that might explain > why some of us have had extreme problems reproducing the problem; namely > we might not be using ecryptfs. See: I was not using ecryptfs on either machine that consistently locked up. The one machine, a laptop, didn't really have any network mounting at all. I did use sshfs but rarely, and never had it mounted when lockups occured. The other machine used CIFS, and might have been in conjunction with lockups. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Ted, I've also not been using ecryptfs. I was one of the earliest reporters of this bug. At that time, (during the beta of Jaunty), the most common use case that triggered the bug was the running of multiple Virtualbox virtual machines WHILE simultaneously moving/deleting files across/from multiple partitions. I must note that, over the last two and a half months, I have had a relatively stable experience with Ext4 on Ubuntu. My Ubuntu laptop sees lots of daily use, and I have only had about five lockups (which may not be the fault of ext4). I still use Virtualbox - although not as much. I still move files around - but not as much. The more recent freezes have all occurred when I have a few large projects open in Eclipse and I am doing an ant build outside on the commandline. The builds and projects are usually large enough to cause a bit of memory pressure (and swapping) on a laptop with 4GB RAM. I am no filesystem expert - so I don't know if this information helps or just clouds things up further. Regards On Mon, Jul 6, 2009 at 4:11 AM, Abraham Smith wrote: > Ted, > I've NOT been using ecryptfs, but I was using NFS and rsync-over-ssh on > ext4 extensively during my lockups. > > -- > Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 > https://bugs.launchpad.net/bugs/330824 > You received this bug notification because you are a direct subscriber > of the bug. > > Status in Ubuntu Release Notes: Fix Released > Status in “linux” package in Ubuntu: Fix Released > Status in linux in Ubuntu Jaunty: Fix Committed > Status in linux in Ubuntu Karmic: Fix Released > > Bug description: > [ > Please read *all* previous comments before posting. > > Mainline kernels are known to not experience this bug, although in general > are not supported (i.e., using one is a workaround, but if they break other > things you're generally out of luck). > > Additional "me-too" comments aren't useful, feel free to select the "This > bug affects me too" option and/or subscribe to this bug instead. > ] > > Binary package hint: linux-image-2.6.28-8-generic > > I'm using 8.10 Kubuntu with all updates done on system. > > System is a clean installed system with EXT4 formating and using 2.6.8-8 > linux kernel. > > System sometimes lock and freeze whole inputs even keyboard or mouse. > I have closed X and kdm and try to reprocedure same bug in console ( not > konsole ) > so i have killed X and kdm. > > And try to compile qt-copy in one console and try to svn up on KDE and on > other console > i tryto apt-get updateto make system under CPU load. and after a while > it happens again. > > No Keyboard response no harddisc response total freeze. > > I have waited a while after freeze and about 4 min later a text appeared on > screen saying : > > BUG: soft locking - CPU#0 stuck for 61s! [uic: 5356] > > after waiting about 4 more minutes a newer but same text appeared unter > this message : > > BUG: soft locking - CPU#0 stuck for 61s! [uic: 5356] > BUG: soft locking - CPU#0 stuck for 61s! [uic: 5356] > > > There isn't any error records on /etc/log/messages releated on hardware > while around freezing/locking times > > And for information : Sometimes i have seen that i'm getting messages like > disc is full but > I'm sure that it isn't. Because df shows me there are more than 7 Gb > freespace. Not always getting this error. > if a file shows this error while i'm updating it i'm deleting it and > downloading a bigger file system won't interrupts me > like saying disk is full. I think it is releated to Ext4. > > But i'm not sure these 2 bugs releated or not. > > Thanks > -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Mon, Jul 06, 2009 at 01:11:40AM -, Abraham Smith wrote: > Ted, > I've NOT been using ecryptfs, but I was using NFS and rsync-over-ssh on ext4 > extensively during my lockups. NFS client or server? If NFS server, were you exporting an ext4 filesystem? - Ted -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Fri, 2009-07-03 at 12:35 +, Stefan Bader wrote: > ** Changed in: linux (Ubuntu Jaunty) >Status: In Progress => Fix Committed Which kernel? Can you paste the changelog entry? -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On 20/06/2009 Michael Rooney wrote: > . Ext4 is not the default file system so > it only affects a small minority of users. See > https://wiki.ubuntu.com/Bugs/Importance for more information. I am not > sure if this would qualify as a severe impact and it is already fixed > in newer kernels as you noted. As I understand priorities, these are per-package, not per distribution. Otherwise a bug that makes e.g. xournal completely unusable would be low priority. Instead it must be high priority, it's high in xournal, not in ubuntu. Likewise, the "medium" designation for a bug that deadlocks the kernel seems a bit wrong also in my opinion. It should be "high" in the kernel. Said this, this bug is being actively worked on, and the priority "medium" does not reflect the effort devs are putting in it, which is "high" so there is nothing to complain about. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sat, 20 Jun 2009 17:49:01 - JoseStefan wrote: > This bug doesn't seem to trigger on ext3 file systems mounted as ext4. > Can anyone confirm? > > Maybe we can pinpoint which of the new ext4 attributes are needed to > trigger the bug? I can confirm that I only saw the bug after converting my filesystems, not before when they were ext3 mounted as ext4. They are all converted now, and I'm running a stock kernel to avoid the crashes so I'm not best placed to test, sorry. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sat, Jun 20, 2009 at 9:27 AM, Andrius Štikonas wrote: > If everybody is using stable defaults then who will test or develop new > software? > Every software community must have early testers and if they report > something it would be in the best interests of community to fix reported bug > and if early testers are ignored, they will just stop testing which > eventually will be very bad for Ubuntu. > You are correct, and these users are definitely important and appreciated! We just must keep in mind the perspective of what we are doing as testers, testing early software, and understand that sometimes it IS going have issues and not to use it for production environments or systems which need to be stable. As such it isn't reasonable or helpful to become infuriated when such an issue is experienced. > No progress is being made on this bug for quite some time and it is likely > to remain so. If nobody knows which patch introduces this bug just apply all > diffs between 2.6.28 and 2.6.29 in fs/[ext4,jbd2] to Jaunty kernel. This is > a huge change, but it unlikely to introduce bugs that are even half as bad > as this one. The problem here is that developers' time is limited. Since the issue is already fixed, their time can be spent benefiting the most users by working on improving things in the default feature set and future release, especially since Jaunty is not an LTS. However of course any contributions by community members are always welcome, including tracking down the commit or getting out a PPA of a fixed kernel, or providing workarounds such as Jason did which have already helped one user on this report (thanks!). -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sat, Jun 20, 2009 at 2:16 AM, Jason Waddle wrote: > Releasing with ext4 (without any sort of cautionary warnings in the > installer!) in this shape was a serious mistake. It is also documented in the release notes: http://www.ubuntu.com/getubuntu/releasenotes/904#Lock-ups%20when%20deleting%20files%20from%20ext4%20filesystems When I decided to give ext4 a try I definitely searched the release notes for ext4 related issues and understood the risk I was taking. Note this isn't something most people are expected to do, but it is definitely a good idea for the minority of people adjusting critical parts of an OS. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
2009/6/20 Brian Shannon > """Ubuntu folks, when am I going to be able to tell my mom it's a good > idea to ditch the Windows and go Ubuntu?""" > > To be blunt, when you decide to use the stable defaults Ubuntu provides. > > Sorry about being off topic but this was just so ridiculous I had to > pass comment. > > -- > Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 > https://bugs.launchpad.net/bugs/330824 > You received this bug notification because you are a direct subscriber > of the bug. > If everybody is using stable defaults then who will test or develop new software? Every software community must have early testers and if they report something it would be in the best interests of community to fix reported bug and if early testers are ignored, they will just stop testing which eventually will be very bad for Ubuntu. No progress is being made on this bug for quite some time and it is likely to remain so. If nobody knows which patch introduces this bug just apply all diffs between 2.6.28 and 2.6.29 in fs/[ext4,jbd2] to Jaunty kernel. This is a huge change, but it unlikely to introduce bugs that are even half as bad as this one. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sat, Jun 20, 2009 at 2:16 AM, Jason Waddle wrote: > I registered just to say that I have been seriously affected by this > bug, and it certainly deserves more than the "medium" designation. While I can definitely understand your frustration, please keep in mind the context of your issue. Ext4 is not the default file system so it only affects a small minority of users. See https://wiki.ubuntu.com/Bugs/Importance for more information. I am not sure if this would qualify as a severe impact and it is already fixed in newer kernels as you noted. Also this isn't affecting all ext4 users, as some people on this report including myself are unable to reproduce it and have had absolutely zero problems with ext4. > > I love Linux in general, and have been very happy with the "hands off" > safe, stable feel of Ubuntu distros, but this latest install was a > freaking disaster since I primarily use this machine remotely. If EXT4 > was known to be so unstable, it should have been marked so. As others have said, if you want the most stable release, go with the file system the OS recommends. The fact that it isn't the default should in itself tell you it is believed to be less stable. The overall stability also was not fully known, which was the point of including it as an option in 9.04, to get testing and learn about important bugs like this so they can be fixed. As with any software, you are not going to know all the bugs until it is released. > I've gone to the mainline kernel 2.6.30 and I had to install the newer > 180 version of nvidia drivers in order to do so. Very helpful > directions for how to do this are given here: > https://bugs.launchpad.net/ubuntu/+source/nvidia-common/+bug/384639. > After the switch to the mainline kernel, no problem with the mass > deletes. Excellent, glad to hear another confirmation that the issues will be addressed in Karmic. > > Ubuntu folks, when am I going to be able to tell my mom it's a good idea > to ditch the Windows and go Ubuntu? Right now, because she is probably not going to select a non-default file system. This is really only relevant more advanced users. Your Mom probably doesn't know or care what a file system is, and shouldn't have to. It is for this exact reason ext4 was not the default in 9.04, and is becoming default in 9.10 (I believe) only after testing and resolving all the issues. So your Mom can happily install Karmic without issues including this one just as she could have Jaunty :) -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
"""Ubuntu folks, when am I going to be able to tell my mom it's a good idea to ditch the Windows and go Ubuntu?""" To be blunt, when you decide to use the stable defaults Ubuntu provides. Sorry about being off topic but this was just so ridiculous I had to pass comment. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Fri, Jun 19, 2009 at 04:49:19PM -, Derek wrote: > >git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git > > for-stable-2.6.28 > > > > http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=shortlog;h=for-stable-2.6.28 > > > > (This is where it's handy to have a file system specialist working at > > the distribution; when I found the problem, I was able to contact Eric > > and he made sure the patch was quickly dropped into the F11 kernel.) > > Is this: > http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=16cb5dd9f53e569130584696909d423b6fe38c1e > > 'cause the machines I was getting lockups on were single core. That's the one; this bug could potentially cause inode table or corruption of block groups, causing data loss. It could *potentially* show up on single processor systems, if CONFIG_PREMPT is defined, but in practice it's unlikely for UP systems to hit the race. - Ted -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Fri, 19 Jun 2009, Theodore Ts'o wrote: > P.S. There is a known ext4 file system corruption bug which is fixed in > the 2.6.30 mainline kernel and in 2.6.29.5. It was found after the > stable kernel series stopped updating for 2.6.28, but I do carry a fix > for it in my for-stable-2.6.28 branch of the ext4 git tree, located > here: > >git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git > for-stable-2.6.28 > > http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=shortlog;h=for-stable-2.6.28 > > (This is where it's handy to have a file system specialist working at > the distribution; when I found the problem, I was able to contact Eric > and he made sure the patch was quickly dropped into the F11 kernel.) Is this: http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=16cb5dd9f53e569130584696909d423b6fe38c1e ? 'cause the machines I was getting lockups on were single core. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Hi There is a better solution as suggested before in the discussion. Download 2.6.30 from here http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.30/. I never had a problem after installing mainline. I wont recommend it if you are using some proprietary drivers from Ubuntu. They may not be supported. My "/" is not on ext4 though but all other partitons which I used for storage are on ext4 now and I was having problem with stock jaunty kernel. On Fri, Jun 19, 2009 at 4:28 PM, Bryan Quigley wrote: > I can not recommend strongly enough that you DO NOT follow jagnet's > advice and upgrade to Karmic to get away from this bug. Yes the bug > is fixed in Karmic, but in development releases many many worse bugs > can be found (especially this early). > > Unfortunately, the only truly supported solution has always been to > use EXT3 (which means reinstall in most cases). > > -- > Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 > https://bugs.launchpad.net/bugs/330824 > You received this bug notification because you are a direct subscriber > of the bug. > > Status in Ubuntu Release Notes: Fix Released > Status in “linux” source package in Ubuntu: Fix Released > Status in linux in Ubuntu Jaunty: In Progress > Status in linux in Ubuntu Karmic: Fix Released > > Bug description: > [ > Please read *all* previous comments before posting. > > Mainline kernels are known to not experience this bug, although in general > are not supported (i.e., using one is a workaround, but if they break other > things you're generally out of luck). > > Additional "me-too" comments aren't useful, feel free to select the "This > bug affects me too" option and/or subscribe to this bug instead. > ] > > Binary package hint: linux-image-2.6.28-8-generic > > I'm using 8.10 Kubuntu with all updates done on system. > > System is a clean installed system with EXT4 formating and using 2.6.8-8 > linux kernel. > > System sometimes lock and freeze whole inputs even keyboard or mouse. > I have closed X and kdm and try to reprocedure same bug in console ( not > konsole ) > so i have killed X and kdm. > > And try to compile qt-copy in one console and try to svn up on KDE and on > other console > i tryto apt-get updateto make system under CPU load. and after a while > it happens again. > > No Keyboard response no harddisc response total freeze. > > I have waited a while after freeze and about 4 min later a text appeared on > screen saying : > > BUG: soft locking - CPU#0 stuck for 61s! [uic: 5356] > > after waiting about 4 more minutes a newer but same text appeared unter > this message : > > BUG: soft locking - CPU#0 stuck for 61s! [uic: 5356] > BUG: soft locking - CPU#0 stuck for 61s! [uic: 5356] > > > There isn't any error records on /etc/log/messages releated on hardware > while around freezing/locking times > > And for information : Sometimes i have seen that i'm getting messages like > disc is full but > I'm sure that it isn't. Because df shows me there are more than 7 Gb > freespace. Not always getting this error. > if a file shows this error while i'm updating it i'm deleting it and > downloading a bigger file system won't interrupts me > like saying disk is full. I think it is releated to Ext4. > > But i'm not sure these 2 bugs releated or not. > > Thanks > -- pritam -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
I can not recommend strongly enough that you DO NOT follow jagnet's advice and upgrade to Karmic to get away from this bug. Yes the bug is fixed in Karmic, but in development releases many many worse bugs can be found (especially this early). Unfortunately, the only truly supported solution has always been to use EXT3 (which means reinstall in most cases). -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
fr., 19.06.2009 kl. 08.57 +, Nicholas Roberts: > Regarding Øyvind's wise words... I agree with what you're saying ! This bug sucks. Also see my reply to dan. And the warning about EXT4 should be more prominent in the release notes. Thankfully, they did not set it as default fs. Sorry for the bug report spamming in general :), I'll stop now. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
fr., 19.06.2009 kl. 08.53 +, skrev _dan_: > I dont want to be rude but the "its your own fault, dont use it attitude" > does not help anyone. > I am pretty sure everyone knows they can use ext3, thats not the point of a > bugreport tho. > If Ubuntu ships with ext4 supoort it should work period. > In this current state its almost unusable. > Yep, and it was not my intention to be rude either. But at the risk of sounding a bit harsh, I'd say that comments like «this should be fixed NOW because it ain't working» also don't help. But I certainly do understand the frustration that many EXT4 Jaunty users must be having wrt. this, and I think this bug is grave. I initially had all three of my machines converted to EXT4 when upgrading to Jaunty. But after observing this bug report for a few weeks, I quikly realised the dangers and converted about 13 partitions on 5 disks back to EXT3 :). My reason for replying was that Nicholas Roberts stated that he was modifying his software to work-around Jaunty EXT4 bugs. And I think he'd be better off leaving his software alone and going back to EXT3 and wait a few more months for EXT4 goodness :). -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
fr., 19.06.2009 kl. 07.26 +, skrev Nicholas Roberts: > I have resisted posting because 'me too' is not helpful. However, I am > operating two very different machines (no two pieces of hardware the > same) with the same operating system (Jaunty [ext4] with all the latest > updates). Both machines exhibit the same 'freezing' problem on deleting > files and the effect is random with seemingly no preference to size or > number of files being deleted. > > As an engineer I hate to give qualitative (versus quantitative) comment, > but I have noticed that the frequency of system freezes has increased > roughly threefold (conservatively) in the last few weeks, particularly > since the move to 2.6.28-13 from 2.6.28-12. > > For the first time in 30 years I am having to modify my software to have > it not delete temporary files just to avoid the operating system > hanging... madness! Dare I say I cannot remember Windoze ever having a > bug this bad; I think this one needs nailing fast gang. > If you need your computers to be stable, then there is no-one stopping you from using EXT3, which is rock solid. EXT4 is not the Jaunty default fs and warnings about EXT4 are clearly available in the release-notes: http://www.ubuntu.com/getubuntu/releasenotes/904#Lock-ups%20when%20deleting%20files%20from%20ext4%20filesystems My advice: don't jump on EXT4 until Karmic. Regards, Øyvind -- < Øyvind Stegard < http://www.oyvind.nu/ -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
loannis, just install a 2.6.30 kernel. I've been running an rc of 2.6.30 on jaunty for months now with nvidia, works fine. DKMS will take care of the restricted modules as necessary. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Confirming the 2.6.28-13.44 kernel from Tim Gardner still has the problem. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkn+NFkACgkQAGHzB9Tvw4zaHgCgofNfbUOmZM6mILTawR+19Y7B SpcAoImn8Ck7t3Ade8tptpJPmwyToifN =vaum -END PGP SIGNATURE- -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
@Tim, Theo: 2.6.28-13.44 from http://kernel.ubuntu.com/~rtg/2.6.28-13.44-lp330824 hung partway through boot with "BUG: soft locking - CPU#0 stuck for 61s! [rmdir:]" -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Sun, May 03, 2009 at 01:32:13PM -, _dan_ wrote: > This issue is *not* fixed in current jaunty. No, it is not fixed in Jaunty, as the status of the bug indicates. "In Progress" means that someone is working on the bug, but it is not yet fixed. Fortunately, it looks like we now have a very good idea where the problem begins, and so hopefully can isolate the root cause now. -- - mdz -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Can reliably trigger this on nvidia, nv and vesa. Theo, did you get my e2image attached above? -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I use the Nvidia binary driver however I've been able to reproduce this from a console without X even running. I logged in from a console, shut down X, then ran hang.py and it froze after about 3 rounds, that's when I saw the "soft lockup on cpu0" message that just repeated over and over. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAknzR5MACgkQAGHzB9Tvw4wd/QCgiwUdCjepeSPS/Yv7D7Dt0U1M UZwAn0FE/VH1jeWxurnaEoLukqqwLn18 =IXdY -END PGP SIGNATURE- -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
I don't use proprietary software on my system, so Nvidia driver shouldn't cause this lockup. Maybe it can trigger it like "rm -rf" but not cause it. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
@Theo That was easy :p I've got a 2.5GB ext4 image which reproduced the issue when mounted on /mnt/test. Attached is the e2image output produced immediately before deleting the files. The hang itself occurred part way into writing the files during the next cycle (as it usually does). ** Attachment added: "ext4-trial0.e2i.gz" http://launchpadlibrarian.net/25694089/ext4-trial0.e2i.gz -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
@Theo, I originally experienced the crash on an external drive (i.e., not the root fs). I'm going to try to reproduce it on a small loop image. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
> So it's still being worked on then? Not to be critical, but isn't this a > bug that's been known for months? Yes, the status is still "In progress". Note that the mainline kernel's don't seem to be exhibiting this issue (or at least orders of magnitude less often), so installing a 2.6.29 kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/ will probably get you up and running (presuming your hardware is supported by a stock kernel + dkms modules). -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
2.5GB of memory, fairly high disk used (~85%), I can reproduce nearly at will. The bisect I performed above may have only found a patch that makes things significantly worse: when running a kernel from the 'good' side of the bisect I still get occasional lockups (on the order of 2-4 days), compared to minutes otherwise. It almost seems that in that case, concurrent activity is a necessary case: firefox's periodic session saving (rewriting a 400kb file), starting audio playback, running a gnome-shell build with io at idle priority, and installing a package was the triggering state the last time I had a 'good' kernel lock up. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
I have 640 MB ant svn up, on KDE repository core modules is enough to lockup. So lockup happens quite easily. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Theodore Ts'o wrote: > @yaztromo, > > Well, the slowest machine I have is an netbook with an Atom N270 1.6GHz > processor with 1.5gigs of memory (Hmm, I wonder if amount of memory has > any significance; are people who can reproduce easily doing so with an > especially large or small amounts of memory?) and a 5400 rpm drive. > > I'll try again with the latest Ubuntu Jaunty kernel, and see if I can > figure out why it wasn't booting on that box with an Intrepid userspace. > The laptop I've been able to easily reproduce this on only has 768MB of RAM. The other laptop has 2GB of RAM and it takes a lot more effort to get that one to lock up. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAknmIzkACgkQAGHzB9Tvw4yGwACfQRwjMcMdcH2OTnG4KG1hXnxT OywAoLAhCnG5RKdj25jUO0NmSteEML1p =90+Y -END PGP SIGNATURE- -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Wed, 2009-04-15 at 17:46 +, Theodore Ts'o wrote: > are people who can reproduce easily doing so with an > especially large or small amounts of memory? MemTotal: 2851296 kB MemFree:164428 kB Buffers:469320 kB Cached: 609504 kB SwapCached: 1780 kB Active:1895404 kB Inactive: 557956 kB HighTotal: 1964992 kB HighFree: 147844 kB LowTotal: 886304 kB LowFree: 16584 kB SwapTotal: 3145720 kB SwapFree: 3119748 kB Dirty:1320 kB Writeback: 0 kB AnonPages: 1374060 kB Mapped: 159116 kB Slab: 145756 kB SReclaimable: 103656 kB SUnreclaim: 42100 kB PageTables: 6940 kB NFS_Unstable:0 kB Bounce: 0 kB WritebackTmp:0 kB CommitLimit: 4571368 kB Committed_AS: 2345724 kB VmallocTotal: 110584 kB VmallocUsed: 60512 kB VmallocChunk:49136 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 4096 kB DirectMap4k:884736 kB DirectMap4M: 32768 kB -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Saïvann Carignan wrote: > Theodore Ts'o : I noticed that this bug is more likely to happen on > partitions that does not have many space left, in case that this give > you some hint. I have a 30 Gb partition which has only 3 Gb left. The > bug can be reproduced immediately each time on that partition. I have a > 700 Gb partition with 500 Gb left, it's pretty hard to reproduce the > issue on it. > The two machines I've been able to reproduce this on have a root partition of around 20GB. One has about 13GB of free space, and the other has about 15GB of free space. hang.py is writing to /tmp so in this case the writing and deleting is happening on the root file system. > One other question --- for the people that have managed to reproduce, do > you know if there was any files getting created or otherwise blocks > getting allocated on the filesystem under test, or is an "rm -rf" of > some hierarchy in the filesystem sufficient on its own to cause the > system to hang? hang.py is creating a bunch of files in /tmp/test and then deleting them all from test/. So it's only deleting files, not a hierarchy of any sort. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAknmF28ACgkQAGHzB9Tvw4wzVwCdHYSivqxE1KQJZ5ANhOMMWanJ VPYAoIKBjVPK27+K39OeBaa4pnGfSrDR =fUB+ -END PGP SIGNATURE- -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Wed, 2009-04-15 at 13:42 +, Theodore Ts'o wrote: > I'll note that part of the problem is that it doesn't seem to be > trivially reproducible even on Ubuntu Jaunty. It seems to reproduce here when two processes are removing files/trees from the same filesystem. And as another comment suggests, my filesystem is pretty damn full (98%) too, so maybe that's the key factor. > On my side, I still need to figure out why a Ubuntu Jaunty kernel built > with my custom "no modules" config causes a black screen lockup when > booting on an Ubuntu Intrepid userspace --- or find room to do an > install of Ubuntu Jaunty beta and try to reproduce the problem myself. > Oh, and I have to get my taxes filed too, and expense reports, and lots > of other things related to my day job (which doesn't include ext4; it's > been a long time since anyone has paid me to work on ext4 as my day job > -- it's something I do in my copious spare time in the evenings or when > I have a few spare moments). I hear ya. I didn't at all mean to suggest that you (Ted) should be working to fix this (indeed, your volunteered contributions on the bug are significant in it's progress), but hopefully somebody who's getting paid to work on Ubuntu Linux could devote some time to it. Ted: As to your other question about allocation during removal, I tend to doubt in my case there was. This is an archive/backup filesystem in my case and the parallel (but not racing) deletes happen after the backup run, so there shouldn't be any allocation happening at that time. Ted: FWIW, you might (or might not) recall we discussed the speed of deleting hardlink trees in ext3 vs. XFS a few months ago when I was switching to XFS specifically for deletes speed... well, a few XFS crashes (and one almost 24 hour xfs_repair) later I am back in the ext* fold on ext4 and happy to report that XFS was just as slow as ext3 and ext4 beats them both hands down at deleting big (many) hard linked trees of files. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Wed, 2009-04-15 at 11:46 +, Carey Underwood wrote: > Because it's still being investigated, and 9.04 isn't actually > released yet. There's a perfectly good chance that the culprit is > nailed down and fixed before release, and if it can't, then simply > removing it from the final release notes and installer would suffice. Ahh, yes. This sounds very good. The previous comment, to which I was replying gave an impression that nothing more was going to be done (i.e. like perhaps it was too close to freeze to make any more changes) and Jaunty would just release "as is". If the decision was made to simply disable it if it cannot be fixed, then yes, this is very good. Doing otherwise, again, IMHO, would further besmirch the reputation of Ubuntu's releases[1]. [1] As if Intrepid didn't do enough besmirching of its own. I know people who are actively searching for alternate distros because of Interepid's bugginess. I personally have not upgraded any of my friends/family to Intrepid because of same said buginess. The jury is still out on Jaunty for me, but including evolution 2.26 (technically anything > 2.22) is not helping Jaunty's case. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Because it's still being investigated, and 9.04 isn't actually released yet. There's a perfectly good chance that the culprit is nailed down and fixed before release, and if it can't, then simply removing it from the final release notes and installer would suffice. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Tue, 2009-04-14 at 23:35 +, Carey Underwood wrote: > The existence of this bug should be noted in the release notes under > known issues as well as in the feature summary for "Ext4 filesystem > support". So basically, the fact that ext4 simply does not work in Ubuntu Jaunty should be in the release notes? Maybe you should just be honest with the upcoming user base and simply disable it until it does work. Delivering something known to be so broken rather than simply failing to deliver, IMHO, is the worse option. If people really want/need ext4 and it's a deciding factor in choosing a distro, they should be allowed to evaluate their choices without the smoke and mirrors that it works in Ubuntu. Don't waste their time. I'm not sure why this is not being resolved anyway. Ted gave some awesome advise into tracking down this bug and some other people spent a significant portion of their own valuable time to do most of the grunt work. Why are the results of those efforts not being put towards fixing the problem and making a release that works? -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm still able to reproduce the freeze with hang.py using the suggested kernel 2.6.28-11 #42 I have a laptop that I have been able to reproduce this bug with rather easily possibly due to it's low system specs. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAknk67wACgkQAGHzB9Tvw4yGvACcD/HRt17cAgx70Dl9uZEGeETM 0yUAn0dCo/jZJve6yfmgAypOrgVxcWtx =VxjD -END PGP SIGNATURE- -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
"UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: cleanup mballoc header files" also fails with the same error: relevant retyped excerpt from dmesg: EXT4-fs: barriers enabled kjournald2 starting. Commit interval 5 seconds EXT4-fs: delayed allocation enabled EXT4-fs: file extents enabled EXT4-fs: mballoc enabled EXT4-fs: mounted filesystem with ordered data mode EXT4 FS on sdb1, interval journal on sdb1:8 EXT4-fs error (device sdb1): ext4_mb_generate_buddy: EXT4-fs: group 232: 15995 blocks in bitmap, 15994 in gd Aborting journal on device sdb1:8 Remounting filesystem read-only EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sdb1) in ext4_ext_remove_space: Journal has aborted EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sdb1) in ext4_orphan_del: Journal has aborted EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sdb1): mb_free_blocks: double-free of inode 0's block 7607748(bit 5572 in group 232) One more kernel to test... -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
Almost done bisecting. git-bisect start # good: [c04f828bd3a42d738f547fe6b0549cf70510a380] relay: fix lock imbalance in relay_late_setup_files git-bisect good c04f828bd3a42d738f547fe6b0549cf70510a380 # bad: [75a9a0bdb7f5d4a9a29711a3232b24fab35eb4e0] cpuidle: Add decaying history logic to menu idle predictor git-bisect bad 75a9a0bdb7f5d4a9a29711a3232b24fab35eb4e0 # bad: [0bfe75ee038b6774197e03990c1e6132c26cc4dc] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Fix race between read_block_bitmap() and mark_diskspace_used() git-bisect bad 0bfe75ee038b6774197e03990c1e6132c26cc4dc # good: [938ded64f043e003a2381b46f890cafb0ebd5e2a] ALSA: hda - More fixes on Gateway entries git-bisect good 938ded64f043e003a2381b46f890cafb0ebd5e2a # good: [7051f08630b7269d548930be358624f2830577df] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Add support for non-native signed/unsigned htree hash algorithms git-bisect good 7051f08630b7269d548930be358624f2830577df # good: [dad87da3db508b0e7befb67c2d7e70219b2bcafc] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] jbd2: Add barrier not supported test to journal_wait_on_commit_record git-bisect good dad87da3db508b0e7befb67c2d7e70219b2bcafc # skip: [dbf8b1c4e8122e705447b69aea9ee6ef3a9caa30] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Use EXT4_GROUP_INFO_NEED_INIT_BIT during resize git-bisect skip dbf8b1c4e8122e705447b69aea9ee6ef3a9caa30 That last one fails on boot to mount the filesystem due to a inode that was double freed (I think, I'll rerun that one after I finish the bisect). This leaves the following patches: [ bad] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Fix race between read_block_bitmap() and mark_diskspace_used() [ ? ] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: don't use blocks freed but not yet committed in buddy cache init [ ? ] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: cleanup mballoc header files [skip] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Use EXT4_GROUP_INFO_NEED_INIT_BIT during resize [ ? ] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Add blocks added during resize to bitmap [ ? ] UBUNTU: SAUCE: (revert before 2.6.28.y update) [PATCH] ext4: Don't overwrite allocation_context ac_status -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
@Theo I'm about halfway through a git bisect of mainline 2.6.28 -> 2.6.29, and (unsurprisingly I now realize) haven't been able to reproduce. I'm pulling down git://kernel.ubuntu.com/ubuntu/ubuntu-jaunty.git right now, and will run a bisect against mainline 2.6.28 and ubuntu's 2.6.28-11.38. On Tue, Mar 31, 2009 at 11:53 AM, Theodore Ts'o wrote: > So what this means is at this point, is that I need some volunteer to > try doing a git bisect, this time between a stock kernel version used as > the base for an official Ubuntu kernel, and the fully applied set of > patches for an Ubuntu kernel, so we can find which patch in the Ubuntu > sauce series seems to make this problem easy to manifest (i.e., > appearing within hours, versus taking a month or so of normal usage > before it shows up). When we find the "problem" patch, it may not be > the guilty patch, but it might unmask the real root cause of the > problem, and so by looking at the patch, maybe we'll get a good hint > about what the real underlying problem might be. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Mon, 2009-03-30 at 17:40 +, Andrius Štikonas wrote: > but my laptop is not > so fast ccache is your friend. Should most certainly be worth it for half a dozen kernel (re-)builds. b. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
> Yes, my feeling has always been that this is an unlink (or generically, > "rm") race as I only saw it when two processes were processing file > removals (with one being "rm -rf") in two different directories. > > Now what might be of relevance is that the two trees would have had > shared a high percentage of hard links. Maybe it's a race on deleting a > hard linked file? I can reproduce with a large number of files in a single directory. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
I'm starting a git bisect now. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
On Mon, 2009-03-30 at 16:56 +, yaztromo wrote: > Theodore, > > I set an rm -rf going of multiple copies of "The High Voltage SID > collection", which in total is around 130,000 small files. At most two > tries is usually enough to trigger the soft lock. > > I'm compiling RC1 now, thanks to Andrius' instructions. Will let you > know the result. Yes, my feeling has always been that this is an unlink (or generically, "rm") race as I only saw it when two processes were processing file removals (with one being "rm -rf") in two different directories. Now what might be of relevance is that the two trees would have had shared a high percentage of hard links. Maybe it's a race on deleting a hard linked file? b. -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 330824] Re: Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28
I've been able to reproduce this consistently on my desktop (2.5gb ram, a...@1.6ghz singlecore, 7200rpm drive) by writing half a meg to a couple thousand different files sequentially, dropping the cache, deleting them, and starting over. Usually the machine hardlocks partway into the second cycle. Under 2.6.29, the test completes fine with no intermittent hanging or otherwise. I haven't tried any other kernels yet. My laptop (1gb ram, in...@1.6ghz, 5400rpm drive) hangs intermittently on the same workload, but doesn't hardlock consistently. ** Attachment added: "hang.py" http://launchpadlibrarian.net/24530324/hang.py -- Soft lockups (freezes) when deleting files from ext4 partitions on 2.6.28 https://bugs.launchpad.net/bugs/330824 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
