Automatic fsck
On Mon, Aug 11, 2008 at 02:09:19PM -0700, Bryce Harrington wrote: > On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: > > == Filesystem checking / AutoFsck == > > > > A suggestion was made to the technical board that Ubuntu could be smarter > > about how and when it performs filesystem integrity checks (fsck). > > > > Decision: This should be discussed more widely in the developer community > > Action: Scott to start a thread on ubuntu-devel/-discuss > > I find the autofsck to be most notable on my laptop, perhaps because I > reboot it more frequently, and because it usually chooses to autofsck at > some inopportune time. I don't know if laptop harddrives need fsck more > than desktop's, but I wouldn't mind seeing the frequency be reduced for > laptops. > > Alternatively, maybe the autofsck could be made to take a few more > factors into account, such as total run time since last fsck, total > absolute time since last fsck, drive age, etc. Total run time sounds like an interesting one to watch. Some of the other ideas which have been proposed are: Run fsck during shutdown (when the user isn't expecting to be able to use the system for a while anyway) rather than at startup. https://blueprints.launchpad.net/ubuntu/+spec/prompt-for-fsck-on-shutdown Allow fsck to be easily skipped. This was implemented in 8.04 as part of https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish Skip fsck when running on battery. Also implemented in 8.04 as part of https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish -- - mdz -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Matt Zimmerman wrote: > On Mon, Aug 11, 2008 at 02:09:19PM -0700, Bryce Harrington wrote: >> On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: > > Some of the other ideas which have been proposed are: > > Run fsck during shutdown (when the user isn't expecting to be able to use > the system for a while anyway) rather than at startup. > https://blueprints.launchpad.net/ubuntu/+spec/prompt-for-fsck-on-shutdown > > Allow fsck to be easily skipped. This was implemented in 8.04 as part of > https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish > If fsck is to run during shutdown, then there definitely needs to be a means to easily skip it, or perhaps defer it to run at the next startup. Many people (like me :P) leave it till the very last minute at work before doing an "express shutdown" to dash out the door for the train. Having the laptop decide that it needs to do a disk scan at that point will not be popular...! -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
ti, 2008-08-12 kello 20:57 +1000, Ian Chennell kirjoitti: > If fsck is to run during shutdown, then there definitely needs to be a > means to easily skip it, or perhaps defer it to run at the next startup. > Many people (like me :P) leave it till the very last minute at work > before doing an "express shutdown" to dash out the door for the train. > Having the laptop decide that it needs to do a disk scan at that point > will not be popular...! I have the same feeling. I often shut down my desktop and file servers at home over night, and I wouldn't want the file server to stay on for several hours doing fsck, while I'm trying to get some sleep. (I also don't like it taking hours to boot up, when fsck-on-boot strikes.) It seems there is not good to run fsck either during boot or during shutdown. As it happens, I made a little experiment to run fsck with -n (supported by ext2/ext3 mainly) while the system was running, to avoid having to run it during boot at all. This would be good also for long-running servers: they could run fsck from cron. Unfortunately, e2fsck does not handle very well the situation of the filesystem changing from underneath it, which will happen when running with -n on a mounted filesystem. A way to avoid that would be to set up systems with LVM, and use an LVM snapshot volume for running fsck. This would give fsck a frozen snapshot of the system, and should work better. However, it requires some free space to be used, and I haven't actually tried it yet. Reserving some disk space just for this probably isn't going to be all that popular, either. However, for systems on which it would be acceptable, it might be worthwhile to investigate this. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 12, 2008 at 08:57:36PM +1000, Ian Chennell wrote: > > > Matt Zimmerman wrote: > > On Mon, Aug 11, 2008 at 02:09:19PM -0700, Bryce Harrington wrote: > >> On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: > > > > > Some of the other ideas which have been proposed are: > > > > Run fsck during shutdown (when the user isn't expecting to be able to use > > the system for a while anyway) rather than at startup. > > https://blueprints.launchpad.net/ubuntu/+spec/prompt-for-fsck-on-shutdown > > > > Allow fsck to be easily skipped. This was implemented in 8.04 as part of > > https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish > > > > > If fsck is to run during shutdown, then there definitely needs to be a > means to easily skip it, or perhaps defer it to run at the next startup. > Many people (like me :P) leave it till the very last minute at work > before doing an "express shutdown" to dash out the door for the train. > Having the laptop decide that it needs to do a disk scan at that point > will not be popular...! See above; it's already possible to skip it. -- - mdz -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
2008/8/12 Lars Wirzenius <[EMAIL PROTECTED]>: > A way to avoid that would be to set up systems with LVM, and use an LVM > snapshot volume for running fsck. This would give fsck a frozen snapshot > of the system, and should work better. However, it requires some free > space to be used, and I haven't actually tried it yet. Reserving some > disk space just for this probably isn't going to be all that popular, > either. However, for systems on which it would be acceptable, it might > be worthwhile to investigate this. And then what happens when it fails the check? Sounds pretty messy to me. I say we look into fixing e2fsck to do online consistency checking without borking over changing filesystem contents. Don't other OS/FS combos do this well? -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 12, 2008 at 02:59:22PM +0100, Alexander Jones wrote: > 2008/8/12 Lars Wirzenius <[EMAIL PROTECTED]>: > > A way to avoid that would be to set up systems with LVM, and use an LVM > > snapshot volume for running fsck. This would give fsck a frozen snapshot > > of the system, and should work better. However, it requires some free > > space to be used, and I haven't actually tried it yet. Reserving some > > disk space just for this probably isn't going to be all that popular, > > either. However, for systems on which it would be acceptable, it might > > be worthwhile to investigate this. > > And then what happens when it fails the check? Sounds pretty messy to me. Indeed. The best we could do in a scenario like this would be to flag the filesystem dirty so that it gets checked the next time it's possible. > I say we look into fixing e2fsck to do online consistency checking > without borking over changing filesystem contents. Don't other OS/FS > combos do this well? This requires the cooperation of the kernel, and I don't think this exists in ext3. -- - mdz -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
I've been bitten bad by e2fsck where it's borked my system such that I've had to reinstall. Since I don't want to be forced into that again, I'm trying to disable it permanently and take my chances on losing a file here or there. However, I can't seem to shut it off. So far, I've used tune2fs -c0 -i 0 to shut it off (interval and count are disabled .. see confirmation below). That didn't work. Next I used dpkg-divert to move e2fsck out of my path and rebuild the initramfs with update-initramfs, but it still runs on boot (I can't figure out how it got a copy of e2fsck). I have to use Esc to abort it. What's the trick to disable it on hardy?? paul :~$ sudo tune2fs -l /dev/mapper/vgsda12-lvrootsda12 tune2fs 1.40.8 (13-Mar-2008) Filesystem volume name: Last mounted on: Filesystem UUID: 8bbf135a-ecb3-4c7d-b9a6-d96fd953b657 Filesystem magic number: 0xEF53 Filesystem revision #:1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: signed_directory_hash Default mount options:(none) Filesystem state: clean with errors Errors behavior: Continue Filesystem OS type: Linux Inode count: 892928 Block count: 3568640 Reserved block count: 178432 Free blocks: 792705 Free inodes: 602839 First block: 0 Block size: 4096 Fragment size:4096 Reserved GDT blocks: 871 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 256 Filesystem created: Sat Jul 19 19:29:03 2008 Last mount time: Sun Aug 10 14:41:31 2008 Last write time: Sun Aug 10 14:41:31 2008 Mount count: 29 Maximum mount count: -1 Last checked: Wed Jul 30 19:53:55 2008 Check interval: 0 () Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode:8 First orphan inode: 869173 Default directory hash: tea Directory Hash Seed: 74af1ac9-4826-4a95-a594-b99f15b0a883 Journal backup: inode blocks paul :~$ dpkg-divert --list | grep e2fs local diversion of /sbin/e2fsck to /diverted/sbin/e2fsck -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 12, 2008 at 10:07 AM, Paul S <[EMAIL PROTECTED]> wrote: > I've been bitten bad by e2fsck where it's borked my system such that > I've had to reinstall. Since I don't want to be forced into that again, > I'm trying to disable it permanently and take my chances on losing a > file here or there. I was under the impression that it's less likely to be "a file here or there" and much more likely to be "corrupted partition" which would lead to having to reinstall in many cases, but that fsck'ing was there to fix problems before they boil all the way up to corrupting the partition table. I'm not sure what kind of problems a fsck could cause, but I know what it can fix. It fixed "system can't boot because Vista shifted Ubuntu's inodes to all sorts of wrongness" a few times. I do not recommend a Linux/Vista dual boot. -- Mackenzie Morgan Linux User #432169 ACM Member #3445683 http://ubuntulinuxtipstricks.blogspot.com <-my blog of Ubuntu stuff apt-get moo -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
2008/8/12 Mackenzie Morgan <[EMAIL PROTECTED]>: > On Tue, Aug 12, 2008 at 10:07 AM, Paul S <[EMAIL PROTECTED]> wrote: >> I've been bitten bad by e2fsck where it's borked my system such that >> I've had to reinstall. Since I don't want to be forced into that again, >> I'm trying to disable it permanently and take my chances on losing a >> file here or there. > > I was under the impression that it's less likely to be "a file here or > there" and much more likely to be "corrupted partition" which would > lead to having to reinstall in many cases, but that fsck'ing was there > to fix problems before they boil all the way up to corrupting the > partition table. I'm not sure what kind of problems a fsck could > cause, but I know what it can fix. It fixed "system can't boot > because Vista shifted Ubuntu's inodes to all sorts of wrongness" a few > times. I do not recommend a Linux/Vista dual boot. There is something /very/ wrong with Linux if a filesystem state can trigger something to break the MBR. I dare say, in fact, that there is no way a filesystem problem can corrupt the partition table. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 12, 2008 at 10:29 AM, Alexander Jones <[EMAIL PROTECTED]> wrote: > 2008/8/12 Mackenzie Morgan <[EMAIL PROTECTED]>: >> On Tue, Aug 12, 2008 at 10:07 AM, Paul S <[EMAIL PROTECTED]> wrote: >>> I've been bitten bad by e2fsck where it's borked my system such that >>> I've had to reinstall. Since I don't want to be forced into that again, >>> I'm trying to disable it permanently and take my chances on losing a >>> file here or there. >> >> I was under the impression that it's less likely to be "a file here or >> there" and much more likely to be "corrupted partition" which would >> lead to having to reinstall in many cases, but that fsck'ing was there >> to fix problems before they boil all the way up to corrupting the >> partition table. I'm not sure what kind of problems a fsck could >> cause, but I know what it can fix. It fixed "system can't boot >> because Vista shifted Ubuntu's inodes to all sorts of wrongness" a few >> times. I do not recommend a Linux/Vista dual boot. > > There is something /very/ wrong with Linux if a filesystem state can > trigger something to break the MBR. I dare say, in fact, that there is > no way a filesystem problem can corrupt the partition table. OK, I'm a worry-wart. -- Mackenzie Morgan Linux User #432169 ACM Member #3445683 http://ubuntulinuxtipstricks.blogspot.com <-my blog of Ubuntu stuff apt-get moo -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
ti, 2008-08-12 kello 15:07 +0100, Matt Zimmerman kirjoitti: > On Tue, Aug 12, 2008 at 02:59:22PM +0100, Alexander Jones wrote: > > 2008/8/12 Lars Wirzenius <[EMAIL PROTECTED]>: > > > A way to avoid that would be to set up systems with LVM, and use an LVM > > > snapshot volume for running fsck. This would give fsck a frozen snapshot > > > of the system, and should work better. However, it requires some free > > > space to be used, and I haven't actually tried it yet. Reserving some > > > disk space just for this probably isn't going to be all that popular, > > > either. However, for systems on which it would be acceptable, it might > > > be worthwhile to investigate this. > > > > And then what happens when it fails the check? Sounds pretty messy to me. > > Indeed. The best we could do in a scenario like this would be to flag the > filesystem dirty so that it gets checked the next time it's possible. I assume you mean the next time the system is rebooted. That might be a long time in the future: servers especially might run for months without a reboot. Even laptops might not reboot until there's a kernel security update that affects them. If the filesystem really is corrupted, it would be best to deal with it as soon as possible, before (more) data is lost. I'd rather notify the server admin via e-mail (which is the standard way of doing such things for servers), and the desktop user via a notification area alert of some kind, triggered in some suitable way. > > I say we look into fixing e2fsck to do online consistency checking > > without borking over changing filesystem contents. Don't other OS/FS > > combos do this well? > > This requires the cooperation of the kernel, and I don't think this exists > in ext3. An ext[234] solution would also be specific to that filesystem. An LVM solution would be generic for all filesystems. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 12, 2008 at 06:17:36PM +0300, Lars Wirzenius wrote: > ti, 2008-08-12 kello 15:07 +0100, Matt Zimmerman kirjoitti: > > Indeed. The best we could do in a scenario like this would be to flag the > > filesystem dirty so that it gets checked the next time it's possible. > > I assume you mean the next time the system is rebooted. That might be a > long time in the future: servers especially might run for months without > a reboot. Even laptops might not reboot until there's a kernel security > update that affects them. If the filesystem really is corrupted, it > would be best to deal with it as soon as possible, before (more) data is > lost. There isn't any reasonable way to deal with this synchronously. > I'd rather notify the server admin via e-mail (which is the standard way > of doing such things for servers), and the desktop user via a > notification area alert of some kind, triggered in some suitable way. That could be done in addition to marking it dirty. > > > I say we look into fixing e2fsck to do online consistency checking > > > without borking over changing filesystem contents. Don't other OS/FS > > > combos do this well? > > > > This requires the cooperation of the kernel, and I don't think this exists > > in ext3. > > An ext[234] solution would also be specific to that filesystem. An LVM > solution would be generic for all filesystems. The LVM solution isn't viable anyway; there's no guarantee that the metadata on disk is in any way consistent while the filesystem is mounted. The problem in your test isn't only that the filesystem is changing from underneath it, it's also that it may not have been consistent in the first place. -- - mdz -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Matt Zimmerman wrote: > The LVM solution isn't viable anyway; there's no guarantee that the metadata > on disk is in any way consistent while the filesystem is mounted. The > problem in your test isn't only that the filesystem is changing from > underneath it, it's also that it may not have been consistent in the first > place. How about making a snapshot when the system boots up (before it's mounted), then fsck'ing the snapshot? I don't know enough about LVM to say whether snapshots can be written back to the main partition, but if so, it would be possible to fix the broken partition from the GUI, then reboot (or just remount the filesystem). The downside of that is that any recent changes would be lost, but the upside is that you'll never need to sit and watch fsck run. - Andrew -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Apologies for replying to myself, but I remembered that Google exists :) Apparently snapshot merging is currently in beta: http://kerneltrap.org/Linux/LVM_Snapshot_Merging Also apparently, LVM2 ensures filesystem integrity when creating snapshots: http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html I assume that the equivalent of "umount $snapshot" is done within the kernel when the snapshot is created, because it gives you a new non-mounted block device. It's therefore possible to do fsck from cron. By way of a test, I fsck'd my mythtv partition while recording TV: # export VOLGROUP=nautilus # name of my volume group # export VOLUME=mythtv # name of my MythTV volume # export BACKUP=$(lvcreate -L1024M -s /dev/$VOLGROUP/$VOLUME \ | cut -d\" -f2) # if e2fsck -f "/dev/$VOLGROUP/$BACKUP" ; \ then tune2fs -C 0 "/dev/$VOLGROUP/$VOLUME" ; \ else echo "warn the user somehow" ; \ fi e2fsck 1.40.8 (13-Mar-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information mythtv: 1268/13107200 files (13.5% non-contiguous), 23812139/26214400 blocks tune2fs 1.40.8 (13-Mar-2008) Setting current mount count to 0 # lvremove -f "/dev/$VOLGROUP/$BACKUP" Logical volume "lvol0" successfully removed In other words, this worked fine for me. The partition was free from errors and the recording was unharmed. - Andrew -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 12, 2008 at 11:15:05AM +0100, Matt Zimmerman wrote: > On Mon, Aug 11, 2008 at 02:09:19PM -0700, Bryce Harrington wrote: > > On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: > > > == Filesystem checking / AutoFsck == > > > > > > A suggestion was made to the technical board that Ubuntu could be smarter > > > about how and when it performs filesystem integrity checks (fsck). > > > > > > Decision: This should be discussed more widely in the developer community > > > Action: Scott to start a thread on ubuntu-devel/-discuss > > > > I find the autofsck to be most notable on my laptop, perhaps because I > > reboot it more frequently, and because it usually chooses to autofsck at > > some inopportune time. I don't know if laptop harddrives need fsck more > > than desktop's, but I wouldn't mind seeing the frequency be reduced for > > laptops. > > > > Alternatively, maybe the autofsck could be made to take a few more > > factors into account, such as total run time since last fsck, total > > absolute time since last fsck, drive age, etc. > > Total run time sounds like an interesting one to watch. > > Some of the other ideas which have been proposed are: > > Run fsck during shutdown (when the user isn't expecting to be able to use > the system for a while anyway) rather than at startup. > https://blueprints.launchpad.net/ubuntu/+spec/prompt-for-fsck-on-shutdown > > Allow fsck to be easily skipped. This was implemented in 8.04 as part of > https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish > > Skip fsck when running on battery. Also implemented in 8.04 as part of > https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish More smaller partitions would be another option (with corresponding trade-offs). Or two partitions - one small one requiring frequent fsck's, and a large one that needs fsck'd only occasionally. Bryce -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Seeing it from the perspective of an common user, fsck is something whose function is unknown and therefore running it is senseless, so they might turn if off directly or skip it always which is not good. My idea: -> Mount the partition read-only -> Capture writes with something like AuFS/Union-FS -> run fsck in between -> When fsck finished, remount the partition as 'rw' and write the changes done in the memory back to the disk This would require a bit more RAM but there might be a other solution such as using a second partition as interim storage for the changed blocks. Greets Uwe -- I'm the one and only XhochY ;-) http://xhochy.org/en/ - my Blog http://xhochy.org/twittering - my Microblog http://schoorbs.xhochy.com/ - my selfmade room and resource booking system http://static.xhochy.com/keys/91B032A5.asc - my PGP key signature.asc Description: OpenPGP digital signature -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
>> On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: >> >>> == Filesystem checking / AutoFsck == >>> >>> A suggestion was made to the technical board that Ubuntu could be smarter >>> about how and when it performs filesystem integrity checks (fsck). >>> >>> Decision: This should be discussed more widely in the developer community >>> Action: Scott to start a thread on ubuntu-devel/-discuss >>> One thing that I have not seen in this discussion is the notion that fsck might be modified to run incrementally. Another that I did not see is the idea that fsck can be run using -n (though ReiserFS and minix aren't supported at the moment). If fsck is run in the background and a notification is sent to the user/administrator if corruption is found, then active intervention can be recommended. At present to me it seems that there are at least three different use cases with different expectations: 1. Ubuntu-desktop user with a desktop machine that is switched off regularly. 2. Ubuntu-desktop user with a laptop that is mostly suspended and is only rebooted on crash or kernel update. Complicated by the fact that a laptop may be running on a battery. 3. Ubuntu-server user with a server that is only rebooted on crash or kernel update. I see with some alarm discussion about reducing the frequency of running fsck. I'm running an ext3 laptop and I'm seeing quite regular corruptions that require an fsck run to fix. (It may be related to a particular kernel, but I've not yet got to the bottom of that.) Fundamentally, in my opinion, fsck is a housekeeping process that is required on a regular basis to ensure the sane state of a file-system, no matter which one you use, errors do happen, even if there are no bugs (ha!), we're talking about tiny magnetic fields affecting the information on a hard-drive - this problem is only going to get bigger with increased storage density. -- Onno Benschop Connected via Optus B3 at S31°54'06" - E115°50'39" (Yokine, WA) -- ()/)/)()..ASCII for Onno.. |>>?..EBCDIC for Onno.. --- -. -. --- ..Morse for Onno.. ITmaze - ABN: 56 178 057 063 - ph: 04 1219 - [EMAIL PROTECTED] -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Wed, Aug 13, 2008 at 07:41:18AM +0800, Onno Benschop wrote: > > >> On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: > >> > >>> == Filesystem checking / AutoFsck == > >>> > >>> A suggestion was made to the technical board that Ubuntu could be smarter > >>> about how and when it performs filesystem integrity checks (fsck). > >>> > >>> Decision: This should be discussed more widely in the developer community > >>> Action: Scott to start a thread on ubuntu-devel/-discuss > >>> > > One thing that I have not seen in this discussion is the notion that > fsck might be modified to run incrementally. That's an interesting idea, though I don't know enough about ext3 to comment on its feasibility. Perhaps something to discuss with upstream? > Another that I did not see is the idea that fsck can be run using -n > (though ReiserFS and minix aren't supported at the moment). If fsck is > run in the background and a notification is sent to the > user/administrator if corruption is found, then active intervention can > be recommended. It's easy to prevent fsck from changing the filesystem, but the trouble is that fsck can't be (usefully) run on a filesystem which is in use (mounted). > I see with some alarm discussion about reducing the frequency of running > fsck. I'm running an ext3 laptop and I'm seeing quite regular > corruptions that require an fsck run to fix. (It may be related to a > particular kernel, but I've not yet got to the bottom of that.) That is a very disconcerting (and atypical) problem. The appearance of errors in fsck indicates a serious problem which should be investigated, not a normal condition of wear and tear. > Fundamentally, in my opinion, fsck is a housekeeping process that is > required on a regular basis to ensure the sane state of a file-system, > no matter which one you use, errors do happen, even if there are no bugs > (ha!), we're talking about tiny magnetic fields affecting the > information on a hard-drive - this problem is only going to get bigger > with increased storage density. I don't think there's any disagreement on this point. The issues are: 1) The frequency at which fsck runs is somewhat arbitrary for many users, depending on their usage pattern, and doesn't relate particularly well to their risk of encountering errors. 2) The fsck runs themselves are very disruptive, blocking access to the computer when the user wants it. -- - mdz -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Andrew Sayers wrote: > I assume that the equivalent of "umount $snapshot" is done within the > kernel when the snapshot is created, because it gives you a new > non-mounted block device. It's therefore possible to do fsck from cron. The snapshot was never mounted in the first place, so there is no need to unmount it. As you mentioned before however, any files changed since the snapshot was made will be lost when you reboot and merge the snapshot back to the main volume. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
ke, 2008-08-13 kello 18:33 -0400, Phillip Susi kirjoitti: > Andrew Sayers wrote: > > > I assume that the equivalent of "umount $snapshot" is done within the > > kernel when the snapshot is created, because it gives you a new > > non-mounted block device. It's therefore possible to do fsck from cron. > > The snapshot was never mounted in the first place, so there is no need > to unmount it. Right. Just to be clear, the following would be a reasonably reasonable scenario for boot-time fsck, in situations in which LVM snapshots are available: * kernel mounts root filesystem read-only * init scripts make LVM copy-on-write snapshot of all filesystems * init scripts re-mount root filesystem read-write, mount any other filesystems * fsck starts on each snapshot, preferably with "ionice -c3" * once a snapshot has been checked, it is destroyed Since the fscks can take a long time, the results can't be reported at boot-time, and an alternative communication channel is needed. For servers, e-mail and syslog seem reasonable. Any problems found with the snapshots will _not_ be fixed in the real filesystem, only in the snapshot. The real filesystem needs to be fixed too. Like Matt suggested, marking the filesystem dirty (so a real fsck is run at next boot) seems like a reasonable way of doing that. For desktops, putting errors into /var/log/background-fsck/foo.log and having something in the GNOME/KDE sessions watch those files and pop up an error message should be feasible. Would that work? -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Phillip Susi wrote: > The snapshot was never mounted in the first place, so there is no need > to unmount it. > > As you mentioned before however, any files changed since the snapshot > was made will be lost when you reboot and merge the snapshot back to the > main volume. > Either I'm not making myself clear or my lack of kernel mojo is showing. The test I did previously was on a running filesystem, while it was in use - not during boot-up as we'd previously discussed. It seems to me that a literal snapshot, capturing a random moment in the life of a filesystem, would be treated by e2fsck as not having been cleanly unmounted. Since e2fsck didn't complain about anything like that, some part of the snapshotting process must cleanly unmount the filesystem. This means that one can take a snapshot while the system is running, fsck the snapshot, and (if the snapshot is error-free) conclude that the main volume doesn't need an automatic check for another few months/reboots. So building on Lars' idea, I propose: At boot-time: 1. kernel mounts root filesystem read-only 2. init scripts check which filesystems have passed their max mount count/interval 2a. init scripts snapshot those filesystems 2b. the main volumes for those systems are mounted (without being checked) 3. init scripts continue as normal 4. fsck starts on each snapshot, preferably with "ionice -c3" 5. if a snapshot is found to be clean, 5a. the main volume has its mount-count/check-time reset 5b. the snapshot is destroyed 5c. the user is /not/ informed 6. if a snapshot is found to have problems, 6a. the main volume is marked to be fsck'd 6b. the snapshot is destroyed 6c. the user is asked to reboot In /etc/cron.weekly/fsck: 1. pick a device in /dev/mapper (i.e. only check one device per week) 2. snapshot that volume 3. continue from (4), above Since merging is currently in beta, and probably a daft idea in this context, it's better to roll out something more practical, and think about being more audacious next time. Remembering my aforementioned lack of kernel mojo, the biggest problems I can see with this approach are that it requires Ubiquity to do LVM by default and to keep a significant chunk of the drive free for snapshots (off the top of my head, I'd say 1-5% of total disk space). - Andrew -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
PLEASE redirect your efforts towards online fscking. This whole idea is absolutely horrible. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Alexander Jones wrote: > PLEASE redirect your efforts towards online fscking. This whole idea > is absolutely horrible. How so? - Andrew -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Because people are talking about snapshotting a FS in a potentially broken state, fscking it in the background---whilst continuing to use it! Assuming that using a broken FS doesnt hose it (admittedly it shouldn't), merging a changeset from a broken state into a repaired state is a process which I think will be extremely fragile and prone to breakage, /especially/ if attempted FS-agnostically. I willfully invite people to prove me wrong, just tell me how to disable it before it eats my data. Or limit your scope a bit and thrash this out properly, for a single filesystem (hint, Ext3!), on LKML. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Alexander Jones wrote: > Because people are talking about snapshotting a FS in a potentially > broken state, fscking it in the background---whilst continuing to use > it! > > Assuming that using a broken FS doesnt hose it (admittedly it > shouldn't), merging a changeset from a broken state into a repaired > state is a process which I think will be extremely fragile and prone > to breakage, /especially/ if attempted FS-agnostically. > > I willfully invite people to prove me wrong, just tell me how to > disable it before it eats my data. Or limit your scope a bit and > thrash this out properly, for a single filesystem (hint, Ext3!), on > LKML. > Right, I understand your objection now. Some points I'd like to make: As I said before, merging back is at most something to consider in version 2. For now it's enough to use the snapshot in read-only mode to confirm whether the FS actually needs to be fixed. I'm only suggesting this be used in place of the regular boot-time fsck's that hardly ever turn up anything of interest. IMHO, when there's any significant probability of breakage, fsck's should be done before the disks are mounted - not in a snapshot, not online. As to the dangers of a broken FS, consider the following: disks are currently fsck'd once every few mounts/months. In order for that regular check to find a problem, you would have to be use a broken FS for anything up to that many mounts/months without noticing. If a cron job checks your disks twice as often (without modifying them), and triggers an old-fashioned boot-time fsck at the user's earliest convenience, that only serves to augment the current system by reducing the amount of time that you're using a broken FS. That said, I'd like to amend my previous proposal slightly based on your objection: 6b. the snapshot is marked to be destroyed _only after_ a successful fsck of the main volume. In the event that the main volume is too badly damaged, the snapshot might aid in recovery. If you didn't go out of your way to manage your disks with LVM when you installed Ubuntu, then this solution would be automatically disabled for you. Snapshotting requires LVM, which is currently only an option if you install from the alternate CD. I'm not aware of any mechanism to upgrade a normal partition table to LVM, so this solution would only work for new installs and systems that already use LVM. - Andrew -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
I've been trying out a script (attached) for the last few days, that
does something similar to the idea in my previous comment. It's a shell
script that can be put in cron.daily and/or called from an @reboot cron
job. The script checks each of your LVM-based filesystems in turn, and
won't start a new check if it's been going for more than 10 minutes.
The short version of the story is that fsck'ing a snapshot of a live
filesystem is possible, but we might want to get at least a little input
from LVM or FS developers first.
The main problem with this script is that it trips over on temporary
files. It's common for programs (via mkstemp(), I think) to create a
temporary file, open it, then delete it. The inode that was previously
associated with the file continues to exist so long as a file descriptor
to it remains open, but when a snapshot of the filesystem is created,
the inodes are never removed, so they become orphans. fsck notices this
minor problem in the snapshot and flags the filesystem as needing to be
checked.
Steps to repeat this problem:
$ sudo /etc/init.d/mysql start # creates temporary files on my system
$ sudo lvcreate -L1024M -s /dev/your-volgroup/your-root-device
$ sudo fsck -v -n -f /dev/your-volgroup/lvol0
$ sudo lvremove /dev/your-volgroup/lvol0
fsck should complain about orphaned files. I get this:
$ sudo fsck -v -n -f /dev/nautilus/lvol0
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 180229 has zero dtime. Fix? no
Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 180230 was part of the orphaned inode list. IGNORED.
Inode 180231 was part of the orphaned inode list. IGNORED.
Inode 180232 was part of the orphaned inode list. IGNORED.
Inode 180233 was part of the orphaned inode list. IGNORED.
Inode 180251 was part of the orphaned inode list. IGNORED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Inode bitmap differences: -(180229--180233) -180251
Fix? no
root: ** WARNING: Filesystem still has errors **
23381 inodes used (8.92%)
518 non-contiguous inodes (2.2%)
# of inodes with ind/dind/tind blocks: 2563/15/0
211424 blocks used (40.33%)
0 bad blocks
1 large file
13390 regular files
2902 directories
1258 character device files
4553 block device files
1 fifo
16 links
1216 symbolic links (1137 fast symbolic links)
46 sockets
23382 files
To my untrained eye, it looks like this could be argued to be a bug in
ext2 or LVM (because they're not deleting inodes properly), or a bug in
fsck (because it doesn't have an "errors remain, but who cares?" return
code). Alternatively, it could be argued that the fsck script I've
written should parse the output of fsck and decide which filesystem
errors are really important.
I've gone as far as I can go with this idea - if someone with more of a
clue is interested, could you suggest the best way of solving this issue?
- Andrew
#!/bin/sh
# Check filesystems without rebooting, using LVM
# Andrew Sayers, 14 August 2008
# [EMAIL PROTECTED]
#
# This script aims to be FS-agnostic, although it currently calls "tune2fs" in
# two places, to reset the mount-count and check-time.
# What to tell the user if an error occurs
TITLE="Filesystem problem detected"
MESSAGE="Your hard disk has a problem,
Please reboot your system to fix it"
check_filesystem() {
# (I think) LVM escapes dashes in volume names by doubling them (--)
# The following gets the volume group, even if it has --s in it
export VOLDEV="$1"
export VOLGROUP=$(echo "$VOLDEV" | sed -e
's/^\(\(\(\([^-]*\)--\)*\)[^-]*\)-\([^-].*\)/\1/' -e 's/--/-/g') \
export VOLUME=$( echo "$VOLDEV" | sed -e
's/^\(\(\(\([^-]*\)--\)*\)[^-]*\)-\([^-].*\)/\5/' -e 's/--/-/g')
export BACKUP=$(lvcreate -L1024M -s "/dev/$VOLGROUP/$VOLUME" | cut -d\"
-f2)
if ERRORS=$(fsck -v -n -f "/dev/$VOLGROUP/$BACKUP" 2>&1)
then
tune2fs -T now -C 0 "/dev/mapper/$VOLDEV" >/dev/null
lvremove -f "/dev/$VOLGROUP/$BACKUP" >/dev/null
# Note: in the success case, success isn't reported until after
tune2fs has completed
# (in case tune2fs fails)
touch "/var/cache/fsck/$VOLDEV"
logger -p cron.info "snapshot fsck of
\"/dev/$VOLGROUP/$VOLUME\" reported a healthy filesystem"
else
RETURN_VALUE=$?
# TODO: check whether $BACKUP has gone away (due to too much FS
activity), and handle that somehow
# TODO: write a co-operating GUI app to handle messages
something like:
# notify-send -u critical -t 6000 --category=device.error
"$TITLE" "$MESSAGE"
# TODO: automatically remove $BACKUP
Re: Automatic fsck
On Tue, 2008-08-12 at 11:15 +0100, Matt Zimmerman wrote: > On Mon, Aug 11, 2008 at 02:09:19PM -0700, Bryce Harrington wrote: > > On Mon, Aug 11, 2008 at 11:52:25AM +0100, Matt Zimmerman wrote: > > > == Filesystem checking / AutoFsck == > > > > > > A suggestion was made to the technical board that Ubuntu could be smarter > > > about how and when it performs filesystem integrity checks (fsck). > > > > > > Decision: This should be discussed more widely in the developer community > > > Action: Scott to start a thread on ubuntu-devel/-discuss > > > > I find the autofsck to be most notable on my laptop, perhaps because I > > reboot it more frequently, and because it usually chooses to autofsck at > > some inopportune time. I don't know if laptop harddrives need fsck more > > than desktop's, but I wouldn't mind seeing the frequency be reduced for > > laptops. > > > > Alternatively, maybe the autofsck could be made to take a few more > > factors into account, such as total run time since last fsck, total > > absolute time since last fsck, drive age, etc. > > Total run time sounds like an interesting one to watch. I suspect fsck times are going to be less of an issue for systems using ext4 in the (hopefully not so distant) future - ext4 is significantly faster to fsck than ext3 (from my understanding, ext3 fsck times are linearly proportional to the number of inodes, ext4 doesn't seem to be). See the table in http://thunk.org/tytso/blog/2008/08/08/fast-ext4-fsck-times/ Mind you, there are some large disks out there nowadays... Colin K. > > Some of the other ideas which have been proposed are: > > Run fsck during shutdown (when the user isn't expecting to be able to use > the system for a while anyway) rather than at startup. > https://blueprints.launchpad.net/ubuntu/+spec/prompt-for-fsck-on-shutdown > > Allow fsck to be easily skipped. This was implemented in 8.04 as part of > https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish > > Skip fsck when running on battery. Also implemented in 8.04 as part of > https://blueprints.launchpad.net/ubuntu/+spec/usplash-polish > > -- > - mdz > -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Although this might start a small side-track of the main subject, I would like to add that it would be cool if the new solution could also incorporate checking of usb mass storage devices. Portable hard disks are becoming more popular, formatting them with ext is becoming more popular, but there doesn't seem to be a good plan to check them. I hope the eventual solution will also be applicable to portable storage devices. Bram -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Alexander Jones wrote: > Because people are talking about snapshotting a FS in a potentially > broken state, fscking it in the background---whilst continuing to use > it! I've been thinking something like this as well. The whole point of fsck is to find and repair damage BEFORE your attempts to write to the filesystem cause further damage. It kind of defeats the purpose if you continue manipulated the damaged volume while you check it. > Assuming that using a broken FS doesnt hose it (admittedly it > shouldn't), merging a changeset from a broken state into a repaired > state is a process which I think will be extremely fragile and prone > to breakage, /especially/ if attempted FS-agnostically. Merging does not mean combining the changes from both copies, it means throwing out the changes in the read-write copy and replacing it with the repaired read-only copy. The downside to this of course, is that any changes you made while the fsck was running are lost. I don't think this is acceptable either. I think the better solution is to fsck on shutdown ( if it must be done at all -- I still say an automatic fsck when no problems have been detected should just be disabled entirely ). It is simpler to implement since it does not require LVM, and doesn't have the above problems. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
Matt Zimmerman wrote: >> One thing that I have not seen in this discussion is the notion that >> fsck might be modified to run incrementally. > > That's an interesting idea, though I don't know enough about ext3 to comment > on its feasibility. Perhaps something to discuss with upstream? Not possible since things that have already been checked and since modified effect the things that have not yet been checked. Anyone remember scandisk in windows 95? Sure, it could run on a mounted disk, if you weren't writing to it at all. As soon as the disk was written to in any way, it had to start over. >> I see with some alarm discussion about reducing the frequency of running >> fsck. I'm running an ext3 laptop and I'm seeing quite regular >> corruptions that require an fsck run to fix. (It may be related to a >> particular kernel, but I've not yet got to the bottom of that.) > > That is a very disconcerting (and atypical) problem. The appearance of > errors in fsck indicates a serious problem which should be investigated, not > a normal condition of wear and tear. Indeed. >> Fundamentally, in my opinion, fsck is a housekeeping process that is >> required on a regular basis to ensure the sane state of a file-system, >> no matter which one you use, errors do happen, even if there are no bugs >> (ha!), we're talking about tiny magnetic fields affecting the >> information on a hard-drive - this problem is only going to get bigger >> with increased storage density. > > I don't think there's any disagreement on this point. The issues are: As you said above, the appearance of errors in fsck is NOT a normal wear and tear condition, so periodic fscks are NOT a housekeeping process. Errors do not creep into filesystems unless something has gone drastically wrong -- like a bug in the kernel overwrote part of the disk with random garbage. Hard disks are also remarkably reliable these days and their internal ECC will detect any errors in the media and either correct them, or return an error when you try to read that sector ( which will cause an emergency remount-ro and a fsck on next boot ). You don't just get a few random bits flipping that you won't notice unless you run a fsck. Reiserfs does not bother doing periodic checks and never seems to suffer for it. I always disable them on ext3 partitions, and used to do so on ext2 partitions throughout the 90s. Fsck still ran when it actually needed to -- after a clean unmount. Just to be sure, I'd still run a manual fsck every once in a while, especially before a defrag, but it never found problems, and that was before journaling. > 1) The frequency at which fsck runs is somewhat arbitrary for many users, > depending on their usage pattern, and doesn't relate particularly well to > their risk of encountering errors. > > 2) The fsck runs themselves are very disruptive, blocking access to the > computer when the user wants it. I think both of these points are best addressed with a simple, non obnoxious prompt on shutdown to run a fsck before shutting down, and also giving the option to put it off until tomorrow, next week, or never ask me again. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 19, 2008 at 12:27 PM, Phillip Susi <[EMAIL PROTECTED]> wrote: > I think both of these points are best addressed with a simple, non > obnoxious prompt on shutdown to run a fsck before shutting down, and > also giving the option to put it off until tomorrow, next week, or never > ask me again. > I like this. If this is implemented, there needs to be a timer (maybe 30 seconds) that does a normal shutdown without fsck when it times out. If the user closes their laptop before the prompt shows up, they don't want their computer on and draining batteries waiting for input. Evan -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Automatic fsck
On Tue, Aug 19, 2008 at 5:23 PM, Evan <[EMAIL PROTECTED]> wrote: > On Tue, Aug 19, 2008 at 12:27 PM, Phillip Susi <[EMAIL PROTECTED]> wrote: >> >> I think both of these points are best addressed with a simple, non >> obnoxious prompt on shutdown to run a fsck before shutting down, and >> also giving the option to put it off until tomorrow, next week, or never >> ask me again. > > I like this. If this is implemented, there needs to be a timer (maybe 30 > seconds) that does a normal shutdown without fsck when it times out. If the > user closes their laptop before the prompt shows up, they don't want their > computer on and draining batteries waiting for input. That's how AutoFsck behaved (minus the "next week, never" part). It had a timeout to normal shutdown, and it put it off til tomorrow/next time if that was reached. -- Mackenzie Morgan Linux User #432169 ACM Member #3445683 http://ubuntulinuxtipstricks.blogspot.com <-my blog of Ubuntu stuff apt-get moo -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
