Re: SU+J Lost files after a power failure
David Demelier wrote: Hello there, I'm writing because after a power failure I was unable to log in on my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some files disappeared, including /etc/pwd.db. Thus I was unable to log in. I've been able to regenerate the password database with a live cd but I'm afraid that more files had disappeared somewhere else... I think this is a serious issue, the journal should not truncate files, so something should have gone wrong somewhere.. Any ideas? Should I open a PR? Not sure there is enough to go on for a PR, but something is weird. Friday morning our power went down at home for about three hours after I had already left for work. When I came home I found the router/gateway box was OK. It is still with the old DOS mbr and disklabel scheme, with softupdates, and is a pair of disks gmirrored. The other box is my first foray into the land of GPT, along with SU+J. It was sitting at the 'couldn't mount... Press return for /bin/sh' line. There was an error indicating that replaying one or more journals had failed. I was able to successfully fsck all the other partitions (besides /), then rebooted and system came back up OK. Both of these machines were recently updated to 9.2 Release from 9.1. It has been approximately 9 months, or so, since I last had a power outage like this one. Back then they were still 8.3 I think, did not have SU+J and recovered just fine on their own. This error about the replay of the journal(s) failing is somewhat disconcerting. Beyond that, however, I do not have any other details or data. Nothing to flesh out a PR, but thought I'd mention what I saw in conjunction with your experience. -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
Michael Powell wrote: [snip] The other box is my first foray into the land of GPT, along with SU+J. It was sitting at the 'couldn't mount... Press return for /bin/sh' line. There was an error indicating that replaying one or more journals had failed. I was able to successfully fsck all the other partitions (besides /), then rebooted and system came back up OK. Meant to include also that I booted from a CD with wddiags and ran the Quick test and it found no errors on the disk. [snip] -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, 14 Oct 2013 05:02:22 -0400 Michael Powell wrote: David Demelier wrote: Hello there, I'm writing because after a power failure I was unable to log in on my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some files disappeared, including /etc/pwd.db. Thus I was unable to log in. I've been able to regenerate the password database with a live cd but I'm afraid that more files had disappeared somewhere else... I think this is a serious issue, the journal should not truncate files, so something should have gone wrong somewhere.. The journalling in SU+J has nothing to do with data integrity. When the system isn't shut-down cleanly, soft-updates are supposed to leave the filesystem in a self-consistent state, except that it may lose track of some freed disk space. The journal allows that space to be recovered without the lengthy background fsck that used to cripple performance. If you are having problems with data integrity you might try gjournal or zfs instead. If you look back at the lists before these were added there was a lot of suspicion about soft-updates and background checks. Some of the problems were explained by some (mostly desktop) drives incorrecty reporting what has been commited to disk - I don't know whether this is still the case. This error about the replay of the journal(s) failing is somewhat disconcerting. I think this is probably a good thing. With background checks you would (if you were looking) occasionally see unexpected soft-update inconsistency during the background check, which would lead to a foreground check on the next boot. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 14.10.2013 14:39, RW wrote: On Mon, 14 Oct 2013 05:02:22 -0400 Michael Powell wrote: David Demelier wrote: Hello there, I'm writing because after a power failure I was unable to log in on my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some files disappeared, including /etc/pwd.db. Thus I was unable to log in. I've been able to regenerate the password database with a live cd but I'm afraid that more files had disappeared somewhere else... I think this is a serious issue, the journal should not truncate files, so something should have gone wrong somewhere.. The journalling in SU+J has nothing to do with data integrity. When the system isn't shut-down cleanly, soft-updates are supposed to leave the filesystem in a self-consistent state, except that it may lose track of some freed disk space. The journal allows that space to be recovered without the lengthy background fsck that used to cripple performance. If you are having problems with data integrity you might try gjournal or zfs instead. Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? On GNU/Linux, on Windows you will not require anything else to recover your data. I don't want to tweak the filesystem or use something different that the default, as it is the default it's the *warranty* that it is the correct way to protect data for new FreeBSD user's installations IMHO. If you look back at the lists before these were added there was a lot of suspicion about soft-updates and background checks. Some of the problems were explained by some (mostly desktop) drives incorrecty reporting what has been commited to disk - I don't know whether this is still the case. This error about the replay of the journal(s) failing is somewhat disconcerting. I think this is probably a good thing. With background checks you would (if you were looking) occasionally see unexpected soft-update inconsistency during the background check, which would lead to a foreground check on the next boot. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 6:34 PM, David Demelier demelier.da...@gmail.com wrote: Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? On GNU/Linux, on Windows you will not require anything else to recover your data. I don't want to tweak the filesystem or use something different that the default, as it is the default it's the *warranty* that it is the correct way to protect data for new FreeBSD user's installations IMHO. Agree :-) SU+J also seems to cause problems on SSD drives: http://lists.freebsd.org/pipermail/freebsd-fs/2013-February/016420.html -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 11:34 AM, David Demelier demelier.da...@gmail.comwrote: Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? As already stated, those measures are to preserve fs integrity eg meta data is in sync. It doesn't ensure that all the outstanding writes are committed to disk in the event of a power outage. On GNU/Linux, on Windows you will not require anything else to recover your data. This is complete garbage when using default settings as you imply below. The default for ext3 on basically every distro still using ext3 is an ordered journal and don't even get started on ext4. NTFS by default can/will also lose data on a power outage. I don't want to tweak the filesystem or use something different that the default, as it is the default it's the *warranty* that it is the correct way to protect data for new FreeBSD user's installations IMHO. There is no *warranty* as explicitly stated in http://www.freebsd.org/copyright/freebsd-license.html The behavior you wish would slow down disk writes by an order of magnitude and is already available to users willing to use non-default settings. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 6:47 PM, Adam Vande More amvandem...@gmail.com wrote: On Mon, Oct 14, 2013 at 11:34 AM, David Demelier demelier.da...@gmail.comwrote: Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? As already stated, those measures are to preserve fs integrity eg meta data is in sync. It doesn't ensure that all the outstanding writes are committed to disk in the event of a power outage. Then why random files gets damaged as well even they are not accessed/written on power loss? :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 11:50 AM, CeDeROM cede...@tlen.pl wrote: Then why random files gets damaged as well even they are not accessed/written on power loss? :-) Prove they weren't. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 6:56 PM, Adam Vande More amvandem...@gmail.com wrote: On Mon, Oct 14, 2013 at 11:50 AM, CeDeROM cede...@tlen.pl wrote: Then why random files gets damaged as well even they are not accessed/written on power loss? :-) Prove they weren't. Hmm, maybe /etc/pwd.db as David mentioned? This is updated on password change, which does not happen all the time.. so why it was damaged when no write occured..? -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 10/14/2013 12:50 PM, CeDeROM wrote: On Mon, Oct 14, 2013 at 6:47 PM, Adam Vande More amvandem...@gmail.com wrote: On Mon, Oct 14, 2013 at 11:34 AM, David Demelier demelier.da...@gmail.comwrote: Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? As already stated, those measures are to preserve fs integrity eg meta data is in sync. It doesn't ensure that all the outstanding writes are committed to disk in the event of a power outage. Then why random files gets damaged as well even they are not accessed/written on power loss? :-) Random files can be affected because the sectors of the hard disk containing the directory entries for those files, not the file data itself, may be damaged (ie: the directory was in the process of being written OR the pointer to that SECTOR was in the process of being written). It doesn't mean a file was in active use, just that a chunk of the disk with data relevant to that file was. Keep in mind, one sector of disk may have data for a dozen files in it (or more). Damage doesn't have to occur because a given file was in use at the time of a crash. If your power grid is prone to failures or blips, I strongly suggest investing in a UPS. Brad ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 7:09 PM, Brad Mettee bmet...@pchotshots.com wrote: On 10/14/2013 12:50 PM, CeDeROM wrote: Then why random files gets damaged as well even they are not accessed/written on power loss? :-) Random files can be affected because the sectors of the hard disk containing the directory entries for those files, not the file data itself, may be damaged (ie: the directory was in the process of being written OR the pointer to that SECTOR was in the process of being written). It doesn't mean a file was in active use, just that a chunk of the disk with data relevant to that file was. Keep in mind, one sector of disk may have data for a dozen files in it (or more). Damage doesn't have to occur because a given file was in use at the time of a crash. Isn't there Journal to prevent and reverse such damage? If your power grid is prone to failures or blips, I strongly suggest investing in a UPS. I have UPS in my desktop and also I am working on a laptop, so the power supply is not the only possible cause of system crash.. this may be faulty driver, hardware failure, kernel panic, etc. -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 10/14/2013 6:16 PM, CeDeROM wrote: Isn't there Journal to prevent and reverse such damage? Unlike other journaling filesystems, UFS+J only protects the metadata, not the data itself - i.e. I think it ensures you won't have to run a manual fsck, but just like plain old UFS files may be truncated as the journal is replayed. For ext3, https://www.kernel.org/doc/Documentation/filesystems/ext3.txt explains the different modes, with 'ordered' being default: Data Mode - There are 3 different data modes: * writeback mode In data=writeback mode, ext3 does not journal data at all. This mode provides a similar level of journaling as that of XFS, JFS, and ReiserFS in its default mode - metadata journaling. A crash+recovery can cause incorrect data to appear in files which were written shortly before the crash. This mode will typically provide the best ext3 performance. * ordered mode In data=ordered mode, ext3 only officially journals metadata, but it logically groups metadata and data blocks into a single unit called a transaction. When it's time to write the new metadata out to disk, the associated data blocks are written first. In general, this mode performs slightly slower than writeback but significantly faster than journal mode. * journal mode data=journal mode provides full data and metadata journaling. All new data is written to the journal first, and then to its final location. In the event of a crash, the journal can be replayed, bringing both data and metadata into a consistent state. This mode is the slowest except when data needs to be read from and written to disk at the same time where it outperforms all other modes. -- Bruce Cran ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, 14 Oct 2013 18:34:36 +0200 David Demelier wrote: On 14.10.2013 14:39, RW wrote: If you are having problems with data integrity you might try gjournal or zfs instead. Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? SU+J isn't a journalled filesytem, it's a filesystem with soft-updates that journals information about free space so it can be recovered without having to go through the whole filesystem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 7:54 PM, Bruce Cran br...@cran.org.uk wrote: On 10/14/2013 6:16 PM, CeDeROM wrote: Isn't there Journal to prevent and reverse such damage? Unlike other journaling filesystems, UFS+J only protects the metadata, not the data itself - i.e. I think it ensures you won't have to run a manual fsck, but just like plain old UFS files may be truncated as the journal is replayed. Thank you for explaining :-) So it looks that it would be sensible to force filesystem check every n-th mount..? Or to do a filesystem check after crash..? Are there any flags like that to mark filesystem unclean and to force fsck after n-th mount? That would assume disabling journal and soft updates journaling I guess..? What would be the best option for best data integrity in case of crash? That would be helpful for development systems I guess :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 10/14/2013 7:33 PM, CeDeROM wrote: Thank you for explaining :-) So it looks that it would be sensible to force filesystem check every n-th mount..? Or to do a filesystem check after crash..? Are there any flags like that to mark filesystem unclean and to force fsck after n-th mount? That would assume disabling journal and soft updates journaling I guess..? What would be the best option for best data integrity in case of crash? That would be helpful for development systems I guess :-) As I understand it UFS+J gives the same reliability as UFS with a normal fsck after a crash, so on a development system the only ways to improve the situation would be to mount with the 'sync' option, disable write caching on the disk or to switch to a different filesystem like ZFS. -- Bruce Cran ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 1:33 PM, CeDeROM cede...@tlen.pl wrote: Thank you for explaining :-) So it looks that it would be sensible to force filesystem check every n-th mount..? Please explain the logic in which this helps anything. Or to do a filesystem check after crash..? Already standard behavior as implicitly seen in this thread. Are there any flags like that to mark filesystem unclean and to force fsck after n-th mount? No and any fs that requires such a system is broken by design. That would assume disabling journal and soft updates journaling I guess..? What would be the best option for best data integrity in case of crash? mount -o sync or use ZFS. Both require hardware that correctly report success to fsync. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, Oct 14, 2013 at 1:43 PM, Adam Vande More amvandem...@gmail.comwrote: mount -o sync should be mount sync -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Oct 14, 2013, at 11:33 AM, CeDeROM cede...@tlen.pl wrote: On Mon, Oct 14, 2013 at 7:54 PM, Bruce Cran br...@cran.org.uk wrote: On 10/14/2013 6:16 PM, CeDeROM wrote: Isn't there Journal to prevent and reverse such damage? Unlike other journaling filesystems, UFS+J only protects the metadata, not the data itself - i.e. I think it ensures you won't have to run a manual fsck, but just like plain old UFS files may be truncated as the journal is replayed. Thank you for explaining :-) So it looks that it would be sensible to force filesystem check every n-th mount..? You shouldn't ever need to recheck the filesystem if it was shutdown cleanly. However, it doesn't hurt to fire off an fsck once a year or so just to look for any unexpected issues. Or to do a filesystem check after crash..? Yes. Without journalling, you'd normally perform the full timeconsuming fsck in the foreground. With journalling, it should be able to do a journal replay to restore the filesystem to an OK state, but sometimes that doesn't restore consistency, in which case it usually fires off a background fsck rather than the foreground fsck. Are there any flags like that to mark filesystem unclean and to force fsck after n-th mount? That would assume disabling journal and soft updates journaling I guess..? /etc/rc.conf should support something like the following to do what you ask: fsck_y_enable=YES background_fsck=NO force_fsck=YES What would be the best option for best data integrity in case of crash? That would be helpful for development systems I guess :-) Well, you can use mount -o sync and disable write caching via hw.ata.wc=0 or similar depending on what kind of drives you use. This will cause a massive loss of write performance, but will greatly improve reliability-- i.e. fsync() and such are not as likely to lie about whether bits have made it to disk, even in the face of hardware which lies about ATA_FLUSHCACHE (or SCSI SYNCHRONIZE CACHE, etc). Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
Thank you all for good hints! This will come handy! :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, 14 Oct 2013, Bruce Cran wrote: On 10/14/2013 6:16 PM, CeDeROM wrote: Isn't there Journal to prevent and reverse such damage? Unlike other journaling filesystems, UFS+J only protects the metadata, not the data itself - i.e. I think it ensures you won't have to run a manual fsck, but just like plain old UFS files may be truncated as the journal is replayed. This discussion skirts the critical issue - are files that are not open for writing endangered? No description of the uses of journaling can be considered informative if it doesn't address that explicitly. As a naive user I have always assumed that once closed, a file was invulnerable to improper shutdowns, but this discussion shakes that belief. I expect the answer may be different for SSD and spinning disks. dan feenberg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, 14 Oct 2013 11:48:18 -0700 Charles Swiger wrote: Yes. Without journalling, you'd normally perform the full timeconsuming fsck in the foreground. Journalling removes the need for the background fsck which only recovers lost space. With journalling, it should be able to do a journal replay to restore the filesystem to an OK state, My understanding is that the journal does nothing to restore the filesystem other than keep track of orphaned memory. In all other respect it's the job of soft-updates to keep the filesystem in an OK state. When it doesn't you need a foreground check. but sometimes that doesn't restore consistency, in which case it usually fires off a background fsck rather than the foreground fsck. I think if the journal fails, you would really need to run at least a foreground preen, maybe a full fsck. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
Charles Swiger wrote: [snip] Yes. Without journalling, you'd normally perform the full timeconsuming fsck in the foreground. With journalling, it should be able to do a journal replay to restore the filesystem to an OK state, but sometimes that doesn't restore consistency, in which case it usually fires off a background fsck rather than the foreground fsck. In my case the journal replay failed, with an error to that effect. All partitions other than / failed to mount and after hitting enter at the .../bin/sh prompt performed manual fsck on all of them, which found and fixed some stuff. Then shutdown -r and everything came up fine (clean) afterwards. Net result was no data loss for me. [snip] -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Mon, 14 Oct 2013 11:48:18 -0700 Charles Swiger wrote: fsck_y_enable=YES One of the most annoying things about SU+J is that fsck asks if you want to use the journal. So fsck -y wont do a proper check unless the journal replay fails. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 14.10.2013 20:08, RW wrote: On Mon, 14 Oct 2013 18:34:36 +0200 David Demelier wrote: On 14.10.2013 14:39, RW wrote: If you are having problems with data integrity you might try gjournal or zfs instead. Why? SU+J is enabled by default. Isn't the purpose of a journaled file system to ensure that any bad shutdown will protect data? SU+J isn't a journalled filesytem, it's a filesystem with soft-updates that journals information about free space so it can be recovered without having to go through the whole filesystem. Okay, but why the fsck didn't run by itself to detect that the journal didn't replayed correctly (if I understanding well) to correct the issues? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 14.10.2013 18:47, Adam Vande More wrote: There is no *warranty* as explicitly stated in http://www.freebsd.org/copyright/freebsd-license.html Aha, please don't play on words ;-). I think you understood I was speaking about the filesystem state not a lawyer issue. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 14.10.2013 20:43, Adam Vande More wrote: On Mon, Oct 14, 2013 at 1:33 PM, CeDeROM cede...@tlen.pl mailto:cede...@tlen.pl wrote: Thank you for explaining :-) So it looks that it would be sensible to force filesystem check every n-th mount..? Please explain the logic in which this helps anything. Or to do a filesystem check after crash..? Already standard behavior as implicitly seen in this thread. Are there any flags like that to mark filesystem unclean and to force fsck after n-th mount? No and any fs that requires such a system is broken by design. That would assume disabling journal and soft updates journaling I guess..? What would be the best option for best data integrity in case of crash? mount -o sync or use ZFS. Both require hardware that correctly report success to fsync. I personnally love ZFS and use it massively on my server, but for a desktop I think this is a real overkill. Also I don't have so much RAM to waste for that. I think UFS is enough, however as a modern operating system I don't expect any data corruption by default using SU+J. The filesystem domain is not a thing I really know deeply, so thanks for all you explanation. PS: the power failure is not the only way that does not shutdown cleanly the system. There are kernel panics, crash and such of course. Those which appears sometimes too. Regards, ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
Hi-- On Oct 14, 2013, at 11:51 AM, Daniel Feenberg feenb...@nber.org wrote: This discussion skirts the critical issue - are files that are not open for writing endangered? No description of the uses of journaling can be considered informative if it doesn't address that explicitly. As a naive user I have always assumed that once closed, a file was invulnerable to improper shutdowns, but this discussion shakes that belief. Well, it's good to be a little paranoid if the data matters. :-) First, unless you call fsync() before close() and your OS and/or drive hardware isn't being deceptive when fsync() returns about whether the bits have made it to permanent storage, then you might be surprised at just how long the unwritten buffers containing the last updates to the file data take to get properly flushed to disk. I expect the answer may be different for SSD and spinning disks. Second, this is an excellent point: however, it also applies to anything where the actual hardware block size does not match the device blocksize that the filesystem thinks it has-- so new 4K sector rotational disks also have some risk. The basic issue with SSDs is that you (or the drive firmware, more precisely) need to read in an entire hardware sector, update the portion with changes in cache memory, do a bulk-erase of that block, and then scribble that back out. Good drive firmware actually writes out to a different block than the original for wear-leveling purposes and only updates the flash translation layer once the new version of that block is written. That makes the drive mostly immune to major data integrity issues even if powered off in the middle of the process. Less-than-good firmware, aka buggy firmware, can lead from power-failure to data loss of files which were not being modified at the time. And may you possess recent working backups if the FTL somehow ever gets confused! Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On Oct 14, 2013, at 12:41 PM, RW rwmailli...@googlemail.com wrote: On Mon, 14 Oct 2013 11:48:18 -0700 Charles Swiger wrote: Yes. Without journalling, you'd normally perform the full timeconsuming fsck in the foreground. Journalling removes the need for the background fsck which only recovers lost space. That and inode link changes (ie, adding or removing files from a directory). With journalling, it should be able to do a journal replay to restore the filesystem to an OK state, My understanding is that the journal does nothing to restore the filesystem other than keep track of orphaned memory. In all other respect it's the job of soft-updates to keep the filesystem in an OK state. Yes, SU is supposed to reorder filesystem operations to provide some level of ACID transaction semantics-- and the journal helps that by avoiding the need for bgfsck. When it doesn't you need a foreground check. but sometimes that doesn't restore consistency, in which case it usually fires off a background fsck rather than the foreground fsck. I think if the journal fails, you would really need to run at least a foreground preen, maybe a full fsck. Yes. Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 13 Oct 2013 11:30, David Demelier demelier.da...@gmail.com wrote: Hello there, I'm writing because after a power failure I was unable to log in on my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some files disappeared, including /etc/pwd.db. Thus I was unable to log in. I've been able to regenerate the password database with a live cd but I'm afraid that more files had disappeared somewhere else... I think this is a serious issue, the journal should not truncate files, so something should have gone wrong somewhere.. Any ideas? Should I open a PR I had similar issues somewhere around 9.0 - although journal check was fine running fsck revealed filesystem inconsistency. I have reported this on the list, but it seemed unnoticed..? For me this is serious issue as well, if you make PR I will give +1 :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 13.10.2013 12:16, CeDeROM wrote: On 13 Oct 2013 11:30, David Demelier demelier.da...@gmail.com mailto:demelier.da...@gmail.com wrote: Hello there, I'm writing because after a power failure I was unable to log in on my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some files disappeared, including /etc/pwd.db. Thus I was unable to log in. I've been able to regenerate the password database with a live cd but I'm afraid that more files had disappeared somewhere else... I think this is a serious issue, the journal should not truncate files, so something should have gone wrong somewhere.. Any ideas? Should I open a PR I had similar issues somewhere around 9.0 - although journal check was fine running fsck revealed filesystem inconsistency. I have reported this on the list, but it seemed unnoticed..? For me this is serious issue as well, if you make PR I will give +1 :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info Yes, I've also ran fsck in single user mode after and lot of incorrect things were corrected, I wait a bit for answers (if any) before sending a PR. Cheers, ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: SU+J Lost files after a power failure
On 13.10.2013 12:16, CeDeROM wrote: On 13 Oct 2013 11:30, David Demelier demelier.da...@gmail.com mailto:demelier.da...@gmail.com wrote: Hello there, I'm writing because after a power failure I was unable to log in on my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some files disappeared, including /etc/pwd.db. Thus I was unable to log in. I've been able to regenerate the password database with a live cd but I'm afraid that more files had disappeared somewhere else... I think this is a serious issue, the journal should not truncate files, so something should have gone wrong somewhere.. Any ideas? Should I open a PR I had similar issues somewhere around 9.0 - although journal check was fine running fsck revealed filesystem inconsistency. I have reported this on the list, but it seemed unnoticed..? For me this is serious issue as well, if you make PR I will give +1 :-) CeDeROM, SQ7MHZ, http://www.tomek.cedro.info Yes, I've also ran fsck in single user mode after and lot of incorrect things were corrected, I wait a bit for answers (if any) before sending a PR. Running fsck in single-user mode may not be sufficient. You may need to run fsck_ffs from a USB-stick installation or live CD. I remember reviving a FreeBSD partition that way, normal root partition not mounted. I once revived a FreeBSD partition with fsck_ffs from a USB-stick installation of NetBSD 5.1_STABLE i386 after FreeBSD couldn't do it. It helps to have a UPS to protect against short power failures and allow graceful shutdown on longer power outages. Tom ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
