Re: backup v2
On Wed, 28 Jun 2000, Robert Graham Merkel wrote: But what if some catastrophic event happens while the modified log file is written to disk? Couldn't you possibly lose the entire log? I think not, but I don't know for sure. I was thinking that GnuCash would open() the log for appending only, and every now and then write a single log entry using a single write() call. What are the chances of a write error destroying data earlier in the log? War stories, anyone? (When chopping off the front of a log, GnuCash would use the same write-and-move procedure as it would when saving the database.) When appending to a file, unless the end of file is exactly aligned on a physical block boundary, the entire last block must be rewritten. This puts the (partial) contents of the last block at risk. Note that physical blocks on modern hard disks are much larger than the nominal 512 bytes oa 1k that Linux uses internally. -- hendrik -- Gnucash Developer's List To unsubscribe send empty email to: [EMAIL PROTECTED]
Re: backup v2
Christopher Browne [EMAIL PROTECTED] writes: But I'd consider having, rather than merely a log file, a log directory, and provide the _option_ of splitting the logs off in some sort of "regular" manner. Whether that's every 5MB, or one each day, or week, it would be somewhat attractive to distinguish between the "old ones we don't touch no more" and the "present one." A recovery option that _I_ like (off top of head...) would be to allow the gentle user to use "cat" to catenate together whatever old logs appear relevant, into one log, and then for GnuCash to be able to read from that log, perhaps with some control kept over this. Options that would be nice: I belive the new format that Bill hinted at that I'm just about finished with will make everything you want here pretty trivial. I'll be submitting a description and committing some initial code soon, so everyone can comment, but not until I finish my current train of thought and have some time to test it. In service of that, I'm going to write something that Linas has wanted for a while: (account-group-= g1 g2). This will let me test for correctness here, and it'll be useful for "make test" which we should start fleshing out soon. -- Rob Browning [EMAIL PROTECTED] PGP=E80E0D04F521A094 532B97F5D64E3930 -- Gnucash Developer's List To unsubscribe send empty email to: [EMAIL PROTECTED]
Re: backup v2
Glen Ditchfield writes: On Mon, 26 Jun 2000, Dave Peticolas wrote: This is the second version of the backup proposal. ... When GnuCash starts, it should check for the presence of the log file and, if it exists and has a later modification time than the main file, GnuCash should prompt the user to load the log file in addition I don't want to trust file modification times; sometimes they get changed. H about logging 'save' events, with matching timestamps or some such in the saved database and the log entry? When GnuCash starts, it checks the log. I the last entry isn't the 'save' for the database, something went wrong. What if we just used the existence of the file? If the save was successful, the log file should have been moved to the historical log file anyway. + GnuCash should maintain log histories, rather than save histories, to handle failure #3. After saving, the current log file should be renamed to be the first log history file, and a new log file should be started. Wouldn't it be simpler to have just one log file, and keep appending to it? Fewer file operations means fewer opportunities for disaster. Logging, log loading, undo, and miscellaneous scheme utilities would be less complicated because they won't have to handle multiple log files, some of which could be missing. (But you would need to give users a way to chop the front off of th log when they want their disk space back.) I'm not sure recovery would be simpler, since if you are recovering from the backup database and the log, you would need to know what position in the log to load from. Since the log may have been 'trimmed' in the interim, this would be problematic. dave -- Gnucash Developer's List To unsubscribe send empty email to: [EMAIL PROTECTED]
Re: backup v2
On Wed, 28 Jun 2000, Robert Graham Merkel wrote: But what if some catastrophic event happens while the modified log file is written to disk? Couldn't you possibly lose the entire log? I think not, but I don't know for sure. I was thinking that GnuCash would open() the log for appending only, and every now and then write a single log entry using a single write() call. What are the chances of a write error destroying data earlier in the log? War stories, anyone? (When chopping off the front of a log, GnuCash would use the same write-and-move procedure as it would when saving the database.) -- Gnucash Developer's List To unsubscribe send empty email to: [EMAIL PROTECTED]
backup v2
This is the second version of the backup proposal. It's quite
a bit different than the first. Thanks to everyone who commented
on the first one, both on the list and in private.
Proposed Backup Strategy for GnuCash v2
===
Goals
-
GnuCash's method for backing up its database files should protect
against three kinds of failures:
1. A crash during the operation of GnuCash that doesn't affect the
database file, but which prevents newly-entered data from being
saved.
2. An error during the writing of a GnuCash database file that
prevents the file being written correctly, resulting in an obviously
incorrect file. This could be caused by, e.g., a power or hardware
failure, running out of disk space, the user hitting ctrl-c or killing
the process in some other way, or a bug in the GnuCash file i/o code.
3. The creation of a GnuCash database file that is a readable but
incorrect in a non-obvious way. Such a problem may not be discovered
for some time. Such a problem could be caused by a bug in GnuCash or
by the user accidentally entering wrong data or performing an operation
that was later determined to be undesirable (like deleting an account).
Current Status
--
GnuCash only writes the main database file when requested by the user.
Currently, when saving, GnuCash writes the main database file and then
writes a time-stamped duplicate of that file. Thus, the user will have
a 'save history' of database files that record the state of the database
at each point the user requested a save. This 'save history' will
protect against failure #3, as you can revert to an earlier, correct
version of the database.
GnuCash also creates an ascii log file which records changes made to
GnuCash entities during operation. Currently, this log file can be
used to manually recover from failure #1, but there is no automated
way to do so, or any other support for protecting against this kind
of error.
Problems
+ Recovering manually from a log is tedious and thus failure #1 is a
real pain.
+ None of the current backup methods will protect against failure #2.
+ Creating a 'save history' will generate lots of large files, especially
for users who save a lot (like me).
Proposed Solution
-
+ To better handle failure #1, GnuCash should be able to automatically
recover from log files.
As changes are made to the database, they will be logged to the
filename "original name.log", where original name is the name
of the database file.
When GnuCash starts, it should check for the presence of the log file
and, if it exists and has a later modification time than the main
file, GnuCash should prompt the user to load the log file in addition
to the main file (since the log file will have the changes to the
main file that occurred after the last save).
+ To handle failure #2, GnuCash should change the way it saves the main
database file. Saving should be done by:
1. Writing the new database to the temporary file.
The temporary file name will be "original name.tmp"
2. Moving or copying the old database to a backup file,
overwriting any existing backup file. The backup file
name will be "original name~"
3. Moving the temporary file to the main database file.
Thus, after successful completion of a save, the original database will
be saved in a backup file.
If there is a writing failure during step 1, the original database is
untouched.
+ GnuCash should maintain log histories, rather than save histories, to
handle failure #3.
After saving, the current log file should be renamed to be the first
log history file, and a new log file should be started.
Log history files should be named as "original name.log.I" where I
is an integer and means the log history file is the Ith file in the
history.
Thus, loading the database original name should be equivalent to
loading (in order) original name.log.1 ... original name.log.N
where N is the number of log history files.
Example
===
The files in the directory where the main database name 'gnucash.gnc'
is located will be as follows:
gnucash.gnc - main database file
gnucash.gnc~ - backup file from last save
gnucash.gnc.tmp - temporary file created during save
This file will only existing during
saving, unless there is an i/o error.
gnucash.gnc.log - The current log file. This file will
only exist while gnucash.gnc is open,
unless there is an i/o error.
gnucash.gnc.log.1 - These are the log history files.
gnucash.gnc.log.2
...
gnucash.gnc.log.N
Pseudo-Code Implementation
--
Saving: /* Save to temporary file */
save_database_to_file("original name.tmp");
if (error) {
report_error_to_gui();
return;
}
/* Make backup file */
