Another candidate for the Wiki site?
David
Suresh Thalamati wrote:
Attached is the functional/design spec based on the input I got from
everyone on this topic. I think, most of the issues are covered in this
doc, except the backup of jars stored inside the database.
Thanks
-suresh
------------------------------------------------------------------------
Online Backup Functional/Design Spec
Overview
Support real-time */online backup/* mechanism that does not block
database operations when backup is in progress. When backup is running,
currently Derby only permits reads from the database. By adding support
to permit writes, it will be easier for the users to schedule backups
without being concerned about the impact of backup operation on the
applications that are accessing the database.
Introduction
New online backup mechanism will not change the way database can be
restored from the backup. In the current system, if there any non-logged
operations when backup starts, they will be rolled back on restore
because database is frozen at the start of the backup. To support
real-time online backups that do not block user operations, it is
necessary to have the transaction log for all the operations that happen
after the backup starts to bring the database to the consistent state
during restore. Because a transaction that has non-logged operations
can commit when backup is in progress. To make a consistent backup of
the database, there should not be any pending transactions with
non-logged operations.
JDBC
Backup operations are support through system procedures. They can be
called using JDBC program or through IJ.
Existing Backup Procedures:
After this enhancement, existing backup procedures
(SYSCS_UTIL.SYSCS_BACKUP_DATABASE ...etc) will allow writes to the
database.
Information about existing online backup procedures can be found in the
Derby admin guide. Derby Admin Guide
<http://db.apache.org/derby/docs/dev/adminguide/> . These backup
procedures will *WAIT *for the transaction with non-logged operations to
commit before proceeding with the backup. __
New Procedures:
Two new procedures are added to make backup wait/return error, if any
non-logged operations are pending at the start of the backup.
Full backup:
*/SYSCS_UTIL.SYSCS_BACKUP_DATABASE(IN BACKUPDIR VARCHAR(), /*
*/ /_IN WAIT_FOR_NONLOGGED_OPERATIONS
SMALLINT_/) /*
Full backup with log archive mode:
*SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE
(IN BACKUPDIR VARCHAR(32672), *
* IN SMALLINT DELETE_ARCHIVED_LOG_FILES,
/_IN WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT_/_))_
*
Arguments
*/BACKUPDIR:/**//*
An input argument of type VARCHAR(32672) that specifies the full system
path to the database directory to be backed up.
*/DELETE_ARCHIVED_LOG_FILES:/*
A non-zero input argument of type SMALLINT will make backup procedures
to delete existing online archived log files that were created before
this backup will be deleted. The log files are deleted only after a
successful backup.
* *
*/WAIT_FOR_NONLOGGED_OPERATIONS/* :
A non-zero input argument of type SMALLINT will make backup procedures
to wait for the non-logged operation that are currently running to
commit, while an argument of 0 will make backup fail if there are any
uncommitted non-logged operations.
Design
Current Online backup copies all the database files (log, seg0 ...Etc)
to the specified location when the database is frozen (writes are blocked).
In the new scheme, goal is to copy the database information to the
backup location, without blocking any user operation for the whole
duration of the backup. Following sections discuss how the information
is copied into the backup.
Containers:
At the start of the backup, all the containers that need to be backed up
are found by finding all the files in the data segment (seg0). Then each
container is copied to the backup by reading through the page cache
instead of the direct file copy.
· Individual pages are latched, when they are being copied to the
backup to prevent modifications to the page.
· Any new containers created when backup is in progress will be
recreated on restore using the information from the transaction log.
· Containers that were in the process of creation will not be
backed up, if the creation is not complete. Container cache will make
sure that backup can not get handle to the containers that are being
created.
· If committed drop occurs on a container that is being backed
up, backup will wait for the committed drop to complete and then copy
the stub to the backup.
· Committed container stubs that are created after the checkpoint
for the backup will not be deleted until the backup is complete. A
checkpoint that occurs after the backup garbage collects them.
· All requests to open containers in non-logged mode will be
automatically converted to logged mode after the backup starts.
Log files:
Transaction log is used to bring the database to the consistent state on
restore from the backup. A checkpoint is made at the start of the
backup; this checkpoint will be starting point for the recovery from the
backup. Log files are copied to the backup using regular file copy.
* Backup Checkpoint information in the log control files is copied
to the backup location at the start.
* All the transaction log files starting from the backup checkpoint
to the log file that is active at the end of the backup are copied
into the backup.
* State of the database in the backup depends on last log file that
goes into the back up. After all the containers are copied to the
backup , last log file that goes into the backup is determined.
* Database engine wills not delete the log files that are to yet to
be backed up, when backup is running even if they are no longer
required for regular crash recovery.
* Checkpoints can occur in parallel when the backup is progress.
Jar files:
Storing Jar files in the database is not a logged operations and they
don’t go through the page cache; they will be copied to the backup
directly as it is done now. I am not sure at the moment, if that will
lead to any issues. Any input about jar files will be helpful.
Other files:
* Database bootstrap file Service.Properties will be to be copied to
the backup. Need to check if any synchronization is required to
copy this file.
* Copy the BACKUP.HISTORY file at the end of the backup.
* Lock files should not be copied to the backup.
* Tmp directory should not be backed up.
Testing:
Most of the current backup tests are single threaded. I will add new
functional tests case that will execute read/writes on the database,
when backup is running. Some of the testing scenarios:
1. Test the new procedure calls.
2. Checkpoint in parallel to the backup.
3. DML operations in parallel to the backup.
4. Grow the container after it is backed up / parallel.
5. Non-logged operations in parallel to the backup and pending when
backup starts.
6. DDL operations in parallel to the backup.
7. In-Place compress in parallel to the backup.
8. ….etc.
References:
· Derby Admin Guide Backup Section
<http://db.apache.org/derby/docs/dev/adminguide/>
· Derby-Dev Thread
<http://nagoya.apache.org/eyebrowse/ReadMsg?listId=270&msgNo=5038>
· Version 1. 10/5/2005 : [EMAIL PROTECTED]
begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:[EMAIL PROTECTED]
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard
