As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also.
Going forward I would like to see more work done to implement the
relaxed durability option suresh and I were discussing. I believe
that with some code changes file allocation, log write and many of
the data write sync's can be eliminated and still guarantee a bootable
database. This mode would not guarantee a committed transaction,
but would guarantee that database on a disk with enough free space
would boot and that the db would represent only complete transactions (either fully applied or not). This option would
be slower than what is available from this patch so I believe both
features are useful.
David Van Couvering wrote:
Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance.
I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such.
Thanks,
David
Sunitha Kambhampati wrote:
A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability
This mode is very useful for unit testing or at development time when recoverability is not required.
Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this.
So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it.
2) Print message to derby.log that this mode is enabled
3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set.
Changes are not much. svn stat
M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
M java\engine\org\apache\derby\iapi\reference\MessageId.java
M java\engine\org\apache\derby\iapi\reference\Property.java
M java\engine\org\apache\derby\loc\messages_en.properties
A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java
M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties
A java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out
M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall
Since this is a longish email , for those who may not read the rest of it, here are the testing results
Testing results:
- I ran derbyall with 1.4.2 jvm and all tests passed. - I ran the storeall suite separately both with setting the derby.system.testMode property to true and without it. with this property set, the storeall suite ran about 2 times as fast.
-------------------------------
Below I explain how the patch addresses the above requirements and try to provide details on how I went about making those changes
0) Note
1) A single property has been added to enable this relaxed durability option. The property name is derby.system.testMode and takes in value true or false. By default it is false. It can be set as a command line option to the JVM when starting the application or within the application or in the derby.properites file. Property is loaded at boot time, so for it to have an effect you need to enable it at boot time. More comments added in code itself. see variable TESTMODE_NO_SYNC = "derby.system.testMode"; in Property.java.
(In this patch, I have used 'no_sync' as suffix in variable names because I think with respect to store - that is what this mode is all about - disabling syncs for the cases already mentioned earlier in this mail)
I chose the property name derby.system.testMode for this
- represents a higher concept as to when one would use relaxed durability (no sync) option.
- mostly it should be enabled only when data is not important and ideally maybe for testing and development purposes,
- also with the hope that the word 'testMode' would prompt users to look this property up before using it in a production like environment where it may not necessarily be appropriate.
Code changes for this: - Added this new property derby.system.testMode in java\engine\org\apache\derby\iapi\reference\Property.java
- Removed the following properties from Property.java as they are no longer required.
derby.storage.dataNotSyncedAtCheckPoint=true ( I'll refer to this as DNSAC in notes below)
derby.storage.dataNotSyncedAtAllocation=true ( I'll refer to this as DNSAA in notes below)
derby.storage.logNotSynced=true (I'll refer to this as LNS in notes below)
- Previously code existed for enabling the DNSAC, DNSAA and LNS options conditional on Performance.MEASURE
So how it works is , the 2 variables in BaseDataFileFactory.java are used to control if syncs happen or not for data pages at allocation and at checkpoint.
dataNotSyncedAtAllocation dataNotSyncedAtCheckpoint
And in java/engine/org/apache/derby/impl/store/raw/log/LogToFile, the logNotSynced variable is used to control if syncs happen for the logs are not.
These variables are enabled (set to true) if Performance.MEASURE is set to true, and if the DNSAC, and DSNAA propertie are enabled.
So all that is required now is
- remove conditional on Performance.MEASURE in these cases, instead use the derby.system.testMode property to enable them . This happens in the boot methods
- make sure to remove any conditionals on Performance.MEASURE that is used to disable syncs but instead control it by the variables mentioned above.. ie e.g. dataNotSyncedAtAllocation in BaseDataFileFactory.java ...
- remove reading of the DNSAC, DNSAA,LNS property as they are not needed anymore
Changes were needed to following files - i) java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java (check boot method)
ii) java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java
iii) java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (check boot method).
2) Write to the derby.log a warning message
Needed to add a new message. Messages need to be translated, so I made the following changes
i) added a variable in java/engine/org/apache/derby/iapi/reference/MessageId.java to represent this new message.
String STORE_TESTMODE_NO_SYNC = "D013";
ii) Message content added to java/engine/org/apache/derby/loc/messages_en.properties
D013=WARNING: {0} is set to true. In this mode, there is no guarantee that the system will recover a system crash or JVM termination. Enable this property only when there is no need for recoverability \n
iii) In the boot process, rawstore boot calls logfactory boot, so the first time the derby.system.testMode property is read, it is in rawstore, so added code to print the warning message in BaseDataFileFactory.boot(..)
3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot if errors happen during recovery report a message that it could have happened because of this mode being set.
- To write a marker out , log.ctrl file is used, it has a few spare bytes so no major changes needed here. ( ie no disk format changes.... )
Check the method readControlFile in java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
- Used the first spare byte available in log control file which will be used to store information about if derby.system.testMode is set or not.
Added variable TESTMODE_NO_SYNC_FLAG_OFFSET to represent this offset in LogToFile.java
- The value written out if this property is enabled is TESTMODE_NO_SYNC_TRUE = 0x1 else a 0 is written out. ( I didnt like to put the 0 but wasnt feeling too strong to adding a new static byte variable either.. and as rest of the code around it in writeControlFile seems to use 0 as is , I just followed the same approach).
- Change made to readControlFile to read in the value for the testmode case when Logfactory is booted, the flag is read and value stored
in variable called wasDBInTestModeNoSync. This variable keeps track of if database was previously booted in the derby.system.testMode or not. Comes in use for error reporting.
- Also added a message to report to the derby.log incase database was previously booted with derby.system.testMode set to true. Change to messages_en.properties and MessageId.java.
L020=WARNING: The database was previously booted with {0} set to true. Note that this mode {0}=true does not guarantee recoverability \n
little on the flow with respect to writeControlFile .. if you look at the code in LogToFile for boot and recover, writeControlFile is called a couple of times.. to read the checkpoint instant etc..
- For my changes, I am just copy/paste the comments from writeControlFile(...) , Note case4 below .. which is the interesting one.
// A spare byte at offset TESTMODE_NO_SYNC_FLAG_OFFSET
// is used to write out if database was in derby.system.testMode=true mode
// or not.
// This byte will have one of the two values
// - TESTMODE_NO_SYNC_TRUE to represent database was booted
// in this mode
// - 0 to represent database was not booted in this mode
// // The sole purpose of this flag in control file is to help
// identify any recovery problems that might happen as a result of this
// mode being set.
//
// Possible scenarios. In table below, cases are listed giving
// the value of derby.system.testMode if it was enabled or not at boot time
// ---------------------------------------------------------------
// Case Previous Current Action , what should the flag be set to in log.ctrl
// Boot Boot file
// --------------------------------------------------------------
// 1 - true write TESTMODE_NO_SYNC_TRUE
// 2 - false write 0
// 3 false false write 0
// 4 true false write TESTMODE_NO_SYNC_TRUE
// 5 true true write TESTMODE_NO_SYNC_TRUE
//
// Case 4 is the interesting case here.
// one cannot write out the derby.system.testMode is disabled just yet
// because it is possible that the database recovery might fail and on subsequent
// restarts this state would be lost. So only when recovery is successful, will
// we update the flag in the log control file to indicate that the derby.system.testMode
// is not enabled. (This will happen at the end of recovery , see recover method)
//
// Case 1, 2 is when database is created fresh with this property enabled.
//
// wasDBInTestModeNoSync has information if database was previously booted
// in derby.system.testMode=true mode
To handle case 4, changes went into recover () method , if database recovery is successful only then we update the flag in the control file to say database is not booted with this property enabled. Since there was no method available to update the log control file, added a new method updateTestModeFlagInCtrlFile(byte value) to do that.
So now that changes are done to write the marker to the control file and to read it, it is necessary to take care of case when error happens.
- added a new message to report that cause of an error could be related to this property being set
added a new variable in MessageId.LOG_TESTMODE_NO_SYNC_ERR in java/engine/org/apache/derby/iapi/reference/MessageId.java
and the message to java/engine/org/apache/derby/loc/messages_en.properties
L021=WARNING: The database was either previously booted or is currently booted with {0} set to true. Note that this mode {0}=true does not guarantee recoverability, and is probably what might have caused this boot error \n
- added MessageId.LOG_TESTMODE_NO_SYNC_ERR in java/engine/org/apache/derby/iapi/reference/MessageId.java
Added a new method logErrMsgForTestModeNoSync() in LogToFile.java to print out this message if any error happens when database is in the influence of this property.
In deciding what the best place would be to call the logErrMsgForTestModeNoSync, I debated between adding it in markCorrupt() versus logErrMsg(..). It seemed like if error happened during boot - the flow was to call markCorrupt which would call logErrMsg. Some error messages seem to call logErrMsg(..) directly. So I picked the logErrMsg as it covered most of the errors thrown during boot/recovery that would be related to something with this property being set. -------------------------------------------
TESTING
1) I added one new java file per the derby test harness, to test the following
- check if the property is enabled correctly by timing some inserts into a table.
- check if the correct flag (TESTMODE_NO_SYNC_TRUE) is written out into the log.ctrl file
- check if the derby.log has the warning message.
Changes include
i)Added new test to java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
ii)Added java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties
TestNoSyncMode_derby.properties has derby.system.testMode=true
This property file is needed so when test is run it picks up this property.
iii)The TestNoSyncMode_derby.properties is added to copyfiles.ant so it will be copied over when compiling derby tests.
iv) To enable this new test as part of the store suite, I modified the
java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall
added store/TestNoSyncMode.java
v) To get my correct output file, first i just put an initial empty file in java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out
Ran the test after compiling source; java org.apache.derbyTesting.functionTests.harness.RunTest store/TestNoSyncMode.java
Then I updated the out file in the master directory, after checking if the TestNoSyncMode.out file was correct.
2) For the error messages on a boot error, it is difficult to put it in a test, so I checked this in the debugger and it works ok.
Meanwhile, Suresh Thalamati posted on the list that he is working on a corruptible storage factory, once that is there - I am hoping I could use that to have a test to check the error messages for boot errors.
As far as I can tell, this does not affect upgrade and such since we are not changing the disk format etc.. ..There is one spare byte in log.ctrl file that is being used now as a result of this change and this byte value should not have been used to represent anything useful by any earlier version
Testing results (already mentioned in the beginning of this email).
Please give your comments. Thanks, Sunitha.
------------------------------------------------------------------------
Index: java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (revision 165265)
+++ java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (working copy)
@@ -265,7 +265,38 @@
/* Log Control file flags. */
private static final byte IS_BETA_FLAG = 0x1;
-
+ + /**
+ * When the derby.system.testMode property is enabled, the store + * system will not force sync calls in the following cases
+ * - for the log file at each commit
+ * - for the log file before data page is forced to disk
+ * - for page allocation when file is grown
+ * - for data writes during checkpoint
+ * This means the recovery system may not work properly
+ * In order that we recognize this case that the database was previously
+ * booted in this mode, this value is written out
+ * into the log control file. This will help prevent us from + * wasting time to resolve issues in such cases. + * @see org.apache.derby.iapi.reference.Property#TESTMODE_NO_SYNC
+ */
+ private static final byte TESTMODE_NO_SYNC_TRUE = 0x1;
+ + /**
+ * A byte in the log control file is used to store information
+ * about if the database is booted in derby.system.testMode mode
+ * The first available spare byte that is available for this purpose is + * after 5 ints ( 4bytes each), a long ( 8 bytes) and 1 byte in the log control
+ * file. Hence the offset is calculated as follows
+ */
+ private static final int TESTMODE_NO_SYNC_FLAG_OFFSET = 5*4 + 8 + 1;
+ + /**
+ * keeps track of if the database was booted previously with + * derby.system.testMode=true + */
+ private static boolean wasDBInTestModeNoSync = false;
+ /* to err on the conservative side, unless otherwise set, assume log
* archive is ON */
@@ -362,7 +393,7 @@
// log goes to the log subdirectory underneath the data directory
String logDevice;
- // debug only flag - disable syncing of log file for debugging performance.
+ // disable syncing of log file when running in derby.system.testMode enabled
private boolean logNotSynced = false;
private boolean logArchived = false;
@@ -403,6 +434,7 @@
private CRC32 checksum = new CRC32(); // holder for the checksum
+ protected static int CHECKSUM_SIZE = 8; // checksum size in bytes
/**
@@ -1194,8 +1226,28 @@
tf.resetTranId();
}
- // done with recovery
-
+ // If we have reached here successfully, that means recovery + // was successful
+ + // Now that database has recovered successfully and if it
+ // is booted with derby.system.testMode=false mode, there is no
+ // need to store information that database was booted previously in + // derby.system.testMode=true mode. + // It is safe to update the marker in log.ctrl file to say
+ // that the derby.system.testMode in not being used
+ + // Need to write to the log.ctrl file as well as the log mirror file
+ if( wasDBInTestModeNoSync && !logNotSynced )
+ {
+ // write to log control file and the mirror file that the testmode + // is not used now
+ if(!updateTestModeFlagInCtrlFile((byte)0))
+ throw StandardException.newException(
+ SQLState.LOG_CONTROL_FILE, getControlFileName());
+ + wasDBInTestModeNoSync = false;
+ } + /////////////////////////////////////////////////////////////
// setup checktpoint daemon
/////////////////////////////////////////////////////////////
@@ -1207,6 +1259,99 @@
}
}
+ /**
+ * Update the flag that represents whether + * derby.system.testMode is enabled or not in both the log control file
+ * as well as the mirror control file
+ * <BR>
+ * Basically, Read the control file, + * update the flag to the new passed in value + * recalculate the checksum and write it out to the log control file
+ * <P> MT- synchronized by caller
+ * @param value to set
+ * @return false if the update didnt happen successfully, else true
+ * @throws IOException
+ * @throws StandardException
+ * @see #readControlFile
+ * @see #writeControlFile(StorageFile, long)
+ */
+ private boolean updateTestModeFlagInCtrlFile(byte value)
+ {
+ StorageRandomAccessFile logControlFile = null;
+ byte barray[] = null;
+ StorageFile ctrlFile = null;
+ + try
+ {
+ ctrlFile = getControlFileName();
+ logControlFile = privRandomAccessFile(ctrlFile, "rw");
+ if (!privCanWrite(ctrlFile))
+ return false;
+ + // retrieve all except the checksum bytes
+ barray = new byte[(int) logControlFile.length() - CHECKSUM_SIZE];
+ logControlFile.readFully(barray);
+ + if (SanityManager.DEBUG)
+ {
+ if (SanityManager.DEBUG_ON(DBG_FLAG))
+ {
+ SanityManager.DEBUG(DBG_FLAG, + "Updating derby.system.testMode marker written out"+
+ "to log control file: prev value = " +
+ barray[TESTMODE_NO_SYNC_FLAG_OFFSET] + " to "+ value); + }
+ }
+ // update the byte at position that represents whether + // derby.system.testMode was used
+ barray[TESTMODE_NO_SYNC_FLAG_OFFSET] = value;
+ + // write back into file the updated value and also compute + // the newchecksum and write it to file
+ logControlFile.seek(0);
+ logControlFile.write(barray);
+ checksum.reset();
+ checksum.update(barray, 0, barray.length);
+ logControlFile.writeLong(checksum.getValue());
+ syncFile(logControlFile);
+ logControlFile.close();
+
+ // write the same data to mirror control file
+ ctrlFile = getMirrorControlFileName();
+ logControlFile =
+ privRandomAccessFile(ctrlFile, "rw");
+ if (!privCanWrite(ctrlFile))
+ return false;
+ logControlFile.seek(0);
+ logControlFile.write(barray);
+ logControlFile.writeLong(checksum.getValue());
+ syncFile(logControlFile);
+ }
+ catch(IOException ioe)
+ {
+ logControlFile = null;
+ return false;
+ }
+ catch(StandardException se)
+ {
+ logControlFile = null;
+ return false;
+ }
+ finally
+ {
+ try
+ { + if (logControlFile != null)
+ logControlFile.close();
+ }
+ catch(IOException ioe2)
+ {
+ logControlFile = null;
+ return false;
+ }
+ }
+ return true;
+ }
/**
Checkpoint the rawStore.
@@ -2142,10 +2287,61 @@
//
// write some spare bytes after 2.0 we have 3 + 2(8) spare bytes.
+ // now with v10.1.0.0 we have 2 +2(8) spare bytes as one of the spare
+ // byte is used as a marker to write out if system is runinng
+ // in derby.system.testMode=true or not.
+ long spare = 0;
+ + // A spare byte at offset TESTMODE_NO_SYNC_FLAG_OFFSET
+ // is used to write out if database was in derby.system.testMode=true mode
+ // or not. + // This byte will have one of the two values
+ // - TESTMODE_NO_SYNC_TRUE to represent database was booted
+ // in this mode + // - 0 to represent database was not booted in this mode
+ // + // The sole purpose of this flag in control file is to help
+ // identify any recovery problems that might happen as a result of this
+ // mode being set. + // + // Possible scenarios. In table below, cases are listed giving + // the value of derby.system.testMode if it was enabled or not at boot time
+ // ---------------------------------------------------------------
+ // Case Previous Current Action , what should the flag be set to in log.ctrl + // Boot Boot file
+ // --------------------------------------------------------------
+ // 1 - true write TESTMODE_NO_SYNC_TRUE + // 2 - false write 0 + // 3 false false write 0
+ // 4 true false write TESTMODE_NO_SYNC_TRUE + // 5 true true write TESTMODE_NO_SYNC_TRUE
+ // + // Case 4 is the interesting case here.
+ // one cannot write out the derby.system.testMode is disabled just yet
+ // because it is possible that the database recovery might fail and on subsequent
+ // restarts this state would be lost. So only when recovery is successful, will + // we update the flag in the log control file to indicate that the derby.system.testMode
+ // is not enabled. (This will happen at the end of recovery , see recover method) + // + // Case 1, 2 is when database is created fresh with this property enabled.
+ // + // wasDBInTestModeNoSync has information if database was previously booted
+ // in derby.system.testMode=true mode
+ if( !wasDBInTestModeNoSync ) // derby.system.testMode=false
+ {
+ // in this case, no need to write out any special marker
+ // for derby.system.testMode=false case.
+ if(!logNotSynced)
+ daos.writeByte(0); + else
+ daos.writeByte(TESTMODE_NO_SYNC_TRUE); + }
+ else
+ daos.writeByte(TESTMODE_NO_SYNC_TRUE);
+ daos.writeByte(0);
daos.writeByte(0);
- daos.writeByte(0);
daos.writeLong(spare);
daos.flush();
// write the checksum for the control data written
@@ -2281,7 +2477,24 @@
onDiskMinorVersion = dais.readInt();
int dbBuildNumber = dais.readInt();
int flags = dais.readByte();
-
+ + // check if the database was running previously with
+ // derby.system.testMode=true mode
+ // if database was running previously in derby.system.testMode=true
+ // and if on a restart - recovery fails we want to be able
+ // to report that the boot error could have happened
+ // because of this property
+ wasDBInTestModeNoSync =(dais.readByte() & TESTMODE_NO_SYNC_TRUE) != 0;
+ + if (SanityManager.DEBUG)
+ {
+ if (SanityManager.DEBUG_ON(LogToFile.DBG_FLAG))
+ SanityManager.DEBUG(LogToFile.DBG_FLAG, + "log control file, was derby.system.testMode enabled = " + + wasDBInTestModeNoSync);
+ }
+ + onDiskBeta = (flags & IS_BETA_FLAG) != 0;
if (onDiskBeta)
@@ -2794,19 +3007,15 @@
isWriteSynced = false;
}
- if (Performance.MEASURE)
+ // if system is running with derby.system.testMode enabled
+ // log records and log file are not synced, so + // set flag to disable log syncs
+ if (PropertyUtil.getSystemBoolean(Property.TESTMODE_NO_SYNC))
{
- // debug only flag - disable syncing of log.
- logNotSynced = - PropertyUtil.getSystemBoolean(Property.STORAGE_LOG_NOT_SYNCED);
-
- if (logNotSynced)
- {
- Monitor.logMessage("logNotSynced = true");
- //if log is Not being synced;files should not be open in write sync mode
- isWriteSynced = false;
- }
- + // disable syncing of log.
+ logNotSynced = true;
+ //if log is Not being synced;files should not be open in write sync mode
+ isWriteSynced = false; }
// try to access the log
@@ -2830,6 +3039,18 @@
if (privExists(logControlFileName))
{
checkpointInstant = readControlFile(logControlFileName, startParams);
+ // in case system was running previously with derby.system.testMode=true
+ // then print a message to the derby log
+ if(wasDBInTestModeNoSync)
+ {
+ // print message stating that the database was
+ // previously running with derby.system.testMode=true
+ // mode
+ Monitor.logMessage(MessageService.getTextMessage(
+ MessageId.LOG_WAS_IN_TESTMODE_NO_SYNC,
+ Property.TESTMODE_NO_SYNC));
+ }
+ if (checkpointInstant == LogCounter.INVALID_LOG_INSTANT &&
getMirrorControlFileName().exists())
checkpointInstant =
@@ -2879,8 +3100,9 @@
// blow away the log file if possible
if (!privDelete(logFile) && logFileNumber == 1)
{
+ logErrMsgForTestModeNoSync();
throw StandardException.newException(
- SQLState.LOG_INCOMPATIBLE_FORMAT, dataDirectory);
+ SQLState.LOG_INCOMPATIBLE_FORMAT,dataDirectory);
}
// If logFileNumber > 1, we are not going to write that file just
@@ -2913,6 +3135,7 @@
if (!privDelete(logFile))
{
+ logErrMsgForTestModeNoSync();
throw StandardException.newException(
SQLState.LOG_INCOMPATIBLE_FORMAT,
dataDirectory);
@@ -3584,15 +3807,8 @@
}
else
{
- if (Performance.MEASURE)
- {
- if (!logNotSynced)
- logOut.syncLogAccessFile();
- }
- else
- {
- logOut.syncLogAccessFile();
- }
+ if (!logNotSynced)
+ logOut.syncLogAccessFile();
}
syncSuceed = true;
@@ -3925,6 +4141,7 @@
*/
protected void logErrMsg(String msg)
{
+ logErrMsgForTestModeNoSync();
Monitor.logTextMessage(MessageId.LOG_BEGIN_ERROR);
Monitor.logMessage(msg);
Monitor.logTextMessage(MessageId.LOG_END_ERROR);
@@ -3936,6 +4153,7 @@
*/
protected void logErrMsg(Throwable t)
{
+ logErrMsgForTestModeNoSync();
if (corrupt != null)
{
Monitor.logTextMessage(MessageId.LOG_BEGIN_CORRUPT_STACK);
@@ -3951,6 +4169,22 @@
}
}
+ /**
+ * In case of boot errors, and if database is either booted
+ * with derby.system.testMode being set or was previously booted in this
+ * mode, mention in the error message that the error is probably + * because the derby.system.testMode was set. + * Dont want to waste time to resolve issues in such
+ * cases
+ * <p>
+ * MT - not needed, informational only
+ */
+ protected void logErrMsgForTestModeNoSync()
+ {
+ if(logNotSynced || wasDBInTestModeNoSync)
+ Monitor.logTextMessage(MessageId.LOG_TESTMODE_NO_SYNC_ERR);
+ }
+
private void printErrorStack(Throwable t)
{
Monitor.logMessage("-------------------------\n");
Index: java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java (revision 165265)
+++ java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java (working copy)
@@ -179,11 +179,11 @@
private static final String LINE = "----------------------------------------------------------------";
- // debug only flag - disable syncing of data during page allocation.
- boolean dataNotSyncedAtAllocation;
+ // disable syncing of data during page allocation.
+ boolean dataNotSyncedAtAllocation = false;
- // debug only flag - disable syncing of data during checkpoint.
- boolean dataNotSyncedAtCheckpoint;
+ // disable syncing of data during checkpoint.
+ boolean dataNotSyncedAtCheckpoint = false;
// these fields can be accessed directly by subclasses if it needs a
// different set of actions
@@ -375,29 +375,24 @@
droppedTableStubInfo = new Hashtable();
- if (Performance.MEASURE)
+ /**
+ * If derby.system.testMode=true then set flags to + * disable sync of data pages at allocation when file is grown,
+ * disable sync of data writes during checkpoint
+ */
+ if (PropertyUtil.getSystemBoolean(Property.TESTMODE_NO_SYNC))
{
- // debug only flag - disable syncing of data during checkpoint.
- dataNotSyncedAtCheckpoint = - PropertyUtil.getSystemBoolean(
- Property.STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT);
+ // - disable syncing of data during checkpoint.
+ dataNotSyncedAtCheckpoint = true;
- // debug only flag - disable syncing of data during page allocation.
- dataNotSyncedAtAllocation = - PropertyUtil.getSystemBoolean(
- Property.STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION);
-
- if (dataNotSyncedAtCheckpoint)
- Monitor.logMessage(
- "Warning: " + - Property.STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT +
- "set to true.");
-
- if (dataNotSyncedAtAllocation)
- Monitor.logMessage(
- "Warning: " + - Property.STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION +
- "set to true.");
+ // - disable syncing of data during page allocation.
+ dataNotSyncedAtAllocation = true;
+ + // log message stating that derby.system.testMode
+ // has been enabled. + Monitor.logMessage(MessageService.getTextMessage(
+ MessageId.STORE_TESTMODE_NO_SYNC,
+ Property.TESTMODE_NO_SYNC));
}
fileHandler = new RFResource( this);
Index: java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java (revision 165265)
+++ java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java (working copy)
@@ -325,15 +325,8 @@
dataFactory.writeInProgress();
try
{
- if (Performance.MEASURE)
- {
- if (!dataFactory.dataNotSyncedAtAllocation)
- fileData.sync( false);
- }
- else
- {
+ if (!dataFactory.dataNotSyncedAtAllocation)
fileData.sync( false);
- }
}
finally
{
@@ -504,16 +497,9 @@
dataFactory.writeInProgress();
inwrite = true;
- if (Performance.MEASURE)
- {
- if (!dataFactory.dataNotSyncedAtAllocation)
- fileData.sync( false);
- }
- else
- {
+ if (!dataFactory.dataNotSyncedAtAllocation)
fileData.sync( false);
- }
- }
+ }
catch (IOException ioe)
{
// The disk may have run out of space. @@ -629,15 +615,9 @@
dataFactory.writeInProgress();
try
{
- if (Performance.MEASURE)
- {
- if (!dataFactory.dataNotSyncedAtCheckpoint)
- file.sync( false);
- }
- else
- {
- file.sync( false);
- }
+ if (!dataFactory.dataNotSyncedAtCheckpoint)
+ file.sync( false);
+
}
finally
{
Index: java/engine/org/apache/derby/iapi/reference/MessageId.java
===================================================================
--- java/engine/org/apache/derby/iapi/reference/MessageId.java (revision 165265)
+++ java/engine/org/apache/derby/iapi/reference/MessageId.java (working copy)
@@ -58,8 +58,10 @@
String LOG_CANNOT_CREATE_NEW_DUETO = "L017";
String LOG_MAYBE_INCONSISTENT = "L018";
String LOG_TRANS_LOG_SCAN = "L019";
+ String LOG_WAS_IN_TESTMODE_NO_SYNC = "L020"; // system was running in derby.system.testMode set + String LOG_TESTMODE_NO_SYNC_ERR = "L021"; // hint that error could be because + // derby.system.testMode was set
-
/*
* Raw Store data
*/
@@ -75,6 +77,7 @@
String STORE_BACKUP_ABORTED = "D010";
String STORE_REMOVED_BACKUP = "D011";
String STORE_BACKUP_COMPLETED = "D012";
+ String STORE_TESTMODE_NO_SYNC = "D013"; // for derby.system.testMode is set
/*
Index: java/engine/org/apache/derby/iapi/reference/Property.java
===================================================================
--- java/engine/org/apache/derby/iapi/reference/Property.java (revision 165265)
+++ java/engine/org/apache/derby/iapi/reference/Property.java (working copy)
@@ -368,55 +368,47 @@
<P>
*/
String STORAGE_TEMP_DIRECTORY = "derby.storage.tempDirectory";
-
+ /**
- * derby.storage.dataNotSyncedAtCheckPoint
+ * derby.system.testMode
* <p>
- * When set, the store system will not force a sync() call on the
- * containers during a checkpoint.
+ * When set, the store system will not force sync calls in the following
+ * cases + * - for the log file at each commit
+ * - for the log file before data page is forced to disk
+ * - for page allocation when file is grown
+ * - for data writes during checkpoint
* <p>
- * An internal debug system only flag. The recovery system will not
- * work properly if this flag is enabled, it is provided to do performance
- * debugging to see whether the system is I/O bound based on checkpoint
- * synchronous I/O.
+ * That means, when this property is set to true
+ * - a commit no longer guarantees that the transaction's modification
+ * will survive a system crash or JVM termination
+ * - the database may not recover successfully upon restart
+ * - a near full disk at runtime may cause unexpected errors
* <p>
- *
- **/
- public static final String STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT = - "derby.storage.dataNotSyncedAtCheckPoint";
-
- /**
- * derby.storage.dataNotSyncedAtAllocation
+ * This property is provided for performance reasons and should ideally
+ * only be used when there is no requirement for the database to recover.
+ * <BR> + * One sample use would be to enable this property when using Derby as a test
+ * database, where recoverability is not a concern but performance is important.
+ * <BR>
+ * Default value is false
+ * <BR>
+ * Valid Values
+ * Valid values are true or false
+ * <BR>
+ * Example
+ * derby.system.testMode=true
+ * One can set this as a command line option to the JVM when starting the
+ * application or in the derby.properties file
+ * <BR>
+ * This property is static; if you change it while Derby is running, + * the change does not take effect until you reboot. * <p>
- * When set, the store system will not force a sync() call on the
- * containers when pages are allocated.
- * <p>
- * An internal debug system only flag. The recovery system will not
- * work properly if this flag is enabled, it is provided to do performance
- * debugging to see whether the system is I/O bound based on page allocation
- * synchronous I/O.
- * <p>
*
**/
- public static final String STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION = - "derby.storage.dataNotSyncedAtAllocation";
-
- /**
- * derby.storage.logNotSynced
- * <p>
- * When set, the store system will not force a sync() call on the log at - * commit.
- * <p>
- * An internal debug system only flag. The recovery system will not
- * work properly if this flag is enabled, it is provided to do performance
- * debugging to see whether the system is I/O bound based on log file
- * synchronous I/O.
- * <p>
- *
- **/
- public static final String STORAGE_LOG_NOT_SYNCED = - "derby.storage.logNotSynced";
-
+ public static final String TESTMODE_NO_SYNC = + "derby.system.testMode";
+ /**
* derby.storage.fileSyncTransactionLog
* <p>
Index: java/engine/org/apache/derby/loc/messages_en.properties
===================================================================
--- java/engine/org/apache/derby/loc/messages_en.properties (revision 165265)
+++ java/engine/org/apache/derby/loc/messages_en.properties (working copy)
@@ -1150,7 +1150,10 @@
L016=cannot create a new log file {0}
L017=cannot create a new log file {0} due to {1}
L018=System may be in a inconsistent state, missing file {0}
+L020=WARNING: The database was previously booted with {0} set to true. Note that this mode {0}=true does not guarantee recoverability \n
+L021=WARNING: The database was either previously booted or is currently booted with {0} set to true. Note that this mode {0}=true does not guarantee recoverability, and is probably what might have caused this boot error \n
+
# Class Manager
C000=Wrote class {0} to file {1}. Please provide support with the file and the following exception information: {2}
C001=Class {0} has a unknown certificate type in jar {1}, expecting X509 certificate.
@@ -1174,6 +1177,7 @@
D010=Backup aborted due to error
D011=removed old backup copy at {0}
D012=Backup completed, log instant at {0} \n
+D013=WARNING: {0} is set to true. In this mode, there is no guarantee that the system will recover a system crash or JVM termination. Enable this property only when there is no need for recoverability \n
# Connectivity J004=database identity
Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java (revision 0)
+++ java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java (revision 0)
@@ -0,0 +1,202 @@
+/*
+ + Derby - Class org.apache.derbyTesting.functionTests.tests.store.TestNoSyncs
+ + Copyright 2002, 2005 The Apache Software Foundation or its licensors, as applicable.
+ + Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ + http://www.apache.org/licenses/LICENSE-2.0
+ + Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ + */
+
+package org.apache.derbyTesting.functionTests.tests.store;
+
+import java.sql.Connection;
+import java.sql.Statement;
+import java.sql.PreparedStatement;
+import java.io.*;
+/**
+ * This program tests the system when the derby.system.testMode property is enabled
+ *
+ * When the derby.system.testMode is set, the system will not do any sync to the disk and
+ * the recovery system will not work property.
+ * It is provided for performance reasons and should ideally only be used
+ * when there is no requirement for the database to recover.
+ * <p>
+ * When set, the system will not do any syncs, the cases namely
+ * - no sync of the log file at each commit
+ * - no sync of the log file before data page is forced to disk
+ * - no sync of page allocation when file is grown
+ * - no sync of data writes during checkpoint
+ * <p>
+ * That means, when this property is set to true,
+ * - a commit no longer guarantees that the transaction's modification
+ * will survive a system crash or JVM termination
+ * - the database may not recover successfully upon restart
+ * - a near full disk at runtime may cause unexpected errors
+ *
+ * This program tests for
+ * 1. enabling the derby.system.testMode is actually not doing the syncs
+ * by timing inserts
+ * 2. check if a warning message exists in derby.log
+ * 3. read log.ctrl file and check if the flag is set or not
+ *
+ *
+ * @author <a href="mailto:[EMAIL PROTECTED]">Sunitha Kambhampati</a>
+ * @version 1.0
+ */
+public class TestNoSyncMode
+{
+ public static void main(String[] args)
+ {
+ Connection conn = null;
+ Statement s = null;
+ PreparedStatement ps = null;
+ try
+ {
+ // use the ij utility to read the property file and
+ // make the initial connection.
+ org.apache.derby.tools.ij.getPropertyArg(args);
+ conn = org.apache.derby.tools.ij.startJBMS();
+ + s = conn.createStatement();
+ s.execute("create table t1 (c1 int, c2 int)");
+ s.close();
+ + // Test 1
+ // this is a quick check incase someone breaks the derby.system.testMode + long timeTaken = doInsertsInAutoCommit(conn);
+ conn.close();
+ + int approxUpperBound = 3000; // approximate upper bound in millisecond
+ if ( timeTaken > approxUpperBound )
+ {
+ report("FAIL -- derby.system.testMode mode seems to be broken.");
+ report(" Time to insert rows in test exceeded the usual limit.");
+ }
+ + // Test 2
+ // Check if derby.log has the warning message
+ report("Is warning message about derby.system.testMode present in derby.log ="+ isMessageInDerbyLog());
+ // Test 3
+ // Check if marker is correctly written out to database + markerInControlFile();
+ + + }
+ catch (Throwable e) {
+ report("FAIL -- unexpected exception: "+e);
+ e.printStackTrace();
+ }
+ + }
+ + + /**
+ * Note doing inserts in autocommit mode is probably the worst case scenario
+ * in terms of performance as each commit will involve a flush/sync to disk
+ * but in case of the derby.system.testMode=true mode, the syncs dont happen
+ * This test case times the inserts and assumes that the inserts on any system
+ * will be less than three second for 500 inserts.
+ * Note this upper bound on time is just an approximate estimation + */
+ public static long doInsertsInAutoCommit(Connection conn)
+ throws Exception
+ {
+ PreparedStatement ps = conn.prepareStatement("insert into t1 values(?,?)");
+ long count = 0;
+ + long start = System.currentTimeMillis();
+ + for ( int i = 0; i < 500; i++ )
+ {
+ ps.setInt(1,i);
+ ps.setInt(2,i);
+ count += ps.executeUpdate();
+ }
+ + long end = System.currentTimeMillis();
+ report("num successful inserts = "+count);
+ + return ( end - start);
+ }
+ + /**
+ * When derby.system.testMode is set, a warning message is written out to derby.log
+ * indicating that the property is set and that it does not guarantee recoverability
+ * This test tests if a message is written out to derby.log or not + */
+ public static boolean isMessageInDerbyLog()
+ throws Exception
+ {
+ BufferedReader reader = null;
+ File derbylog = null;
+ try
+ {
+ String derbyHome = System.getProperty("derby.system.home");
+ derbylog = new File(derbyHome,"derby.log");
+ reader = new BufferedReader(new FileReader(derbylog));
+ + String line = null;
+ while((line = reader.readLine())!= null)
+ {
+ if(line.indexOf("derby.system.testMode") != -1)
+ return true;
+ + }
+ return false;
+ }
+ finally
+ {
+ if (reader != null)
+ {
+ reader.close();
+ }
+ derbylog = null;
+ }
+ }
+
+ /**
+ * if derby.system.testMode is set, a marker is written out into log control file
+ * to recognize that the database was previously booted in this mode + * Test if the marker byte is set correctly or not
+ * See comments in org.apache.derby.impl.store.log.LogToFile for IS_TESTMODE_NO_SYNC_FLAG
+ */
+ public static void markerInControlFile()
+ throws Exception
+ {
+ RandomAccessFile controlFile = null;
+ try
+ {
+ int testModeNoSyncMarkerPosition = 29;
+ String derbyHome = System.getProperty("derby.system.home");
+ controlFile = new RandomAccessFile(derbyHome+"/wombat/log/log.ctrl","r");
+ controlFile.seek(testModeNoSyncMarkerPosition);
+ report("log.ctrl file has testMode NoSync marker value = "+ controlFile.readByte());
+ }
+ finally
+ {
+ if(controlFile != null)
+ controlFile.close();
+ + }
+ }
+ + public static void report(String msg)
+ {
+ System.out.println(msg);
+ }
+ +}
+
+
+
Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
___________________________________________________________________ Name: svn:eol-style + native
Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant (revision 165265)
+++ java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant (working copy)
@@ -23,6 +23,7 @@
TransactionTable.sql
TransactionTable_app.properties
TransactionTable_derby.properties
+TestNoSyncMode_derby.properties
access.sql
access_app.properties
access_derby.properties
Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties (revision 0)
+++ java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties (revision 0)
@@ -0,0 +1 @@
+derby.system.testMode=true
Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties
___________________________________________________________________ Name: svn:eol-style + native
Index: java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out (revision 0)
+++ java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out (revision 0)
@@ -0,0 +1,3 @@
+num successful inserts = 500
+Is warning message about derby.system.testMode present in derby.log =true
+log.ctrl file has testMode NoSync marker value = 1
Index: java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall (revision 165265)
+++ java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall (working copy)
@@ -4,3 +4,4 @@
store/longColumn.sql
store/madhare.sql
store/streamingColumn.java
+store/TestNoSyncMode.java