[PATCH] Derby 218 - Add relaxed durability option
A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. Changes are not much. svn stat M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\reference\MessageId.java M java\engine\org\apache\derby\iapi\reference\Property.java M java\engine\org\apache\derby\loc\messages_en.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties A java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall Since this is a longish email , for those who may not read the rest of it, here are the testing results Testing results: - I ran derbyall with 1.4.2 jvm and all tests passed. - I ran the storeall suite separately both with setting the derby.system.testMode property to true and without it. with this property set, the storeall suite ran about 2 times as fast. --- Below I explain how the patch addresses the above requirements and try to provide details on how I went about making those changes 0) Note 1) A single property has been added to enable this relaxed durability option. The property name is derby.system.testMode and takes in value true or false. By default it is false. It can be set as a command line option to the JVM when starting the application or within the application or in the derby.properites file. Property is loaded at boot time, so for it to have an effect you need to enable it at boot time. More comments added in code itself. see variable TESTMODE_NO_SYNC = "derby.system.testMode"; in Property.java. (In this patch, I have used 'no_sync' as suffix in variable names because I think with respect to store - that is what this mode is all about - disabling syncs for the cases already mentioned earlier in this mail) I chose the property name derby.system.testMode for this - represents a higher concept as to when one would use relaxed durability (no sync) option. - mostly it should be enabled only when data is not important and ideally maybe for testing and development purposes, - also with the hope that the word 'testMode' would prompt users to look this property up before using it in a production like environment where it may not necessarily be appropriate. Code changes for this: - Added this new property derby.system.testMode in java\engine\org\apache\derby\iapi\reference\Property.java - Removed the following properties from Property.java as they are no longer required. derby.storage.dataNotSyncedAtCheckPoint=true( I'll refer to this as DNSAC in notes below) derby.storage.dataNotSyncedAtAllocation=true ( I'll refer to this as DNSAA in notes below) derby.storage.logNotSynced=true(I'll refer to this as LNS in notes below) - Previously code existed for enabling the DNSAC, DNSAA and LNS options conditional on Performance.MEASURE So how it works is , the 2 variables in BaseDataFileFactory.java are used to control if syncs happen or not for data pages at allocation and at checkpoint. dataNotSyncedAtAllocation dataNotSyncedAtCheckpoint And in java/engine/org/apache/derby/impl/store/raw/log/LogToFile, the logNotSynced variable is used to control if syncs happen for the logs are not. These variables are enabled (set to true) if Performance.MEASURE is set to true, and if the DNSAC, and DSNAA propertie are enabled. So all that is required now is - rem
Re: [PATCH] Derby 218 - Add relaxed durability option
Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. Changes are not much. svn stat M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\reference\MessageId.java M java\engine\org\apache\derby\iapi\reference\Property.java M java\engine\org\apache\derby\loc\messages_en.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties A java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall Since this is a longish email , for those who may not read the rest of it, here are the testing results Testing results: - I ran derbyall with 1.4.2 jvm and all tests passed. - I ran the storeall suite separately both with setting the derby.system.testMode property to true and without it. with this property set, the storeall suite ran about 2 times as fast. --- Below I explain how the patch addresses the above requirements and try to provide details on how I went about making those changes 0) Note 1) A single property has been added to enable this relaxed durability option. The property name is derby.system.testMode and takes in value true or false. By default it is false. It can be set as a command line option to the JVM when starting the application or within the application or in the derby.properites file. Property is loaded at boot time, so for it to have an effect you need to enable it at boot time. More comments added in code itself. see variable TESTMODE_NO_SYNC = "derby.system.testMode"; in Property.java. (In this patch, I have used 'no_sync' as suffix in variable names because I think with respect to store - that is what this mode is all about - disabling syncs for the cases already mentioned earlier in this mail) I chose the property name derby.system.testMode for this - represents a higher concept as to when one would use relaxed durability (no sync) option. - mostly it should be enabled only when data is not important and ideally maybe for testing and development purposes, - also with the hope that the word 'testMode' would prompt users to look this property up before using it in a production like environment where it may not necessarily be appropriate. Code changes for this: - Added this new property derby.system.testMode in java\engine\org\apache\derby\iapi\reference\Property.java - Removed the following properties from Property.java as they are no longer required. derby.storage.dataNotSyncedAtCheckPoint=true( I'll refer to this as DNSAC in notes below) derby.storage.dataNotSyncedAtAllocation=true ( I'll refer to this as DNSAA in notes below) derby.storage.logNo
Re: [PATCH] Derby 218 - Add relaxed durability option
I am looking at committing this patch. Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. > ... rest deleted
Re: [PATCH] Derby 218 - Add relaxed durability option
As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also. Going forward I would like to see more work done to implement the relaxed durability option suresh and I were discussing. I believe that with some code changes file allocation, log write and many of the data write sync's can be eliminated and still guarantee a bootable database. This mode would not guarantee a committed transaction, but would guarantee that database on a disk with enough free space would boot and that the db would represent only complete transactions (either fully applied or not). This option would be slower than what is available from this patch so I believe both features are useful. David Van Couvering wrote: Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. Changes are not much. svn stat M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\reference\MessageId.java M java\engine\org\apache\derby\iapi\reference\Property.java M java\engine\org\apache\derby\loc\messages_en.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties A java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall Since this is a longish email , for those who may not read the rest of it, here are the testing results Testing results: - I ran derbyall with 1.4.2 jvm and all tests passed. - I ran the storeall suite separately both with setting the derby.system.testMode property to true and without it. with this property set, the storeall suite ran about 2 times as fast. --- Below I explain how the patch addresses the above requirements and try to provide details on how I went about making those changes 0) Note 1) A single property has been added to enable this relaxed durability option. The property name is derby.system.testMode and takes in value true or false. By default it is false. It can be set as a command line option to the JVM when starting the application or within the application or in the derby.properites file. Property is loaded at boot time, so for it to have an effect you need to enable it at boot time. More comments added in code itself. see variable TESTMODE_NO_SYNC = "derby.system.testMode"; in Property.java. (In this patch, I have used 'no_sync' as s
Re: [PATCH] Derby 218 - Add relaxed durability option
suresh, you have done a lot of work in this area could you review the patch and post your comments? Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. > ... deleted ...
Re: [PATCH] Derby 218 - Add relaxed durability option
After I wrote my email, I wondered if the feature as currently implemented wasn't quite what I was thinking we should have. I agree that if you potentially can't boot your database, this is not a production-level feature. I second the motion to see more work done to implement the relaxed durability option that guarantees a bootable database, that's what I was thinking of when I sent my previous email. Thanks, David Mike Matrigali wrote: As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also. Going forward I would like to see more work done to implement the relaxed durability option suresh and I were discussing. I believe that with some code changes file allocation, log write and many of the data write sync's can be eliminated and still guarantee a bootable database. This mode would not guarantee a committed transaction, but would guarantee that database on a disk with enough free space would boot and that the db would represent only complete transactions (either fully applied or not). This option would be slower than what is available from this patch so I believe both features are useful. David Van Couvering wrote: Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. Changes are not much. svn stat M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\reference\MessageId.java M java\engine\org\apache\derby\iapi\reference\Property.java M java\engine\org\apache\derby\loc\messages_en.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties A java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall Since this is a longish email , for those who may not read the rest of it, here are the testing results Testing results: - I ran derbyall with 1.4.2 jvm and all tests passed. - I ran the storeall suite separately both with setting the derby.system.testMode property to true and without it. with this property set, the storeall suite ran about 2 times as fast. --- Below I explain how the patch addresses the above requirements and try to provide details on how I went about making those changes 0) Note 1) A single property has been added to enable this relaxed durability option. The property name is derby.system.testMode and
Re: [PATCH] Derby 218 - Add relaxed durability option
sure. I will review it. -suresh Mike Matrigali wrote: suresh, you have done a lot of work in this area could you review the patch and post your comments? Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set. > ... deleted ...
Re: [PATCH] Derby 218 - Add relaxed durability option
This patch disables syncs for - 1)the log file at each commit - 2)the log file before data page makes it to disk - 3)data page allocation when file is grown - 4)data writes during checkpoint. Thus, when this mode is enabled, syncs are not enforcing proper WAL protocol and might lead to database not being able to boot. Also earlier when I did a search on the web for the term relaxed durability , it seemed to me that the term was used for cases when it is ok to lose committed transactions but the database should be able to boot successfully. So does derby.system.testMode seem ok given that it might not be appropriate in a production like environment... Or, maybe another name - call this mode of no syncs as derby.system.durable=none ( or derby.storage.durable=none). and in the future when someone implements the case that was discussed earlier on list (http://mail-archives.apache.org/mod_mbox/db-derby-dev/200504.mbox/[EMAIL PROTECTED] ) where only #1 and #3 syncs are disabled, that would ensure that database would boot ok, with losing some committed transactions, this setting could be derby.system.durable=partial or something similar. Any comments. Thanks, Sunitha. David Van Couvering wrote: After I wrote my email, I wondered if the feature as currently implemented wasn't quite what I was thinking we should have. I agree that if you potentially can't boot your database, this is not a production-level feature. I second the motion to see more work done to implement the relaxed durability option that guarantees a bootable database, that's what I was thinking of when I sent my previous email. Thanks, David Mike Matrigali wrote: As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also. Going forward I would like to see more work done to implement the relaxed durability option suresh and I were discussing. I believe that with some code changes file allocation, log write and many of the data write sync's can be eliminated and still guarantee a bootable database. This mode would not guarantee a committed transaction, but would guarantee that database on a disk with enough free space would boot and that the db would represent only complete transactions (either fully applied or not). This option would be slower than what is available from this patch so I believe both features are useful. David Van Couvering wrote: Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A way to report boot errors that may be because of this option being enabled. What this maps to is - have a marker to recognize that database was booted in this mode, and then on subsequent boot; if errors happen during recovery - report a message that it could have happened because of this mode being set.
Re: [PATCH] Derby 218 - Add relaxed durability option
I think testMode isn't very clear what it's doing. I agree that relaxedDurability is not correct because it doesn't communicate that you may not be able to boot the database. How about derby.system.fast-and-loose :) Seriously, how about derby.system.durability=test (you can't really say "none" because data pages are being written to disk) or derby.system.durability.testMode=true I personally like derby.system.durability=test, because then later we can add derby.system.durability=relaxed for true relaxed durability where the database is durable, and derby.system.durability=none for when we implement in-memory storage. David Sunitha Kambhampati wrote: This patch disables syncs for - 1)the log file at each commit - 2)the log file before data page makes it to disk - 3)data page allocation when file is grown - 4)data writes during checkpoint. Thus, when this mode is enabled, syncs are not enforcing proper WAL protocol and might lead to database not being able to boot. Also earlier when I did a search on the web for the term relaxed durability , it seemed to me that the term was used for cases when it is ok to lose committed transactions but the database should be able to boot successfully. So does derby.system.testMode seem ok given that it might not be appropriate in a production like environment... Or, maybe another name - call this mode of no syncs as derby.system.durable=none ( or derby.storage.durable=none). and in the future when someone implements the case that was discussed earlier on list (http://mail-archives.apache.org/mod_mbox/db-derby-dev/200504.mbox/[EMAIL PROTECTED] ) where only #1 and #3 syncs are disabled, that would ensure that database would boot ok, with losing some committed transactions, this setting could be derby.system.durable=partial or something similar. Any comments. Thanks, Sunitha. David Van Couvering wrote: After I wrote my email, I wondered if the feature as currently implemented wasn't quite what I was thinking we should have. I agree that if you potentially can't boot your database, this is not a production-level feature. I second the motion to see more work done to implement the relaxed durability option that guarantees a bootable database, that's what I was thinking of when I sent my previous email. Thanks, David Mike Matrigali wrote: As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also. Going forward I would like to see more work done to implement the relaxed durability option suresh and I were discussing. I believe that with some code changes file allocation, log write and many of the data write sync's can be eliminated and still guarantee a bootable database. This mode would not guarantee a committed transaction, but would guarantee that database on a disk with enough free space would boot and that the db would represent only complete transactions (either fully applied or not). This option would be slower than what is available from this patch so I believe both features are useful. David Van Couvering wrote: Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to do this. So for Derby 218, This patch addresses the following requirements: 1) Have a single property to enable this relaxed durability mode, so it is easy for users to enable it. 2) Print message to derby.log that this mode is enabled 3) A
Re: [PATCH] Derby 218 - Add relaxed durability option
I like this name : derby.system.durability=test I will wait for Suresh's comments and then change the name of the property after that unless someone else has a better suggestion. Thanks for your input. Sunitha. David Van Couvering wrote: I think testMode isn't very clear what it's doing. I agree that relaxedDurability is not correct because it doesn't communicate that you may not be able to boot the database. How about derby.system.fast-and-loose :) Seriously, how about derby.system.durability=test (you can't really say "none" because data pages are being written to disk) or derby.system.durability.testMode=true I personally like derby.system.durability=test, because then later we can add derby.system.durability=relaxed for true relaxed durability where the database is durable, and derby.system.durability=none for when we implement in-memory storage. David Sunitha Kambhampati wrote: This patch disables syncs for - 1)the log file at each commit - 2)the log file before data page makes it to disk - 3)data page allocation when file is grown - 4)data writes during checkpoint. Thus, when this mode is enabled, syncs are not enforcing proper WAL protocol and might lead to database not being able to boot. Also earlier when I did a search on the web for the term relaxed durability , it seemed to me that the term was used for cases when it is ok to lose committed transactions but the database should be able to boot successfully. So does derby.system.testMode seem ok given that it might not be appropriate in a production like environment... Or, maybe another name - call this mode of no syncs as derby.system.durable=none ( or derby.storage.durable=none). and in the future when someone implements the case that was discussed earlier on list (http://mail-archives.apache.org/mod_mbox/db-derby-dev/200504.mbox/[EMAIL PROTECTED] ) where only #1 and #3 syncs are disabled, that would ensure that database would boot ok, with losing some committed transactions, this setting could be derby.system.durable=partial or something similar. Any comments. Thanks, Sunitha. David Van Couvering wrote: After I wrote my email, I wondered if the feature as currently implemented wasn't quite what I was thinking we should have. I agree that if you potentially can't boot your database, this is not a production-level feature. I second the motion to see more work done to implement the relaxed durability option that guarantees a bootable database, that's what I was thinking of when I sent my previous email. Thanks, David Mike Matrigali wrote: As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also. Going forward I would like to see more work done to implement the relaxed durability option suresh and I were discussing. I believe that with some code changes file allocation, log write and many of the data write sync's can be eliminated and still guarantee a bootable database. This mode would not guarantee a committed transaction, but would guarantee that database on a disk with enough free space would boot and that the db would represent only complete transactions (either fully applied or not). This option would be slower than what is available from this patch so I believe both features are useful. David Van Couvering wrote: Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability This mode is very useful for unit testing or at development time when recoverability is not required. Basically in this mode, syncs are disabled for logs, data writes at checkpoint, page allocation when file is grown; - what this means is that data is not flushed all the way to the disk and in most cases i/o cost is not involved. Note, code already exists in Derby to d
Re: [PATCH] Derby 218 - Add relaxed durability option
this seems fine with me. I agree none does not seem right, none seems more appropriate if an in memory implementation is ever provided - though derby.system.durability=memory seems better than none in that case. I like that a setting that says "test" forces someone to think about what they are setting - and hopefully look up what the consequences can be. Sunitha Kambhampati wrote: I like this name : derby.system.durability=test I will wait for Suresh's comments and then change the name of the property after that unless someone else has a better suggestion. Thanks for your input. Sunitha. David Van Couvering wrote: I think testMode isn't very clear what it's doing. I agree that relaxedDurability is not correct because it doesn't communicate that you may not be able to boot the database. How about derby.system.fast-and-loose :) Seriously, how about derby.system.durability=test (you can't really say "none" because data pages are being written to disk) or derby.system.durability.testMode=true I personally like derby.system.durability=test, because then later we can add derby.system.durability=relaxed for true relaxed durability where the database is durable, and derby.system.durability=none for when we implement in-memory storage. David Sunitha Kambhampati wrote: This patch disables syncs for - 1)the log file at each commit - 2)the log file before data page makes it to disk - 3)data page allocation when file is grown - 4)data writes during checkpoint. Thus, when this mode is enabled, syncs are not enforcing proper WAL protocol and might lead to database not being able to boot. Also earlier when I did a search on the web for the term relaxed durability , it seemed to me that the term was used for cases when it is ok to lose committed transactions but the database should be able to boot successfully. So does derby.system.testMode seem ok given that it might not be appropriate in a production like environment... Or, maybe another name - call this mode of no syncs as derby.system.durable=none ( or derby.storage.durable=none). and in the future when someone implements the case that was discussed earlier on list (http://mail-archives.apache.org/mod_mbox/db-derby-dev/200504.mbox/[EMAIL PROTECTED] ) where only #1 and #3 syncs are disabled, that would ensure that database would boot ok, with losing some committed transactions, this setting could be derby.system.durable=partial or something similar. Any comments. Thanks, Sunitha. David Van Couvering wrote: After I wrote my email, I wondered if the feature as currently implemented wasn't quite what I was thinking we should have. I agree that if you potentially can't boot your database, this is not a production-level feature. I second the motion to see more work done to implement the relaxed durability option that guarantees a bootable database, that's what I was thinking of when I sent my previous email. Thanks, David Mike Matrigali wrote: As I have posted before I have concerns about this mode, as it can definitely produce a database which can not boot. But the discussions here convinced me that it was useful for a number of applications, so now believe it should be checked in. I definitely have no career path in marketing as my first take at a name was corrupt_your_db=true :-) . So testMode seemed ok to me, but if another name is better that is fine with me also. Going forward I would like to see more work done to implement the relaxed durability option suresh and I were discussing. I believe that with some code changes file allocation, log write and many of the data write sync's can be eliminated and still guarantee a bootable database. This mode would not guarantee a committed transaction, but would guarantee that database on a disk with enough free space would boot and that the db would represent only complete transactions (either fully applied or not). This option would be slower than what is available from this patch so I believe both features are useful. David Van Couvering wrote: Hi, Sunitha. Thanks for your work on this. My comment is more a "marketing" comment than a technical one. I disagree that this is useful only for testing and development. As I mentioned in previous emails, there are a number of customers who may find this very useful indeed, who are willing to trade off 100% guarantee of all transactions being available on recovery for significant increases in performance. I wouldn't get into this debate were it not for the fact that the property you have created is called "testMode." I would much rather have it not imply it's only for testing, with a name like "delayedWrite" or "relaxedDurability" or some such. Thanks, David Sunitha Kambhampati wrote: A little background: Sometime earlier on the list, Dan posted a fix to make derby go faster with relaxed durability with some flags. The post is at http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durab
Re: [PATCH] Derby 218 - Add relaxed durability option
David Van Couvering wrote: > I think testMode isn't very clear what it's doing. I agree that > relaxedDurability is not correct because it doesn't communicate that you > may not be able to boot the database. > > How about derby.system.fast-and-loose :) > > Seriously, how about > > derby.system.durability=test > > (you can't really say "none" because data pages are being written to disk) but they are not guaranteed to be written to disk, that's what I meant by 'none' originally. I personally think 'test' describes how it might be used and doesn't describe the durability, which is what the value of such a property should be doing. However 'Test' is probably fine, another term could be added if someone comes up with a better description, so that something like these two options would mean the same thing derby.system.durability=test derby.system.durability=all_bets_are_off Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
My two cents: How about cryptic one like derby.system.durabilityLevel = 0 0 - mean none. That way some one using it will be forced to read the doc and find out what it mean :-) -suresht Daniel John Debrunner wrote: David Van Couvering wrote: I think testMode isn't very clear what it's doing. I agree that relaxedDurability is not correct because it doesn't communicate that you may not be able to boot the database. How about derby.system.fast-and-loose :) Seriously, how about derby.system.durability=test (you can't really say "none" because data pages are being written to disk) but they are not guaranteed to be written to disk, that's what I meant by 'none' originally. I personally think 'test' describes how it might be used and doesn't describe the durability, which is what the value of such a property should be doing. However 'Test' is probably fine, another term could be added if someone comes up with a better description, so that something like these two options would mean the same thing derby.system.durability=test derby.system.durability=all_bets_are_off Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
Hi Sunitha,
Good comments. I am surprised to find that this patch doing more than
what I imagined it will do, your changes
seems to allow switching between durable test mode(No syncs) and the
default durability level (all sync). I don't get why
these mode switch functionality is required, if this mode is just meant
to run the tests faster in non-durable mode! . Not sure what
kind of new problem scenarios this mode switching will introduce.
But for some reason if some one wants, I am not sure your fix will
actually make a non-durable database
to durable/consistent one correctly. What I learn from your changes
is that data is not synced to disk on checkpoints,
if you don't do that you can not assume database is is in consistent
state if recovery went through fine. Let's say:
1) user started the database with derby.system.testMode=true
2) did some insert/update/deletes ...
3) crashed immediately after the checpoint record got on to disk...
4) Now user switches derby.system.testMode=false and boot the database.
Recovery will work fine because there is no log to replay. So you
unmark the
Test mode flag in the control file and declare that this database is in
durable/consistent mode.
But that is not the case because only some of the data pages might
have reached to the
disk before the crash, Which will actually leave the database
inconsistent state.
(This is different from losing some transactions).
Some general comments on the code:
1) Why don't we just introduce a new flag private static final byte
TESTMODE_NO_SYNC_TRUE = 0x2 and
write as part of the "flags" byte which is currently used to
indicate the IS_BETA_FLAG instead of using
another spare byte.
2) in few places indentation is not matching the rest of the code(
that's how my emacs shows it):
a)
// if system is running with derby.system.testMode enabled
// log records and log file are not synced, so
// set flag to disable log syncs
b)
{
// print message stating that the database was
// previously running with derby.system.testMode=true
// mode
c) if( wasDBInTestModeNoSync && !logNotSynced )
{
// write to log control file and the mirror file that the testmode
// is not used now
d)
if(wasDBInTestModeNoSync)
{
// print message stating that the database was
// previously running with derby.system.testMode=true
..etc.
3) I think the logical place for the method private boolean
updateTestModeFlagInCtrlFile(byte value)
is next to writeControlFile(...) .
4) protected logErrMsgForTestModeNoSync() may be marked as a private.
Thanks
-suresht
Sunitha Kambhampati wrote:
A little background: Sometime earlier on the list, Dan posted a fix to
make derby go faster with relaxed durability with some flags. The
post is at
http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability
This mode is very useful for unit testing or at development time when
recoverability is not required.
Basically in this mode, syncs are disabled for logs, data writes at
checkpoint, page allocation when file is grown; - what this means is
that data is not flushed all the way to the disk and in most cases i/o
cost is not involved. Note, code already exists in Derby to do this.
So for Derby 218, This patch addresses the following requirements:
1) Have a single property to enable this relaxed durability mode, so
it is easy for users to enable it.
2) Print message to derby.log that this mode is enabled
3) A way to report boot errors that may be because of this option
being enabled. What this maps to is - have a marker to recognize that
database was booted in this mode, and then on subsequent boot; if
errors happen during recovery - report a message that it could have
happened because of this mode being set.
Ch
Re: [PATCH] Derby 218 - Add relaxed durability option
I think that once the db is run in this mode it should be marked, and
not worry about ever unmarking it. At first I just thought this was
necessary to explain lost transactions, but I agree with the case below
- successful recovery/booting does not guarantee consistent database.
The case that you describe below can easily result in an index entry
pointing to a non-existent base table row - with all the subsequent
runtime errors later on.
So I would lean toward changing to single flag, set once and never
untoggled. Change the property name to something like
derby.system.durability=test. Fix the comments below. resubmit
patch.
If anyone else is planning on testing/review the patch let the list
know.
/mikem
Suresh Thalamati wrote:
> Hi Sunitha,
>
> Good comments. I am surprised to find that this patch doing more than
> what I imagined it will do, your changes
> seems to allow switching between durable test mode(No syncs) and the
> default durability level (all sync). I don't get why
> these mode switch functionality is required, if this mode is just meant
> to run the tests faster in non-durable mode! . Not sure what
> kind of new problem scenarios this mode switching will introduce.
>
> But for some reason if some one wants, I am not sure your fix will
> actually make a non-durable database
> to durable/consistent one correctly. What I learn from your changes
> is that data is not synced to disk on checkpoints,
> if you don't do that you can not assume database is is in consistent
> state if recovery went through fine. Let's say:
>
> 1) user started the database with derby.system.testMode=true
> 2) did some insert/update/deletes ...
> 3) crashed immediately after the checpoint record got on to disk...
> 4) Now user switches derby.system.testMode=false and boot the database.
>
> Recovery will work fine because there is no log to replay. So you
> unmark the
> Test mode flag in the control file and declare that this database is in
> durable/consistent mode.
> But that is not the case because only some of the data pages might
> have reached to the
> disk before the crash, Which will actually leave the database
> inconsistent state.
> (This is different from losing some transactions).
>
> Some general comments on the code:
> 1) Why don't we just introduce a new flag private static final byte
> TESTMODE_NO_SYNC_TRUE = 0x2 and
> write as part of the "flags" byte which is currently used to
> indicate the IS_BETA_FLAG instead of using
> another spare byte.
> 2) in few places indentation is not matching the rest of the code(
> that's how my emacs shows it):
>a)
> // if system is running with derby.system.testMode enabled
>// log records and log file are not synced, so
>// set flag to disable log syncs
> b)
> {
>// print message stating that the database was
>// previously running with derby.system.testMode=true
>// mode
> c) if( wasDBInTestModeNoSync && !logNotSynced )
>{
>// write to log control file and the mirror file that the testmode
>// is not used now
> d) if(wasDBInTestModeNoSync)
>{
>// print message stating that the database was
>// previously running with derby.system.testMode=true
>..etc.
>
> 3) I think the logical place for the method private boolean
> updateTestModeFlagInCtrlFile(byte value)
> is next to writeControlFile(...) .
>
> 4) protected logErrMsgForTestModeNoSync() may be marked as a private.
>
>
> Thanks
> -suresht
>
>
> Sunitha Kambhampati wrote:
>
>> A little background: Sometime earlier on the list, Dan posted a fix to
>> make derby go faster with relaxed durability with some flags. The
>> post is at
>> http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability
>>
>> This mode is very useful for unit testing or at development time when
>> recoverability is not required.
>> Basically in this mode, syncs are disabled for logs, data writes at
>> checkpoint, page allocation when file is grown; - what this means is
>> that data is not flushed all the way to the disk and in most cases i/o
>> cost is not involved. Note, code already exists in Derby to do this.
>> So for Derby 218, This patch addresses the following requirements:
>> 1) Have a single property to enable this relaxed durability mode, so
>> it is easy for users to enable it.
>> 2) Print message to derby.log that this mode is enabled
>> 3) A way to report boot errors that may be because of this option
>> being enabled. What this maps to is - have a marker to recognize that
>> database was booted in this mode, and then on subsequent boot; if
>> errors happen during recovery - report a message that it could have
>> happened because of this mode being set.
>>
>> Ch
>
>
>
>
>
Re: [PATCH] Derby 218 - Add relaxed durability option
Thanks Suresh for your comments. I have attached a new patch that takes
care of them and as well as the property name change as discussed on the
list.
So now,
1)To enable this no syncs mode, the property setting is
derby.system.durability=test
If it is set to any other value other than test, then this property
setting is ignored and the system will default to the usual mode where
all syncs happen correctly.
2) If database is booted in this mode then the flag to indicate this
mode is being used is set in log.ctrl file and it will not change after
that.Thus there wont be any switching between this mode ( no syncs) to
default ( all syncs) mode because we cannot guarantee that the database
will be in a consistent state once its booted in this no sync mode..
3) This flag is now written out as part of the log control flags in
log.ctrl file instead of using a spare byte.
___
svn stat:
M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
M
java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
M java\engine\org\apache\derby\iapi\reference\MessageId.java
M java\engine\org\apache\derby\iapi\reference\Property.java
M java\engine\org\apache\derby\loc\messages_en.properties
A
java\testing\org\apache\derbyTesting\functionTests\tests\store\TestDurabilityProperty.java
A
java\testing\org\apache\derbyTesting\functionTests\master\TestDurabilityProperty.out
M
java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall
I use eclipse IDE and have it set to put 4 spaces for tab ( as per
discussion on formatting on the list). I did verify by opening the files
in viw ( with tab set to 4) and the formatting looks ok. But even when
I sent the patch last time, in eclipse the formatting looked ok, so not
totally sure what may be happening. But if someone finds any issue with
formatting please let me know.
Can someone review it and if it looks ok, can a committer commit it.
Thanks,
Sunitha.
Mike Matrigali wrote:
<>I think that once the db is run in this mode it should be marked, and
not worry about ever unmarking it. At first I just thought this was
necessary to explain lost transactions, but I agree with the case below
- successful recovery/booting does not guarantee consistent database.
The case that you describe below can easily result in an index entry
pointing to a non-existent base table row - with all the subsequent
runtime errors later on.
So I would lean toward changing to single flag, set once and never
untoggled. ...
<>Suresh Thalamati wrote:
Hi Sunitha,
Good comments. I am surprised to find that this patch doing more than
what I imagined it will do, your changes
seems to allow switching between durable test mode(No syncs) and the
default durability level (all sync). I don't get why
these mode switch functionality is required, if this mode is just meant
to run the tests faster in non-durable mode! . Not sure what
kind of new problem scenarios this mode switching will introduce.
But for some reason if some one wants, I am not sure your fix will
actually make a non-durable database
to durable/consistent one correctly. What I learn from your changes
is that data is not synced to disk on checkpoints,
if you don't do that you can not assume database is is in consistent
state if recovery went through fine. Let's say:
1) user started the database with derby.system.testMode=true
2) did some insert/update/deletes ...
3) crashed immediately after the checpoint record got on to disk...
4) Now user switches derby.system.testMode=false and boot the database.
Recovery will work fine because there is no log to replay. So you
unmark the
Test mode flag in the control file and declare that this database is in
durable/consistent mode.
But that is not the case because only some of the data pages might
have reached to the
disk before the crash, Which will actually leave the database
inconsistent state.
(This is different from losing some transactions).
Some general comments on the code:
1) Why don't we just introduce a new flag private static final byte
TESTMODE_NO_SYNC_TRUE = 0x2 and
write as part of the "flags" byte which is currently used to
indicate the IS_BETA_FLAG instead of using
another spare byte.
2) in few places indentation is not matching the rest of the code(
that's how my emacs shows it):
a)
// if system is running with derby.system.testMode enabled
// log records and log file are not synced, so
// set flag to disable log syncs
b)
{
// print message stating that the database was
// previously running with derby.system.testMode=true
// mode
c) if( wasDBInTestModeNoSync && !logNotSynced )
{
// write to log control file and the mirror file that the testmode
// is not used no
Re: [PATCH] Derby 218 - Add relaxed durability option
Could you resolve the conflict in LogToFile.java in the current trunk revision and resubmit the patch? Sunitha Kambhampati wrote: Thanks Suresh for your comments. I have attached a new patch that takes care of them and as well as the property name change as discussed on the list. So now, 1)To enable this no syncs mode, the property setting is derby.system.durability=test If it is set to any other value other than test, then this property setting is ignored and the system will default to the usual mode where all syncs happen correctly. 2) If database is booted in this mode then the flag to indicate this mode is being used is set in log.ctrl file and it will not change after that.Thus there wont be any switching between this mode ( no syncs) to default ( all syncs) mode because we cannot guarantee that the database will be in a consistent state once its booted in this no sync mode.. 3) This flag is now written out as part of the log control flags in log.ctrl file instead of using a spare byte. ___ svn stat: M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\reference\MessageId.java M java\engine\org\apache\derby\iapi\reference\Property.java M java\engine\org\apache\derby\loc\messages_en.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestDurabilityProperty.java A java\testing\org\apache\derbyTesting\functionTests\master\TestDurabilityProperty.out M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall
Re: [PATCH] Derby 218 - Add relaxed durability option
Mike Matrigali wrote: Could you resolve the conflict in LogToFile.java in the current trunk revision and resubmit the patch? I resolved the conflict and regenerated the patch(DurabilityTest_0510.txt). Let me know if you have any problems applying this patch. Thanks, Sunitha. Sunitha Kambhampati wrote: Thanks Suresh for your comments. I have attached a new patch that takes care of them and as well as the property name change as discussed on the list. So now, 1)To enable this no syncs mode, the property setting is derby.system.durability=test If it is set to any other value other than test, then this property setting is ignored and the system will default to the usual mode where all syncs happen correctly. 2) If database is booted in this mode then the flag to indicate this mode is being used is set in log.ctrl file and it will not change after that.Thus there wont be any switching between this mode ( no syncs) to default ( all syncs) mode because we cannot guarantee that the database will be in a consistent state once its booted in this no sync mode.. 3) This flag is now written out as part of the log control flags in log.ctrl file instead of using a spare byte. ___ svn stat: M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\reference\MessageId.java M java\engine\org\apache\derby\iapi\reference\Property.java M java\engine\org\apache\derby\loc\messages_en.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\TestDurabilityProperty.java A java\testing\org\apache\derbyTesting\functionTests\master\TestDurabilityProperty.out M java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall Index: java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java === --- java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (revision 169506) +++ java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (working copy) @@ -265,7 +265,32 @@ /* Log Control file flags. */ private static final byte IS_BETA_FLAG = 0x1; - + +/** + * When the derby.system.durability property is set to 'test', the store + * system will not force sync calls in the following cases + * - for the log file at each commit + * - for the log file before data page is forced to disk + * - for page allocation when file is grown + * - for data writes during checkpoint + * This means it is possible that the recovery system may not work properly, + * committed transactions may be lost, and/or database may not + * be in a consistent state. + * In order that we recognize this case that the database was previously + * at any time booted in this mode, this value is written out + * into the log control file. This will help prevent us from + * wasting time to resolve issues in such cases. + * @see org.apache.derby.iapi.reference.Property#DURABILITY_PROPERTY + * This value is written as part of the log control file flags byte. + */ +private static final byte IS_DURABILITY_TESTMODE_NO_SYNC_FLAG = 0x2; + +/** + * keeps track of if the database was booted previously at any time with + * derby.system.durability=test + */ +private static boolean wasDBInDurabilityTestModeNoSync = false; + /* to err on the conservative side, unless otherwise set, assume log * archive is ON */ @@ -362,7 +387,7 @@ // log goes to the log subdirectory underneath the data directory String logDevice; -// debug only flag - disable syncing of log file for debugging performance. +// disable syncing of log file when running in derby.system.durability=test private boolean logNotSynced = false; private boolean logArchived = false; @@ -403,8 +428,7 @@ private CRC32 checksum = new CRC32(); // holder for the checksum - - + /** * Note: Why logging system support file sync and write sync ? * Note : The reason to support file and write sync of logs is @@ -1194,8 +1218,8 @@ tf.resetTranId(); } - // done with recovery - +// done with recovery + / // setup checktpoint daemon / @@ -1207,6 +1231,7 @@ } } + /** Checkpoint the rawStore. @@ -2138,14 +2163,27 @@ daos.writeInt(jbmsVersion.getBuildNumberAsInt()); byte flag
Re: [PATCH] Derby 218 - Add relaxed durability option
I committed this change with svn 169538. Sunitha Kambhampati wrote: Mike Matrigali wrote: Could you resolve the conflict in LogToFile.java in the current trunk revision and resubmit the patch? I resolved the conflict and regenerated the patch(DurabilityTest_0510.txt). Let me know if you have any problems applying this patch. Thanks, Sunitha. Sunitha Kambhampati wrote: Thanks Suresh for your comments. I have attached a new patch that takes care of them and as well as the property name change as discussed on the list. So now, 1)To enable this no syncs mode, the property setting is derby.system.durability=test If it is set to any other value other than test, then this property setting is ignored and the system will default to the usual mode where all syncs happen correctly. 2) If database is booted in this mode then the flag to indicate this mode is being used is set in log.ctrl file and it will not change after that.Thus there wont be any switching between this mode ( no syncs) to default ( all syncs) mode because we cannot guarantee that the database will be in a consistent state once its booted in this no sync mode.. 3) This flag is now written out as part of the log control flags in log.ctrl file instead of using a spare byte. ___ deleted rest ...
Re: [PATCH] Derby 218 - Add relaxed durability option
Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? Thanks, David Mike Matrigali wrote: I committed this change with svn 169538. Sunitha Kambhampati wrote: Mike Matrigali wrote: Could you resolve the conflict in LogToFile.java in the current trunk revision and resubmit the patch? I resolved the conflict and regenerated the patch(DurabilityTest_0510.txt). Let me know if you have any problems applying this patch. Thanks, Sunitha. Sunitha Kambhampati wrote: Thanks Suresh for your comments. I have attached a new patch that takes care of them and as well as the property name change as discussed on the list. So now, 1)To enable this no syncs mode, the property setting is derby.system.durability=test If it is set to any other value other than test, then this property setting is ignored and the system will default to the usual mode where all syncs happen correctly. 2) If database is booted in this mode then the flag to indicate this mode is being used is set in log.ctrl file and it will not change after that.Thus there wont be any switching between this mode ( no syncs) to default ( all syncs) mode because we cannot guarantee that the database will be in a consistent state once its booted in this no sync mode.. 3) This flag is now written out as part of the log control flags in log.ctrl file instead of using a spare byte. ___ deleted rest ...
Re: [PATCH] Derby 218 - Add relaxed durability option
David Van Couvering wrote: Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? To enable this mode for derbyall suite. put the following property in java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties derby.system.durability=test build it ( ant testing). Then I believe, when you run derbyall, this property will be picked up by all the tests run as part of derbyall. Sunitha. Mike Matrigali wrote: I committed this change with svn 169538. Sunitha Kambhampati wrote: Mike Matrigali wrote: Could you resolve the conflict in LogToFile.java in the current trunk revision and resubmit the patch? I resolved the conflict and regenerated the patch(DurabilityTest_0510.txt). Let me know if you have any problems applying this patch. Thanks, Sunitha. Sunitha Kambhampati wrote: Thanks Suresh for your comments. I have attached a new patch that takes care of them and as well as the property name change as discussed on the list. So now, 1)To enable this no syncs mode, the property setting is derby.system.durability=test If it is set to any other value other than test, then this property setting is ignored and the system will default to the usual mode where all syncs happen correctly. 2) If database is booted in this mode then the flag to indicate this mode is being used is set in log.ctrl file and it will not change after that.Thus there wont be any switching between this mode ( no syncs) to default ( all syncs) mode because we cannot guarantee that the database will be in a consistent state once its booted in this no sync mode.. 3) This flag is now written out as part of the log control flags in log.ctrl file instead of using a spare byte. ___ deleted rest ...
Re: [PATCH] Derby 218 - Add relaxed durability option
Sunitha Kambhampati wrote: David Van Couvering wrote: Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? To enable this mode for derbyall suite. put the following property in java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties derby.system.durability=test build it ( ant testing). Then I believe, when you run derbyall, this property will be picked up by all the tests run as part of derbyall. Also, if you want to doublecheck if this property is picked up correctly or not, in the window where you are running the derbyall suite, you should see the derby.system.durability=test property passed as part of testSpecialProps for the tests.. Execute command: java -DjavaCmd=java -Doutputdir=C:\workghm\svnclient\trunk\test ing\derbyall\derbylang -Dtopsuitedir=C:\workghm\svnclient\trunk\testing\derbyall -Dtopreportdir=C:\workghm\svnclient\trunk\testing\derbyall -Drundir=C:\workghm\ svnclient\trunk\testing -DtestSpecialProps=derby.debug.true=enableBtreeConsisten cyCheck^*derby.system.durability=test*^derby.stream.error.logSeverityLevel=0 -Dsui tename=derbyall:derbylang -Dtopsuitename=derbyall org.apache.derbyTesting.functi onTests.harness.RunTest lang/errorCode.sql Also, you should also see the following warning in derby.log : " WARNING: The database is booted with derby.system.durability=test. In this mode, it is possible that database may not be able to recover, committed transactions may be lost, database may be in an inconsistent state. Please use this mode only when these consequences are acceptable " _ Sunitha.
Re: [PATCH] Derby 218 - Add relaxed durability option
Sunitha Kambhampati wrote: > David Van Couvering wrote: > >> Great! I have a question: if I wanted to use this to speed up >> testing, how do I do it within the context of the test harness? >> > To enable this mode for derbyall suite. > put the following property in > java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties > > derby.system.durability=test I think this will work as an alternative, and means you don't have to modify any files java -Dderby.system.durability=test -Djvmflags="-Dderby.system.durability=test" org.apache.derbyTesting.functionTests.harness.RunSuite derbyall The -Dderby.system.durability to the java command ensures the property will be set for any tests run in-process, e.g. nist The setting in the jvmflags ensures it is passed on to the JVM's spawned to run most tests. Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
let us know if this works. There are some recovery/crash tests in the full suite. My guess is that they will work since the I/O is sent to the OS, just not sync's -- but they are not guaranteed to work under this mode. Sunitha Kambhampati wrote: David Van Couvering wrote: Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? To enable this mode for derbyall suite. put the following property in java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties derby.system.durability=test build it ( ant testing). Then I believe, when you run derbyall, this property will be picked up by all the tests run as part of derbyall. Sunitha. Mike Matrigali wrote: I committed this change with svn 169538. Sunitha Kambhampati wrote: Mike Matrigali wrote: Could you resolve the conflict in LogToFile.java in the current trunk revision and resubmit the patch? I resolved the conflict and regenerated the patch(DurabilityTest_0510.txt). Let me know if you have any problems applying this patch. Thanks, Sunitha. Sunitha Kambhampati wrote: Thanks Suresh for your comments. I have attached a new patch that takes care of them and as well as the property name change as discussed on the list. So now, 1)To enable this no syncs mode, the property setting is derby.system.durability=test If it is set to any other value other than test, then this property setting is ignored and the system will default to the usual mode where all syncs happen correctly. 2) If database is booted in this mode then the flag to indicate this mode is being used is set in log.ctrl file and it will not change after that.Thus there wont be any switching between this mode ( no syncs) to default ( all syncs) mode because we cannot guarantee that the database will be in a consistent state once its booted in this no sync mode.. 3) This flag is now written out as part of the log control flags in log.ctrl file instead of using a spare byte. ___ deleted rest ...
Re: [PATCH] Derby 218 - Add relaxed durability option
Thanks. Are there any tests that *shouldn't* be run in this mode, e.g. recovery tests? David Daniel John Debrunner wrote: Sunitha Kambhampati wrote: David Van Couvering wrote: Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? To enable this mode for derbyall suite. put the following property in java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties derby.system.durability=test I think this will work as an alternative, and means you don't have to modify any files java -Dderby.system.durability=test -Djvmflags="-Dderby.system.durability=test" org.apache.derbyTesting.functionTests.harness.RunSuite derbyall The -Dderby.system.durability to the java command ensures the property will be set for any tests run in-process, e.g. nist The setting in the jvmflags ensures it is passed on to the JVM's spawned to run most tests. Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
FYI, your suggestion worked great on XP, but on Linux I got a strange error in tests that spawn another JVM: Exception in thread "main" java.lang.NoClassDefFoundError: "-Dderby/system/durability=test" I am will try to track this down, but meanwhile I'm modifying derbyall.properties... David Daniel John Debrunner wrote: Sunitha Kambhampati wrote: David Van Couvering wrote: Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? To enable this mode for derbyall suite. put the following property in java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties derby.system.durability=test I think this will work as an alternative, and means you don't have to modify any files java -Dderby.system.durability=test -Djvmflags="-Dderby.system.durability=test" org.apache.derbyTesting.functionTests.harness.RunSuite derbyall The -Dderby.system.durability to the java command ensures the property will be set for any tests run in-process, e.g. nist The setting in the jvmflags ensures it is passed on to the JVM's spawned to run most tests. Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
FYI, your suggestion worked great on XP, but on Linux I got a strange error in tests that spawn another JVM: Exception in thread "main" java.lang.NoClassDefFoundError: "-Dderby/system/durability=test" I am working on tracking this down, but meanwhile I'm modifying derbyall.properties... David Daniel John Debrunner wrote: Sunitha Kambhampati wrote: David Van Couvering wrote: Great! I have a question: if I wanted to use this to speed up testing, how do I do it within the context of the test harness? To enable this mode for derbyall suite. put the following property in java\testing\org\apache\derbyTesting\functionTests\suites\derbyall.properties derby.system.durability=test I think this will work as an alternative, and means you don't have to modify any files java -Dderby.system.durability=test -Djvmflags="-Dderby.system.durability=test" org.apache.derbyTesting.functionTests.harness.RunSuite derbyall The -Dderby.system.durability to the java command ensures the property will be set for any tests run in-process, e.g. nist The setting in the jvmflags ensures it is passed on to the JVM's spawned to run most tests. Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
David Van Couvering wrote: > FYI, your suggestion worked great on XP, but on Linux I got a strange > error in tests that spawn another JVM: > > Exception in thread "main" java.lang.NoClassDefFoundError: > "-Dderby/system/durability=test" Getting the quoting to work when passing setting jvmflags seems a major pain. Take a look at the j9 thread to see how I managed to get jvmflags to work when it included multiple -D options, may help, but it was on XP as well. You can use -Dverbose=true on the outer jvm to see the command line for the spawned jvm, that might give clues. Now if we used JUnit this wouldn't be an issue ... (I'll just keep repeating this ... :-) Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
Thanks, Dan. Yes, I would love to move to JUnit too, that has my vote. David Daniel John Debrunner wrote: David Van Couvering wrote: FYI, your suggestion worked great on XP, but on Linux I got a strange error in tests that spawn another JVM: Exception in thread "main" java.lang.NoClassDefFoundError: "-Dderby/system/durability=test" Getting the quoting to work when passing setting jvmflags seems a major pain. Take a look at the j9 thread to see how I managed to get jvmflags to work when it included multiple -D options, may help, but it was on XP as well. You can use -Dverbose=true on the outer jvm to see the command line for the spawned jvm, that might give clues. Now if we used JUnit this wouldn't be an issue ... (I'll just keep repeating this ... :-) Dan.
Re: [PATCH] Derby 218 - Add relaxed durability option
> "SK" == Sunitha Kambhampati <[EMAIL PROTECTED]> writes: SK> A little background: Sometime earlier on the list, Dan posted a fix to SK> make derby go faster with relaxed durability with some flags. The SK> post is at SK> http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability I was originally planning to review this before it was committed, but I did not get around to sending my comments before I went on a vacation. Even if it is a bit late, here are some questions/comments to this patch and the follow-up patch for the test program: - I am not sure about the value of this mode compared to a mode with relaxed durability, but guaranteed consistency. The large performance improvements comes from not syncing transaction commit. How much extra performance do you really get from not doing one sync per checkpoint? - Unless I am missing something, it does not seem more difficult to implement the alternative that guarantees consistency. Do you have any plans to do that? - With respect to upgrade, does previous versions properly initialize the spare byte in the log.ctrl file? Otherwise, one could risk false detections on upgrade. - I am a bit sceptical to the value of having a test that tests that this feature give a certain performance advantage. I have seen that over time such tests often have false failures when run on new platforms. I see that your latest patch addresses this. However, should the performance of disk io increase beyond your threshold, the test would not be able to detect any failures to turn off syncing. I think that a performance regression test would be better for this prupose. - The test will only test whether the syncing at commit is turned off. The other syncs will not decrease performance so much that it would be detected by the threshold of the test. -- Øystein
Re: [PATCH] Derby 218 - Add relaxed durability option
Thanks for the feedback. Øystein Grøvlen wrote: <>Even if it is a bit late, here are some questions/commentsto this patch and the follow-up patch for the test program: - I am not sure about the value of this mode compared to a mode with relaxed durability, but guaranteed consistency. The large performance improvements comes from not syncing transaction commit. How much extra performance do you really get from not doing one sync per checkpoint? I think both the options 1) derby.system.durability=test 2)actual relaxed durability with consistency and recoverability are useful. I would probably use 1 for testing purposes and earlier responses to the list seemed to indicate #1 was good to support. I agree with you that #2 is a good and useful option to have and I think other databases support this option. <> - Unless I am missing something, it does not seem more difficult to implement the alternative that guarantees consistency. Do you have any plans to do that? I have filed a jira issue (Derby-306). But I am tied up with other things at the moment so I dont have plans to do this right now. <> - With respect to upgrade, does previous versions properly initialize the spare byte in the log.ctrl file? Otherwise, one could risk false detections on upgrade. -- the spare byte (that I used in my initial patch) is initialized correctly in previous version., but in the final patch per review comments - the log control flags byte was used to store this flag and yes it is initialized in previous versions correctly. So I think there should not be any false detections on upgrade. <> - I am a bit sceptical to the value of having a test that tests that this feature give a certain performance advantage. I have seen that over time such tests often have false failures when run on new platforms. I see that your latest patch addresses this. However, should the performance of disk io increase beyond your threshold, the test would not be able to detect any failures to turn off syncing.I think that a performance regression test would be better for this prupose. Yes, you are right. The best way to detect issues with this case would be a performance regression test. This test was (is) tricky :) specially adding the timing part as a functional test.. But I think currently what the test does will atleast send off an alarm if the inserts take really long. I added lots of comments in the test and as well as when the test might fail with warnings about it being approximate times. But since we dont require any perf test to be run when submitting patches, I think maybe this test is probably ok for now .. I welcome any suggestions on this to make it better.. Thanks, Sunitha.
