[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_6.diff DERBY -1156 (partial) (re) encryption of an existing database. This patch adds code required to correctly handle crash/recovery during encryption of an un-encrypted database and encryption of an encrypted database with a new key. Crashes before updating the database with the new encryption key are handled by recovery, those log records are encrypted old key. Crashes after updates to the service.properties with a new encryption key is handled by using the new status flag (derby.storage.databaseEncryptionStatus) to track the changes and by writing the transaction log encrypted with new key to a new log file. Logging system can handle only one encryption key, which means the recovery can read the log with a new key or the old key. Checkpoint log record is also encrypted. So it is necessary for the system to find a checkpoint in the log that it can decrypt using the encryption key stored in the service.properties or the one provided by the user. This is ensured by by writing COMMIT and CHECKPOINT log records into a new log file and delete that file on-reboot if there is a crash before checkpoint records are updated. CHECKPOINT and COMMIT is done after setting the databaseEncryptionStatus to IN-PROGRESS in the service.properties.On a reboot if databaseEncryptionStatus is IN-PROGRESS, then engine first checks if checkpoint is in the last log file , it it is then (re) encryption is complete otherwise it deletes the last log file before doing recovery. Because the last log file also had the commit record , it is also gone; Now recovery sees log only encrypted with the old key and no end for re-encryption transaction, so it the (re) encryption work rolled back and database is brought to the state it was before (re) encryption started. If engine find a checkpoint in the last log file when databaseEncryptionStatus is IN-PROGRESS , the it is clear that checkpoint is encrypted with the new key; so it does any cleanup required. Added new test cases using debug flags to crash at critical point during (re) encryption and recovery of failed (re)encryption. svn stat : M java\engine\org\apache\derby\impl\store\raw\log\ReadOnly.java M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\RawStore.java M java\engine\org\apache\derby\impl\store\raw\data\EncryptData.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\iapi\services\io\FileUtil.java M java\engine\org\apache\derby\iapi\store\raw\log\LogFactory.java M java\engine\org\apache\derby\iapi\store\raw\data\DataFactory.java M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\ReEncryptCrashRecovery.java > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.1.2.3 >Reporter: Mike Matrigali > Assigned To: Suresh Thalamati >Priority: Minor > Fix For: 10.2.0.0 > > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff, reencrypt_5.diff, reencrypt_6.diff, > reencryptspec_1.html > > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There w
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencryptspec_1.html Minor updated to the spec. > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.1.2.3 >Reporter: Mike Matrigali > Assigned To: Suresh Thalamati >Priority: Minor > Fix For: 10.2.0.0 > > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff, reencrypt_5.diff, reencryptspec_1.html > > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Rick Hillegas updated DERBY-1156: - Urgency: Low > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.1.2.3 >Reporter: Mike Matrigali > Assigned To: Suresh Thalamati >Priority: Minor > Fix For: 10.2.0.0 > > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff, reencrypt_5.diff > > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_5.diff DERBY -1156 (partial) re-encryption of the database. This patch adds a new test that tests recovery of the database if the engine crashes just before committing the re-encryption with a new password phrase. Crash is simulated using the debug flags. svn stat: A java\testing\org\apache\derbyTesting\functionTests\tests\store\ReEncryptCrashRecovery.java A java\testing\org\apache\derbyTesting\functionTests\master\ReEncryptCrashRecovery.out M java\testing\org\apache\derbyTesting\functionTests\suites\encryptionAll.runall > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.1.2.3 >Reporter: Mike Matrigali > Assigned To: Suresh Thalamati >Priority: Minor > Fix For: 10.2.0.0 > > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff, reencrypt_5.diff > > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
Suresh Thalamati wrote: I think returning Ok (true) is not a right thing to do unless I really check the versions by reading the versions from the control files ..etc. Current usage of this function is to make sure database is at the right version before doing any writes that will break the soft-upgrade. If some one in the future implements a read-only feature that requires a version check , they can implement this method. Not my itch at the moment :-) That's fine just leave it, I just looked again and see that unimplemented is standard for the readonly implmentation. I was confused that you were counting on an exception, now I see it just never will be called for for readonly - only there to fulfill interface. I was thinking that Readonly extended the default impl, and didn't understand why it didn't just inherit impl. Before 10.2 goes out, would like to see test showing what user gets when they try to reencryt a read only db, but no need for this incremental commit.
Re: [jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
Thanks for taking time to review the patch Mike. My comments are in-line. Mike Matrigali (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Mike Matrigali updated DERBY-1156: -- Here are my comments on review of the reencrypt_4.diff, also I am running storeall but it has not finished yet.: minor typo's: XactFactory.java line 835 - trasaction --> transactions TransactionTable.java: add comment that hasPreparedXact(boolean recovered) is also MT unsafe. I will fix the comments. Is there a test for readonly db? There is a store/TurnsReadOnly.java test , but that is not included in the regresssion test suites, because cleanup will fail. May be there are some other tests or may be not :-) It is a little wierd to throw an exception from the ReadOnly impl of checkVersion(), from the name of the routine. I understand what the comment is saying. It seems unexpected for this routine to not return ok for a "current" db. I think returning Ok (true) is not a right thing to do unless I really check the versions by reading the versions from the control files ..etc. Current usage of this function is to make sure database is at the right version before doing any writes that will break the soft-upgrade. If some one in the future implements a read-only feature that requires a version check , they can implement this method. Not my itch at the moment :-) Other choices are, throw the error : 1) StandardException.newException(SQLState.DATABASE_READ_ONLY); instead of unimplmented. 2) Make sure checkVersion(...) is not called by doing the readOnly db check in calling method. I was planning to do the option 2 later. If you think option 1 is better I am ok with it too. stuff for now or later: o may be interesting to add following to your tests on XA o have a global xact in the log that is aborted during recovery (ie. not yet prepared). o have a global xact in the log that is prepared and committed. I don't think there are code issues with these paths, mostly just easy cases to add and will verify future code doesn't break it. o add test for readonly db and reencrypt. o add test for upgrade fail. Is there an existing framework for soft upgrade testing in 10.2? Thanks, those are good tests cases , I will work on them as a different patch. -suresh
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Mike Matrigali updated DERBY-1156: -- storeall ran fine with sun jdk1.4.2 on xp - with patch reencrypt_4 > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Mike Matrigali updated DERBY-1156: -- Here are my comments on review of the reencrypt_4.diff, also I am running storeall but it has not finished yet.: minor typo's: XactFactory.java line 835 - trasaction --> transactions TransactionTable.java: add comment that hasPreparedXact(boolean recovered) is also MT unsafe. Is there a test for readonly db? It is a little wierd to throw an exception from the ReadOnly impl of checkVersion(), from the name of the routine. I understand what the comment is saying. It seems unexpected for this routine to not return ok for a "current" db. stuff for now or later: o may be interesting to add following to your tests on XA o have a global xact in the log that is aborted during recovery (ie. not yet prepared). o have a global xact in the log that is prepared and committed. I don't think there are code issues with these paths, mostly just easy cases to add and will verify future code doesn't break it. o add test for readonly db and reencrypt. o add test for upgrade fail. Is there an existing framework for soft upgrade testing in 10.2? > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_4.diff DERBY -1156 (partial) This patch adds some code required to support reconfigure(rencryption) of an already existing encrypted database with a new password(secret key) or an external user specified encryption key. -- disables encryption/re-encryption of an existing database if there are any global transaction in the prepared state after recovery. -- disables encryption/re-encryption of an existing database if database is soft-upgraded to 10.2. -- Added test that tests re-encryption of an encrypted database when global transaction are in the prepared state after recovery. Tested the upgrade manually, will add the test case later. TESTS : derbyall test suite passed on Windows XP/JDK142 It would be great if some can review this patch. svn stat: M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java M java\engine\org\apache\derby\impl\store\raw\xact\TransactionTable.java M java\engine\org\apache\derby\impl\store\raw\log\ReadOnly.java M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\RawStore.java M java\engine\org\apache\derby\iapi\store\raw\log\LogFactory.java M java\engine\org\apache\derby\iapi\store\raw\xact\TransactionFactory.java M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java M java\engine\org\apache\derby\loc\messages_en.properties M java\shared\org\apache\derby\shared\common\reference\SQLState.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles. ant A java\testing\org\apache\derbyTesting\functionTests\tests\store\encryptDatabaseTest2_app.properties A java\testing\org\apache\derbyTesting\functionTests\tests\store\encryptDatabaseTest2.sql A java\testing\org\apache\derbyTesting\functionTests\master\encryptDatabaseTest2.out M java\testing\org\apache\derbyTesting\functionTests\suites\encryptionAll.runall > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff, reencrypt_4.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this
Re: [jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
Mike Matrigali (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Mike Matrigali updated DERBY-1156: -- i reviewed and tested reencrypt_3.diff patch. it looks fine, i will let you commit. Still would like to see more testing, especially exercising the abort paths when failure happens while reencrypting. Thanks Mike. I will add the tests for abort paths. -suresh
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Mike Matrigali updated DERBY-1156: -- i reviewed and tested reencrypt_3.diff patch. it looks fine, i will let you commit. Still would like to see more testing, especially exercising the abort paths when failure happens while reencrypting. > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_3.diff DERBY -1156 (partial) This patch adds some code required to support reconfigure(rencryption) of an already existing encrypted database with a new password(secret key) or an external user specified encryption key. Two new attributes "newBootPassword" and "newEncryptionkey" are introduced to support this functionality. -- modified the code to support two have instance of two cipher factories to exist. So that the existing data can decrypted with the old encryption key using one cipher factory and rewrite the data with new encryption keys using another cipher factory. -- reenryption of the database with new keys is similar to encrypting an already existing database. All the container data is read through the page cache and rewritten using the new encryption keys. -- Added test case to test the re-encryption of an encrypted database. TESTS : derbyall test suite passed on Windows XP/JDK142 It would be great if some can review this patch. svn stat: M java\engine\org\apache\derby\impl\services\jce\JCECipherFactory.java A java\engine\org\apache\derby\impl\services\jce\JCECipherFactoryBuilder.java M java\engine\org\apache\derby\impl\store\raw\RawStore.java A java\engine\org\apache\derby\iapi\services\crypto\CipherFactoryBuilder.java M java\engine\org\apache\derby\iapi\reference\Attribute.java M java\engine\org\apache\derby\iapi\reference\Module.java M java\engine\org\apache\derby\modules.properties M java\testing\org\apache\derbyTesting\unitTests\crypto\T_Cipher.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\encryptDatabaseTest1.sql M java\testing\org\apache\derbyTesting\functionTests\master\encryptDatabaseTest1.out > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db w
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_2.diff Attached patch (reencrypt_2,.diff ) fixes the encryptDatabaseTest1.sql test failure on Solaris regression runs. This test was using 16 byte encyrption key , which causes the test to fail on Solaris10 (with security provider 'SunPCKS11-Solaris') . This patch modifies the test to use 8 byte key instead of 16 bytes encryption key. Please refer to DERBY-788 and DERBY-1088 for more details on this topic. Test passed on jdk142 and jdk15 on Windows. It would be great if some one can commit this patch. Thanks -suresh > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Mike Matrigali updated DERBY-1156: -- I have reviewed this patch and run a full set of tests on windows XP / sun jdk1.4.2 which passed. I think this patch should be committed as is. It is a good incremental checkin for this feature. It provides a set of tools for the rest of the project, and gets the code to the point that it can encrypt an existing unencrypted db. Future checkins should address a couple of things: 1) obviously more testing. Some quick notes, the current tests may want to show that accessing the newly encrypted db with a bad password does not work - just to insure all the url;s are not just being totally ignored. Also want to test abort - probably need a testing codepoint to cause the failure at the right time. 2) fix up the comments, there are some typo's . 3) usual nit - some greater than 80 char lines. I would have committed this change now, but svn still is not back. > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_1.diff DERBY -1156 ( first patch) This patch adds some code required to support to configure an existing database for encryption of an un-encrypted database. Users can now specify the dataEncryption=true attribute along with other encryption attributed to encrypt an existing un-encrypted database at boot time. This patch implements encryption of an existing data in the database with new encryption key and also added new log operation requried to restore the database to its original state incase of crash/error. This patch also adds a test to test encryption of an un-encrypted database. TESTS : derbyall test suite passed on Windows XP/JDK142 It would be great if some can review and commit this patch. svn stat: M java\engine\org\apache\derby\impl\services\jce\JCECipherFactory.java M java\engine\org\apache\derby\impl\store\raw\log\ReadOnly.java M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java M java\engine\org\apache\derby\impl\store\raw\RawStore.java A java\engine\org\apache\derby\impl\store\raw\data\EncryptData.java M java\engine\org\apache\derby\impl\store\raw\data\InputStreamContainer.java M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java M java\engine\org\apache\derby\impl\store\raw\data\StreamFileContainer.java M java\engine\org\apache\derby\impl\store\raw\data\FileContainer.java A java\engine\org\apache\derby\impl\store\raw\data\EncryptContainerUndoOperation.java A java\engine\org\apache\derby\impl\store\raw\data\EncryptContainerOperation.java M java\engine\org\apache\derby\impl\store\raw\data\BaseContainer.java M java\engine\org\apache\derby\impl\store\raw\data\BaseContainerHandle.java M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java M java\engine\org\apache\derby\iapi\services\crypto\CipherFactory.java M java\engine\org\apache\derby\iapi\services\io\RegisteredFormatIds.java M java\engine\org\apache\derby\iapi\services\io\StoredFormatIds.java M java\engine\org\apache\derby\iapi\store\raw\log\LogFactory.java M java\engine\org\apache\derby\iapi\store\raw\data\DataFactory.java M java\engine\org\apache\derby\iapi\store\raw\data\RawContainerHandle.java M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java A java\testing\org\apache\derbyTesting\functionTests\tests\store\encryptDatabaseTest1.sql M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant A java\testing\org\apache\derbyTesting\functionTests\tests\store\encryptDatabaseTest1_app.properties A java\testing\org\apache\derbyTesting\functionTests\master\encryptDatabaseTest1.out M java\testing\org\apache\derbyTesting\functionTests\suites\encryptionAll.runall > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk befor
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: encryptspec.html Attached is a functional/design spec that describes database encryption enhancements, I plan to implement for 10.2 release. Any comments/suggestions ? Thanks -suresht > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db was already encrypted with. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
