[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239752#comment-17239752 ] Hudson commented on HBASE-25263: Results for branch branch-2 [build #113 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/]: (x) *{color:red}-1 overall{color}* details (if available): (x) {color:red}-1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Change encryption key generation algorithm used in the HBase shell > -- > > Key: HBASE-25263 > URL: https://issues.apache.org/jira/browse/HBASE-25263 > Project: HBase > Issue Type: Improvement > Components: encryption, shell >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > This PR is a follow-up of HBASE-25181 (#2539), where several issues were > discussed on the PR: > 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to > generate a secret key for HFile / WalFile encryption, when the user is > defining a string encryption key in the hbase shell. This algorithm is not > secure enough and not allowed in certain environments (e.g. on FIPS compliant > clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break > backward-compatibility, as even the tables created by the shell using the new > algorithm will be able to load (e.g. during bulkload / replication) the > HFiles serialized with the key generated by an old algorithm, as the HFiles > themselves already contain the key necessary for their decryption. > Smaller issues to be fixed: > 2. Improve the documentation e.g. with the changes introduced by HBASE-25181 > and also by some points discussed on the Jira ticket of HBASE-25263. > 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption > config checks should throw {{IllegalStateExceptions}} instead of > {{RuntimeExceptions}}. > 4. Test cases in {{TestEncryptionTest.java}} should be broken down into > smaller tests. > 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit > rule to validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239642#comment-17239642 ] Hudson commented on HBASE-25263: Results for branch master [build #140 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/]: (x) *{color:red}-1 overall{color}* details (if available): (x) {color:red}-1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Change encryption key generation algorithm used in the HBase shell > -- > > Key: HBASE-25263 > URL: https://issues.apache.org/jira/browse/HBASE-25263 > Project: HBase > Issue Type: Improvement > Components: encryption, shell >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > This PR is a follow-up of HBASE-25181 (#2539), where several issues were > discussed on the PR: > 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to > generate a secret key for HFile / WalFile encryption, when the user is > defining a string encryption key in the hbase shell. This algorithm is not > secure enough and not allowed in certain environments (e.g. on FIPS compliant > clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break > backward-compatibility, as even the tables created by the shell using the new > algorithm will be able to load (e.g. during bulkload / replication) the > HFiles serialized with the key generated by an old algorithm, as the HFiles > themselves already contain the key necessary for their decryption. > Smaller issues to be fixed: > 2. Improve the documentation e.g. with the changes introduced by HBASE-25181 > and also by some points discussed on the Jira ticket of HBASE-25263. > 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption > config checks should throw {{IllegalStateExceptions}} instead of > {{RuntimeExceptions}}. > 4. Test cases in {{TestEncryptionTest.java}} should be broken down into > smaller tests. > 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit > rule to validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230669#comment-17230669 ] Mate Szalay-Beko commented on HBASE-25263: -- These are very good points, thanks! {quote}... each HFile has all of the necessary metadata for supporting its decryption stored within it ... {quote} Yes, I didn't think it through deep enough before asking the question. You are right and this also means that we can change the derivation function without causing problems with backward compatibility. We can even bulkload HFiles / move snapshots to other cluster and all the "old" hfiles will be readable by the "new" cluster (assuming the master encryption key is the same / still known). {quote}If our older key derivation function has been deemed insufficient then all of the derived DEKs should probably also be treated as insufficient. The guidance we give there is compaction will rewrite the files with newly derived DEKs. Major compaction will rewrite most files, but not all, so there is scope for further improvement here... We may want an unconditional major compaction to ensure all DEKs are rotated. {quote} This is a valid point. Also an unconditional major compaction could be useful for other things too (like changing HFile compression maybe?). I can check the code and create a follow-up ticket later. BTW in the documentation ([http://hbase.apache.org/book.html#hbase.encryption.server)] we only recommend major compaction, we don't mention that this won't necessary rewrite all the HFiles (I might need to update this): {quote}Rotate the Data Key To rotate the data key, first change the ColumnFamily key in the column descriptor, then trigger a major compaction. When compaction is complete, all HFiles will be re-encrypted using the new data key. Until the compaction completes, the old HFiles will still be readable using the old key. {quote} Also a note for me: I should also update the documentation mentioning the key derivation function we use in the HBase shell (and also mention that the users will have a better control over the encryption key if they provide it from Java code and not from the shell) > Change encryption key generation algorithm used in the HBase shell > -- > > Key: HBASE-25263 > URL: https://issues.apache.org/jira/browse/HBASE-25263 > Project: HBase > Issue Type: Improvement > Components: encryption, shell >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > This ticket is a follow-up of HBASE-25181, where several issues were > discussed on the PR: > 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to > generate a secret key for HFile / WalFile encryption, when the user is > defining a string encryption key in the hbase shell. This algorithm is not > secure enough and not allowed in certain environments (e.g. on FIPS compliant > clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If > this would break backward compatibility, then we should make this algorithm > configurable. > Smaller issues: > 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption > config checks should throw IllegalStateExceptions instead of > RuntimeExceptions. > 3. Test cases in {{TestEncryptionTest.java}} should be broken down into > smaller tests. > 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit > rule to validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229408#comment-17229408 ] Andrew Kyle Purtell commented on HBASE-25263: - I can tell you the intent of the shell facility but not what people might actually be doing with it, my crystal ball is in the shop. :) So I suppose with the advantage of more birthdays and maturity, I have to say I cannot say. Worth noting that changing the algorithm does not affect existing files, though, because each HFile has all of the necessary metadata for supporting its decryption stored within it. As you know we don't re-transform the shell password to read, we only use the shell password to derive a data encryption key (DEK), which is then encrypted with the master encryption key and stored into an HFile meta block. Whenever we need to read the HFile, the DEK is extracted from the meta block, decrypted with the master key, and the decrypted DEK is used to decrypt block data. Whatever the shell password is, or whatever algorithm is used to derive a key from it, is immaterial to the decryption process. I don't think backwards compatibility as proposed would be meaningful to users, because users of this feature care that their data is encrypted but are very very unlikely to care about precisely how the DEK is derived except that the choice of derivation function meets best practice and is sufficiently strong. Time has moved on since the original patch landed and the derivation function needs updating. Hence this proposal. +1 to updating the derivation function. If our older key derivation function has been deemed insufficient then all of the derived DEKs should probably also be treated as insufficient. The guidance we give there is compaction will rewrite the files with newly derived DEKs. Major compaction will rewrite most files, but not all, so there is scope for further improvement here... We may want an unconditional major compaction to ensure all DEKs are rotated. > Change encryption key generation algorithm used in the HBase shell > -- > > Key: HBASE-25263 > URL: https://issues.apache.org/jira/browse/HBASE-25263 > Project: HBase > Issue Type: Improvement > Components: encryption, shell >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > This ticket is a follow-up of HBASE-25181, where several issues were > discussed on the PR: > 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to > generate a secret key for HFile / WalFile encryption, when the user is > defining a string encryption key in the hbase shell. This algorithm is not > secure enough and not allowed in certain environments (e.g. on FIPS compliant > clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If > this would break backward compatibility, then we should make this algorithm > configurable. > Smaller issues: > 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption > config checks should throw IllegalStateExceptions instead of > RuntimeExceptions. > 3. Test cases in {{TestEncryptionTest.java}} should be broken down into > smaller tests. > 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit > rule to validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229080#comment-17229080 ] Mate Szalay-Beko commented on HBASE-25263: -- regarding changing {{PBKDF2WithHmacSHA1}} to {{PBKDF2WithHmacSHA384}}: [~apurtell] I saw you did something similar in HBASE-10951. On that ticket the assumption was that there is no compatibility issue with simply changing this algorithm: {quote}No compatability issues that I can see given this isn't the way to generate data keys for production. This is so one can use the shell to create a schema with all encryption related attributes set up properly for basic functional testing or integration tests. {quote} Do you think this is still true? Isn't is possible that someone is relying in production on the schema defined with the HBase shell? Alternatively (instead of simply changing this argument) I can introduce a configuration variable, defaulting to the old algorithm. What do you think? > Change encryption key generation algorithm used in the HBase shell > -- > > Key: HBASE-25263 > URL: https://issues.apache.org/jira/browse/HBASE-25263 > Project: HBase > Issue Type: Improvement > Components: encryption, shell >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > This ticket is a follow-up of HBASE-25181, where several issues were > discussed on the PR: > 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to > generate a secret key for HFile / WalFile encryption, when the user is > defining a string encryption key in the hbase shell. This algorithm is not > secure enough and not allowed in certain environments (e.g. on FIPS compliant > clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If > this would break backward compatibility, then we should make this algorithm > configurable. > Smaller issues: > 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption > config checks should throw IllegalStateExceptions instead of > RuntimeExceptions. > 3. Test cases in {{TestEncryptionTest.java}} should be broken down into > smaller tests. > 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit > rule to validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)