[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239752#comment-17239752
]
Hudson commented on HBASE-25263:
Results for branch branch-2
[build #113 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/113/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> This PR is a follow-up of HBASE-25181 (#2539), where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break
> backward-compatibility, as even the tables created by the shell using the new
> algorithm will be able to load (e.g. during bulkload / replication) the
> HFiles serialized with the key generated by an old algorithm, as the HFiles
> themselves already contain the key necessary for their decryption.
> Smaller issues to be fixed:
> 2. Improve the documentation e.g. with the changes introduced by HBASE-25181
> and also by some points discussed on the Jira ticket of HBASE-25263.
> 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw {{IllegalStateExceptions}} instead of
> {{RuntimeExceptions}}.
> 4. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239642#comment-17239642
]
Hudson commented on HBASE-25263:
Results for branch master
[build #140 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/140/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> This PR is a follow-up of HBASE-25181 (#2539), where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break
> backward-compatibility, as even the tables created by the shell using the new
> algorithm will be able to load (e.g. during bulkload / replication) the
> HFiles serialized with the key generated by an old algorithm, as the HFiles
> themselves already contain the key necessary for their decryption.
> Smaller issues to be fixed:
> 2. Improve the documentation e.g. with the changes introduced by HBASE-25181
> and also by some points discussed on the Jira ticket of HBASE-25263.
> 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw {{IllegalStateExceptions}} instead of
> {{RuntimeExceptions}}.
> 4. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230669#comment-17230669
]
Mate Szalay-Beko commented on HBASE-25263:
--
These are very good points, thanks!
{quote}... each HFile has all of the necessary metadata for supporting its
decryption stored within it ...
{quote}
Yes, I didn't think it through deep enough before asking the question. You are
right and this also means that we can change the derivation function without
causing problems with backward compatibility. We can even bulkload HFiles /
move snapshots to other cluster and all the "old" hfiles will be readable by
the "new" cluster (assuming the master encryption key is the same / still
known).
{quote}If our older key derivation function has been deemed insufficient then
all of the derived DEKs should probably also be treated as insufficient. The
guidance we give there is compaction will rewrite the files with newly derived
DEKs. Major compaction will rewrite most files, but not all, so there is scope
for further improvement here... We may want an unconditional major compaction
to ensure all DEKs are rotated.
{quote}
This is a valid point. Also an unconditional major compaction could be useful
for other things too (like changing HFile compression maybe?). I can check the
code and create a follow-up ticket later.
BTW in the documentation
([http://hbase.apache.org/book.html#hbase.encryption.server)] we only recommend
major compaction, we don't mention that this won't necessary rewrite all the
HFiles (I might need to update this):
{quote}Rotate the Data Key
To rotate the data key, first change the ColumnFamily key in the column
descriptor, then trigger a major compaction. When compaction is complete, all
HFiles will be re-encrypted using the new data key. Until the compaction
completes, the old HFiles will still be readable using the old key.
{quote}
Also a note for me: I should also update the documentation mentioning the key
derivation function we use in the HBase shell (and also mention that the users
will have a better control over the encryption key if they provide it from Java
code and not from the shell)
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229408#comment-17229408
]
Andrew Kyle Purtell commented on HBASE-25263:
-
I can tell you the intent of the shell facility but not what people might
actually be doing with it, my crystal ball is in the shop. :) So I suppose with
the advantage of more birthdays and maturity, I have to say I cannot say.
Worth noting that changing the algorithm does not affect existing files,
though, because each HFile has all of the necessary metadata for supporting its
decryption stored within it. As you know we don't re-transform the shell
password to read, we only use the shell password to derive a data encryption
key (DEK), which is then encrypted with the master encryption key and stored
into an HFile meta block. Whenever we need to read the HFile, the DEK is
extracted from the meta block, decrypted with the master key, and the decrypted
DEK is used to decrypt block data. Whatever the shell password is, or whatever
algorithm is used to derive a key from it, is immaterial to the decryption
process.
I don't think backwards compatibility as proposed would be meaningful to users,
because users of this feature care that their data is encrypted but are very
very unlikely to care about precisely how the DEK is derived except that the
choice of derivation function meets best practice and is sufficiently strong.
Time has moved on since the original patch landed and the derivation function
needs updating. Hence this proposal. +1 to updating the derivation function.
If our older key derivation function has been deemed insufficient then all of
the derived DEKs should probably also be treated as insufficient. The guidance
we give there is compaction will rewrite the files with newly derived DEKs.
Major compaction will rewrite most files, but not all, so there is scope for
further improvement here... We may want an unconditional major compaction to
ensure all DEKs are rotated.
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229080#comment-17229080
]
Mate Szalay-Beko commented on HBASE-25263:
--
regarding changing {{PBKDF2WithHmacSHA1}} to {{PBKDF2WithHmacSHA384}}:
[~apurtell] I saw you did something similar in HBASE-10951. On that ticket the
assumption was that there is no compatibility issue with simply changing this
algorithm:
{quote}No compatability issues that I can see given this isn't the way to
generate data keys for production. This is so one can use the shell to create a
schema with all encryption related attributes set up properly for basic
functional testing or integration tests.
{quote}
Do you think this is still true? Isn't is possible that someone is relying in
production on the schema defined with the HBase shell?
Alternatively (instead of simply changing this argument) I can introduce a
configuration variable, defaulting to the old algorithm.
What do you think?
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
