This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new b8e2498007a0 [SPARK-47318][CORE][3.5] Adds HKDF round to AuthEngine
key derivation to follow standard KEX practices
b8e2498007a0 is described below
commit b8e2498007a0b4e9a8790ce8a3511427093fa6a7
Author: Dongjoon Hyun
AuthorDate: Sat Apr 13 16:54:20 2024 -0700
[SPARK-47318][CORE][3.5] Adds HKDF round to AuthEngine key derivation to
follow standard KEX practices
### What changes were proposed in this pull request?
Backport of SPARK-47318 to v3.5.0
This change adds an additional pass through a key derivation function (KDF)
to the key exchange protocol in `AuthEngine`. Currently, it uses the shared
secret from a bespoke key negotiation protocol directly. This is an encoded X
coordinate on the X25519 curve. It is atypical and not recommended to use that
coordinate directly as a key, but rather to pass it to an KDF.
Note, Spark now supports TLS for RPC calls. It is preferable to use that
rather than the bespoke AES RPC encryption implemented by `AuthEngine` and
`TransportCipher`.
### Why are the changes needed?
This follows best practices of key negotiation protocols. The encoded X
coordinate is not guaranteed to be uniformly distributed over the 32-byte key
space. Rather, we pass it through a HKDF function to map it uniformly to a
16-byte key space.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Exiting tests under:
`build/sbt "network-common/test:testOnly"`
Specifically:
`build/sbt "network-common/test:testOnly
org.apache.spark.network.crypto.AuthEngineSuite"`
`build/sbt "network-common/test:testOnly
org.apache.spark.network.crypto.AuthIntegrationSuite"`
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46014 from sweisdb/SPARK-47318-v3.5.0.
Lead-authored-by: Dongjoon Hyun
Co-authored-by: Steve Weis
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/network/crypto/AuthEngine.java| 15 +-
.../java/org/apache/spark/network/crypto/README.md | 17 ++-
.../apache/spark/network/util/TransportConf.java | 9 ++
.../spark/network/crypto/AuthEngineSuite.java | 34 --
docs/security.md | 12
5 files changed, 82 insertions(+), 5 deletions(-)
diff --git
a/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java
b/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java
index 078d9ceb317b..14f0c23fd05f 100644
---
a/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java
+++
b/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java
@@ -41,16 +41,19 @@ import org.apache.spark.network.util.TransportConf;
* Exchange, using a pre-shared key to derive an AES-GCM key encrypting key.
*/
class AuthEngine implements Closeable {
+ public static final byte[] DERIVED_KEY_INFO = "derivedKey".getBytes(UTF_8);
public static final byte[] INPUT_IV_INFO = "inputIv".getBytes(UTF_8);
public static final byte[] OUTPUT_IV_INFO = "outputIv".getBytes(UTF_8);
private static final String MAC_ALGORITHM = "HMACSHA256";
private static final int AES_GCM_KEY_SIZE_BYTES = 16;
private static final byte[] EMPTY_TRANSCRIPT = new byte[0];
+ private static final int UNSAFE_SKIP_HKDF_VERSION = 1;
private final String appId;
private final byte[] preSharedSecret;
private final TransportConf conf;
private final Properties cryptoConf;
+ private final boolean unsafeSkipFinalHkdf;
private byte[] clientPrivateKey;
private TransportCipher sessionCipher;
@@ -62,6 +65,9 @@ class AuthEngine implements Closeable {
this.preSharedSecret = preSharedSecret.getBytes(UTF_8);
this.conf = conf;
this.cryptoConf = conf.cryptoConf();
+// This is for backward compatibility with version 1.0 of this protocol,
+// which did not perform a final HKDF round.
+this.unsafeSkipFinalHkdf = conf.authEngineVersion() ==
UNSAFE_SKIP_HKDF_VERSION;
}
@VisibleForTesting
@@ -201,6 +207,13 @@ class AuthEngine implements Closeable {
byte[] sharedSecret,
boolean isClient,
byte[] transcript) throws GeneralSecurityException {
+byte[] derivedKey = unsafeSkipFinalHkdf ? sharedSecret : // This is for
backwards compatibility
+ Hkdf.computeHkdf(
+MAC_ALGORITHM,
+sharedSecret,
+transcript,
+DERIVED_KEY_INFO,
+AES_GCM_KEY_SIZE_BYTES);
byte[] clientIv = Hkdf.computeHkdf(
MAC_ALGORITHM,
sharedSecret,
@@ -213,7 +226,7 @@ class
