This is an automated email from the ASF dual-hosted git repository.
zaynt4606 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 0bf4b44314 [CELEBORN-2388] Decouple inter-master Ratis TLS into its
own SSL module
0bf4b44314 is described below
commit 0bf4b44314b8f5b89367e9051a572109b3f1486a
Author: hzheng182 <[email protected]>
AuthorDate: Sun Nov 1 16:04:55 2026 +0800
[CELEBORN-2388] Decouple inter-master Ratis TLS into its own SSL module
### What changes were proposed in this pull request?
Adds a dedicated `ratis` SSL module so inter-master Ratis (Raft consensus)
gRPC TLS can use its own certificate/keystore, independent of the client-facing
`rpc_service` module — with a fully backward-compatible fallback.
- `TransportModuleConstants`: add `RATIS_MODULE = "ratis"`.
- `CelebornConf`: add `sslEnabledExplicitly(module)`, true only when
`celeborn.ssl.<module>.enabled` itself is set to `true`, ignoring the global
`celeborn.ssl.enabled` default and parent module fallbacks.
- `MasterClusterInfo.scala`: select the Ratis SSL module in one place:
- `ratisSslModule = ratis` only if `sslEnabledExplicitly(ratis)`, i.e.
`celeborn.ssl.ratis.enabled` is explicitly `true`, else `rpc_service`. The
global `celeborn.ssl.enabled` default does not select `ratis`, because the
`ratis` module does not inherit the `rpc_service` keystore.
- Ratis TLS is enabled when `sslEnabled(ratisSslModule)` is true.
- `HARaftServer.configureSsl`: build the `SSLFactory` from
`MasterClusterInfo.ratisSslModule`.
- Docs: add the `ratis` module to `docs/configuration/network-module.md`
and describe opting in under `docs/security.md`.
Config keys (`celeborn.ssl.ratis.enabled/.keyStore/.trustStore/...`) come
from the existing `celeborn.ssl.<module>.*` machinery — no enumerated module
list needed.
### Why are the changes needed?
Master↔master Ratis TLS is currently derived from the `rpc_service` module
(`MasterClusterInfo.scala` → `HARaftServer.configureSsl`). But `rpc_service` is
also the client-facing control cert, so Ratis and external clients are forced
to share **one** presented cert. That cert must simultaneously satisfy:
- **clients**: signed by a CA they trust (e.g. a corporate CA) + the
gateway/SNI hostname, and
- **Ratis peers**: gRPC hostname verification against the internal master
pod FQDNs (`*.<svc>-master-svc.<ns>.svc.cluster.local`).
In a cross-cluster / TLS-passthrough-gateway deployment these requirements
conflict (the client cert has no internal SANs), so HA masters can't complete
the Ratis TLS handshake ("UNAVAILABLE: io exception" during leader election).
Decoupling lets operators give Ratis its own cert (internal SANs) while leaving
the client-facing cert untouched.
Note: CELEBORN-1356 already split the unified `rpc` module into
`rpc_app`/`rpc_service` for client-vs-server separation, but never separated
Ratis inter-master traffic from `rpc_service` specifically — this PR closes
that remaining gap.
### Does this PR introduce _any_ user-facing change?
Default behavior is unchanged: unless `celeborn.ssl.ratis.enabled=true` is
explicitly set, Ratis TLS enablement and certificates come from `rpc_service`
exactly as today, including when SSL is enabled through the global
`celeborn.ssl.enabled` default. The new `ratis` module only takes effect when
`celeborn.ssl.ratis.enabled=true` is explicitly set.
### How was this patch tested?
`CelebornConfSuite` covers `sslEnabledExplicitly` against the global and
parent module defaults. `MasterClusterInfoSuite` covers module selection and
enablement, including:
- global `celeborn.ssl.enabled=true` with the keystore only under
`rpc_service` selects `rpc_service` and its certificate;
- global `celeborn.ssl.enabled=true` with `rpc_service.enabled=false`
leaves Ratis TLS disabled;
- an explicitly disabled `ratis` module falls back to `rpc_service`;
- an explicitly enabled `ratis` module uses its own certificate rather than
the `rpc_service` one.
The certificate cases assert the certificate presented by the selected key
manager. Existing `SSLRatisMasterStatusSystemSuiteJ` and
`MasterRatisServerSuiteJ` pass.
Closes #3767 from shellfish007/upstream-pr/ratis-tls-module.
Authored-by: hzheng182 <[email protected]>
Signed-off-by: zhengtao <[email protected]>
AI-Contributed/Feature: 0/36
AI-Contributed/UT: 0/155
---
.../common/protocol/TransportModuleConstants.java | 5 +
.../org/apache/celeborn/common/CelebornConf.scala | 8 ++
.../apache/celeborn/common/CelebornConfSuite.scala | 15 +++
docs/configuration/network-module.md | 1 +
docs/security.md | 2 +
.../deploy/master/clustermeta/ha/HARaftServer.java | 3 +-
.../master/clustermeta/ha/MasterClusterInfo.scala | 17 ++-
.../clustermeta/ha/MasterClusterInfoSuite.scala | 140 +++++++++++++++++++++
8 files changed, 187 insertions(+), 4 deletions(-)
diff --git
a/common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java
b/common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java
index ae4e77ed54..befcead4df 100644
---
a/common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java
+++
b/common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java
@@ -45,6 +45,11 @@ public class TransportModuleConstants {
public static final String DATA_MODULE = "data";
+ // SSL module for the inter-master Ratis channel, letting it use a
certificate separate from
+ // the client-facing RPC_SERVICE_MODULE. Used only when
celeborn.ssl.ratis.enabled is explicitly
+ // true, otherwise Ratis keeps using RPC_SERVICE_MODULE.
+ public static final String RATIS_MODULE = "ratis";
+
// When using null, InetSocketAddress will resolve to a wildcard address for
bind operations.
public static final String WILDCARD_BIND_ADDRESS = null;
}
diff --git
a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
index 23bb955c58..f66e4f251f 100644
--- a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
+++ b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
@@ -1594,6 +1594,14 @@ class CelebornConf(loadDefaults: Boolean) extends
Cloneable with Logging with Se
getSslConfig(SSL_ENABLED, module)
}
+ /**
+ * Whether SSL is explicitly enabled for the module itself, ignoring the
global
+ * `celeborn.ssl.enabled` default and parent module fallbacks.
+ */
+ def sslEnabledExplicitly(module: String): Boolean = {
+ getOption(SSL_ENABLED.key.replace("<module>",
module)).exists(SSL_ENABLED.valueConverter)
+ }
+
/**
* SSL protocol (remember that SSLv3 was compromised) supported by Java
*/
diff --git
a/common/src/test/scala/org/apache/celeborn/common/CelebornConfSuite.scala
b/common/src/test/scala/org/apache/celeborn/common/CelebornConfSuite.scala
index 392d5edcda..faa15ff26d 100644
--- a/common/src/test/scala/org/apache/celeborn/common/CelebornConfSuite.scala
+++ b/common/src/test/scala/org/apache/celeborn/common/CelebornConfSuite.scala
@@ -423,6 +423,21 @@ class CelebornConfSuite extends CelebornFunSuite {
validateDefaultTransportConfValue(conf, "rpc_app")
}
+ test("sslEnabledExplicitly ignores global and parent module defaults") {
+ val conf = new CelebornConf()
+ conf.set("celeborn.ssl.enabled", "true")
+ conf.set("celeborn.ssl.rpc.enabled", "true")
+ assert(conf.sslEnabled("rpc_service"))
+ assert(!conf.sslEnabledExplicitly("rpc_service"))
+ assert(!conf.sslEnabledExplicitly("ratis"))
+
+ conf.set("celeborn.ssl.ratis.enabled", "true")
+ assert(conf.sslEnabledExplicitly("ratis"))
+
+ conf.set("celeborn.ssl.ratis.enabled", "false")
+ assert(!conf.sslEnabledExplicitly("ratis"))
+ }
+
test("Test fallback config works even with parent") {
// Using NETWORK_IO_CONNECT_TIMEOUT since it has fallback to
NETWORK_CONNECT_TIMEOUT
diff --git a/docs/configuration/network-module.md
b/docs/configuration/network-module.md
index 18cee80982..8b66c7f3b8 100644
--- a/docs/configuration/network-module.md
+++ b/docs/configuration/network-module.md
@@ -29,6 +29,7 @@ The various transport modules which can be configured are:
| fetch | - | Configure transport module for handling data fetch at Celeborn
workers |
| data | - | Configure transport module for handling data push and fetch at
Celeborn apps |
| replicate | - | Configure transport module for handling data replication
between Celeborn workers |
+| ratis | - | Configure SSL for Raft communication between Celeborn masters.
Used only when `celeborn.ssl.ratis.enabled` is explicitly set to `true`,
otherwise Raft communication uses the `rpc_service` SSL configuration |
<!--end-include-->
diff --git a/docs/security.md b/docs/security.md
index 8fa7b62fd1..588ee985e3 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -36,6 +36,8 @@ end="<!--end-include-->"
When SSL is enabled for `rpc_service`, Raft communication between masters are
secured **only when** `celeborn.master.ha.ratis.raft.rpc.type` is set to `grpc`.
+By default, Raft communication uses the `rpc_service` SSL configuration. To
secure it with a separate certificate, set `celeborn.ssl.ratis.enabled` to
`true` explicitly and configure the `ratis` module. The global
`celeborn.ssl.enabled` default does not enable the `ratis` module. Setting
`celeborn.ssl.ratis.enabled` to `false` does not disable TLS for Raft
communication; it keeps using the `rpc_service` SSL configuration.
+
Note that `celeborn.ssl`, **without any module**, can be used to set SSL
default values which applies to all modules.
Also note that `data` module at application side, maps to `push` and `fetch`
at worker - hence, for SSL configuration, worker configuration for `push` and
`fetch` should be compatible with each other and with `data` at application
side.
diff --git
a/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java
b/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java
index c36aa321de..86f090c4dd 100644
---
a/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java
+++
b/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java
@@ -62,7 +62,6 @@ import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.client.MasterClient;
import org.apache.celeborn.common.exception.CelebornRuntimeException;
import org.apache.celeborn.common.network.ssl.SSLFactory;
-import org.apache.celeborn.common.protocol.TransportModuleConstants;
import org.apache.celeborn.common.util.ThreadUtils;
import org.apache.celeborn.common.util.Utils;
import org.apache.celeborn.service.deploy.master.clustermeta.ResourceProtos;
@@ -515,7 +514,7 @@ public class HARaftServer {
// also why nThreads does not matter
SSLFactory factory =
SSLFactory.createSslFactory(
- Utils.fromCelebornConf(conf,
TransportModuleConstants.RPC_SERVICE_MODULE, 1));
+ Utils.fromCelebornConf(conf,
MasterClusterInfo.ratisSslModule(conf), 1));
assert (null != factory);
assert (factory.hasKeyManagers());
diff --git
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfo.scala
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfo.scala
index 83b6f92830..18ee5043c2 100644
---
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfo.scala
+++
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfo.scala
@@ -38,8 +38,7 @@ object MasterClusterInfo extends Logging {
def loadHAConfig(conf: CelebornConf): MasterClusterInfo = {
val localNodeIdOpt = conf.haMasterNodeId
val clusterNodeIds = conf.haMasterNodeIds
- // If ssl is enabled, we enable it for ratis as well
- val sslEnabled =
conf.sslEnabled(TransportModuleConstants.RPC_SERVICE_MODULE)
+ val sslEnabled = conf.sslEnabled(ratisSslModule(conf))
val masterNodes = clusterNodeIds.map { nodeId =>
val ratisHost = conf.haMasterRatisHost(nodeId)
@@ -72,6 +71,20 @@ object MasterClusterInfo extends Logging {
MasterClusterInfo(localNodes.head, peerNodes.toList.asJava)
}
+ /**
+ * SSL module for the inter-master Ratis channel. The dedicated `ratis`
module is used only when
+ * `celeborn.ssl.ratis.enabled` is explicitly true; the global
`celeborn.ssl.enabled` default
+ * does not select it, so deployments configuring only `rpc_service` keep
its enablement and
+ * certificates.
+ */
+ def ratisSslModule(conf: CelebornConf): String = {
+ if (conf.sslEnabledExplicitly(TransportModuleConstants.RATIS_MODULE)) {
+ TransportModuleConstants.RATIS_MODULE
+ } else {
+ TransportModuleConstants.RPC_SERVICE_MODULE
+ }
+ }
+
private def isLocalAddress(addr: InetAddress): Boolean = {
if (addr.isAnyLocalAddress || addr.isLoopbackAddress) {
return true
diff --git
a/master/src/test/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfoSuite.scala
b/master/src/test/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfoSuite.scala
new file mode 100644
index 0000000000..6dfe9b82f1
--- /dev/null
+++
b/master/src/test/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfoSuite.scala
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.service.deploy.master.clustermeta.ha
+
+import java.io.FileInputStream
+import java.security.KeyStore
+import java.security.cert.Certificate
+import javax.net.ssl.X509KeyManager
+
+import scala.collection.JavaConverters._
+
+import org.apache.celeborn.CelebornFunSuite
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.network.ssl.{SSLFactory, SslSampleConfigs}
+import org.apache.celeborn.common.protocol.TransportModuleConstants
+import org.apache.celeborn.common.util.Utils
+
+class MasterClusterInfoSuite extends CelebornFunSuite {
+
+ private def sslEnabledKey(module: String): String =
s"celeborn.ssl.$module.enabled"
+
+ private val globalSslKey = "celeborn.ssl.enabled"
+ private val ratisKey = sslEnabledKey(TransportModuleConstants.RATIS_MODULE)
+ private val rpcServiceKey =
sslEnabledKey(TransportModuleConstants.RPC_SERVICE_MODULE)
+
+ private def setAll(conf: CelebornConf, entries: java.util.Map[String,
String]): Unit = {
+ entries.asScala.foreach { case (k, v) => conf.set(k, v) }
+ }
+
+ private def ratisSslEnabled(conf: CelebornConf): Boolean = {
+ conf.sslEnabled(MasterClusterInfo.ratisSslModule(conf))
+ }
+
+ // Certificate presented by the key manager that Ratis TLS would be
configured with.
+ private def selectedCertificate(conf: CelebornConf): Certificate = {
+ val module = MasterClusterInfo.ratisSslModule(conf)
+ val factory = SSLFactory.createSslFactory(Utils.fromCelebornConf(conf,
module, 1))
+ try {
+ assert(factory.hasKeyManagers)
+ val keyManager =
factory.getKeyManagers.get(0).asInstanceOf[X509KeyManager]
+ val alias = keyManager.getServerAliases("RSA", null).head
+ keyManager.getCertificateChain(alias).head
+ } finally {
+ factory.destroy()
+ }
+ }
+
+ private def keyStoreCertificate(path: String): Certificate = {
+ val keyStore = KeyStore.getInstance(KeyStore.getDefaultType)
+ val in = new FileInputStream(path)
+ try {
+ keyStore.load(in, "password".toCharArray)
+ } finally {
+ in.close()
+ }
+ keyStore.getCertificate(keyStore.aliases().nextElement())
+ }
+
+ test("ratis ssl is disabled when neither ratis nor rpc_service module is
enabled") {
+ assert(!ratisSslEnabled(new CelebornConf()))
+ }
+
+ test("ratis ssl stays enabled when only rpc_service is enabled") {
+ // Legacy deployments configure only rpc_service; they must keep securing
Ratis as before.
+ val conf = new CelebornConf()
+ conf.set(rpcServiceKey, "true")
+ assert(ratisSslEnabled(conf))
+ assert(MasterClusterInfo.ratisSslModule(conf) ===
TransportModuleConstants.RPC_SERVICE_MODULE)
+ }
+
+ test("ratis ssl is enabled by the dedicated ratis module alone") {
+ val conf = new CelebornConf()
+ conf.set(ratisKey, "true")
+ assert(ratisSslEnabled(conf))
+ assert(MasterClusterInfo.ratisSslModule(conf) ===
TransportModuleConstants.RATIS_MODULE)
+ }
+
+ test("ratis ssl is enabled by the ratis module even when rpc_service is
explicitly disabled") {
+ val conf = new CelebornConf()
+ conf.set(ratisKey, "true")
+ conf.set(rpcServiceKey, "false")
+ assert(ratisSslEnabled(conf))
+ }
+
+ test("explicitly disabled ratis module falls back to rpc_service") {
+ val conf = new CelebornConf()
+ conf.set(ratisKey, "false")
+ conf.set(rpcServiceKey, "true")
+ assert(ratisSslEnabled(conf))
+ assert(MasterClusterInfo.ratisSslModule(conf) ===
TransportModuleConstants.RPC_SERVICE_MODULE)
+ }
+
+ test("global ssl default does not select the ratis module") {
+ val conf = new CelebornConf()
+ conf.set(globalSslKey, "true")
+ setAll(
+ conf,
+
SslSampleConfigs.createDefaultConfigMapForModule(TransportModuleConstants.RPC_SERVICE_MODULE))
+ assert(MasterClusterInfo.ratisSslModule(conf) ===
TransportModuleConstants.RPC_SERVICE_MODULE)
+ assert(ratisSslEnabled(conf))
+ assert(
+ selectedCertificate(conf) ===
keyStoreCertificate(SslSampleConfigs.DEFAULT_KEY_STORE_PATH))
+ }
+
+ test("global ssl default does not enable ratis ssl when rpc_service is
disabled") {
+ val conf = new CelebornConf()
+ conf.set(globalSslKey, "true")
+ conf.set(rpcServiceKey, "false")
+ assert(MasterClusterInfo.ratisSslModule(conf) ===
TransportModuleConstants.RPC_SERVICE_MODULE)
+ assert(!ratisSslEnabled(conf))
+ }
+
+ test("explicitly enabled ratis module uses its own certificate") {
+ val conf = new CelebornConf()
+ setAll(
+ conf,
+
SslSampleConfigs.createDefaultConfigMapForModule(TransportModuleConstants.RPC_SERVICE_MODULE))
+ setAll(
+ conf,
+
SslSampleConfigs.createAnotherConfigMapForModule(TransportModuleConstants.RATIS_MODULE))
+ assert(MasterClusterInfo.ratisSslModule(conf) ===
TransportModuleConstants.RATIS_MODULE)
+ assert(
+ selectedCertificate(conf) ===
keyStoreCertificate(SslSampleConfigs.SECOND_KEY_STORE_PATH))
+ }
+}
