This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 1127bf9 [SPARK-57543] Make ConfigOption dynamic override opt-in and
filter dynamic config refresh
1127bf9 is described below
commit 1127bf9fee08b55251401b86c847e6646b520c98
Author: Zhou JIANG <[email protected]>
AuthorDate: Tue Jul 7 17:03:44 2026 -0700
[SPARK-57543] Make ConfigOption dynamic override opt-in and filter dynamic
config refresh
### What changes were proposed in this pull request?
This PR makes runtime dynamic config override opt-in and enforces it
centrally when dynamic config is refreshed:
1. `ConfigOption`
- `enableDynamicOverride` now defaults to `false` — an option must
explicitly opt in via `.enableDynamicOverride(true)` to be overridable at
runtime.
- Each option registers itself (keyed by its config key) into a static
registry on construction, exposing `dynamicOverrideEnabledKeys()` as the
allow-list of overridable keys. `enableDynamicOverride` on each option remains
the single source of truth.
2. `SparkOperatorConfManager.refresh(Map)` now filters the incoming
ConfigMap data against `ConfigOption.dynamicOverrideEnabledKeys()`, applying
only overridable keys and logging any dropped (unknown or non-overridable) keys
at WARN. `getValue(String)` and `getAll()` therefore no longer surface override
values for keys that are never honored.
3. `SparkOperatorConf` options now set `enableDynamicOverride`explicitly;
this also fixes docs for `LOG_CONF` and `PERIODIC_GC_INTERVAL_SECONDS`, which
are only read at startup and are non-overridable.
The existing per-option check in `ConfigOption.resolveValue()` is kept as
defense-in-depth.
### Why are the changes needed?
Previously `refresh()` stored the entire ConfigMap into the override map,
and `enableDynamicOverride` defaulted to `true`. As a result, an override value
could be stored (and surfaced via `getValue`/`getAll`/the `LOG_CONF` dump) for
keys that are not meant to be hot-reloaded, and any new option was overridable
by default. Defaulting to opt-in and dropping non-allow-listed keys at refresh
time makes the override surface explicit, safer, and self-documenting, and
avoids silently ignored [...]
### Does this PR introduce _any_ user-facing change?
Yes. Dynamic config updates for keys that are not explicitly marked
`enableDynamicOverride(true)` are now ignored (and logged at WARN) instead of
being stored. In particular `spark.logConf` and
`spark.kubernetes.operator.periodicGC.intervalSeconds` are no longer
dynamically overridable (both are only read at startup). The generated config
documentation's "Allow Hot Reloading" column reflects each option's flag.
### How was this patch tested?
Updated `ConfigOptionTest` and `SparkOperatorConfManagerTest`, and added
`SparkOperatorConfManagerTest#testRefreshDropsUnknownAndNonOverridableKeys`
covering the allow-list filtering. Existing
`SparkOperatorConfigMapReconcilerTest` continues to exercise the dynamic-config
refresh path with an overridable key.
### Was this patch authored or co-authored using generative AI tooling?
Co-Authored-By: Claude Opus 4.8
Closes #713 from jiangzho/conf_wl.
Authored-by: Zhou JIANG <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/config_properties.md | 6 +-
.../spark/k8s/operator/config/ConfigOption.java | 70 ++++++++++++++++++----
.../k8s/operator/config/SparkOperatorConf.java | 20 +++++++
.../operator/config/SparkOperatorConfManager.java | 31 +++++++++-
.../spark/k8s/operator/SparkOperatorTest.java | 8 +++
.../k8s/operator/config/ConfigOptionTest.java | 12 +++-
.../config/SparkOperatorConfManagerTest.java | 39 +++++++++++-
.../apache/spark/k8s/operator/utils/UtilsTest.java | 8 ++-
8 files changed, 174 insertions(+), 20 deletions(-)
diff --git a/docs/config_properties.md b/docs/config_properties.md
index 9c14468..1a9ff64 100644
--- a/docs/config_properties.md
+++ b/docs/config_properties.md
@@ -35,7 +35,7 @@
| spark.kubernetes.operator.metrics.sanitizePrometheusMetricsNameEnabled |
Boolean | true | false | Whether or not to enable automatic name sanitizing for
all metrics based on best-practice guide from Prometheus
https://prometheus.io/docs/practices/naming/ |
| spark.kubernetes.operator.name | String | spark-kubernetes-operator | false
| Name of the operator. |
| spark.kubernetes.operator.namespace | String | default | false | Namespace
that operator is deployed within. |
- | spark.kubernetes.operator.periodicGC.intervalSeconds | Long | 1800 | true |
Interval (in seconds) between periodic System.gc() invocations. Set to 0 or a
negative value to disable. Note that System.gc() is a no-op if JVM is started
with -XX:+DisableExplicitGC. |
+ | spark.kubernetes.operator.periodicGC.intervalSeconds | Long | 1800 | false
| Interval (in seconds) between periodic System.gc() invocations. Set to 0 or a
negative value to disable. Note that System.gc() is a no-op if JVM is started
with -XX:+DisableExplicitGC. |
| spark.kubernetes.operator.reconciler.appStatusListenerClassNames | String |
| false | Comma-separated names of SparkAppStatusListener class
implementations |
| spark.kubernetes.operator.reconciler.clusterStatusListenerClassNames |
String | | false | Comma-separated names of SparkClusterStatusListener class
implementations |
| spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds | Long
| 60 | true | Timeout (in seconds) for requests made to API server. This
applies only to foreground requests. |
@@ -50,6 +50,6 @@
| spark.kubernetes.operator.reconciler.terminationTimeoutSeconds | Integer |
30 | false | Grace period for operator shutdown before reconciliation threads
are killed. |
| spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled |
Boolean | true | true | When enabled, operator would trim state transition
history when a new attempt starts, keeping previous attempt summary only. |
| spark.kubernetes.operator.watchedNamespaces | String | default | true |
Comma-separated list of namespaces that the operator would be watching for
Spark resources. If set to '*', operator would watch all namespaces. |
- | spark.logConf | Boolean | false | true | When enabled, operator will print
configurations |
- | spark.redaction.regex | String | (?i)secret\|password\|token\|access[.]?key
| true | Regex to decide which parts of configuration properties contain
sensitive information, whose values would be redacted in operator logs. This
reuses Spark's redaction configuration key and default pattern. |
+ | spark.logConf | Boolean | false | false | When enabled, operator will print
configurations |
+ | spark.redaction.regex | String | (?i)secret\|password\|token\|access[.]?key
| false | Regex to decide which parts of configuration properties contain
sensitive information, whose values would be redacted in operator logs. This
reuses Spark's redaction configuration key and default pattern. |
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
index 1130866..d9c4aae 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
@@ -19,12 +19,15 @@
package org.apache.spark.k8s.operator.config;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
import com.fasterxml.jackson.core.JsonProcessingException;
-import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
-import lombok.RequiredArgsConstructor;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
@@ -34,20 +37,67 @@ import org.apache.spark.k8s.operator.utils.StringUtils;
/**
* Config options for Spark Operator. Supports primitive and serialized JSON.
*
+ * <p>Every option registers itself, keyed by {@link #key}, into a static
registry on construction.
+ * {@link SparkOperatorConfManager#refresh(Map)} consults {@link
#dynamicOverrideEnabledKeys()} to
+ * drop any dynamic-config update that targets an unknown key or one whose
{@link
+ * #enableDynamicOverride} is {@code false}.
+ *
* @param <T> The type of the config option's value.
*/
-@RequiredArgsConstructor
-@AllArgsConstructor
@EqualsAndHashCode
@ToString
-@Builder
@Slf4j
public class ConfigOption<T> {
- @Getter @Builder.Default private final boolean enableDynamicOverride = true;
- @Getter private String key;
- @Getter private String description;
- @Getter private T defaultValue;
- @Getter private Class<T> typeParameterClass;
+ /** Indexes every declared option so dynamic-config refresh can check
overridability. */
+ private static final Map<String, ConfigOption<?>> REGISTRY = new
ConcurrentHashMap<>();
+
+ /**
+ * Whether this option may be overridden at runtime via dynamic config.
Defaults to {@code false}
+ * (opt-in): dynamic override must be explicitly enabled through the builder.
+ */
+ @Getter private final boolean enableDynamicOverride;
+ @Getter private final String key;
+ @Getter private final String description;
+ @Getter private final T defaultValue;
+ @Getter private final Class<T> typeParameterClass;
+
+ @Builder
+ ConfigOption(
+ boolean enableDynamicOverride,
+ String key,
+ String description,
+ T defaultValue,
+ Class<T> typeParameterClass) {
+ this.enableDynamicOverride = enableDynamicOverride;
+ this.key = key;
+ this.description = description;
+ this.defaultValue = defaultValue;
+ this.typeParameterClass = typeParameterClass;
+ if (StringUtils.isNotEmpty(key)) {
+ REGISTRY.put(key, this);
+ }
+ }
+
+ /**
+ * Returns the keys of all declared options that permit runtime dynamic
override. Used as the
+ * allow-list when refreshing dynamic config.
+ *
+ * <p><b>Note:</b> the returned set only includes options that have already
been constructed.
+ * Options are {@code static final} fields (e.g. on {@link
SparkOperatorConf}) that register
+ * themselves lazily on class initialization, so a caller that reads this
before those classes
+ * are initialized would see an empty or partial set. {@link
+ * SparkOperatorConfManager#refresh(Map)} forces initialization via {@link
+ * SparkOperatorConf#ensureOptionsRegistered()} first to avoid that; other
callers must ensure
+ * the relevant option classes are initialized.
+ *
+ * @return An immutable set of dynamically overridable config keys.
+ */
+ public static Set<String> dynamicOverrideEnabledKeys() {
+ return REGISTRY.values().stream()
+ .filter(ConfigOption::isEnableDynamicOverride)
+ .map(ConfigOption::getKey)
+ .collect(Collectors.toUnmodifiableSet());
+ }
/**
* Returns the resolved value of the config option.
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
index 03ab285..a1a4c10 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
@@ -35,6 +35,7 @@ public final class SparkOperatorConf {
public static final ConfigOption<Boolean> LOG_CONF =
ConfigOption.<Boolean>builder()
.key("spark.logConf")
+ .enableDynamicOverride(false)
.description("When enabled, operator will print configurations")
.typeParameterClass(Boolean.class)
.defaultValue(false)
@@ -48,6 +49,9 @@ public final class SparkOperatorConf {
public static final ConfigOption<String> REDACTION_REGEX =
ConfigOption.<String>builder()
.key("spark.redaction.regex")
+ // Intentionally not dynamically overridable: a runtime override of
the redaction
+ // pattern could weaken or disable log redaction, so this is only
read at startup.
+ .enableDynamicOverride(false)
.description(
"Regex to decide which parts of configuration properties contain
sensitive "
+ "information, whose values would be redacted in operator
logs. This reuses "
@@ -63,6 +67,7 @@ public final class SparkOperatorConf {
public static final ConfigOption<Long> PERIODIC_GC_INTERVAL_SECONDS =
ConfigOption.<Long>builder()
.key("spark.kubernetes.operator.periodicGC.intervalSeconds")
+ .enableDynamicOverride(false)
.description(
"Interval (in seconds) between periodic System.gc() invocations.
"
+ "Set to 0 or a negative value to disable. Note that
System.gc() is a no-op "
@@ -868,4 +873,19 @@ public final class SparkOperatorConf {
}
return value;
}
+
+ /**
+ * Forces static initialization of this class so that every {@link
ConfigOption} declared here
+ * has registered itself with {@link ConfigOption}'s registry.
+ *
+ * <p>{@link ConfigOption#dynamicOverrideEnabledKeys()} only reflects
options that have already
+ * been constructed, and the options are {@code static final} fields that
register lazily on
+ * class init. Invoking this method (an active use of the class) guarantees
the allow-list is
+ * complete before dynamic config is refreshed, independent of operator boot
ordering — see
+ * {@link SparkOperatorConfManager#refresh(java.util.Map)}. This method is
intentionally a no-op;
+ * the side effect is triggering class initialization.
+ */
+ public static void ensureOptionsRegistered() {
+ // No-op: the mere invocation triggers this class's static initializer.
+ }
}
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
index a0f4bfe..15f8118 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
@@ -23,8 +23,11 @@ import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
import lombok.extern.slf4j.Slf4j;
@@ -99,14 +102,36 @@ public class SparkOperatorConfManager {
}
/**
- * Refreshes the configuration overrides with new values from a map.
+ * Refreshes the configuration overrides with new values from a map. Only
keys that are declared
+ * as dynamically overridable options (see {@link
ConfigOption#dynamicOverrideEnabledKeys()}) are
+ * applied; unknown keys and keys whose option disables dynamic override are
dropped and logged so
+ * an ineffective update is not silently ignored.
*
* @param updatedConfig A map containing the updated configuration
properties.
*/
public void refresh(Map<String, String> updatedConfig) {
+ // Ensure every ConfigOption has registered before we read the allow-list.
The options are
+ // static final fields on SparkOperatorConf and register lazily on class
init, so without this
+ // the allow-list could be empty or partial if refresh() runs before
SparkOperatorConf is
+ // touched, silently dropping every override. Forcing init here makes
correctness independent
+ // of boot ordering rather than relying on an earlier incidental read of
SparkOperatorConf.
+ SparkOperatorConf.ensureOptionsRegistered();
+ Set<String> allowedKeys = ConfigOption.dynamicOverrideEnabledKeys();
+ Properties filtered = new Properties();
+ List<String> droppedKeys = new ArrayList<>();
+ for (Map.Entry<String, String> entry : updatedConfig.entrySet()) {
+ if (allowedKeys.contains(entry.getKey())) {
+ filtered.put(entry.getKey(), entry.getValue());
+ } else {
+ droppedKeys.add(entry.getKey());
+ }
+ }
+ if (!droppedKeys.isEmpty()) {
+ log.warn(
+ "Ignoring dynamic config override for unknown or non-overridable
keys: {}", droppedKeys);
+ }
synchronized (this) {
- this.configOverrides = new Properties();
- configOverrides.putAll(updatedConfig);
+ this.configOverrides = filtered;
}
}
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
index 7c55c33..7e97d92 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
@@ -53,6 +53,7 @@ import org.mockito.MockedConstruction;
import org.mockito.MockedStatic;
import org.apache.spark.k8s.operator.client.KubernetesClientFactory;
+import org.apache.spark.k8s.operator.config.ConfigOption;
import org.apache.spark.k8s.operator.config.DynamicConfigMonitor;
import org.apache.spark.k8s.operator.config.SparkOperatorConf;
import org.apache.spark.k8s.operator.config.SparkOperatorConfManager;
@@ -285,6 +286,13 @@ class SparkOperatorTest {
MockedConstruction<KubernetesMetricsInterceptor>
interceptorMockedConstruction =
mockConstruction(KubernetesMetricsInterceptor.class)) {
setConfigKey(SparkOperatorConf.LOG_CONF, true);
+ // Register the sensitive key as dynamically overridable so refresh()
keeps it; refresh now
+ // drops any key that is not in the dynamic-override allow-list.
+ ConfigOption.<String>builder()
+ .key("spark.dummy.db.password")
+ .enableDynamicOverride(true)
+ .typeParameterClass(String.class)
+ .build();
SparkOperatorConfManager.INSTANCE.refresh(
Map.of("spark.dummy.db.password", "super-sensitive-value"));
mockMetricsSystemFactory
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
index d1a9827..c380cc3 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
@@ -129,10 +129,10 @@ class ConfigOptionTest {
configOverride.put("foodouble", "10.0");
configOverride.put("foobool", "true");
configOverride.put("foo", "barbar");
- SparkOperatorConfManager.INSTANCE.refresh(configOverride);
ConfigOption<String> testStrConf =
ConfigOption.<String>builder()
.key("foo")
+ .enableDynamicOverride(true)
.typeParameterClass(String.class)
.description("foo foo.")
.defaultValue(defaultStringValue)
@@ -140,6 +140,7 @@ class ConfigOptionTest {
ConfigOption<Integer> testIntConf =
ConfigOption.<Integer>builder()
.key("fooint")
+ .enableDynamicOverride(true)
.typeParameterClass(Integer.class)
.description("foo foo.")
.defaultValue(defaultIntValue)
@@ -147,6 +148,7 @@ class ConfigOptionTest {
ConfigOption<Short> testShortConf =
ConfigOption.<Short>builder()
.key("fooshort")
+ .enableDynamicOverride(true)
.typeParameterClass(Short.class)
.description("foo foo.")
.defaultValue(defaultShortValue)
@@ -154,6 +156,7 @@ class ConfigOptionTest {
ConfigOption<Long> testLongConf =
ConfigOption.<Long>builder()
.key("foolong")
+ .enableDynamicOverride(true)
.typeParameterClass(Long.class)
.description("foo foo.")
.defaultValue(defaultLongValue)
@@ -161,6 +164,7 @@ class ConfigOptionTest {
ConfigOption<Boolean> testBooleanConf =
ConfigOption.<Boolean>builder()
.key("foobool")
+ .enableDynamicOverride(true)
.typeParameterClass(Boolean.class)
.description("foo foo.")
.defaultValue(defaultBooleanValue)
@@ -168,6 +172,7 @@ class ConfigOptionTest {
ConfigOption<Float> testFloatConf =
ConfigOption.<Float>builder()
.key("foofloat")
+ .enableDynamicOverride(true)
.typeParameterClass(Float.class)
.description("foo foo.")
.defaultValue(defaultFloatValue)
@@ -175,6 +180,7 @@ class ConfigOptionTest {
ConfigOption<Double> testDoubleConf =
ConfigOption.<Double>builder()
.key("foodouble")
+ .enableDynamicOverride(true)
.typeParameterClass(Double.class)
.description("foo foo.")
.defaultValue(defaultDoubleValue)
@@ -182,10 +188,14 @@ class ConfigOptionTest {
ConfigOption<Byte> testByteConf =
ConfigOption.<Byte>builder()
.key("foobyte")
+ .enableDynamicOverride(true)
.typeParameterClass(Byte.class)
.description("foo foo.")
.defaultValue(defaultByteValue)
.build();
+ // The options above opt in to dynamic override, so refreshing after they
are built lets the
+ // allow-list filter retain these keys.
+ SparkOperatorConfManager.INSTANCE.refresh(configOverride);
Assertions.assertEquals(overrideStringValue, testStrConf.getValue());
Assertions.assertEquals(overrideIntValue, testIntConf.getValue());
Assertions.assertEquals(overrideLongValue, testLongConf.getValue());
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
index 7d0538f..924b6e0 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
@@ -54,6 +54,7 @@ class SparkOperatorConfManagerTest {
String propBackUp = System.getProperty("spark.kubernetes.operator.foo");
System.setProperty("spark.kubernetes.operator.foo", "bar");
try {
+ registerDynamicOption("spark.kubernetes.operator.foo");
SparkOperatorConfManager confManager = new SparkOperatorConfManager();
Assertions.assertEquals("bar",
confManager.getInitialValue("spark.kubernetes.operator.foo"));
Assertions.assertEquals("bar",
confManager.getValue("spark.kubernetes.operator.foo"));
@@ -80,8 +81,10 @@ class SparkOperatorConfManagerTest {
String propBackUp = System.getProperty("spark.kubernetes.operator.foo");
System.setProperty("spark.kubernetes.operator.foo", "bar");
try {
+ registerDynamicOption("spark.kubernetes.operator.foo");
+ registerDynamicOption("k1");
+ registerDynamicOption("k2");
SparkOperatorConfManager confManager = new SparkOperatorConfManager();
-
// Check initial configurations.
int initialSize = confManager.getAll().size();
Assertions.assertEquals(initialSize, confManager.initialConfig.size());
@@ -111,4 +114,38 @@ class SparkOperatorConfManagerTest {
}
}
}
+
+ @Test
+ void testRefreshDropsUnknownAndNonOverridableKeys() {
+ String overridableKey = "spark.kubernetes.operator.test.overridable";
+ String nonOverridableKey = "spark.kubernetes.operator.test.nonOverridable";
+ String unknownKey = "spark.kubernetes.operator.test.unknown";
+ registerDynamicOption(overridableKey);
+ ConfigOption.<String>builder()
+ .key(nonOverridableKey)
+ .enableDynamicOverride(false)
+ .typeParameterClass(String.class)
+ .build();
+
+ SparkOperatorConfManager confManager = new SparkOperatorConfManager();
+ confManager.refresh(
+ Map.of(
+ overridableKey, "kept",
+ nonOverridableKey, "dropped",
+ unknownKey, "dropped"));
+
+ Assertions.assertEquals(1, confManager.configOverrides.size());
+ Assertions.assertEquals("kept", confManager.getValue(overridableKey));
+ Assertions.assertNull(confManager.getValue(nonOverridableKey));
+ Assertions.assertNull(confManager.getValue(unknownKey));
+ }
+
+ /** Builds a throwaway {@link ConfigOption} so {@code key} is registered as
overridable. */
+ private static void registerDynamicOption(String key) {
+ ConfigOption.<String>builder()
+ .key(key)
+ .enableDynamicOverride(true)
+ .typeParameterClass(String.class)
+ .build();
+ }
}
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
index 40e7e50..42c591d 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
@@ -94,14 +94,18 @@ class UtilsTest {
}
@Test
- void redactSensitiveInfoHonorsRedactionRegexFromOperatorConfig() {
+ void redactSensitiveInfoIgnoresDynamicRedactionRegexOverride() {
+ // spark.redaction.regex is intentionally not dynamically overridable: a
runtime override
+ // could weaken or disable redaction, so refresh() drops it and the
startup default pattern
+ // continues to apply.
SparkOperatorConfManager.INSTANCE.refresh(Map.of(REDACTION_REGEX_KEY,
"(?i)confidential"));
Map<String, String> props =
Map.of(
"custom.confidential.conf", "custom-value",
"spark.kubernetes.operator.namespace", "default");
Map<String, String> redacted = Utils.redactSensitiveInfo(props);
- assertEquals(REDACTED_TEXT, redacted.get("custom.confidential.conf"));
+ // The override is ignored, so "confidential" is not treated as sensitive.
+ assertEquals("custom-value", redacted.get("custom.confidential.conf"));
assertEquals("default",
redacted.get("spark.kubernetes.operator.namespace"));
assertEquals(props.keySet(), redacted.keySet());
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
