This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new eec38cd54e1 [fix](kerberos) Preserve catalog realm mappings in shared
Hadoop runtimes (#69608)
eec38cd54e1 is described below
commit eec38cd54e10a2cf8972afcdace05e48aca1549e
Author: Gabriel <[email protected]>
AuthorDate: Mon Oct 26 16:17:02 2026 +0800
[fix](kerberos) Preserve catalog realm mappings in shared Hadoop runtimes
(#69608)
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
The external regression run `Doris_x64_Master_Doris_Regression_External
#1122` accessed Docker Kerberos catalogs before EMR catalogs in the same
FE process. Three suites created and switched their catalogs
successfully, then failed at line 43 (`USE`) while initializing external
metadata:
| Failed suite | Catalog path | Failure point |
| --- | --- | --- |
| `test_hive_migrate_partition_iceberg` | Iceberg HMS | Build the
Iceberg catalog |
| `test_hive_migrate_iceberg` | Iceberg HMS | Build the Iceberg catalog
|
| `test_orc_merge_io_input_streams` | Hive HMS | Create the HMS client |
All three FE stacks end in `getUGIFromSubject ->
HadoopLoginModule.commit -> KerberosName.getShortName ->
NoMatchingRule`. Their catalog properties contained explicit rules for
the EMR principal, but the shared Hadoop mapping state did not match it.
Each suite aborted before its migration-table or ORC queries ran.
Hadoop UGI and `KerberosName` are shared across FE connector plugins,
while the old `configuredAuthMethod` flag belonged to each loaded
authenticator class. After that class's first initialization, later
calls skipped configuration updates entirely, including valid
`auth_to_local` rules; the authentication-method comparison only
controlled a warning. A different plugin's first initialization could
also replace the shared rules. This explains both later-realm failures
and initialization-order dependence. The logs establish the failure
location and suite order; controlled tests reproduce both mechanisms
without assuming which catalog first published the global rules.
The same run's later `test_hive_migrate_paimon` succeeded with the EMR
configuration. This is consistent with a separate plugin's first
initialization replacing shared state, rather than demonstrating that
the failed suites had invalid credentials.
The fix:
- Share the Kerberos helper through the connector and filesystem parent
loaders. Coordinate configuration and UGI construction on
`UserGroupInformation.class`, inspecting actual Hadoop state and
tracking failed full initialization so a corrected catalog can retry
even if Hadoop already changed its authentication mode.
- Append compatible explicit Hadoop mapping rules after existing rules,
including `DEFAULT`, preserving existing first-match precedence.
Canonicalize and deduplicate rules so repeated ticket refreshes do not
accumulate copies.
- Check `isInitialized()` before the lazy security predicate so the
first catalog initializes Hadoop's group mapping service from its own
configuration. Restore previous rules and mechanism if configuration or
principal mapping fails; publish rule extensions only after successful
UGI construction.
- Validate mapping syntax, regular expressions, and mechanisms before
initializing UGI. Reject incompatible mechanisms and MIT rule extensions
without publishing changes: MIT accepts unmatched principals unchanged,
so extending its rules could rename existing users.
- Route the filesystem authenticator through the same initialization
logic and protect its initial and refreshed UGI construction with the
same monitor. JAAS/keytab login remains outside the shared monitor.
Mapping policy remains shared within each Hadoop runtime. Existing
mappings take precedence over overlapping later rules; independent
conflicting policies require separate Hadoop runtimes or a common policy
configured from startup.
### Release note
Fix `NoMatchingRule` failures when Kerberos external catalogs for
different realms share an FE Hadoop runtime.
### Check List (For Author)
- Test
- [x] Unit Test: related Kerberos and HDFS authenticator suites,
including multiple realms in both orders, independent plugin
classloaders, concurrent initialization/refresh, rule deduplication,
filesystem ticket refresh, and invalid configurations, failed principal
mapping rollback, first-catalog group settings, corrected initialization
retries, and actual connector/filesystem loader policies. 76 tests
passed with no failures or errors (30 in fe-kerberos, 31 in hdfs-base, 5
existing HDFS authenticator tests, and 10 plugin-manager tests).
- [x] Manual test: replay the original Docker/EMR mapping configurations
against Hadoop JARs from the failing build with the modified
authenticator; both realm sequences and separate plugin copies succeed.
- FE Checkstyle: full reactor and the three affected modules passed.
- The complete external regression pipeline and environment-dependent
KDC/HDFS integration tests were not run locally.
- Behavior changed:
- [x] Yes. Later compatible realm rules are retained alongside earlier
mappings; incompatible rule mechanisms fail explicitly.
- Does this need documentation?
- [x] No. No new catalog properties or configuration options.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../doris/connector/ConnectorPluginManager.java | 3 +-
.../apache/doris/fs/FileSystemPluginManager.java | 5 +-
.../connector/ConnectorPluginHadoopPatchTest.java | 51 +++
.../doris/fs/FileSystemPluginManagerTest.java | 17 +
.../hdfs/KerberosHadoopAuthenticator.java | 59 +---
.../hdfs/KerberosHadoopAuthenticatorTest.java | 129 +++++++
.../kerberos/HadoopKerberosAuthenticator.java | 134 ++++++--
.../kerberos/HadoopKerberosAuthenticatorTest.java | 370 +++++++++++++++++++++
8 files changed, 694 insertions(+), 74 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java
b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java
index 4eddb056346..e660a042585 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java
@@ -122,9 +122,10 @@ public class ConnectorPluginManager {
// of hadoop-common in the plugin directory (see
be-java-extensions/jni-bootstrap PluginRuntime).
//
// Package-private so ConnectorPluginHadoopPatchTest asserts against this
list, not a copy of it.
+ // Kerberos setup failure state must share Hadoop's loader so another
plugin can retry partial setup.
static final List<String> CONNECTOR_PARENT_FIRST_PREFIXES =
Arrays.asList("org.apache.doris.connector.",
"org.apache.doris.filesystem.",
- "org.apache.hadoop.");
+ "org.apache.doris.kerberos.", "org.apache.hadoop.");
/** Family label in the process-wide {@link PluginRegistry}. */
private static final String PLUGIN_FAMILY = "CONNECTOR";
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java
b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java
index 4c3d5954830..08534c04957 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java
@@ -86,9 +86,10 @@ public class FileSystemPluginManager {
// different Class objects for the type ExecutionAuthenticator used in
the signature"
// (and cross-loader catch blocks for StoragePropertiesException would
silently stop
// matching). Foundation is a shared contract layer by definition —
always parent-first.
- private static final List<String> FS_PARENT_FIRST_PREFIXES =
+ // Share Kerberos setup failure state with connectors that use the same
parent-loaded Hadoop runtime.
+ static final List<String> FS_PARENT_FIRST_PREFIXES =
Arrays.asList("org.apache.doris.filesystem.",
"org.apache.doris.foundation.",
- "software.amazon.awssdk.", "org.apache.hadoop.");
+ "org.apache.doris.kerberos.", "software.amazon.awssdk.",
"org.apache.hadoop.");
/** Family label in the process-wide {@link PluginRegistry}. */
private static final String PLUGIN_FAMILY = "FILESYSTEM";
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginHadoopPatchTest.java
b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginHadoopPatchTest.java
index 4e055c2d850..20b159c1c38 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginHadoopPatchTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginHadoopPatchTest.java
@@ -19,13 +19,18 @@ package org.apache.doris.connector;
import org.apache.doris.extension.loader.ClassLoadingPolicy;
import org.apache.doris.extension.loader.PluginLoader;
+import org.apache.doris.kerberos.HadoopKerberosAuthenticator;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.IOException;
import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Files;
@@ -80,6 +85,44 @@ public class ConnectorPluginHadoopPatchTest {
Assertions.assertSame(pluginLoader, pluginCopy.getClassLoader());
}
+ @Test
+ public void kerberosInitializationCanRetryThroughAnotherPlugin() throws
Exception {
+ String originalRealm = System.getProperty("java.security.krb5.realm");
+ String originalKdc = System.getProperty("java.security.krb5.kdc");
+ System.setProperty("java.security.krb5.realm", "DEFAULT.EXAMPLE");
+ System.setProperty("java.security.krb5.kdc", "localhost");
+ UserGroupInformation.reset();
+ String entry =
"org/apache/doris/kerberos/HadoopKerberosAuthenticator.class";
+ Path jar = jarWith(entry, classBytes(entry, false), "kerberos.jar");
+ try (URLClassLoader first = (URLClassLoader) loaderOver(jar,
+ ConnectorPluginManager.CONNECTOR_PARENT_FIRST_PREFIXES,
getClass().getClassLoader());
+ URLClassLoader second = (URLClassLoader) loaderOver(jar,
+
ConnectorPluginManager.CONNECTOR_PARENT_FIRST_PREFIXES,
getClass().getClassLoader())) {
+ Class<?> firstAuth =
first.loadClass(HadoopKerberosAuthenticator.class.getName());
+ Class<?> secondAuth =
second.loadClass(HadoopKerberosAuthenticator.class.getName());
+ Assertions.assertSame(HadoopKerberosAuthenticator.class,
firstAuth);
+ Assertions.assertSame(firstAuth, secondAuth);
+ UserGroupInformation.setConfiguration(new Configuration(false));
+ Configuration invalid = new Configuration(false);
+ invalid.set("hadoop.security.authentication", "kerberos");
+ invalid.set("hadoop.security.auth_to_local",
"RULE:[2:$1]s/^hdfs$/old-user/");
+ invalid.set("hadoop.kerberos.min.seconds.before.relogin", "oops");
+ InvocationTargetException failure =
Assertions.assertThrows(InvocationTargetException.class,
+ () -> firstAuth.getMethod("initializeAuthConfig",
Configuration.class).invoke(null, invalid));
+ Assertions.assertInstanceOf(IllegalArgumentException.class,
failure.getCause());
+
+ Configuration corrected = new Configuration(invalid);
+ corrected.unset("hadoop.kerberos.min.seconds.before.relogin");
+ corrected.set("hadoop.security.auth_to_local",
"RULE:[2:$1]s/^hdfs$/new-user/");
+ secondAuth.getMethod("initializeAuthConfig",
Configuration.class).invoke(null, corrected);
+ Assertions.assertEquals("new-user", new
KerberosName("hdfs/[email protected]").getShortName());
+ } finally {
+ UserGroupInformation.reset();
+ restoreProperty("java.security.krb5.realm", originalRealm);
+ restoreProperty("java.security.krb5.kdc", originalKdc);
+ }
+ }
+
@Test
public void
testParentFirstFallsBackToThePluginWhenTheKernelHasNothingToOffer() throws
Exception {
// The prefix is a delegation ORDER, not an exclusive claim. It is
what keeps the entry from
@@ -103,6 +146,14 @@ public class ConnectorPluginHadoopPatchTest {
return loaderOver(vanillaFileSystemJar(), parentFirstPrefixes,
getClass().getClassLoader());
}
+ private static void restoreProperty(String name, String value) {
+ if (value == null) {
+ System.clearProperty(name);
+ } else {
+ System.setProperty(name, value);
+ }
+ }
+
private ClassLoader loaderOver(Path jar, List<String> parentFirstPrefixes,
ClassLoader parent)
throws IOException {
URL[] urls = {jar.toUri().toURL()};
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java
index 29feec06d54..190ac4737ec 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java
@@ -18,15 +18,20 @@
package org.apache.doris.fs;
import org.apache.doris.common.util.DatasourcePrintableMap;
+import org.apache.doris.extension.loader.ClassLoadingPolicy;
+import org.apache.doris.extension.loader.PluginLoader;
import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.filesystem.FileSystemType;
import org.apache.doris.filesystem.properties.FileSystemProperties;
import org.apache.doris.filesystem.properties.StorageKind;
import org.apache.doris.filesystem.spi.FileSystemProvider;
+import org.apache.doris.kerberos.HadoopKerberosAuthenticator;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.net.URL;
+import java.net.URLClassLoader;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
@@ -35,6 +40,18 @@ import java.util.Set;
public class FileSystemPluginManagerTest {
+ @Test
+ public void filesystemPluginSharesKerberosInitializationState() throws
Exception {
+ URL classes =
HadoopKerberosAuthenticator.class.getProtectionDomain().getCodeSource().getLocation();
+ PluginLoader loader = new PluginLoader(new ClassLoadingPolicy(
+
FileSystemPluginManager.FS_PARENT_FIRST_PREFIXES).toParentFirstPackages());
+ try (URLClassLoader plugin = (URLClassLoader) loader.createClassLoader(
+ new URL[] {classes}, getClass().getClassLoader())) {
+ Assertions.assertSame(HadoopKerberosAuthenticator.class,
+
plugin.loadClass(HadoopKerberosAuthenticator.class.getName()));
+ }
+ }
+
@Test
public void registerProvider_registersProviderSensitiveKeysForMasking() {
FileSystemPluginManager manager = new FileSystemPluginManager();
diff --git
a/fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticator.java
b/fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticator.java
index ac8370a6a7a..0c8e39d3bea 100644
---
a/fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticator.java
+++
b/fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticator.java
@@ -23,11 +23,10 @@ package org.apache.doris.filesystem.hdfs;
import org.apache.doris.foundation.security.ExecutionAuthenticator;
import org.apache.doris.foundation.security.KerberosTicketUtils;
+import org.apache.doris.kerberos.HadoopKerberosAuthenticator;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
-import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -58,21 +57,14 @@ import javax.security.auth.login.LoginException;
* SASL/GSS layer falls back to the JVM-default interactive JAAS login and
fails with
* "LoginException: Cannot read from System.in".
*
- * <p>Note: {@link UserGroupInformation#setConfiguration(Configuration)}
mutates
- * JVM-global state — all UGI instances in the process share the same
authentication
- * mode. Multiple HDFS catalogs with differing {@code
hadoop.security.authentication}
- * values therefore cannot truly coexist: the first writer wins. We serialise
the
- * setup via {@link #UGI_INIT_LOCK} and skip the {@code setConfiguration} call
when
- * the existing auth method already matches, so concurrent Kerberos catalogs
don't
- * stomp on each other. When modes disagree a WARN is logged so operators
notice.
+ * <p>UGI authentication and auth_to_local rules are shared with connector
plugins.
+ * {@link HadoopKerberosAuthenticator#getUGIFromSubject(Configuration,
Subject)} initializes
+ * that state and retains later catalogs' rules only after successful
principal mapping.
*/
public class KerberosHadoopAuthenticator implements ExecutionAuthenticator {
private static final Logger LOG =
LogManager.getLogger(KerberosHadoopAuthenticator.class);
- /** Process-wide lock for serialising UGI static setup. */
- private static final Object UGI_INIT_LOCK = new Object();
-
private final String principal;
private final String keytab;
@@ -88,15 +80,9 @@ public class KerberosHadoopAuthenticator implements
ExecutionAuthenticator {
this.principal = principal;
this.keytab = keytab;
try {
- synchronized (UGI_INIT_LOCK) {
- AuthenticationMethod desired =
SecurityUtil.getAuthenticationMethod(conf);
- if (!shouldSkipSetConfiguration(desired)) {
- UserGroupInformation.setConfiguration(conf);
- }
- this.subject = loginSubject();
- this.ugi =
Objects.requireNonNull(UserGroupInformation.getUGIFromSubject(subject),
- "getUGIFromSubject returned null");
- }
+ this.subject = loginSubject();
+ this.ugi =
Objects.requireNonNull(HadoopKerberosAuthenticator.getUGIFromSubject(conf,
subject),
+ "getUGIFromSubject returned null");
this.nextRefreshTime = KerberosTicketUtils.getRefreshTime(
KerberosTicketUtils.getTicketGrantingTicket(subject));
LOG.info("Kerberos login succeeded for principal={}", principal);
@@ -106,31 +92,6 @@ public class KerberosHadoopAuthenticator implements
ExecutionAuthenticator {
}
}
- /**
- * Returns true when the JVM-global UGI configuration already matches what
this
- * authenticator needs, so we can safely skip the {@code setConfiguration}
call.
- * If security is on but the existing auth mode differs, a WARN is logged
and
- * we still skip to preserve first-writer-wins semantics — the operator is
then
- * responsible for aligning catalog configs.
- */
- private boolean shouldSkipSetConfiguration(AuthenticationMethod desired) {
- if (!UserGroupInformation.isSecurityEnabled()) {
- return false;
- }
- AuthenticationMethod current;
- try {
- current =
UserGroupInformation.getLoginUser().getAuthenticationMethod();
- } catch (IOException e) {
- return false;
- }
- if (current == desired) {
- return true;
- }
- LOG.warn("UGI already configured with authentication={} but this
catalog requests {}; "
- + "keeping existing JVM-global setting (first-writer-wins).",
current, desired);
- return true;
- }
-
@Override
public <T> T execute(Callable<T> task) throws Exception {
UserGroupInformation currentUgi;
@@ -161,8 +122,10 @@ public class KerberosHadoopAuthenticator implements
ExecutionAuthenticator {
private synchronized UserGroupInformation getUGI() throws IOException {
if (nextRefreshTime < System.currentTimeMillis()) {
Subject newSubject = loginSubject();
-
Objects.requireNonNull(UserGroupInformation.getUGIFromSubject(newSubject),
- "getUGIFromSubject returned null");
+ synchronized (UserGroupInformation.class) {
+
Objects.requireNonNull(UserGroupInformation.getUGIFromSubject(newSubject),
+ "getUGIFromSubject returned null");
+ }
// We modify the existing UGI's credentials in-place instead of
returning a new UGI
// because some parts of Hadoop code reuse UGI (e.g. DFSClient).
// We also need to clear the old credentials because the JDK
assumes that the first
diff --git
a/fe/fe-filesystem/fe-filesystem-hdfs-base/src/test/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticatorTest.java
b/fe/fe-filesystem/fe-filesystem-hdfs-base/src/test/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticatorTest.java
new file mode 100644
index 00000000000..1b3dfe9dee7
--- /dev/null
+++
b/fe/fe-filesystem/fe-filesystem-hdfs-base/src/test/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticatorTest.java
@@ -0,0 +1,129 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.hdfs;
+
+import org.apache.doris.kerberos.HadoopKerberosAuthenticator;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Date;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.security.auth.Subject;
+import javax.security.auth.kerberos.KerberosPrincipal;
+import javax.security.auth.kerberos.KerberosTicket;
+
+class KerberosHadoopAuthenticatorTest {
+ private String originalRealm;
+ private String originalKdc;
+
+ @BeforeEach
+ void setUp() {
+ originalRealm = System.getProperty("java.security.krb5.realm");
+ originalKdc = System.getProperty("java.security.krb5.kdc");
+ System.setProperty("java.security.krb5.realm", "DEFAULT.EXAMPLE");
+ System.setProperty("java.security.krb5.kdc", "localhost");
+ UserGroupInformation.reset();
+ UserGroupInformation.setConfiguration(configuration("FIRST.EXAMPLE"));
+ }
+
+ @AfterEach
+ void tearDown() {
+ UserGroupInformation.reset();
+ restoreProperty("java.security.krb5.realm", originalRealm);
+ restoreProperty("java.security.krb5.kdc", originalKdc);
+ }
+
+ @Test
+ void filesystemAndConnectorRealmsSurviveTicketRefresh() throws Exception {
+ KerberosHadoopAuthenticator first = authenticator("FIRST.EXAMPLE", new
AtomicInteger());
+ AtomicInteger secondLogins = new AtomicInteger();
+ KerberosHadoopAuthenticator second = Assertions.assertDoesNotThrow(
+ () -> authenticator("SECOND.EXAMPLE", secondLogins));
+ Assertions.assertEquals(1, secondLogins.get());
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("THIRD.EXAMPLE"));
+
+ // The fabricated TGTs have expired, so execute() must rebuild the UGI
before running the action.
+ Assertions.assertEquals("hdfs/[email protected]",
+ first.execute(() ->
UserGroupInformation.getCurrentUser().getUserName()));
+ Assertions.assertEquals("hdfs/[email protected]",
+ second.execute(() ->
UserGroupInformation.getCurrentUser().getUserName()));
+ Assertions.assertEquals(2, secondLogins.get());
+ Subject third = subject("THIRD.EXAMPLE");
+ Assertions.assertEquals("hdfs",
UserGroupInformation.getUGIFromSubject(third).getShortUserName());
+ }
+
+ @Test
+ void failedFilesystemMappingDoesNotPoisonCorrectedCatalogs() throws
Exception {
+ Configuration invalid = configuration("SECOND.EXAMPLE");
+ invalid.set("hadoop.security.auth_to_local", "RULE:[2:$3]");
+ String rules = KerberosName.getRules();
+ Assertions.assertThrows(RuntimeException.class, () -> new
KerberosHadoopAuthenticator(
+ "hdfs/[email protected]", "unused.keytab", invalid) {
+ @Override
+ Subject loginSubject() {
+ return subject("SECOND.EXAMPLE");
+ }
+ });
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ KerberosHadoopAuthenticator corrected =
authenticator("SECOND.EXAMPLE", new AtomicInteger());
+ Assertions.assertEquals("hdfs",
+ corrected.execute(() ->
UserGroupInformation.getCurrentUser().getShortUserName()));
+ Assertions.assertEquals("hdfs", HadoopKerberosAuthenticator
+ .getUGIFromSubject(configuration("THIRD.EXAMPLE"),
subject("THIRD.EXAMPLE")).getShortUserName());
+ }
+
+ private static KerberosHadoopAuthenticator authenticator(String realm,
AtomicInteger logins) {
+ return new KerberosHadoopAuthenticator("hdfs/host@" + realm,
"unused.keytab", configuration(realm)) {
+ @Override
+ Subject loginSubject() {
+ logins.incrementAndGet();
+ return subject(realm);
+ }
+ };
+ }
+
+ private static Subject subject(String realm) {
+ KerberosPrincipal client = new KerberosPrincipal("hdfs/host@" + realm);
+ KerberosTicket ticket = new KerberosTicket(new byte[] {1}, client,
+ new KerberosPrincipal("krbtgt/" + realm + "@" + realm), new
byte[8], 1,
+ null, null, new Date(1000L), new Date(2000L), null, null);
+ return new Subject(false, Set.of(client), Set.of(), Set.of(ticket));
+ }
+
+ private static Configuration configuration(String realm) {
+ Configuration conf = new Configuration(false);
+ conf.set("hadoop.security.authentication", "kerberos");
+ conf.set("hadoop.security.auth_to_local", "RULE:[2:$1@$0](.*@" + realm
+ ")s/@.*// DEFAULT");
+ return conf;
+ }
+
+ private static void restoreProperty(String name, String value) {
+ if (value == null) {
+ System.clearProperty(name);
+ } else {
+ System.setProperty(name, value);
+ }
+ }
+}
diff --git
a/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java
b/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java
index 599f3dbb995..4145264ec68 100644
---
a/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java
+++
b/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java
@@ -22,17 +22,20 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
-import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.io.IOException;
import java.util.Collections;
import java.util.Date;
+import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import javax.security.auth.Subject;
import javax.security.auth.kerberos.KerberosPrincipal;
import javax.security.auth.kerberos.KerberosTicket;
@@ -47,9 +50,14 @@ public class HadoopKerberosAuthenticator implements
HadoopAuthenticator {
private long nextRefreshTime;
private UserGroupInformation ugi;
- // The authentication method the first caller published to the
process-wide UGI configuration.
- // Guarded by HadoopKerberosAuthenticator.class; null until the first
setConfiguration.
- private static UserGroupInformation.AuthenticationMethod
configuredAuthMethod = null;
+ // Shared with Hadoop through the plugin parent loader; guarded by
UserGroupInformation.class.
+ // Hadoop can publish its auth mode before setConfiguration fails, so that
mode alone cannot prove success.
+ private static boolean authConfigurationFailed;
+
+ // Hadoop's rule grammar keeps whitespace inside a regex or substitution
distinct from rule separators.
+ private static final Pattern AUTH_TO_LOCAL_RULE = Pattern.compile(
+ "\\s*((DEFAULT)|(RULE:\\[(\\d*):([^\\]]*)](\\(([^)]*)\\))?"
+ + "(s/([^/]*)/([^/]*)/(g)?)?))/?(L)?");
public HadoopKerberosAuthenticator(KerberosAuthenticationConfig config) {
this.config = config;
@@ -57,26 +65,107 @@ public class HadoopKerberosAuthenticator implements
HadoopAuthenticator {
public static void initializeAuthConfig(Configuration hadoopConf) {
hadoopConf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION,
"true");
- // UserGroupInformation.setConfiguration mutates a single process-wide
global, so multiple HDFS
- // catalogs with differing hadoop.security.authentication cannot truly
coexist. First-writer-wins:
- // only the first caller publishes the global config; later catalogs
(and every ticket refresh,
- // which re-enters this method via login()) skip it so they don't
stomp on each other. A WARN
- // fires only on a genuine auth-method mismatch. We compare against a
remembered method rather than
- // UserGroupInformation.getLoginUser() on purpose: Doris never
establishes a process-wide login user
- // (per-instance getUGIFromSubject only), and getLoginUser() would
trigger exactly that.
- UserGroupInformation.AuthenticationMethod desired =
SecurityUtil.getAuthenticationMethod(hadoopConf);
- synchronized (HadoopKerberosAuthenticator.class) {
- if (configuredAuthMethod == null) {
- UserGroupInformation.setConfiguration(hadoopConf);
- configuredAuthMethod = desired;
+ String additionalRules =
hadoopConf.get(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTH_TO_LOCAL);
+ String mechanism =
hadoopConf.get(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTH_TO_LOCAL_MECHANISM,
+ KerberosName.DEFAULT_MECHANISM);
+ // UGI changes its auth mode before parsing rules. Reject malformed
input before it can leave
+ // an enabled UGI with the permissive rules inherited from a previous
SIMPLE initialization.
+
Preconditions.checkArgument(KerberosName.MECHANISM_HADOOP.equalsIgnoreCase(mechanism)
+ ||
KerberosName.MECHANISM_MIT.equalsIgnoreCase(mechanism),
+ "Invalid auth_to_local mechanism: %s", mechanism);
+ Set<String> requestedRules = additionalRules == null
+ ? Collections.emptySet() :
parseAuthToLocalRules(additionalRules);
+ // Connector and filesystem plugins share Hadoop and this helper
through their parent loader.
+ // Use Hadoop's monitor for both policy changes and the principal
mapping that validates them.
+ synchronized (UserGroupInformation.class) {
+ // isSecurityEnabled() lazily initializes UGI and its Groups
singleton with default settings.
+ if (!UserGroupInformation.isInitialized() ||
authConfigurationFailed
+ || !UserGroupInformation.isSecurityEnabled()) {
+ String previousRules = KerberosName.getRules();
+ String previousMechanism = KerberosName.getRuleMechanism();
+ try {
+ UserGroupInformation.setConfiguration(hadoopConf);
+ authConfigurationFailed = false;
+ } catch (RuntimeException e) {
+ authConfigurationFailed = true;
+ KerberosName.setRules(previousRules);
+ KerberosName.setRuleMechanism(previousMechanism);
+ throw e;
+ }
return;
}
- if (configuredAuthMethod != desired) {
- LOG.warn("UGI already configured with authentication={} but
this catalog requests {}; "
- + "keeping existing JVM-global setting
(first-writer-wins).",
- configuredAuthMethod, desired);
+ if (additionalRules == null) {
+ return;
+ }
+
Preconditions.checkArgument(mechanism.equalsIgnoreCase(KerberosName.getRuleMechanism()),
+ "Kerberos catalogs sharing a Hadoop runtime must use the
same auth_to_local mechanism "
+ + "(existing=%s, requested=%s)",
KerberosName.getRuleMechanism(), mechanism);
+ Set<String> rules = parseAuthToLocalRules(KerberosName.getRules());
+ int previousSize = rules.size();
+ // Append after all existing rules, including DEFAULT: a later
realm must not change an
+ // earlier principal's mapping. Replacing the global configuration
would break its next login.
+ if (rules.addAll(requestedRules)) {
+ // MIT accepts an unmatched principal unchanged; extending its
rules could rename a user.
+
Preconditions.checkArgument(KerberosName.MECHANISM_HADOOP.equalsIgnoreCase(mechanism),
+ "MIT auth_to_local rules cannot be extended after
Hadoop initialization; "
+ + "configure a common policy before
initializing catalogs");
+ KerberosName.setRules(String.join("\n", rules));
+ LOG.debug("Extended JVM-global Kerberos auth_to_local rules
from {} to {} rules",
+ previousSize, rules.size());
+ }
+ }
+ }
+
+ public static UserGroupInformation getUGIFromSubject(Configuration
hadoopConf, Subject subject) throws IOException {
+ synchronized (UserGroupInformation.class) {
+ String previousRules = KerberosName.getRules();
+ String previousMechanism = KerberosName.getRuleMechanism();
+ boolean configuring = !UserGroupInformation.isInitialized() ||
authConfigurationFailed
+ || !UserGroupInformation.isSecurityEnabled();
+ try {
+ initializeAuthConfig(hadoopConf);
+ return UserGroupInformation.getUGIFromSubject(subject);
+ } catch (IOException | RuntimeException e) {
+ // Some valid rule syntax (e.g. an out-of-range $ parameter)
fails only when a principal
+ // is mapped. A failed catalog must not leave that rule ahead
of later corrected rules.
+ KerberosName.setRules(previousRules);
+ KerberosName.setRuleMechanism(previousMechanism);
+ if (configuring) {
+ authConfigurationFailed = true;
+ }
+ throw e;
+ }
+ }
+ }
+
+ private static Set<String> parseAuthToLocalRules(String rawRules) {
+ String rules = rawRules.trim();
+ Set<String> result = new LinkedHashSet<>();
+ Matcher matcher = AUTH_TO_LOCAL_RULE.matcher(rules);
+ int position = 0;
+ while (position < rules.length()) {
+ matcher.region(position, rules.length());
+ if (!matcher.lookingAt()) {
+ throw new IllegalArgumentException("Invalid auth_to_local
rule: " + rules.substring(position));
+ }
+ String rule = matcher.group(1);
+ if (matcher.group(4) != null) {
+ if (matcher.group(7) != null) {
+ Pattern.compile(matcher.group(7));
+ }
+ if (matcher.group(9) != null) {
+ Pattern.compile(matcher.group(9));
+ }
+ rule = "RULE:[" + Integer.parseInt(matcher.group(4)) + ":" +
matcher.group(5) + "]"
+ + Objects.toString(matcher.group(6), "") +
Objects.toString(matcher.group(8), "");
+ if (matcher.group(12) != null) {
+ rule += "/L";
+ }
}
+ result.add(rule);
+ position = matcher.end();
}
+ return result;
}
@Override
@@ -132,13 +221,12 @@ public class HadoopKerberosAuthenticator implements
HadoopAuthenticator {
}
private UserGroupInformation login(Subject subject) throws IOException {
- // login and get ugi when catalog is initialized
- initializeAuthConfig(config.getConf());
+ // Login and get UGI when the catalog is initialized.
String principal = config.getKerberosPrincipal();
if (LOG.isDebugEnabled()) {
LOG.debug("Login by kerberos authentication with principal: {}",
principal);
}
- return UserGroupInformation.getUGIFromSubject(subject);
+ return getUGIFromSubject(config.getConf(), subject);
}
private static long calculateNextRefreshTime(Subject subject) {
diff --git
a/fe/fe-kerberos/src/test/java/org/apache/doris/kerberos/HadoopKerberosAuthenticatorTest.java
b/fe/fe-kerberos/src/test/java/org/apache/doris/kerberos/HadoopKerberosAuthenticatorTest.java
new file mode 100644
index 00000000000..332110824b8
--- /dev/null
+++
b/fe/fe-kerberos/src/test/java/org/apache/doris/kerberos/HadoopKerberosAuthenticatorTest.java
@@ -0,0 +1,370 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.kerberos;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.Groups;
+import org.apache.hadoop.security.KerberosAuthException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import javax.security.auth.Subject;
+import javax.security.auth.kerberos.KerberosPrincipal;
+
+class HadoopKerberosAuthenticatorTest {
+ private static final String AUTH_TO_LOCAL =
"hadoop.security.auth_to_local";
+ private static final String RULE_MECHANISM =
"hadoop.security.auth_to_local.mechanism";
+ private String originalRealm;
+ private String originalKdc;
+
+ @BeforeEach
+ void setUp() {
+ originalRealm = System.getProperty("java.security.krb5.realm");
+ originalKdc = System.getProperty("java.security.krb5.kdc");
+ System.setProperty("java.security.krb5.realm", "DEFAULT.EXAMPLE");
+ System.setProperty("java.security.krb5.kdc", "localhost");
+ UserGroupInformation.reset();
+ UserGroupInformation.setConfiguration(configuration("FIRST.EXAMPLE"));
+ }
+
+ @AfterEach
+ void tearDown() {
+ UserGroupInformation.reset();
+ restoreProperty("java.security.krb5.realm", originalRealm);
+ restoreProperty("java.security.krb5.kdc", originalKdc);
+ }
+
+ @Test
+ void laterRealmCanCreateUgiWithoutBreakingEarlierRealm() throws Exception {
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("FIRST.EXAMPLE"));
+ assertUser("FIRST.EXAMPLE");
+
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("SECOND.EXAMPLE"));
+ assertUser("SECOND.EXAMPLE");
+ assertUser("FIRST.EXAMPLE");
+ }
+
+ @Test
+ void firstKerberosCatalogInitializesSharedUgi() {
+ UserGroupInformation.reset();
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("FIRST.EXAMPLE"));
+ Assertions.assertTrue(UserGroupInformation.isSecurityEnabled());
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("SECOND.EXAMPLE"));
+ assertUser("FIRST.EXAMPLE");
+ assertUser("SECOND.EXAMPLE");
+ }
+
+ @Test
+ void firstCatalogKeepsItsGroupMappingConfiguration() throws Exception {
+ UserGroupInformation.reset();
+ Groups.reset();
+ try {
+ Configuration conf = configuration("FIRST.EXAMPLE");
+ conf.set("hadoop.user.group.static.mapping.overrides",
"hdfs=analytics");
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+ Assertions.assertArrayEquals(new String[] {"analytics"},
user("FIRST.EXAMPLE").getGroupNames());
+ } finally {
+ Groups.reset();
+ }
+ }
+
+ @Test
+ void failedInitialConfigurationCanBeCorrected() throws Exception {
+ UserGroupInformation.setConfiguration(new Configuration(false));
+ Configuration invalid = configuration("FIRST.EXAMPLE");
+ invalid.set(AUTH_TO_LOCAL, "RULE:[2:$1]s/^hdfs$/old-user/");
+ invalid.set("hadoop.kerberos.min.seconds.before.relogin", "oops");
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () ->
HadoopKerberosAuthenticator.initializeAuthConfig(invalid));
+
+ Configuration corrected = configuration("FIRST.EXAMPLE");
+ corrected.set(AUTH_TO_LOCAL, "RULE:[2:$1]s/^hdfs$/new-user/");
+ HadoopKerberosAuthenticator.initializeAuthConfig(corrected);
+ Assertions.assertEquals("new-user",
user("FIRST.EXAMPLE").getShortUserName());
+ }
+
+ @Test
+ void failedPrincipalMappingDoesNotPoisonLaterCatalogs() throws Exception {
+ Configuration invalid = configuration("SECOND.EXAMPLE");
+ invalid.set(AUTH_TO_LOCAL, "RULE:[2:$3]");
+ String rules = KerberosName.getRules();
+ Subject subject = new Subject(false, Set.of(new
KerberosPrincipal("hdfs/[email protected]")),
+ Set.of(), Set.of());
+ Assertions.assertThrows(KerberosAuthException.class,
+ () -> HadoopKerberosAuthenticator.getUGIFromSubject(invalid,
subject));
+ Assertions.assertEquals(rules, KerberosName.getRules());
+
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("SECOND.EXAMPLE"));
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("THIRD.EXAMPLE"));
+ assertUser("FIRST.EXAMPLE");
+ assertUser("SECOND.EXAMPLE");
+ assertUser("THIRD.EXAMPLE");
+ }
+
+ @Test
+ void failedFirstPrincipalMappingCanBeCorrected() throws Exception {
+ UserGroupInformation.reset();
+ Configuration invalid = configuration("FIRST.EXAMPLE");
+ invalid.set(AUTH_TO_LOCAL, "RULE:[2:$3]");
+ Subject subject = new Subject(false, Set.of(new
KerberosPrincipal("hdfs/[email protected]")),
+ Set.of(), Set.of());
+ Assertions.assertThrows(KerberosAuthException.class,
+ () -> HadoopKerberosAuthenticator.getUGIFromSubject(invalid,
subject));
+ Assertions.assertNull(KerberosName.getRules());
+ Assertions.assertEquals("hdfs", HadoopKerberosAuthenticator
+ .getUGIFromSubject(configuration("FIRST.EXAMPLE"),
subject).getShortUserName());
+ }
+
+ @Test
+ void reverseInitializationOrderKeepsBothRealms() {
+ UserGroupInformation.setConfiguration(configuration("SECOND.EXAMPLE"));
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("FIRST.EXAMPLE"));
+ assertUser("FIRST.EXAMPLE");
+ assertUser("SECOND.EXAMPLE");
+ }
+
+ @Test
+ void independentPluginCopiesDoNotReplaceSharedRules() throws Exception {
+ URL classes =
HadoopKerberosAuthenticator.class.getProtectionDomain().getCodeSource().getLocation();
+ try (URLClassLoader first = pluginLoader(classes); URLClassLoader
second = pluginLoader(classes)) {
+ Class<?> firstAuth =
first.loadClass(HadoopKerberosAuthenticator.class.getName());
+ Class<?> secondAuth =
second.loadClass(HadoopKerberosAuthenticator.class.getName());
+ Assertions.assertNotSame(firstAuth, secondAuth);
+
Assertions.assertSame(first.loadClass(UserGroupInformation.class.getName()),
+ second.loadClass(UserGroupInformation.class.getName()));
+ firstAuth.getMethod("initializeAuthConfig", Configuration.class)
+ .invoke(null, configuration("FIRST.EXAMPLE"));
+ secondAuth.getMethod("initializeAuthConfig", Configuration.class)
+ .invoke(null, configuration("SECOND.EXAMPLE"));
+
+ assertUser("FIRST.EXAMPLE");
+ assertUser("SECOND.EXAMPLE");
+ }
+ }
+
+ @Test
+ void concurrentInitializationAndRefreshKeepEveryRealm() throws Exception {
+ int catalogs = 8;
+ ExecutorService executor = Executors.newFixedThreadPool(catalogs);
+ CountDownLatch start = new CountDownLatch(1);
+ List<Future<?>> futures = new ArrayList<>();
+ try {
+ for (int i = 0; i < catalogs; i++) {
+ String realm = "REALM" + i + ".EXAMPLE";
+ futures.add(executor.submit(() -> {
+ Assertions.assertTrue(start.await(10, TimeUnit.SECONDS));
+ for (int refresh = 0; refresh < 20; refresh++) {
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration(realm));
+ assertUser(realm);
+ assertUser("FIRST.EXAMPLE");
+ }
+ return null;
+ }));
+ }
+ start.countDown();
+ for (Future<?> future : futures) {
+ Assertions.assertDoesNotThrow(() -> future.get(30,
TimeUnit.SECONDS));
+ }
+ } finally {
+ executor.shutdownNow();
+ Assertions.assertTrue(executor.awaitTermination(10,
TimeUnit.SECONDS));
+ }
+ }
+
+ @Test
+ void refreshDoesNotAccumulateDuplicateRules() throws Exception {
+ Configuration conf = configuration("SECOND.EXAMPLE");
+ conf.set(AUTH_TO_LOCAL, " RULE:[02:$1@$0](.*@SECOND.EXAMPLE)s/@.*//
DEFAULT ");
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+ String rules = KerberosName.getRules();
+ assertUser("SECOND.EXAMPLE");
+ for (int refresh = 0; refresh < 20; refresh++) {
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+ }
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ Assertions.assertEquals(3, rules.lines().count());
+ }
+
+ @Test
+ void lowercaseFlagsAndDefaultRulesAreCanonicalized() {
+ Configuration conf = configuration("SECOND.EXAMPLE");
+ conf.set(AUTH_TO_LOCAL, "RULE:[02:$1@$0](.*@SECOND.EXAMPLE)s/@.*//g/L
DEFAULT/L");
+ for (int refresh = 0; refresh < 20; refresh++) {
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+ }
+ Assertions.assertEquals(3, KerberosName.getRules().lines().count());
+ assertUser("FIRST.EXAMPLE");
+ assertUser("SECOND.EXAMPLE");
+ }
+
+ @Test
+ void whitespaceInsideSubstitutionIsPreserved() throws Exception {
+ Configuration conf = configuration("SECOND.EXAMPLE");
+ conf.set(AUTH_TO_LOCAL,
+ "RULE:[2:$1@$0](.*@SECOND.EXAMPLE)s/(hdfs|user
name)@.*/mapped-user/ DEFAULT");
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+ Assertions.assertEquals("mapped-user",
user("SECOND.EXAMPLE").getShortUserName());
+ String rules = KerberosName.getRules();
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ }
+
+ @Test
+ void extendingMitRulesIsRejectedWithoutChangingFallbackMappings() throws
Exception {
+ Configuration first = configuration("FIRST.EXAMPLE");
+ first.set(RULE_MECHANISM, "mit");
+ UserGroupInformation.setConfiguration(first);
+ Configuration second = configuration("SECOND.EXAMPLE");
+ second.set(RULE_MECHANISM, "mit");
+ String rules = KerberosName.getRules();
+ String originalUser = user("SECOND.EXAMPLE").getShortUserName();
+
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () ->
HadoopKerberosAuthenticator.initializeAuthConfig(second));
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ Assertions.assertEquals(originalUser,
user("SECOND.EXAMPLE").getShortUserName());
+ Assertions.assertDoesNotThrow(() ->
HadoopKerberosAuthenticator.initializeAuthConfig(first));
+ }
+
+ @Test
+ void existingRulesKeepTheirPrecedence() throws Exception {
+ Configuration conf = configuration("FIRST.EXAMPLE");
+ conf.set(AUTH_TO_LOCAL,
"RULE:[2:$1@$0](.*@FIRST.EXAMPLE)s/.*/another-user/ DEFAULT");
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+
+ assertUser("FIRST.EXAMPLE");
+ Assertions.assertThrows(KerberosAuthException.class, () ->
user("UNCONFIGURED.EXAMPLE"));
+ }
+
+ @Test
+ void invalidRulesDoNotChangePublishedRules() {
+ Configuration conf = configuration("SECOND.EXAMPLE");
+ conf.set(AUTH_TO_LOCAL, "INVALID");
+ String rules = KerberosName.getRules();
+
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HadoopKerberosAuthenticator.initializeAuthConfig(conf));
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ }
+
+ @Test
+ void invalidFirstCatalogCannotPublishPermissiveFallback() {
+ Configuration invalidSyntax = configuration("FIRST.EXAMPLE");
+ invalidSyntax.set(AUTH_TO_LOCAL, "INVALID");
+ Configuration invalidRegex = configuration("FIRST.EXAMPLE");
+ invalidRegex.set(AUTH_TO_LOCAL, "RULE:[2:$1@$0]s/[/x/");
+ Configuration invalidMechanism = configuration("FIRST.EXAMPLE");
+ invalidMechanism.set(RULE_MECHANISM, "unknown");
+ for (Configuration invalid : List.of(invalidSyntax, invalidRegex,
invalidMechanism)) {
+ UserGroupInformation.reset();
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () ->
HadoopKerberosAuthenticator.initializeAuthConfig(invalid));
+ Assertions.assertFalse(UserGroupInformation.isSecurityEnabled());
+
HadoopKerberosAuthenticator.initializeAuthConfig(configuration("SECOND.EXAMPLE"));
+ assertUser("SECOND.EXAMPLE");
+ Assertions.assertThrows(KerberosAuthException.class, () ->
user("FIRST.EXAMPLE"));
+ }
+ }
+
+ @Test
+ void incompatibleRuleMechanismDoesNotChangePublishedRules() {
+ Configuration conf = configuration("SECOND.EXAMPLE");
+ conf.set(RULE_MECHANISM, "mit");
+ String rules = KerberosName.getRules();
+
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HadoopKerberosAuthenticator.initializeAuthConfig(conf));
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ Assertions.assertEquals("hadoop", KerberosName.getRuleMechanism());
+ }
+
+ @Test
+ void absentRulesDoNotBroadenExistingPolicy() throws Exception {
+ Configuration conf = configuration("SECOND.EXAMPLE");
+ conf.unset(AUTH_TO_LOCAL);
+ String rules = KerberosName.getRules();
+ HadoopKerberosAuthenticator.initializeAuthConfig(conf);
+
+ Assertions.assertEquals(rules, KerberosName.getRules());
+ assertUser("FIRST.EXAMPLE");
+ Assertions.assertThrows(KerberosAuthException.class, () ->
user("SECOND.EXAMPLE"));
+ }
+
+ private static Configuration configuration(String realm) {
+ Configuration conf = new Configuration(false);
+ conf.set("hadoop.security.authentication", "kerberos");
+ conf.set(AUTH_TO_LOCAL, "RULE:[2:$1@$0](.*@" + realm +
")s/@.*//\nDEFAULT");
+ return conf;
+ }
+
+ private static void assertUser(String realm) {
+ UserGroupInformation ugi = Assertions.assertDoesNotThrow(() ->
user(realm));
+ Assertions.assertEquals("hdfs", ugi.getShortUserName());
+ }
+
+ private static UserGroupInformation user(String realm) throws Exception {
+ Subject subject = new Subject(false, Set.of(new
KerberosPrincipal("hdfs/host@" + realm)),
+ Set.of(), Set.of());
+ synchronized (UserGroupInformation.class) {
+ return UserGroupInformation.getUGIFromSubject(subject);
+ }
+ }
+
+ private static URLClassLoader pluginLoader(URL classes) {
+ return new URLClassLoader(new URL[] {classes},
HadoopKerberosAuthenticatorTest.class.getClassLoader()) {
+ @Override
+ protected Class<?> loadClass(String name, boolean resolve) throws
ClassNotFoundException {
+ synchronized (getClassLoadingLock(name)) {
+ if (name.startsWith("org.apache.doris.kerberos.")) {
+ Class<?> loaded = findLoadedClass(name);
+ if (loaded == null) {
+ loaded = findClass(name);
+ }
+ if (resolve) {
+ resolveClass(loaded);
+ }
+ return loaded;
+ }
+ return super.loadClass(name, resolve);
+ }
+ }
+ };
+ }
+
+ private static void restoreProperty(String name, String value) {
+ if (value == null) {
+ System.clearProperty(name);
+ } else {
+ System.setProperty(name, value);
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
