This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 4a54c47c4ef [fix](regression) Backport external and Kerberos fixes to
branch-4.1 (#68150, #66433) (#68869)
4a54c47c4ef is described below
commit 4a54c47c4eff245cee51b3427e85e0b2e09303c9
Author: Gabriel <[email protected]>
AuthorDate: Sat Oct 10 18:32:58 2026 +0800
[fix](regression) Backport external and Kerberos fixes to branch-4.1
(#68150, #66433) (#68869)
Backport #68150 and #66433 to branch-4.1.
Trino startup/query failures and unfinished HDFS Broker Load jobs could
appear as empty-result mismatches. Kerberized HDFS clients lacked the
DataNode SASL setting, and the lightweight Kerberos metastore lacked the
Paimon fixture. Iceberg metadata write failures could expose only a
self-suppression wrapper.
This backport adds bounded readiness/load checks with failure
diagnostics, configures HDFS transfer protection, restores the Paimon
fixture before the Kerberos readiness marker, and reports the root cause
of failed Iceberg metadata commits.
Branch adaptation: branch-4.1 uses `IcebergMetadataOps`, so the
root-cause changes are applied to its create-table, schema-update, and
branch/tag commit paths. Its partition-spec update helper already
reports root causes. The newer connector implementation is not
introduced.
Validation:
- FE Checkstyle passes: `mvn -pl fe-core -am checkstyle:check`.
- All six changed suites compile with Groovy 4.0.19.
- Eight local control-flow scenarios pass for Trino readiness/query
errors and Broker Load completion/cancellation/timeouts.
- Shell syntax, Kerberos settings isolation, both container role/switch
combinations, env/config rendering, and rendered XML checks pass.
- An isolated harness compiles the actual adapted
create-table/schema-commit methods with stubbed dependencies: the
original masks the storage failure, while the backport preserves the
root message, operation context, and exception cause.
- `git diff --check` passes. Full FE build and external integration runs
remain pending CI.
Release note: Improved external regression setup and diagnostics;
Iceberg metadata commit errors now expose the underlying cause.
Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
---
.../kerberos/entrypoint-hive-master.sh | 26 ++++++++++++
.../docker-compose/kerberos/hadoop-hive.env.tpl | 4 ++
.../docker-compose/kerberos/kerberos.yaml.tpl | 6 +++
.../kerberos/sql/create_paimon_hive_table.hql | 20 +++++++++
docker/thirdparties/run-thirdparties-docker.sh | 47 +++++++++++++++++++++-
.../datasource/iceberg/IcebergMetadataOps.java | 16 ++++----
.../iceberg/test_iceberg_deletion_vector.groovy | 42 ++++++++++++-------
.../refactor_storage_param/hdfs_all_test.groovy | 2 +
.../tvf/test_s3_tvf_number_range.groovy | 32 +++++++--------
.../paimon/test_paimon_hms_catalog.groovy | 4 +-
.../hive_on_hms_and_dlf.groovy | 4 +-
.../iceberg_on_hms_and_filesystem_and_dlf.groovy | 4 +-
12 files changed, 165 insertions(+), 42 deletions(-)
diff --git
a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
index 6735ae7ee40..885a415d996 100644
--- a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
+++ b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
@@ -136,9 +136,35 @@ kdestroy
report_stage "initialize-hive-metastore"
schematool -dbType derby -initSchema
+# The Paimon table declares no columns, so the metastore itself resolves its
+# schema through the storage handler. The handler class must therefore be on
the
+# metastore service's own classpath, not merely on the DDL client's.
+if [[ -d /opt/doris/auxlib ]]; then
+ export HIVE_AUX_JARS_PATH=/opt/doris/auxlib
+fi
start_service hive --service metastore -p "${HMS_PORT}"
wait_for_port "${HOST}" "${HMS_PORT}" "Hive Metastore"
+# Register the Paimon fixture consumed by test_paimon_hms_catalog. This runs
+# before the readiness marker below on purpose: run-thirdparties-docker.sh
+# releases the pipeline on DORIS_KERBEROS_READY, so anything published later
+# would race the suites. A failure here aborts the container (set -e) instead
of
+# handing out an environment that is silently missing hdfs_db.
+#
+# Both kerberos containers run this entrypoint with the same env switch, but
the
+# fixture and its mounts (sql/, paimon_data/, auxlib/) belong to kerberos1
only -
+# its metastore (9583) is the one the suite talks to. Gate on the container
role,
+# not on the mounts, so a broken mount on kerberos1 still fails loudly.
+if [[ "${enablePaimonHms:-false}" == "true" && "${HOST:-}" == "hadoop-master"
]]; then
+ report_stage "load-paimon-hms"
+ export KRB5CCNAME=FILE:/tmp/hive-admin.ccache
+ kinit -kt /data/keytabs/hive.keytab "${HIVE_PRINCIPAL}"
+ hdfs dfs -mkdir -p /user/hive/warehouse
+ hdfs dfs -put -f /opt/doris/paimon_data/* /user/hive/warehouse/
+ hive -f /opt/doris/sql/create_paimon_hive_table.hql
+ kdestroy
+fi
+
touch /tmp/SUCCESS
echo "Minimal Kerberos HDFS and Hive Metastore environment is ready"
echo "DORIS_KERBEROS_READY"
diff --git a/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
b/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
index 792d0c0491d..ff95e56f1ae 100644
--- a/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
@@ -29,3 +29,7 @@ PRESTO_CLIENT_KEYTAB=${PRESTO_CLIENT_KEYTAB}
HADOOP_CONF_DIR=/opt/doris/conf
HIVE_CONF_DIR=/opt/doris/conf
KRB5_CONFIG=/etc/krb5.conf
+# Paimon-on-kerberized-HMS fixture switch. Read from hive-3x_settings.env by
+# start_kerberos() rather than duplicated into kerberos*_settings.env, so that
+# the pipeline keeps patching exactly one file and the two stacks cannot drift.
+enablePaimonHms=${enablePaimonHms}
diff --git a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
index d21496516af..8e903ed6d7f 100644
--- a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
@@ -33,11 +33,17 @@ services:
<<: *kerberos-service
container_name: doris-${CONTAINER_UID}-kerberos1
hostname: hadoop-master
+ # auxlib/sql/paimon_data are mounted on kerberos1 only:
test_paimon_hms_catalog
+ # is the sole consumer and it talks to this metastore (9583). Leaving
kerberos2
+ # untouched keeps the second container as light as it was.
volumes:
- ./conf/kerberos1:/opt/doris/conf:ro
- ./conf/kerberos1/krb5.conf:/etc/krb5.conf:ro
- ./data/kerberos1:/data
- ./two-kerberos-hives:/keytabs
+ - ./auxlib:/opt/doris/auxlib:ro
+ - ./sql:/opt/doris/sql:ro
+ - ./paimon_data:/opt/doris/paimon_data:ro
env_file:
- ./hadoop-hive-1.env
diff --git
a/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
b/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
new file mode 100644
index 00000000000..69a6f476286
--- /dev/null
+++
b/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
@@ -0,0 +1,20 @@
+-- Paimon table registered in the kerberized Hive Metastore, consumed by the
+-- hdfs_kerberos / hdfs_new_kerberos cases of
+-- external_table_p2/paimon/test_paimon_hms_catalog.groovy.
+--
+-- hdfs_db is backed by ../paimon_data, which the entrypoint uploads into HDFS.
+--
+-- The table declares no columns, so the metastore derives them through the
+-- Paimon storage handler (metastore.storage.schema.reader.impl is
+-- SerDeStorageSchemaReader) -- the Paimon jar has to be on the metastore's own
+-- classpath, not just on the classpath of whoever runs this script.
+
+CREATE DATABASE IF NOT EXISTS hdfs_db;
+
+USE hdfs_db;
+
+DROP TABLE IF EXISTS external_test_table;
+
+CREATE EXTERNAL TABLE external_test_table
+ STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
+LOCATION 'hdfs:///user/hive/warehouse/hdfs_db.db/external_test_table';
diff --git a/docker/thirdparties/run-thirdparties-docker.sh
b/docker/thirdparties/run-thirdparties-docker.sh
index 63154a15473..14babf73026 100755
--- a/docker/thirdparties/run-thirdparties-docker.sh
+++ b/docker/thirdparties/run-thirdparties-docker.sh
@@ -1574,6 +1574,46 @@ validate_kerberos_container() {
fi
}
+# Aux jar for the Paimon fixture in the kerberized metastore: the Paimon
storage
+# handler the metastore loads to derive the fixture table's schema. Same source
+# and same skip-if-present caching the Hive3 stack uses in
prepare-hive-data.sh.
+download_kerberos_paimon_jars() {
+ local auxlib_dir="$1"
+ local
url_prefix="https://${s3BucketName}.${s3Endpoint}/regression/docker/hive3"
+ local jars=(
+ paimon-hive-connector-3.1-1.3-SNAPSHOT.jar
+ )
+ local jar
+
+ for jar in "${jars[@]}"; do
+ if [[ -f "${auxlib_dir}/${jar}" ]]; then
+ echo "Reuse cached kerberos aux jar ${jar}"
+ continue
+ fi
+ echo "Download kerberos aux jar ${jar}"
+ curl -sSfL -o "${auxlib_dir}/${jar}" "${url_prefix}/${jar}"
+ done
+}
+
+# enablePaimonHms lives in hive-3x_settings.env, which the pipeline already
+# patches with the real value. Read it from there instead of duplicating the
+# switch into kerberos*_settings.env, where the two copies would drift and one
+# stack would silently lose the fixture. Sourced in a subshell so that hive3's
+# FS_PORT / HMS_PORT cannot leak into the kerberos settings sourced a few lines
+# below.
+resolve_kerberos_paimon_env() {
+ local hive3_settings="${ROOT}/docker-compose/hive/hive-3x_settings.env"
+
+ enablePaimonHms="false"
+ if [[ -f "${hive3_settings}" ]]; then
+ read -r enablePaimonHms < <(
+ . "${hive3_settings}" >/dev/null 2>&1
+ printf '%s\n' "${enablePaimonHms:-false}"
+ )
+ fi
+ export enablePaimonHms
+}
+
start_kerberos() {
echo "RUN_KERBEROS"
local KERBEROS_DIR="${ROOT}/docker-compose/kerberos"
@@ -1588,8 +1628,13 @@ start_kerberos() {
export CONTAINER_UID=${CONTAINER_UID}
envsubst <"${KERBEROS_DIR}/kerberos.yaml.tpl"
>"${KERBEROS_DIR}/kerberos.yaml"
+ # auxlib must exist even when the fixture is off: kerberos1 mounts it
read-only.
mkdir -p "${KERBEROS_DIR}/conf/kerberos1" "${KERBEROS_DIR}/conf/kerberos2"
\
- "${KERBEROS_DIR}/two-kerberos-hives"
+ "${KERBEROS_DIR}/two-kerberos-hives" "${KERBEROS_DIR}/auxlib"
+ resolve_kerberos_paimon_env
+ if [[ "${enablePaimonHms}" == "true" && "${STOP}" -ne 1 ]]; then
+ download_kerberos_paimon_jars "${KERBEROS_DIR}/auxlib"
+ fi
for i in {1..2}; do
. "${KERBEROS_DIR}/kerberos${i}_settings.env"
envsubst <"${KERBEROS_DIR}/hadoop-hive.env.tpl"
>"${KERBEROS_DIR}/hadoop-hive-${i}.env"
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
index ccc693f4ad2..0e605f9790d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
@@ -345,9 +345,10 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
try {
return executeCatalogOperation(() ->
performCreateTable(createTableInfo));
} catch (Exception e) {
+ // Metadata writes can be wrapped by a self-suppression error;
expose the underlying failure.
throw new DdlException(
- "Failed to create table: " + createTableInfo.getTableName() +
", error message is:" + e.getMessage(),
- e);
+ "Failed to create table: " +
createTableInfo.getTableName() + ", error message is:"
+ + ExceptionUtils.getRootCauseMessage(e), e);
}
}
@@ -582,7 +583,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to create or replace branch: " + branchName +
" in table: " + icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
}
@@ -650,7 +651,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to create or replace tag: " + tagName + " in
table: " + icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
}
@@ -672,7 +673,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to drop tag: " + tagName + " in table: " +
icebergTable.name()
- + ", error message is: " + e.getMessage(),
e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
}
@@ -695,7 +696,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to drop branch: " + branchName + " in
table: " + icebergTable.name()
- + ", error message is: " + e.getMessage(),
e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
}
@@ -792,7 +793,8 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
try {
lease.getAuthenticator().execute(commit);
} catch (Exception e) {
- throw new UserException(failureMessage + ", error message is: " +
e.getMessage(), e);
+ // Preserve the storage failure hidden by metadata writer cleanup
exceptions.
+ throw new UserException(failureMessage + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy
index 9b06efc8a85..8d513f0c9f5 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+import java.util.concurrent.TimeUnit
+
suite("test_iceberg_deletion_vector", "p0,external,nonConcurrent") {
String enabled = context.config.otherConfigs.get("enableIcebergTest")
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
@@ -495,22 +497,34 @@ s3.path-style-access=true
30
)
executeCommand("${dockerCommand} restart ${trinoContainerName}", true, 60)
- String trinoRows = ""
- for (int i = 0; i < 12; i++) {
- Thread.sleep(5000)
- trinoRows = normalizeExternalRows(executeCommand(
- "${dockerCommand} exec ${trinoContainerName} trino
--output-format TSV " +
- "--catalog iceberg --schema format_v3 --execute " +
- "\"SELECT id, batch, data " +
- "FROM dv_delete_matrix_equality_and_dv ORDER BY id\"",
- false,
- 120
- ))
- if (!trinoRows.isEmpty()) {
+ String trinoCommand = "${dockerCommand} exec ${trinoContainerName} trino
--output-format TSV " +
+ "--catalog iceberg --schema format_v3 --execute "
+ // A running container does not imply a ready coordinator. Retry startup
separately so a
+ // failed data query cannot be mistaken for a successful query returning
no rows.
+ long trinoReadyDeadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(300)
+ def readiness = [exitCode: -1, stdout: "", stderr: "Readiness probe has
not run"]
+ while (System.nanoTime() < trinoReadyDeadline) {
+ readiness = executeCommandWithStatus(trinoCommand + '"SELECT 1"', 10,
false, false)
+ if (readiness.exitCode == 0 && readiness.stdout.trim() == "1") {
break
- }
+ }
+ Thread.sleep(1000)
}
- assertEquals(expectedRows, trinoRows)
+ assertTrue(readiness.exitCode == 0 && readiness.stdout.trim() == "1",
+ "Trino did not become ready within 300 seconds. Exit code:
${readiness.exitCode}\n" +
+
"stdout:\n${readiness.stdout}\nstderr:\n${readiness.stderr}")
+
+ def trinoResult = executeCommandWithStatus(
+ trinoCommand + '"SELECT id, batch, data ' +
+ 'FROM dv_delete_matrix_equality_and_dv ORDER BY id"',
+ 120
+ )
+ assertEquals(0, trinoResult.exitCode,
+ "Trino comparison query failed.
stdout:\n${trinoResult.stdout}\nstderr:\n${trinoResult.stderr}")
+ String trinoRows = normalizeExternalRows(trinoResult.stdout)
+ assertEquals(expectedRows, trinoRows,
+ "Trino comparison query returned unexpected rows. " +
+
"stdout:\n${trinoResult.stdout}\nstderr:\n${trinoResult.stderr}")
def profileCounterValues = { String profileText, String counterName ->
def values = []
diff --git
a/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
b/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
index 25ca5ebf6da..c2a8a8cdec5 100644
---
a/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
+++
b/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
@@ -74,9 +74,11 @@ suite("refactor_params_hdfs_all_test",
"p0,external,kerberos,external_docker,ext
assert insertResult.get(0).get(0) == 1
}
+ // Explicit client properties must match the DataNode SASL protection to
allow writes.
def hdfsNonXmlParams = "\"fs.defaultFS\" =
\"hdfs://${externalEnvIp}:8520\",\n" +
"\"dfs.namenode.kerberos.principal\" =
\"hdfs/[email protected]\",\n" +
"\"dfs.client.use.datanode.hostname\" = \"true\",\n" +
+ "\"dfs.data.transfer.protection\" = \"authentication\",\n" +
"\"hadoop.security.token.service.use_ip\" = \"false\",\n" +
"\"hadoop.kerberos.min.seconds.before.relogin\" = \"5\",\n" +
"\"hadoop.security.authentication\" = \"kerberos\",\n" +
diff --git
a/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy
b/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy
index 3e84bce35ba..216c869f131 100644
---
a/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy
+++
b/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+import java.util.concurrent.TimeUnit
+
suite("test_s3_tvf_number_range", "p0,external") {
String ak = getS3AK()
@@ -351,29 +353,25 @@ suite("test_s3_tvf_number_range", "p0,external") {
// Helper closure to check load result
def check_hdfs_load_result = {checklabel ->
- def max_try_milli_secs = 10000
- def success = false
- while(max_try_milli_secs) {
- def result = sql """ SHOW LOAD WHERE LABEL =
'${checklabel}' """
+ // Broker Load is asynchronous: cancellation or an unfinished
job must fail here,
+ // before the caller compares rows in the freshly truncated
table.
+ long deadline = System.nanoTime() +
TimeUnit.SECONDS.toNanos(120)
+ def result = []
+ while (System.nanoTime() < deadline) {
+ result = sql """ SHOW LOAD WHERE LABEL = '${checklabel}'
"""
if (result.size() > 0) {
def state = result[0][2] // State column
if (state == "FINISHED") {
sql "sync"
- success = true
- break
- } else if (state == "CANCELLED") {
- logger.error("HDFS load job ${checklabel} was
cancelled: ${result[0]}")
- break
+ return
}
+ assertTrue(state != "CANCELLED",
+ "HDFS load job ${checklabel} was cancelled.
Status: ${result}")
}
- sleep(1000) // wait 1 second every time
- max_try_milli_secs-=1000
- }
-
- if (!success) {
- def result = sql """ SHOW LOAD WHERE LABEL =
'${checklabel}' """
- logger.error("HDFS load job ${checklabel} failed or
timeout. Status: ${result}")
+ sleep(1000)
}
+ assertTrue(false,
+ "HDFS load job ${checklabel} did not finish within 120
seconds. Last status: ${result}")
}
// Test 12: HDFS Broker Load Single range {1..3} - should load
{1,2,3}
@@ -407,4 +405,4 @@ suite("test_s3_tvf_number_range", "p0,external") {
}
}
sql """ DROP TABLE IF EXISTS ${test_table} """
-}
\ No newline at end of file
+}
diff --git
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
index 01b98dc02b2..c1810ca24be 100644
---
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
+++
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
@@ -119,11 +119,12 @@ suite("test_paimon_hms_catalog",
"p2,external,paimon,new_catalog_property") {
'fs.defaultFS' = 'hdfs://${extHiveHmsHost}:${extHiveHdfsHost}'
"""
- // kerberos
+ // Both Kerberos property forms must match the DataNode SASL protection to
allow writes.
String hdfs_kerberos_properties = """
"fs.defaultFS" = "hdfs://${extHiveHmsHost}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hadoop.security.authentication" = "kerberos",
"hadoop.kerberos.principal"="hive/[email protected]",
@@ -134,6 +135,7 @@ suite("test_paimon_hms_catalog",
"p2,external,paimon,new_catalog_property") {
"fs.defaultFS" = "hdfs://${extHiveHmsHost}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hdfs.authentication.type" = "kerberos",
"hdfs.authentication.kerberos.principal"="hive/[email protected]",
diff --git
a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
index 62771e46ebc..4055a5aaa0c 100644
---
a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
+++
b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
@@ -421,11 +421,12 @@ suite("hive_on_hms_and_dlf",
"p2,external,new_catalog_property") {
String hdfs_properties = """
"hadoop.username" = "doris"
"""
- // kerberos
+ // Both Kerberos property forms must match the DataNode SASL protection to
allow writes.
String hdfs_kerberos_properties = """
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hadoop.security.authentication" = "kerberos",
@@ -436,6 +437,7 @@ suite("hive_on_hms_and_dlf",
"p2,external,new_catalog_property") {
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hdfs.authentication.type" = "kerberos",
"hdfs.authentication.kerberos.principal"="hive/[email protected]",
diff --git
a/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
b/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
index fa3cff25572..98ab7ad6039 100644
---
a/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
+++
b/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
@@ -474,11 +474,12 @@ suite("iceberg_on_hms_and_filesystem_and_dlf",
"p2,external,new_catalog_property
String hdfs_properties = """
"hadoop.username" = "doris"
"""
- // kerberos
+ // Both Kerberos property forms must match the DataNode SASL protection to
allow writes.
String hdfs_kerberos_properties = """
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hadoop.security.authentication" = "kerberos",
"io-impl" =
"org.apache.doris.datasource.iceberg.fileio.DelegateFileIO",
@@ -490,6 +491,7 @@ suite("iceberg_on_hms_and_filesystem_and_dlf",
"p2,external,new_catalog_property
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"io-impl" =
"org.apache.doris.datasource.iceberg.fileio.DelegateFileIO",
"hdfs.authentication.type" = "kerberos",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]