This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 6e79ef0c714 branch-4.0: [fix](docker)(case) Fix the remaining 4
external regression failures: kerberos HDFS SASL, Paimon fixture, file cache
flakiness (#66417)
6e79ef0c714 is described below
commit 6e79ef0c714871bb8bcf991d9423533801258b6a
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Aug 4 23:04:21 2026 +0800
branch-4.0: [fix](docker)(case) Fix the remaining 4 external regression
failures: kerberos HDFS SASL, Paimon fixture, file cache flakiness (#66417)
Related PR: #65939, #66254
Problem Summary:
Follow-up to #66254. The daily `Regression External` pipeline for
`branch-4.0`
(build 482/483) still fails 4 cases. None of them is a runtime problem:
the
cluster is healthy, 522 cases pass, and there is no crash, OOM or core
dump.
Two root causes remain, both of them the tail of #65939.
**1. Kerberos HDFS writes fail (3 p2 cases, red on every run since
#65939)**
`hive_on_hms_and_dlf` fails with
could only be written to 0 of the 1 minReplication nodes.
There are 1 datanode(s) running and 1 node(s) are excluded in this
operation.
and `iceberg_on_hms_and_filesystem_and_dlf` fails with
`IllegalArgumentException: Self-suppression not permitted`, which is
what
`TableMetadataParser.internalWrite` turns the same failure into.
The DataNode is alive — the client excluded it. `jni.log` shows why:
`java.net.SocketException: Connection reset` in
`DataStreamer.createBlockOutputStream`. Connection **reset**, not
refused: the
socket was established and the DataNode closed it. #65939 created
`kerberos/conf/hdfs-site.xml.tpl` — the previous environment carried its
server
configuration inside the image, so this file is new — and it sets
`dfs.data.transfer.protection=authentication`. A client that does not
set the
same property sends a plain `writeBlock` op, `SaslDataTransferServer`
does not
find the SASL magic number and closes the socket, the client excludes
the only
DataNode, and the next `addBlock` has no target left. Metadata
operations only
reach the NameNode and the HMS, so `CREATE DATABASE` / `CREATE TABLE`
still
succeed and only the write fails.
The same commit added the client property to the four
`external_table_p0/kerberos`
cases but not to the p2 cases that talk to the same kerberized HDFS.
Both suites
were green through build 468 and red from build 469, the first run after
#65939.
They were held at `GeneralSecurityException: Checksum failed` until
#66254 fixed
the keytab, which is why this second gap only surfaced now.
Fix: set `dfs.data.transfer.protection` in every property block that
points at
the kerberized HDFS. A scan of `:8520` / `:8620` across the whole suite
tree found
seven such blocks still missing it: two each in `hive_on_hms_and_dlf`,
`iceberg_on_hms_and_filesystem_and_dlf` and `test_paimon_hms_catalog`,
plus one
in `hdfs_all_test`, which writes to HDFS but currently returns early
because
`refactor_params_hdfs_kerberos_test` is unset — it would fail the moment
it is
enabled. The kerberos block of `test_information_schema_timezone` has
the same
gap but is disabled by an unrelated TODO, so it is left alone.
**2. `test_paimon_hms_catalog`: `Unknown database 'hdfs_db'`**
The lightweight Kerberos environment from #65939 starts an empty
metastore —
`schematool -dbType derby -initSchema` and nothing else. The environment
it
replaced provisioned the Paimon fixture on every start (`hadoop fs -put
/tmp/paimon_data/*`, `create_paimon_hive_table.hql`, and the
Paimon/jindo jars in
the image's hive auxlib). None of that survived;
`paimon_data/hdfs_db.db/` is
still in the tree but the compose file never mounted it.
This restores the fixture on `kerberos1` only — it owns metastore 9583,
the only
one the suite talks to, so `kerberos2` stays as light as it is today.
`hdfs_db` is
backed by `paimon_data`; `ali_db` carries no data and points at the same
OSS
warehouse the Hive3 stack registers under that name, so the two
metastores share
one copy. Registering it still needs the `oss://` scheme to resolve,
because
`HiveMetaStore.create_table_core` puts any explicit LOCATION through
`Warehouse.getDnsPath` → `Path.getFileSystem`; hence the jindo
filesystem settings
in `hive-site.xml` and the aliyun jars next to the Paimon one. The
fixture runs
before `DORIS_KERBEROS_READY`, so nothing races the suites, and a
failure aborts
the container instead of handing out an environment silently missing the
two
databases.
The golden output already contains the `hdfs_kerberos`,
`hdfs_new_kerberos` and
`oss_hms_kerberos` blocks, so no baseline has to be regenerated.
**3. `test_file_cache_statistics` (flaky, ~25% pass rate)**
#66254 relaxed the upper bound after build 474 reported
`normal_queue_curr_size`
2.44e8 above a `max_size` of 1.68e8. Build 482 fails the other way
round, with
`normal_queue_curr_size` exactly 0 while `hits_ratio` is 3.76 — the
queue is empty
rather than over its share.
`test_file_cache_features` runs immediately before it and drives every
backend into
`disk_resource_limit_mode` and `need_evict_cache_in_advance` on purpose,
by lowering
the enter thresholds to 2%. Both modes drain the shared cache, and while
`disk_resource_limit_mode` is on `BlockFileCache::try_reserve` admits
nothing: it
multiplies the requested size by 5 and only succeeds if it evicted that
much, which
an already drained cache never can. `setBeConfigTemporary` restores the
configs, but
the backend only recomputes the two flags in `run_background_monitor`,
once per
`file_cache_background_monitor_interval_ms` (5s in this pipeline). The
next suite
starts 37ms later, so its warm-up queries can land inside that window
and cache
nothing — and since the case then only polls a passive metric, it can
never recover.
Both polls time out 30s later, which matches the log exactly.
Fix, in two parts:
- `test_file_cache_features` waits until both metrics are back to 0
before it returns,
so it leaves the cluster in the state it found. The wait uses `max()`
rather than
`limit 1`, because one row is one arbitrary (backend, cache_path) pair
and any path
still in the mode reports a 1.
- `test_file_cache_statistics` re-runs the read on every poll of the
normal queue
metrics, the same self-healing shape its hit/read-count section already
uses.
The upper bound from #66254 is kept: build 474 shows that mode is real
too.
**4. `IcebergMetadataOps`: report the root cause of failed metadata
commits**
While diagnosing (1), the failure surfaced to the user as
`Failed to create table: ..., error message is:Self-suppression not
permitted`,
which says nothing about what went wrong.
`TableMetadataParser.internalWrite`
writes the metadata JSON inside try-with-resources, and
`DFSOutputStream` throws
the same exception instance from `write` and from `close` once its
pipeline has
failed, so the generated `addSuppressed` call becomes
`addSuppressed(this)` and
`Throwable` replaces the failure with `IllegalArgumentException`. Every
path that
commits Iceberg metadata goes through the same parser and can be masked
the same
way. `addPartitionField`, `dropPartitionField` and
`replacePartitionField` already
report `ExceptionUtils.getRootCauseMessage(e)`; this applies it to the
remaining
eleven commit paths so all fourteen agree. The surrounding message is a
literal
prefix that already names the operation and the object, so replacing
only the
trailing part loses no context, and no regression case asserts on this
text.
Signed-off-by: morningman <[email protected]>
---
.../docker-compose/kerberos/conf/hive-site.xml.tpl | 27 +++++++++
.../kerberos/entrypoint-hive-master.sh | 31 +++++++++++
.../docker-compose/kerberos/hadoop-hive.env.tpl | 8 +++
.../docker-compose/kerberos/kerberos.yaml.tpl | 6 ++
.../kerberos/sql/create_paimon_hive_table.hql | 32 +++++++++++
docker/thirdparties/run-thirdparties-docker.sh | 65 +++++++++++++++++++++-
.../datasource/iceberg/IcebergMetadataOps.java | 26 ++++-----
.../cache/test_file_cache_features.groovy | 34 +++++++++++
.../cache/test_file_cache_statistics.groovy | 26 ++++++++-
.../refactor_storage_param/hdfs_all_test.groovy | 1 +
.../paimon/test_paimon_hms_catalog.groovy | 2 +
.../hive_on_hms_and_dlf.groovy | 2 +
.../iceberg_on_hms_and_filesystem_and_dlf.groovy | 2 +
13 files changed, 245 insertions(+), 17 deletions(-)
diff --git a/docker/thirdparties/docker-compose/kerberos/conf/hive-site.xml.tpl
b/docker/thirdparties/docker-compose/kerberos/conf/hive-site.xml.tpl
index 2b4b41e20a7..9c46c326f24 100644
--- a/docker/thirdparties/docker-compose/kerberos/conf/hive-site.xml.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/conf/hive-site.xml.tpl
@@ -62,4 +62,31 @@ under the License.
<name>metastore.storage.schema.reader.impl</name>
<value>org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader</value>
</property>
+ <!--
+ Needed by the ali_db Paimon fixture: HiveMetaStore.create_table_core
resolves
+ any explicit LOCATION through Warehouse.getDnsPath ->
Path.getFileSystem, so
+ registering a table at oss:// requires the scheme to be resolvable at
DDL time.
+ Matches the Hive3 stack (HIVE_SITE_CONF_fs_oss_impl in
hadoop-hive-3x.env.tpl);
+ the jindo jars come from the auxlib mount. Inert when nothing touches
oss://.
+ -->
+ <property>
+ <name>fs.oss.impl</name>
+ <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
+ </property>
+ <property>
+ <name>fs.AbstractFileSystem.oss.impl</name>
+ <value>com.aliyun.jindodata.oss.JindoOSS</value>
+ </property>
+ <property>
+ <name>fs.oss.accessKeyId</name>
+ <value>${OSSAk}</value>
+ </property>
+ <property>
+ <name>fs.oss.accessKeySecret</name>
+ <value>${OSSSk}</value>
+ </property>
+ <property>
+ <name>fs.oss.endpoint</name>
+ <value>${OSSEndpoint}</value>
+ </property>
</configuration>
diff --git
a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
index 6735ae7ee40..db7f642c0cd 100644
--- a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
+++ b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
@@ -136,9 +136,40 @@ kdestroy
report_stage "initialize-hive-metastore"
schematool -dbType derby -initSchema
+# The Paimon tables declare no columns, so the metastore itself resolves their
+# schema through the storage handler, and registering ali_db at an oss://
location
+# makes it resolve that scheme too. Both classes must therefore be on the
+# metastore service's own classpath, not merely on the DDL client's.
+if [[ -d /opt/doris/auxlib ]]; then
+ export HIVE_AUX_JARS_PATH=/opt/doris/auxlib
+fi
start_service hive --service metastore -p "${HMS_PORT}"
wait_for_port "${HOST}" "${HMS_PORT}" "Hive Metastore"
+# Register the Paimon fixture consumed by test_paimon_hms_catalog. This runs
+# before the readiness marker below on purpose: run-thirdparties-docker.sh
+# releases the pipeline on DORIS_KERBEROS_READY, so anything published later
+# would race the suites. A failure here aborts the container (set -e) instead
of
+# handing out an environment that is silently missing hdfs_db / ali_db.
+#
+# Both kerberos containers run this entrypoint with the same rendered env
+# switch, but the fixture and its mounts (sql/, paimon_data/, auxlib/) belong
to
+# kerberos1 only - its metastore (9583) is the one the suite talks to. Gate on
+# the container role, not on the mounts, so a broken mount on kerberos1 still
+# fails loudly while kerberos2 skips deterministically.
+if [[ "${enablePaimonHms:-false}" == "true" && "${HOST:-}" == "hadoop-master"
]]; then
+ report_stage "load-paimon-hms"
+ export KRB5CCNAME=FILE:/tmp/hive-admin.ccache
+ kinit -kt /data/keytabs/hive.keytab "${HIVE_PRINCIPAL}"
+ hdfs dfs -mkdir -p /user/hive/warehouse
+ hdfs dfs -put -f /opt/doris/paimon_data/* /user/hive/warehouse/
+ paimon_hql="$(mktemp /tmp/create_paimon_hive_table.XXXXXX.hql)"
+ sed "s|__OSS_BUCKET__|${OSSBucket}|g"
/opt/doris/sql/create_paimon_hive_table.hql >"${paimon_hql}"
+ hive -f "${paimon_hql}"
+ rm -f "${paimon_hql}"
+ kdestroy
+fi
+
touch /tmp/SUCCESS
echo "Minimal Kerberos HDFS and Hive Metastore environment is ready"
echo "DORIS_KERBEROS_READY"
diff --git a/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
b/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
index 792d0c0491d..e1f5e7863f9 100644
--- a/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl
@@ -29,3 +29,11 @@ PRESTO_CLIENT_KEYTAB=${PRESTO_CLIENT_KEYTAB}
HADOOP_CONF_DIR=/opt/doris/conf
HIVE_CONF_DIR=/opt/doris/conf
KRB5_CONFIG=/etc/krb5.conf
+# Paimon-on-kerberized-HMS fixture. These four are read from
hive-3x_settings.env
+# by start_kerberos() rather than duplicated into kerberos*_settings.env, so
that
+# the pipeline keeps patching exactly one file and the two stacks cannot drift.
+enablePaimonHms=${enablePaimonHms}
+OSSBucket=${OSSBucket}
+OSSAk=${OSSAk}
+OSSSk=${OSSSk}
+OSSEndpoint=${OSSEndpoint}
diff --git a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
index d21496516af..8e903ed6d7f 100644
--- a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
@@ -33,11 +33,17 @@ services:
<<: *kerberos-service
container_name: doris-${CONTAINER_UID}-kerberos1
hostname: hadoop-master
+ # auxlib/sql/paimon_data are mounted on kerberos1 only:
test_paimon_hms_catalog
+ # is the sole consumer and it talks to this metastore (9583). Leaving
kerberos2
+ # untouched keeps the second container as light as it was.
volumes:
- ./conf/kerberos1:/opt/doris/conf:ro
- ./conf/kerberos1/krb5.conf:/etc/krb5.conf:ro
- ./data/kerberos1:/data
- ./two-kerberos-hives:/keytabs
+ - ./auxlib:/opt/doris/auxlib:ro
+ - ./sql:/opt/doris/sql:ro
+ - ./paimon_data:/opt/doris/paimon_data:ro
env_file:
- ./hadoop-hive-1.env
diff --git
a/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
b/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
new file mode 100644
index 00000000000..6bdbfc9f882
--- /dev/null
+++
b/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
@@ -0,0 +1,32 @@
+-- Paimon tables registered in the kerberized Hive Metastore, consumed by
+-- external_table_p2/paimon/test_paimon_hms_catalog.groovy.
+--
+-- hdfs_db is backed by ../paimon_data, which the entrypoint uploads into HDFS.
+-- ali_db carries no data of its own: it points at the same OSS warehouse the
+-- Hive3 stack registers as ali_db, so this is a second metastore entry over
one
+-- copy of the data. __OSS_BUCKET__ is substituted with ${OSSBucket}.
+--
+-- Neither table declares columns, so the metastore derives them through the
+-- Paimon storage handler (metastore.storage.schema.reader.impl is
+-- SerDeStorageSchemaReader) -- the Paimon jar has to be on the metastore's own
+-- classpath, not just on the classpath of whoever runs this script.
+
+CREATE DATABASE IF NOT EXISTS hdfs_db;
+
+USE hdfs_db;
+
+DROP TABLE IF EXISTS external_test_table;
+
+CREATE EXTERNAL TABLE external_test_table
+ STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
+LOCATION 'hdfs:///user/hive/warehouse/hdfs_db.db/external_test_table';
+
+CREATE DATABASE IF NOT EXISTS ali_db;
+
+USE ali_db;
+
+DROP TABLE IF EXISTS external_test_table;
+
+CREATE EXTERNAL TABLE external_test_table
+ STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
+LOCATION
'oss://__OSS_BUCKET__/regression/paimon_warehouse/ali_db.db/hive_test_table';
diff --git a/docker/thirdparties/run-thirdparties-docker.sh
b/docker/thirdparties/run-thirdparties-docker.sh
index 9ad5239e4ab..1c803e502d7 100755
--- a/docker/thirdparties/run-thirdparties-docker.sh
+++ b/docker/thirdparties/run-thirdparties-docker.sh
@@ -1575,6 +1575,64 @@ validate_kerberos_container() {
fi
}
+# Aux jars for the Paimon fixture in the kerberized metastore: the Paimon
storage
+# handler, plus the jindo OSS filesystem that ali_db's oss:// location has to
+# resolve at DDL time. Same source and same skip-if-present caching the Hive3
+# stack uses in prepare-hive-data.sh.
+download_kerberos_paimon_jars() {
+ local auxlib_dir="$1"
+ local
url_prefix="https://${s3BucketName}.${s3Endpoint}/regression/docker/hive3";
+ local jars=(
+ paimon-hive-connector-3.1-1.3-SNAPSHOT.jar
+ jdom-1.1.jar
+ aliyun-java-sdk-core-3.4.0.jar
+ aliyun-java-sdk-ecs-4.2.0.jar
+ aliyun-java-sdk-ram-3.0.0.jar
+ aliyun-java-sdk-sts-3.0.0.jar
+ jindo-core-6.3.4.jar
+ jindo-core-linux-el7-aarch64-6.3.4.jar
+ jindo-sdk-6.3.4.jar
+ )
+ local jar
+
+ for jar in "${jars[@]}"; do
+ if [[ -f "${auxlib_dir}/${jar}" ]]; then
+ echo "Reuse cached kerberos aux jar ${jar}"
+ continue
+ fi
+ echo "Download kerberos aux jar ${jar}"
+ curl -sSfL -o "${auxlib_dir}/${jar}" "${url_prefix}/${jar}"
+ done
+}
+
+# enablePaimonHms and the OSS credentials live in hive-3x_settings.env, which
the
+# pipeline already patches with the real values. Read them from there instead
of
+# duplicating the switch into kerberos*_settings.env, where the two copies
would
+# drift and one stack would silently lose the fixture. Sourced in a subshell so
+# that hive3's FS_PORT / HMS_PORT cannot leak into the kerberos settings
sourced
+# a few lines below.
+resolve_kerberos_paimon_env() {
+ local hive3_settings="${ROOT}/docker-compose/hive/hive-3x_settings.env"
+
+ enablePaimonHms="false"
+ OSSAk=""
+ OSSSk=""
+ OSSEndpoint=""
+ if [[ -f "${hive3_settings}" ]]; then
+ {
+ read -r enablePaimonHms
+ read -r OSSAk
+ read -r OSSSk
+ read -r OSSEndpoint
+ } < <(
+ . "${hive3_settings}" >/dev/null 2>&1
+ printf '%s\n%s\n%s\n%s\n' \
+ "${enablePaimonHms:-false}" "${OSSAk:-}" "${OSSSk:-}"
"${OSSEndpoint:-}"
+ )
+ fi
+ export enablePaimonHms OSSAk OSSSk OSSEndpoint
+}
+
start_kerberos() {
echo "RUN_KERBEROS"
local KERBEROS_DIR="${ROOT}/docker-compose/kerberos"
@@ -1589,8 +1647,13 @@ start_kerberos() {
export CONTAINER_UID=${CONTAINER_UID}
envsubst <"${KERBEROS_DIR}/kerberos.yaml.tpl"
>"${KERBEROS_DIR}/kerberos.yaml"
+ # auxlib must exist even when the fixture is off: kerberos1 mounts it
read-only.
mkdir -p "${KERBEROS_DIR}/conf/kerberos1" "${KERBEROS_DIR}/conf/kerberos2"
\
- "${KERBEROS_DIR}/two-kerberos-hives"
+ "${KERBEROS_DIR}/two-kerberos-hives" "${KERBEROS_DIR}/auxlib"
+ resolve_kerberos_paimon_env
+ if [[ "${enablePaimonHms}" == "true" && "${STOP}" -ne 1 ]]; then
+ download_kerberos_paimon_jars "${KERBEROS_DIR}/auxlib"
+ fi
for i in {1..2}; do
. "${KERBEROS_DIR}/kerberos${i}_settings.env"
envsubst <"${KERBEROS_DIR}/hadoop-hive.env.tpl"
>"${KERBEROS_DIR}/hadoop-hive-${i}.env"
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
index 3aa26f3572c..e2e659695b6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
@@ -310,9 +310,8 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
try {
return executionAuthenticator.execute(() ->
performCreateTable(createTableInfo));
} catch (Exception e) {
- throw new DdlException(
- "Failed to create table: " + createTableInfo.getTableName() +
", error message is:" + e.getMessage(),
- e);
+ throw new DdlException("Failed to create table: " +
createTableInfo.getTableName()
+ + ", error message is:" +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
@@ -514,7 +513,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to create or replace branch: " + branchName + " in
table: " + icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
@@ -578,7 +577,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to create or replace tag: " + tagName + " in
table: " + icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
@@ -598,7 +597,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to drop tag: " + tagName + " in table: " +
icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
}
@@ -619,7 +618,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
} catch (Exception e) {
throw new RuntimeException(
"Failed to drop branch: " + branchName + " in table: "
+ icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
}
}
@@ -666,7 +665,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
executionAuthenticator.execute(() -> updateSchema.commit());
} catch (Exception e) {
throw new UserException("Failed to add column: " +
column.getName() + " to table: "
- + icebergTable.name() + ", error message is: " +
e.getMessage(), e);
+ + icebergTable.name() + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
refreshTable(dorisTable, updateTime);
}
@@ -683,7 +682,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
executionAuthenticator.execute(() -> updateSchema.commit());
} catch (Exception e) {
throw new UserException("Failed to add columns to table: " +
icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
refreshTable(dorisTable, updateTime);
}
@@ -697,7 +696,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
executionAuthenticator.execute(() -> updateSchema.commit());
} catch (Exception e) {
throw new UserException("Failed to drop column: " + columnName + "
from table: "
- + icebergTable.name() + ", error message is: " +
e.getMessage(), e);
+ + icebergTable.name() + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
refreshTable(dorisTable, updateTime);
}
@@ -712,7 +711,8 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
executionAuthenticator.execute(() -> updateSchema.commit());
} catch (Exception e) {
throw new UserException("Failed to rename column: " + oldName + "
to " + newName
- + " in table: " + icebergTable.name() + ", error message
is: " + e.getMessage(), e);
+ + " in table: " + icebergTable.name()
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
refreshTable(dorisTable, updateTime);
}
@@ -758,7 +758,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
executionAuthenticator.execute(() -> updateSchema.commit());
} catch (Exception e) {
throw new UserException("Failed to modify column: " +
column.getName() + " in table: "
- + icebergTable.name() + ", error message is: " +
e.getMessage(), e);
+ + icebergTable.name() + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
refreshTable(dorisTable, updateTime);
}
@@ -967,7 +967,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
executionAuthenticator.execute(() -> updateSchema.commit());
} catch (Exception e) {
throw new UserException("Failed to reorder columns in table: " +
icebergTable.name()
- + ", error message is: " + e.getMessage(), e);
+ + ", error message is: " +
ExceptionUtils.getRootCauseMessage(e), e);
}
refreshTable(dorisTable, updateTime);
}
diff --git
a/regression-test/suites/external_table_p0/cache/test_file_cache_features.groovy
b/regression-test/suites/external_table_p0/cache/test_file_cache_features.groovy
index a3485f5ff6b..4b20a927c47 100644
---
a/regression-test/suites/external_table_p0/cache/test_file_cache_features.groovy
+++
b/regression-test/suites/external_table_p0/cache/test_file_cache_features.groovy
@@ -236,6 +236,40 @@ suite("test_file_cache_features",
"external_docker,hive,external_docker_hive,p0,
}
// ===== End File Cache Features Metrics Check =====
+ // Both setBeConfigTemporary blocks above deliberately push EVERY backend
into
+ // disk_resource_limit_mode and need_evict_cache_in_advance. Those two
modes drain the shared
+ // file cache, and while disk_resource_limit_mode is on
BlockFileCache::try_reserve refuses
+ // every new block: it multiplies the requested size by 5 and admits only
if it managed to evict
+ // that much, which an already-drained cache never can.
setBeConfigTemporary restores the
+ // configs, but the BE only recomputes the two flags in
run_background_monitor, i.e. once per
+ // file_cache_background_monitor_interval_ms. Returning right after the
restore therefore hands
+ // a drained, admit-nothing cache to whatever suite runs next -- that is
what made
+ // test_file_cache_statistics see normal_queue_curr_size == 0 and fail.
Wait for the cluster to
+ // come back to normal before finishing so this suite leaves the state it
found.
+ //
+ // max() (not "limit 1") because a single row is one arbitrary (backend,
cache_path) pair: the
+ // wait has to see every path leave the mode, and any path still in it is
a 1.
+ def cacheMetricMax = { String metricName ->
+ def r = sql """select max(cast(METRIC_VALUE as double)) from
information_schema.file_cache_statistics
+ where METRIC_NAME = '${metricName}';"""
+ if (r.size() == 0 || r[0][0] == null) {
+ return null
+ }
+ return Double.valueOf(r[0][0].toString())
+ }
+
+ Awaitility.await()
+ .atMost((totalWaitTime * 4 + 10) as long, TimeUnit.SECONDS)
+ .pollInterval(1, TimeUnit.SECONDS)
+ .until {
+ def limitMode = cacheMetricMax('disk_resource_limit_mode')
+ def evictInAdvance =
cacheMetricMax('need_evict_cache_in_advance')
+ logger.info("waiting for file cache modes to reset -
disk_resource_limit_mode: " +
+ "${limitMode}, need_evict_cache_in_advance:
${evictInAdvance}")
+ return limitMode != null && evictInAdvance != null &&
+ limitMode == 0.0 && evictInAdvance == 0.0
+ }
+
sql """set global enable_file_cache=false"""
return true
}
\ No newline at end of file
diff --git
a/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
b/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
index 34f4948c510..cb3f11e16c5 100644
---
a/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
+++
b/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
@@ -97,12 +97,17 @@ suite("test_file_cache_statistics",
"external_docker,hive,external_docker_hive,p
// interval after the query races the refresh. Awaitility polling waits
only as long as needed
// and avoids reading too soon. On timeout we swallow the exception so the
caller's own
// metric-specific assert below can surface the precise failure message.
- def pollMetric = { String metricName, Closure predicate, long
timeoutSeconds ->
+ // `warmUp`, when given, re-runs the caching query on every poll: a metric
that only a fresh
+ // read can move must not be polled passively (see the normal-queue
section below).
+ def pollMetric = { String metricName, Closure predicate, long
timeoutSeconds, Closure warmUp = null ->
try {
Awaitility.await()
.atMost(timeoutSeconds, TimeUnit.SECONDS)
.pollInterval(1, TimeUnit.SECONDS)
.until {
+ if (warmUp != null) {
+ warmUp()
+ }
def v = cacheMetricSum(metricName)
return v != null && predicate(v)
}
@@ -188,8 +193,23 @@ suite("test_file_cache_statistics",
"external_docker,hive,external_docker_hive,p
// The hard bound the BE actually enforces is the per-cache _capacity, and
by construction in
// get_file_cache_settings() capacity == normal + index + ttl + disposable
max sizes (the
// normal/query queue is defined as the remainder). Sum across paths and
assert against that.
- pollMetric('normal_queue_curr_size', { it > 0 }, metricPollTimeoutSeconds)
- pollMetric('normal_queue_curr_elements', { it > 0 },
metricPollTimeoutSeconds)
+ //
+ // The warm-up queries above are NOT enough on their own to guarantee the
queue is populated.
+ // This suite shares one file cache with every other case on the cluster,
and the case that runs
+ // right before it (test_file_cache_features) deliberately forces the
backends into
+ // disk_resource_limit_mode / need_evict_cache_in_advance. Those modes
drain the cache, and while
+ // disk_resource_limit_mode is on try_reserve admits nothing (it must
evict 5x the block size
+ // first and a drained cache has nothing to evict). The flags only clear
on the next background
+ // monitor tick, so a warm-up that lands inside that window caches zero
blocks -- and polling a
+ // passive metric afterwards can never recover, which is exactly how this
case failed with
+ // normal_queue_curr_size == 0. Re-run the read on every poll so the first
tick after the modes
+ // clear repopulates the queue, the same self-healing shape the
hit/read-count section uses.
+ def warmUpFileCache = {
+ sql """select * from
${catalog_name}.${ex_db_name}.parquet_partition_table
+ where l_orderkey=1 and l_partkey=1534 limit 1;"""
+ }
+ pollMetric('normal_queue_curr_size', { it > 0 }, metricPollTimeoutSeconds,
warmUpFileCache)
+ pollMetric('normal_queue_curr_elements', { it > 0 },
metricPollTimeoutSeconds, warmUpFileCache)
def normalQueueCurrSizeSum = cacheMetricSum('normal_queue_curr_size')
logger.info("normal_queue_curr_size sum: " + normalQueueCurrSizeSum)
diff --git
a/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
b/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
index 25ca5ebf6da..dc5eb7ea662 100644
---
a/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
+++
b/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy
@@ -77,6 +77,7 @@ suite("refactor_params_hdfs_all_test",
"p0,external,kerberos,external_docker,ext
def hdfsNonXmlParams = "\"fs.defaultFS\" =
\"hdfs://${externalEnvIp}:8520\",\n" +
"\"dfs.namenode.kerberos.principal\" =
\"hdfs/[email protected]\",\n" +
"\"dfs.client.use.datanode.hostname\" = \"true\",\n" +
+ "\"dfs.data.transfer.protection\" = \"authentication\",\n" +
"\"hadoop.security.token.service.use_ip\" = \"false\",\n" +
"\"hadoop.kerberos.min.seconds.before.relogin\" = \"5\",\n" +
"\"hadoop.security.authentication\" = \"kerberos\",\n" +
diff --git
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
index 01b98dc02b2..b1b8c2d155e 100644
---
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
+++
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
@@ -124,6 +124,7 @@ suite("test_paimon_hms_catalog",
"p2,external,paimon,new_catalog_property") {
"fs.defaultFS" = "hdfs://${extHiveHmsHost}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hadoop.security.authentication" = "kerberos",
"hadoop.kerberos.principal"="hive/[email protected]",
@@ -134,6 +135,7 @@ suite("test_paimon_hms_catalog",
"p2,external,paimon,new_catalog_property") {
"fs.defaultFS" = "hdfs://${extHiveHmsHost}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hdfs.authentication.type" = "kerberos",
"hdfs.authentication.kerberos.principal"="hive/[email protected]",
diff --git
a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
index 74844ee1145..e074e27c352 100644
---
a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
+++
b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
@@ -420,6 +420,7 @@ suite("hive_on_hms_and_dlf",
"p2,external,new_catalog_property") {
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hadoop.security.authentication" = "kerberos",
@@ -430,6 +431,7 @@ suite("hive_on_hms_and_dlf",
"p2,external,new_catalog_property") {
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hdfs.authentication.type" = "kerberos",
"hdfs.authentication.kerberos.principal"="hive/[email protected]",
diff --git
a/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
b/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
index 32ea498a9db..65dd50e580c 100644
---
a/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
+++
b/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy
@@ -485,6 +485,7 @@ suite("iceberg_on_hms_and_filesystem_and_dlf",
"p2,external,new_catalog_property
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"hadoop.security.authentication" = "kerberos",
"io-impl" =
"org.apache.doris.datasource.iceberg.fileio.DelegateFileIO",
@@ -496,6 +497,7 @@ suite("iceberg_on_hms_and_filesystem_and_dlf",
"p2,external,new_catalog_property
"fs.defaultFS" = "hdfs://${externalEnvIp}:8520",
"dfs.namenode.kerberos.principal" =
"hdfs/[email protected]",
"dfs.client.use.datanode.hostname" = "true",
+ "dfs.data.transfer.protection" = "authentication",
"hadoop.security.token.service.use_ip" = "false",
"io-impl" =
"org.apache.doris.datasource.iceberg.fileio.DelegateFileIO",
"hdfs.authentication.type" = "kerberos",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
