This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit ceb33843d4b [chore](handoff) record the 2026-07-27 rebase onto
5b3ac63f8b4
omit 7837c5698df [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
omit ffe6c96cab7 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
omit 5243176d83f [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
omit 8bd8d058b03 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
omit a122189799c [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
omit eac42bdacd3 [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
omit ea47ac1664a [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
omit 1bc292b02a5 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
omit 4193cf645fb [fix](be) fail the query instead of aborting BE on a
missing projected column
omit 17efb90f377 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
omit 5c081c7d3c2 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
omit 0a1696314a5 [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
omit ef874a130e6 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
omit eaa4680b35c [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
omit f15bda5c8d9 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
omit 553d74b8666 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
omit 6e850ad8cd7 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
omit bc3637d5f58 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
omit e65d0f6feca [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit 97d7e179e4e [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit 815ca38973e [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit ece8eb53f52 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit d7c4c577638 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 8a55a725022 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit ce8805db052 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 09b31a13c29 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit ee64bada109 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit 385ad651e56 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 4ea88483636 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 515cab2e104 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit f9c6fa86cae [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 931642e4ed6 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit ba1b51933d6 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit a901b285253 [refactor](catalog) Catalog spi 11 hive (#65473)
omit 909a8438b38 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit 49e4a6e67d0 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 83c8eb0a397 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit afbc33471d8 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 8ddd852117c [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 128c1d4a6a4 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 78945041686 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit f62031a4510 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit 7cf5d8c2701 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 20b186263eb [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 0e33af61629 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 25a92be3f8a [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 38bc1a9f580 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 2195048a827 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit fa11340244b [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit a26d2e6d325 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 863a7e3016c [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit a953ddd71d5 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit a92320fa4e4 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 2498943fe2f [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit bddddbd4d50 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 116ea6d7f46 update P5 handoff and fix compile issue
omit 3df0ec8d2ab [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit bba524d6ef0 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit ff80ee37fcb [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 9d533cf9d72 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit e7a8409b2d5 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit 29e98c098af [doc](connector) add project tracking system for catalog
SPI migration
add 973f93f51f8 [fix](build) Let ScopedLSANDisabler compile without a
sanitizer (#66049)
add faa6aec3896 [test](feut) fix
`testShouldWarmUpPerformanceRepeatedCycles200k` (#66081)
add bab2a3a3176 [fix](mv) Fix scan materialization pattern context check
(#65414)
add fb19e0da0c0 [fix](be) Finish pending scanners after shared limit
(#65814)
add 74227a80e46 [feature](paimon) Support Paimon table option passthrough
(#65955)
add 042e613b134 [test](regression) Complete Paimon read/write P0 coverage
(#66065)
add 73da94b6dc7 [doc](connector) add project tracking system for catalog
SPI migration
add 08c05bb371e [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add d904e005f08 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 1bb6dea1ec8 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add d5890d009c8 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 46153e26be6 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 423b488e6e3 update P5 handoff and fix compile issue
add 2d65237c5f8 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 23ab32e2e1d [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 5fc8cb65e9f [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add c276f8ed737 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add f7caa067ead [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add dcb4744a0ef [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add ea05668f6a8 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add a6b27dfad8e [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add affc033a871 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add e5e07ae7a80 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 25ba722b60c [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 117342a4805 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 04020f0d516 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 7f42a457bb3 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add b54307ef471 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add a51040dcc14 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 7f45de86b2b [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 6cf66158cb6 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 2169ac305d1 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add c3deb797c51 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 33da408760c [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 246eb5de2c3 [refactor](catalog) Catalog spi 11 hive (#65473)
add 4e39a39f2bc [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 0fc7007b35c [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 9f20e42243c [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add f4693453ce1 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add be89f06bab0 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add e4466b50526 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add 3229456925d [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 1616ecc9d92 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add 8ece644991f [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add a83813add05 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 3b56b794c55 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add 3a033b5c8dc [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 71e7d44a446 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add be13de2eb98 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add f2a14d4a7f9 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add bf53596fbad [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add 95af37d0e5d [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add 22ca0f168ba [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add 16d93ea35d8 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add 1edd78e318e [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add 755051b4cba [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
add bc82352fe9a [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
add ec7380334e3 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
add 50e77e247e7 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
add b94d9ea3e81 [fix](be) fail the query instead of aborting BE on a
missing projected column
add bb83f846f85 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
add cae06ef3942 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
add 7e547346a4f [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
add 1978fe6ec36 [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
add 957b670c9f9 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
add dbbab3652e6 [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
add 4ed4714d3a1 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
add d09b1b51503 [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
add 8aec7238ddf [chore](handoff) record the 2026-07-27 rebase onto
5b3ac63f8b4
add e48a96ae488 [fix](catalog) port #65955's paimon table-option
passthrough + jni.* rename to the connector
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ceb33843d4b)
\
N -- N -- N refs/heads/branch-catalog-spi (e48a96ae488)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/exec/scan/scanner_context.cpp | 18 +-
be/src/exec/scan/scanner_context.h | 1 +
be/src/format/table/paimon_jni_reader.cpp | 4 +-
be/src/format_v2/jni/paimon_jni_reader.cpp | 4 +-
be/src/util/debug/leak_annotations.h | 16 ++
be/test/format_v2/table/paimon_reader_test.cpp | 18 +-
.../org/apache/doris/paimon/PaimonJniScanner.java | 19 +-
.../connector/paimon/PaimonCatalogFactory.java | 10 +-
.../doris/connector/paimon/PaimonCatalogOps.java | 21 +-
.../doris/connector/paimon/PaimonConnector.java | 12 +-
.../connector/paimon/PaimonConnectorProvider.java | 4 +
.../connector/paimon/PaimonScanPlanProvider.java | 14 +-
.../doris/connector/paimon/PaimonTableOptions.java | 139 ++++++++++++
.../paimon/PaimonScanPlanProviderTest.java | 39 ++--
.../connector/paimon/PaimonTableOptionsTest.java | 240 +++++++++++++++++++++
...terializedViewAggregateOnNoneAggregateRule.java | 4 +-
.../cloud/CacheHotspotManagerTableFilterTest.java | 4 +-
plan-doc/HANDOFF.md | 87 +++++++-
.../test_shared_scan_limit_pending_tasks.out | 7 +
.../paimon/test_paimon_merge_engine_matrix.out | 105 +++++++++
.../paimon/test_paimon_write_boundary.out | 15 ++
.../test_shared_scan_limit_pending_tasks.groovy | 175 +++++++++++++++
.../PAIMON_ICEBERG_READ_WRITE_P0_COVERAGE.md | 102 +++++++++
.../test_paimon_ctas_atomicity_negative.groovy | 77 +++++++
.../paimon/test_paimon_merge_engine_matrix.groovy | 223 +++++++++++++++++++
.../paimon/test_paimon_write_boundary.groovy | 115 ++++++++++
26 files changed, 1403 insertions(+), 70 deletions(-)
create mode 100644
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTableOptions.java
create mode 100644
fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableOptionsTest.java
create mode 100644
regression-test/data/correctness_p0/test_shared_scan_limit_pending_tasks.out
create mode 100644
regression-test/data/external_table_p0/paimon/test_paimon_merge_engine_matrix.out
create mode 100644
regression-test/data/external_table_p0/paimon/test_paimon_write_boundary.out
create mode 100644
regression-test/suites/correctness_p0/test_shared_scan_limit_pending_tasks.groovy
create mode 100644
regression-test/suites/external_table_p0/PAIMON_ICEBERG_READ_WRITE_P0_COVERAGE.md
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_ctas_atomicity_negative.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_merge_engine_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_write_boundary.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
