This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard bae5aae719c [chore](ci) waive the unpatchable jackson-mapper-asl
advisories in dependency review
discard 3236254ccea [chore](license) satisfy the ASF header check on this
branch
discard f8ce4c159c8 [fix](ci) hand the license-check file list over as a file,
not an env var
omit f71245098db [fix](build) keep CONTAINER_UID assignment literal so
pipeline sed still matches
omit a0c10f0672b [chore](handoff) record the 2026-07-27c rebase onto
e7b7f1d1359 (upstream #66004 storage facade)
omit e29884df07f [fix](catalog) reconcile the connector SPI with upstream
#66004's fe-filesystem storage facade
omit e1a8963dadb [fix](build) Support Iceberg Docker stacks on Apple Silicon
omit fce5af4e041 [refactor](catalog) connector public interface cleanup: no
common-module edit to add a connector, dead-surface deletion, 11 correctness
fixes (#66135)
omit 77caeba35a9 [fix](catalog) port #65955's paimon table-option
passthrough + jni.* rename to the connector
omit 079dbace4a5 [chore](handoff) record the 2026-07-27 rebase onto
5b3ac63f8b4
omit 5f3fdbc63d9 [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
omit 638109c0946 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
omit e473ac77e0e [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
omit 7f778ebfded [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
omit f1afb5fe7bc [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
omit d88fbbda243 [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
omit 18df253ae94 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
omit a78045b3ea3 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
omit ff3251d135d [fix](be) fail the query instead of aborting BE on a
missing projected column
omit 8da6076f178 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
omit c2155b300fe [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
omit be71c254653 [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
omit f9cbd1b8ebb [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
omit bdfff9c2a57 [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
omit fe9fa1a3298 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
omit fffc74f1020 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
omit 51d694fc4fa [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
omit 8559a82e0f2 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
omit 009c59ecafe [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit 9ad4d4935eb [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit cadf02e5706 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit 594fa9d1f0c [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit 942829ddd9d [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit d1df9809fe2 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit 2f092cafd89 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 2582ace313f [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit 087097b5b60 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit a5ada44b263 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 63a82f6ff51 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit be22a7bd293 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit f889c080151 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 4839f614fc1 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 675a56cfc27 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 2844bc17962 [refactor](catalog) Catalog spi 11 hive (#65473)
omit d90b695dab1 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit 0e9ccf93b2b [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 83467da912a [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 0ae42404f18 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 21dd757efff [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 63cbee44c1f [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 4968cf14eaa [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit 8902d92169f [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit ccc2f936682 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit cf5c690b801 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 89a7397ccce [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit b79a3cecaf4 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 32a1c3c030c [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 78e2853e660 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit f68004b6909 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 3624a117877 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit e16c8341583 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 8700870bfa6 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 93d5754117a [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 2f2a47b787c [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 142cc821883 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit c043691f607 update P5 handoff and fix compile issue
omit 457257ef2ae [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit 7232e1c6ddb [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit 8a0554b5011 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 6b624aef7ff [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 9e5c596059f [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit cb5f9d70c41 [doc](connector) add project tracking system for catalog
SPI migration
add 6467a0ff168 [improvement](ci) Update Codex auth object pool (#66118)
add 1284cd5b1f3 [improvement](parquet) Optimize V2 sparse runtime-filter
scans (#66056)
add d74ebad82e2 [feature](paimon) Support query-level dynamic options
(#65984)
add 749290cb041 [opt](repeat) deduplicate grouping scalar functions from
projetion inrepeat (#65880)
add e3468bef0e2 [doc](connector) add project tracking system for catalog
SPI migration
add ac55d1fe196 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 1ff9ca84032 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 5b24843405a [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 9e2f31038e3 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 93464e01015 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 9bbcf522e29 update P5 handoff and fix compile issue
add 2637c3dc861 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 664978fbd54 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add aada073d8d7 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 3560db37ef8 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add f8fea631fdd [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add ee1318003d5 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add c78d32dadae [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add e90d4c4704e [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add 03f37f70a00 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 195fa2ed030 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 112398fb9e2 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add b37feb1e0ff [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 560c8a30d32 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 6a461b1da61 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 87c0ae3c1f2 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add 8c01fd24fae [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add c6ce6828b8e [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 2afd9216baf [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 18ec6e36bf1 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 1e7393dc5cb [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 302f04a00c4 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 282f9f6a839 [refactor](catalog) Catalog spi 11 hive (#65473)
add 88f65e6bd8b [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add c14c76ca2be [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add bccb5081f5f [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add ea3bb7eb538 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 833b39220f2 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add fd76f6f7afb [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add dff60958dbe [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 6d88664322c [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add 2a999f51444 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add c64ce20af2f [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 835e17d71b5 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add abe843bf994 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 7d4e235e8ff [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add dcaf73cd359 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add 60839879c9a [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add 1bbaea80fc8 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add 2035f7e4212 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add ab46b26d5c2 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add 7612abf0595 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add 3483e97e3a9 [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add b1a6847694f [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
add cbf995259fb [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
add 7f56f06b67e [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
add 7a2800e4f28 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
add c23437b6156 [fix](be) fail the query instead of aborting BE on a
missing projected column
add 9628129cf3f [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
add d7196fa51eb [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
add e3d404ec737 [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
add 8ea9e1440ba [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
add 30a6cd62f52 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
add 4187f9b396c [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
add 2651f7f1374 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
add d11a69b8564 [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
add 2116f98c146 [chore](handoff) record the 2026-07-27 rebase onto
5b3ac63f8b4
add 5335a5dfbfe [fix](catalog) port #65955's paimon table-option
passthrough + jni.* rename to the connector
add 9b57fe09c21 [refactor](catalog) connector public interface cleanup: no
common-module edit to add a connector, dead-surface deletion, 11 correctness
fixes (#66135)
add e391df4bc1b [fix](build) Support Iceberg Docker stacks on Apple Silicon
add 40624de7374 [fix](catalog) reconcile the connector SPI with upstream
#66004's fe-filesystem storage facade
add 0b4d5a6d076 [chore](handoff) record the 2026-07-27c rebase onto
e7b7f1d1359 (upstream #66004 storage facade)
add e7c5693421a [fix](build) keep CONTAINER_UID assignment literal so
pipeline sed still matches
add 6f5b80ce39e [fix](ci) hand the license-check file list over as a file,
not an env var
add b94234c03c4 [chore](license) satisfy the ASF header check on this
branch
add 5902f7a1c8c [chore](ci) waive the unpatchable jackson-mapper-asl
advisories in dependency review
add d55a4fb458c [refactor](catalog) port #65984 paimon query-level
@options into fe-connector-paimon
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (bae5aae719c)
\
N -- N -- N refs/heads/branch-catalog-spi (d55a4fb458c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/code-review-runner.yml | 3 +-
.../core/data_type_serde/parquet_decode_source.cpp | 23 +-
be/src/exprs/bloom_filter_func.h | 60 +++
be/src/exprs/hybrid_set.h | 19 +
be/src/exprs/runtime_filter_expr.cpp | 31 ++
be/src/exprs/runtime_filter_expr.h | 7 +
be/src/exprs/vbloom_predicate.cpp | 69 +++-
be/src/exprs/vbloom_predicate.h | 8 +
be/src/exprs/vdirect_in_predicate.h | 79 ++++
be/src/format_v2/parquet/parquet_scan.cpp | 61 ++-
be/src/format_v2/parquet/reader/column_reader.cpp | 2 +-
be/src/format_v2/parquet/reader/column_reader.h | 3 +-
.../parquet/reader/native/column_chunk_reader.cpp | 41 +-
.../parquet/reader/native/column_chunk_reader.h | 4 +-
.../parquet/reader/native/column_reader.cpp | 19 +-
.../parquet/reader/native/column_reader.h | 16 +-
be/src/format_v2/parquet/reader/native/decoder.h | 61 +++
.../parquet/reader/native_column_reader.cpp | 52 +--
.../parquet/reader/native_column_reader.h | 7 +-
be/src/format_v2/parquet/selection_vector.h | 22 +-
be/test/exprs/bloom_filter_func_test.cpp | 129 ++++++
be/test/exprs/expr_zonemap_filter_test.cpp | 95 +++++
be/test/format_v2/parquet/native_decoder_test.cpp | 169 ++++++++
.../parquet/parquet_reader_control_test.cpp | 7 +
be/test/format_v2/parquet/parquet_reader_test.cpp | 28 ++
be/test/format_v2/parquet/parquet_scan_test.cpp | 218 +++++++++++
.../doris/connector/api/ConnectorCapability.java | 19 +-
.../api/mvcc/ConnectorTimeTravelSpec.java | 35 +-
.../api/scan/ConnectorScanPlanProvider.java | 39 +-
.../doris/connector/paimon/PaimonConnector.java | 10 +-
.../connector/paimon/PaimonConnectorMetadata.java | 66 +++-
.../paimon/PaimonIncrementalScanParams.java | 7 +-
.../doris/connector/paimon/PaimonScanParams.java | 434 +++++++++++++++++++++
.../connector/paimon/PaimonScanPlanProvider.java | 140 ++++++-
.../connector/paimon/PaimonScanParamsTest.java | 394 +++++++++++++++++++
.../PaimonScanPlanProviderCapabilityTest.java | 34 ++
.../org/apache/doris/analysis/TableScanParams.java | 46 ++-
.../mvcc/PluginDrivenMvccExternalTable.java | 6 +
.../plugin/PluginDrivenExternalTable.java | 14 +
.../datasource/scan/PluginDrivenScanNode.java | 78 +++-
.../doris/nereids/ExternalTablePreloadInfo.java | 3 +-
.../org/apache/doris/nereids/StatementContext.java | 6 +
.../doris/nereids/analyzer/UnboundRelation.java | 8 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 6 +
.../doris/nereids/rules/analysis/BindRelation.java | 34 ++
.../nereids/rules/analysis/NormalizeRepeat.java | 34 +-
.../exploration/mv/MaterializedViewUtils.java | 9 +-
.../doris/nereids/trees/plans/algebra/Repeat.java | 2 +-
.../trees/plans/commands/ExecuteCommand.java | 25 ++
.../apache/doris/nereids/util/ExpressionUtils.java | 4 +-
.../PluginDrivenScanNodeSysTableGuardTest.java | 59 ++-
.../apache/doris/nereids/StatementContextTest.java | 46 +++
.../doris/nereids/parser/NereidsParserTest.java | 103 +++++
.../nereids/rules/analysis/BindRelationTest.java | 27 ++
.../rules/analysis/NormalizeRepeatTest.java | 30 ++
.../exploration/mv/MaterializedViewUtilsTest.java | 19 +
.../trees/plans/commands/ExecuteCommandTest.java | 99 +++++
plan-doc/HANDOFF.md | 135 ++++++-
.../paimon/paimon_data_system_table.out | 10 +-
.../external_table_p0/paimon/paimon_incr_read.out | 40 ++
.../paimon/paimon_system_table.out | 9 +
.../paimon/paimon_time_travel.out | 6 +
.../test_paimon_schema_time_travel_matrix.out | 27 ++
.../paimon/paimon_data_system_table.groovy | 21 +
.../paimon/paimon_incr_read.groovy | 57 ++-
.../paimon/paimon_system_table.groovy | 71 +++-
.../paimon/paimon_time_travel.groovy | 8 +
.../test_paimon_schema_time_travel_matrix.groovy | 150 +++++++
68 files changed, 3449 insertions(+), 154 deletions(-)
create mode 100644
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java
create mode 100644
fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanParamsTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommandTest.java
create mode 100644
regression-test/data/external_table_p0/paimon/test_paimon_schema_time_travel_matrix.out
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
