This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard c9540902dd1 [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
discard 565d31eb0ce [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
discard 3b7efafceeb [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
discard 609adeb6261 [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
discard 16004bace48 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
discard adb77ec5e74 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
discard 5429366e9f7 [fix](be) fail the query instead of aborting BE on a
missing projected column
discard aa2098ad1fd [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
discard cbdab97ba74 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
discard 595d93b2f17 [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
discard 1b330930cc2 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
discard 7309efcf48c [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
discard 1e4ade63c71 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
discard 924793f9ee9 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
discard c28691a9d0e [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
discard 3f010fea648 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
discard f9c298b102b [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
discard f82809a5b31 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
discard 4867144b30d [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
discard 0944b636339 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
discard 7ac3ca7e1e4 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
discard bb27ae9f951 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
discard 907894ea410 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
discard d42868bd9a7 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
discard 21f6821241f [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
discard 06ab813ab39 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
discard a39b4f89c1f [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
discard 5c2ff1d996d [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
discard 91f874f2a65 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
discard e7812791026 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
discard 5ecafcbeb0f [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
discard a7b45c6e2fc [refactor](catalog) Catalog spi 11 hive (#65473)
discard caca4c90e46 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
discard 772991ebc13 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
discard e801a8ff0ac [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
discard b68a583530e [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
discard b68b7ca51dd [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
discard db68d14b210 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
discard 8206bc5b4b7 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
discard 7b4cccce6cd [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
discard aabba45fc15 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
discard 11e040a0fac [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
discard f04b54371eb [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
discard 4dfad2a30c7 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
discard 3b9092bcc42 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
discard 3fe9fe8aa9f [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
discard 9d48b21cc8b [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
discard 0b42d96e1a8 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
discard bd433112564 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
discard 57e1ce46b7c [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
discard ca853d78d24 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
discard 1ee79930faf [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
discard ed56bff25f1 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
discard 7c4ae8d9d38 update P5 handoff and fix compile issue
discard 76d15eb8a4e [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
discard da1dc51652d [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
discard e3a23fed8ef [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
discard 61e7f4e66cb [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
discard 53becdd76b8 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
discard e068ced0f80 [doc](connector) add project tracking system for catalog
SPI migration
add 97f7f957a81 [improvement](parquet) Optimize typed dictionary range
filtering (#66036)
add 962bd5b28c7 [Feature](file scanner) Support JNI and WAL in V2 (#66008)
add a6cb29ab2ba [doc](connector) add project tracking system for catalog
SPI migration
add 2cf57f79638 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 8f230a093ae [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 5c658ea072e [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add de930d575ce [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 76e279bbcc7 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 090596c4844 update P5 handoff and fix compile issue
add b156b0f304f [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add ff3088dfc7a [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add e5de5f00873 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 76ecd62be3a [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add a106157a2ab [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add ed05170c7e7 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add ff7dd1a945d [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 9aaf6686e7b [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add d64ad6d85f7 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 4fd80cd6e67 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 335012d1e16 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add c66fcc06749 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add e55d3a68e77 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add e97aad2f57f [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 25c9199706c [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add 0e75080be3e [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 7516fe68a43 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 96fd92c1acd [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add d8b105aee3d [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 7b09e195ee7 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 48c3e6966a5 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 53766ceb32b [refactor](catalog) Catalog spi 11 hive (#65473)
add 6117bcb22bb [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 02284e682c7 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 7932e374909 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 16c3278e93f [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add ed2206c2c69 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add 3fd609552c0 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add e027acdb600 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 20de518986d [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add da9ce090f3b [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add 0ef0fbeab7d [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 2ee432deb94 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add 13638182a41 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 8352e6b02b6 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add 231f8fedec3 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add 2b8f84b5c29 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add 989a10bc96d [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add 336556289df [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add 0a2568a4aa4 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add 6a6d2f4a06a [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add 645a18e86b0 [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add 4dc44f71292 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
add f66d3be2f7b [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
add ead8a9924cd [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
add 19a90a7dba1 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
add 4b918b025db [fix](be) fail the query instead of aborting BE on a
missing projected column
add f2d579fbda4 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
add 4417b12c338 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
add 5d49b866d6a [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
add 710ed241df6 [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
add 827dc4f0ca6 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
add b68cf015a57 [opt](catalog) fe-connector: unify the connector cache
foundation, memoize per-statement hot paths, fix iceberg case-insensitive
column DDL (#66029)
add 0d88b8dec20 [fix](iceberg) dedupe partition columns so a
multi-transform spec cannot crash partition pruning
add 1aa5ae9597e [fix](catalog) port #66008's paimon-cpp removal to the
paimon connector scan path
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (c9540902dd1)
\
N -- N -- N refs/heads/branch-catalog-spi (1aa5ae9597e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/benchmark/parquet/benchmark_parquet_reader.hpp | 118 +++++++-
be/benchmark/parquet/parquet_benchmark_scenarios.h | 49 +++-
.../core/data_type_serde/parquet_decode_source.cpp | 47 ++++
be/src/exec/operator/file_scan_operator.cpp | 9 +-
be/src/exec/scan/file_scanner_v2.cpp | 40 ++-
be/src/exprs/function/functions_comparison.h | 41 ++-
be/src/format_v2/file_reader.h | 1 +
be/src/format_v2/jni/jni_table_reader.cpp | 14 +-
be/src/format_v2/jni/paimon_jni_reader.cpp | 2 +-
be/src/format_v2/parquet/parquet_profile.cpp | 18 ++
be/src/format_v2/parquet/parquet_profile.h | 18 +-
be/src/format_v2/parquet/parquet_scan.cpp | 223 +++++++++++++--
be/src/format_v2/parquet/reader/column_reader.cpp | 2 +-
be/src/format_v2/parquet/reader/column_reader.h | 2 +-
.../parquet/reader/native/column_chunk_reader.cpp | 195 +++++++++++++
.../parquet/reader/native/column_chunk_reader.h | 17 +-
.../parquet/reader/native/column_reader.cpp | 225 ++++++++++++++-
.../parquet/reader/native/column_reader.h | 38 +++
be/src/format_v2/parquet/reader/native/decoder.cpp | 3 +
.../parquet/reader/native_column_reader.cpp | 169 ++++++++++--
.../parquet/reader/native_column_reader.h | 8 +-
be/src/format_v2/table/paimon_reader.cpp | 35 ++-
be/src/format_v2/table_reader.cpp | 2 +
be/src/format_v2/table_reader.h | 1 +
be/src/format_v2/wal/wal_reader.cpp | 301 +++++++++++++++++++++
.../{native/native_reader.h => wal/wal_reader.h} | 56 ++--
be/src/format_v2/wal/wal_table_reader.cpp | 47 ++++
.../ticker.h => format_v2/wal/wal_table_reader.h} | 32 +--
be/test/exec/scan/file_scanner_v2_test.cpp | 35 +--
be/test/exprs/expr_zonemap_filter_test.cpp | 33 ++-
be/test/format_v2/jni/paimon_jni_reader_test.cpp | 20 +-
be/test/format_v2/parquet/native_decoder_test.cpp | 99 ++++---
.../parquet/parquet_benchmark_scenarios_test.cpp | 22 +-
be/test/format_v2/parquet/parquet_scan_test.cpp | 289 +++++++++++++++++++-
be/test/format_v2/table/paimon_reader_test.cpp | 59 +++-
be/test/format_v2/wal/wal_reader_test.cpp | 171 ++++++++++++
docs/file-scanner-v2-parquet-scan-design.md | 25 +-
.../iceberg/IcebergConnectorMetadata.java | 15 +-
.../connector/iceberg/IcebergPartitionUtils.java | 21 +-
.../iceberg/IcebergConnectorMetadataTest.java | 26 ++
.../iceberg/IcebergPartitionUtilsTest.java | 86 ++++++
.../connector/paimon/PaimonScanPlanProvider.java | 118 +++-----
.../doris/connector/paimon/PaimonScanRange.java | 39 +--
.../connector/paimon/PaimonScanExplainTest.java | 2 -
.../paimon/PaimonScanPlanProviderTest.java | 156 +++++++----
.../paimon/PaimonScanRangeReaderTypeTest.java | 60 ++--
.../rules/rewrite/PruneFileScanPartition.java | 20 ++
plan-doc/HANDOFF.md | 33 ++-
48 files changed, 2606 insertions(+), 436 deletions(-)
create mode 100644 be/src/format_v2/wal/wal_reader.cpp
copy be/src/format_v2/{native/native_reader.h => wal/wal_reader.h} (50%)
create mode 100644 be/src/format_v2/wal/wal_table_reader.cpp
copy be/src/{exec/scan/task_executor/ticker.h =>
format_v2/wal/wal_table_reader.h} (59%)
create mode 100644 be/test/format_v2/wal/wal_reader_test.cpp
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
