This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 5f009592035 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit 61a8b380fa5 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit 7b3821fe170 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 26c5b8612c7 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 9b9d71c466f [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 25af7a74f10 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit d279444880f [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 4612d60ec18 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 38a877d7c20 [refactor](catalog) Catalog spi 11 hive (#65473)
omit 3e9998ca647 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit ce71c9da55b [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 745fa94c390 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit eab026fc43e [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 46cc9df4413 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit c09960dcb29 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit b19ac5e7b72 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit b8c21a43e43 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit 0799b9d7855 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 502f675ff5d [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 569010962c6 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 7c91f661efa [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 1aed06fe1fa [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 5e9fb3e0c21 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit a9ea7d3a43f [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit eb9c0a89119 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 3b0c9a2596f [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit e176a7baa21 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 14272ec8240 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 7c88f49bd54 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 17c45263461 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 523b1294955 update P5 handoff and fix compile issue
omit 70397f44126 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit b8b7374383f [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit e7e88922dbc [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 01285e6f315 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 20ce63809f0 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit da98221815e [doc](connector) add project tracking system for catalog
SPI migration
add f39116498dc [fix](scanner)(nereids) Harden FileScannerV2 and fix
external COUNT pushdown semantics (#65548)
add 91a12e13170 [fix](be) Fix macOS BE compilation errors (#65783)
add 6ab33331cb7 [doc](connector) add project tracking system for catalog
SPI migration
add d66c2023cfa [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 538d21a47eb [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 52d3886bb66 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 5771264af95 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 30625653cfa [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 73394ebd8c5 update P5 handoff and fix compile issue
add 0b4e497fa03 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add adf258f4a1d [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add d1c3db89c4b [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add c7cda5c1147 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 3e3442c8011 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 2693b19324e [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 3af05086513 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 1283fbd255e [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add fdc2986dca3 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 0657f800a56 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add b25082a5d33 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 606cea86f07 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 7e55cb8a1b0 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add b6f9ed5d1cd [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 4302d1e3525 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add d29810c29d8 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add c9579dc1a72 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 0592a16110c [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add fdc15137538 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add a1c945e88d9 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 16fdb1808b5 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 92a53a0aac5 [refactor](catalog) Catalog spi 11 hive (#65473)
add d3f98a787b1 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add e5af2dbcaa1 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 14b35d647c2 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add f88027e75ca [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add efe7cbb311a [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add d0093c680c6 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add 1586b7608a6 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add d73cfdb08ab [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add 3fb9cae390f [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5f009592035)
\
N -- N -- N refs/heads/branch-catalog-spi (3fb9cae390f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/common/phdr_cache.h | 2 +
be/src/exec/operator/scan_operator.cpp | 9 +
be/src/exec/operator/scan_operator.h | 13 +
be/src/exec/scan/file_scanner.h | 28 +-
be/src/exec/scan/file_scanner_v2.cpp | 78 ++++-
be/src/exec/scan/file_scanner_v2.h | 10 +
be/src/format_v2/column_mapper.cpp | 282 ++++++++++++++++---
be/src/format_v2/file_reader.cpp | 9 +-
be/src/format_v2/file_reader.h | 12 +
be/src/format_v2/native/native_reader.cpp | 28 +-
be/src/format_v2/parquet/parquet_column_schema.cpp | 13 +-
be/src/format_v2/parquet/parquet_file_context.cpp | 154 ++++++----
be/src/format_v2/parquet/parquet_file_context.h | 36 +++
be/src/format_v2/parquet/parquet_reader.cpp | 89 +++++-
be/src/format_v2/parquet/parquet_scan.cpp | 45 ++-
be/src/format_v2/parquet/parquet_statistics.cpp | 4 +-
be/src/format_v2/parquet/parquet_type.cpp | 3 +-
be/src/format_v2/parquet/parquet_type.h | 3 +
be/src/format_v2/table/hudi_reader.cpp | 6 +
be/src/format_v2/table/hudi_reader.h | 1 +
be/src/format_v2/table/iceberg_reader.cpp | 15 +-
be/src/format_v2/table/paimon_reader.cpp | 6 +
be/src/format_v2/table/paimon_reader.h | 1 +
be/src/format_v2/table_reader.cpp | 10 +-
be/src/format_v2/table_reader.h | 102 +++++--
be/test/exec/scan/file_scanner_v2_test.cpp | 33 +++
be/test/format_v2/column_mapper_test.cpp | 313 +++++++++++++++++++++
be/test/format_v2/native/native_reader_test.cpp | 37 ++-
.../parquet/parquet_page_cache_range_test.cpp | 39 +++
.../parquet/parquet_reader_control_test.cpp | 14 +
be/test/format_v2/parquet/parquet_scan_test.cpp | 128 +++++++++
be/test/format_v2/parquet/parquet_schema_test.cpp | 26 +-
be/test/format_v2/table/hudi_reader_test.cpp | 99 +++++++
be/test/format_v2/table/iceberg_reader_test.cpp | 76 +++++
be/test/format_v2/table/paimon_reader_test.cpp | 53 ++++
be/test/format_v2/table_reader_test.cpp | 297 ++++++++++++++++++-
.../apache/doris/datasource/scan/FileScanNode.java | 17 ++
.../datasource/scan/PluginDrivenScanNode.java | 7 +-
.../doris/datasource/tvf/source/TVFScanNode.java | 5 +-
.../glue/translator/PhysicalPlanTranslator.java | 9 +
.../glue/translator/PlanTranslatorContext.java | 10 +
.../rules/implementation/AggregateStrategies.java | 30 +-
.../physical/PhysicalStorageLayerAggregate.java | 23 +-
.../java/org/apache/doris/planner/PlanNode.java | 7 +
.../datasource/tvf/source/TVFScanNodeTest.java | 55 ++++
.../rewrite/PhysicalStorageLayerAggregateTest.java | 25 +-
gensrc/thrift/PlanNodes.thrift | 4 +
.../tvf/file_scanner_v2_empty.csv | 0
.../tvf/file_scanner_v2_struct_0_bigint.parquet | Bin 0 -> 965 bytes
.../tvf/file_scanner_v2_struct_1_int.parquet | Bin 0 -> 923 bytes
.../tvf/file_scanner_v2_unsupported_time.parquet | Bin 0 -> 673 bytes
.../tvf/test_file_scanner_v2_review_fixes.out | 17 ++
.../tvf/test_file_scanner_v2_review_fixes.groovy | 113 ++++++++
53 files changed, 2210 insertions(+), 186 deletions(-)
copy be/src/pch/pch.cc =>
regression-test/data/external_table_p0/tvf/file_scanner_v2_empty.csv (100%)
create mode 100644
regression-test/data/external_table_p0/tvf/file_scanner_v2_struct_0_bigint.parquet
create mode 100644
regression-test/data/external_table_p0/tvf/file_scanner_v2_struct_1_int.parquet
create mode 100644
regression-test/data/external_table_p0/tvf/file_scanner_v2_unsupported_time.parquet
create mode 100644
regression-test/data/external_table_p0/tvf/test_file_scanner_v2_review_fixes.out
create mode 100644
regression-test/suites/external_table_p0/tvf/test_file_scanner_v2_review_fixes.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
