This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 78db0c21cdb [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 174c1a6029f [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit ec93c2263e2 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 282256e88b3 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit afb59b58a0a [refactor](catalog) Catalog spi 11 hive (#65473)
omit 8f22dd86ad9 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit 9520659bf4f [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit 09d62d1a77c [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 046ccc229b6 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit e931e9bc750 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 64a3228ffc6 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit a3b61ca903d [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit 7ecb54860aa [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit dda855c608f [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 65f0628ffbc [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 4f80cfb6f2e [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit a9701b55f4b [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 1afb9c18cc4 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit dbd1b996ad8 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit 843844aa6a9 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 3d8e8e17a27 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 3b8d5dde804 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 8450d7c4bd0 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 5e005323ceb [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 3effe0b416d [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 65ddc62a550 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 3ab9005a4ba update P5 handoff and fix compile issue
omit e0703a557ac [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit 6021582d496 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit dfc58d6e794 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit ea61b536723 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 4235de9bb8d [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit fe3175e35c4 [doc](connector) add project tracking system for catalog
SPI migration
add ecf7732adbd [fix](function) Support cross-dimensional casts for null
arrays (#65456)
add 4df43e95c39 [chore](thirdparty) introduce thridparty about lance-c
(#65304)
add 2ec58216929 [fix](be) Guard empty stack trace formatting (#65731)
add 589fd3e9a1e [opt](local shuffle) support bucket shuffle for set
operation (#65129)
add f697dfa973c [fix](fe) Revert semi join constraint matching fix (#65546)
add a99acfc7a40 [improvement](be-java-extensions) Drop hive-catalog-shade
from be-java-extensions (#65733)
add 007e75666b5 [refactor](be) move storage field type conversion to core
(#65717)
add 9dd68334c5d [test](fe) Cover internal query audit failure in FE unit
test (#65696)
add 14331e4974d [enhance](cloud) Increase get version timeout to 30
seconds (#65625)
add 3a75387e613 [fix](catalog) re-auth Iceberg REST catalog on 401 instead
of wedging until FE restart (#64966)
add 9a5afd4958f [fix](arrow-flight-sql) Close temporary VectorSchemaRoot
in createPreparedStatement to fix FE direct memory leak (#65311)
add b7931405b35 [fix](aggregate) Check all aggregate function arguments
(#65537)
add b357fc249cc [fix](eager-agg)Forbid pushdown when the child aggregate
has no GROUP BY keys (#65703)
add 039a1f0909f [refine](exec) simplify DataQueue eos handling (#65324)
add a18295c11c7 [doc](connector) add project tracking system for catalog
SPI migration
add 703bb3b7591 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add b4046230334 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add daaedec3e75 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add fa7c54b431a [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 2ac79416c50 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add c6ffc4d3451 update P5 handoff and fix compile issue
add 20cc2da1586 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 2cb35776ba4 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 4581a4ad235 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 4fd2bc21268 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 60129926159 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 1f63fe58fbe [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 819716f90f0 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 9e33a312b4f [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add ad1ffd851ac [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 2d991b348be [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add d65e0f152d3 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 4996ec6acb6 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add e7ecc2af910 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 2e2b85ebe9a [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add b5733067fe7 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add ecbc2d9db5e [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 5cb3f1495d5 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add f66f80f50d1 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 88bb211f706 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 0f68a7cd860 [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 4d46863b7f1 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 83585fd5097 [refactor](catalog) Catalog spi 11 hive (#65473)
add 427c956dda2 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add dc993c8edfa [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add b0244748a4b [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 45025882317 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 21420349783 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add b2dff681aad [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (78db0c21cdb)
\
N -- N -- N refs/heads/branch-catalog-spi (b2dff681aad)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/cmake/thirdparty.cmake | 4 +
be/src/common/stack_trace.cpp | 23 +-
be/src/core/column/column_variant.cpp | 3 +-
be/src/core/data_type/data_type.cpp | 4 +-
be/src/core/data_type/data_type.h | 1 +
be/src/core/data_type/define_primitive_type.h | 3 +
be/src/core/data_type/storage_field_type.cpp | 202 +++++++
.../data_type/storage_field_type.h} | 17 +-
.../data_type_serde/data_type_decimal_serde.cpp | 4 +-
.../data_type_serde/data_type_number_serde.cpp | 3 +-
be/src/core/data_type_serde/data_type_serde.cpp | 4 +-
be/src/exec/common/variant_util.cpp | 9 +-
be/src/exec/operator/cache_sink_operator.cpp | 13 +-
be/src/exec/operator/cache_source_operator.cpp | 17 +-
be/src/exec/operator/cache_source_operator.h | 4 -
be/src/exec/operator/data_queue.cpp | 103 ++--
be/src/exec/operator/data_queue.h | 31 +-
be/src/exec/operator/union_sink_operator.cpp | 37 +-
be/src/exec/operator/union_sink_operator.h | 8 +-
be/src/exec/operator/union_source_operator.cpp | 41 +-
be/src/exec/operator/union_source_operator.h | 9 +-
be/src/exprs/function/cast/cast_to_array.h | 17 +-
be/src/storage/olap_common.h | 4 +-
be/src/storage/segment/column_reader.h | 3 +-
be/src/storage/tablet/tablet_schema.cpp | 130 +----
be/src/storage/tablet/tablet_schema.h | 2 -
be/test/common/status_test.cpp | 29 +
be/test/core/data_type/storage_field_type_test.cpp | 171 ++++++
be/test/exec/operator/union_operator_test.cpp | 6 +-
be/test/exec/pipeline/data_queue_test.cpp | 148 ++---
be/test/exprs/function/cast/cast_to_array_test.cpp | 91 +++
build.sh | 4 +-
dist/LICENSE-dist.txt | 3 +
dist/licenses/LICENSE-lance-c-rust-crates.txt | 625 +++++++++++++++++++++
fe/be-java-extensions/avro-scanner/pom.xml | 116 ----
.../org/apache/doris/avro/AvroColumnValue.java | 204 -------
.../org/apache/doris/avro/AvroFileContext.java | 61 --
.../java/org/apache/doris/avro/AvroJNIScanner.java | 225 --------
.../java/org/apache/doris/avro/AvroProperties.java | 45 --
.../java/org/apache/doris/avro/AvroReader.java | 111 ----
.../java/org/apache/doris/avro/AvroTypeUtils.java | 126 -----
.../java/org/apache/doris/avro/HDFSFileReader.java | 80 ---
.../java/org/apache/doris/avro/S3FileReader.java | 103 ----
.../main/java/org/apache/doris/avro/S3Utils.java | 102 ----
.../avro-scanner/src/main/resources/package.xml | 41 --
.../org/apache/doris/avro/AvroTypeUtilsTest.java | 105 ----
fe/be-java-extensions/hive-udf-shade/pom.xml | 114 ++++
fe/be-java-extensions/java-udf/pom.xml | 3 +-
fe/be-java-extensions/pom.xml | 2 +-
.../main/java/org/apache/doris/common/Config.java | 2 +-
.../doris/connector/iceberg/IcebergConnector.java | 123 +++-
.../iceberg/IcebergSessionCatalogAdapter.java | 16 +-
.../ReauthenticatingRestSessionCatalog.java | 283 ++++++++++
.../ReauthenticatingRestSessionCatalogTest.java | 191 +++++++
.../glue/translator/PhysicalPlanTranslator.java | 36 +-
.../org/apache/doris/nereids/hint/LeadingHint.java | 17 +-
.../properties/ChildOutputPropertyDeriver.java | 121 ++--
.../properties/ChildrenPropertiesRegulator.java | 195 ++++---
.../nereids/properties/RequestPropertyDeriver.java | 55 +-
.../nereids/rules/analysis/NormalizeAggregate.java | 11 +-
.../rewrite/eageraggregation/EagerAggRewriter.java | 37 ++
.../job/UnassignedScanBucketOlapTableJob.java | 12 +-
.../trees/plans/physical/PhysicalSetOperation.java | 1 -
.../apache/doris/planner/LocalExchangeNode.java | 11 +-
.../org/apache/doris/planner/SetOperationNode.java | 10 +-
.../arrowflight/DorisFlightSqlProducer.java | 16 +-
.../arrowflight/results/FlightSqlChannel.java | 6 +
.../apache/doris/nereids/hint/LeadingHintTest.java | 219 --------
.../rules/analysis/NormalizeAggregateTest.java | 10 +
.../job/UnassignedScanBucketOlapTableJobTest.java | 40 ++
.../planner/LocalShuffleNodeCoverageTest.java | 49 ++
.../doris/qe/StmtExecutorInternalQueryTest.java | 52 +-
.../arrowflight/DorisFlightSqlProducerTest.java | 145 +++++
.../iceberg-rest-reauth-connector-port-design.md | 75 +++
.../data/external_table_p0/tvf/test_tvf_avro.out | 97 ----
.../cast/test_cast_null_array_to_nested_array.out | 37 ++
.../query_p0/eager_agg/scalar_agg_pushdown.out | 41 ++
.../bucket_shuffle_set_operation.out | 185 ++++++
.../external_table_p0/tvf/test_tvf_avro.groovy | 211 -------
.../test_audit_log_internal_query_failure.groovy | 98 ----
.../test_cast_null_array_to_nested_array.groovy | 120 ++++
.../query_p0/aggregate/agg_group_concat.groovy | 8 +
.../query_p0/eager_agg/scalar_agg_pushdown.groovy | 126 +++++
.../bucket_shuffle_set_operation.groovy | 315 ++++++++++-
thirdparty/LICENSE.txt | 10 +
thirdparty/build-thirdparty.sh | 75 +++
thirdparty/vars.sh | 7 +
87 files changed, 3740 insertions(+), 2534 deletions(-)
create mode 100644 be/src/core/data_type/storage_field_type.cpp
copy be/src/{exprs/aggregate/aggregate_function_avg_weighted.cpp =>
core/data_type/storage_field_type.h} (69%)
create mode 100644 be/test/core/data_type/storage_field_type_test.cpp
create mode 100644 dist/licenses/LICENSE-lance-c-rust-crates.txt
delete mode 100644 fe/be-java-extensions/avro-scanner/pom.xml
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroColumnValue.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroFileContext.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroProperties.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroReader.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroTypeUtils.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/HDFSFileReader.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/S3FileReader.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/S3Utils.java
delete mode 100644
fe/be-java-extensions/avro-scanner/src/main/resources/package.xml
delete mode 100644
fe/be-java-extensions/avro-scanner/src/test/java/org/apache/doris/avro/AvroTypeUtilsTest.java
create mode 100644 fe/be-java-extensions/hive-udf-shade/pom.xml
create mode 100644
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/ReauthenticatingRestSessionCatalog.java
create mode 100644
fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/ReauthenticatingRestSessionCatalogTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/hint/LeadingHintTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducerTest.java
create mode 100644
plan-doc/tasks/designs/iceberg-rest-reauth-connector-port-design.md
delete mode 100644 regression-test/data/external_table_p0/tvf/test_tvf_avro.out
create mode 100644
regression-test/data/function_p0/cast/test_cast_null_array_to_nested_array.out
create mode 100644
regression-test/data/query_p0/eager_agg/scalar_agg_pushdown.out
delete mode 100644
regression-test/suites/external_table_p0/tvf/test_tvf_avro.groovy
delete mode 100644
regression-test/suites/fault_injection_p0/test_audit_log_internal_query_failure.groovy
create mode 100644
regression-test/suites/function_p0/cast/test_cast_null_array_to_nested_array.groovy
create mode 100644
regression-test/suites/query_p0/eager_agg/scalar_agg_pushdown.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
