This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
from 4865cb73eb0 [chore](build) Reduce redundant Actions on branch-4.2
(#68026)
new 48fe6020d7b branch-4.1: [fix](cloud) Decode transaction IDs safely
through Versionstamp #67641 (#67954)
new 18baef9aa26 [fix](fe) Reject unsafe casts in storage MIN/MAX pushdown
(branch-4.1) (#67980)
new 42bf1581939 [Feature](lance) Support nested Arrow Null types (#67921)
new 0dd17a3bb12 branch-4.1: [improvement](storage) Release unused memory
earlier, rather than leaving it to the memory pool for guaranteed release
#57215 (#67956)
new 9712b7b7704 branch-4.1: [fix](cdc) Use streaming job creator identity
for CDC operations #67851 (#67937)
new b872a1b25fc branch-4.1 [fix](runtime filter) do not push runtime
filter below repeat when target slot is not in all grouping sets (#67992)
new 5f98d6c0382 [fix](lance) Select metric-compatible vector index (#67553)
new 5eb01bb3d82 branch-4.1: [fix](zonemap) Keep every byte of a
STRING/VARCHAR zone map bound #67949 (#67998)
new dbb274e1502 branch-4.1:[fix](iceberg) Preserve nullability in schema
metadata (#67856)
new 865adb83fee branch-4.1: [test](statistics) Make analyze MV row count
assertion stable #67947 (#67988)
new fc68393b41e branch-4.1: [fix](fe) Fix ReadListener leak on rejected
worker task (#62679) (#67990)
new 5d885d57d2d branch-4.1: [fix](be) Return null for TRY_CAST arithmetic
overflow #67697 (#67985)
new 4ba21d3d019 branch-4.1: [fix](variant) Keep integer element_at indexes
out of variant sub-column paths #67955 (#68041)
new 1e02c27ef94 branch-4.1: [fix](cloud) refresh tablet meta of
continuously ingested tablets #67972 (#68044)
new 1a3cfc3cb24 branch-4.1: [fix](be) Avoid per-fd stat when counting
process fds in metrics #67945 (#68046)
new f9c46259d73 [fix](be) Backport Parquet metadata size limit to
branch-4.1 (#67631) (#68047)
new 876e2f05f59 branch-4.1: [fix](be) Fix SIGSEGV in bvar::take_sample
caused by ~AgentCombiner walking freed TLS agents #67276 (#68045)
new 746d3779191 branch-4.1: [fix](regression) Wait for sync MV row count
#68015 (#68053)
new f48a453ae21 branch-4.1: [fix](fe) Keep rollup jobs waiting on conflict
txn abort failure (#67661) (#68055)
new 0d0acda94fe branch-4.1: [fix](test) make test_show_index_data_p2 wait
for loaded data to be visible #67939 (#68058)
new 6640614be23 branch-4.1: [fix](statistics) Treat invalid column
statistics as UNKNOWN instead of aborting analyze job (#66756) (#68061)
new 608f2f533ca branch-4.1: [fix](fe) Upgrade gRPC and remove unused LZ4
dependencies (#68060)
new dd84c95967b branch-4.1: [feature](lance) Add Lance index admission,
job inspection SQL and catalog DDL guard (#67630)
new d092429a3c3 branch-4.1: [fix](mv) Compensate complete invalid roll-up
partition buckets #67882 (#68067)
new ad01732816b branch-4.1: [fix](aggregate) Normalize projected count
slots before null safety checks #67732 (#68068)
new 8e79827c98e branch-4.1: [fix](variable) Track user variables in SQL
cache during binding #67787 (#68069)
The 26 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/cloud/cloud_base_compaction.cpp | 6 +-
be/src/cloud/cloud_cumulative_compaction.cpp | 6 +-
be/src/cloud/cloud_full_compaction.cpp | 2 +-
be/src/cloud/cloud_index_change_compaction.cpp | 4 +-
be/src/cloud/cloud_meta_mgr.cpp | 2 +-
be/src/cloud/cloud_tablet.cpp | 1 +
be/src/cloud/cloud_tablet.h | 9 +-
be/src/cloud/cloud_tablet_mgr.cpp | 85 +-
be/src/common/config.cpp | 4 +
be/src/common/config.h | 2 +
be/src/common/metrics/doris_metrics.cpp | 14 +-
be/src/core/data_type/data_type.h | 4 +-
.../core/data_type_serde/data_type_map_serde.cpp | 2 +-
.../data_type_serde/data_type_nullable_serde.cpp | 12 +
.../data_type_serde/data_type_string_serde.cpp | 15 +-
.../data_type_serde/data_type_struct_serde.cpp | 2 +-
be/src/exprs/function/cast/cast_to_date.h | 2 +-
be/src/exprs/function/cast/cast_to_timestamptz.h | 4 +-
.../cast/variant_v2/cast_array_to_variant.cpp | 22 +-
.../cast/variant_v2/cast_variant_to_array.cpp | 2 +-
.../cast/variant_v2/cast_variant_to_jsonb.cpp | 8 +-
.../cast/variant_v2/cast_variant_to_scalar.cpp | 20 +-
.../cast/variant_v2/cast_variant_to_string.cpp | 6 +-
.../function/cast/variant_v2/cast_variant_v2.cpp | 13 +-
be/src/exprs/vcast_expr.cpp | 52 +-
be/src/exprs/vcast_expr.h | 1 -
be/src/format_v2/lance/lance_reader_helper.cpp | 24 +-
be/src/format_v2/parquet/parquet_file_context.cpp | 14 +-
be/src/format_v2/parquet/parquet_file_context.h | 3 +-
be/src/load/memtable/memtable.cpp | 4 +-
be/src/storage/compaction/compaction.cpp | 2 +-
be/test/cloud/cloud_compaction_test.cpp | 16 +-
be/test/cloud/cloud_tablet_mgr_test.cpp | 101 ++
.../function/cast/cast_variant_v2_from_test.cpp | 2 +-
.../function/cast/cast_variant_v2_to_test.cpp | 2 +-
be/test/exprs/try_cast_expr_test.cpp | 294 +++++-
be/test/format_v2/lance/lance_nested_null_test.cpp | 292 ++++++
be/test/format_v2/parquet/parquet_reader_test.cpp | 26 +-
be/test/format_v2/table/lance_reader_test.cpp | 19 -
.../cloud_index_change_compaction_test.cpp | 4 +-
be/test/storage/olap_type_test.cpp | 38 +-
be/test/storage/segment/zone_map_index_test.cpp | 79 ++
cloud/src/meta-service/doris_txn.cpp | 34 +-
.../src/meta-service/meta_service_tablet_stats.cpp | 1 +
cloud/src/meta-store/versionstamp.h | 14 +-
cloud/test/doris_txn_test.cpp | 176 ++++
cloud/test/versionstamp_test.cpp | 23 +
.../iceberg/scripts/lance_build_nested_null.py | 70 ++
.../scripts/lance_build_preinstalled_catalog.py | 27 +-
.../0-d6a5ba97-9164-4b3e-b773-15de06005c54.txn | Bin 0 -> 643 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 1363 bytes
.../_versions/latest_version_hint.json | 0
...1011111010110112b5b4742fd82cf05955fa974a3.lance | Bin 0 -> 3278 bytes
.../max-compute-connector/pom.xml | 3 +
.../doris/maxcompute/MaxComputeJniWriterTest.java | 52 +
.../main/java/org/apache/doris/common/Config.java | 40 +
.../doris/common/LanceIndexConfigValidator.java | 76 ++
.../doris/job/cdc/request/WriteRecordRequest.java | 1 +
.../common/LanceIndexConfigValidatorTest.java | 196 ++++
fe/fe-core/pom.xml | 1 +
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 1 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 3 +
.../java/org/apache/doris/alter/RollupJobV2.java | 10 +-
.../main/java/org/apache/doris/catalog/Env.java | 3 +-
.../java/org/apache/doris/common/ErrorCode.java | 5 +
.../apache/doris/common/proc/IndexInfoProcDir.java | 3 +
.../org/apache/doris/datasource/CatalogMgr.java | 138 ++-
.../datasource/iceberg/IcebergExternalTable.java | 41 +-
.../doris/datasource/iceberg/IcebergUtils.java | 10 +
.../datasource/lance/LanceExternalCatalog.java | 93 ++
.../datasource/lance/LanceIndexAdmission.java | 544 ++++++++++
.../lance/LanceIndexAdmissionSnapshot.java | 202 ++++
.../doris/datasource/lance/LanceIndexFamilies.java | 116 +++
.../datasource/lance/LanceIndexMetadataLoader.java | 99 +-
.../lance/LanceIndexMutationValidator.java | 46 +
.../lance/LanceSchemaContractBuilder.java | 271 +++++
.../doris/datasource/lance/LanceTypeConverter.java | 27 +-
.../datasource/lance/job/LanceIndexJobManager.java | 37 +
.../datasource/lance/source/LanceScanNode.java | 59 +-
.../doris/datasource/tvf/source/TVFScanNode.java | 14 +-
.../doris/httpv2/rest/StreamingJobAction.java | 92 +-
.../doris/httpv2/rest/TableSchemaAction.java | 7 +-
.../insert/streaming/StreamingMultiTblTask.java | 1 +
.../java/org/apache/doris/mysql/ReadListener.java | 37 +-
.../glue/translator/PhysicalPlanTranslator.java | 35 +-
.../doris/nereids/jobs/executor/Analyzer.java | 12 -
.../doris/nereids/parser/LogicalPlanBuilder.java | 20 +
.../post/RuntimeFilterPushDownVisitor.java | 17 +
.../post/runtimefilterv2/PushDownVisitor.java | 17 +
.../nereids/rules/analysis/ExpressionAnalyzer.java | 7 +-
.../nereids/rules/analysis/VariableToLiteral.java | 40 -
.../rules/exploration/mv/PartitionCompensator.java | 7 +-
.../rules/expression/ExpressionRuleType.java | 1 -
.../expression/rules/ReplaceVariableByLiteral.java | 53 -
.../rules/implementation/AggregateStrategies.java | 18 +-
.../rules/rewrite/VariantSubPathPruning.java | 10 +-
.../expressions/functions/ExpressionTrait.java | 29 +-
.../apache/doris/nereids/trees/plans/PlanType.java | 2 +
.../trees/plans/commands/AlterTableCommand.java | 103 +-
.../trees/plans/commands/ShowColumnsCommand.java | 4 +-
.../plans/commands/ShowLanceIndexJobCommand.java | 174 ++++
.../plans/commands/ShowLanceIndexJobsCommand.java | 378 +++++++
.../trees/plans/commands/info/CreateTableInfo.java | 8 +-
.../trees/plans/commands/info/IndexDefinition.java | 4 +
.../physical/PhysicalLazyMaterializeTVFScan.java | 6 +-
.../trees/plans/visitor/CommandVisitor.java | 10 +
.../apache/doris/nereids/util/ExpressionUtils.java | 9 +-
.../apache/doris/service/FrontendServiceImpl.java | 16 +-
.../apache/doris/statistics/BaseAnalysisTask.java | 9 +-
.../org/apache/doris/statistics/ColStatsData.java | 4 +
.../apache/doris/statistics/ColumnStatistic.java | 5 +
.../apache/doris/statistics/StatisticsCache.java | 11 +-
.../ExternalFileTableValuedFunction.java | 5 +
.../tablefunction/FileTableValuedFunction.java | 12 +
.../tablefunction/LocalTableValuedFunction.java | 2 +-
.../doris/transaction/GlobalTransactionMgr.java | 4 +
.../org/apache/doris/alter/RollupJobV2Test.java | 39 +
.../doris/datasource/CatalogMgrLanceGuardTest.java | 495 ++++++++++
.../iceberg/IcebergSchemaDisplayTest.java | 297 ++++++
.../LanceExternalCatalogIndexAdmissionTest.java | 133 +++
.../lance/LanceIndexAdmissionSnapshotTest.java | 281 ++++++
.../datasource/lance/LanceIndexAdmissionTest.java | 1042 ++++++++++++++++++++
.../datasource/lance/LanceIndexFamiliesTest.java | 178 ++++
.../lance/LanceIndexMetadataLoaderTest.java | 128 +++
.../lance/LanceIndexMutationValidatorTest.java | 79 ++
.../datasource/lance/LanceJniTestSupport.java | 57 ++
.../LanceSchemaContractBuilderRealDatasetTest.java | 133 +++
.../lance/LanceSchemaContractBuilderTest.java | 403 ++++++++
.../datasource/lance/LanceTypeConverterTest.java | 100 +-
.../lance/job/LanceIndexJobManagerQueryTest.java | 131 +++
.../datasource/lance/source/LanceScanNodeTest.java | 128 +++
.../datasource/tvf/source/TVFScanNodeTest.java | 95 ++
.../rest/StreamingJobActionSchemaChangeTest.java | 167 ++++
.../apache/doris/mysql/ConnectionExceedTest.java | 37 +
.../LanceLazyMaterializationUpgradeTest.java | 159 +++
.../parser/ShowLanceIndexJobsParserTest.java | 183 ++++
.../nereids/postprocess/RuntimeFilterTest.java | 74 ++
.../rules/analysis/UserVariableAnalysisTest.java | 29 +
.../exploration/mv/PartitionCompensatorTest.java | 78 ++
.../rewrite/PhysicalStorageLayerAggregateTest.java | 163 ++-
.../rules/rewrite/PruneNestedColumnTest.java | 17 +
.../rules/rewrite/VariantPruningLogicTest.java | 4 +-
.../expressions/functions/ExpressionTraitTest.java | 86 --
.../AlterTableCommandLanceAdmissionTest.java | 627 ++++++++++++
.../commands/AlterTableCommandLanceIndexTest.java | 27 +-
.../commands/ShowLanceIndexJobCommandTest.java | 565 +++++++++++
.../commands/ShowLanceIndexJobsCommandTest.java | 821 +++++++++++++++
.../doris/statistics/BaseAnalysisTaskTest.java | 47 +-
.../transaction/GlobalTransactionMgrTest.java | 21 +
fe/pom.xml | 17 +-
.../cdcclient/service/PipelineCoordinator.java | 10 +-
.../doris/cdcclient/sink/DorisBatchStreamLoad.java | 2 +
.../doris/cdcclient/sink/HttpPutBuilder.java | 2 -
.../org/apache/doris/cdcclient/utils/HttpUtil.java | 4 -
.../doris/cdcclient/utils/SchemaChangeManager.java | 31 +-
.../cdcclient/itcase/CdcClientWriteHarness.java | 1 +
.../doris/cdcclient/itcase/MockDorisServer.java | 16 +-
.../cdcclient/utils/SchemaChangeManagerTest.java | 20 +-
.../string/test_string_embedded_nul_zonemap.out | 73 ++
.../iceberg/iceberg_schema_change_ddl.out | 72 +-
.../lance/test_lance_nested_null.out | 11 +
.../cast/test_try_cast_decimal_overflow.out | 22 +
.../stats/invalid_stats/invalid_stats.out | 2 +-
.../data/variant_p0/element_function.out | 34 +
.../string/test_string_embedded_nul_zonemap.groovy | 79 ++
.../fault_tolerance_nereids.groovy | 12 +-
.../iceberg/test_iceberg_show_nullable.groovy | 117 +++
.../lance/test_lance_index_admission.groovy | 271 +++++
.../lance/test_lance_index_ddl.groovy | 17 +-
.../lance/test_lance_nested_null.groovy | 67 ++
.../test_lance_vector_search_index_matrix.groovy | 9 +-
.../paimon/test_paimon_minmax_cast.groovy | 105 ++
.../cast/test_try_cast_complex_overflow.groovy | 75 ++
.../cast/test_try_cast_conversion_errors.groovy | 73 ++
.../cast/test_try_cast_decimal_overflow.groovy | 110 +++
.../show_data/test_show_index_data_p2.groovy | 31 +-
.../cdc/test_streaming_mysql_job_priv.groovy | 41 +-
.../cache/parse_sql_from_sql_cache.groovy | 23 +
.../nereids_p0/test_minmax_cast_pushdown.groovy | 64 ++
.../rollup_p0/test_create_mv_and_mtmv.groovy | 13 +
.../suites/statistics/analyze_stats.groovy | 4 +-
.../suites/statistics/test_analyze_all_null.groovy | 4 +-
.../suites/statistics/test_analyze_mv.groovy | 10 +-
.../test_analyze_sample_almost_all_null.groovy | 138 +++
.../suites/variant_p0/element_function.groovy | 73 ++
...c-1.4.0-fix-agent-combiner-use-after-free.patch | 86 ++
186 files changed, 12605 insertions(+), 783 deletions(-)
create mode 100644 be/test/format_v2/lance/lance_nested_null_test.cpp
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/lance_build_nested_null.py
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/nested_null.lance/_transactions/0-d6a5ba97-9164-4b3e-b773-15de06005c54.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/nested_null.lance/_versions/18446744073709551614.manifest
copy
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/{all_types.lance
=> nested_null.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/nested_null.lance/data/0110100101011111010110112b5b4742fd82cf05955fa974a3.lance
create mode 100644
fe/fe-common/src/main/java/org/apache/doris/common/LanceIndexConfigValidator.java
create mode 100644
fe/fe-common/src/test/java/org/apache/doris/common/LanceIndexConfigValidatorTest.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexAdmission.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexAdmissionSnapshot.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexFamilies.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilder.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/VariableToLiteral.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/ReplaceVariableByLiteral.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommand.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommand.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrLanceGuardTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergSchemaDisplayTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceExternalCatalogIndexAdmissionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexAdmissionSnapshotTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexAdmissionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexFamiliesTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceJniTestSupport.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderRealDatasetTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerQueryTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/httpv2/rest/StreamingJobActionSchemaChangeTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/LanceLazyMaterializationUpgradeTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ShowLanceIndexJobsParserTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/ExpressionTraitTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceAdmissionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommandTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommandTest.java
create mode 100644
regression-test/data/datatype_p0/string/test_string_embedded_nul_zonemap.out
create mode 100644
regression-test/data/external_table_p0/lance/test_lance_nested_null.out
create mode 100644
regression-test/data/function_p0/cast/test_try_cast_decimal_overflow.out
create mode 100644
regression-test/suites/datatype_p0/string/test_string_embedded_nul_zonemap.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_show_nullable.groovy
create mode 100644
regression-test/suites/external_table_p0/lance/test_lance_index_admission.groovy
create mode 100644
regression-test/suites/external_table_p0/lance/test_lance_nested_null.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_minmax_cast.groovy
create mode 100644
regression-test/suites/function_p0/cast/test_try_cast_complex_overflow.groovy
create mode 100644
regression-test/suites/function_p0/cast/test_try_cast_conversion_errors.groovy
create mode 100644
regression-test/suites/function_p0/cast/test_try_cast_decimal_overflow.groovy
create mode 100644
regression-test/suites/nereids_p0/test_minmax_cast_pushdown.groovy
create mode 100644
regression-test/suites/statistics/test_analyze_sample_almost_all_null.groovy
create mode 100644
thirdparty/patches/brpc-1.4.0-fix-agent-combiner-use-after-free.patch
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
