This is an automated email from the ASF dual-hosted git repository.
924060929 pushed a change to branch auto-pick-64160-branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 0cbc0b40bb0 [fix](fe) Harden getDbForReplay against a missing metadata
cache
omit 31cfeed0e06 [test](fe) Make external row count cache refresh test
deterministic
omit fdf93c6d9ba [fix](fe) Fence cold partition event row counts
omit 18ecae819aa [fix](fe) Preserve committed cache invalidation
omit 438b2d4f681 [fix](fe) Close remaining external row count invalidation
gaps
omit 434a25e9ab1 [fix](fe) Preserve row count fences on refresh failures
omit 446a9242e94 [fix](fe) Fence row count cache on metadata events
omit 41d1009caf9 [fix](fe) Invalidate external row counts after metadata
changes
add ccc81563d3d [opt](lance) reduce redundant work in LanceTableReader
(#68335)
add e6bb8f591c4 branch-4.1: [fix](filecache) Unify the file cache TTL
deadline across all paths (#68313)
add 884ae5a1458 branch-4.1: [fix](workload group) Drain adaptive flush
callbacks before pool teardown #68264 (#68326)
add 78a67f84769 [branch-4.1][fix](ngram-bf) Give up the NGRAM index when a
LIKE pattern can carry an escape (#68021) (#68319)
add 899884296d0 branch-4.1: [fix](fetype) Commit FE type only after
transition completes #68302 (#68325)
add 69681aa70f7 branch-4.1: [test](regression) Remove remote Doris variant
select case (#68355)
add c9fd8ec0ec6 [fix](types) Fix binary value ownership and timestamp
primitives (#68297)
add eda5d8772fe branch-4.1: [fix](compaction) Avoid repeatedly compacting
large cumulative rowset #64954 (#65601)
add a334145fee5 branch-4.1: [fix](compaction) Count empty rowsets in
approximate compaction score #66610 (#66656)
add 85c9e0aab94 branch-4.1: [fix](metacache) release Iceberg FileIO and
Hudi fsView on cache eviction #66914 (#66913)
add ee39bd05c59 branch-4.1: [fix](be) Avoid rowset map race in backend
tablets scanner (#68306) (#68316)
add c1f200dabb4 [fix](fe) Invalidate external row counts after metadata
changes
add aca1272fd9e [fix](fe) Fence row count cache on metadata events
add 6605593c438 [fix](fe) Preserve row count fences on refresh failures
add 12bc5cce73b [fix](fe) Close remaining external row count invalidation
gaps
add 58d9bd6f6db [fix](fe) Preserve committed cache invalidation
add 288a8360783 [fix](fe) Fence cold partition event row counts
add a2ef2dd09b3 [test](fe) Make external row count cache refresh test
deterministic
add 62f09af1197 [fix](fe) Harden getDbForReplay against a missing metadata
cache
add 20ca42025e8 [fix](fe) Close remaining row-count invalidation gaps
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (0cbc0b40bb0)
\
N -- N -- N refs/heads/auto-pick-64160-branch-4.1 (20ca42025e8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../cloud/cloud_cumulative_compaction_policy.cpp | 17 +-
be/src/cloud/cloud_delete_task.cpp | 1 +
be/src/cloud/cloud_internal_service.cpp | 2 +-
be/src/cloud/cloud_rowset_builder.cpp | 6 +-
be/src/cloud/cloud_schema_change_job.cpp | 2 +-
be/src/cloud/cloud_tablet.cpp | 6 +-
be/src/cloud/cloud_warm_up_manager.cpp | 2 +-
be/src/core/column/column_varbinary.cpp | 20 +
be/src/core/column/column_varbinary.h | 6 +
be/src/core/data_type/data_type_factory.cpp | 4 +
.../data_type_serde/data_type_varbinary_serde.cpp | 80 ++
.../data_type_serde/data_type_varbinary_serde.h | 16 +
be/src/core/field.cpp | 73 +-
be/src/core/field.h | 7 +-
be/src/core/value/timestamptz_value.cpp | 16 +
be/src/exec/common/hash_table/hash_key_type.h | 7 +
be/src/exec/operator/olap_scan_operator.cpp | 1 -
be/src/exec/scan/olap_scanner.cpp | 2 +-
.../sink/writer/iceberg/partition_transformers.cpp | 6 +
.../aggregate/aggregate_function_min_max_impl.h | 4 +
be/src/exprs/create_predicate_function.h | 3 +
be/src/exprs/function/cast/cast_to_date.h | 6 +-
.../function/cast/cast_to_datetimev2_impl.hpp | 89 +-
be/src/exprs/function/cast/cast_to_string.h | 19 +-
be/src/exprs/function/cast/cast_to_timestamptz.h | 8 +-
be/src/exprs/function/in.h | 4 +
be/src/exprs/vexpr.cpp | 5 +-
be/src/exprs/vin_predicate.cpp | 4 +-
be/src/format_v2/lance/lance_reader_helper.cpp | 251 ++---
be/src/format_v2/lance/lance_reader_helper.h | 31 +-
.../lance/lance_record_batch_converter.cpp | 261 +++++
.../format_v2/lance/lance_record_batch_converter.h | 91 ++
.../lance/lance_runtime_filter_helper.cpp | 6 +
.../format_v2/lance/lance_runtime_filter_helper.h | 4 +
be/src/format_v2/table/lance_reader.cpp | 391 ++-----
be/src/format_v2/table/lance_reader.h | 19 +-
.../information_schema/schema_tablets_scanner.cpp | 161 ++-
be/src/io/fs/file_writer.h | 2 +-
be/src/runtime/exec_env_init.cpp | 7 +
be/src/runtime/workload_group/workload_group.cpp | 51 +-
be/src/runtime/workload_group/workload_group.h | 4 +
.../workload_group/workload_group_manager.cpp | 9 +-
be/src/storage/adaptive_thread_pool_controller.cpp | 57 +-
be/src/storage/adaptive_thread_pool_controller.h | 23 +-
be/src/storage/compaction/compaction.cpp | 2 +-
.../compaction/cumulative_compaction_policy.cpp | 16 +-
be/src/storage/predicate/like_column_predicate.h | 13 +-
be/src/storage/rowset/beta_rowset_reader.cpp | 8 +-
be/src/storage/rowset/rowset_reader_context.h | 5 +-
be/src/storage/rowset/rowset_writer_context.h | 12 +-
be/src/storage/tablet/base_tablet.h | 4 +
be/src/storage/tablet/tablet_meta.cpp | 19 +
be/src/storage/tablet/tablet_meta.h | 11 +
be/src/storage/tablet/tablet_reader.cpp | 5 +-
be/src/util/raw_value.h | 6 +
be/src/util/timezone_utils.cpp | 138 ++-
be/src/util/timezone_utils.h | 6 +
be/test/cloud/cloud_compaction_test.cpp | 45 +
.../cloud_cumulative_compaction_policy_test.cpp | 118 ++-
be/test/core/column/column_varbinary_test.cpp | 154 +++
.../core/data_type/data_type_varbinary_test.cpp | 18 +-
.../data_type_serde_varbinary_test.cpp | 69 ++
.../exec/common/hash_table/hash_key_type_test.cpp | 10 +
.../writer/iceberg/partition_transformers_test.cpp | 14 +
be/test/exprs/aggregate/agg_min_max_test.cpp | 12 +
be/test/exprs/expr_zonemap_filter_test.cpp | 17 +
.../function/cast/cast_to_string_api_test.cpp | 46 +
.../function/cast/cast_to_timestamptz_test.cpp | 45 +
be/test/exprs/function/function_varbinary_test.cpp | 15 +
be/test/exprs/vexpr_test.cpp | 24 +
be/test/format_v2/lance/lance_nested_null_test.cpp | 5 +-
be/test/format_v2/table/lance_reader_test.cpp | 20 +-
be/test/runtime/timestamptz_value_test.cpp | 133 +++
.../workload_group/workload_group_manager_test.cpp | 127 +++
.../adaptive_thread_pool_controller_test.cpp | 162 +++
.../cumulative_compaction_policy_test.cpp | 230 +++-
be/test/storage/tablet/tablet_meta_test.cpp | 39 +
be/test/util/timezone_utils_test.cpp | 66 +-
.../main/java/org/apache/doris/catalog/Env.java | 34 +-
.../org/apache/doris/catalog/RefreshManager.java | 4 +
.../org/apache/doris/common/ThreadPoolManager.java | 26 +-
.../org/apache/doris/datasource/CatalogMgr.java | 2 -
.../doris/datasource/ExternalMetaCacheMgr.java | 11 +-
.../apache/doris/datasource/SplitAssignment.java | 34 +-
.../doris/datasource/hive/HMSExternalCatalog.java | 273 ++++-
.../doris/datasource/hive/HMSExternalTable.java | 18 +-
.../doris/datasource/hive/IcebergDlaTable.java | 30 +-
.../datasource/hudi/HudiExternalMetaCache.java | 124 ++-
.../datasource/hudi/HudiFsViewCacheValue.java | 112 ++
.../doris/datasource/hudi/source/HudiScanNode.java | 642 ++++++++++--
.../datasource/iceberg/DorisHadoopCatalog.java | 96 ++
.../doris/datasource/iceberg/DorisHiveCatalog.java | 96 ++
.../iceberg/IcebergCatalogResourceTracker.java | 121 +++
.../datasource/iceberg/IcebergExternalCatalog.java | 205 +++-
.../iceberg/IcebergExternalDatabase.java | 11 +-
.../iceberg/IcebergExternalMetaCache.java | 762 ++++++++++++--
.../datasource/iceberg/IcebergExternalTable.java | 88 +-
.../datasource/iceberg/IcebergMetadataOps.java | 1100 +++++++++++---------
.../datasource/iceberg/IcebergRuntimeContext.java | 77 ++
.../datasource/iceberg/IcebergSchemaCacheKey.java | 59 +-
.../iceberg/IcebergSnapshotCacheValue.java | 70 ++
.../iceberg/IcebergSysExternalTable.java | 27 +-
.../datasource/iceberg/IcebergTableCacheValue.java | 210 +++-
.../datasource/iceberg/IcebergTransaction.java | 131 ++-
.../doris/datasource/iceberg/IcebergUtils.java | 237 ++++-
.../iceberg/IcebergWriteSchemaContext.java | 47 +-
.../iceberg/action/BaseIcebergAction.java | 32 +-
.../action/IcebergCherrypickSnapshotAction.java | 3 +-
.../action/IcebergExpireSnapshotsAction.java | 4 +-
.../iceberg/action/IcebergFastForwardAction.java | 4 +-
.../action/IcebergPublishChangesAction.java | 3 +-
.../action/IcebergRewriteDataFilesAction.java | 10 +-
.../action/IcebergRewriteManifestsAction.java | 3 +-
.../action/IcebergRollbackToSnapshotAction.java | 3 +-
.../action/IcebergRollbackToTimestampAction.java | 4 +-
.../action/IcebergSetCurrentSnapshotAction.java | 4 +-
.../iceberg/cache/IcebergManifestCacheLoader.java | 15 +-
.../iceberg/rewrite/RewriteDataFileExecutor.java | 182 ++--
.../iceberg/rewrite/RewriteGroupTask.java | 56 +-
.../datasource/iceberg/source/IcebergScanNode.java | 205 +++-
.../metacache/AbstractExternalMetaCache.java | 3 +-
.../doris/datasource/metacache/MetaCacheEntry.java | 131 ++-
.../datasource/metacache/MetaCacheEntryDef.java | 41 +-
.../IcebergFileSystemMetaStoreProperties.java | 4 +-
.../metastore/IcebergHMSMetaStoreProperties.java | 6 +-
.../org/apache/doris/nereids/StatementContext.java | 60 +-
.../expressions/functions/agg/CollectSet.java | 11 +
.../functions/scalar/ArrayContains.java | 5 +
.../functions/scalar/ArrayContainsAll.java | 5 +
.../functions/scalar/ArrayDistinct.java | 1 +
.../functions/scalar/ArrayEnumerateUniq.java | 1 +
.../expressions/functions/scalar/ArrayExcept.java | 5 +
.../functions/scalar/ArrayFunctionUtils.java | 42 +
.../functions/scalar/ArrayIntersect.java | 1 +
.../functions/scalar/ArrayPosition.java | 1 +
.../expressions/functions/scalar/ArrayRemove.java | 1 +
.../expressions/functions/scalar/ArrayUnion.java | 1 +
.../functions/scalar/ArraysOverlap.java | 1 +
.../expressions/functions/scalar/CountEqual.java | 1 +
.../expressions/literal/TimestampTzLiteral.java | 4 +
.../commands/insert/AbstractInsertExecutor.java | 4 +-
.../insert/BaseExternalTableInsertExecutor.java | 18 +-
.../plans/commands/insert/HiveInsertExecutor.java | 42 +-
.../apache/doris/catalog/EnvStateListenerTest.java | 80 ++
.../apache/doris/catalog/RefreshManagerTest.java | 25 +
.../apache/doris/common/ThreadPoolManagerTest.java | 39 +
.../apache/doris/datasource/CatalogMgrTest.java | 31 +
.../datasource/ExternalCatalogDeadlockTest.java | 8 +
.../ExternalCatalogRuntimeStateTest.java | 118 +++
.../doris/datasource/FileQueryScanNodeTest.java | 29 +
.../doris/datasource/SplitAssignmentTest.java | 26 +
.../hive/HMSExternalCatalogLifecycleTest.java | 98 ++
.../datasource/hive/HMSExternalTableTest.java | 36 +
.../datasource/hudi/HudiExternalMetaCacheTest.java | 213 ++++
.../datasource/hudi/HudiFsViewCacheValueTest.java | 77 ++
.../hudi/source/HudiBatchFsViewOwnerTest.java | 487 +++++++++
.../datasource/hudi/source/HudiScanNodeTest.java | 119 ++-
.../datasource/iceberg/CreateIcebergTableTest.java | 2 +-
.../datasource/iceberg/DorisHadoopCatalogTest.java | 84 ++
.../datasource/iceberg/DorisHiveCatalogTest.java | 94 ++
.../iceberg/IcebergCatalogResourceTrackerTest.java | 79 ++
.../iceberg/IcebergDDLAndDMLPlanTest.java | 7 +
.../iceberg/IcebergExternalMetaCacheTest.java | 718 ++++++++++++-
.../IcebergExternalTableBranchAndTagTest.java | 7 +
.../iceberg/IcebergExternalTableTest.java | 48 +-
.../datasource/iceberg/IcebergMetadataOpTest.java | 255 ++++-
.../iceberg/IcebergMetadataOpsValidationTest.java | 146 ++-
.../iceberg/IcebergSchemaDisplayTest.java | 45 +
.../iceberg/IcebergSysExternalTableTest.java | 83 +-
.../iceberg/IcebergTableCacheValueTest.java | 380 +++++++
.../datasource/iceberg/IcebergTransactionTest.java | 187 ++--
.../doris/datasource/iceberg/IcebergUtilsTest.java | 83 +-
.../iceberg/IcebergWriteSchemaContextTest.java | 68 +-
.../iceberg/action/BaseIcebergActionTest.java | 116 +++
.../rewrite/RewriteDataFileExecutorTest.java | 169 ++-
.../iceberg/rewrite/RewriteGroupTaskTest.java | 17 +
.../iceberg/source/IcebergScanNodeTest.java | 322 ++++++
.../lance/LanceCatalogLifecycleTest.java | 30 +
.../datasource/metacache/MetaCacheEntryTest.java | 198 +++-
.../apache/doris/nereids/StatementContextTest.java | 24 +
.../functions/VarBinaryCollectionLegalityTest.java | 106 ++
.../literal/TimestampTzLiteralTest.java | 39 +
.../plans/commands/ExecuteActionCommandTest.java | 24 +-
.../commands/insert/HiveInsertExecutorTest.java | 99 ++
.../insert/IcebergInsertFailureLifecycleTest.java | 149 +++
.../plans/commands/insert/InsertUtilsTest.java | 14 +-
.../types/VarBinaryUnsupportedCollectionTest.java | 62 ++
.../stream_load/test_timestamptz_stream_load.out | 8 +-
.../timestamptz/test_cast_timestamptz.out | 6 +
.../timestamptz/test_timestamptz_binary_output.out | 16 +-
.../cloud_p0/cache/ttl/alter_ttl_seconds.groovy | 5 +-
.../cache/ttl/test_ttl_expired_tablet.groovy | 207 ++++
.../test_timestamptz_stream_load.groovy | 2 +
.../timestamptz/test_cast_timestamptz.groovy | 14 +-
.../test_timestamptz_binary_output.groovy | 4 +
.../test_timestamptz_historical_offset.groovy | 53 +
.../test_timestamptz_null_string.groovy | 48 +
.../test_timestamptz_output_boundary.groovy | 71 ++
.../test_remote_doris_variant_select.groovy | 177 ----
...test_ngram_bloomfilter_index_like_escape.groovy | 165 +++
200 files changed, 12568 insertions(+), 2211 deletions(-)
create mode 100644 be/src/format_v2/lance/lance_record_batch_converter.cpp
create mode 100644 be/src/format_v2/lance/lance_record_batch_converter.h
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiFsViewCacheValue.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisHadoopCatalog.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisHiveCatalog.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergCatalogResourceTracker.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRuntimeContext.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayFunctionUtils.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/catalog/EnvStateListenerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalCatalogRuntimeStateTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HMSExternalCatalogLifecycleTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/HudiFsViewCacheValueTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/source/HudiBatchFsViewOwnerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/DorisHadoopCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/DorisHiveCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergCatalogResourceTrackerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergTableCacheValueTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/action/BaseIcebergActionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/VarBinaryCollectionLegalityTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/IcebergInsertFailureLifecycleTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/types/VarBinaryUnsupportedCollectionTest.java
create mode 100644
regression-test/suites/cloud_p0/cache/ttl/test_ttl_expired_tablet.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_historical_offset.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_null_string.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_output_boundary.groovy
delete mode 100644
regression-test/suites/external_table_p0/remote_doris/test_remote_doris_variant_select.groovy
create mode 100644
regression-test/suites/index_p0/test_ngram_bloomfilter_index_like_escape.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
