This is an automated email from the ASF dual-hosted git repository.
englefly pushed a change to branch set-runtime-filter
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 534d89de9be update shape cases
omit 85d65170386 add-license
omit 0549a7cbbe6 setop rf nullAware=true
omit 637b9858865 set legacy rf in RuntimeFilterContextV2
omit 5fd9c4897ec update fix
omit f3b936876a1 fix
omit e0412b520cd setop rf (fe part) (#49399)
omit 0fe6a84210a support rf on set operator: be part
add 5abb4c6d6e6 [fix](nereids) project child output to union output in
correct order after eliminate empty relation (#49257)
add 6d42a93ef62 [improve](function) support lead/lag function input column
as third params (#49381)
add dffbd5108db [Enhancement] Support some math functions (#48476)
add bd157e6a62e [fix](jdbc catalog) ensure initialization before fetching
row count (#49442)
add 6d75dbc87b6 [fix](multi-catalog) Fix bug: "Can not create a Path from
an empty string" (#49382)
add 8f46bf97b21 [fix](spill) Fix `WorkloadGroupManager` in spill (#49373)
add 386dad71893 [Fix](testcase) Change big txn load of auto partition test
to not-primary table (#49435)
add ed87894e525 [chore](ci) pr for dev branch can run cloud_p0 and
performance (#49466)
add 77a2b22ba6e [opt](nereids) remove redundant hash join condition
(#48363)
add 7b6d2e21226 [fix](nereids)canInferNotNullForMarkSlot method get wrong
result if fold constant rule is disabled (#49428)
add 5834faebe73 [Opt](cloud-sc) Clear stop token when `commit_tablet_job`
fails (#49275)
add 3708b43c564 [fix](inverted index) Fix for NullPointerException in
Ordered Sloppy Phrase Query (#49487)
add 3668945816a [fix](cloud) fix misuse of
file_cache_evict_in_advance_batch_bytes (#49336)
add b3b00eb67da [Bug] Fix accidental table deletion during restore job
(#48820)
add d9afcecc4ca [follow-up](nereids) speedup huge InPredicate for tablet
pruning (#49386)
add a61d7c44695 [chore](metrics) Add agent batch task metrics (#49480)
add 44133312747 [chore](task) log the thrift message size if the broken
pipe is occurred (#49492)
add c602ed5f146 [fix](nereids) EliminateGroupByConstant should replace
agg's group by after removing constant group by keys (#49473)
add 79a6dd1ff49 [fix](catalog)when checkpoint,use cacheThreadPool (#49097)
add 58393b6160c [fix](Nereids) fix double literal to string literal cast
problem (#49416)
add a9054a82170 [Bug](udf) should delete local ref when return error msg
(#48609)
add 0653b90f041 [chore](binlog) record the ingest binlog elapsed in each
phase (#49499)
add 6099f0c5ed6 [chore](test) add ut for sql cache (#49389)
add cb56a46289a [feat](deps) add jindofs ubuntu jar (#49502)
add 8b268dcda4d [fix](schema-change) Nested types should only support
enlarging varchar length with light schema change (#49452)
add 63cabdd9259 [Refactor]remove useless code (#49526)
add 5b4b121fbc9 [Fix](Variant) return status if dict key not found when
`sync_tablet_rowsets` (#49417)
add edee8c1b8d7 [fix](struct)Fixed the issue of inserting into a struct
type string literal with one more subfield causing BE coredump (#49485)
add 4b3758899d3 [Chore](runtime-filter) add limitation for
runtime_filter_type (#49404)
add 311e1d39ce2 [chore](information schema) rename routine_load_job system
table to routine_load_jobs (#49515)
add 5ae66acdb29 [enhancement](cloud) set the connection timeout and
request timeout for FE requests to the meta service (#49546)
add 0312b925f8f [fix](system) fix alter system modify hostname with valid
ip exception (#49551)
add 1eafee1cb20 [refactor](spill) Refactor spill dependency (#49554)
add 71d9c934e2b [fix](spill) Fix potential hanging pipeline task due to
spilling (#49555)
add 8660611bedd [fix](jindofs) add jindofs ubuntu jar in build.sh (#49503)
add 4dd3b2ec096 [fix](build) Fix compilation error on MacOS caused by
unsupported include (#49531)
add 0f857f4a465 [fix](mac) Fix Mac compile (#49521)
add 8ccfc9d442b [Feat](nereids) support show tablets from table command
(#49426)
add 89f1c835542 [enhancement](compaction) generate multiple compaction
tasks each round (#49547)
add a7d61489db2 [Enhancement] (nereids) implement
drop/killAnalyzeJobCommand in nereids (#48942)
add cf723a05f7b [Fix](RowStore) fix partial columns as row store (#49542)
add 9d8c32d5af4 [Improve](Variant) only `merge_schema` when `sync_tablets`
or scan in… (#48570)
add 58d0558203d [fix](mow) remove agg cache version for unused rowsets
(#49511)
add 094f2dd02a9 [fix](auth)Prohibit other users from modifying the root
(#48752)
add dc3afaab6bc [Fix](txn) Remove `TabletTxnInfo` if version exists when
publish version (#49564)
add 93c4b6bbe71 [Chore](case) add nonConcurrent for some inject case
(#49577)
add 6cc2d4a3e5c [Improve](TabletSchema) optimize finding TabletIndex
performance in TabletSchema (#48699)
add 97d985c0e16 [opt](connection) add connection num in error msg (#49471)
add 765296fee53 [fix](meta) do not check replica allocation when replay
(#49569)
add 9056b62b3f6 [fix](backup) Save snapshot meta during replay (#49550)
add 910297f598e support rf on set operator: be part
add 416aeda047a setop rf (fe part) (#49399)
add 74f4016dbdf fix
add d2db96cf09a update fix
add 1f0770700e0 set legacy rf in RuntimeFilterContextV2
add ea3ef998f2b setop rf nullAware=true
add 7142f3ea88f add-license
add 7a14543f7e7 update shape cases
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (534d89de9be)
\
N -- N -- N refs/heads/set-runtime-filter (7a14543f7e7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/cloud/cloud_backend_service.cpp | 4 +-
be/src/cloud/cloud_meta_mgr.cpp | 34 +-
be/src/cloud/cloud_meta_mgr.h | 4 +-
be/src/cloud/cloud_schema_change_job.cpp | 24 +-
be/src/cloud/cloud_storage_engine.cpp | 16 +-
be/src/cloud/cloud_storage_engine.h | 2 +-
be/src/cloud/cloud_tablet.cpp | 12 +-
be/src/cloud/cloud_tablet.h | 10 +-
be/src/cloud/cloud_tablet_mgr.cpp | 12 +-
be/src/common/config.cpp | 10 +-
be/src/common/config.h | 8 +
be/src/common/daemon.cpp | 20 +-
be/src/exec/schema_scanner.cpp | 2 +-
.../schema_routine_load_job_scanner.cpp | 2 +-
be/src/http/action/calc_file_crc_action.cpp | 2 +-
be/src/http/action/delete_bitmap_action.cpp | 6 +-
be/src/io/cache/block_file_cache.cpp | 8 +-
be/src/olap/memtable_flush_executor.cpp | 4 +-
.../inverted_index/query/phrase_query.cpp | 4 +-
be/src/olap/storage_engine.cpp | 6 +
be/src/olap/tablet_meta.cpp | 51 +-
be/src/olap/tablet_meta.h | 2 +
be/src/olap/tablet_schema.cpp | 91 +--
be/src/olap/tablet_schema.h | 17 +
be/src/olap/task/engine_publish_version_task.cpp | 3 +
be/src/olap/txn_manager.cpp | 29 +-
be/src/olap/txn_manager.h | 8 +
be/src/pipeline/exec/olap_scan_operator.cpp | 5 +-
be/src/pipeline/exec/operator.h | 5 +
.../exec/partitioned_aggregation_sink_operator.cpp | 2 -
.../partitioned_aggregation_source_operator.cpp | 2 -
.../exec/partitioned_aggregation_source_operator.h | 1 -
.../exec/partitioned_hash_join_probe_operator.cpp | 2 -
.../exec/partitioned_hash_join_probe_operator.h | 2 -
.../exec/partitioned_hash_join_sink_operator.cpp | 2 -
be/src/pipeline/exec/spill_sort_sink_operator.cpp | 2 -
.../pipeline/exec/spill_sort_source_operator.cpp | 2 -
be/src/pipeline/exec/spill_sort_source_operator.h | 2 -
be/src/pipeline/pipeline_task.cpp | 25 +-
be/src/pipeline/pipeline_task.h | 6 -
be/src/runtime/memory/global_memory_arbitrator.cpp | 11 +-
be/src/runtime/memory/global_memory_arbitrator.h | 21 +-
be/src/runtime/memory/mem_tracker_limiter.cpp | 4 +
be/src/runtime/memory/mem_tracker_limiter.h | 5 +
be/src/runtime/memory/memory_profile.cpp | 3 +-
be/src/runtime/memory/thread_mem_tracker_mgr.h | 5 +-
be/src/runtime/thread_context.h | 44 +-
be/src/runtime/workload_group/workload_group.cpp | 10 +
.../workload_group/workload_group_manager.cpp | 169 +++--
.../workload_group/workload_group_manager.h | 2 +-
be/src/service/backend_service.cpp | 29 +-
be/src/service/point_query_executor.cpp | 6 +-
.../aggregate_function_java_udaf.h | 13 +-
.../aggregate_function_window.h | 66 +-
be/src/vec/common/allocator.cpp | 4 +-
.../data_types/serde/data_type_struct_serde.cpp | 6 +-
be/src/vec/exec/scan/scanner_scheduler.cpp | 13 +-
.../exprs/table_function/udf_table_function.cpp | 7 +-
.../vec/functions/function_datetime_floor_ceil.cpp | 2 +
be/src/vec/functions/function_java_udf.cpp | 3 +-
be/src/vec/functions/math.cpp | 28 +
be/test/io/cache/block_file_cache_test.cpp | 106 ++-
.../inverted_index/query/phrase_query_test.cpp | 35 +
be/test/olap/tablet_schema_index_test.cpp | 174 +++++
.../runtime/memory/thread_mem_tracker_mgr_test.cpp | 16 +-
be/test/vec/function/function_math_test.cpp | 60 ++
build.sh | 2 +
.../scripts/create_preinstalled_scripts/run74.hql | 9 +
.../partition_col=,/000000_0 | Bin 0 -> 408 bytes
.../partition_col=a, b, c/000000_0 | Bin 0 -> 408 bytes
.../partition_col=a, b/000000_0 | Bin 0 -> 408 bytes
.../main/java/org/apache/doris/common/Config.java | 12 +
fe/fe-core/pom.xml | 4 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 10 +-
.../apache/doris/alter/CloudSchemaChangeJobV2.java | 23 -
.../apache/doris/alter/SchemaChangeHandler.java | 17 +-
.../org/apache/doris/analysis/AlterUserStmt.java | 8 +
.../doris/analysis/BitmapFilterPredicate.java | 4 +-
.../doris/analysis/ModifyNodeHostNameClause.java | 2 -
.../org/apache/doris/analysis/SchemaTableType.java | 4 +-
.../java/org/apache/doris/backup/BackupJob.java | 12 +-
.../java/org/apache/doris/backup/RestoreJob.java | 12 +-
.../doris/catalog/BuiltinScalarFunctions.java | 8 +
.../apache/doris/catalog/ColocateTableIndex.java | 8 +-
.../java/org/apache/doris/catalog/ColumnType.java | 9 +-
.../main/java/org/apache/doris/catalog/Env.java | 4 +-
.../java/org/apache/doris/catalog/SchemaTable.java | 4 +-
.../org/apache/doris/catalog/TableProperty.java | 3 +
.../apache/doris/cloud/rpc/MetaServiceClient.java | 204 ++++--
.../apache/doris/cloud/rpc/MetaServiceProxy.java | 12 +-
.../doris/common/util/DynamicPartitionUtil.java | 19 +-
.../doris/datasource/ExternalMetaCacheMgr.java | 25 +-
.../doris/datasource/hive/HiveMetaStoreCache.java | 3 +-
.../doris/datasource/jdbc/JdbcExternalTable.java | 1 +
.../datasource/property/PropertyConverter.java | 3 +-
.../org/apache/doris/master/ReportHandler.java | 1 +
.../java/org/apache/doris/metric/MetricRepo.java | 13 +
.../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 1 +
.../org/apache/doris/mysql/AcceptListener.java | 11 +-
.../doris/nereids/jobs/executor/Rewriter.java | 8 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 55 ++
.../processor/post/RuntimeFilterGenerator.java | 6 +-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../rules/analysis/EliminateGroupByConstant.java | 5 +-
.../rules/exploration/join/JoinCommute.java | 2 +-
.../exploration/mv/MaterializedViewUtils.java | 3 +-
.../rewrite/EliminateConstHashJoinCondition.java | 91 +++
.../rules/rewrite/EliminateEmptyRelation.java | 22 +-
.../functions/executable/NumericArithmetic.java | 50 ++
.../functions/scalar/{Ln.java => Acosh.java} | 15 +-
.../functions/scalar/{Exp.java => Asinh.java} | 15 +-
.../functions/scalar/{Ln.java => Atanh.java} | 15 +-
.../functions/scalar/{Exp.java => Sinh.java} | 14 +-
.../trees/expressions/functions/window/Lag.java | 5 +-
.../trees/expressions/functions/window/Lead.java | 5 +-
.../functions/window/WindowFunction.java | 6 +-
.../nereids/trees/expressions/literal/Literal.java | 25 +
.../expressions/visitor/ScalarFunctionVisitor.java | 20 +
.../apache/doris/nereids/trees/plans/PlanType.java | 5 +-
...SyncCommand.java => DropAnalyzeJobCommand.java} | 29 +-
...SyncCommand.java => KillAnalyzeJobCommand.java} | 30 +-
.../{DropCommand.java => KillCommand.java} | 9 +-
.../trees/plans/commands/ShowTabletIdCommand.java | 10 +
.../commands/ShowTabletsFromTableCommand.java | 298 +++++++++
.../info/ModifyFrontendOrBackendHostNameOp.java | 2 -
.../nereids/trees/plans/logical/LogicalJoin.java | 7 +
.../trees/plans/visitor/CommandVisitor.java | 15 +
.../apache/doris/nereids/util/ExpressionUtils.java | 11 +-
.../doris/planner/RuntimeFilterGenerator.java | 8 +-
.../java/org/apache/doris/qe/ConnectScheduler.java | 14 +-
.../apache/doris/qe/RuntimeFilterTypeHelper.java | 18 +
.../java/org/apache/doris/qe/SessionVariable.java | 37 +-
.../sessions/FlightSessionsWithTokenManager.java | 20 +-
.../apache/doris/statistics/AnalysisManager.java | 32 +
.../java/org/apache/doris/task/AgentBatchTask.java | 22 +-
.../joinorder/hypergraph/PullupExpressionTest.java | 10 +-
.../exploration/mv/MaterializedViewUtilsTest.java | 2 +-
.../EliminateConstHashJoinConditionTest.java} | 44 +-
.../rules/rewrite/EliminateEmptyRelationTest.java | 92 +++
.../plans/commands/AlterSystemCommandTest.java | 34 +-
...t.java => ShowTabletsFromTableCommandTest.java} | 137 ++--
.../util/CanInferNotNullForMarkSlotTest.java | 100 +++
.../doris/planner/RuntimeFilterGeneratorTest.java | 711 +++++++++++++++++++++
.../doris/qe/RuntimeFilterTypeHelperTest.java | 51 +-
.../java/org/apache/doris/qe/SqlCacheTest.java | 60 ++
.../java/org/apache/doris/task/AgentTaskTest.java | 2 +
fe/pom.xml | 7 +
gensrc/thrift/Descriptors.thrift | 2 +-
.../data/correctness_p0/test_lag_lead_window.out | Bin 1402 -> 1797 bytes
.../hive/test_hive_partitions.out | Bin 3171 -> 3333 bytes
...> test_cloud_sc_self_retry_with_stop_token.out} | Bin 171 -> 167 bytes
.../data/insert_p0/test_struct_insert.out | Bin 625 -> 4698 bytes
.../nereids_p0/runtime_filter/runtime_filter.out | Bin 868 -> 742 bytes
.../extend_infer_equal_predicate.out | Bin 30317 -> 30195 bytes
.../infer_predicate/pull_up_predicate_literal.out | Bin 82024 -> 78088 bytes
.../infer_predicate/pull_up_predicate_set_op.out | Bin 19703 -> 19652 bytes
.../data/nereids_rules_p0/pkfk/eliminate_inner.out | Bin 7333 -> 6965 bytes
regression-test/data/point_query_p0/load.out | Bin 5193 -> 5239 bytes
.../test_math_unary_always_nullable.out | Bin 1032 -> 1769 bytes
.../schema_change_p0/test_type_length_change.out | Bin 352 -> 362 bytes
regression-test/pipeline/cloud_p0/prepare.sh | 10 +-
regression-test/pipeline/performance/prepare.sh | 20 +-
.../test_root_user.groovy} | 30 +-
.../test_backup_restore_get_snapshot.groovy | 102 +++
.../correctness_p0/test_lag_lead_window.groovy | 39 ++
.../correctness_p0/test_runtime_filter.groovy | 17 +
.../suites/empty_relation/eliminate_empty.groovy | 173 +++++
.../hive/test_hive_partitions.groovy | 6 +-
...est_cloud_sc_self_retry_with_stop_token.groovy} | 53 +-
.../suites/insert_p0/test_struct_insert.groovy | 101 ++-
.../test_routine_load_job_info_system_table.groovy | 2 +-
.../aggregate/aggregate_groupby_constant.groovy | 26 +
.../nereids_p0/cte/test_cte_filter_pushdown.groovy | 2 +
.../fold_constant/fold_constant_cast.groovy | 49 ++
.../fold_constant_numeric_arithmatic.groovy | 48 +-
.../cte/test_cte_filter_pushdown.groovy | 2 +-
.../filter_push_down/push_filter_through.groovy | 2 +-
.../pull_up_predicate_literal.groovy | 2 +-
.../agg_optimize_when_uniform.groovy | 10 +-
.../agg_with_roll_up/aggregate_with_roll_up.groovy | 1 +
.../aggregate_without_roll_up.groovy | 3 +-
.../mv/dimension/dimension_1.groovy | 1 +
.../mv/dimension/dimension_2_full_join.groovy | 1 +
.../mv/dimension/dimension_2_inner_join.groovy | 1 +
.../mv/dimension/dimension_2_left_join.groovy | 1 +
.../mv/dimension/dimension_2_right_join.groovy | 1 +
.../mv/join/dphyp_inner/inner_join_dphyp.groovy | 1 +
.../mv/join/dphyp_outer/outer_join_dphyp.groovy | 2 +
.../mv/join/inner/inner_join.groovy | 1 +
.../mv/join/left_outer/outer_join.groovy | 1 +
.../inner_join_infer_and_derive.groovy | 1 +
.../mv/nested/nested_materialized_view.groovy | 1 +
.../predicate_infer/infer_predicate.groovy | 2 +-
.../suites/nereids_syntax_p0/runtime_filter.groovy | 40 +-
.../ddl/create_list_part_data_table.sql | 2 +-
.../ddl/create_range_part_data_table.sql | 2 +-
.../ddl/stream_load_list_test_table.sql | 2 +-
.../ddl/stream_load_range_test_table.sql | 2 +-
regression-test/suites/point_query_p0/load.groovy | 58 +-
.../suites/point_query_p0/load_ck.groovy | 6 +-
.../test_inject_send_filter_size_fail.groovy | 2 +-
.../join/test_low_bucket/test_low_bucket.groovy | 1 +
.../join/test_slow_close/test_slow_close.groovy | 2 +-
.../test_math_unary_always_nullable.groovy | 30 +-
.../test_type_length_change.groovy | 8 +-
.../test_varchar_sc_in_complex.groovy | 55 ++
...d_stats.groovy => test_drop_analyze_job.groovy} | 43 +-
thirdparty/CHANGELOG.md | 4 +
thirdparty/vars.sh | 8 +-
209 files changed, 4032 insertions(+), 866 deletions(-)
create mode 100644 be/test/olap/tablet_schema_index_test.cpp
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/partition_tables/string_partition_table_with_comma/partition_col=,/000000_0
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/partition_tables/string_partition_table_with_comma/partition_col=a,
b, c/000000_0
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/partition_tables/string_partition_table_with_comma/partition_col=a,
b/000000_0
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateConstHashJoinCondition.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{Ln.java
=> Acosh.java} (87%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{Exp.java
=> Asinh.java} (86%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{Ln.java
=> Atanh.java} (87%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{Exp.java
=> Sinh.java} (86%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{SyncCommand.java
=> DropAnalyzeJobCommand.java} (70%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{SyncCommand.java
=> KillAnalyzeJobCommand.java} (70%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{DropCommand.java
=> KillCommand.java} (88%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowTabletsFromTableCommand.java
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/{jobs/joinorder/hypergraph/InferPredicateTest.java
=> rules/rewrite/EliminateConstHashJoinConditionTest.java} (52%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateEmptyRelationTest.java
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/{ShowTableStatsCommandTest.java
=> ShowTabletsFromTableCommandTest.java} (54%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/util/CanInferNotNullForMarkSlotTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/planner/RuntimeFilterGeneratorTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/qe/SqlCacheTest.java
copy
regression-test/data/fault_injection_p0/cloud/{test_cloud_partial_update_update_tmp_rowset_fail.out
=> test_cloud_sc_self_retry_with_stop_token.out} (88%)
copy regression-test/suites/{auth_p0/test_drop_materialized_view_auth.groovy
=> account_p0/test_root_user.groovy} (64%)
create mode 100644
regression-test/suites/backup_restore/test_backup_restore_get_snapshot.groovy
copy
regression-test/suites/fault_injection_p0/cloud/{test_cloud_partial_update_update_tmp_rowset_fail.groovy
=> test_cloud_sc_self_retry_with_stop_token.groovy} (52%)
create mode 100644
regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_cast.groovy
copy regression-test/suites/statistics/{test_drop_cached_stats.groovy =>
test_drop_analyze_job.groovy} (53%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
