This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch runtimefilter_multi_send
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 254ec1ba53 [feature](runtime_filter) MultiCastSender supporting
runtime filter on be (#21304)
add 76620c21aa [improvement](nereids) prune hash join output slot ids list
(#20789)
add 283fd2903f [typo](doc)json document optimization (#20753)
add f77c69ab95 [fix](test) case bug, streamload without sync. (#21161)
add 016870b673 [opt](nereids) use Expression's isConstant to check whether
could be remove from group by key (#21195)
add 325504deeb [bugfix](recover) do not need dynamic partition recover
except olap table (#21290)
add 4e082a803f [typo](docs) improvement lakehouse doc sidebar (#21270)
add a6b51ec19a [Feature](avro) Support Apache Avro file format (#19990)
add 274203a59c [typo](storage)Fixed wrong description about
Storage_root_path parameter (#20641)
add e7dd65f551 [fix](test) fix PlannerTest testEliminatingSortNode (#21112)
add 449c8d4568 [fix](jdbc) Handling Zero DateTime Values in Non-nullable
Columns for JDBC Catalog Reading MySQL (#21296)
add 86af533e83 [Enhancement](heartbeat) make heartbeat ok when config
repeated host-ip pairs (#21228)
add 884c908e25 [Enhancement](multi-catalog) try to reuse existed ugi.
(#21274)
add 73bce9e750 [typo](doc) add params description and example for
accessing hdfs in ha mode by tvf #21277
add 9af714bceb [fix](catalog) disble FileSystem Cache to avoid too many fs
cache (#21283)
add 64ffb06a79 [fix](Nereids) olap scan should not be gather since
coordinator chould not process (#21298)
add 54e2e2f7ee [typo](doc)FlinkCDC access to multi-table or whole database
example document mod… (#21295)
add 30b1b93353 [dependency](fe)Dependency version upgrade (#21191)
add f8cfe5e579 [Bug](pipeline) add DCHECK for
_instance_to_sending_by_pipeline = false on _send_rpc (#21169)
add 3f99b91ddf [fix](gc_binlog) Fix tablet gc_binlogs nullptr (#21158)
add 7f0e37069f [improvement](olap) filter the whole segment by dictionary
(#21239)
add 45f1909bc3 [Bug](lateral-view) make lateral view function's nullable
mode work (#21242)
add 3a12b67517 [Improvement](statistics, multi catalog)Implement hive
table statistic connector (#21053)
add 87e64115ae [Chore](materialized-view) add case about insert data
imidiately after create mv(#21281)
add f5668ac1a0 [fix](doc) Fix table typo in star schema benchmark
documentation and join optimization (#19181)
add 16c218fde5 [feature](nereids) support bind external relation out of
Doris fe environment (#21123)
add a518ea5063 [Bug](pipeline) do not call cancelPlanFragmentAsync when
instance finished (#21193)
add 64e9eab0dd [fix](nereids)update Agg stats estimation #21300
add 59198ed59e [improvement](nereids) Support rf into cte (#21114)
new 111160e19d [feature](runtime_filter) MultiCastSender supporting
runtime filter on be (#21304)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (254ec1ba53)
\
N -- N -- N refs/heads/runtimefilter_multi_send (111160e19d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/olap/block_column_predicate.cpp | 15 +
be/src/olap/block_column_predicate.h | 9 +
be/src/olap/column_predicate.h | 4 +
be/src/olap/comparison_predicate.h | 13 +
be/src/olap/in_list_predicate.h | 11 +
be/src/olap/olap_common.h | 1 +
be/src/olap/rowset/segment_v2/column_reader.cpp | 57 +-
be/src/olap/rowset/segment_v2/column_reader.h | 10 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 20 +
be/src/olap/storage_engine.cpp | 6 +-
be/src/pipeline/exec/exchange_sink_buffer.cpp | 7 +-
be/src/runtime/descriptors.cpp | 17 +-
be/src/runtime/descriptors.h | 1 -
be/src/service/internal_service.cpp | 9 +
be/src/util/network_util.cpp | 10 +-
be/src/util/ref_count_closure.h | 2 +-
.../aggregate_functions/aggregate_function_avg.h | 5 +-
.../aggregate_functions/aggregate_function_count.h | 5 +-
.../aggregate_function_min_max.h | 5 +-
.../aggregate_functions/aggregate_function_sum.h | 5 +-
be/src/vec/data_types/data_type_agg_state.h | 4 +-
be/src/vec/exec/jni_connector.cpp | 21 +-
be/src/vec/exec/jni_connector.h | 24 +
be/src/vec/exec/scan/avro_jni_reader.cpp | 165 ++++++
be/src/vec/exec/scan/avro_jni_reader.h | 96 ++++
be/src/vec/exec/scan/new_olap_scan_node.cpp | 1 +
be/src/vec/exec/scan/new_olap_scan_node.h | 1 +
be/src/vec/exec/scan/new_olap_scanner.cpp | 1 +
be/src/vec/exec/scan/vfile_scanner.cpp | 7 +
build.sh | 2 +
conf/be.conf | 10 +-
.../docker-compose/mysql/init/03-create-table.sql | 22 +-
.../docker-compose/mysql/init/04-insert.sql | 3 +
docs/en/docs/admin-manual/config/be-config.md | 6 +
docs/en/docs/benchmark/ssb.md | 8 +-
docs/en/docs/ecosystem/flink-doris-connector.md | 2 +
docs/en/docs/install/standard-deployment.md | 81 ++-
docs/en/docs/lakehouse/{multi-catalog => }/faq.md | 0
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 14 +-
.../docs/lakehouse/multi-catalog/multi-catalog.md | 4 +-
.../join-optimization/doris-join-optimization.md | 2 +-
docs/en/docs/query-acceleration/statistics.md | 3 +-
.../sql-functions/table-functions/hdfs.md | 29 +
.../sql-manual/sql-functions/table-functions/s3.md | 14 +-
.../sql-manual/sql-reference/Data-Types/JSON.md | 2 +-
docs/sidebars.json | 8 +-
docs/zh-CN/docs/admin-manual/config/be-config.md | 6 +
docs/zh-CN/docs/benchmark/ssb.md | 8 +-
docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 2 +
docs/zh-CN/docs/install/standard-deployment.md | 28 +-
.../docs/lakehouse/{multi-catalog => }/faq.md | 0
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 14 +-
.../docs/lakehouse/multi-catalog/multi-catalog.md | 4 +-
.../join-optimization/doris-join-optimization.md | 2 +-
docs/zh-CN/docs/query-acceleration/statistics.md | 3 +-
.../sql-functions/table-functions/hdfs.md | 29 +
.../sql-manual/sql-functions/table-functions/s3.md | 10 +-
.../sql-manual/sql-reference/Data-Types/JSON.md | 2 +-
.../{hudi-scanner => avro-scanner}/pom.xml | 87 +--
.../org/apache/doris/avro/AvroColumnValue.java | 162 ++++++
.../java/org/apache/doris/avro/AvroJNIScanner.java | 247 +++++++++
.../java/org/apache/doris/avro/AvroProperties.java | 40 ++
.../java/org/apache/doris/avro/AvroReader.java | 42 +-
.../java/org/apache/doris/avro/HDFSFileReader.java | 73 +++
.../java/org/apache/doris/avro/S3FileReader.java | 91 ++++
.../src/main/resources/package.xml | 0
fe/be-java-extensions/hudi-scanner/pom.xml | 19 +-
.../java/org/apache/doris/hudi/HudiJniScanner.java | 6 +
fe/be-java-extensions/java-common/pom.xml | 4 +
.../org/apache/doris/common/jni/JniScanner.java | 13 +-
.../apache/doris/common/jni/MockJniScanner.java | 8 +-
.../apache/doris/common/jni/vec/TableSchema.java | 83 +++
.../doris/maxcompute/MaxComputeJniScanner.java | 7 +
.../org/apache/doris/paimon/PaimonJniScanner.java | 7 +
fe/be-java-extensions/pom.xml | 1 +
fe/fe-core/pom.xml | 2 -
fe/fe-core/src/main/cup/sql_parser.cup | 19 +-
.../org/apache/doris/analysis/DropStatsStmt.java | 4 -
.../org/apache/doris/analysis/LateralViewRef.java | 11 +-
.../apache/doris/analysis/ShowColumnStatsStmt.java | 12 +-
.../org/apache/doris/analysis/SlotDescriptor.java | 36 +-
.../org/apache/doris/analysis/TupleDescriptor.java | 17 +-
.../apache/doris/catalog/CatalogRecycleBin.java | 5 +-
.../org/apache/doris/catalog/HdfsResource.java | 1 +
.../main/java/org/apache/doris/catalog/Table.java | 2 +-
.../java/org/apache/doris/catalog/TableIf.java | 2 +-
.../doris/catalog/external/ExternalTable.java | 8 +-
.../doris/catalog/external/HMSExternalTable.java | 140 +++++
.../doris/datasource/hive/HiveMetaStoreCache.java | 19 +
.../property/constants/S3Properties.java | 2 +
.../doris/external/jdbc/JdbcMySQLClient.java | 12 +
.../apache/doris/fs/remote/RemoteFileSystem.java | 3 +-
.../apache/doris/fs/remote/dfs/DFSFileSystem.java | 70 ++-
.../src/main/java/org/apache/doris/load/Load.java | 3 -
.../doris/load/loadv2/LoadingTaskPlanner.java | 3 -
.../org/apache/doris/nereids/CascadesContext.java | 9 +
.../glue/translator/PhysicalPlanTranslator.java | 60 ++-
.../glue/translator/RuntimeFilterTranslator.java | 4 +
.../doris/nereids/jobs/executor/Analyzer.java | 61 ++-
.../processor/post/RuntimeFilterContext.java | 38 ++
.../processor/post/RuntimeFilterGenerator.java | 585 +++++++++++++++++----
.../doris/nereids/rules/analysis/BindRelation.java | 26 +-
.../LogicalOlapScanToPhysicalOlapScan.java | 10 +-
.../rules/rewrite/EliminateGroupByConstant.java | 3 +-
.../doris/nereids/stats/StatsCalculator.java | 3 +-
.../trees/plans/physical/PhysicalCTEConsumer.java | 18 +-
.../org/apache/doris/planner/HashJoinNode.java | 21 +-
.../doris/planner/MultiCastPlanFragment.java | 3 +-
.../org/apache/doris/planner/OriginalPlanner.java | 3 -
.../org/apache/doris/planner/PlanFragment.java | 8 +
.../apache/doris/planner/StreamLoadPlanner.java | 6 -
.../main/java/org/apache/doris/qe/Coordinator.java | 11 +-
.../java/org/apache/doris/qe/ShowExecutor.java | 9 +-
.../statistics/ColumnStatisticsCacheLoader.java | 2 +-
.../apache/doris/statistics/HiveAnalysisTask.java | 121 +----
.../ExternalFileTableValuedFunction.java | 3 +
.../doris/tablefunction/S3TableValuedFunction.java | 14 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
.../datasource/property/PropertyConverterTest.java | 4 +-
.../nereids/rules/analysis/BindRelationTest.java | 65 ++-
.../org/apache/doris/nereids/util/PlanChecker.java | 27 +-
.../java/org/apache/doris/planner/PlannerTest.java | 68 +--
.../doris/planner/TableFunctionPlanTest.java | 14 +-
.../org/apache/doris/statistics/CacheTest.java | 2 +-
fe/pom.xml | 66 ++-
gensrc/thrift/PlanNodes.thrift | 1 +
.../test_agg_state_group_concat.out | 0
.../agg_state/max/test_agg_state_max.out | 22 +
.../hive/test_hive_statistic_cache.out | 58 ++
.../jdbc_catalog_p0/test_mysql_jdbc_catalog.out | 4 +
regression-test/data/mv_p0/no_await/no_await.out | 511 ++++++++++++++++++
.../nereids_tpcds_shape_sf100_p0/shape/query24.out | 23 +-
.../nereids_tpcds_shape_sf100_p0/shape/query31.out | 46 +-
.../test_agg_state_group_concat.groovy | 0
.../agg_state/max/test_agg_state_max.groovy | 62 +++
.../suites/datatype_p0/decimalv3/test_load.groovy | 4 +-
.../hive/test_hive_statistic_cache.groovy | 58 ++
.../suites/github_events_p2/load.groovy | 1 +
.../jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy | 4 +-
.../test_map_load_and_compaction.groovy | 2 +
.../suites/mv_p0/no_await/no_await.groovy | 142 +++++
.../suites/nereids_syntax_p0/agg_with_const.groovy | 5 +
.../aggregate/push_filter_through_agg.groovy | 3 +-
143 files changed, 3576 insertions(+), 746 deletions(-)
create mode 100644 be/src/vec/exec/scan/avro_jni_reader.cpp
create mode 100644 be/src/vec/exec/scan/avro_jni_reader.h
rename docs/en/docs/lakehouse/{multi-catalog => }/faq.md (100%)
rename docs/zh-CN/docs/lakehouse/{multi-catalog => }/faq.md (100%)
copy fe/be-java-extensions/{hudi-scanner => avro-scanner}/pom.xml (51%)
create mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroColumnValue.java
create mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java
create mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroProperties.java
copy be/src/util/ref_count_closure.h =>
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroReader.java
(51%)
create mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/HDFSFileReader.java
create mode 100644
fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/S3FileReader.java
copy fe/be-java-extensions/{hudi-scanner =>
avro-scanner}/src/main/resources/package.xml (100%)
create mode 100644
fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/vec/TableSchema.java
rename regression-test/data/datatype_p0/agg_state/{ =>
group_concat}/test_agg_state_group_concat.out (100%)
create mode 100644
regression-test/data/datatype_p0/agg_state/max/test_agg_state_max.out
create mode 100644
regression-test/data/external_table_emr_p2/hive/test_hive_statistic_cache.out
create mode 100644 regression-test/data/mv_p0/no_await/no_await.out
rename regression-test/suites/datatype_p0/agg_state/{ =>
group_concat}/test_agg_state_group_concat.groovy (100%)
create mode 100644
regression-test/suites/datatype_p0/agg_state/max/test_agg_state_max.groovy
create mode 100644
regression-test/suites/external_table_emr_p2/hive/test_hive_statistic_cache.groovy
create mode 100644 regression-test/suites/mv_p0/no_await/no_await.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
