This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/proto-33bc8ff79f
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit 59d59f8d79 chore(deps): bump pbjson-types from 0.8.0 to 0.9.0 in the
proto group
add 11ef486e6c Runs-on for extended CI checks (#20511)
add 3aa34b33f5 chore(deps): bump strum from 0.27.2 to 0.28.0 (#20520)
add 4c0a6531ca chore(deps): bump taiki-e/install-action from 2.68.6 to
2.68.8 (#20518)
add 6c793694e9 chore(deps): bump the all-other-cargo-deps group with 2
updates (#20519)
add 0dfa542201 fix: HashJoin panic with dictionary-encoded columns in
multi-key joins (#20441)
add 4a41587bdf Make `custom_file_casts` example schema nullable to allow
null `id` values during casting (#20486)
add a9c090141d Add support for FFI config extensions (#19469)
add 9c85ac608f perf: Fix quadratic behavior of `to_array_of_size` (#20459)
add 17d770d6e5 fix: handle out of range errors in DATE_BIN instead of
panicking (#20221)
add 670dbf481c fix: prevent duplicate alias collision with user-provided
__datafusion_extracted names (#20432)
add e71e7a39bf chore: Cleanup code to use `repeat_n` in a few places
(#20527)
add 932418b20c chore(deps): bump strum_macros from 0.27.2 to 0.28.0
(#20521)
add db5197b742 chore: Replace `matches!` on fieldless enums with `==`
(#20525)
add b16ad9badc fix: SortMergeJoin don't wait for all input before emitting
(#20482)
add fdd36d0d21 Update comments on OptimizerRule about function name
matching (#20346)
add e80694e369 Remove recursive const check in `simplify_const_expr`
(#20234)
add b8cebdde2a Fix incorrect regex pattern in regex_replace_posix_groups
(#19827)
add 34dad2ccee Cache `PlanProperties`, add fast-path for
`with_new_children` (#19792)
add 585bbf35d3 perf: Optimize `array_has_any()` with scalar arg (#20385)
add 387e20cc58 Improve `HashJoinExecBuilder` to save state from previous
fields (#20276)
add 2347306943 [Minor] Fix error messages for `shrink` and `try_shrink`
(#20422)
add d75fcb83e3 Fix physical expr adapter to resolve physical fields by
name, not column index (#20485)
add e937cadbcc [fix] Add type coercion from NULL to Interval to make
date_bin more postgres compatible (#20499)
add d7d646164d feat: Implement Spark `bin` function (#20479)
add e6849945bf fix: `cardinality()` of an empty array should be zero
(#20533)
add e894a03bea perf: Use Hashbrown for array_distinct (#20538)
add 3a970c58ff Clamp early aggregation emit to the sort boundary when
using partial group ordering (#20446)
add 33b86fe02e perf: Cache num_output_rows in sort merge join to avoid
O(n) recount (#20478)
add bcd42b090a fix: Unaccounted spill sort in row_hash (#20314)
add a026e7da2f perf: Optimize heap handling in TopK operator (#20556)
add d6fb3608b0 perf: Optimize `array_position` for scalar needle (#20532)
add e76f0eebe3 fix: IS NULL panic with invalid function without input
arguments (#20306)
add 3ab1301c53 fix: handle empty delimiter in split_part (closes #20503)
(#20542)
add a257c29c26 add redirect for old upgrading.html URL to fix broken
changelog links (#20582)
add a79e6e6b39 fix(substrait): Correctly parse field references in
subqueries (#20439)
add bc600b3090 Split `push_down_filter.slt` into standalone sqllogictest
files to reduce long-tail runtime (#20566)
add e583fe9d22 Add deterministic per-file timing summary to sqllogictest
runner (#20569)
add c63ca330d7 fix: increase ROUND decimal precision to prevent overflow
truncation (#19926)
add 7ef62b988d chore: Enable workspace lint for all workspace members
(#20577)
add 451c79fc00 fix: Fix `array_to_string` with columnar third arg (#20536)
add e567cb91f4 Fix serde of window lead/lag defaults (#20608)
add 5d8249ff16 fix: Fix and Refactor Spark `shuffle` function (#20484)
add acec058cb5 perf: Use Arrow vectorized eq kernel for IN list with
column references (#20528)
add 73fbd48070 Upgrade DataFusion to arrow-rs/parquet 58.0.0 /
`object_store` 0.13.0 (#19728)
add 3a23bb2531 perf: Optimize `array_agg()` using `GroupsAccumulator`
(#20504)
add 8df75c3f04 Document guidance on how to evaluate breaking API changes
(#20584)
add 8482e2e448 feat: support extension planner for `TableScan` (#20548)
add 6713439497 perf: Optimize `array_to_string()`, support more types
(#20553)
add 95de1bf410 Add metrics for parquet sink (#20307)
add 1f37a33ce5 Update DataFusion meetups page on docs (#20629)
add 93d177db4e Extend dynamic filter to joins that preserve probe side ON
(#20447)
add 0af9ff5998 Improve sqllogicteset speed by creating only a single large
file rather than 2 (#20586)
add 43584caf11 cli: Fix datafusion-cli hint edge cases (#20609)
add 02dae77420 Speedup sqllogictests by running long running tests first
(#20576)
add a5f490e4c3 Add `ExecutionPlan::apply_expressions()` (#20337)
add 12314c5564 perf: Optimize `array_to_string` to avoid a copy (#20639)
add 0bf376735e fix: make the `sql` feature truly optional (#20625)
add 657887dca9 Fix custom metric display (#20643)
new 843f5a86d6 chore(deps): bump pbjson-types from 0.8.0 to 0.9.0 in the
proto group
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (59d59f8d79)
\
N -- N -- N refs/heads/dependabot/cargo/main/proto-33bc8ff79f
(843f5a86d6)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/audit.yml | 2 +-
.github/workflows/extended.yml | 22 +-
.github/workflows/rust.yml | 4 +-
Cargo.lock | 204 +++---
Cargo.toml | 23 +-
datafusion-cli/Cargo.toml | 3 +
datafusion-cli/src/exec.rs | 11 +-
datafusion-cli/src/functions.rs | 8 +-
datafusion-cli/src/helper.rs | 33 +-
datafusion-cli/src/highlighter.rs | 3 +-
datafusion-cli/src/main.rs | 12 +-
datafusion-cli/src/object_storage.rs | 17 +-
datafusion-cli/src/object_storage/instrumented.rs | 128 ++--
datafusion-cli/src/print_format.rs | 20 +-
datafusion-cli/src/print_options.rs | 16 +-
.../custom_data_source/adapter_serialization.rs | 2 +-
.../examples/custom_data_source/csv_json_opener.rs | 2 +-
.../custom_data_source/custom_datasource.rs | 23 +-
.../custom_data_source/custom_file_casts.rs | 6 +-
.../custom_data_source/default_column_values.rs | 4 +-
.../examples/data_io/json_shredding.rs | 4 +-
.../examples/data_io/parquet_advanced_index.rs | 8 +-
.../memory_pool_execution_plan.rs | 21 +-
.../ffi/ffi_example_table_provider/Cargo.toml | 3 +
.../examples/ffi/ffi_module_interface/Cargo.toml | 3 +
.../examples/ffi/ffi_module_loader/Cargo.toml | 3 +
.../examples/proto/composed_extension_codec.rs | 23 +-
.../examples/relation_planner/table_sample.rs | 24 +-
datafusion/catalog/src/memory/table.rs | 21 +-
datafusion/common/Cargo.toml | 4 +
datafusion/common/benches/scalar_to_array.rs | 107 +++
datafusion/common/src/alias.rs | 10 +
datafusion/common/src/config.rs | 35 +-
datafusion/common/src/file_options/mod.rs | 2 +-
.../common/src/file_options/parquet_writer.rs | 8 +-
datafusion/common/src/join_type.rs | 29 +
datafusion/common/src/scalar/mod.rs | 96 ++-
datafusion/common/src/stats.rs | 2 +-
datafusion/common/src/types/native.rs | 4 +-
datafusion/core/Cargo.toml | 4 +-
datafusion/core/benches/aggregate_query_sql.rs | 11 +
datafusion/core/benches/data_utils/mod.rs | 18 +-
datafusion/core/benches/parquet_query_sql.rs | 4 +-
datafusion/core/benches/parquet_struct_query.rs | 4 +-
datafusion/core/benches/push_down_filter.rs | 2 +-
datafusion/core/benches/reset_plan_states.rs | 2 +
datafusion/core/benches/spm.rs | 5 +-
datafusion/core/benches/sql_query_with_io.rs | 2 +-
datafusion/core/src/dataframe/parquet.rs | 133 ++++
datafusion/core/src/datasource/file_format/csv.rs | 29 +-
.../core/src/datasource/file_format/parquet.rs | 38 +-
.../core/src/datasource/physical_plan/csv.rs | 2 +-
datafusion/core/src/physical_planner.rs | 316 +++++++--
datafusion/core/src/test/object_store.rs | 65 +-
.../tests/catalog_listing/pruned_partition_list.rs | 2 +-
datafusion/core/tests/custom_sources_cases/mod.rs | 26 +-
.../provider_filter_pushdown.rs | 26 +-
.../core/tests/custom_sources_cases/statistics.rs | 23 +-
datafusion/core/tests/dataframe/mod.rs | 9 +-
.../core/tests/datasource/object_store_access.rs | 113 +---
datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs | 2 +-
datafusion/core/tests/fuzz_cases/once_exec.rs | 23 +-
datafusion/core/tests/fuzz_cases/pruning.rs | 4 +-
.../spilling_fuzz_in_memory_constrained_env.rs | 16 +-
.../core/tests/fuzz_cases/topk_filter_pushdown.rs | 2 +-
datafusion/core/tests/fuzz_cases/window_fuzz.rs | 2 +-
datafusion/core/tests/parquet/custom_reader.rs | 2 +-
datafusion/core/tests/parquet/expr_adapter.rs | 2 +-
.../core/tests/parquet/external_access_plan.rs | 2 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 4 +-
datafusion/core/tests/parquet/mod.rs | 2 +-
.../physical_optimizer/enforce_distribution.rs | 24 +-
.../tests/physical_optimizer/filter_pushdown.rs | 375 +++++++++++
.../tests/physical_optimizer/join_selection.rs | 41 +-
.../tests/physical_optimizer/pushdown_utils.rs | 31 +-
.../replace_with_order_preserving_variants.rs | 2 +-
.../core/tests/physical_optimizer/test_utils.rs | 48 +-
datafusion/core/tests/sql/path_partition.rs | 65 +-
.../core/tests/tracing/traceable_object_store.rs | 34 +-
.../core/tests/user_defined/insert_operation.rs | 37 +-
.../core/tests/user_defined/user_defined_plan.rs | 30 +-
datafusion/datasource-arrow/src/file_format.rs | 3 +-
datafusion/datasource-arrow/src/source.rs | 17 +-
datafusion/datasource-avro/src/file_format.rs | 2 +-
datafusion/datasource-avro/src/source.rs | 17 +-
datafusion/datasource-csv/src/file_format.rs | 4 +-
datafusion/datasource-csv/src/source.rs | 15 +
datafusion/datasource-json/src/file_format.rs | 2 +-
datafusion/datasource-json/src/source.rs | 17 +-
.../benches/parquet_nested_filter_pushdown.rs | 12 +-
datafusion/datasource-parquet/src/file_format.rs | 45 +-
datafusion/datasource-parquet/src/opener.rs | 11 +-
.../datasource-parquet/src/row_group_filter.rs | 3 +-
datafusion/datasource-parquet/src/source.rs | 21 +
datafusion/datasource/src/file.rs | 22 +
datafusion/datasource/src/file_scan_config.rs | 19 +
datafusion/datasource/src/memory.rs | 15 +
datafusion/datasource/src/mod.rs | 2 +-
datafusion/datasource/src/sink.rs | 22 +-
datafusion/datasource/src/source.rs | 49 +-
datafusion/datasource/src/test_util.rs | 9 +-
datafusion/datasource/src/url.rs | 42 +-
datafusion/execution/src/memory_pool/mod.rs | 7 +-
datafusion/expr-common/src/signature.rs | 2 +-
datafusion/expr-common/src/type_coercion/binary.rs | 2 +-
datafusion/expr/src/logical_plan/plan.rs | 2 +-
datafusion/expr/src/type_coercion/functions.rs | 2 +-
datafusion/ffi/src/config/extension_options.rs | 288 ++++++++
datafusion/ffi/src/config/mod.rs | 169 +++++
datafusion/ffi/src/execution_plan.rs | 53 +-
datafusion/ffi/src/lib.rs | 1 +
datafusion/ffi/src/session/config.rs | 39 +-
datafusion/ffi/src/tests/async_provider.rs | 25 +-
datafusion/ffi/src/tests/config.rs | 51 ++
datafusion/ffi/src/tests/mod.rs | 6 +
datafusion/ffi/tests/ffi_config.rs | 124 ++++
.../src/aggregate/groups_accumulator/nulls.rs | 2 +-
datafusion/functions-aggregate/src/array_agg.rs | 718 +++++++++++++++++++-
datafusion/functions-aggregate/src/regr.rs | 2 +-
datafusion/functions-nested/Cargo.toml | 10 +
datafusion/functions-nested/benches/array_has.rs | 273 +++++++-
.../functions-nested/benches/array_position.rs | 237 +++++++
.../functions-nested/benches/array_to_string.rs | 188 ++++++
datafusion/functions-nested/src/array_has.rs | 252 ++++++-
datafusion/functions-nested/src/cardinality.rs | 13 +-
datafusion/functions-nested/src/position.rs | 163 ++++-
datafusion/functions-nested/src/range.rs | 25 +-
datafusion/functions-nested/src/set_ops.rs | 2 +-
datafusion/functions-nested/src/string.rs | 450 +++++++------
datafusion/functions-window/src/nth_value.rs | 2 +-
datafusion/functions/src/core/getfield.rs | 2 +-
datafusion/functions/src/datetime/date_bin.rs | 291 +++++---
datafusion/functions/src/math/round.rs | 472 +++++++++++--
datafusion/functions/src/regex/regexpreplace.rs | 48 +-
datafusion/functions/src/string/split_part.rs | 130 +++-
datafusion/optimizer/src/analyzer/type_coercion.rs | 4 -
.../src/decorrelate_predicate_subquery.rs | 2 +-
.../optimizer/src/extract_leaf_expressions.rs | 41 +-
datafusion/optimizer/src/optimizer.rs | 46 +-
datafusion/optimizer/src/push_down_filter.rs | 24 +-
datafusion/physical-expr-adapter/Cargo.toml | 3 +
.../physical-expr-adapter/src/schema_rewriter.rs | 83 ++-
.../physical-expr-common/src/metrics/value.rs | 10 +-
.../src/equivalence/properties/mod.rs | 7 +-
datafusion/physical-expr/src/expressions/case.rs | 2 +-
.../physical-expr/src/expressions/in_list.rs | 245 ++++++-
datafusion/physical-expr/src/partitioning.rs | 2 +-
.../src/simplifier/const_evaluator.rs | 102 ++-
datafusion/physical-expr/src/simplifier/mod.rs | 9 +-
datafusion/physical-expr/src/simplifier/not.rs | 4 +
.../src/combine_partial_final_agg.rs | 2 +-
.../physical-optimizer/src/enforce_distribution.rs | 85 +--
datafusion/physical-optimizer/src/ensure_coop.rs | 16 +-
.../physical-optimizer/src/join_selection.rs | 11 +-
.../physical-optimizer/src/output_requirements.rs | 40 +-
datafusion/physical-plan/src/aggregates/mod.rs | 214 +++++-
.../physical-plan/src/aggregates/row_hash.rs | 171 ++++-
datafusion/physical-plan/src/analyze.rs | 15 +-
datafusion/physical-plan/src/async_func.rs | 37 +-
datafusion/physical-plan/src/buffer.rs | 31 +-
datafusion/physical-plan/src/coalesce_batches.rs | 33 +-
.../physical-plan/src/coalesce_partitions.rs | 34 +-
datafusion/physical-plan/src/coop.rs | 32 +-
datafusion/physical-plan/src/display.rs | 14 +-
datafusion/physical-plan/src/empty.rs | 18 +-
datafusion/physical-plan/src/execution_plan.rs | 270 +++++++-
datafusion/physical-plan/src/explain.rs | 16 +-
datafusion/physical-plan/src/filter.rs | 39 +-
datafusion/physical-plan/src/joins/cross_join.rs | 38 +-
.../physical-plan/src/joins/hash_join/exec.rs | 430 ++++++------
.../src/joins/hash_join/inlist_builder.rs | 58 +-
.../physical-plan/src/joins/nested_loop_join.rs | 48 +-
.../src/joins/piecewise_merge_join/classic_join.rs | 2 +-
.../src/joins/piecewise_merge_join/exec.rs | 99 ++-
.../src/joins/sort_merge_join/exec.rs | 45 +-
.../src/joins/sort_merge_join/stream.rs | 81 ++-
.../src/joins/sort_merge_join/tests.rs | 453 ++++++++++++-
.../physical-plan/src/joins/symmetric_hash_join.rs | 40 +-
datafusion/physical-plan/src/joins/utils.rs | 2 +-
datafusion/physical-plan/src/limit.rs | 78 ++-
datafusion/physical-plan/src/memory.rs | 29 +-
datafusion/physical-plan/src/placeholder_row.rs | 17 +-
datafusion/physical-plan/src/projection.rs | 31 +-
datafusion/physical-plan/src/recursive_query.rs | 15 +-
datafusion/physical-plan/src/repartition/mod.rs | 48 +-
datafusion/physical-plan/src/sorts/mod.rs | 2 +
datafusion/physical-plan/src/sorts/partial_sort.rs | 41 +-
datafusion/physical-plan/src/sorts/sort.rs | 129 ++--
.../src/sorts/sort_preserving_merge.rs | 55 +-
datafusion/physical-plan/src/sorts/stream.rs | 164 ++++-
datafusion/physical-plan/src/spill/mod.rs | 14 +-
.../physical-plan/src/spill/spill_manager.rs | 51 +-
datafusion/physical-plan/src/streaming.rs | 17 +-
datafusion/physical-plan/src/test.rs | 33 +-
datafusion/physical-plan/src/test/exec.rs | 194 +++++-
datafusion/physical-plan/src/topk/mod.rs | 44 +-
datafusion/physical-plan/src/union.rs | 56 +-
datafusion/physical-plan/src/unnest.rs | 32 +-
.../src/windows/bounded_window_agg_exec.rs | 34 +-
datafusion/physical-plan/src/windows/mod.rs | 12 +
.../physical-plan/src/windows/window_agg_exec.rs | 38 +-
datafusion/physical-plan/src/work_table.rs | 18 +-
datafusion/proto/src/physical_plan/to_proto.rs | 3 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 45 ++
datafusion/pruning/src/pruning_predicate.rs | 2 +-
datafusion/spark/src/function/array/shuffle.rs | 16 +-
datafusion/spark/src/function/datetime/last_day.rs | 6 +-
datafusion/spark/src/function/datetime/next_day.rs | 2 +-
.../spark/src/function/map/map_from_arrays.rs | 4 +-
datafusion/spark/src/function/map/utils.rs | 4 +-
datafusion/spark/src/function/math/bin.rs | 109 +++
datafusion/spark/src/function/math/mod.rs | 8 +
datafusion/spark/src/function/math/width_bucket.rs | 4 +-
.../spark/src/function/string/format_string.rs | 2 +-
datafusion/sql/src/expr/function.rs | 2 +-
datafusion/sql/src/planner.rs | 8 +-
datafusion/sqllogictest/README.md | 30 +
datafusion/sqllogictest/bin/sqllogictests.rs | 287 +++++++-
.../test_files/aggregate_skip_partial.slt | 41 +-
datafusion/sqllogictest/test_files/array.slt | 228 ++++++-
datafusion/sqllogictest/test_files/copy.slt | 11 +
.../sqllogictest/test_files/date_bin_errors.slt | 60 ++
.../test_files/datetime/timestamps.slt | 12 +
datafusion/sqllogictest/test_files/decimal.slt | 2 +-
.../test_files/dynamic_filter_pushdown_config.slt | 101 ++-
.../sqllogictest/test_files/explain_tree.slt | 14 +-
datafusion/sqllogictest/test_files/expr.slt | 20 +
datafusion/sqllogictest/test_files/joins.slt | 28 +
datafusion/sqllogictest/test_files/limit.slt | 8 +-
.../test_files/projection_pushdown.slt | 43 +-
.../sqllogictest/test_files/push_down_filter.slt | 745 ---------------------
.../test_files/push_down_filter_outer_joins.slt | 264 ++++++++
.../test_files/push_down_filter_parquet.slt | 188 ++++++
.../test_files/push_down_filter_regression.slt | 193 ++++++
.../test_files/push_down_filter_unnest.slt | 148 ++++
datafusion/sqllogictest/test_files/scalar.slt | 68 +-
.../test_files/spark/array/shuffle.slt | 10 +
.../test_files/spark/datetime/make_interval.slt | 10 +-
.../sqllogictest/test_files/spark/math/bin.slt | 79 ++-
.../sqllogictest/test_files/type_coercion.slt | 21 +
datafusion/sqllogictest/test_files/window.slt | 8 +-
.../logical_plan/consumer/expr/field_reference.rs | 53 +-
.../src/logical_plan/consumer/expr/mod.rs | 5 +-
.../src/logical_plan/consumer/expr/subquery.rs | 44 +-
.../substrait/src/logical_plan/consumer/plan.rs | 5 +-
.../src/logical_plan/consumer/rel/exchange_rel.rs | 3 +-
.../logical_plan/consumer/substrait_consumer.rs | 123 +++-
.../src/logical_plan/producer/expr/cast.rs | 2 +-
.../substrait/tests/cases/consumer_integration.rs | 78 ++-
.../nested_correlated_subquery.substrait.json | 265 ++++++++
datafusion/wasmtest/Cargo.toml | 2 +-
datafusion/wasmtest/src/lib.rs | 2 +-
docs/source/conf.py | 1 +
docs/source/contributor-guide/api-health.md | 71 +-
.../library-user-guide/custom-table-providers.md | 36 +-
docs/source/library-user-guide/upgrading/53.0.0.md | 74 ++
docs/source/library-user-guide/upgrading/54.0.0.md | 93 +++
docs/source/library-user-guide/upgrading/index.rst | 1 +
docs/source/user-guide/concepts-readings-events.md | 7 +
docs/source/user-guide/sql/scalar_functions.md | 14 +-
260 files changed, 11840 insertions(+), 2915 deletions(-)
create mode 100644 datafusion/common/benches/scalar_to_array.rs
create mode 100644 datafusion/ffi/src/config/extension_options.rs
create mode 100644 datafusion/ffi/src/config/mod.rs
create mode 100644 datafusion/ffi/src/tests/config.rs
create mode 100644 datafusion/ffi/tests/ffi_config.rs
create mode 100644 datafusion/functions-nested/benches/array_position.rs
create mode 100644 datafusion/functions-nested/benches/array_to_string.rs
create mode 100644 datafusion/spark/src/function/math/bin.rs
create mode 100644 datafusion/sqllogictest/test_files/date_bin_errors.slt
delete mode 100644 datafusion/sqllogictest/test_files/push_down_filter.slt
create mode 100644
datafusion/sqllogictest/test_files/push_down_filter_outer_joins.slt
create mode 100644
datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
create mode 100644
datafusion/sqllogictest/test_files/push_down_filter_regression.slt
create mode 100644
datafusion/sqllogictest/test_files/push_down_filter_unnest.slt
create mode 100644
datafusion/substrait/tests/testdata/test_plans/nested_correlated_subquery.substrait.json
create mode 100644 docs/source/library-user-guide/upgrading/54.0.0.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
