This is an automated email from the ASF dual-hosted git repository.
blaginin pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-sandbox.git
from 4a9f34d1d Try to fix
add 8f9081483 refactor: update cmp and nested data in binary operator
(#18256)
add c1965b662 Fix: topk_aggregate benchmark failing (#18502)
add c728d5481 refactor: Add `assert_or_internal_err!` macro for more
ergonomic internal invariant checks (#18511)
add 8e3f157ce chore: enforce clippy lint needless_pass_by_value to
datafusion-physical-optimizer (#18555)
add 2ae3818f3 chore: enforce clippy lint needless_pass_by_value for
datafusion-sql (#18554)
add f10fcbecc chore: enforce clippy lint needless_pass_by_value to
physical-expr-common (#18556)
add a216d4aee chore: Enforce lint rule `clippy::needless_pass_by_value` to
`datafusion-physical-expr` (#18557)
add 6ab4d216b Force `FileSource` to be constructed with a `Schema` (#18386)
add 49782435f Fix out-of-bounds access in SLT runner (#18562)
add a6fab6bbe feat: support complex expr for prepared statement argument
(#18383)
add 0e6be3045 feat: Implement
`SessionState::create_logical_expr_from_sql_expr` (#18423)
add 1d8bc9b0e [main] Update version to 51.0.0, add Changelog (#18551)
(#18565)
add 812bb357c Make array_reverse faster for List and FixedSizeList (#18500)
add b131cac1b Consolidate custom data source examples (#18142) (#18553)
add 28755b1d7 Normalize partitioned and flat object listing (#18146)
add f162fd325 refactor: include metric output_batches into BaselineMetrics
(#18491)
add 1586cabeb feat: added clippy::needless_pass_by_value lint rule to
datafusion/expr (#18532)
add d8845a616 feat: support nested key for get_field (#18394)
add 900ee658d Support Arrow IPC Stream Files (#18457)
add 1988ffc61 chore(deps): bump taiki-e/install-action from 2.62.47 to
2.62.49 (#18581)
add dc000b89e chore(deps): bump maturin from 1.9.6 to 1.10.0 in /docs
(#18590)
add e4066898b chore: Remove unused `tokio` dependency and clippy (#18598)
add 3fcbd023a minor: enforce `clippy::needless_pass_by_value` for crates
that don't require code changes. (#18586)
add 552dbe4f1 fix: Eliminate consecutive repartitions (#18521)
add 9029ff183 refactor: merge CoalesceAsyncExecInput into CoalesceBatches
(#18540)
add 0c0b5fe7d Update release download links on download page (#18550)
add 6d9ab4539 Enhance the help message for invalid command in
datafusion-cli (#18603)
add d57e215f9 Update Release README.md with latest process (#18549)
add cbf3f50fb Add timezone to date_trunc fast path (#18596)
add 65bd13daa Coalesce batches inside FilterExec (#18604)
add a3a5ca4fb Fix misleading boolean 'null' interval tests (#18620)
add 206656813 Clarify tests for `Interval::and`, `Interval::not`, and add
`Interval::or` tests (#18621)
add a8f0d59de bugfix: correct regression on TableType for into_view
(#18617)
add 03d197464 Separating Benchmarks for physical sorted union over large
columns in SQL planner based on Datatype (#18599)
add cb1f01966 Add RunEndEncoded type coercion (#18561)
add fb765c234 minor: refactor with `assert_or_internal_err!()` in
`datafusion/sql` (#18614)
add a94ec8fc1 docs: fix rustup cmd for adding rust-analyzer (#18605)
add 0d52a1eeb chore: ASF tracking process on `.asf.yaml` (#18636)
add becc71be0 feat: Add `ansi` enable parameter for execution config
(#18635)
add 76b4156af feat: Add evaluate_to_arrays function (#18446)
add aa74ed086 Refactor bit aggregate functions signature (#18593)
add 92a3a3302 fix: `with_param_values` on `LogicalPlan::EmptyRelation`
returns incorrect schema (#18286)
add 62f5cd6ac feat: support named variables & defaults for `CREATE
FUNCTION` (#18450)
add 65f369dcf chore(deps): bump taiki-e/install-action from 2.62.49 to
2.62.50 (#18645)
add 30eaff315 bugfix: select_columns should validate column names (#18623)
add 8226ebf2e Consolidate data io examples (#18142) (#18591)
add 6af5c954b feat: Add new() constructor for CachedParquetFileReader
(#18575)
add e661b33ee Correct implementations of `NullableInterval::and` and
`NullableInterval::or`. (#18625)
add cf0357478 chore: ASF tracking process on `.asf.yaml` (#18652)
add d8c63eef7 feat: support decimal for math functions: power (#18032)
add 2a6f3aafc perf: Improve NLJ for very small right side case (#17562)
add dae99acc3 feat: selectivity metrics (for Explain Analyze) in Hash Join
(#18488)
add 35b536377 Refactor Spark bitshift signature (#18649)
add 8ef884d3c Change default of
`AggregateUDFImpl::supports_null_handling_clause` to `false` (#18441)
add ae9e1dc0f chore(deps): bump crate-ci/typos from 1.39.0 to 1.39.1
(#18667)
add 5053c71af Update docs for aggregate repartition test (#18650)
add 3e2972ae1 chore: Enforce lint rule `clippy::needless_pass_by_value` to
`datafusion-catalog` (#18638)
add e29009fbc [main] Update Changelog (#18592) (#18616)
add e42a0b626 Refactor distinct aggregate implementations to use common
buffer (#18348)
add 377c0fce4 chore: enforce lint rule `clippy::needless_pass_by_value` to
`datafusion-datasource-avro` (#18641)
add 37dbf9ebe Refactor Spark expm1 signature (#18655)
add b477816c8 Enforce explicit opt-in for `WITHIN GROUP` syntax in
aggregate UDAFs (#18607)
add a9ad712d1 chore(core): Enforce lint rule
`clippy::needless_pass_by_value` to `datafusion-core` (#18640)
add 6685bbe50 Refactor substr signature (#18653)
add d88a3b20f minor: Use allow->expect to explicitly suppress Clippy lint
checks (#18686)
add 3bc82adbe chore(deps): bump taiki-e/install-action from 2.62.50 to
2.62.51 (#18693)
add eefb680fe chore(deps): bump crate-ci/typos from 1.39.1 to 1.39.2
(#18694)
add f17cc09fb Remove FilterExec from CoalesceBatches optimization rule,
add fetch support (#18630)
add 0850cecc6 minor: refactor with `assert_or_internal_err!()` in
`datafusion/datasource` (#18697)
add 1dddf034f chore: Enforce lint rule `clippy::needless_pass_by_value` to
datafusion-datasource (#18682)
add 2b3b22083 feat: Handle edge case with `corr` with single row and `NaN`
(#18677)
add cfb26b6c0 [main] Update changelog for 51.0.0 RC2 (#18710)
add c39334f65 Refactor Spark crc32/sha1 signatures (#18662)
add 729bf2631 feat: support spark csc (#18642)
add 302dbc8e4 CI: try free up space in `Rust / cargo test (amd64)` action
(#18709)
add e730101c4 chore: enforce clippy lint needless_pass_by_value to
datafusion-proto (#18715)
add 0cfc1fefa chore: enforce clippy lint needless_pass_by_value to
datafusion-spark (#18714)
add 17e6c8815 minor: refactor with `assert_or_internal_err!()` in
`datafusion/optimizer` (#18699)
add a113a83ef minor: refactor with `assert_or_internal_err!()` in
`datafusion/functions` (#18700)
add 1b1cd0d1e minor: refactor with `assert_or_internal_err!()` in
`datafusion/expr-common` (#18702)
add 31b297578 minor: refactor with `assert_or_internal_err!()` in
`datafusion/functions-aggregate` (#18716)
add 899653f69 chore: enforce clippy lint needless_pass_by_value to
datafusion-execution (#18723)
add 30b0982f2 minor: refactor with `assert_or_internal_err!()` in
`datafusion/functions-nested` (#18724)
add 0cd6a19bc chore: enforce clippy lint needless_pass_by_value to
datafusion-substrait (#18703)
add 17050a9c6 chore: Refactor with assert_or_internal_err!() in
datafusion/spark. (#18674)
add 3ac3d0edf Minor: Add docs to release/README.md about rate limits
(#18704)
add 91eba1dfc feat: support spark sec (#18728)
add 3863267af Consolidate query planning examples (#18142) (#18690)
add a26afcea7 minor: refactor with `assert_or_internal_err!()` in
`datafusion/physical-expr-common` (#18735)
add 63545ae59 minor: refactor with `assert_or_internal_err!()` in
`datafusion/physical-expr` (#18736)
add 97af468f3 Consolidate ArrowFileSource and ArrowStreamFileSource
(#18720)
add ac9bb17d1 minor: refactor with `assert_or_internal_err!()` in
`datafusion/physical-optimizer` (#18732)
add 6cb3d7ec4 refactor: reduce duplication in make_udf_function macro
(#18733)
add b0015d3a9 feat(parquet): Implement `scan_efficiency_ratio` metric for
parquet reading (#18577)
add 85777c4f7 minor: refactor with `assert_or_internal_err!()` in
`datafusion/physical-plan` (#18730)
add 79f2b14dc chore: enforce clippy lint needless_pass_by_value to
datafusion-functions-aggregate-common (#18741)
add fad5f8a62 Optimize NullState::build (#18737)
add 8a75075a0 [Minor] Remove RawTableAllocExt (#18748)
add 5791822e4 chore: enforce clippy lint needless_pass_by_value to
datafusion-datasource-parquet (#18695)
add 4198c5a74 minor: refactor with `assert_or_internal_err!()` in
`datafusion/expr` (#18731)
add 0424eaa4d fix: Nested arrays should not get a field in lookup (#18745)
add e5b6d9700 minor: Fix an example in the `PruningPredicate`
documentation (#18742)
add 2437e0303 docs: fix broken catalog example links (#18765)
add e557d0aaa chore(deps): bump indicatif from 0.18.2 to 0.18.3 (#18756)
add b05d81114 doc: Add documentation for error handling (#18762)
add 7fa63789f Consolidate `EliminateNestedUnion` and `EliminateOneUnion`
optimizer rules' (#18678)
add 5d5a276f1 Fix map_query_sql benchmark duplicate key error (#18427)
add 8b8bdc4ad minor: enforce lint rule clippy::needless_pass_by_value to
datafusion-ffi (#18764)
add af2233675 perf: improve performance of `vectorized_equal_to` for
`PrimitiveGroupValueBuilder` in multi group by aggregation (#17977)
add e4bc51481 feat: Enhance map handling to support NULL map values
(#18531)
add bd30fe237 Rename boolean `Interval` constants to match
`NullableInterval` (#18654)
add 71fcd03d6 feat: add RESET statement for configuration variabless
(#18408)
add e64de0f2f chore(deps): bump bytes from 1.10.1 to 1.11.0 (#18755)
add 82181ac48 fix: update schema's data type for `LogicalPlan::Values`
after placeholder substitution (#18740)
add a732dc307 feat: add human-readable formatting to EXPLAIN ANALYZE
metrics #18689 (#18734)
add 0304cda4f CI: Fix `main` branch CI test failure (#18792)
add 958a6de9d chore: Enforce 'clippy::needless_pass_by_value' to
datafusion-expr-common (#18775)
add 19865b327 chore: Finish refactor with `assert_or_internal_err!()`
(#18790)
add 5765a08c8 Switch from xz2 to liblzma to reduce duplicate dependencies
(#17509)
add a07ddb21c chore(deps): bump taiki-e/install-action from 2.62.51 to
2.62.53 (#18796)
add 1b42a7ca7 chore(deps): bump actions/checkout from 5.0.0 to 5.0.1
(#18797)
add b3ff6d83b Misc improvements to ProjectionExprs (#18719)
add 984d21003 fix: Pick correct columns in Sort Merge Equijoin (#18772)
add b2e99fc09 Fix incorrect link for sql_query.rs example in README
(#18807)
add 45e93529d docs: Fix the examples for char_length() and
character_length() (#18808)
add 6601959fe Adds prefix filtering for table URLs (#18780)
add 486c5d860 Refactor InListExpr to support structs by re-using existing
hashing infrastructure (#18449)
add 2dd17b9e6 chore: Add script to protect RC branches during the release
(#18660)
add 373b0ad0f fix: remove `WorkTableExec` special case in
`reset_plan_states` (#18803)
add 794660482 Prevent overflow and panics when casting DATE to TIMESTAMP
by validating bounds (#18761)
add ac9c6b44e feat: support Spark-compatible `abs` math function part 1 -
non-ANSI mode (#18205)
add 307ec5093 chore(deps): bump taiki-e/install-action from 2.62.53 to
2.62.54 (#18815)
add d9d847e22 CI : Enforce clippy: :needless_pass_by_value rule to
datafusion-functions-aggregate (#18805)
add b1af8fe9c Consolidate sql operations examples (#18142) (#18743)
add d01c0d39b chore: Support 'untake' for unassigning github issues
(#18637)
add f9231fc12 Move `GuaranteeRewriter` to datafusion_expr (#18821)
add 5b0aa37c8 Refactor state management in `HashJoinExec` and use CASE
expressions for more precise filters (#18451)
add 3b390d782 Refactor avg & sum signatures away from user defined (#18769)
add 068f96f04 Hash UnionArrays (#18718)
add 5dbbac698 CI: add clippy::needless_pass_by_value rule to
datafusion-functions-window crate (#18838)
add 7fa2a694b Add field to DynamicPhysicalExpr to indicate when the filter
is complete or updated (#18799)
add 0bd127fd9 #17801 Improve nullability reporting of case expressions
(#17813)
add 356616e0c Consolidate execution monitoring examples (#18142) (#18846)
add 99ab87c57 Implement CatalogProviderList in FFI (#18657)
add dc6e7cf86 fix: display the failed sqllogictest file and query that
failed in case of a panic (#18785)
add 1833093ce Removed incorrect union check in enforce_sorting and updated
tests (#18661)
add 7d8b8602a Change `CacheAccessor::remove` to take `&self` rather than
`&mut self` (#18726)
add 8a91db56b Move statistics handling into FileScanConfig (#18721)
add f1ecaccd1 chore: Add filtered pending PRs link to main page (#18854)
add 195bd5cb3 chore(deps): bump actions/checkout from 5.0.1 to 6.0.0
(#18865)
add d65fb860d Remove unnecessary bit counting code from spark `bit_count`
(#18841)
add 78a4bbab6 Fix async_udf batch size behaviour (#18819)
add a7b2e8556 Docs: Enhance contributor guide with testing section (#18852)
add e6ddb48f4 Fix Partial AggregateExec correctness issue dropping rows
(#18712)
add e6d1773ef chore: Add missing boolean tests to `bit_count` Spark
function (#18871)
add c7b339e0b Consolidate proto examples (#18142) (#18861)
add 12cb4cae1 Use logical null count in `case_when_with_expr` (#18872)
add c141fca12 chore: enforce `clippy::needless_pass_by_value` to
`datafusion-physical-plan` (#18864)
add 5e36b052e Refactor spark `bit_get()` signature away from user defined
(#18836)
add 1e9c1f426 minor: enforce lint rule clippy::needless_pass_by_value to
datafusion-functions (#18768)
add 240e2e39a chore: enforce clippy lint needless_pass_by_value to
datafusion-functions-nested (#18839)
add 8b3725aec chore: fix CI on main (#18876)
add c8d26ba01 feat: Support Show runtime settings (#18564)
add 5f2272219 chore: update Repartition DisplayAs to indicate maintained
sort order (#18673)
add 6ed22bb7c implement sum for durations (#18853)
add fbe1ae6ab feat(small): Support `<slt:ignore>` marker in `sqllogictest`
for non-deterministic expected parts (#18857)
add d3ae1a785 Consolidate dataframe examples (#18142) (#18862)
add e50b9398d Docs: Enhance testing documentation with examples and links
(#18851)
add 7d87976e1 Avoid the need to rewrite expressions when evaluating
logical case nullability (#18849)
add fc77be945 chore: remove `pyarrow` feature (#18528)
add b0b43a311 Merge branch 'main' into sandbox-main
No new revisions were added by this update.
Summary of changes:
.github/workflows/audit.yml | 4 +-
.github/workflows/dependencies.yml | 4 +-
.github/workflows/dev.yml | 4 +-
.github/workflows/docs.yaml | 4 +-
.github/workflows/docs_pr.yaml | 2 +-
.github/workflows/extended.yml | 8 +-
.github/workflows/labeler.yml | 2 +-
.github/workflows/large_files.yml | 2 +-
.github/workflows/rust.yml | 94 +-
.github/workflows/take.yml | 32 +-
Cargo.lock | 317 ++--
Cargo.toml | 81 +-
ci/scripts/rust_clippy.sh | 2 +-
datafusion-cli/src/exec.rs | 4 +-
datafusion-examples/Cargo.toml | 5 -
datafusion-examples/README.md | 69 +-
.../examples/builtin_functions/date_time.rs | 2 +
.../examples/builtin_functions/function_factory.rs | 2 +
.../examples/builtin_functions/main.rs | 5 +
.../examples/builtin_functions/regexp.rs | 2 +
.../{ => custom_data_source}/csv_json_opener.rs | 27 +-
.../{ => custom_data_source}/csv_sql_streaming.rs | 5 +-
.../{ => custom_data_source}/custom_datasource.rs | 5 +-
.../{ => custom_data_source}/custom_file_casts.rs | 6 +-
.../{ => custom_data_source}/custom_file_format.rs | 81 +-
.../default_column_values.rs | 48 +-
.../file_stream_provider.rs | 38 +-
.../examples/custom_data_source/main.rs | 135 ++
.../examples/{ => data_io}/catalog.rs | 17 +-
.../examples/{ => data_io}/json_shredding.rs | 5 +-
datafusion-examples/examples/data_io/main.rs | 160 ++
.../parquet_advanced_index.rs} | 20 +-
.../{ => data_io}/parquet_embedded_index.rs | 58 +-
.../examples/{ => data_io}/parquet_encrypted.rs | 6 +-
.../{ => data_io}/parquet_encrypted_with_kms.rs | 5 +-
.../examples/{ => data_io}/parquet_exec_visitor.rs | 15 +-
.../examples/{ => data_io}/parquet_index.rs | 10 +-
.../query_http_csv.rs} | 8 +-
.../examples/{ => data_io}/remote_catalog.rs | 6 +-
.../examples/{ => dataframe}/dataframe.rs | 13 +-
.../{ => dataframe}/deserialize_to_struct.rs | 5 +-
.../{builtin_functions => dataframe}/main.rs | 45 +-
.../main.rs | 57 +-
.../memory_pool_execution_plan.rs | 6 +-
.../memory_pool_tracking.rs | 9 +-
.../examples/{ => execution_monitoring}/tracing.rs | 6 +-
datafusion-examples/examples/flight/client.rs | 2 +
datafusion-examples/examples/flight/main.rs | 5 +
datafusion-examples/examples/flight/server.rs | 2 +
datafusion-examples/examples/flight/sql_server.rs | 2 +
.../{ => proto}/composed_extension_codec.rs | 16 +-
.../examples/{builtin_functions => proto}/main.rs | 39 +-
.../examples/{ => query_planning}/analyzer_rule.rs | 5 +-
.../examples/{ => query_planning}/expr_api.rs | 5 +-
.../examples/query_planning/main.rs | 138 ++
.../{ => query_planning}/optimizer_rule.rs | 5 +-
.../{ => query_planning}/parse_sql_expr.rs | 10 +-
.../examples/{ => query_planning}/plan_to_sql.rs | 20 +-
.../examples/{ => query_planning}/planner_api.rs | 5 +-
.../examples/{ => query_planning}/pruning.rs | 10 +-
.../examples/{ => query_planning}/thread_pools.rs | 7 +-
.../{sql_analysis.rs => sql_ops/analysis.rs} | 273 ++--
.../{sql_dialect.rs => sql_ops/dialect.rs} | 5 +-
.../{sql_frontend.rs => sql_ops/frontend.rs} | 4 +-
.../{builtin_functions => sql_ops}/main.rs | 55 +-
.../examples/{sql_query.rs => sql_ops/query.rs} | 5 +-
datafusion-examples/examples/udf/advanced_udaf.rs | 2 +
datafusion-examples/examples/udf/advanced_udf.rs | 2 +
datafusion-examples/examples/udf/advanced_udwf.rs | 2 +
datafusion-examples/examples/udf/async_udf.rs | 2 +
datafusion-examples/examples/udf/main.rs | 5 +
datafusion-examples/examples/udf/simple_udaf.rs | 2 +
datafusion-examples/examples/udf/simple_udf.rs | 2 +
datafusion-examples/examples/udf/simple_udtf.rs | 2 +
datafusion-examples/examples/udf/simple_udwf.rs | 2 +
datafusion/catalog-listing/Cargo.toml | 1 -
datafusion/catalog-listing/src/helpers.rs | 573 ++-----
datafusion/catalog-listing/src/mod.rs | 3 +
datafusion/catalog-listing/src/table.rs | 38 +-
datafusion/catalog/src/async.rs | 2 +-
datafusion/catalog/src/catalog.rs | 4 +-
datafusion/catalog/src/information_schema.rs | 37 +-
datafusion/catalog/src/lib.rs | 2 +
datafusion/common-runtime/src/lib.rs | 3 +
datafusion/common/Cargo.toml | 4 +-
datafusion/common/src/config.rs | 144 +-
datafusion/common/src/error.rs | 239 ++-
datafusion/common/src/hash_utils.rs | 333 +++-
datafusion/common/src/lib.rs | 2 -
datafusion/common/src/pyarrow.rs | 169 --
datafusion/common/src/scalar/mod.rs | 281 +++-
datafusion/common/src/utils/mod.rs | 9 +-
datafusion/common/src/utils/proxy.rs | 76 +-
datafusion/core/Cargo.toml | 5 +-
datafusion/core/benches/map_query_sql.rs | 10 +-
datafusion/core/benches/sql_planner.rs | 71 +-
datafusion/core/benches/topk_aggregate.rs | 30 +-
datafusion/core/src/dataframe/mod.rs | 22 +-
datafusion/core/src/datasource/file_format/mod.rs | 6 +-
datafusion/core/src/datasource/mod.rs | 13 +-
.../core/src/datasource/physical_plan/avro.rs | 29 +-
.../core/src/datasource/physical_plan/csv.rs | 185 ++-
.../core/src/datasource/physical_plan/json.rs | 24 +-
.../core/src/datasource/physical_plan/parquet.rs | 61 +-
datafusion/core/src/execution/context/mod.rs | 96 +-
datafusion/core/src/execution/session_state.rs | 40 +
datafusion/core/src/lib.rs | 2 +
datafusion/core/src/physical_planner.rs | 101 +-
datafusion/core/src/test/mod.rs | 31 +-
datafusion/core/src/test_util/parquet.rs | 37 +-
.../core/tests/{catalog => catalog_listing}/mod.rs | 2 +-
.../tests/catalog_listing/pruned_partition_list.rs | 251 +++
datafusion/core/tests/core_integration.rs | 3 +
.../part=123/data.arrow | Bin 0 -> 1608 bytes
.../part=456/data.arrow | Bin 0 -> 1608 bytes
datafusion/core/tests/dataframe/mod.rs | 251 ++-
.../core/tests/datasource/object_store_access.rs | 63 +-
datafusion/core/tests/execution/mod.rs | 1 +
datafusion/core/tests/execution/register_arrow.rs | 90 ++
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 697 ++++----
datafusion/core/tests/fuzz_cases/pruning.rs | 5 +-
datafusion/core/tests/macro_hygiene/mod.rs | 1 +
datafusion/core/tests/optimizer/mod.rs | 26 +-
datafusion/core/tests/parquet/custom_reader.rs | 3 +-
.../core/tests/parquet/external_access_plan.rs | 11 +-
datafusion/core/tests/parquet/page_pruning.rs | 4 +-
datafusion/core/tests/parquet/schema_adapter.rs | 22 +-
datafusion/core/tests/parquet/schema_coercion.rs | 15 +-
.../physical_optimizer/enforce_distribution.rs | 523 +++---
.../tests/physical_optimizer/enforce_sorting.rs | 49 +-
.../physical_optimizer/filter_pushdown/mod.rs | 318 +++-
.../physical_optimizer/filter_pushdown/util.rs | 113 +-
.../tests/physical_optimizer/limit_pushdown.rs | 7 +-
.../physical_optimizer/projection_pushdown.rs | 59 +-
.../replace_with_order_preserving_variants.rs | 120 +-
.../tests/physical_optimizer/sanity_checker.rs | 10 +-
.../core/tests/physical_optimizer/test_utils.rs | 14 +-
.../schema_adapter_integration_tests.rs | 42 +-
datafusion/core/tests/sql/explain_analyze.rs | 165 +-
datafusion/core/tests/sql/joins.rs | 30 +-
datafusion/core/tests/sql/mod.rs | 18 +-
datafusion/core/tests/sql/select.rs | 42 +-
datafusion/core/tests/tpcds_planning.rs | 5 +-
datafusion/core/tests/user_defined/mod.rs | 3 +
.../user_defined_async_scalar_functions.rs | 139 ++
.../core/tests/user_defined/user_defined_plan.rs | 19 +-
.../user_defined/user_defined_scalar_functions.rs | 233 ++-
datafusion/datasource-arrow/src/file_format.rs | 436 +++--
datafusion/datasource-arrow/src/mod.rs | 5 +
datafusion/datasource-arrow/src/source.rs | 566 +++++--
.../tests/data/example_stream.arrow | Bin 0 -> 1480 bytes
.../example_stream_corrupted_metadata_length.arrow | Bin 0 -> 1480 bytes
.../tests/data/example_stream_empty.arrow | Bin 0 -> 776 bytes
.../src/avro_to_arrow/arrow_array_reader.rs | 98 +-
.../datasource-avro/src/avro_to_arrow/reader.rs | 8 +-
datafusion/datasource-avro/src/file_format.rs | 10 +-
datafusion/datasource-avro/src/mod.rs | 2 +
datafusion/datasource-avro/src/source.rs | 47 +-
datafusion/datasource-csv/src/file_format.rs | 36 +-
datafusion/datasource-csv/src/mod.rs | 7 +-
datafusion/datasource-csv/src/source.rs | 143 +-
datafusion/datasource-json/src/file_format.rs | 11 +-
datafusion/datasource-json/src/mod.rs | 3 +
datafusion/datasource-json/src/source.rs | 37 +-
datafusion/datasource-parquet/src/access_plan.rs | 17 +-
datafusion/datasource-parquet/src/file_format.rs | 20 +-
datafusion/datasource-parquet/src/metadata.rs | 32 +-
datafusion/datasource-parquet/src/metrics.rs | 15 +-
datafusion/datasource-parquet/src/mod.rs | 3 +
datafusion/datasource-parquet/src/opener.rs | 2 +-
datafusion/datasource-parquet/src/page_filter.rs | 1 +
datafusion/datasource-parquet/src/reader.rs | 56 +-
.../datasource-parquet/src/row_group_filter.rs | 16 +-
datafusion/datasource-parquet/src/source.rs | 103 +-
datafusion/datasource/Cargo.toml | 6 +-
datafusion/datasource/src/file.rs | 13 +-
datafusion/datasource/src/file_compression_type.rs | 4 +-
datafusion/datasource/src/file_format.rs | 5 +-
datafusion/datasource/src/file_scan_config.rs | 221 ++-
datafusion/datasource/src/file_stream.rs | 4 +-
datafusion/datasource/src/memory.rs | 25 +-
datafusion/datasource/src/mod.rs | 4 +
datafusion/datasource/src/sink.rs | 10 +-
datafusion/datasource/src/source.rs | 9 +-
datafusion/datasource/src/statistics.rs | 39 +-
datafusion/datasource/src/table_schema.rs | 6 +
datafusion/datasource/src/test_util.rs | 52 +-
datafusion/datasource/src/url.rs | 85 +-
datafusion/datasource/src/write/demux.rs | 4 +-
datafusion/doc/src/lib.rs | 3 +
datafusion/execution/src/cache/cache_manager.rs | 2 +-
datafusion/execution/src/cache/cache_unit.rs | 12 +-
datafusion/execution/src/cache/mod.rs | 2 +-
datafusion/execution/src/disk_manager.rs | 26 +-
datafusion/execution/src/lib.rs | 3 +
datafusion/execution/src/memory_pool/mod.rs | 107 +-
datafusion/execution/src/memory_pool/pool.rs | 8 +-
datafusion/execution/src/runtime_env.rs | 121 +-
datafusion/expr-common/src/columnar_value.rs | 206 ++-
datafusion/expr-common/src/interval_arithmetic.rs | 1101 ++++++++++---
datafusion/expr-common/src/lib.rs | 3 +
datafusion/expr-common/src/statistics.rs | 160 +-
datafusion/expr-common/src/type_coercion/binary.rs | 84 +-
.../src/type_coercion/binary/tests/mod.rs | 1 +
.../type_coercion/binary/tests/run_end_encoded.rs | 99 ++
datafusion/expr/src/execution_props.rs | 1 +
datafusion/expr/src/expr_fn.rs | 5 +
datafusion/expr/src/expr_rewriter/guarantees.rs | 668 ++++++++
datafusion/expr/src/expr_rewriter/mod.rs | 5 +
datafusion/expr/src/expr_rewriter/order_by.rs | 6 +-
datafusion/expr/src/expr_schema.rs | 220 ++-
datafusion/expr/src/lib.rs | 4 +
datafusion/expr/src/literal.rs | 3 +
datafusion/expr/src/logical_plan/invariants.rs | 20 +-
datafusion/expr/src/logical_plan/mod.rs | 5 +-
datafusion/expr/src/logical_plan/plan.rs | 126 +-
datafusion/expr/src/logical_plan/statement.rs | 12 +
datafusion/expr/src/predicate_bounds.rs | 681 ++++++++
datafusion/expr/src/udaf.rs | 10 +-
datafusion/expr/src/udf.rs | 21 +-
datafusion/expr/src/utils.rs | 7 +-
datafusion/ffi/src/arrow_wrappers.rs | 5 +-
datafusion/ffi/src/catalog_provider.rs | 2 +-
datafusion/ffi/src/catalog_provider_list.rs | 283 ++++
datafusion/ffi/src/lib.rs | 4 +
datafusion/ffi/src/tests/async_provider.rs | 4 +-
datafusion/ffi/src/tests/catalog.rs | 57 +-
datafusion/ffi/src/tests/mod.rs | 6 +
.../tests/{ffi_integration.rs => ffi_catalog.rs} | 74 +-
datafusion/ffi/tests/ffi_integration.rs | 27 -
.../src/aggregate/count_distinct/native.rs | 70 +-
.../src/aggregate/groups_accumulator/accumulate.rs | 10 +-
.../src/aggregate/sum_distinct/numeric.rs | 78 +-
datafusion/functions-aggregate-common/src/lib.rs | 4 +
.../src/noop_accumulator.rs | 70 +
.../functions-aggregate-common/src/tdigest.rs | 1 +
datafusion/functions-aggregate-common/src/utils.rs | 99 +-
.../functions-aggregate/src/approx_distinct.rs | 52 +-
.../src/approx_percentile_cont.rs | 8 +-
.../src/approx_percentile_cont_with_weight.rs | 6 +-
datafusion/functions-aggregate/src/array_agg.rs | 20 +-
datafusion/functions-aggregate/src/average.rs | 55 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 33 +-
datafusion/functions-aggregate/src/correlation.rs | 137 +-
datafusion/functions-aggregate/src/first_last.rs | 116 +-
datafusion/functions-aggregate/src/lib.rs | 3 +
datafusion/functions-aggregate/src/median.rs | 73 +-
datafusion/functions-aggregate/src/nth_value.rs | 22 +-
.../functions-aggregate/src/percentile_cont.rs | 93 +-
datafusion/functions-aggregate/src/sum.rs | 136 +-
.../functions-nested/benches/array_reverse.rs | 76 +-
datafusion/functions-nested/src/array_has.rs | 35 +-
datafusion/functions-nested/src/flatten.rs | 12 +-
datafusion/functions-nested/src/lib.rs | 3 +
datafusion/functions-nested/src/map.rs | 410 ++++-
datafusion/functions-nested/src/position.rs | 20 +-
datafusion/functions-nested/src/remove.rs | 10 +-
datafusion/functions-nested/src/replace.rs | 14 +-
datafusion/functions-nested/src/reverse.rs | 117 +-
datafusion/functions-nested/src/set_ops.rs | 15 +-
datafusion/functions-nested/src/string.rs | 60 +-
datafusion/functions-table/src/lib.rs | 3 +
datafusion/functions-window-common/src/lib.rs | 3 +
datafusion/functions-window/src/lead_lag.rs | 2 +-
datafusion/functions-window/src/lib.rs | 3 +
datafusion/functions-window/src/nth_value.rs | 2 +-
datafusion/functions-window/src/ntile.rs | 4 +-
datafusion/functions-window/src/utils.rs | 4 +-
datafusion/functions/src/core/getfield.rs | 52 +-
datafusion/functions/src/core/greatest.rs | 14 +-
.../functions/src/core/greatest_least_utils.rs | 27 +-
datafusion/functions/src/core/least.rs | 14 +-
datafusion/functions/src/core/mod.rs | 2 +
datafusion/functions/src/crypto/basic.rs | 13 +-
datafusion/functions/src/datetime/common.rs | 8 +-
datafusion/functions/src/datetime/date_trunc.rs | 177 ++-
datafusion/functions/src/datetime/to_char.rs | 14 +-
datafusion/functions/src/lib.rs | 3 +
datafusion/functions/src/macros.rs | 10 +-
datafusion/functions/src/math/abs.rs | 16 +-
datafusion/functions/src/math/monotonicity.rs | 20 +-
datafusion/functions/src/math/pi.rs | 10 +-
datafusion/functions/src/math/power.rs | 448 +++++-
datafusion/functions/src/math/random.rs | 10 +-
datafusion/functions/src/regex/regexpcount.rs | 38 +-
datafusion/functions/src/regex/regexpinstr.rs | 1 +
datafusion/functions/src/regex/regexpreplace.rs | 1 +
datafusion/functions/src/string/ascii.rs | 8 +-
datafusion/functions/src/string/common.rs | 1 +
datafusion/functions/src/string/repeat.rs | 8 +-
datafusion/functions/src/string/split_part.rs | 1 +
datafusion/functions/src/string/uuid.rs | 10 +-
.../functions/src/unicode/character_length.rs | 8 +-
datafusion/functions/src/unicode/find_in_set.rs | 29 +-
datafusion/functions/src/unicode/lpad.rs | 12 +-
datafusion/functions/src/unicode/reverse.rs | 6 +-
datafusion/functions/src/unicode/rpad.rs | 12 +-
datafusion/functions/src/unicode/strpos.rs | 22 +-
datafusion/functions/src/unicode/substr.rs | 106 +-
datafusion/functions/src/utils.rs | 62 +-
.../src/decorrelate_predicate_subquery.rs | 13 +-
datafusion/optimizer/src/eliminate_one_union.rs | 121 --
.../optimizer/src/extract_equijoin_predicate.rs | 15 +-
datafusion/optimizer/src/lib.rs | 9 +-
.../optimizer/src/optimize_projections/mod.rs | 28 +-
...liminate_nested_union.rs => optimize_unions.rs} | 43 +-
datafusion/optimizer/src/optimizer.rs | 7 +-
.../optimizer/src/propagate_empty_relation.rs | 4 +-
datafusion/optimizer/src/push_down_filter.rs | 20 +-
.../optimizer/src/scalar_subquery_to_join.rs | 18 +-
.../src/simplify_expressions/expr_simplifier.rs | 10 +-
.../src/simplify_expressions/guarantees.rs | 476 ------
.../optimizer/src/simplify_expressions/mod.rs | 3 +-
.../optimizer/src/single_distinct_to_groupby.rs | 18 +-
datafusion/physical-expr-adapter/src/lib.rs | 3 +
datafusion/physical-expr-common/src/datum.rs | 83 +-
datafusion/physical-expr-common/src/lib.rs | 3 +
.../physical-expr-common/src/physical_expr.rs | 23 +-
datafusion/physical-expr-common/src/utils.rs | 21 +
datafusion/physical-expr/Cargo.toml | 4 +
datafusion/physical-expr/src/aggregate.rs | 9 +-
datafusion/physical-expr/src/analysis.rs | 43 +-
.../physical-expr/src/async_scalar_function.rs | 25 +-
.../src/equivalence/properties/mod.rs | 2 +-
datafusion/physical-expr/src/expressions/binary.rs | 117 +-
.../src/expressions/binary/kernels.rs | 6 +-
datafusion/physical-expr/src/expressions/case.rs | 404 ++++-
.../src/expressions/dynamic_filters.rs | 102 +-
.../physical-expr/src/expressions/in_list.rs | 1677 +++++++++++++++++---
datafusion/physical-expr/src/expressions/like.rs | 22 +-
.../physical-expr/src/expressions/literal.rs | 1 +
datafusion/physical-expr/src/expressions/not.rs | 8 +-
.../physical-expr/src/intervals/cp_solver.rs | 79 +-
datafusion/physical-expr/src/lib.rs | 3 +
datafusion/physical-expr/src/projection.rs | 194 ++-
.../physical-expr/src/statistics/stats_solver.rs | 2 +-
datafusion/physical-expr/src/utils/guarantee.rs | 4 +-
.../src/window/standard_window_function_expr.rs | 9 +-
datafusion/physical-expr/src/window/window_expr.rs | 9 +-
.../src/coalesce_async_exec_input.rs | 71 -
.../physical-optimizer/src/coalesce_batches.rs | 31 +-
.../physical-optimizer/src/enforce_distribution.rs | 3 +-
.../physical-optimizer/src/enforce_sorting/mod.rs | 7 +-
.../replace_with_order_preserving_variants.rs | 23 +-
.../src/enforce_sorting/sort_pushdown.rs | 4 +-
.../physical-optimizer/src/filter_pushdown.rs | 58 +-
datafusion/physical-optimizer/src/lib.rs | 4 +-
datafusion/physical-optimizer/src/optimizer.rs | 2 -
.../physical-optimizer/src/projection_pushdown.rs | 4 +-
.../group_values/multi_group_by/primitive.rs | 114 +-
.../src/aggregates/group_values/null_builder.rs | 8 +
.../src/aggregates/group_values/row.rs | 14 +-
datafusion/physical-plan/src/aggregates/mod.rs | 71 +-
.../physical-plan/src/aggregates/no_grouping.rs | 23 +-
.../physical-plan/src/aggregates/row_hash.rs | 225 ++-
.../physical-plan/src/aggregates/topk/heap.rs | 14 +-
.../physical-plan/src/aggregates/topk_stream.rs | 11 +-
datafusion/physical-plan/src/analyze.rs | 22 +-
datafusion/physical-plan/src/async_func.rs | 10 +-
datafusion/physical-plan/src/coalesce/mod.rs | 11 +-
.../physical-plan/src/coalesce_partitions.rs | 12 +-
datafusion/physical-plan/src/coop.rs | 10 +-
datafusion/physical-plan/src/empty.rs | 28 +-
datafusion/physical-plan/src/execution_plan.rs | 58 +-
datafusion/physical-plan/src/explain.rs | 10 +-
datafusion/physical-plan/src/filter.rs | 168 +-
datafusion/physical-plan/src/filter_pushdown.rs | 1 +
datafusion/physical-plan/src/joins/cross_join.rs | 16 +-
.../physical-plan/src/joins/hash_join/exec.rs | 257 ++-
.../physical-plan/src/joins/hash_join/mod.rs | 1 +
.../src/joins/hash_join/partitioned_hash_eval.rs | 158 ++
.../src/joins/hash_join/shared_bounds.rs | 409 +++--
.../physical-plan/src/joins/hash_join/stream.rs | 112 +-
.../physical-plan/src/joins/nested_loop_join.rs | 332 +++-
.../src/joins/piecewise_merge_join/classic_join.rs | 20 +-
.../src/joins/sort_merge_join/exec.rs | 15 +-
.../src/joins/sort_merge_join/metrics.rs | 8 -
.../src/joins/sort_merge_join/stream.rs | 66 +-
.../src/joins/sort_merge_join/tests.rs | 234 +++
.../physical-plan/src/joins/stream_join_utils.rs | 6 -
.../physical-plan/src/joins/symmetric_hash_join.rs | 51 +-
datafusion/physical-plan/src/joins/test_utils.rs | 6 +-
datafusion/physical-plan/src/joins/utils.rs | 62 +-
datafusion/physical-plan/src/lib.rs | 3 +
datafusion/physical-plan/src/limit.rs | 20 +-
datafusion/physical-plan/src/memory.rs | 48 +-
datafusion/physical-plan/src/metrics/baseline.rs | 14 +
datafusion/physical-plan/src/metrics/builder.rs | 22 +-
datafusion/physical-plan/src/metrics/mod.rs | 5 +-
datafusion/physical-plan/src/metrics/value.rs | 300 +++-
datafusion/physical-plan/src/placeholder_row.rs | 14 +-
datafusion/physical-plan/src/projection.rs | 95 +-
datafusion/physical-plan/src/recursive_query.rs | 17 +-
datafusion/physical-plan/src/repartition/mod.rs | 147 +-
datafusion/physical-plan/src/sorts/sort.rs | 43 +-
.../src/sorts/sort_preserving_merge.rs | 14 +-
datafusion/physical-plan/src/sorts/stream.rs | 13 +-
.../physical-plan/src/sorts/streaming_merge.rs | 11 +-
datafusion/physical-plan/src/spill/mod.rs | 1 +
datafusion/physical-plan/src/test.rs | 19 +-
datafusion/physical-plan/src/topk/mod.rs | 67 +-
datafusion/physical-plan/src/union.rs | 52 +-
datafusion/physical-plan/src/unnest.rs | 15 +-
.../physical-plan/src/windows/window_agg_exec.rs | 16 +-
datafusion/physical-plan/src/work_table.rs | 14 +-
datafusion/proto-common/src/lib.rs | 3 +
datafusion/proto/src/common.rs | 14 +-
datafusion/proto/src/lib.rs | 3 +
datafusion/proto/src/logical_plan/mod.rs | 13 +-
datafusion/proto/src/physical_plan/from_proto.rs | 58 +-
datafusion/proto/src/physical_plan/mod.rs | 41 +-
datafusion/proto/src/physical_plan/to_proto.rs | 3 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 4 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 131 +-
datafusion/pruning/src/lib.rs | 4 +
datafusion/pruning/src/pruning_predicate.rs | 22 +-
datafusion/session/src/lib.rs | 4 +
datafusion/spark/src/function/aggregate/avg.rs | 35 +-
datafusion/spark/src/function/bitwise/bit_count.rs | 25 +-
datafusion/spark/src/function/bitwise/bit_get.rs | 237 +--
datafusion/spark/src/function/bitwise/bit_shift.rs | 727 ++-------
datafusion/spark/src/function/bitwise/mod.rs | 18 +-
datafusion/spark/src/function/datetime/date_add.rs | 8 +-
datafusion/spark/src/function/datetime/last_day.rs | 10 +-
.../spark/src/function/datetime/make_interval.rs | 43 +-
datafusion/spark/src/function/hash/crc32.rs | 56 +-
datafusion/spark/src/function/hash/sha1.rs | 58 +-
datafusion/spark/src/function/math/abs.rs | 378 +++++
datafusion/spark/src/function/math/expm1.rs | 63 +-
datafusion/spark/src/function/math/factorial.rs | 10 +-
datafusion/spark/src/function/math/hex.rs | 13 +-
datafusion/spark/src/function/math/mod.rs | 11 +
datafusion/spark/src/function/math/modulus.rs | 28 +-
datafusion/spark/src/function/math/rint.rs | 7 +-
datafusion/spark/src/function/math/trigonometry.rs | 167 ++
.../spark/src/function/string/format_string.rs | 25 +-
datafusion/spark/src/function/string/length.rs | 16 +-
datafusion/spark/src/lib.rs | 3 +
datafusion/sql/src/cte.rs | 12 +-
datafusion/sql/src/expr/binary_op.rs | 4 +-
datafusion/sql/src/expr/function.rs | 93 +-
datafusion/sql/src/expr/identifier.rs | 9 +-
datafusion/sql/src/expr/mod.rs | 16 +-
datafusion/sql/src/expr/subquery.rs | 8 +-
datafusion/sql/src/expr/value.rs | 22 +-
datafusion/sql/src/lib.rs | 3 +
datafusion/sql/src/parser.rs | 103 +-
datafusion/sql/src/resolve.rs | 1 +
datafusion/sql/src/statement.rs | 87 +-
datafusion/sql/src/unparser/expr.rs | 35 +-
datafusion/sql/src/unparser/plan.rs | 20 +-
datafusion/sql/src/unparser/utils.rs | 26 +-
datafusion/sql/src/utils.rs | 11 +-
datafusion/sql/tests/cases/params.rs | 2 +-
datafusion/sql/tests/sql_integration.rs | 24 +-
datafusion/sqllogictest/README.md | 11 +
datafusion/sqllogictest/bin/sqllogictests.rs | 84 +-
.../src/engines/currently_executed_sql.rs | 85 +
.../src/engines/datafusion_engine/runner.rs | 25 +-
.../runner.rs | 20 +
datafusion/sqllogictest/src/engines/mod.rs | 3 +
.../src/engines/postgres_engine/mod.rs | 37 +-
datafusion/sqllogictest/src/lib.rs | 1 +
datafusion/sqllogictest/src/util.rs | 49 +-
.../test_files/agg_func_substitute.slt | 6 +-
datafusion/sqllogictest/test_files/aggregate.slt | 166 +-
.../test_files/aggregate_repartition.slt | 10 +-
.../sqllogictest/test_files/aggregates_topk.slt | 49 +-
datafusion/sqllogictest/test_files/array.slt | 42 +-
datafusion/sqllogictest/test_files/arrow_files.slt | 260 +++
datafusion/sqllogictest/test_files/async_udf.slt | 11 +-
datafusion/sqllogictest/test_files/case.slt | 20 +
.../sqllogictest/test_files/count_star_rule.slt | 23 +-
datafusion/sqllogictest/test_files/cte.slt | 39 +-
datafusion/sqllogictest/test_files/decimal.slt | 139 ++
datafusion/sqllogictest/test_files/dictionary.slt | 15 +-
datafusion/sqllogictest/test_files/explain.slt | 23 +-
.../drop_tables.slt.part => explain_analyze.slt} | 29 +-
.../sqllogictest/test_files/explain_tree.slt | 871 +++++-----
datafusion/sqllogictest/test_files/expr.slt | 207 +++
.../test_files/filter_without_sort_exec.slt | 42 +-
datafusion/sqllogictest/test_files/functions.slt | 19 +-
datafusion/sqllogictest/test_files/group_by.slt | 40 +-
.../sqllogictest/test_files/information_schema.slt | 10 +
datafusion/sqllogictest/test_files/insert.slt | 15 +-
.../sqllogictest/test_files/insert_to_external.slt | 10 +-
datafusion/sqllogictest/test_files/join.slt.part | 37 +-
.../test_files/join_disable_repartition_joins.slt | 9 +-
.../test_files/join_is_not_distinct_from.slt | 9 +-
datafusion/sqllogictest/test_files/joins.slt | 188 +--
datafusion/sqllogictest/test_files/limit.slt | 14 +-
datafusion/sqllogictest/test_files/map.slt | 25 +-
.../test_files/monotonic_projection_test.slt | 10 +-
.../sqllogictest/test_files/named_arguments.slt | 2 +-
datafusion/sqllogictest/test_files/operator.slt | 40 +-
datafusion/sqllogictest/test_files/options.slt | 11 +-
datafusion/sqllogictest/test_files/order.slt | 52 +-
datafusion/sqllogictest/test_files/parquet.slt | 28 +-
.../test_files/parquet_filter_pushdown.slt | 42 +-
.../sqllogictest/test_files/parquet_statistics.slt | 21 +-
datafusion/sqllogictest/test_files/predicates.slt | 49 +-
datafusion/sqllogictest/test_files/prepare.slt | 22 +-
datafusion/sqllogictest/test_files/projection.slt | 7 +-
.../sqllogictest/test_files/push_down_filter.slt | 63 +-
datafusion/sqllogictest/test_files/pwmj.slt | 46 +-
datafusion/sqllogictest/test_files/qualify.slt | 71 +-
.../sqllogictest/test_files/regexp/regexp_like.slt | 5 +-
datafusion/sqllogictest/test_files/repartition.slt | 14 +-
.../sqllogictest/test_files/repartition_scan.slt | 30 +-
datafusion/sqllogictest/test_files/scalar.slt | 2 +-
datafusion/sqllogictest/test_files/select.slt | 44 +-
.../sqllogictest/test_files/set_variable.slt | 171 ++
.../sqllogictest/test_files/simplify_expr.slt | 21 +-
.../{spark/string/concat.slt => slt_features.slt} | 56 +-
.../sqllogictest/test_files/sort_merge_join.slt | 49 +
.../test_files/spark/bitwise/bit_count.slt | 15 +
.../test_files/spark/bitwise/bit_get.slt | 25 +
.../test_files/spark/bitwise/shiftright.slt | 115 ++
.../spark/bitwise/shiftrightunsigned.slt | 103 ++
.../sqllogictest/test_files/spark/hash/crc32.slt | 31 +-
.../sqllogictest/test_files/spark/hash/sha1.slt | 25 +-
.../sqllogictest/test_files/spark/math/abs.slt | 69 +-
.../sqllogictest/test_files/spark/math/csc.slt | 23 +-
.../sqllogictest/test_files/spark/math/expm1.slt | 5 +
.../sqllogictest/test_files/spark/math/sec.slt | 22 +-
.../test_files/spark/math/shiftleft.slt | 100 ++
.../test_files/string/string_literal.slt | 4 +-
datafusion/sqllogictest/test_files/subquery.slt | 32 +-
datafusion/sqllogictest/test_files/timestamps.slt | 55 +-
datafusion/sqllogictest/test_files/topk.slt | 2 +-
.../sqllogictest/test_files/tpch/plans/q1.slt.part | 5 +-
.../test_files/tpch/plans/q10.slt.part | 30 +-
.../test_files/tpch/plans/q11.slt.part | 60 +-
.../test_files/tpch/plans/q12.slt.part | 11 +-
.../test_files/tpch/plans/q13.slt.part | 14 +-
.../test_files/tpch/plans/q14.slt.part | 12 +-
.../test_files/tpch/plans/q15.slt.part | 31 +-
.../test_files/tpch/plans/q16.slt.part | 18 +-
.../test_files/tpch/plans/q17.slt.part | 19 +-
.../test_files/tpch/plans/q18.slt.part | 30 +-
.../test_files/tpch/plans/q19.slt.part | 18 +-
.../sqllogictest/test_files/tpch/plans/q2.slt.part | 103 +-
.../test_files/tpch/plans/q20.slt.part | 64 +-
.../test_files/tpch/plans/q21.slt.part | 49 +-
.../test_files/tpch/plans/q22.slt.part | 26 +-
.../sqllogictest/test_files/tpch/plans/q3.slt.part | 25 +-
.../sqllogictest/test_files/tpch/plans/q4.slt.part | 14 +-
.../sqllogictest/test_files/tpch/plans/q5.slt.part | 45 +-
.../sqllogictest/test_files/tpch/plans/q6.slt.part | 5 +-
.../sqllogictest/test_files/tpch/plans/q7.slt.part | 49 +-
.../sqllogictest/test_files/tpch/plans/q8.slt.part | 61 +-
.../sqllogictest/test_files/tpch/plans/q9.slt.part | 39 +-
datafusion/sqllogictest/test_files/union.slt | 48 +-
datafusion/sqllogictest/test_files/unnest.slt | 6 +-
datafusion/sqllogictest/test_files/window.slt | 177 +--
.../sqllogictest/test_files/window_limits.slt | 10 +-
datafusion/substrait/src/extensions.rs | 4 +-
datafusion/substrait/src/lib.rs | 3 +
.../src/logical_plan/consumer/expr/mod.rs | 4 +-
.../logical_plan/producer/substrait_producer.rs | 8 +-
datafusion/substrait/src/physical_plan/consumer.rs | 4 +-
.../tests/cases/roundtrip_physical_plan.rs | 34 +-
datafusion/wasmtest/src/lib.rs | 3 +
dev/changelog/51.0.0.md | 717 +++++++++
dev/release/README.md | 340 ++--
dev/release/add-branch-protection.sh | 160 ++
dev/update_function_docs.sh | 30 +
docs/requirements.txt | 2 +-
.../contributor-guide/development_environment.md | 2 +-
docs/source/contributor-guide/index.md | 8 +-
docs/source/contributor-guide/testing.md | 32 +-
docs/source/download.md | 26 +-
docs/source/library-user-guide/catalogs.md | 2 +-
docs/source/library-user-guide/upgrading.md | 229 ++-
docs/source/user-guide/configs.md | 237 +--
docs/source/user-guide/expressions.md | 4 +-
docs/source/user-guide/metrics.md | 1 +
docs/source/user-guide/sql/aggregate_functions.md | 30 +
docs/source/user-guide/sql/explain.md | 18 +-
579 files changed, 23539 insertions(+), 11034 deletions(-)
rename datafusion-examples/examples/{ =>
custom_data_source}/csv_json_opener.rs (89%)
rename datafusion-examples/examples/{ =>
custom_data_source}/csv_sql_streaming.rs (96%)
rename datafusion-examples/examples/{ =>
custom_data_source}/custom_datasource.rs (98%)
rename datafusion-examples/examples/{ =>
custom_data_source}/custom_file_casts.rs (98%)
rename datafusion-examples/examples/{ =>
custom_data_source}/custom_file_format.rs (94%)
rename datafusion-examples/examples/{ =>
custom_data_source}/default_column_values.rs (91%)
rename datafusion-examples/examples/{ =>
custom_data_source}/file_stream_provider.rs (91%)
create mode 100644 datafusion-examples/examples/custom_data_source/main.rs
rename datafusion-examples/examples/{ => data_io}/catalog.rs (98%)
rename datafusion-examples/examples/{ => data_io}/json_shredding.rs (99%)
create mode 100644 datafusion-examples/examples/data_io/main.rs
rename datafusion-examples/examples/{advanced_parquet_index.rs =>
data_io/parquet_advanced_index.rs} (98%)
rename datafusion-examples/examples/{ => data_io}/parquet_embedded_index.rs
(98%)
rename datafusion-examples/examples/{ => data_io}/parquet_encrypted.rs (96%)
rename datafusion-examples/examples/{ =>
data_io}/parquet_encrypted_with_kms.rs (99%)
rename datafusion-examples/examples/{ => data_io}/parquet_exec_visitor.rs (93%)
rename datafusion-examples/examples/{ => data_io}/parquet_index.rs (99%)
rename datafusion-examples/examples/{query-http-csv.rs =>
data_io/query_http_csv.rs} (91%)
rename datafusion-examples/examples/{ => data_io}/remote_catalog.rs (98%)
rename datafusion-examples/examples/{ => dataframe}/dataframe.rs (97%)
rename datafusion-examples/examples/{ => dataframe}/deserialize_to_struct.rs
(98%)
copy datafusion-examples/examples/{builtin_functions => dataframe}/main.rs
(63%)
copy datafusion-examples/examples/{builtin_functions =>
execution_monitoring}/main.rs (53%)
rename datafusion-examples/examples/{ =>
execution_monitoring}/memory_pool_execution_plan.rs (98%)
rename datafusion-examples/examples/{ =>
execution_monitoring}/memory_pool_tracking.rs (95%)
rename datafusion-examples/examples/{ => execution_monitoring}/tracing.rs (97%)
rename datafusion-examples/examples/{ => proto}/composed_extension_codec.rs
(95%)
copy datafusion-examples/examples/{builtin_functions => proto}/main.rs (63%)
rename datafusion-examples/examples/{ => query_planning}/analyzer_rule.rs (98%)
rename datafusion-examples/examples/{ => query_planning}/expr_api.rs (99%)
create mode 100644 datafusion-examples/examples/query_planning/main.rs
rename datafusion-examples/examples/{ => query_planning}/optimizer_rule.rs
(98%)
rename datafusion-examples/examples/{ => query_planning}/parse_sql_expr.rs
(96%)
rename datafusion-examples/examples/{ => query_planning}/plan_to_sql.rs (95%)
rename datafusion-examples/examples/{ => query_planning}/planner_api.rs (98%)
rename datafusion-examples/examples/{ => query_planning}/pruning.rs (97%)
rename datafusion-examples/examples/{ => query_planning}/thread_pools.rs (99%)
rename datafusion-examples/examples/{sql_analysis.rs => sql_ops/analysis.rs}
(98%)
rename datafusion-examples/examples/{sql_dialect.rs => sql_ops/dialect.rs}
(98%)
rename datafusion-examples/examples/{sql_frontend.rs => sql_ops/frontend.rs}
(99%)
copy datafusion-examples/examples/{builtin_functions => sql_ops}/main.rs (59%)
rename datafusion-examples/examples/{sql_query.rs => sql_ops/query.rs} (99%)
delete mode 100644 datafusion/common/src/pyarrow.rs
copy datafusion/core/tests/{catalog => catalog_listing}/mod.rs (96%)
create mode 100644
datafusion/core/tests/catalog_listing/pruned_partition_list.rs
create mode 100644
datafusion/core/tests/data/partitioned_table_arrow_stream/part=123/data.arrow
create mode 100644
datafusion/core/tests/data/partitioned_table_arrow_stream/part=456/data.arrow
create mode 100644 datafusion/core/tests/execution/register_arrow.rs
create mode 100644
datafusion/core/tests/user_defined/user_defined_async_scalar_functions.rs
create mode 100644 datafusion/datasource-arrow/tests/data/example_stream.arrow
create mode 100644
datafusion/datasource-arrow/tests/data/example_stream_corrupted_metadata_length.arrow
create mode 100644
datafusion/datasource-arrow/tests/data/example_stream_empty.arrow
create mode 100644
datafusion/expr-common/src/type_coercion/binary/tests/run_end_encoded.rs
create mode 100644 datafusion/expr/src/expr_rewriter/guarantees.rs
create mode 100644 datafusion/expr/src/predicate_bounds.rs
create mode 100644 datafusion/ffi/src/catalog_provider_list.rs
copy datafusion/ffi/tests/{ffi_integration.rs => ffi_catalog.rs} (50%)
create mode 100644
datafusion/functions-aggregate-common/src/noop_accumulator.rs
delete mode 100644 datafusion/optimizer/src/eliminate_one_union.rs
rename datafusion/optimizer/src/{eliminate_nested_union.rs =>
optimize_unions.rs} (90%)
delete mode 100644 datafusion/optimizer/src/simplify_expressions/guarantees.rs
delete mode 100644
datafusion/physical-optimizer/src/coalesce_async_exec_input.rs
create mode 100644
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs
create mode 100644 datafusion/spark/src/function/math/abs.rs
create mode 100644 datafusion/spark/src/function/math/trigonometry.rs
create mode 100644
datafusion/sqllogictest/src/engines/currently_executed_sql.rs
copy datafusion/sqllogictest/test_files/{tpch/drop_tables.slt.part =>
explain_analyze.slt} (69%)
copy datafusion/sqllogictest/test_files/{spark/string/concat.slt =>
slt_features.slt} (56%)
create mode 100644 dev/changelog/51.0.0.md
create mode 100755 dev/release/add-branch-protection.sh
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
