This is an automated email from the ASF dual-hosted git repository.
blaginin pushed a change to branch dorny-path-filter
in repository https://gitbox.apache.org/repos/asf/datafusion-sandbox.git
from 2c04105da Merge branch 'sandbox-main' into dorny-path-filter
add c2ba08717 Avoid skew in Roundrobin repartition (#18880)
add 76b9e123e Limit visibility of internal impl functions in function
crates (#18877)
add 3230048fa Add benchmark for array_has/array_has_all/array_has_any
(#18729)
add d041ba2ab chore(deps): bump maturin from 1.10.0 to 1.10.2 in /docs
(#18905)
add 91bd6966e chore(deps): bump taiki-e/install-action from 2.62.54 to
2.62.56 (#18899)
add faca8d7e9 chore(deps): bump indicatif from 0.18.0 to 0.18.3 (#18897)
add 710d14b0d chore(deps): bump tokio-util from 0.7.16 to 0.7.17 (#18898)
add 1cc9bcd52 FFI: return underlying trait type when converting from FFI
structs (#18672)
add 2dbd9c7e4 Support Non-Literal Expressions in Substrait VirtualTable
Values and Improve Round-Trip Robustness (#18866)
add 5c24b80cb chore(deps): bump indexmap from 2.12.0 to 2.12.1 (#18895)
add ec51b94ee chore(deps): bump aws-config from 1.8.7 to 1.8.11 (#18896)
add d24eb4a23 chore(deps): bump flate2 from 1.1.4 to 1.1.5 (#18900)
add 6856dc423 Add iter() method to `Extensions` (#18887)
add 6ea305ec4 feat: allow custom caching via logical node (#18688)
add 2db3aeaa5 chore: Enforce `clippy::needless_pass_by_value` globally
across the workspace (#18904)
add 39d4a3876 Consolidate external dependency examples (#18142) (#18747)
add c83b9d722 feat: add `array_slice` benchmark (#18879)
add 48cc4c8af Optimize planning for projected nested union (#18713)
add 838e1dea8 Update links in documentation to point at new example
locations (#18931)
add d5f1149bf chore(deps): bump taiki-e/install-action from 2.62.56 to
2.62.57 (#18927)
add c7b6afd20 chore(deps): bump actions/setup-python from 6.0.0 to 6.1.0
(#18925)
add 10706aeb3 Fix `map` function alias handling in SQL planner (#18914)
add 3ba7350d5 feat: Support recursive queries with a distinct 'UNION'
(#18254)
add 7cbb4432b minor: add builder setting
`NdJsonReadOptions::schema_infer_max_records` (#18920)
add 14f34f641 fix: preserve byte-size statistics in AggregateExec (#18885)
add 087174102 Add Kubeflow Trainer to known users (#18935)
add 8ab83425c Implement Substrait Support for `GROUPING SET CUBE` (#18798)
add 0366f0ef5 chore: unify common dependencies as workspace dependencies
(#18665)
add 54a78689c Fix bug where binary types were incorrectly being casted for
coercible signatures (#18750)
add 211877f1a Refactor approx_median signature & support f16 (#18647)
add 473cbdab8 Refactor `to_local_time()` signature away from user_defined
(#18707)
add 4eb293344 Refactor crypto functions code (#18664)
add 769f36732 fix: Track elapsed_compute metric for CSV scans (#18901)
add 8615ef4a9 chore(deps-dev): bump node-forge from 1.3.1 to 1.3.2 in
/datafusion/wasmtest/datafusion-wasm-app (#18958)
add 9f725d9c7 move projection handling into FileSource (#18627)
add b990987bd Add PhysicalOptimizerRule::optimize_plan to allow passing
more context into optimizer rules (#18739)
add c6f73636b Support LikeMatch, ILikeMatch, NotLikeMatch, NotILikeMatch
operators in protobuf serialization (#18961)
add fb14d7cb1 fix: Implement Substrait consumer support for like_match,
like_imatch, and negated variants (#18929)
add cb218dc38 chore: cargo fmt to fix CI (#18969)
add 7516b642a chore(deps): bump Swatinem/rust-cache from 2.8.1 to 2.8.2
(#18963)
add 53a89b20e chore(deps): bump taiki-e/install-action from 2.62.57 to
2.62.58 (#18964)
add 70edcc1f3 chore(deps): bump crate-ci/typos from 1.39.2 to 1.40.0
(#18965)
add 02a7181a7 [Minor] Refactor `traverse_chain` macro to function (#18951)
add 525835280 fix: Initialize CsvOptions::double_quote from
proto_opts.double_quote (#18967)
add 79257353b Enable clippy::allow_attributes lint for datafusion-catalog
(#18973)
add 36ec9f1de Feat/update rust 2024 edition (#18915)
add 487de9db6 chore(deps): bump taiki-e/install-action from 2.62.58 to
2.62.59 (#18978)
add 73562e8ea Simplify percentile_cont for 0/1 percentiles (#18837)
add 1f03ac502 chore: enforce clippy::allow_attributes for functions-*
crates (#18986)
add 2bfa64d28 chore: enforce clippy::allow_attributes for common crates
(#18988)
add e54eb4238 Fix predicate_rows_pruned & predicate_rows_matched metrics
(#18980)
add 4cfe20ffe Allocate a buffer of the correct length for
ScalarValue::FixedSizeBinary in ScalarValue::to_array_of_size (#18903)
add e4dcf0c85 Fix error planning aggregates with duplicated names in
select list (#18831)
add 96a5f21c4 optimizer: Support dynamic filter in `MIN/MAX` aggregates
(#18644)
add 81512da2b feat: Makes error macros hygienic (#18995)
add 1a9821e30 chore: remove `deny`s of `needless_pass_by_value` in
`lib.rs` files (#18996)
add c89cb7078 Add Explicit Error Handling for Unsupported SQL `FETCH`
Clause in Planner and CLI (#18691)
add 033ce4301 chore(deps): bump criterion from 0.7.0 to 0.8.0 (#19009)
add a3f1250d7 chore(deps): bump syn from 2.0.108 to 2.0.111 (#19011)
add 6425e40c3 chore(deps): bump taiki-e/install-action from 2.62.59 to
2.62.60 (#19012)
add 901aacaec chore: remove redundant clone code (#18997)
add 8a999e189 fix: `rstest` is a DEV dependency (#19014)
add a06073913 Update to `arrow`, `parquet` to `57.1.0` (#18820)
add 0f1133e9f fix: partition pruning stats pruning when multiple values
are present (#18923)
add 3b6e2fbfa fix: deprecate data_type_and_nullable and simplify API usage
(#18869)
add bc47cbe73 Add PGO documentation section to crate configuration (#18959)
add 5519b6145 deny on allow_attributes lint in physical-plan (#18983)
add 477053d83 Optimize planning / stop cloning Strings / Fields so much
(2-3% faster planning time) (#18415)
add e43e8bf13 Add additional test coverage of multi-value
PartitionPruningStats (#19021)
add fa3048969 perf: use `new_repeated` when converting scalar to an array
(#19018)
add 2c24deb0e Fix tpch benchmark harness (#19033)
add da36ad83c Fix data for tpch_csv and tpch_csv10 (#19034)
add ed836d739 chore: update group of 3 crates to rust 2024 edition (#19001)
add cab28a6a7 chore(deps-dev): bump express from 4.21.2 to 4.22.1 in
/datafusion/wasmtest/datafusion-wasm-app (#19040)
add 7b4593f36 Add upgrade guide for PhysicalOptimizerRule::optimize_plan
(#19030)
add a43f33db5 Allow repartitioning on files with ranges (#18948)
add 0490aec7e Support simplify not for physical expr (#18970)
add a1bb74b40 dev: Add typos check to the local `dev/rust_lint.sh` (#17863)
add 653caa143 doc: add `FilterExec` metrics to `user-guide/metrics.md`
(#19043)
add 5842e23c4 Implement FFI_PhysicalExpr and the structs it needs to
support it. (#18916)
add 4d86ae091 fix: pre-warm listing file statistics cache during listing
table creation (#18971)
add 9ee687276 chore(deps): bump actions/setup-node from 6.0.0 to 6.1.0
(#19063)
add e30ca2e45 chore(deps): bump taiki-e/install-action from 2.62.60 to
2.62.61 (#19062)
add 09fe307ca chore(deps): bump actions/stale from 10.1.0 to 10.1.1
(#19061)
add 2217c2b83 chore: merge make_array and spark array (#19006)
add b402dff94 chore(deps): bump actions/checkout from 6.0.0 to 6.0.1
(#19060)
add 11c9789b0 Add documentation example for `PartitionPruningStatistics`
(#19020)
add 9af6858ad Add `force_filter_selections` to restore `pushdown_filters`
behavior prior to parquet 57.1.0 upgrade (#19003)
add 91e623bf8 chore: upgrade expr and execution crates to rust 2024
edition (#19047)
add d4820d171 fix: log metadata differences when comparing physical and
logical schema (#19070)
add 0c6b65414 Implement FFI task context and task context provider (#18918)
add 6751f441f refactor: Refactor spark make_interval signature away from
user defined (#19027)
add 434a23b1b Fix: Align sort_merge_join filter output with join schema to
fix right-anti panic (#18800)
add 90aa47e19 Support Substrait Round-Trip of `EmptyRelation` Including
`produce_one_row` Semantics (#18842)
add 00e79523b chore(deps): bump taiki-e/install-action from 2.62.61 to
2.62.62 (#19081)
add 71fdad080 chore: enforce clippy::allow_attributes for datasource
crates (#19068)
add 63a8c659e common: Add hashing support for REE arrays (#18981)
add f22a3f395 Use `tpchgen-cli` to generate tpch data in bench.sh (#19035)
add 8dac8f1c4 Minor: fix link errors in docs (#19088)
add 388db0e52 Update aggregate probe to be locked only if skipping
aggregation (#18766)
add d150af7b2 Adds memory-bound DefaultListFilesCache (#18855)
add 107cb5e8f Fix function doc CI check (#19093)
add a3a020f6b Fix Schema Duplication Errors in Self‑Referential
INTERSECT/EXCEPT by Requalifying Input Sides (#18814)
add e01fa0ce4 Cut `Parquet` over to PhysicalExprAdapter, remove
`SchemaAdapter` (#18998)
add a33ae5e57 run cargo fmt to fix after #18998 (#19102)
add a5fc3c704 bench: set test_util as required feature for
aggregate_vectorized (#19101)
add 1844c88ed Update Committer / PMC list (#19105)
add 03c1a8df9 use ProjectionExprs:project_statistics in FileScanConfig
(#19094)
add f04a0eb26 Temporarily ignore test_cache_with_ttl_and_lru test (#19115)
add 482c6b8bd refactor: move human readable display utilities to
`datafusion-common` crate (#19080)
add 2b05b09b6 feat: Add builder API for CreateExternalTable to reduce
verbosity (#19066)
add 1800d150f Always remove unecessary software from github runners for
all jobs (fix intermittent out of space on runners) (#19122)
add c293854a3 fix: fix panic when lo is greater than hi (#19099)
add 340a28ca6 fix: escape underscores when simplifying `starts_with`
(#19077)
add cf9d07851 [datafusion-spark]: Refactor make_dt_interval's signature
away from user defined (#19083)
add 674600782 feat(spark): Implement Spark functions `url_encode`,
`url_decode` and `try_url_decode` (#17399)
add c479deeb6 fix deprecation notes with incorrect versions from #13083
(#19135)
add 944f7f2f2 Run the examples in the new format (#18946)
add cb2f3d2a8 Add constant expression evaluator to physical expression
simplifier (#19130)
add fc6d0a428 Fix shuffle function to report nullability correctly (#19184)
add 41f713758 chore: enforce clippy::allow_attributes for physical crates
(#19185)
add 3a0064d82 fix: custom nullability for length (#19175) (#19182)
add 998f534aa Update 5 crates to rust 2024 edition (#19091)
add 215cdffb6 Coalesce batches inside hash join, reuse indices buffer
(#18972)
add dd35f7c65 slt test coverage for `CASE` exprs with constant value
lookup tables (#19143)
add 4c3e3c1c6 Fix fmt after logical conflict (#19208)
add e8384fbbf chore: Add TPCDS benchmarks (#19138)
add adaed426a Arc partition values in TableSchema (#19137)
add 0812d2731 feat: Move DefaultMetadataCache into its own module (#19125)
add 662a3bad6 fix: inverted null_percent logic in in_list benchmark
(#19204)
add cde6dfab1 Add sorted data benchmark. (#19042)
add f2b476f39 Refactor PhysicalExprSimplfier to &self instead of &mut self
(#19212)
add 3fabee770 Revert adding PhysicalOptimizerRule::optimize_plan (#19186)
add 4504d941b chore(deps): bump uuid from 1.18.1 to 1.19.0 (#19199)
add af3ab52a7 chore(deps): bump async-compression from 0.4.34 to 0.4.35
(#19201)
add dcc09c345 chore(deps): bump taiki-e/install-action from 2.62.62 to
2.62.63 (#19198)
add 7d958b040 chore(deps): bump tracing-subscriber from 0.3.20 to 0.3.22
(#19200)
add 71a698285 chore(deps): bump wasm-bindgen-test from 0.3.55 to 0.3.56
(#19202)
add 55a38d4a6 feat: Add `remove_optimizer_rule` to `SessionContext`
(#19209)
add ab7fe0eb5 bench: add dedicated Utf8View benchmarks for InList (#19211)
add 83736efc4 Fix PruningPredicate interaction with
DynamicFilterPhysicalExpr that references partition columns (#19129)
add ad9b779b6 Implement physical and logical codecs in FFI (#19079)
add 4ddee14ee refactor: Refactor spark width bucket signature away from
user defined (#19065)
add 2626fc202 fix: Ensure column names do not change with
`expand_views_at_output` (#19019)
add 4fb36b23d fix: bitmap_count should report nullability correctly
(#19195)
add 1a6df66e2 fix: bit_count function to report nullability correctly
(#19197)
add ea9f35412 fix: derive custom nullability for spark `bit_shift` (#19222)
add 7ea5066fe Sort Merge Join: Reduce batch concatenation, use
`BatchCoalescer`, new benchmarks (TPC-H Q21 SMJ up to ~4000x faster) (#18875)
add a30cf3709 Add relation planner extension support to customize SQL
planning (#17843)
add 20870c18a feat: integrate batch coalescer with repartition exec
(#19002)
add c0e8bb501 Push down InList or hash table references from HashJoinExec
depending on the size of the build side (#18393)
add 21a16e44b Add additional tests for InListExpr (#19050)
add 7ac40d796 chore(deps): bump taiki-e/install-action from 2.62.63 to
2.62.64 (#19226)
add dc786138c Move partition handling out of PhysicalExprAdapter (#19128)
add cd160de0b fix: spark elt custom nullability (#19207)
add dc6a71235 Use strum in the examples (#19126) (#19205)
add d85595e03 perf: optimize CASE WHEN lookup table (2.5-22.5 times
faster) (#18183)
add e8a0829b7 Allow Logical expression ScalarVariable to represent an
extension type or metadata (#18243)
add dc4e3ab47 feat: Implement the `statistics_cache` function (#19054)
add ca67edcf9 [Proto]: Serialization support for `AsyncFuncExec` (#19118)
add 26af02a48 chore: add test case for decimal overflow (#19255)
add 79a2f5e11 chore(deps): bump taiki-e/install-action from 2.62.64 to
2.62.65 (#19251)
add 2a08013af chore: update 6 crates to rust edition 2024 (#19196)
add 021188e71 Push down projection expressions into ParquetOpener (#19111)
add 321401ce6 fix: `array_remove`/`array_remove_n`/`array_remove_all` not
using the same nullability as the input (#19259)
add c1aa1b530 Track column sizes in Statistics; propagate through
projections (#19113)
add 39a1f7575 Implement FFI_Session (#19223)
add 89a9c9d06 Feat: Add an option for fast tests by gating slow tests to
extended_tests feature (#19237)
add 5496c3043 fix: typo in sql/ddl (#19276)
add e91493566 chore: enforce clippy::allow_attributes for 7 crates (#19133)
add bde16083a feat: Preserve File Partitioning From File Scans (#19124)
add 58345f94a Improve ProjectionExpr documentation and comments (#19263)
add 85d8a88c1 add specialized InList implementations for common scalar
types (#18832)
add c8add5c1d Update README occording to the new examples (#18529) (#19257)
add 044a4a736 Add make_time function (#19183)
add 33b3f0216 dev: Add CI doc prettier check to local `rust_lint.sh`
(#19254)
add a3b3eb558 bug: Eliminate dead round-robin insertion in enforce
distribution (#19132)
add f9c030a5f Automatically download tpcds benchmark data to the right
place (#19244)
add 86011519b fix: flaky cache test (#19140)
add 10c933cc7 [datafusion-spark]: Refactor hex's signature away from
user_defined (#19235)
add 06f27e94d fix : correct nullability propagation for spark.bitwise_not
(#19224)
add aebae990d fix: Add custom nullability for Spark ILIKE function (#19206)
add 96ddd558f fix: derive custom nullability for spark `map_from_arrays`
(#19275)
add 18e4a0caf fix: derive custom nullability for spark map_from_entries
(#19274)
add 2c6f6d369 fix: derive custom nullable for spark `make_dt_interval`
(#19236)
add c4ca946ca fix: derive custome nullable for the spark last_day (#19232)
add 49cfee05c fix: derive custom nullable for spark `date_sub` (#19225)
add 845894641 added custom nullability for char (#19268)
add 5a01e6864 Update to_date udf function to support a consistent set of
argument types (#19134)
add d61f1a715 replace HashTableLookupExpr with lit(true) in proto
serialization (#19300)
add fedddbcae chore: fix return_field_from_args doc (#19307)
add 4088192fc chore: enforce clippy::allow_attributes for
spark,sql,sustrait (#19309)
add 94d9c8e68 Simplify make_date & fix null handling (#19296)
add 033116e8a fix: Fix a few minor issues with join metrics (#19283)
add 9163feed6 fix: derive nullability for spark `bit_get` (#19220)
add 16fa35fa2 Allow base64 encoding of fixedsizebinary arrays (#18950)
add befaf93a1 chore: update 11 crates to Rust 2024 edition (#19258)
add ec11f4250 feat: Add constant column extraction and rewriting for
projections in ParquetOpener (#19136)
add dbf9265e5 fix: pow() with integer base and negative float exponent
returns error (#19303)
add 2bea79689 Minor: remove unnecessary unit tests for fixed size binary
(#19318)
add 899a76223 Populate partition column statistics for PartitionedFile
(#19284)
add 7c05b208e refactor: move metrics module to `datafusion-common` crate
(#19247)
add 9d4fe1589 Add library user guide for extending SQL syntax (#19265)
add b828fbfa5 chore(deps): bump taiki-e/install-action from 2.62.65 to
2.62.67 (#19295)
add 58377bf2e chore(deps): bump ctor from 0.6.1 to 0.6.3 (#19328)
add fc8824011 Refactor `power()` signature away from user defined (#18968)
add 933657e2c feat: Support sliding window queries for MedianAccumulator
by implementing `retract_batch` (#19278)
add 02c647ae1 chore: enforce `clippy::allow_attributes` for optimizer and
macros (#19310)
add 3a67d86c9 fix(concat): correct nullability inference (nullable only if
all arguments nullable) (#19189)
add 266a7d060 chore(deps): bump taiki-e/install-action from 2.62.67 to
2.63.3 (#19349)
add ea8087d77 chore(deps): bump clap from 4.5.50 to 4.5.53 (#19326)
add 199cf06d2 chore(deps): bump insta from 1.43.2 to 1.44.3 (#19327)
add 79cfe8e92 Add runtime config options for `list_files_cache_limit` and
`list_files_cache_ttl` (#19108)
add 89af69411 remove repartition exec from coalesce batches optimizer
(#19239)
add be0cf05f9 minor: cleanup unnecessary config in `decimal.slt` (#19352)
add c53a44891 Fix panic for `GROUPING SETS(())` and handle empty-grouping
aggregates (#19252)
add 775277ae3 Update datafusion-core crate to Rust 2024 edition (#19332)
add efd793ba3 Minor: clean up titles and links n extending operators and
optimizer pages (#19317)
add 1f2671628 Update 4 crates to rust 2024 edition (#19357)
add 50d20dd55 preserve Field metadata in first_value/last_value (#19335)
add 3a41cc607 Establish the high level API for sort pushdown and the
optimizer rule and support reverse files and row groups (#19064)
add b806117bf Fix flaky SpillPool channel test by synchronizing reader and
writer tasks (#19110)
add 2b67ba81c [minor] Upgrade rust version (#19363)
add 401345b6f Minor: fix cargo fmt (#19368)
add 7900cd6e9 chore: enforce clippy::allow_attributes for proto, pruning,
session (#19350)
add 59dcc362c Update remaining crates to rust 2024 edition (#19361)
add 2e95f2673 Minor: Make `ProjectionExpr::new` easier to use with
constants (#19343)
add fe11ad6b4 fix: Added nullable return from date_add(#19151) (#19229)
add 3879b2450 fix: spark sha1 nullability reporting (#19242)
add 1e4bd7565 Feat: DefaultListFilesCache prefix-aware for partition
pruning optimization (#19298)
add 14cd71eab feat: add compression level configuration for JSON/CSV
writers (#18954)
add 4e7bba490 Extend in_list benchmark coverage (#19376)
add d68b629dc Add hashing microbenchmark `with_hashes` (#19373)
add cb3fa1c2c fix: derive custom nullability for the spark `next_day`
(#19253)
add 0a0416dbf Move `newlines_in_values` from `FileScanConfig` to
`CsvSource` (#19313)
add 2c9077a8b [datafusion-cli] Implement average LIST duration for object
store profiling (#19127)
add 8120da40a chore(deps): bump taiki-e/install-action from 2.63.3 to
2.64.0 (#19382)
add ead820980 update insta snapshots (#19381)
add d59ebac9f feat(spark): implement Spark `try_sum` function (#18569)
add e5c4c9702 Merge branch 'main' into sandbox-main
add e6dcd5210 Merge branch 'main' into dorny-path-filter
No new revisions were added by this update.
Summary of changes:
.github/actions/setup-builder/action.yaml | 14 +
.github/workflows/audit.yml | 4 +-
.github/workflows/dependencies.yml | 4 +-
.github/workflows/dev.yml | 34 +-
.github/workflows/docs.yaml | 6 +-
.github/workflows/docs_pr.yaml | 4 +-
.github/workflows/extended.yml | 8 +-
.github/workflows/labeler.yml | 2 +-
.github/workflows/large_files.yml | 2 +-
.github/workflows/rust.yml | 99 +-
.github/workflows/stale.yml | 2 +-
Cargo.lock | 492 +++--
Cargo.toml | 33 +-
benchmarks/Cargo.toml | 2 +-
benchmarks/README.md | 143 +-
benchmarks/bench.sh | 283 ++-
.../regen.sh => benchmarks/compare_tpcds.sh | 41 +-
.../regen.sh => benchmarks/compare_tpch.sh | 39 +-
.../clickbench/queries/{ => sorted_data}/q0.sql | 3 +-
benchmarks/src/bin/dfbench.rs | 8 +-
benchmarks/src/bin/external_aggr.rs | 6 +-
benchmarks/src/bin/mem_profile.rs | 27 +-
benchmarks/src/bin/tpch.rs | 65 -
benchmarks/src/cancellation.rs | 10 +-
benchmarks/src/clickbench.rs | 119 +-
benchmarks/src/h2o.rs | 4 +-
benchmarks/src/hj.rs | 2 +-
benchmarks/src/imdb/convert.rs | 2 +-
benchmarks/src/imdb/run.rs | 8 +-
benchmarks/src/lib.rs | 2 +
benchmarks/src/nlj.rs | 6 +-
benchmarks/src/smj.rs | 524 +++++
benchmarks/src/sort_tpch.rs | 4 +-
.../tests/catalog => benchmarks/src/tpcds}/mod.rs | 3 +-
benchmarks/src/tpcds/run.rs | 356 +++
benchmarks/src/tpch/convert.rs | 162 --
benchmarks/src/tpch/mod.rs | 3 -
benchmarks/src/tpch/run.rs | 12 +-
benchmarks/src/util/memory.rs | 2 +-
benchmarks/src/util/options.rs | 2 +-
benchmarks/src/util/run.rs | 2 +-
ci/scripts/doc_prettier_check.sh | 57 +
ci/scripts/rust_example.sh | 30 +-
ci/scripts/{rust_docs.sh => typos_check.sh} | 5 +-
datafusion-cli/Cargo.toml | 4 +-
datafusion-cli/examples/cli-session-context.rs | 2 +-
datafusion-cli/src/catalog.rs | 50 +-
datafusion-cli/src/cli_context.rs | 2 +-
datafusion-cli/src/command.rs | 2 +-
datafusion-cli/src/exec.rs | 27 +-
datafusion-cli/src/functions.rs | 120 +-
datafusion-cli/src/helper.rs | 10 +-
datafusion-cli/src/highlighter.rs | 4 +-
datafusion-cli/src/main.rs | 142 +-
datafusion-cli/src/object_storage.rs | 51 +-
datafusion-cli/src/object_storage/instrumented.rs | 114 +-
datafusion-cli/src/print_format.rs | 66 +-
datafusion-cli/src/print_options.rs | 8 +-
datafusion-cli/tests/cli_integration.rs | 34 +-
...errides@explain_plan_environment_overrides.snap | 1 -
.../tests/snapshots/[email protected] | 2 +-
datafusion-cli/tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
datafusion-cli/tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/cli_quick_test@batch_size.snap | 2 +-
.../cli_quick_test@default_explain_plan.snap | 1 -
.../tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
datafusion-examples/Cargo.toml | 16 +-
datafusion-examples/README.md | 229 +-
.../examples/builtin_functions/date_time.rs | 7 +-
.../examples/builtin_functions/function_factory.rs | 2 +-
.../examples/builtin_functions/main.rs | 73 +-
.../examples/builtin_functions/regexp.rs | 93 +-
.../examples/custom_data_source/csv_json_opener.rs | 30 +-
.../custom_data_source/custom_datasource.rs | 9 +-
.../custom_data_source/custom_file_casts.rs | 31 +-
.../custom_data_source/custom_file_format.rs | 6 +-
.../custom_data_source/default_column_values.rs | 142 +-
.../custom_data_source/file_stream_provider.rs | 6 +-
.../examples/custom_data_source/main.rs | 115 +-
datafusion-examples/examples/data_io/catalog.rs | 4 +-
.../examples/data_io/json_shredding.rs | 114 +-
datafusion-examples/examples/data_io/main.rs | 128 +-
.../examples/data_io/parquet_advanced_index.rs | 16 +-
.../examples/data_io/parquet_embedded_index.rs | 30 +-
.../examples/data_io/parquet_encrypted.rs | 4 +-
.../examples/data_io/parquet_exec_visitor.rs | 23 +-
.../examples/data_io/parquet_index.rs | 12 +-
.../examples/data_io/remote_catalog.rs | 2 +-
.../examples/dataframe/dataframe.rs | 41 +-
datafusion-examples/examples/dataframe/main.rs | 75 +-
.../examples/execution_monitoring/main.rs | 89 +-
.../memory_pool_execution_plan.rs | 3 +-
.../examples/execution_monitoring/tracing.rs | 14 +-
.../{dataframe-to-s3.rs => dataframe_to_s3.rs} | 22 +-
.../examples/external_dependency/main.rs | 84 +
.../{query-aws-s3.rs => query_aws_s3.rs} | 13 +-
.../examples/ffi/ffi_module_interface/Cargo.toml | 2 +-
.../examples/ffi/ffi_module_interface/src/lib.rs | 3 +-
.../examples/ffi/ffi_module_loader/Cargo.toml | 3 +-
.../examples/ffi/ffi_module_loader/src/main.rs | 10 +-
datafusion-examples/examples/flight/main.rs | 80 +-
datafusion-examples/examples/flight/server.rs | 3 +-
datafusion-examples/examples/flight/sql_server.rs | 4 +-
.../examples/proto/composed_extension_codec.rs | 2 +-
datafusion-examples/examples/proto/main.rs | 69 +-
.../examples/query_planning/analyzer_rule.rs | 4 +-
.../examples/query_planning/expr_api.rs | 31 +-
.../examples/query_planning/main.rs | 102 +-
.../examples/query_planning/optimizer_rule.rs | 2 +-
.../examples/query_planning/plan_to_sql.rs | 2 +-
.../examples/query_planning/pruning.rs | 1 +
.../examples/relation_planner/main.rs | 121 ++
.../examples/relation_planner/match_recognize.rs | 406 ++++
.../examples/relation_planner/pivot_unpivot.rs | 567 +++++
.../examples/relation_planner/table_sample.rs | 845 ++++++++
datafusion-examples/examples/sql_ops/analysis.rs | 2 +-
datafusion-examples/examples/sql_ops/frontend.rs | 2 +-
datafusion-examples/examples/sql_ops/main.rs | 77 +-
datafusion-examples/examples/sql_ops/query.rs | 10 +-
datafusion-examples/examples/udf/advanced_udaf.rs | 4 +-
datafusion-examples/examples/udf/advanced_udf.rs | 6 +-
datafusion-examples/examples/udf/advanced_udwf.rs | 49 +-
datafusion-examples/examples/udf/async_udf.rs | 31 +-
datafusion-examples/examples/udf/main.rs | 101 +-
datafusion-examples/examples/udf/simple_udtf.rs | 9 +-
datafusion-examples/examples/udf/simple_udwf.rs | 48 +-
datafusion/catalog-listing/src/config.rs | 2 +-
datafusion/catalog-listing/src/helpers.rs | 23 +-
datafusion/catalog-listing/src/mod.rs | 5 +-
datafusion/catalog-listing/src/options.rs | 4 +-
datafusion/catalog-listing/src/table.rs | 155 +-
datafusion/catalog/src/async.rs | 10 +-
datafusion/catalog/src/catalog.rs | 2 +-
datafusion/catalog/src/default_table_source.rs | 2 +-
datafusion/catalog/src/information_schema.rs | 24 +-
datafusion/catalog/src/lib.rs | 4 +-
datafusion/catalog/src/listing_schema.rs | 23 +-
datafusion/catalog/src/memory/schema.rs | 2 +-
datafusion/catalog/src/memory/table.rs | 6 +-
datafusion/catalog/src/schema.rs | 6 +-
datafusion/catalog/src/stream.rs | 2 +-
datafusion/catalog/src/streaming.rs | 6 +-
datafusion/catalog/src/table.rs | 2 +-
datafusion/catalog/src/view.rs | 2 +-
datafusion/common-runtime/src/common.rs | 8 +-
datafusion/common-runtime/src/lib.rs | 5 +-
datafusion/common-runtime/src/trace_utils.rs | 2 +-
datafusion/common/Cargo.toml | 7 +-
datafusion/common/benches/with_hashes.rs | 209 ++
datafusion/common/src/cast.rs | 8 +-
datafusion/common/src/config.rs | 155 +-
datafusion/common/src/cse.rs | 8 +-
datafusion/common/src/datatype.rs | 114 +-
datafusion/common/src/dfschema.rs | 157 +-
datafusion/common/src/display/human_readable.rs | 139 ++
datafusion/common/src/display/mod.rs | 1 +
datafusion/common/src/error.rs | 77 +-
datafusion/common/src/file_options/csv_writer.rs | 17 +
datafusion/common/src/file_options/json_writer.rs | 18 +-
datafusion/common/src/file_options/mod.rs | 2 +-
.../common/src/file_options/parquet_writer.rs | 42 +-
datafusion/common/src/format.rs | 6 +-
datafusion/common/src/hash_utils.rs | 248 ++-
datafusion/common/src/instant.rs | 2 +-
datafusion/common/src/lib.rs | 32 +-
datafusion/common/src/metadata.rs | 14 +-
datafusion/common/src/nested_struct.rs | 6 +-
datafusion/common/src/param_value.rs | 2 +-
datafusion/common/src/pruning.rs | 145 +-
datafusion/common/src/rounding.rs | 2 +-
datafusion/common/src/scalar/cache.rs | 6 +-
datafusion/common/src/scalar/mod.rs | 238 +-
datafusion/common/src/stats.rs | 285 ++-
datafusion/common/src/test_util.rs | 54 +-
datafusion/common/src/tree_node.rs | 54 +-
datafusion/common/src/types/builtin.rs | 15 +
datafusion/common/src/types/native.rs | 14 +-
datafusion/common/src/utils/memory.rs | 200 +-
datafusion/common/src/utils/mod.rs | 22 +-
datafusion/core/Cargo.toml | 18 +-
datafusion/core/benches/aggregate_query_sql.rs | 1 +
datafusion/core/benches/csv_load.rs | 1 +
datafusion/core/benches/data_utils/mod.rs | 3 +-
datafusion/core/benches/dataframe.rs | 1 +
datafusion/core/benches/distinct_query_sql.rs | 6 +-
datafusion/core/benches/filter_query_sql.rs | 2 +-
datafusion/core/benches/map_query_sql.rs | 4 +-
datafusion/core/benches/math_query_sql.rs | 1 +
datafusion/core/benches/parquet_query_sql.rs | 4 +-
datafusion/core/benches/physical_plan.rs | 3 +-
.../core/benches/preserve_file_partitioning.rs | 838 +++++++
datafusion/core/benches/push_down_filter.rs | 6 +-
datafusion/core/benches/scalar.rs | 2 +-
datafusion/core/benches/sort.rs | 66 +-
datafusion/core/benches/sort_limit_query_sql.rs | 4 +-
datafusion/core/benches/spm.rs | 6 +-
datafusion/core/benches/sql_planner.rs | 11 +-
datafusion/core/benches/sql_planner_extended.rs | 2 +-
datafusion/core/benches/sql_query_with_io.rs | 6 +-
datafusion/core/benches/struct_query_sql.rs | 2 +-
datafusion/core/benches/topk_aggregate.rs | 11 +-
datafusion/core/benches/window_query_sql.rs | 1 +
datafusion/core/src/bin/print_functions_docs.rs | 11 +-
datafusion/core/src/dataframe/mod.rs | 54 +-
datafusion/core/src/dataframe/parquet.rs | 2 +-
datafusion/core/src/datasource/dynamic_file.rs | 2 +-
datafusion/core/src/datasource/empty.rs | 2 +-
datafusion/core/src/datasource/file_format/avro.rs | 40 +-
datafusion/core/src/datasource/file_format/csv.rs | 159 +-
datafusion/core/src/datasource/file_format/json.rs | 68 +-
datafusion/core/src/datasource/file_format/mod.rs | 31 +-
.../core/src/datasource/file_format/options.rs | 8 +-
.../core/src/datasource/file_format/parquet.rs | 48 +-
datafusion/core/src/datasource/listing/table.rs | 425 +---
.../core/src/datasource/listing_table_factory.rs | 252 ++-
datafusion/core/src/datasource/memory_test.rs | 11 +-
datafusion/core/src/datasource/mod.rs | 124 +-
.../core/src/datasource/physical_plan/avro.rs | 96 +-
.../core/src/datasource/physical_plan/csv.rs | 182 +-
.../core/src/datasource/physical_plan/json.rs | 32 +-
.../core/src/datasource/physical_plan/mod.rs | 8 +-
.../core/src/datasource/physical_plan/parquet.rs | 324 ++-
datafusion/core/src/datasource/view_test.rs | 41 +-
datafusion/core/src/execution/context/mod.rs | 179 +-
datafusion/core/src/execution/context/parquet.rs | 6 +-
datafusion/core/src/execution/session_state.rs | 192 +-
.../core/src/execution/session_state_defaults.rs | 6 +-
datafusion/core/src/lib.rs | 8 +-
datafusion/core/src/physical_planner.rs | 434 +++-
datafusion/core/src/prelude.rs | 2 +-
datafusion/core/src/test/mod.rs | 19 +-
datafusion/core/src/test/object_store.rs | 10 +-
datafusion/core/src/test_util/mod.rs | 75 +-
datafusion/core/src/test_util/parquet.rs | 13 +-
datafusion/core/tests/catalog/memory.rs | 10 +-
.../tests/catalog_listing/pruned_partition_list.rs | 4 +-
datafusion/core/tests/config_from_env.rs | 60 +-
datafusion/core/tests/custom_sources_cases/mod.rs | 9 +-
.../provider_filter_pushdown.rs | 14 +-
.../core/tests/custom_sources_cases/statistics.rs | 2 +
.../core/tests/data/recursive_cte/closure.csv | 6 +
.../core/tests/dataframe/dataframe_functions.rs | 12 +-
datafusion/core/tests/dataframe/describe.rs | 2 +-
datafusion/core/tests/dataframe/mod.rs | 919 ++++----
.../core/tests/datasource/object_store_access.rs | 53 +
datafusion/core/tests/execution/coop.rs | 24 +-
.../core/tests/execution/datasource_split.rs | 2 +-
datafusion/core/tests/execution/logical_plan.rs | 2 +-
datafusion/core/tests/expr_api/mod.rs | 6 +-
datafusion/core/tests/expr_api/parse_sql_expr.rs | 2 +-
datafusion/core/tests/expr_api/simplification.rs | 22 +-
datafusion/core/tests/fifo/mod.rs | 8 +-
datafusion/core/tests/fuzz.rs | 5 +-
datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs | 37 +-
.../aggregation_fuzzer/context_generator.rs | 2 +-
.../aggregation_fuzzer/data_generator.rs | 6 +-
.../tests/fuzz_cases/aggregation_fuzzer/fuzzer.rs | 4 +-
.../fuzz_cases/aggregation_fuzzer/query_builder.rs | 2 +-
.../tests/fuzz_cases/distinct_count_string_fuzz.rs | 2 +-
.../core/tests/fuzz_cases/equivalence/ordering.rs | 10 +-
.../tests/fuzz_cases/equivalence/projection.rs | 8 +-
.../tests/fuzz_cases/equivalence/properties.rs | 6 +-
.../core/tests/fuzz_cases/equivalence/utils.rs | 8 +-
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 100 +-
datafusion/core/tests/fuzz_cases/limit_fuzz.rs | 2 +-
datafusion/core/tests/fuzz_cases/merge_fuzz.rs | 2 +-
datafusion/core/tests/fuzz_cases/mod.rs | 6 +
datafusion/core/tests/fuzz_cases/pruning.rs | 4 +-
.../tests/fuzz_cases/record_batch_generator.rs | 20 +-
datafusion/core/tests/fuzz_cases/sort_fuzz.rs | 4 +-
.../fuzz_cases/sort_preserving_repartition_fuzz.rs | 11 +-
.../core/tests/fuzz_cases/sort_query_fuzz.rs | 32 +-
.../spilling_fuzz_in_memory_constrained_env.rs | 42 +-
datafusion/core/tests/fuzz_cases/window_fuzz.rs | 51 +-
.../memory_limit/memory_limit_validation/utils.rs | 12 +-
datafusion/core/tests/memory_limit/mod.rs | 28 +-
.../tests/memory_limit/repartition_mem_limit.rs | 15 +-
datafusion/core/tests/optimizer/mod.rs | 6 +-
datafusion/core/tests/parquet/custom_reader.rs | 6 +-
datafusion/core/tests/parquet/encryption.rs | 5 +-
.../core/tests/parquet/external_access_plan.rs | 10 +-
datafusion/core/tests/parquet/file_statistics.rs | 28 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 23 +-
datafusion/core/tests/parquet/mod.rs | 31 +-
datafusion/core/tests/parquet/page_pruning.rs | 8 +-
datafusion/core/tests/parquet/schema_adapter.rs | 409 +---
datafusion/core/tests/parquet/schema_coercion.rs | 7 +-
datafusion/core/tests/parquet/utils.rs | 11 +-
.../physical_optimizer/aggregate_statistics.rs | 6 +-
.../combine_partial_final_agg.rs | 8 +-
.../physical_optimizer/enforce_distribution.rs | 830 ++++---
.../tests/physical_optimizer/enforce_sorting.rs | 33 +-
.../enforce_sorting_monotonicity.rs | 2 +-
.../physical_optimizer/filter_pushdown/mod.rs | 1247 +++++++++--
.../physical_optimizer/filter_pushdown/util.rs | 22 +-
.../tests/physical_optimizer/join_selection.rs | 25 +-
.../tests/physical_optimizer/limit_pushdown.rs | 148 +-
.../limited_distinct_aggregation.rs | 6 +-
datafusion/core/tests/physical_optimizer/mod.rs | 6 +
.../physical_optimizer/partition_statistics.rs | 326 ++-
.../physical_optimizer/projection_pushdown.rs | 242 +--
.../core/tests/physical_optimizer/pushdown_sort.rs | 672 ++++++
.../replace_with_order_preserving_variants.rs | 10 +-
.../tests/physical_optimizer/sanity_checker.rs | 6 +-
.../core/tests/physical_optimizer/test_utils.rs | 96 +-
.../tests/physical_optimizer/window_optimize.rs | 4 +-
.../schema_adapter_integration_tests.rs | 537 ++++-
datafusion/core/tests/sql/aggregates/basic.rs | 8 +-
datafusion/core/tests/sql/aggregates/dict_nulls.rs | 24 +-
datafusion/core/tests/sql/aggregates/mod.rs | 12 +-
datafusion/core/tests/sql/explain_analyze.rs | 59 +-
datafusion/core/tests/sql/joins.rs | 79 +-
datafusion/core/tests/sql/mod.rs | 7 +-
datafusion/core/tests/sql/path_partition.rs | 38 +-
datafusion/core/tests/sql/runtime_config.rs | 92 +
datafusion/core/tests/sql/select.rs | 10 +-
datafusion/core/tests/tpc-ds/30.sql | 4 +-
datafusion/core/tests/tracing/asserting_tracer.rs | 2 +-
.../core/tests/tracing/traceable_object_store.rs | 4 +-
datafusion/core/tests/user_defined/expr_planner.rs | 18 +-
.../core/tests/user_defined/insert_operation.rs | 4 +-
datafusion/core/tests/user_defined/mod.rs | 3 +
.../core/tests/user_defined/relation_planner.rs | 527 +++++
.../tests/user_defined/user_defined_aggregates.rs | 81 +-
.../user_defined_async_scalar_functions.rs | 2 +-
.../core/tests/user_defined/user_defined_plan.rs | 73 +-
.../user_defined/user_defined_scalar_functions.rs | 69 +-
.../user_defined/user_defined_table_functions.rs | 16 +-
.../user_defined/user_defined_window_functions.rs | 303 +--
datafusion/datasource-arrow/src/file_format.rs | 34 +-
datafusion/datasource-arrow/src/mod.rs | 3 +-
datafusion/datasource-arrow/src/source.rs | 195 +-
.../src/avro_to_arrow/arrow_array_reader.rs | 58 +-
.../datasource-avro/src/avro_to_arrow/schema.rs | 16 +-
datafusion/datasource-avro/src/file_format.rs | 12 +-
datafusion/datasource-avro/src/mod.rs | 2 +-
datafusion/datasource-avro/src/source.rs | 53 +-
datafusion/datasource-csv/Cargo.toml | 2 +-
datafusion/datasource-csv/src/file_format.rs | 17 +-
datafusion/datasource-csv/src/mod.rs | 14 +-
datafusion/datasource-csv/src/source.rs | 95 +-
datafusion/datasource-json/src/file_format.rs | 17 +-
datafusion/datasource-json/src/mod.rs | 3 +-
datafusion/datasource-json/src/source.rs | 51 +-
datafusion/datasource-parquet/src/access_plan.rs | 18 +-
datafusion/datasource-parquet/src/file_format.rs | 37 +-
datafusion/datasource-parquet/src/metadata.rs | 209 +-
datafusion/datasource-parquet/src/mod.rs | 4 +-
datafusion/datasource-parquet/src/opener.rs | 1038 ++++++---
datafusion/datasource-parquet/src/page_filter.rs | 18 +-
datafusion/datasource-parquet/src/reader.rs | 7 +-
datafusion/datasource-parquet/src/row_filter.rs | 287 ++-
.../datasource-parquet/src/row_group_filter.rs | 11 +-
datafusion/datasource-parquet/src/sort.rs | 407 ++++
datafusion/datasource-parquet/src/source.rs | 326 ++-
datafusion/datasource/Cargo.toml | 12 +-
.../benches/split_groups_by_statistics.rs | 2 +-
datafusion/datasource/src/decoder.rs | 28 +-
datafusion/datasource/src/display.rs | 2 +-
datafusion/datasource/src/file.rs | 95 +-
datafusion/datasource/src/file_compression_type.rs | 60 +-
datafusion/datasource/src/file_format.rs | 2 +-
datafusion/datasource/src/file_groups.rs | 379 +++-
datafusion/datasource/src/file_scan_config.rs | 1446 +++++--------
datafusion/datasource/src/file_sink_config.rs | 4 +-
datafusion/datasource/src/file_stream.rs | 143 +-
datafusion/datasource/src/memory.rs | 33 +-
datafusion/datasource/src/mod.rs | 153 +-
datafusion/datasource/src/projection.rs | 630 ++++++
datafusion/datasource/src/schema_adapter.rs | 15 +-
datafusion/datasource/src/sink.rs | 6 +-
datafusion/datasource/src/source.rs | 52 +-
datafusion/datasource/src/statistics.rs | 12 +-
datafusion/datasource/src/table_schema.rs | 107 +-
datafusion/datasource/src/test_util.rs | 41 +-
datafusion/datasource/src/url.rs | 287 ++-
datafusion/datasource/src/write/demux.rs | 16 +-
datafusion/datasource/src/write/mod.rs | 25 +-
datafusion/datasource/src/write/orchestration.rs | 24 +-
datafusion/doc/src/lib.rs | 11 +-
datafusion/doc/src/udf.rs | 4 +-
datafusion/execution/Cargo.toml | 2 +-
datafusion/execution/src/cache/cache_manager.rs | 152 +-
datafusion/execution/src/cache/cache_unit.rs | 842 +-------
.../{cache_unit.rs => file_metadata_cache.rs} | 251 +--
datafusion/execution/src/cache/list_files_cache.rs | 1072 +++++++++
datafusion/execution/src/cache/mod.rs | 15 +-
datafusion/execution/src/config.rs | 9 +-
datafusion/execution/src/disk_manager.rs | 20 +-
datafusion/execution/src/lib.rs | 6 +-
datafusion/execution/src/memory_pool/mod.rs | 138 +-
datafusion/execution/src/memory_pool/pool.rs | 57 +-
.../src/metrics/baseline.rs | 8 +-
.../src/metrics/builder.rs | 4 +-
.../src/metrics/custom.rs | 2 +-
.../src/metrics/mod.rs | 32 +-
.../src/metrics/value.rs | 16 +-
datafusion/execution/src/object_store.rs | 30 +-
datafusion/execution/src/parquet_encryption.rs | 2 +-
datafusion/execution/src/runtime_env.rs | 68 +-
datafusion/execution/src/task.rs | 7 +-
datafusion/expr-common/Cargo.toml | 2 +-
datafusion/expr-common/src/accumulator.rs | 2 +-
datafusion/expr-common/src/casts.rs | 9 +-
datafusion/expr-common/src/columnar_value.rs | 4 +-
datafusion/expr-common/src/dyn_eq.rs | 4 +-
datafusion/expr-common/src/groups_accumulator.rs | 2 +-
datafusion/expr-common/src/interval_arithmetic.rs | 58 +-
datafusion/expr-common/src/lib.rs | 3 +-
datafusion/expr-common/src/signature.rs | 190 +-
datafusion/expr-common/src/statistics.rs | 10 +-
.../expr-common/src/type_coercion/aggregates.rs | 2 +-
datafusion/expr-common/src/type_coercion/binary.rs | 24 +-
.../src/type_coercion/binary/tests/arithmetic.rs | 13 +-
.../src/type_coercion/binary/tests/comparison.rs | 10 +-
datafusion/expr/Cargo.toml | 2 +-
datafusion/expr/src/arguments.rs | 42 +-
datafusion/expr/src/async_udf.rs | 2 +-
datafusion/expr/src/conditional_expressions.rs | 4 +-
datafusion/expr/src/execution_props.rs | 7 +-
datafusion/expr/src/expr.rs | 61 +-
datafusion/expr/src/expr_fn.rs | 4 +-
datafusion/expr/src/expr_rewriter/guarantees.rs | 14 +-
datafusion/expr/src/expr_rewriter/mod.rs | 14 +-
datafusion/expr/src/expr_rewriter/order_by.rs | 6 +-
datafusion/expr/src/expr_schema.rs | 170 +-
datafusion/expr/src/lib.rs | 17 +-
datafusion/expr/src/literal.rs | 2 +-
datafusion/expr/src/logical_plan/builder.rs | 172 +-
datafusion/expr/src/logical_plan/ddl.rs | 159 +-
datafusion/expr/src/logical_plan/display.rs | 43 +-
datafusion/expr/src/logical_plan/invariants.rs | 16 +-
datafusion/expr/src/logical_plan/mod.rs | 15 +-
datafusion/expr/src/logical_plan/plan.rs | 186 +-
datafusion/expr/src/logical_plan/statement.rs | 2 +-
datafusion/expr/src/logical_plan/tree_node.rs | 12 +-
datafusion/expr/src/partition_evaluator.rs | 2 +-
datafusion/expr/src/planner.rs | 107 +-
datafusion/expr/src/predicate_bounds.rs | 12 +-
datafusion/expr/src/ptr_eq.rs | 2 +-
datafusion/expr/src/registry.rs | 2 +-
datafusion/expr/src/select_expr.rs | 2 +-
datafusion/expr/src/simplify.rs | 14 +-
datafusion/expr/src/test/function_stub.rs | 12 +-
datafusion/expr/src/tree_node.rs | 4 +-
datafusion/expr/src/type_coercion/functions.rs | 96 +-
datafusion/expr/src/udaf.rs | 27 +-
datafusion/expr/src/udf.rs | 18 +-
datafusion/expr/src/udf_eq.rs | 2 +-
datafusion/expr/src/udwf.rs | 12 +-
datafusion/expr/src/utils.rs | 67 +-
datafusion/expr/src/window_frame.rs | 54 +-
datafusion/expr/src/window_state.rs | 17 +-
datafusion/ffi/Cargo.toml | 13 +
datafusion/ffi/README.md | 92 +
datafusion/ffi/src/arrow_wrappers.rs | 31 +-
datafusion/ffi/src/catalog_provider.rs | 209 +-
datafusion/ffi/src/catalog_provider_list.rs | 155 +-
.../equivalence => ffi/src/execution}/mod.rs | 9 +-
datafusion/ffi/src/execution/task_ctx.rs | 288 +++
datafusion/ffi/src/execution/task_ctx_provider.rs | 229 ++
datafusion/ffi/src/execution_plan.rs | 189 +-
datafusion/ffi/src/expr/columnar_value.rs | 85 +
datafusion/ffi/src/expr/distribution.rs | 215 ++
datafusion/ffi/src/expr/expr_properties.rs | 116 +
datafusion/ffi/src/expr/interval.rs | 58 +
.../{core/tests/catalog => ffi/src/expr}/mod.rs | 5 +-
datafusion/ffi/src/lib.rs | 37 +-
datafusion/ffi/src/physical_expr/mod.rs | 995 +++++++++
datafusion/ffi/src/physical_expr/partitioning.rs | 98 +
datafusion/ffi/src/physical_expr/sort.rs | 75 +
datafusion/ffi/src/plan_properties.rs | 106 +-
.../ffi/src/proto/logical_extension_codec.rs | 710 ++++++
.../{core/tests/catalog => ffi/src/proto}/mod.rs | 3 +-
.../ffi/src/proto/physical_extension_codec.rs | 682 ++++++
datafusion/ffi/src/record_batch_stream.rs | 72 +-
datafusion/ffi/src/schema_provider.rs | 225 +-
.../src/{session_config.rs => session/config.rs} | 106 +-
datafusion/ffi/src/session/mod.rs | 623 ++++++
datafusion/ffi/src/table_provider.rs | 216 +-
datafusion/ffi/src/tests/catalog.rs | 8 +-
datafusion/ffi/src/tests/mod.rs | 3 +-
datafusion/ffi/src/udaf/accumulator.rs | 211 +-
datafusion/ffi/src/udaf/accumulator_args.rs | 8 +-
datafusion/ffi/src/udaf/groups_accumulator.rs | 223 +-
datafusion/ffi/src/udaf/mod.rs | 350 +--
datafusion/ffi/src/udf/mod.rs | 235 +-
datafusion/ffi/src/udf/return_type_args.rs | 6 +-
datafusion/ffi/src/udtf.rs | 75 +-
datafusion/ffi/src/udwf/mod.rs | 294 +--
datafusion/ffi/src/udwf/partition_evaluator.rs | 231 +-
.../ffi/src/udwf/partition_evaluator_args.rs | 12 +-
datafusion/ffi/src/util.rs | 59 +-
datafusion/ffi/tests/ffi_catalog.rs | 12 +-
datafusion/ffi/tests/ffi_integration.rs | 8 +-
datafusion/ffi/tests/ffi_udaf.rs | 14 +-
datafusion/ffi/tests/ffi_udf.rs | 15 +-
datafusion/ffi/tests/ffi_udtf.rs | 10 +-
datafusion/ffi/tests/ffi_udwf.rs | 10 +-
.../benches/accumulate.rs | 2 +-
.../src/aggregate/avg_distinct/decimal.rs | 4 +-
.../src/aggregate/count_distinct/bytes.rs | 10 +-
.../src/aggregate/count_distinct/dict.rs | 4 +-
.../src/aggregate/count_distinct/native.rs | 10 +-
.../src/aggregate/groups_accumulator.rs | 2 +-
.../src/aggregate/groups_accumulator/accumulate.rs | 64 +-
.../src/aggregate/groups_accumulator/nulls.rs | 2 +-
.../src/aggregate/groups_accumulator/prim_op.rs | 2 +-
datafusion/functions-aggregate-common/src/lib.rs | 4 +-
.../functions-aggregate-common/src/merge_arrays.rs | 2 +-
.../functions-aggregate-common/src/min_max.rs | 12 +-
.../functions-aggregate-common/src/tdigest.rs | 62 +-
datafusion/functions-aggregate-common/src/utils.rs | 8 +-
datafusion/functions-aggregate/Cargo.toml | 2 +-
.../functions-aggregate/benches/array_agg.rs | 7 +-
datafusion/functions-aggregate/benches/count.rs | 3 +-
.../functions-aggregate/benches/min_max_bytes.rs | 4 +-
datafusion/functions-aggregate/benches/sum.rs | 5 +-
.../functions-aggregate/src/approx_distinct.rs | 44 +-
.../functions-aggregate/src/approx_median.rs | 90 +-
.../src/approx_percentile_cont.rs | 104 +-
.../src/approx_percentile_cont_with_weight.rs | 9 +-
datafusion/functions-aggregate/src/array_agg.rs | 40 +-
datafusion/functions-aggregate/src/average.rs | 30 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 58 +-
datafusion/functions-aggregate/src/bool_and_or.rs | 32 +-
datafusion/functions-aggregate/src/correlation.rs | 28 +-
datafusion/functions-aggregate/src/count.rs | 99 +-
datafusion/functions-aggregate/src/covariance.rs | 5 +-
datafusion/functions-aggregate/src/first_last.rs | 137 +-
datafusion/functions-aggregate/src/grouping.rs | 16 +-
datafusion/functions-aggregate/src/lib.rs | 4 +-
datafusion/functions-aggregate/src/median.rs | 80 +-
datafusion/functions-aggregate/src/min_max.rs | 28 +-
.../src/min_max/min_max_bytes.rs | 2 +-
.../src/min_max/min_max_struct.rs | 3 +-
datafusion/functions-aggregate/src/nth_value.rs | 12 +-
.../functions-aggregate/src/percentile_cont.rs | 118 +-
datafusion/functions-aggregate/src/planner.rs | 2 +-
datafusion/functions-aggregate/src/regr.rs | 4 +-
datafusion/functions-aggregate/src/stddev.rs | 20 +-
datafusion/functions-aggregate/src/string_agg.rs | 23 +-
datafusion/functions-aggregate/src/sum.rs | 48 +-
datafusion/functions-aggregate/src/utils.rs | 4 +-
datafusion/functions-aggregate/src/variance.rs | 6 +-
datafusion/functions-nested/Cargo.toml | 10 +-
datafusion/functions-nested/benches/array_has.rs | 377 ++++
datafusion/functions-nested/benches/array_slice.rs | 230 ++
datafusion/functions-nested/benches/map.rs | 6 +-
datafusion/functions-nested/src/array_has.rs | 26 +-
datafusion/functions-nested/src/cardinality.rs | 7 +-
datafusion/functions-nested/src/concat.rs | 15 +-
datafusion/functions-nested/src/dimension.rs | 8 +-
datafusion/functions-nested/src/distance.rs | 6 +-
datafusion/functions-nested/src/empty.rs | 5 +-
datafusion/functions-nested/src/except.rs | 9 +-
datafusion/functions-nested/src/extract.rs | 14 +-
datafusion/functions-nested/src/flatten.rs | 5 +-
datafusion/functions-nested/src/length.rs | 5 +-
datafusion/functions-nested/src/lib.rs | 4 +-
datafusion/functions-nested/src/make_array.rs | 45 +-
datafusion/functions-nested/src/map.rs | 2 +-
datafusion/functions-nested/src/map_entries.rs | 2 +-
datafusion/functions-nested/src/map_extract.rs | 4 +-
datafusion/functions-nested/src/map_keys.rs | 2 +-
datafusion/functions-nested/src/map_values.rs | 2 +-
datafusion/functions-nested/src/min_max.rs | 15 +-
datafusion/functions-nested/src/planner.rs | 6 +-
datafusion/functions-nested/src/position.rs | 13 +-
datafusion/functions-nested/src/range.rs | 16 +-
datafusion/functions-nested/src/remove.rs | 491 ++++-
datafusion/functions-nested/src/repeat.rs | 9 +-
datafusion/functions-nested/src/replace.rs | 12 +-
datafusion/functions-nested/src/resize.rs | 11 +-
datafusion/functions-nested/src/reverse.rs | 2 +-
datafusion/functions-nested/src/set_ops.rs | 17 +-
datafusion/functions-nested/src/sort.rs | 10 +-
datafusion/functions-nested/src/string.rs | 79 +-
datafusion/functions-nested/src/utils.rs | 2 +-
datafusion/functions-table/Cargo.toml | 2 +-
datafusion/functions-table/src/generate_series.rs | 40 +-
datafusion/functions-table/src/lib.rs | 4 +-
datafusion/functions-window-common/src/lib.rs | 4 +-
datafusion/functions-window/Cargo.toml | 2 +-
datafusion/functions-window/src/cume_dist.rs | 4 +-
datafusion/functions-window/src/lead_lag.rs | 4 +-
datafusion/functions-window/src/lib.rs | 6 +-
datafusion/functions-window/src/nth_value.rs | 41 +-
datafusion/functions-window/src/ntile.rs | 2 +-
datafusion/functions-window/src/planner.rs | 2 +-
datafusion/functions-window/src/rank.rs | 7 +-
datafusion/functions-window/src/utils.rs | 2 +-
datafusion/functions/Cargo.toml | 4 +-
datafusion/functions/benches/ascii.rs | 2 +-
datafusion/functions/benches/character_length.rs | 2 +-
datafusion/functions/benches/chr.rs | 2 +-
datafusion/functions/benches/concat.rs | 4 +-
datafusion/functions/benches/cot.rs | 2 +-
datafusion/functions/benches/date_bin.rs | 6 +-
datafusion/functions/benches/date_trunc.rs | 6 +-
datafusion/functions/benches/encoding.rs | 2 +-
datafusion/functions/benches/find_in_set.rs | 4 +-
datafusion/functions/benches/gcd.rs | 4 +-
datafusion/functions/benches/helper.rs | 2 +-
datafusion/functions/benches/initcap.rs | 2 +-
datafusion/functions/benches/isnan.rs | 2 +-
datafusion/functions/benches/iszero.rs | 2 +-
datafusion/functions/benches/lower.rs | 2 +-
datafusion/functions/benches/ltrim.rs | 8 +-
datafusion/functions/benches/make_date.rs | 6 +-
datafusion/functions/benches/nullif.rs | 4 +-
datafusion/functions/benches/pad.rs | 7 +-
datafusion/functions/benches/random.rs | 2 +-
datafusion/functions/benches/regx.rs | 4 +-
datafusion/functions/benches/repeat.rs | 4 +-
datafusion/functions/benches/reverse.rs | 2 +-
datafusion/functions/benches/signum.rs | 2 +-
datafusion/functions/benches/strpos.rs | 2 +-
datafusion/functions/benches/substr.rs | 5 +-
datafusion/functions/benches/substr_index.rs | 4 +-
datafusion/functions/benches/to_char.rs | 8 +-
datafusion/functions/benches/to_hex.rs | 2 +-
datafusion/functions/benches/to_timestamp.rs | 2 +-
datafusion/functions/benches/trunc.rs | 2 +-
datafusion/functions/benches/upper.rs | 2 +-
datafusion/functions/benches/uuid.rs | 2 +-
datafusion/functions/src/core/arrow_cast.rs | 6 +-
datafusion/functions/src/core/arrowtypeof.rs | 2 +-
datafusion/functions/src/core/coalesce.rs | 2 +-
datafusion/functions/src/core/getfield.rs | 74 +-
datafusion/functions/src/core/greatest.rs | 14 +-
.../functions/src/core/greatest_least_utils.rs | 4 +-
datafusion/functions/src/core/least.rs | 14 +-
datafusion/functions/src/core/named_struct.rs | 2 +-
datafusion/functions/src/core/nullif.rs | 2 +-
datafusion/functions/src/core/nvl2.rs | 6 +-
datafusion/functions/src/core/overlay.rs | 6 +-
datafusion/functions/src/core/planner.rs | 2 +-
datafusion/functions/src/core/struct.rs | 2 +-
datafusion/functions/src/core/union_extract.rs | 15 +-
datafusion/functions/src/core/union_tag.rs | 4 +-
datafusion/functions/src/core/version.rs | 4 +-
datafusion/functions/src/crypto/basic.rs | 163 +-
datafusion/functions/src/crypto/digest.rs | 48 +-
datafusion/functions/src/crypto/md5.rs | 52 +-
datafusion/functions/src/crypto/mod.rs | 13 +-
datafusion/functions/src/crypto/sha.rs | 175 ++
datafusion/functions/src/crypto/sha224.rs | 104 -
datafusion/functions/src/crypto/sha256.rs | 102 -
datafusion/functions/src/crypto/sha384.rs | 102 -
datafusion/functions/src/crypto/sha512.rs | 102 -
datafusion/functions/src/datetime/common.rs | 26 +-
datafusion/functions/src/datetime/current_date.rs | 2 +-
datafusion/functions/src/datetime/current_time.rs | 7 +-
datafusion/functions/src/datetime/date_bin.rs | 16 +-
datafusion/functions/src/datetime/date_part.rs | 6 +-
datafusion/functions/src/datetime/date_trunc.rs | 14 +-
datafusion/functions/src/datetime/from_unixtime.rs | 4 +-
datafusion/functions/src/datetime/make_date.rs | 337 +--
datafusion/functions/src/datetime/make_time.rs | 267 +++
datafusion/functions/src/datetime/mod.rs | 7 +
datafusion/functions/src/datetime/now.rs | 4 +-
datafusion/functions/src/datetime/planner.rs | 2 +-
datafusion/functions/src/datetime/to_char.rs | 10 +-
datafusion/functions/src/datetime/to_date.rs | 53 +-
datafusion/functions/src/datetime/to_local_time.rs | 347 ++-
datafusion/functions/src/datetime/to_timestamp.rs | 31 +-
datafusion/functions/src/datetime/to_unixtime.rs | 2 +-
datafusion/functions/src/encoding/inner.rs | 31 +-
datafusion/functions/src/lib.rs | 4 +-
datafusion/functions/src/macros.rs | 16 +-
datafusion/functions/src/math/abs.rs | 8 +-
datafusion/functions/src/math/cot.rs | 2 +-
datafusion/functions/src/math/factorial.rs | 2 +-
datafusion/functions/src/math/gcd.rs | 35 +-
datafusion/functions/src/math/iszero.rs | 4 +-
datafusion/functions/src/math/lcm.rs | 2 +-
datafusion/functions/src/math/log.rs | 17 +-
datafusion/functions/src/math/monotonicity.rs | 4 +-
datafusion/functions/src/math/nans.rs | 4 +-
datafusion/functions/src/math/nanvl.rs | 14 +-
datafusion/functions/src/math/pi.rs | 2 +-
datafusion/functions/src/math/power.rs | 497 ++---
datafusion/functions/src/math/random.rs | 4 +-
datafusion/functions/src/math/round.rs | 10 +-
datafusion/functions/src/math/signum.rs | 16 +-
datafusion/functions/src/math/trunc.rs | 16 +-
datafusion/functions/src/planner.rs | 2 +-
datafusion/functions/src/regex/mod.rs | 2 +-
datafusion/functions/src/regex/regexpcount.rs | 16 +-
datafusion/functions/src/regex/regexpinstr.rs | 56 +-
datafusion/functions/src/regex/regexplike.rs | 51 +-
datafusion/functions/src/regex/regexpmatch.rs | 33 +-
datafusion/functions/src/regex/regexpreplace.rs | 69 +-
datafusion/functions/src/string/ascii.rs | 2 +-
datafusion/functions/src/string/btrim.rs | 2 +-
datafusion/functions/src/string/chr.rs | 14 +-
datafusion/functions/src/string/common.rs | 14 +-
datafusion/functions/src/string/concat.rs | 37 +-
datafusion/functions/src/string/concat_ws.rs | 64 +-
datafusion/functions/src/string/contains.rs | 4 +-
datafusion/functions/src/string/ends_with.rs | 8 +-
datafusion/functions/src/string/levenshtein.rs | 10 +-
datafusion/functions/src/string/lower.rs | 2 +-
datafusion/functions/src/string/ltrim.rs | 2 +-
datafusion/functions/src/string/octet_length.rs | 2 +-
datafusion/functions/src/string/repeat.rs | 6 +-
datafusion/functions/src/string/replace.rs | 6 +-
datafusion/functions/src/string/rtrim.rs | 2 +-
datafusion/functions/src/string/split_part.rs | 50 +-
datafusion/functions/src/string/starts_with.rs | 27 +-
datafusion/functions/src/string/to_hex.rs | 14 +-
datafusion/functions/src/string/upper.rs | 2 +-
datafusion/functions/src/string/uuid.rs | 4 +-
datafusion/functions/src/strings.rs | 4 +-
.../functions/src/unicode/character_length.rs | 4 +-
datafusion/functions/src/unicode/find_in_set.rs | 19 +-
datafusion/functions/src/unicode/initcap.rs | 2 +-
datafusion/functions/src/unicode/left.rs | 4 +-
datafusion/functions/src/unicode/lpad.rs | 14 +-
datafusion/functions/src/unicode/planner.rs | 2 +-
datafusion/functions/src/unicode/reverse.rs | 6 +-
datafusion/functions/src/unicode/right.rs | 6 +-
datafusion/functions/src/unicode/rpad.rs | 10 +-
datafusion/functions/src/unicode/strpos.rs | 2 +-
datafusion/functions/src/unicode/substr.rs | 10 +-
datafusion/functions/src/unicode/substrindex.rs | 4 +-
datafusion/functions/src/unicode/translate.rs | 2 +-
datafusion/functions/src/utils.rs | 5 +-
datafusion/macros/Cargo.toml | 2 +-
datafusion/macros/src/user_doc.rs | 3 +-
.../optimizer/benches/projection_unnecessary.rs | 4 +-
.../optimizer/src/analyzer/function_rewrite.rs | 2 +-
datafusion/optimizer/src/analyzer/mod.rs | 2 +-
.../src/analyzer/resolve_grouping_function.rs | 8 +-
datafusion/optimizer/src/analyzer/type_coercion.rs | 51 +-
.../optimizer/src/common_subexpr_eliminate.rs | 14 +-
datafusion/optimizer/src/decorrelate.rs | 6 +-
.../optimizer/src/decorrelate_lateral_join.rs | 6 +-
.../src/decorrelate_predicate_subquery.rs | 18 +-
datafusion/optimizer/src/eliminate_cross_join.rs | 19 +-
.../optimizer/src/eliminate_duplicated_expr.rs | 6 +-
datafusion/optimizer/src/eliminate_filter.rs | 6 +-
.../optimizer/src/eliminate_group_by_constant.rs | 8 +-
datafusion/optimizer/src/eliminate_join.rs | 4 +-
datafusion/optimizer/src/eliminate_limit.rs | 10 +-
datafusion/optimizer/src/eliminate_outer_join.rs | 6 +-
.../optimizer/src/extract_equijoin_predicate.rs | 6 +-
datafusion/optimizer/src/filter_null_join_keys.rs | 6 +-
datafusion/optimizer/src/join_key_set.rs | 2 +-
datafusion/optimizer/src/lib.rs | 3 +-
.../optimizer/src/optimize_projections/mod.rs | 31 +-
datafusion/optimizer/src/optimize_unions.rs | 67 +-
datafusion/optimizer/src/optimizer.rs | 16 +-
datafusion/optimizer/src/plan_signature.rs | 2 +-
.../optimizer/src/propagate_empty_relation.rs | 18 +-
datafusion/optimizer/src/push_down_filter.rs | 24 +-
datafusion/optimizer/src/push_down_limit.rs | 16 +-
.../optimizer/src/replace_distinct_aggregate.rs | 6 +-
.../optimizer/src/scalar_subquery_to_join.rs | 52 +-
.../src/simplify_expressions/expr_simplifier.rs | 38 +-
.../src/simplify_expressions/inlist_simplifier.rs | 92 +-
.../optimizer/src/simplify_expressions/regex.rs | 20 +-
.../src/simplify_expressions/simplify_exprs.rs | 27 +-
.../src/simplify_expressions/unwrap_cast.rs | 6 +-
.../optimizer/src/simplify_expressions/utils.rs | 4 +-
.../optimizer/src/single_distinct_to_groupby.rs | 9 +-
datafusion/optimizer/src/test/mod.rs | 4 +-
datafusion/optimizer/src/test/user_defined.rs | 2 +-
datafusion/optimizer/src/utils.rs | 6 +-
.../optimizer/tests/optimizer_integration.rs | 369 ++--
datafusion/physical-expr-adapter/src/lib.rs | 6 +-
.../physical-expr-adapter/src/schema_rewriter.rs | 449 ++--
datafusion/physical-expr-common/src/binary_map.rs | 4 +-
datafusion/physical-expr-common/src/datum.rs | 7 +-
datafusion/physical-expr-common/src/lib.rs | 4 +-
.../physical-expr-common/src/physical_expr.rs | 56 +-
datafusion/physical-expr-common/src/sort_expr.rs | 105 +-
datafusion/physical-expr-common/src/tree_node.rs | 4 +-
datafusion/physical-expr-common/src/utils.rs | 4 +-
datafusion/physical-expr/Cargo.toml | 4 +-
datafusion/physical-expr/benches/binary_op.rs | 7 +-
datafusion/physical-expr/benches/case_when.rs | 12 +-
datafusion/physical-expr/benches/in_list.rs | 205 +-
datafusion/physical-expr/benches/is_null.rs | 4 +-
datafusion/physical-expr/src/aggregate.rs | 37 +-
datafusion/physical-expr/src/analysis.rs | 52 +-
.../physical-expr/src/async_scalar_function.rs | 13 +-
datafusion/physical-expr/src/equivalence/class.rs | 35 +-
datafusion/physical-expr/src/equivalence/mod.rs | 6 +-
.../physical-expr/src/equivalence/ordering.rs | 15 +-
.../src/equivalence/properties/dependency.rs | 6 +-
.../src/equivalence/properties/joins.rs | 2 +-
.../src/equivalence/properties/mod.rs | 8 +-
.../src/equivalence/properties/union.rs | 4 +-
datafusion/physical-expr/src/expressions/binary.rs | 98 +-
.../src/expressions/binary/kernels.rs | 43 +-
datafusion/physical-expr/src/expressions/case.rs | 732 ++++++-
.../literal_lookup_table/boolean_lookup_table.rs | 122 ++
.../bytes_like_lookup_table.rs | 223 ++
.../expressions/case/literal_lookup_table/mod.rs | 327 +++
.../literal_lookup_table/primitive_lookup_table.rs | 229 ++
datafusion/physical-expr/src/expressions/cast.rs | 34 +-
.../physical-expr/src/expressions/cast_column.rs | 4 +-
datafusion/physical-expr/src/expressions/column.rs | 8 +-
.../src/expressions/dynamic_filters.rs | 88 +-
.../physical-expr/src/expressions/in_list.rs | 1788 +++++++++------
datafusion/physical-expr/src/expressions/like.rs | 2 +-
datafusion/physical-expr/src/expressions/mod.rs | 26 +-
.../physical-expr/src/expressions/negative.rs | 20 +-
datafusion/physical-expr/src/expressions/no_op.rs | 2 +-
datafusion/physical-expr/src/expressions/not.rs | 41 +-
.../physical-expr/src/expressions/try_cast.rs | 6 +-
.../src/expressions/unknown_column.rs | 2 +-
.../physical-expr/src/intervals/cp_solver.rs | 12 +-
.../physical-expr/src/intervals/test_utils.rs | 6 +-
datafusion/physical-expr/src/intervals/utils.rs | 6 +-
datafusion/physical-expr/src/lib.rs | 8 +-
datafusion/physical-expr/src/partitioning.rs | 4 +-
datafusion/physical-expr/src/physical_expr.rs | 4 +-
datafusion/physical-expr/src/planner.rs | 8 +-
datafusion/physical-expr/src/projection.rs | 228 +-
datafusion/physical-expr/src/scalar_function.rs | 37 +-
.../src/simplifier/const_evaluator.rs | 103 +
datafusion/physical-expr/src/simplifier/mod.rs | 538 ++++-
datafusion/physical-expr/src/simplifier/not.rs | 124 ++
.../physical-expr/src/simplifier/unwrap_cast.rs | 53 +-
.../physical-expr/src/statistics/stats_solver.rs | 6 +-
datafusion/physical-expr/src/utils/guarantee.rs | 4 +-
datafusion/physical-expr/src/utils/mod.rs | 8 +-
datafusion/physical-expr/src/window/aggregate.rs | 4 +-
.../physical-expr/src/window/sliding_aggregate.rs | 4 +-
datafusion/physical-expr/src/window/standard.rs | 6 +-
.../src/window/standard_window_function_expr.rs | 3 +-
datafusion/physical-expr/src/window/window_expr.rs | 9 +-
.../physical-optimizer/src/aggregate_statistics.rs | 45 +-
.../physical-optimizer/src/coalesce_batches.rs | 30 +-
.../src/combine_partial_final_agg.rs | 6 +-
.../physical-optimizer/src/enforce_distribution.rs | 248 +--
.../physical-optimizer/src/enforce_sorting/mod.rs | 12 +-
.../replace_with_order_preserving_variants.rs | 4 +-
.../src/enforce_sorting/sort_pushdown.rs | 6 +-
datafusion/physical-optimizer/src/ensure_coop.rs | 12 +-
.../physical-optimizer/src/filter_pushdown.rs | 8 +-
.../physical-optimizer/src/join_selection.rs | 28 +-
datafusion/physical-optimizer/src/lib.rs | 5 +-
.../physical-optimizer/src/limit_pushdown.rs | 2 +-
.../src/limit_pushdown_past_window.rs | 10 +-
.../src/limited_distinct_aggregation.rs | 20 +-
datafusion/physical-optimizer/src/optimizer.rs | 5 +-
.../physical-optimizer/src/output_requirements.rs | 2 +-
.../physical-optimizer/src/projection_pushdown.rs | 10 +-
datafusion/physical-optimizer/src/pushdown_sort.rs | 129 ++
.../physical-optimizer/src/sanity_checker.rs | 17 +-
.../physical-optimizer/src/topk_aggregation.rs | 4 +-
.../physical-optimizer/src/update_aggr_exprs.rs | 6 +-
datafusion/physical-plan/Cargo.toml | 4 +-
.../physical-plan/benches/aggregate_vectorized.rs | 5 +-
.../physical-plan/benches/partial_ordering.rs | 2 +-
.../physical-plan/benches/sort_preserving_merge.rs | 4 +-
datafusion/physical-plan/benches/spill_io.rs | 7 +-
.../src/aggregates/group_values/metrics.rs | 2 +-
.../src/aggregates/group_values/mod.rs | 2 +-
.../group_values/multi_group_by/boolean.rs | 2 +-
.../group_values/multi_group_by/bytes.rs | 10 +-
.../group_values/multi_group_by/bytes_view.rs | 4 +-
.../aggregates/group_values/multi_group_by/mod.rs | 14 +-
.../group_values/multi_group_by/primitive.rs | 4 +-
.../src/aggregates/group_values/row.rs | 2 +-
.../group_values/single_group_by/primitive.rs | 6 +-
datafusion/physical-plan/src/aggregates/mod.rs | 744 +++++--
.../physical-plan/src/aggregates/no_grouping.rs | 263 ++-
.../physical-plan/src/aggregates/row_hash.rs | 185 +-
.../src/aggregates/topk/hash_table.rs | 224 +-
.../physical-plan/src/aggregates/topk/heap.rs | 113 +-
.../src/aggregates/topk/priority_map.rs | 146 +-
.../physical-plan/src/aggregates/topk_stream.rs | 8 +-
datafusion/physical-plan/src/analyze.rs | 4 +-
datafusion/physical-plan/src/async_func.rs | 27 +-
datafusion/physical-plan/src/coalesce/mod.rs | 2 +-
datafusion/physical-plan/src/coalesce_batches.rs | 16 +
.../physical-plan/src/coalesce_partitions.rs | 46 +-
datafusion/physical-plan/src/common.rs | 5 +-
datafusion/physical-plan/src/coop.rs | 4 +-
datafusion/physical-plan/src/display.rs | 4 +-
datafusion/physical-plan/src/empty.rs | 13 +-
datafusion/physical-plan/src/execution_plan.rs | 42 +-
datafusion/physical-plan/src/explain.rs | 15 +-
datafusion/physical-plan/src/filter.rs | 66 +-
datafusion/physical-plan/src/joins/cross_join.rs | 65 +-
.../physical-plan/src/joins/hash_join/exec.rs | 357 +--
.../src/joins/hash_join/inlist_builder.rs | 133 ++
.../physical-plan/src/joins/hash_join/mod.rs | 2 +
.../src/joins/hash_join/partitioned_hash_eval.rs | 146 +-
.../src/joins/hash_join/shared_bounds.rs | 401 +++-
.../physical-plan/src/joins/hash_join/stream.rs | 210 +-
.../physical-plan/src/joins/join_hash_map.rs | 165 +-
datafusion/physical-plan/src/joins/mod.rs | 8 +-
.../physical-plan/src/joins/nested_loop_join.rs | 299 +--
.../src/joins/piecewise_merge_join/classic_join.rs | 147 +-
.../src/joins/piecewise_merge_join/exec.rs | 23 +-
.../src/joins/sort_merge_join/exec.rs | 20 +-
.../src/joins/sort_merge_join/metrics.rs | 2 -
.../src/joins/sort_merge_join/stream.rs | 713 +++---
.../src/joins/sort_merge_join/tests.rs | 823 +++----
.../physical-plan/src/joins/stream_join_utils.rs | 118 +-
.../physical-plan/src/joins/symmetric_hash_join.rs | 34 +-
datafusion/physical-plan/src/joins/test_utils.rs | 6 +-
datafusion/physical-plan/src/joins/utils.rs | 106 +-
datafusion/physical-plan/src/lib.rs | 20 +-
datafusion/physical-plan/src/limit.rs | 13 +-
datafusion/physical-plan/src/memory.rs | 6 +-
.../src/metrics.rs} | 8 +-
datafusion/physical-plan/src/placeholder_row.rs | 13 +-
datafusion/physical-plan/src/projection.rs | 29 +-
datafusion/physical-plan/src/recursive_query.rs | 108 +-
datafusion/physical-plan/src/render_tree.rs | 3 +-
.../src/repartition/distributor_channels.rs | 4 +-
datafusion/physical-plan/src/repartition/mod.rs | 223 +-
datafusion/physical-plan/src/sort_pushdown.rs | 120 ++
datafusion/physical-plan/src/sorts/cursor.rs | 4 +-
datafusion/physical-plan/src/sorts/merge.rs | 4 +-
.../physical-plan/src/sorts/multi_level_merge.rs | 8 +-
datafusion/physical-plan/src/sorts/partial_sort.rs | 159 +-
datafusion/physical-plan/src/sorts/sort.rs | 98 +-
.../src/sorts/sort_preserving_merge.rs | 174 +-
datafusion/physical-plan/src/sorts/stream.rs | 6 +-
.../physical-plan/src/sorts/streaming_merge.rs | 6 +-
.../src/spill/in_progress_spill_file.rs | 2 +-
datafusion/physical-plan/src/spill/mod.rs | 214 +-
.../physical-plan/src/spill/spill_manager.rs | 6 +-
datafusion/physical-plan/src/spill/spill_pool.rs | 67 +-
datafusion/physical-plan/src/stream.rs | 4 +-
datafusion/physical-plan/src/streaming.rs | 8 +-
datafusion/physical-plan/src/test.rs | 13 +-
datafusion/physical-plan/src/test/exec.rs | 12 +-
datafusion/physical-plan/src/topk/mod.rs | 18 +-
datafusion/physical-plan/src/tree_node.rs | 4 +-
datafusion/physical-plan/src/union.rs | 62 +-
datafusion/physical-plan/src/unnest.rs | 64 +-
.../src/windows/bounded_window_agg_exec.rs | 144 +-
datafusion/physical-plan/src/windows/mod.rs | 58 +-
.../physical-plan/src/windows/window_agg_exec.rs | 4 +-
datafusion/physical-plan/src/work_table.rs | 7 +-
.../proto-common/proto/datafusion_common.proto | 6 +
datafusion/proto-common/src/common.rs | 2 +-
datafusion/proto-common/src/from_proto/mod.rs | 20 +-
datafusion/proto-common/src/generated/mod.rs | 1 +
datafusion/proto-common/src/generated/pbjson.rs | 76 +
datafusion/proto-common/src/generated/prost.rs | 11 +
datafusion/proto-common/src/lib.rs | 3 +-
datafusion/proto-common/src/to_proto/mod.rs | 14 +-
datafusion/proto/Cargo.toml | 1 +
datafusion/proto/proto/datafusion.proto | 19 +
datafusion/proto/src/bytes/mod.rs | 8 +-
datafusion/proto/src/bytes/registry.rs | 29 +-
datafusion/proto/src/common.rs | 4 +-
datafusion/proto/src/generated/datafusion.rs | 1 +
.../proto/src/generated/datafusion_proto_common.rs | 11 +
datafusion/proto/src/generated/mod.rs | 2 +
datafusion/proto/src/generated/pbjson.rs | 376 ++++
datafusion/proto/src/generated/prost.rs | 29 +-
datafusion/proto/src/lib.rs | 3 +-
datafusion/proto/src/logical_plan/file_formats.rs | 23 +-
datafusion/proto/src/logical_plan/from_proto.rs | 19 +-
datafusion/proto/src/logical_plan/mod.rs | 83 +-
datafusion/proto/src/logical_plan/to_proto.rs | 45 +-
datafusion/proto/src/physical_plan/from_proto.rs | 45 +-
datafusion/proto/src/physical_plan/mod.rs | 137 +-
datafusion/proto/src/physical_plan/to_proto.rs | 63 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 42 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 210 +-
datafusion/proto/tests/cases/serialize.rs | 6 +-
datafusion/pruning/src/file_pruner.rs | 131 +-
datafusion/pruning/src/lib.rs | 7 +-
datafusion/pruning/src/pruning_predicate.rs | 373 +++-
datafusion/session/src/lib.rs | 3 +-
datafusion/session/src/session.rs | 4 +-
datafusion/spark/Cargo.toml | 2 +
datafusion/spark/benches/char.rs | 2 +-
datafusion/spark/src/function/aggregate/avg.rs | 6 +-
datafusion/spark/src/function/aggregate/mod.rs | 12 +-
datafusion/spark/src/function/aggregate/try_sum.rs | 660 ++++++
datafusion/spark/src/function/array/shuffle.rs | 66 +-
datafusion/spark/src/function/array/spark_array.rs | 135 +-
.../spark/src/function/bitmap/bitmap_count.rs | 73 +-
datafusion/spark/src/function/bitwise/bit_count.rs | 60 +-
datafusion/spark/src/function/bitwise/bit_get.rs | 62 +-
datafusion/spark/src/function/bitwise/bit_shift.rs | 78 +-
.../spark/src/function/bitwise/bitwise_not.rs | 130 +-
datafusion/spark/src/function/conditional/if.rs | 6 +-
datafusion/spark/src/function/datetime/date_add.rs | 92 +-
datafusion/spark/src/function/datetime/date_sub.rs | 84 +-
datafusion/spark/src/function/datetime/last_day.rs | 82 +-
.../src/function/datetime/make_dt_interval.rs | 175 +-
.../spark/src/function/datetime/make_interval.rs | 89 +-
datafusion/spark/src/function/datetime/next_day.rs | 89 +-
datafusion/spark/src/function/error_utils.rs | 6 +-
datafusion/spark/src/function/hash/crc32.rs | 4 +-
datafusion/spark/src/function/hash/sha1.rs | 63 +-
datafusion/spark/src/function/hash/sha2.rs | 35 +-
.../spark/src/function/map/map_from_arrays.rs | 88 +-
.../spark/src/function/map/map_from_entries.rs | 103 +-
datafusion/spark/src/function/map/utils.rs | 11 +-
datafusion/spark/src/function/math/abs.rs | 2 +-
datafusion/spark/src/function/math/factorial.rs | 4 +-
datafusion/spark/src/function/math/hex.rs | 99 +-
datafusion/spark/src/function/math/mod.rs | 6 +-
datafusion/spark/src/function/math/modulus.rs | 12 +-
datafusion/spark/src/function/math/rint.rs | 5 +-
datafusion/spark/src/function/math/width_bucket.rs | 149 +-
datafusion/spark/src/function/string/char.rs | 63 +-
datafusion/spark/src/function/string/concat.rs | 77 +-
datafusion/spark/src/function/string/elt.rs | 67 +-
.../spark/src/function/string/format_string.rs | 24 +-
datafusion/spark/src/function/string/ilike.rs | 90 +-
datafusion/spark/src/function/string/length.rs | 54 +-
datafusion/spark/src/function/string/like.rs | 2 +-
datafusion/spark/src/function/string/luhn_check.rs | 2 +-
datafusion/spark/src/function/url/mod.rs | 29 +-
datafusion/spark/src/function/url/parse_url.rs | 11 +-
datafusion/spark/src/function/url/try_parse_url.rs | 2 +-
.../url/{try_parse_url.rs => try_url_decode.rs} | 64 +-
datafusion/spark/src/function/url/url_decode.rs | 261 +++
datafusion/spark/src/function/url/url_encode.rs | 131 ++
datafusion/spark/src/lib.rs | 7 +-
datafusion/sql/Cargo.toml | 2 +-
datafusion/sql/examples/sql.rs | 6 +-
datafusion/sql/src/cte.rs | 15 +-
datafusion/sql/src/expr/binary_op.rs | 2 +-
datafusion/sql/src/expr/function.rs | 78 +-
datafusion/sql/src/expr/identifier.rs | 86 +-
datafusion/sql/src/expr/mod.rs | 26 +-
datafusion/sql/src/expr/order_by.rs | 2 +-
datafusion/sql/src/expr/subquery.rs | 18 +-
datafusion/sql/src/expr/substring.rs | 10 +-
datafusion/sql/src/expr/unary_op.rs | 11 +-
datafusion/sql/src/expr/value.rs | 35 +-
datafusion/sql/src/lib.rs | 3 +-
datafusion/sql/src/parser.rs | 65 +-
datafusion/sql/src/planner.rs | 22 +-
datafusion/sql/src/query.rs | 33 +-
datafusion/sql/src/relation/join.rs | 2 +-
datafusion/sql/src/relation/mod.rs | 123 +-
datafusion/sql/src/select.rs | 206 +-
datafusion/sql/src/set_expr.rs | 2 +-
datafusion/sql/src/statement.rs | 100 +-
datafusion/sql/src/unparser/ast.rs | 20 +-
datafusion/sql/src/unparser/dialect.rs | 4 +-
datafusion/sql/src/unparser/expr.rs | 53 +-
datafusion/sql/src/unparser/extension_unparser.rs | 4 +-
datafusion/sql/src/unparser/plan.rs | 160 +-
datafusion/sql/src/unparser/rewrite.rs | 13 +-
datafusion/sql/src/unparser/utils.rs | 48 +-
datafusion/sql/src/utils.rs | 67 +-
datafusion/sql/tests/cases/collection.rs | 10 +-
datafusion/sql/tests/cases/diagnostic.rs | 3 +-
datafusion/sql/tests/cases/params.rs | 87 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 204 +-
datafusion/sql/tests/common/mod.rs | 2 +-
datafusion/sql/tests/sql_integration.rs | 2280 ++++++++++----------
datafusion/sqllogictest/Cargo.toml | 2 +-
datafusion/sqllogictest/bin/postgres_container.rs | 10 +-
datafusion/sqllogictest/bin/sqllogictests.rs | 41 +-
datafusion/sqllogictest/src/engines/conversion.rs | 5 +-
.../src/engines/datafusion_engine/runner.rs | 2 +-
.../runner.rs | 3 +-
datafusion/sqllogictest/src/engines/mod.rs | 4 +-
.../src/engines/postgres_engine/mod.rs | 4 +-
datafusion/sqllogictest/src/filters.rs | 19 +-
datafusion/sqllogictest/src/lib.rs | 4 +-
datafusion/sqllogictest/src/test_context.rs | 62 +-
datafusion/sqllogictest/src/util.rs | 2 +-
.../test_files/agg_func_substitute.slt | 27 +-
datafusion/sqllogictest/test_files/aggregate.slt | 230 +-
.../test_files/aggregate_repartition.slt | 16 +-
.../sqllogictest/test_files/aggregates_topk.slt | 49 +-
datafusion/sqllogictest/test_files/array.slt | 54 +-
.../sqllogictest/test_files/arrow_typeof.slt | 10 +-
datafusion/sqllogictest/test_files/async_udf.slt | 15 +-
datafusion/sqllogictest/test_files/case.slt | 134 ++
datafusion/sqllogictest/test_files/cast.slt | 36 +
datafusion/sqllogictest/test_files/coalesce.slt | 6 +-
.../sqllogictest/test_files/count_star_rule.slt | 14 +-
.../test_files/create_external_table.slt | 2 +-
datafusion/sqllogictest/test_files/cte.slt | 92 +-
datafusion/sqllogictest/test_files/dates.slt | 102 +
datafusion/sqllogictest/test_files/decimal.slt | 34 +-
datafusion/sqllogictest/test_files/distinct_on.slt | 9 +-
.../test_files/dynamic_filter_pushdown_config.slt | 471 +++-
datafusion/sqllogictest/test_files/encoding.slt | 13 +
datafusion/sqllogictest/test_files/errors.slt | 5 +
datafusion/sqllogictest/test_files/explain.slt | 30 +-
.../sqllogictest/test_files/explain_tree.slt | 360 ++--
datafusion/sqllogictest/test_files/expr.slt | 7 +-
datafusion/sqllogictest/test_files/group_by.slt | 218 +-
datafusion/sqllogictest/test_files/grouping.slt | 12 +
.../sqllogictest/test_files/information_schema.slt | 18 +-
datafusion/sqllogictest/test_files/insert.slt | 15 +-
.../sqllogictest/test_files/insert_to_external.slt | 10 +-
datafusion/sqllogictest/test_files/join.slt.part | 85 +-
.../test_files/join_disable_repartition_joins.slt | 20 +-
.../test_files/join_is_not_distinct_from.slt | 93 +-
datafusion/sqllogictest/test_files/joins.slt | 772 ++++---
datafusion/sqllogictest/test_files/limit.slt | 16 +-
.../test_files/listing_table_statistics.slt | 2 +-
datafusion/sqllogictest/test_files/map.slt | 24 +-
datafusion/sqllogictest/test_files/math.slt | 101 +-
datafusion/sqllogictest/test_files/metadata.slt | 49 +
.../test_files/monotonic_projection_test.slt | 8 +-
datafusion/sqllogictest/test_files/order.slt | 57 +-
.../test_files/parquet_sorted_statistics.slt | 3 +-
.../sqllogictest/test_files/parquet_statistics.slt | 12 +-
datafusion/sqllogictest/test_files/predicates.slt | 43 +-
.../test_files/preserve_file_partitioning.slt | 596 +++++
.../sqllogictest/test_files/push_down_filter.slt | 85 +-
datafusion/sqllogictest/test_files/qualify.slt | 25 +-
datafusion/sqllogictest/test_files/repartition.slt | 14 +-
datafusion/sqllogictest/test_files/scalar.slt | 14 +-
datafusion/sqllogictest/test_files/select.slt | 17 +-
.../sqllogictest/test_files/set_variable.slt | 14 +
.../sqllogictest/test_files/slt_features.slt | 2 +-
.../test_files/spark/aggregate/try_sum.slt | 140 ++
.../test_files/spark/bitwise/bit_count.slt | 2 +-
.../test_files/spark/datetime/make_dt_interval.slt | 3 +-
.../sqllogictest/test_files/spark/math/hex.slt | 15 +
.../url/try_url_decode.slt} | 65 +-
.../test_files/spark/url/url_decode.slt | 60 +-
.../test_files/spark/url/url_encode.slt | 24 +-
.../test_files/string/string_literal.slt | 39 +
.../sqllogictest/test_files/string/string_view.slt | 2 +-
datafusion/sqllogictest/test_files/struct.slt | 38 +-
datafusion/sqllogictest/test_files/subquery.slt | 99 +-
.../sqllogictest/test_files/subquery_sort.slt | 9 +-
datafusion/sqllogictest/test_files/timestamps.slt | 447 +++-
datafusion/sqllogictest/test_files/topk.slt | 5 +-
.../sqllogictest/test_files/tpch/plans/q1.slt.part | 11 +-
.../test_files/tpch/plans/q10.slt.part | 44 +-
.../test_files/tpch/plans/q11.slt.part | 69 +-
.../test_files/tpch/plans/q12.slt.part | 20 +-
.../test_files/tpch/plans/q13.slt.part | 24 +-
.../test_files/tpch/plans/q14.slt.part | 15 +-
.../test_files/tpch/plans/q15.slt.part | 43 +-
.../test_files/tpch/plans/q16.slt.part | 44 +-
.../test_files/tpch/plans/q17.slt.part | 31 +-
.../test_files/tpch/plans/q18.slt.part | 43 +-
.../test_files/tpch/plans/q19.slt.part | 19 +-
.../sqllogictest/test_files/tpch/plans/q2.slt.part | 113 +-
.../test_files/tpch/plans/q20.slt.part | 62 +-
.../test_files/tpch/plans/q21.slt.part | 69 +-
.../test_files/tpch/plans/q22.slt.part | 44 +-
.../sqllogictest/test_files/tpch/plans/q3.slt.part | 37 +-
.../sqllogictest/test_files/tpch/plans/q4.slt.part | 22 +-
.../sqllogictest/test_files/tpch/plans/q5.slt.part | 68 +-
.../sqllogictest/test_files/tpch/plans/q7.slt.part | 74 +-
.../sqllogictest/test_files/tpch/plans/q8.slt.part | 96 +-
.../sqllogictest/test_files/tpch/plans/q9.slt.part | 68 +-
datafusion/sqllogictest/test_files/union.slt | 147 +-
datafusion/sqllogictest/test_files/unnest.slt | 32 +-
datafusion/sqllogictest/test_files/window.slt | 240 +--
.../sqllogictest/test_files/window_limits.slt | 10 +-
datafusion/substrait/Cargo.toml | 4 +-
datafusion/substrait/src/extensions.rs | 6 +-
datafusion/substrait/src/lib.rs | 3 +-
.../consumer/expr/aggregate_function.rs | 6 +-
.../src/logical_plan/consumer/expr/cast.rs | 4 +-
.../logical_plan/consumer/expr/field_reference.rs | 4 +-
.../consumer/expr/function_arguments.rs | 4 +-
.../src/logical_plan/consumer/expr/literal.rs | 48 +-
.../src/logical_plan/consumer/expr/mod.rs | 26 +-
.../logical_plan/consumer/expr/scalar_function.rs | 133 +-
.../logical_plan/consumer/expr/singular_or_list.rs | 4 +-
.../src/logical_plan/consumer/expr/subquery.rs | 8 +-
.../logical_plan/consumer/expr/window_function.rs | 10 +-
.../substrait/src/logical_plan/consumer/plan.rs | 67 +-
.../src/logical_plan/consumer/rel/aggregate_rel.rs | 8 +-
.../src/logical_plan/consumer/rel/exchange_rel.rs | 4 +-
.../src/logical_plan/consumer/rel/fetch_rel.rs | 6 +-
.../src/logical_plan/consumer/rel/join_rel.rs | 6 +-
.../substrait/src/logical_plan/consumer/rel/mod.rs | 6 +-
.../src/logical_plan/consumer/rel/project_rel.rs | 4 +-
.../src/logical_plan/consumer/rel/read_rel.rs | 124 +-
.../src/logical_plan/consumer/rel/set_rel.rs | 5 +-
.../src/logical_plan/consumer/rel/sort_rel.rs | 2 +-
.../logical_plan/consumer/substrait_consumer.rs | 12 +-
.../substrait/src/logical_plan/consumer/types.rs | 40 +-
.../substrait/src/logical_plan/consumer/utils.rs | 37 +-
.../producer/expr/aggregate_function.rs | 2 +-
.../src/logical_plan/producer/expr/cast.rs | 6 +-
.../logical_plan/producer/expr/field_reference.rs | 6 +-
.../src/logical_plan/producer/expr/if_then.rs | 2 +-
.../src/logical_plan/producer/expr/literal.rs | 8 +-
.../src/logical_plan/producer/expr/mod.rs | 8 +-
.../logical_plan/producer/expr/scalar_function.rs | 14 +-
.../logical_plan/producer/expr/singular_or_list.rs | 2 +-
.../src/logical_plan/producer/expr/subquery.rs | 2 +-
.../logical_plan/producer/expr/window_function.rs | 10 +-
.../substrait/src/logical_plan/producer/plan.rs | 6 +-
.../src/logical_plan/producer/rel/aggregate_rel.rs | 40 +-
.../src/logical_plan/producer/rel/exchange_rel.rs | 6 +-
.../src/logical_plan/producer/rel/fetch_rel.rs | 2 +-
.../src/logical_plan/producer/rel/join.rs | 6 +-
.../src/logical_plan/producer/rel/project_rel.rs | 4 +-
.../src/logical_plan/producer/rel/read_rel.rs | 169 +-
.../src/logical_plan/producer/rel/set_rel.rs | 2 +-
.../src/logical_plan/producer/rel/sort_rel.rs | 4 +-
.../logical_plan/producer/substrait_producer.rs | 14 +-
.../substrait/src/logical_plan/producer/types.rs | 10 +-
.../substrait/src/logical_plan/producer/utils.rs | 4 +-
datafusion/substrait/src/physical_plan/consumer.rs | 26 +-
datafusion/substrait/src/physical_plan/producer.rs | 157 +-
.../substrait/tests/cases/aggregation_tests.rs | 16 +-
.../substrait/tests/cases/consumer_integration.rs | 420 ++--
.../substrait/tests/cases/emit_kind_tests.rs | 44 +-
datafusion/substrait/tests/cases/function_test.rs | 8 +-
datafusion/substrait/tests/cases/logical_plans.rs | 65 +-
.../tests/cases/roundtrip_logical_plan.rs | 316 ++-
.../tests/cases/roundtrip_physical_plan.rs | 2 +-
datafusion/substrait/tests/cases/serialize.rs | 20 +-
.../substrait/tests/cases/substrait_validations.rs | 24 +-
datafusion/substrait/tests/utils.rs | 22 +-
datafusion/wasmtest/Cargo.toml | 2 +-
.../wasmtest/datafusion-wasm-app/package-lock.json | 102 +-
datafusion/wasmtest/src/lib.rs | 4 +-
dev/depcheck/Cargo.toml | 3 +-
.../depcheck/rust-toolchain.toml | 2 +-
dev/depcheck/src/main.rs | 2 +-
dev/rust_lint.sh | 14 +-
docs/requirements.txt | 2 +-
docs/source/contributor-guide/governance.md | 5 +-
docs/source/contributor-guide/howtos.md | 8 +-
docs/source/contributor-guide/inviting.md | 2 +-
docs/source/index.rst | 1 +
.../library-user-guide/custom-table-providers.md | 2 +-
.../library-user-guide/extending-operators.md | 7 +-
docs/source/library-user-guide/extending-sql.md | 339 +++
.../library-user-guide/functions/adding-udfs.md | 28 +-
docs/source/library-user-guide/query-optimizer.md | 20 +-
docs/source/library-user-guide/upgrading.md | 369 +++-
.../library-user-guide/using-the-dataframe-api.md | 2 +-
.../library-user-guide/working-with-exprs.md | 8 +-
docs/source/user-guide/cli/functions.md | 32 +
docs/source/user-guide/configs.md | 10 +-
docs/source/user-guide/crate-configuration.md | 30 +
docs/source/user-guide/expressions.md | 2 +-
docs/source/user-guide/introduction.md | 3 +
docs/source/user-guide/metrics.md | 10 +-
docs/source/user-guide/sql/ddl.md | 2 +-
docs/source/user-guide/sql/scalar_functions.md | 120 +-
rust-toolchain.toml | 2 +-
rustfmt.toml | 2 +-
test-utils/src/array_gen/binary.rs | 2 +-
test-utils/src/array_gen/boolean.rs | 2 +-
test-utils/src/array_gen/decimal.rs | 2 +-
test-utils/src/array_gen/primitive.rs | 4 +-
test-utils/src/array_gen/random_data.rs | 16 +-
test-utils/src/array_gen/string.rs | 2 +-
test-utils/src/string_gen.rs | 2 +-
test-utils/src/tpcds.rs | 2 +-
typos.toml | 6 +-
1264 files changed, 55450 insertions(+), 24791 deletions(-)
copy datafusion/proto-common/regen.sh => benchmarks/compare_tpcds.sh (52%)
copy datafusion/proto-common/regen.sh => benchmarks/compare_tpch.sh (55%)
copy benchmarks/queries/clickbench/queries/{ => sorted_data}/q0.sql (75%)
delete mode 100644 benchmarks/src/bin/tpch.rs
create mode 100644 benchmarks/src/smj.rs
copy {datafusion/core/tests/catalog => benchmarks/src/tpcds}/mod.rs (96%)
create mode 100644 benchmarks/src/tpcds/run.rs
delete mode 100644 benchmarks/src/tpch/convert.rs
create mode 100755 ci/scripts/doc_prettier_check.sh
copy ci/scripts/{rust_docs.sh => typos_check.sh} (84%)
rename datafusion-examples/examples/external_dependency/{dataframe-to-s3.rs =>
dataframe_to_s3.rs} (87%)
create mode 100644 datafusion-examples/examples/external_dependency/main.rs
rename datafusion-examples/examples/external_dependency/{query-aws-s3.rs =>
query_aws_s3.rs} (90%)
create mode 100644 datafusion-examples/examples/relation_planner/main.rs
create mode 100644
datafusion-examples/examples/relation_planner/match_recognize.rs
create mode 100644
datafusion-examples/examples/relation_planner/pivot_unpivot.rs
create mode 100644
datafusion-examples/examples/relation_planner/table_sample.rs
create mode 100644 datafusion/common/benches/with_hashes.rs
create mode 100644 datafusion/common/src/display/human_readable.rs
create mode 100644 datafusion/core/benches/preserve_file_partitioning.rs
create mode 100644 datafusion/core/tests/data/recursive_cte/closure.csv
create mode 100644 datafusion/core/tests/physical_optimizer/pushdown_sort.rs
create mode 100644 datafusion/core/tests/user_defined/relation_planner.rs
create mode 100644 datafusion/datasource-parquet/src/sort.rs
create mode 100644 datafusion/datasource/src/projection.rs
copy datafusion/execution/src/cache/{cache_unit.rs => file_metadata_cache.rs}
(78%)
create mode 100644 datafusion/execution/src/cache/list_files_cache.rs
rename datafusion/{physical-plan => execution}/src/metrics/baseline.rs (97%)
rename datafusion/{physical-plan => execution}/src/metrics/builder.rs (99%)
rename datafusion/{physical-plan => execution}/src/metrics/custom.rs (98%)
rename datafusion/{physical-plan => execution}/src/metrics/mod.rs (96%)
rename datafusion/{physical-plan => execution}/src/metrics/value.rs (99%)
copy datafusion/{core/tests/fuzz_cases/equivalence =>
ffi/src/execution}/mod.rs (86%)
create mode 100644 datafusion/ffi/src/execution/task_ctx.rs
create mode 100644 datafusion/ffi/src/execution/task_ctx_provider.rs
create mode 100644 datafusion/ffi/src/expr/columnar_value.rs
create mode 100644 datafusion/ffi/src/expr/distribution.rs
create mode 100644 datafusion/ffi/src/expr/expr_properties.rs
create mode 100644 datafusion/ffi/src/expr/interval.rs
copy datafusion/{core/tests/catalog => ffi/src/expr}/mod.rs (90%)
create mode 100644 datafusion/ffi/src/physical_expr/mod.rs
create mode 100644 datafusion/ffi/src/physical_expr/partitioning.rs
create mode 100644 datafusion/ffi/src/physical_expr/sort.rs
create mode 100644 datafusion/ffi/src/proto/logical_extension_codec.rs
copy datafusion/{core/tests/catalog => ffi/src/proto}/mod.rs (92%)
create mode 100644 datafusion/ffi/src/proto/physical_extension_codec.rs
rename datafusion/ffi/src/{session_config.rs => session/config.rs} (63%)
create mode 100644 datafusion/ffi/src/session/mod.rs
create mode 100644 datafusion/functions-nested/benches/array_has.rs
create mode 100644 datafusion/functions-nested/benches/array_slice.rs
create mode 100644 datafusion/functions/src/crypto/sha.rs
delete mode 100644 datafusion/functions/src/crypto/sha224.rs
delete mode 100644 datafusion/functions/src/crypto/sha256.rs
delete mode 100644 datafusion/functions/src/crypto/sha384.rs
delete mode 100644 datafusion/functions/src/crypto/sha512.rs
create mode 100644 datafusion/functions/src/datetime/make_time.rs
create mode 100644
datafusion/physical-expr/src/expressions/case/literal_lookup_table/boolean_lookup_table.rs
create mode 100644
datafusion/physical-expr/src/expressions/case/literal_lookup_table/bytes_like_lookup_table.rs
create mode 100644
datafusion/physical-expr/src/expressions/case/literal_lookup_table/mod.rs
create mode 100644
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs
create mode 100644 datafusion/physical-expr/src/simplifier/const_evaluator.rs
create mode 100644 datafusion/physical-expr/src/simplifier/not.rs
create mode 100644 datafusion/physical-optimizer/src/pushdown_sort.rs
create mode 100644
datafusion/physical-plan/src/joins/hash_join/inlist_builder.rs
copy datafusion/{core/src/bin/print_config_docs.rs =>
physical-plan/src/metrics.rs} (84%)
create mode 100644 datafusion/physical-plan/src/sort_pushdown.rs
create mode 100644 datafusion/proto/src/generated/datafusion.rs
create mode 100644 datafusion/spark/src/function/aggregate/try_sum.rs
copy datafusion/spark/src/function/url/{try_parse_url.rs => try_url_decode.rs}
(55%)
create mode 100644 datafusion/spark/src/function/url/url_decode.rs
create mode 100644 datafusion/spark/src/function/url/url_encode.rs
create mode 100644
datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
create mode 100644
datafusion/sqllogictest/test_files/spark/aggregate/try_sum.slt
copy datafusion/sqllogictest/test_files/{slt_features.slt =>
spark/url/try_url_decode.slt} (52%)
copy rust-toolchain.toml => dev/depcheck/rust-toolchain.toml (98%)
create mode 100644 docs/source/library-user-guide/extending-sql.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
