This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/github_actions/main/arduino/setup-protoc-3
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
omit 03c79d1 build(deps): bump arduino/setup-protoc from 1 to 3
add 5296c0c Allow PyDataFrame to be used from other projects (#582)
add 476ca22 docs: Add ASF attribution (#580)
add 697ca2c Add array functions (#560)
add 27a9264 feat: Add `flatten` array function (#562)
add dd4ed97 Prepare 36.0.0 Release (#599)
add 3a82be0 Python3.11 support (#603)
add 76ecf56 Switch to Ruff for Python linting (#529)
add 1ff22bb remove SQL om polars/pandas/cudf experiment (#602)
add 18ac182 build(deps): bump object_store from 0.9.0 to 0.9.1 (#611)
add 6a895c6 More missing array funcs (#605)
add 5e97701 feat: add execute_stream and execute_stream_partitioned (#610)
add 7d08ec9 build(deps): bump uuid from 1.7.0 to 1.8.0 (#615)
add 6e570e2 Bind SQLOptions and relative ctx method #567 (#588)
add 7204a35 bugfix: no panic on empty table (#613)
add 84415dd Expose `register_listing_table` (#618)
add 7366f89 Expose unnest feature (#641)
add 13d34a5 update domain names and paths in asf yaml (#643)
add 88ce6e6 use python 3.11 to publish docs (#645)
add 67d4cfb docs: update docs CI to install python-311 requirements (#661)
add ee93cdd Upgrade Datafusion to v37.1.0 (#669)
add 7f27651 Generate changelog (#673)
add 7fd0c96 Add document about basics of working with expressions (#668)
add c6bdc9f chore: Update Python release process now that DataFusion is
TLP (#674)
add 7d4a40c Fix Docs (#676)
add d71c436 Add examples from TPC-H (#666)
add d41eba4 ci: clean conda cache before building the packages (#689)
add 01a370e Upgrade to datafusion 38 (#691)
add 0c82b3f chore: update to maturin's recommended project layout for
rust/python… (#695)
add 1da28b5 chore: update cargo deps (#698)
add 344ebad feat: add python bindings for ends_with function (#693)
add d6c42b4 feat: expose `named_struct` in python (#700)
add 856b310 Website fixes (#702)
add f12a487 Update changelog for 38.0.0 (#704)
add dfbb3ca tsaucer/run TPC-H examples in CI (#711)
add e029559 test: parametrize test_array_functions (#678)
add 0d42e77 Use int64 for TPC-H keys and set input schema to not nullable
(#714)
add ea4a4dc change version to 38.0.1 (#716)
add ef88921 ci: add substrait feature to linux builds (#720)
add 84c1195 Docs deploy action (#721)
add 1d4743f update deps (#723)
add 860283a Upgrade maturin (#725)
add b5446ef Upgrade datafusion 39 (#728)
add 1f49d46 use ScalarValue::to_pyarrow to convert to python object (#731)
add c7ea90d Pyo3 `Bound<'py, T>` api (#734)
add 532dc38 github test action: drop python 3.7, add python 3.12 (#736)
add 45a6844 Pyarrow filter pushdowns (#735)
add faa26b2 build(deps): bump syn from 2.0.66 to 2.0.67 (#738)
add 32d6975 Pyo3 refactorings (#740)
add ec835ab UDAF `sum` workaround (#741)
add f374b44 changelog (#742)
add 75896e5 build(deps): bump uuid from 1.8.0 to 1.9.1 (#744)
add 594655f build(deps): bump mimalloc from 0.1.42 to 0.1.43 (#745)
add a3908ed build(deps): bump syn from 2.0.67 to 2.0.68 (#746)
add faa5a3f Tsaucer/find window fn (#747)
add aa8aa9c Python wrapper classes for all user interfaces (#750)
add f00b8ee Expose array sort (#764)
add fd6b4df Upgrade protobuf and remove GH Action googletest-installer
(#773)
add f580155 Upgrade Datafusion 40 (#771)
add 66bfe36 Bugfix: Calling count with None arguments (#768)
add 951d6b9 Add in user example that compares a two different approaches
to UDFs (#770)
add 9a6805e Add missing exports for wrapper modules (#782)
add 3eb198b Add PyExpr to_variant conversions (#793)
add 1d61548 Add missing expressions to wrapper export (#795)
add bd0e820 Doc/cross reference (#791)
add 2205e05 Re-Enable `num_centroids` to `approx_percentile_cont` (#798)
add b6f06f7 UDAF process all state variables (#799)
add e206a0c chore: Prepare for 40.0.0 release (#801)
add b2982ec Add typing-extensions dependency to pyproject (#805)
add 805183b feat: enable list of paths for read_csv (#824)
add 3b5085e Upgrade deps to datafusion 41 (#802)
add 02414d8 Fix SessionContext init with only SessionConfig (#827)
add 22c70ef build(deps): upgrade actions/{upload,download}-artifact@v3 to
v4 (#829)
add 766e2ed Run ruff format in CI (#837)
add 69ed7fe Add PyCapsule support for Arrow import and export (#825)
add 003eea8 Feature/expose when function (#836)
add 90f5b5b Add Window Functions for use with function builder (#808)
add 57eb959 chore: fix typos (#844)
add e8ebc4f build(ci): use proper mac runners (#841)
add 909b809 Set of small features (#839)
add fe0738a feat: better exception and message for table not found (#851)
add 859acb4 feat: make cast accept built-in Python types (#858)
add a0913c7 chore: fix docstrings, typos (#852)
add 6154ef9 chore: Use datafusion re-exported dependencies (#856)
add 4ea0032 add section to the contributor guide on separating python and
rust code (#860)
add fd8224e Update Aggregate functions to take builder parameters (#859)
add 89b77ab prepare release of `datafusion-python` 41.0.0 (#866)
add 02d4453 feat: expose between (#868)
add 6c8bf5f Upgrade datafusion (#867)
add 7bd8327 build(deps): bump arduino/setup-protoc from 1 to 3
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (03c79d1)
\
N -- N -- N
refs/heads/dependabot/github_actions/main/arduino/setup-protoc-3 (7bd8327)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 14 +-
.github/workflows/build.yml | 120 +-
.github/workflows/conda.yml | 11 +-
.github/workflows/dev.yml | 4 +-
.github/workflows/docs.yaml | 27 +-
.github/workflows/test.yaml | 57 +-
.gitignore | 5 +
.pre-commit-config.yaml | 21 +-
CHANGELOG.md | 782 +++---
Cargo.lock | 1833 ++++++++------
Cargo.toml | 39 +-
README.md | 84 +-
benchmarks/db-benchmark/db-benchmark.dockerfile | 2 +-
benchmarks/db-benchmark/groupby-datafusion.py | 16 +-
benchmarks/db-benchmark/join-datafusion.py | 27 +-
benchmarks/tpch/tpch-gen.sh | 15 +-
benchmarks/tpch/tpch.py | 4 +-
ci/scripts/{rust_fmt.sh => python_lint.sh} | 3 +-
conda/environments/datafusion-dev.yaml | 2 +-
conda/recipes/meta.yaml | 5 +-
datafusion/__init__.py | 225 --
datafusion/context.py | 142 --
datafusion/cudf.py | 99 -
datafusion/functions.py | 23 -
datafusion/pandas.py | 95 -
datafusion/polars.py | 106 -
datafusion/substrait.py | 23 -
datafusion/tests/test_aggregation.py | 161 --
datafusion/tests/test_expr.py | 118 -
datafusion/tests/test_functions.py | 602 -----
dev/release/README.md | 59 +-
dev/release/check-rat-report.py | 4 +-
dev/release/create-tarball.sh | 34 +-
dev/release/generate-changelog.py | 119 +-
dev/release/release-tarball.sh | 12 +-
dev/release/update_change_log-datafusion-python.sh | 33 -
dev/release/update_change_log.sh | 87 -
dev/release/verify-release-candidate.sh | 28 +-
docs/.gitignore | 2 +
docs/README.md | 47 +-
docs/mdbook/src/index.md | 2 +-
docs/requirements.txt | 6 +-
docs/source/_static/images/2x_bgwhite_original.png | Bin 0 -> 60451 bytes
.../images/DataFusion-Logo-Background-White.png | Bin 12401 -> 0 bytes
.../images/DataFusion-Logo-Background-White.svg | 1 -
.../source/_static/images/DataFusion-Logo-Dark.png | Bin 20134 -> 0 bytes
.../source/_static/images/DataFusion-Logo-Dark.svg | 1 -
.../_static/images/DataFusion-Logo-Light.png | Bin 19102 -> 0 bytes
.../_static/images/DataFusion-Logo-Light.svg | 1 -
docs/source/_static/images/original.png | Bin 0 -> 26337 bytes
docs/source/_static/images/original.svg | 31 +
docs/source/_static/images/original2x.png | Bin 0 -> 66517 bytes
docs/source/_static/theme_overrides.css | 2 +-
docs/source/_templates/docs-sidebar.html | 2 +-
docs/source/_templates/layout.html | 21 +
docs/source/api.rst | 31 -
docs/source/api/dataframe.rst | 27 -
docs/source/api/execution_context.rst | 29 -
docs/source/api/expression.rst | 27 -
docs/source/api/functions.rst | 27 -
docs/source/api/object_store.rst | 27 -
docs/source/conf.py | 57 +-
docs/source/contributor-guide/introduction.rst | 11 +
docs/source/index.rst | 10 +-
docs/source/user-guide/basics.rst | 14 +-
.../user-guide/common-operations/aggregations.rst | 208 +-
.../user-guide/common-operations/basic-info.rst | 8 +-
.../user-guide/common-operations/expressions.rst | 133 +
.../user-guide/common-operations/functions.rst | 14 +-
docs/source/user-guide/common-operations/index.rst | 1 +
docs/source/user-guide/common-operations/joins.rst | 4 +-
.../common-operations/select-and-filter.rst | 10 +-
.../user-guide/common-operations/udf-and-udfa.rst | 9 +-
.../user-guide/common-operations/windows.rst | 187 +-
docs/source/user-guide/configuration.rst | 51 +
docs/source/user-guide/io/arrow.rst | 73 +
docs/source/user-guide/io/avro.rst | 2 +-
docs/source/user-guide/io/csv.rst | 6 +-
docs/source/user-guide/io/index.rst | 6 +-
docs/source/user-guide/io/json.rst | 2 +-
docs/source/user-guide/io/parquet.rst | 6 +-
examples/README.md | 64 +
examples/export.py | 2 +-
examples/import.py | 2 +-
examples/python-udf-comparisons.py | 186 ++
examples/sql-on-cudf.py | 24 -
examples/sql-on-pandas.py | 24 -
examples/sql-on-polars.py | 26 -
examples/sql-using-python-udaf.py | 8 +-
examples/substrait.py | 21 +-
examples/tpch/.gitignore | 2 +
examples/tpch/README.md | 57 +
examples/tpch/_tests.py | 125 +
examples/tpch/convert_data_to_parquet.py | 143 ++
examples/tpch/q01_pricing_summary_report.py | 89 +
examples/tpch/q02_minimum_cost_supplier.py | 141 ++
examples/tpch/q03_shipping_priority.py | 87 +
examples/tpch/q04_order_priority_checking.py | 79 +
examples/tpch/q05_local_supplier_volume.py | 101 +
examples/tpch/q06_forecasting_revenue_change.py | 86 +
examples/tpch/q07_volume_shipping.py | 124 +
examples/tpch/q08_market_share.py | 178 ++
examples/tpch/q09_product_type_profit_measure.py | 96 +
examples/tpch/q10_returned_item_reporting.py | 107 +
.../tpch/q11_important_stock_identification.py | 83 +
examples/tpch/q12_ship_mode_order_priority.py | 111 +
examples/tpch/q13_customer_distribution.py | 67 +
examples/tpch/q14_promotion_effect.py | 83 +
examples/tpch/q15_top_supplier.py | 87 +
examples/tpch/q16_part_supplier_relationship.py | 87 +
examples/tpch/q17_small_quantity_order.py | 76 +
examples/tpch/q18_large_volume_customer.py | 66 +
examples/tpch/q19_discounted_revenue.py | 137 +
examples/tpch/q20_potential_part_promotion.py | 98 +
examples/tpch/q21_suppliers_kept_orders_waiting.py | 115 +
examples/tpch/q22_global_sales_opportunity.py | 80 +
datafusion/common.py => examples/tpch/util.py | 19 +-
pyproject.toml | 28 +-
python/datafusion/__init__.py | 115 +
python/datafusion/catalog.py | 76 +
python/datafusion/common.py | 62 +
python/datafusion/context.py | 1029 ++++++++
python/datafusion/dataframe.py | 572 +++++
python/datafusion/expr.py | 718 ++++++
python/datafusion/functions.py | 2659 ++++++++++++++++++++
.../expr.py => python/datafusion/input/__init__.py | 10 +-
{datafusion => python/datafusion}/input/base.py | 25 +-
.../datafusion}/input/location.py | 18 +-
{datafusion => python/datafusion}/object_store.py | 14 +-
.../__init__.py => python/datafusion/py.typed | 8 +-
python/datafusion/record_batch.py | 76 +
python/datafusion/substrait.py | 193 ++
.../datafusion}/tests/__init__.py | 0
.../datafusion}/tests/conftest.py | 3 +-
.../datafusion}/tests/data_test_context/data.json | 0
{datafusion => python/datafusion}/tests/generic.py | 14 +-
python/datafusion/tests/test_aggregation.py | 437 ++++
.../datafusion}/tests/test_catalog.py | 0
.../datafusion}/tests/test_config.py | 5 +-
.../datafusion}/tests/test_context.py | 290 ++-
.../datafusion}/tests/test_dataframe.py | 507 +++-
python/datafusion/tests/test_expr.py | 218 ++
python/datafusion/tests/test_functions.py | 1057 ++++++++
.../datafusion}/tests/test_imports.py | 24 +-
.../datafusion}/tests/test_indexing.py | 0
.../datafusion}/tests/test_input.py | 4 +-
.../datafusion}/tests/test_sql.py | 185 +-
.../datafusion}/tests/test_store.py | 0
.../datafusion}/tests/test_substrait.py | 46 +-
.../datafusion}/tests/test_udaf.py | 8 +-
python/datafusion/tests/test_wrapper_coverage.py | 63 +
python/datafusion/udf.py | 248 ++
requirements-310.txt | 400 ++-
requirements-311.txt | 322 ++-
requirements-312.txt | 184 ++
requirements-37.txt | 264 --
requirements.in | 7 +-
src/catalog.rs | 4 +-
src/common.rs | 6 +-
src/common/data_type.rs | 87 +-
src/common/df_field.rs | 111 -
src/common/df_schema.rs | 2 +-
src/common/schema.rs | 47 +-
src/config.rs | 4 +-
src/context.rs | 356 ++-
src/dataframe.rs | 384 ++-
src/dataset.rs | 37 +-
src/dataset_exec.rs | 98 +-
src/expr.rs | 286 ++-
src/expr/aggregate.rs | 18 +-
src/expr/aggregate_expr.rs | 6 +-
src/expr/alias.rs | 32 +-
src/expr/analyze.rs | 2 +-
src/expr/between.rs | 2 +-
src/expr/binary_expr.rs | 2 +-
src/expr/bool_expr.rs | 2 +-
src/expr/case.rs | 2 +-
src/expr/cast.rs | 2 +-
src/expr/column.rs | 2 +-
src/expr/conditional_expr.rs | 2 +-
src/expr/create_memory_table.rs | 2 +-
src/expr/create_view.rs | 2 +-
src/expr/cross_join.rs | 2 +-
src/expr/distinct.rs | 2 +-
src/expr/drop_table.rs | 2 +-
src/expr/empty_relation.rs | 2 +-
src/expr/exists.rs | 15 +-
src/expr/explain.rs | 2 +-
src/expr/extension.rs | 2 +-
src/expr/filter.rs | 2 +-
src/expr/grouping_set.rs | 2 +-
src/expr/in_list.rs | 22 +-
src/expr/in_subquery.rs | 22 +-
src/expr/indexed_field.rs | 2 +-
src/expr/join.rs | 2 +-
src/expr/like.rs | 2 +-
src/expr/limit.rs | 2 +-
src/expr/literal.rs | 9 +-
src/expr/placeholder.rs | 21 +-
src/expr/projection.rs | 4 +-
src/expr/repartition.rs | 2 +-
src/expr/scalar_function.rs | 65 -
src/expr/scalar_subquery.rs | 2 +-
src/expr/signature.rs | 3 +-
src/expr/sort.rs | 10 +-
src/expr/sort_expr.rs | 90 +
src/expr/subquery.rs | 2 +-
src/expr/subquery_alias.rs | 2 +-
src/expr/table_scan.rs | 6 +-
src/expr/union.rs | 2 +-
src/expr/{union.rs => unnest.rs} | 46 +-
src/expr/{alias.rs => unnest_expr.rs} | 58 +-
src/expr/window.rs | 23 +-
src/functions.rs | 998 ++++++--
src/lib.rs | 50 +-
src/physical_plan.rs | 4 +-
src/pyarrow_filter_expression.rs | 54 +-
src/record_batch.rs | 9 +
src/sql/logical.rs | 23 +-
src/store.rs | 2 +-
src/substrait.rs | 16 +-
src/udaf.rs | 55 +-
src/udf.rs | 20 +-
src/utils.rs | 10 +-
224 files changed, 16455 insertions(+), 5619 deletions(-)
copy ci/scripts/{rust_fmt.sh => python_lint.sh} (94%)
delete mode 100644 datafusion/__init__.py
delete mode 100644 datafusion/context.py
delete mode 100644 datafusion/cudf.py
delete mode 100644 datafusion/functions.py
delete mode 100644 datafusion/pandas.py
delete mode 100644 datafusion/polars.py
delete mode 100644 datafusion/substrait.py
delete mode 100644 datafusion/tests/test_aggregation.py
delete mode 100644 datafusion/tests/test_expr.py
delete mode 100644 datafusion/tests/test_functions.py
delete mode 100755 dev/release/update_change_log-datafusion-python.sh
delete mode 100755 dev/release/update_change_log.sh
create mode 100644 docs/.gitignore
create mode 100644 docs/source/_static/images/2x_bgwhite_original.png
delete mode 100644
docs/source/_static/images/DataFusion-Logo-Background-White.png
delete mode 100644
docs/source/_static/images/DataFusion-Logo-Background-White.svg
delete mode 100644 docs/source/_static/images/DataFusion-Logo-Dark.png
delete mode 100644 docs/source/_static/images/DataFusion-Logo-Dark.svg
delete mode 100644 docs/source/_static/images/DataFusion-Logo-Light.png
delete mode 100644 docs/source/_static/images/DataFusion-Logo-Light.svg
create mode 100644 docs/source/_static/images/original.png
create mode 100644 docs/source/_static/images/original.svg
create mode 100644 docs/source/_static/images/original2x.png
delete mode 100644 docs/source/api.rst
delete mode 100644 docs/source/api/dataframe.rst
delete mode 100644 docs/source/api/execution_context.rst
delete mode 100644 docs/source/api/expression.rst
delete mode 100644 docs/source/api/functions.rst
delete mode 100644 docs/source/api/object_store.rst
create mode 100644 docs/source/user-guide/common-operations/expressions.rst
create mode 100644 docs/source/user-guide/configuration.rst
create mode 100644 docs/source/user-guide/io/arrow.rst
create mode 100644 examples/python-udf-comparisons.py
delete mode 100644 examples/sql-on-cudf.py
delete mode 100644 examples/sql-on-pandas.py
delete mode 100644 examples/sql-on-polars.py
create mode 100644 examples/tpch/.gitignore
create mode 100644 examples/tpch/README.md
create mode 100644 examples/tpch/_tests.py
create mode 100644 examples/tpch/convert_data_to_parquet.py
create mode 100644 examples/tpch/q01_pricing_summary_report.py
create mode 100644 examples/tpch/q02_minimum_cost_supplier.py
create mode 100644 examples/tpch/q03_shipping_priority.py
create mode 100644 examples/tpch/q04_order_priority_checking.py
create mode 100644 examples/tpch/q05_local_supplier_volume.py
create mode 100644 examples/tpch/q06_forecasting_revenue_change.py
create mode 100644 examples/tpch/q07_volume_shipping.py
create mode 100644 examples/tpch/q08_market_share.py
create mode 100644 examples/tpch/q09_product_type_profit_measure.py
create mode 100644 examples/tpch/q10_returned_item_reporting.py
create mode 100644 examples/tpch/q11_important_stock_identification.py
create mode 100644 examples/tpch/q12_ship_mode_order_priority.py
create mode 100644 examples/tpch/q13_customer_distribution.py
create mode 100644 examples/tpch/q14_promotion_effect.py
create mode 100644 examples/tpch/q15_top_supplier.py
create mode 100644 examples/tpch/q16_part_supplier_relationship.py
create mode 100644 examples/tpch/q17_small_quantity_order.py
create mode 100644 examples/tpch/q18_large_volume_customer.py
create mode 100644 examples/tpch/q19_discounted_revenue.py
create mode 100644 examples/tpch/q20_potential_part_promotion.py
create mode 100644 examples/tpch/q21_suppliers_kept_orders_waiting.py
create mode 100644 examples/tpch/q22_global_sales_opportunity.py
rename datafusion/common.py => examples/tpch/util.py (65%)
create mode 100644 python/datafusion/__init__.py
create mode 100644 python/datafusion/catalog.py
create mode 100644 python/datafusion/common.py
create mode 100644 python/datafusion/context.py
create mode 100644 python/datafusion/dataframe.py
create mode 100644 python/datafusion/expr.py
create mode 100644 python/datafusion/functions.py
rename datafusion/expr.py => python/datafusion/input/__init__.py (79%)
rename {datafusion => python/datafusion}/input/base.py (68%)
rename {datafusion => python/datafusion}/input/location.py (87%)
rename {datafusion => python/datafusion}/object_store.py (75%)
rename datafusion/input/__init__.py => python/datafusion/py.typed (88%)
create mode 100644 python/datafusion/record_batch.py
create mode 100644 python/datafusion/substrait.py
rename {datafusion => python/datafusion}/tests/__init__.py (100%)
rename {datafusion => python/datafusion}/tests/conftest.py (95%)
rename {datafusion => python/datafusion}/tests/data_test_context/data.json
(100%)
rename {datafusion => python/datafusion}/tests/generic.py (86%)
create mode 100644 python/datafusion/tests/test_aggregation.py
rename {datafusion => python/datafusion}/tests/test_catalog.py (100%)
rename {datafusion => python/datafusion}/tests/test_config.py (91%)
rename {datafusion => python/datafusion}/tests/test_context.py (54%)
rename {datafusion => python/datafusion}/tests/test_dataframe.py (56%)
create mode 100644 python/datafusion/tests/test_expr.py
create mode 100644 python/datafusion/tests/test_functions.py
rename {datafusion => python/datafusion}/tests/test_imports.py (92%)
rename {datafusion => python/datafusion}/tests/test_indexing.py (100%)
rename {datafusion => python/datafusion}/tests/test_input.py (91%)
rename {datafusion => python/datafusion}/tests/test_sql.py (65%)
rename {datafusion => python/datafusion}/tests/test_store.py (100%)
rename {datafusion => python/datafusion}/tests/test_substrait.py (52%)
rename {datafusion => python/datafusion}/tests/test_udaf.py (94%)
create mode 100644 python/datafusion/tests/test_wrapper_coverage.py
create mode 100644 python/datafusion/udf.py
create mode 100644 requirements-312.txt
delete mode 100644 requirements-37.txt
delete mode 100644 src/common/df_field.rs
delete mode 100644 src/expr/scalar_function.rs
create mode 100644 src/expr/sort_expr.rs
copy src/expr/{union.rs => unnest.rs} (65%)
copy src/expr/{alias.rs => unnest_expr.rs} (57%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
