Will Jones created ARROW-14730: ---------------------------------- Summary: [C++][R][Python] Support reading from Delta Lake tables Key: ARROW-14730 URL: https://issues.apache.org/jira/browse/ARROW-14730 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Will Jones
[Delta Lake|https://delta.io/] is a parquet table format that supports ACID transactions. It's popularized by Databricks, which uses it as the default table format in their platform. Previously, it's only been readable from Spark, but now there is an effort in [delta-rs|https://github.com/delta-io/delta-rs] to make it accessible from elsewhere. There is already some integration with DataFusion (see: https://github.com/apache/arrow-datafusion/issues/525). There does already exist [a method to read Delta Lake tables into Arrow tables in Python|https://delta-io.github.io/delta-rs/python/api_reference.html#deltalake.table.DeltaTable.to_pyarrow_table] in the delta-rs Python bindings. This includes filtering by partitions. Is there a good way we could integrate this functionality with Arrow C++ Dataset and expose that in Python and R? Would that be something that should be implemented in Arrow libraries or in delta-rs? -- This message was sent by Atlassian Jira (v8.20.1#820001)