andygrove opened a new issue, #5108: URL: https://github.com/apache/arrow-datafusion/issues/5108
**Describe the bug** Original bug was filed against Python bindings: https://github.com/apache/arrow-datafusion-python/issues/157 Describe the bug try a sort and export a parquet file using Colab generate an Out of memory error To Reproduce !curl -L 'https://drive.google.com/uc?export=download&id=18gv0Yd_a-Zc7CSolol8qeYVAAzSthnSN&confirm=t' > lineitem.parquet from datafusion import SessionContext ctx = SessionContext() ctx.register_parquet('lineitem', 'lineitem.parquet') df = ctx.sql("select * from lineitem order by l_shipdate") df.write_parquet("lineitem_Datafusion.parquet") Expected behavior I expected to use only the available memory here is the link comparing the same using Polars and DuckDB https://colab.research.google.com/drive/1pfAPpIG7jpvGB_aHj-PXX66vRaRT0xlj#scrollTo=O8-lyg1y6RT2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
