alamb commented on issue #18411: URL: https://github.com/apache/datafusion/issues/18411#issuecomment-3473952514
I can reproduce some speed difference locally with the tpchgen dataset: Make dataset with https://github.com/clflushopt/tpchgen-rs ```shell tpchgen-cli --tables=lineitem --format=parquet --scale-factor 1000 ``` Dataset is 246GB: ```shell andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ du -s -h lineitem.parquet 246G lineitem.parquet ``` datafusion-cli ``` time datafusion-cli -c "select l_returnflag,l_linestatus, count(*) as count_order from 'lineitem.parquet' group by l_returnflag, l_linestatus;" DataFusion CLI v50.3.0 +--------------+--------------+-------------+ | l_returnflag | l_linestatus | count_order | +--------------+--------------+-------------+ | N | F | 38657144 | | N | O | 2999975927 | | A | F | 1480681438 | | R | F | 1480675200 | +--------------+--------------+-------------+ 4 row(s) fetched. Elapsed 12.473 seconds. real 0m12.711s user 2m53.323s sys 0m5.760s ``` Duckdb completes it almost 2x faster: ``` andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ time duckdb -c "select l_returnflag,l_linestatus, count(*) as count_order from 'lineitem.parquet' group by l_returnflag, l_linestatus;" 100% ▕██████████████████████████████████████▏ (00:00:06.37 elapsed) ┌──────────────┬──────────────┬─────────────┐ │ l_returnflag │ l_linestatus │ count_order │ │ varchar │ varchar │ int64 │ ├──────────────┼──────────────┼─────────────┤ │ R │ F │ 1480675200 │ │ N │ F │ 38657144 │ │ A │ F │ 1480681438 │ │ N │ O │ 2999975927 │ └──────────────┴──────────────┴─────────────┘ real 0m7.220s user 1m34.961s sys 0m4.062s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
