jayzhan211 commented on issue #26696: URL: https://github.com/apache/datafusion/issues/26696#issuecomment-6395394520
The same problem shows up in arrow's row format: apache/arrow-rs#11901. `RowConverter` encodes every dictionary value for each batch, so once a GROUP BY on a high-cardinality dictionary column spills, merging the spilled runs barely makes progress. ClickBench Q33 with an 8 GB memory limit was still running after 5 hours; it takes about 15 s without a limit. So far this issue covers: - hashing and the string functions: #26703 - `concat_dictionaries` in the repartition coalescer: apache/arrow-rs#11885 - row conversion in spill merges: apache/arrow-rs#11901 I'll keep tracking them here, so let me know if you'd rather have a separate issue for the spill case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
