sunchao commented on PR #5650: URL: https://github.com/apache/datafusion-comet/pull/5650#issuecomment-5520138751
There may be a useful follow-up optimization here: **preserve dictionary encoding where downstream consumers support it, and unpack only where necessary.** DataFusion can hash dictionary values once and reuse the hashes by key, Arrow filtering preserves dictionaries, and `CometDictionaryVector` supports lazy lookups for some layouts. For the native path, this would require preserving dictionary types consistently in physical schemas and checking downstream operator support. The JVM path could retain supported layouts while decoding others. It would be worth benchmarking CPU and peak memory on low-cardinality string columns, with coverage for dictionaries changing between batches, nested types, and null dictionary values. This could be explored separately from the compatibility fix in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
