andygrove opened a new issue, #7297: URL: https://github.com/apache/datafusion-comet/issues/7297
### What is the problem the feature request solves? Part of #6834. Comet's cache format stores a dictionary-encoded column as plain values, so a low-cardinality string column pays the full inflate and decode cost on every read, and holds the full values in memory, even though the data has only a handful of distinct values. Storing the dictionary would shrink the cached batch and let the read path hand native code a dictionary array instead of materializing every value. It needs a way to carry the dictionary alongside the batch, either a per-batch schema message or a dictionary message in the IPC stream, plus handling for batches in the same relation whose dictionaries differ. ### Describe the potential solution Measure a low-cardinality string column with the cache benchmark first, to put a number on the size and decode cost at stake. If it is worth it, extend `CachedBatchIpc` to write and read dictionary batches, and keep the plain layout for columns whose dictionary does not pay for itself, such as a high-cardinality string column. ### Additional context Found in the cache audit (#6834). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
