adriangb opened a new issue, #24710: URL: https://github.com/apache/datafusion/issues/24710
Follow-up from #24526. That PR uses the compact sorted-domain form only for lists strictly larger than `MAX_IN_LIST_SIZE` (20); at or below that, the per-value OR tree is kept. The PR documents this as a scope and compatibility choice rather than a measured threshold. The open question from review: since the compact form looks both cheaper and precision-equivalent, is there a reason to keep a lower bound at all? The answer at the time was that the benchmark keeps the 20-value case on the legacy path, so it doesn't establish anything about small lists — a focused comparison could justify broadening it. Possible outcomes: - The compact form wins everywhere, and the lower bound can go - There's a real crossover, in which case it's worth measuring — and it likely depends on container count as well as list length - 20 stays, in which case it might be worth giving the crossover its own named constant. `MAX_IN_LIST_SIZE` currently doubles as the config default and the representation threshold, so the two would move together, which doesn't seem intended. Probably wants the benchmark work first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
