milenkovicm opened a new issue, #1827: URL: https://github.com/apache/datafusion-ballista/issues/1827
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Currently, shuffle collects basic statistics like the number of rows and bytes. This data is used downstream for some optimisation. However, I understand those statistics aren’t sufficient for join estimations, leading to incorrect decisions in some cases. No column based statistics such as null count or distinct count is collected during exchange. **Describe the solution you'd like** investigate what would be appropriate approach to collect null and distinct count for columns used in repartition expressions. Collecting null count may not be big problem but distinct count might and approximation may be used for it **Describe alternatives you've considered** **Additional context** - collected statistic should be propagated to planners -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
