andygrove commented on PR #4926: URL: https://github.com/apache/arrow-datafusion/pull/4926#issuecomment-1384372325
> The repartitioning isn't really redundant though? Suppose we only read from one parquet file and do repartition by hash after, we would gain parallelism by first doing a (fast) round robin partition, and a (slower) hash repartition which can now be parallelized. That's a good point. The round-robin partitioning is not so fast in Ballista because it introduces an extra query stage with shuffles. Maybe this optimization should just be added in Ballista rather than in DataFusion? > > The round robin repartitioning however would benefit from being also down the `FilterExec` in this case, so the filtering can benefit from more parallelism as well. Yes, I also just noticed that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
