advancedxy commented on issue #407: URL: https://github.com/apache/incubator-uniffle/issues/407#issuecomment-1348618906
> We can just cache the partitions of applicaitions which enable AQE AQE is enabled by default in later spark versions such as Spark 3.3 and also it may turned on by default in some production envs. So I believe by simply cache index file for all the AQE applications might not be sufficient. The cache behavior might still be triggered by access pattern. Also another question, in which cases AQE would trigger reading the same partition many times? I know the `OptimizeSkewedJoin` would split the same partition into multiple parts, and therefore trigger the behavior you described. Is there any other cases? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
