BlakeOrth commented on PR #18146: URL: https://github.com/apache/datafusion/pull/18146#issuecomment-3493054476
@alamb I have moved the tests that rely in a `SessionState` into the `core_integration` tests to appease the CI gods. Thank you for your thorough and thoughtful review for this PR! Even though it ends up taking a bit longer, I genuinely appreciate the drive to use evidence-based approaches for measurable performance improvements. > though this PR won't interleave IO and CPU the way the previous one did Yes, I agree with this. While I didn't summarize and submit them, the memory based benchmarks in https://github.com/apache/datafusion/pull/18361 seemed to show evidence of this. Hopefully all of this nuance becomes moot when we get some caching implemented soon :smile: > Note you don't have to be the only one implementing this -- I think we could implement caching in parallel with smarter use of partitioning values for `LIST`ing For sure! I'm in no way trying to hoard work/issues. Unless something comes up my personal plan is to continue chipping away at this effort in whatever way I can help, regardless of whether that's implementations, reviews etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
