gaogaotiantian commented on PR #55552: URL: https://github.com/apache/spark/pull/55552#issuecomment-4348066509
Before a more detailed review of the code itself - we do have a more dedicated benchmark framework based on `asv` now. The code lives in `python/benchmarks` and is mostly done by @Yicong-Huang . Can we use that framework for this and maybe get some more trustworthy data? The current benchmark is meaningful of course, but there could be some factors that impact the result, like garbage collection. Also `asv` provides memory consumption metrics too. It would be nice to write all of our benchmarks with the same framework. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
