Yicong-Huang commented on PR #55552: URL: https://github.com/apache/spark/pull/55552#issuecomment-4348730331
> However, the current ASV benchmarks in python/benchmarks/ (e.g., bench_eval_type.py) work by directly calling worker_main(infile, outfile) with a mock protocol — they bypass the JVM and socket communication entirely. This means they can only measure the Python worker's internal processing time. Those benchmark are designed to be micro benchmark without JVM: so there is no plan to extend it. But I think we definitely can use ASV to measure PySpark performance directly, end to end. We just need to create one bench file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
