voonhous commented on PR #18201: URL: https://github.com/apache/hudi/pull/18201#issuecomment-5631687402
Did some tests on my own. The intuition here is that on master, each test that require spark context is starting a spark session and closing it after. So if there are _N_ tests. _N_ spark sessions will be started and closed, each will pay _X_ seconds of wait time. So total time spent waiting is _N * X_. However, a local-mode SparkContext on a JVM that has already loaded Spark once starts and stops in tens of milliseconds from what i can observe in my local runs. Doing an A/B test: 234 start-stop pairs removed, 2 seconds saved in total, which is pretty much insignificant. (Not sure if i am doing the tests wrongly) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
