Baunsgaard opened a new pull request, #2516: URL: https://github.com/apache/systemds/pull/2516
The JVM-wide singleton `SparkContext` in `SparkExecutionContext` was stopped unconditionally at the end of every `DMLScript.execute()`. With surefire parallel tests (`parallel=classes`, `threadCount=2`), one finishing execution would stop the context while a concurrent execution still had an in-flight Spark job. ## Change - Make `close()` reference-counted. `SparkExecutionContext.enterSparkExecution()` is called from `DMLScript.execute()` when the execution context is a `SparkExecutionContext`, and `close()` only stops the shared context once the active-execution count reaches zero. ## Notes - Single-run behaviour is unchanged (enter -> 1, close -> 0 -> stop). - The counter is balanced: enter and close run under the same condition at the same single site, with `close()` always in the `finally` block. - Only the `DMLScript.execute()` path that owns the context is covered; MLContext manages an externally-owned context separately and is unaffected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
