tkaymak opened a new pull request, #40787: URL: https://github.com/apache/beam/pull/40787
Fixes #40120. Part of #36841. `SparkStructuredStreamingPipelineResult.getState()` now resolves a terminal state from a finished execution without blocking or throwing, CANCELLED when a cancel was requested, otherwise DONE or FAILED. `cancel()` only requests cancellation while the execution is still running and returns the current state, so it reports RUNNING right after the request and CANCELLED once the execution ended, as discussed on #40103. `waitUntilFinish` resolves through the same path, so a terminal state is never rewritten. A cancel after the pipeline already ended reports the real DONE or FAILED and no longer runs the cancel hook. MetricsPusher now stops once the execution ends instead of running until `waitUntilFinish` is called. The legacy `SparkPipelineResult` keeps its behavior. Tests assert CANCELLED behind `waitUntilFinish` on Spark 3 and 4. `testCancelRunsJobCancelHookOnce`, `testCancelIsAsynchronous`, `testCancelAfterNormalCompletionReturnsDone` and `testCancelAfterExceptionalCompletionReturnsFailed` fail on master and pass now. Full `:runners:spark:4:test` and `:runners:spark:3:test` pass locally. The trigger files run the Spark 4 and SparkStructuredStreaming ValidatesRunner suites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
