tkaymak commented on issue #40120: URL: https://github.com/apache/beam/issues/40120#issuecomment-5797784734
Hi @singhmrinal02, I am back. Please go ahead, the issue is yours. I am not touching this code for now, my next work on #36841 is the state internals for slice 5. A few pointers: 1. The main change is in `SparkStructuredStreamingPipelineResult.cancel()` and `getState()`. The state should come from the execution, so a cancel after the job already ended reports DONE or FAILED. 2. `StructuredStreamingPipelineStateTest` asserts CANCELLED right after `cancel()`. Those assertions need to move behind `waitUntilFinish()`. 3. The code is shared by Spark 3 and 4, so please run the tests on both, `:runners:spark:3:test` and `:runners:spark:4:test`. 4. Every test that cancels must also call `waitUntilFinish()`. Otherwise the session gets stopped under the next test class. This broke the Spark Versions precommit once on #40103. Abacn reviewed #40103 and will likely review this one. He prefers no blocking waits or extra locks in `cancel()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
