shrirangmhalgi commented on issue #58211: URL: https://github.com/apache/spark/issues/58211#issuecomment-5399577223
Thank you @Kimahriman for reporting the issue. I was able to reproduce the issue and trace the root cause. The `StateStoreCoordinator` RPC endpoint is only registered when `StreamingQueryManager` is instantiated - this was intentionally a `lazy val` in `SessionState` (since SPARK-29423). PR #50123 added `reportSnapshotUploadToCoordinator()` to `HDFSBackedStateStoreProvider.loadMap()`, creating an unconditional dependency on the coordinator from the state store read path. The existing test suite masks this because `StateDataSourceTestBase.beforeEach()` explicitly calls `spark.streams.stateStoreCoordinator` before every test. I will implement the fix and raise a PR for the same. I am thinking of making `StateStoreProvider.coordinatorRef` gracefully return `None` when the coordinator endpoint hasn't been registered. The snapshot upload reporting is best-effort and shouldn't fail batch reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
