shrirangmhalgi opened a new pull request, #58258:
URL: https://github.com/apache/spark/pull/58258
### What changes were proposed in this pull request?
Make `StateStoreProvider.coordinatorRef` gracefully return `None` when the
`StateStoreCoordinator` RPC endpoint is not available, instead of propagating
`SparkException`. This is done by wrapping the `forExecutor` call in a
try-catch that catches `SparkException` (which wraps
`RpcEndpointNotFoundException` from `awaitResult`), logs a warning, and returns
`None`.
### Why are the changes needed?
PR #50123 added `reportSnapshotUploadToCoordinator()` to
`HDFSBackedStateStoreProvider.loadMap()`, creating an unconditional dependency
on the `StateStoreCoordinator` from the state store read path. The coordinator
endpoint is only registered when `StreamingQueryManager` is instantiated —
which is intentionally a `lazy val` in `SessionState` (since SPARK-29423). In a
fresh session that only does batch reads of state store data (e.g.,
`spark.read.format("statestore").load(path)`), the endpoint doesn't exist and
the read fails with `[CANNOT_LOAD_STATE_STORE.UNCATEGORIZED]`.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.read.format("statestore").load(checkpointDir)` now works in
sessions that have never started a streaming query, without requiring the
workaround of accessing `spark.streams` first or setting
`spark.sql.streaming.stateStore.coordinatorReportSnapshotUploadLag=false`.
### How was this patch tested?
Added `StateStoreCoordinatorBatchReadSuite` with two tests:
1. Unit test verifying `StateStoreProvider.coordinatorRef` returns `None`
after coordinator shutdown
2. Integration test verifying batch state store reads succeed without the
coordinator
### Was this patch authored or co-authored using generative AI tooling?
Yes. Co-Authored using Claude Opus 4.8
Closes #58211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
