HeartSaVioR opened a new pull request, #59389: URL: https://github.com/apache/spark/pull/59389
### What changes were proposed in this pull request? For a new stream-stream join checkpoint, align the configured state format and merge operator with the state store: - Fall back from join state format V3 or V4 to V2 with a non-RocksDB-compatible provider. RocksDB subclasses remain compatible. - When V4 is retained with RocksDB, promote merge operator V1 to V2 before recording the configuration in the offset log. Existing checkpoints keep their pinned settings. If a checkpoint pins an incompatible V4/merge-V1 combination, physical planning fails with a structured error; changing the merge version requires a new checkpoint. ### Why are the changes needed? V3 requires virtual column families. V4 also depends on RocksDB-supported StateStore APIs (including `rangeScanWithMultiValues` for bounded timestamp scans) and requires merge operator V2. A query that explicitly requests V3/V4 with an incompatible provider, or V4 with merge operator V1, cannot run as configured. Spark's default HDFS provider and join state format V2 are unchanged. ### Does this PR introduce _any_ user-facing change? Yes. New stream-stream joins configured with V3/V4 and an incompatible provider use V2. New V4 joins with RocksDB and merge operator V1 use merge operator V2 instead. Existing checkpoints retain their recorded settings; an incompatible pinned V4/merge-V1 combination fails early with a structured error rather than changing stored state. ### How was this patch tested? `build/sbt 'sql/testOnly org.apache.spark.sql.streaming.StreamingInnerJoinWithVCFSuite org.apache.spark.sql.streaming.StreamingInnerJoinV4Suite -- -z "non-RocksDB provider uses compatible format" -z "promotes RocksDB merge operator version 1" -z "state format version 4 rejects an incompatible merge operator"'` (6 passed). `build/sbt sql/scalastyle sql/Test/scalastyle` (0 errors). The tests cover the effective state format and merge operator in the plan and offset log, the unchanged caller session, a restart from the same checkpoint, and fail-fast validation of an incompatible physical plan. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5); GPT-5.2 (Codex CLI). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
