cloud-fan opened a new pull request, #57293: URL: https://github.com/apache/spark/pull/57293
### What changes were proposed in this pull request? Follow-up to #56692 (SPARK-56972), which rejected streaming sink evolution combined with async progress tracking because the async path writes only V1 commit metadata and would silently drop the per-sink metadata that sink evolution persists via the V3 commit log. Continuous processing has the same gap. Sink evolution's V3 commit-log write lives entirely in `MicroBatchExecution.markMicroBatchEnd`. `ContinuousExecution` extends `StreamExecution` (not `MicroBatchExecution`), and its `commit` writes a bare `CommitMetadata()` (V1). So enabling `spark.sql.streaming.queryEvolution.enableSinkEvolution` together with `Trigger.Continuous` runs the query while silently never persisting the sink metadata, defeating the feature's durability guarantee with no signal. This PR rejects the combination explicitly at query start, in the `ContinuousTrigger` match arm of `StreamingQueryManager.createQuery`, mirroring the async guard. A new error subclass `STREAMING_QUERY_EVOLUTION_ERROR.CONTINUOUS_PROCESSING_NOT_SUPPORTED` is added. ### Why are the changes needed? Correctness: a user enabling sink evolution together with continuous processing would get no error and no persisted sink metadata. This makes the durability gap fail loudly instead of silently, consistent with the async-path fix. ### Does this PR introduce _any_ user-facing change? Yes, but only for the unreleased sink-evolution feature (off by default, and internal). A streaming query that sets `spark.sql.streaming.queryEvolution.enableSinkEvolution=true` and uses `Trigger.Continuous` now fails at start with a `STREAMING_QUERY_EVOLUTION_ERROR.CONTINUOUS_PROCESSING_NOT_SUPPORTED` error instead of running while silently not persisting the sink metadata. ### How was this patch tested? Added `StreamingSinkEvolutionSuite."fail with continuous processing enabled"`, which asserts the new validation error. Existing `StreamingSinkEvolutionSuite` tests pass (13 tests total). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-4-8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
