HeartSaVioR opened a new pull request, #59629: URL: https://github.com/apache/spark/pull/59629
### What changes were proposed in this pull request? Fix V4 timestamp-index eviction so it includes newly admitted rows below the previous state cutoff. Use that cutoff as the scan lower bound only when it does not exceed the late-input watermark; otherwise scan from the beginning of the index through the current eviction cutoff. ### Why are the changes needed? For a left outer stream-stream join with zero-second watermarks on both event-time columns: ```sql leftKey = rightKey AND leftTime > rightTime + INTERVAL 5 SECONDS ``` The previous derived state cutoff can be ahead of the watermark used to reject late input. A new row can therefore be accepted yet already be eligible for eviction. Starting the V4 scan above that cutoff skips the row and loses its outer-null output. The regression disables no-data micro-batches and uses these input batches (times in seconds): 1. Left `(9, 20)`, right `(9, 10)`: emits their match. 2. Right `(2, 20)`: advances the watermark. 3. Left `(1, 12)`, right `(2, 30)`: must emit `(1, null, 12, null)`. In batch 3, the late-input watermark is 10 seconds and the previous state cutoff is 15 seconds. The accepted 12-second row falls below the old scan lower bound. ### Does this PR introduce _any_ user-facing change? Yes. V4 joins finalize these eligible rows instead of skipping them. V1-V3 do not use the timestamp-index scan lower bound. State encoding and the eviction predicate are unchanged. For affected join conditions, scanning from the beginning can read more state; the current eviction cutoff still bounds the scan. No performance benchmark was run. ### How was this patch tested? - Four E2E regression cases passed, covering continuous execution and an ordinary restart, each with RocksDB changelog checkpointing enabled and disabled. A subsequent batch checks that the outer-null result is not duplicated. - SQL Compile/Test scalastyle passed with zero errors and warnings; ASCII/100-column checks passed. - `git diff --check` passed. ```text build/sbt 'sql/testOnly *StreamingOuterJoinV4Suite -- -z "newly admitted rows"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
