HeartSaVioR commented on issue #23634: [SPARK-26154][SS] Streaming left/right outer join should not return outer nulls for already matched rows URL: https://github.com/apache/spark/pull/23634#issuecomment-458915512 I just realized current approach is not compatible with old state, and it is non-trivial to address the issue. I would end up with applying either one to support backward compatibility: 1) Store state data with empty map for all of previous batches on `matched` state store. It still requires leaving old logic as fail-back mechanism to handle outer join correctly with old state. (8e1a2f4 actually does this, but I forgot I should fill up state checkpoints for matched state store.) 2) Versioning state (old state = version 1 / new state = version 2), and separate logic against state version. This means we give up fixing bug with state version 1. As we have been addressing state compatibility issue via 2), I would try to tackle 2) first.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
