Jungtaek Lim created SPARK-36480: ------------------------------------ Summary: SessionWindowStateStoreSaveExec should not filter input rows against watermark Key: SPARK-36480 URL: https://issues.apache.org/jira/browse/SPARK-36480 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 3.2.0 Reporter: Jungtaek Lim
SessionWindowStateStoreSaveExec receives all sessions including existing sessions into input rows and stores as they are. That said, we should not filter out input rows before storing into state store, but we do. Fortunately it hasn't showed any actual problem due to the nature how we deal with watermark against micro-batch and it seems hard to come up with the broken case, but it should be better to fix it before someone succeeds to touch the possible edge case. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org