HeartSaVioR edited a comment on pull request #31989: URL: https://github.com/apache/spark/pull/31989#issuecomment-809067233
Except the test suite, one more thing worths to address here is write amplification; we "blindly" replace all start times and all sessions. This could bring unnecessary writes on "unmodified" existing sessions. In many cases we expect the new inputs will be bound and expanding to the existing sessions, but with very long watermark gap and old inputs which have various timestamps, the case could still happen. EDIT: I realized the logic is bound to the physical plan. Though it seems OK to move the logic to here so that the logic to store new session windows efficiently can be bound to the state format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
