Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21366 @foxish @liyinan926 @dvogelbacher ok, I think what we have here is closer to what we want. I primarily addressed the situations outlined by the most recent comments and observations made about the previous design. The PR is getting pretty lengthy, so I wouldn't object to a request to break down the patch into smaller chunks if anyone requests that for ease of review and legibility. In [the latest patch](https://github.com/apache/spark/pull/21366/commits/8615c067328c0c64d0d048922b221477580acdb4) we preserve the idea of observables and having multiple modules poll the state of the cluster, but instead of conceiving of state changes as updates to pods, the modules now always receive a full snapshot of the current state of the cluster. More formally, the watch pushes snapshots that are just the previous state with the state of the updated executor applied. The polling, on the contrary, pushes fresh snapshots. The aim is to get as close to a level-based paradigm as possible. I think this gets us closest, but I've left some fringe unlikely edge cases for future patches.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org