HeartSaVioR edited a comment on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-525986491 > At least for now, KVStore is always empty when AppStatusListener startup, right ? > I mean, what do we expect to get from the KVStore since we've already know it's empty ? In my perspective the assumption shouldn't be there, or at least documented. We do neither. (Let's at least document it if we don't want to address it soon.) We're planning to break the assumption soon so I'm not sure it gives a value to continue relying on current assumption. You may be able to just rely on current for now, and I have to just modify it afterwards. Just a matter of who will deal with that. If you're thinking about having flag to provide hint to AppStatusListener whether KVStore is empty or not, please no more flag, unless we figure out initialization with empty KVStore is considerably slow. (couple of "seconds" for example.) Let's not doing premature optimization. > Do we only need snapshot for completed apps in SHS since snapshot for in-completed apps could be generated by live AppStatusListener ? Assuming you're referring single huge application event log, you may be right if there's no failure at all, but support any failure in SHS or even just a restart due to maintenance, without snapshotting in-completed apps we'll lose live AppStatusListener and end up with replaying all again. I guess that's not what we expect from incremental reload, right?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
