Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/10966#issuecomment-176502936 Thanks a lot @zsxwing for your comments. Yes I agree with you that current approach of replaying event logs suffers a lot for streaming application. The reason I brought out the patch here is to hear more feedbacks about this approach or what is the right way to address it. Below is what I though about: * This problem is not only streaming specific problem, but also affects all the long running applications like thrift server. Unless we change the current replay mechanism for long running service, this problem will always exist. * Beside this scalability issue, I think the modification of Streaming UI and `StreamingJobProgressListener` is still valid to decouple from live streaming context, like what other UI did.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org