Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 Right so from my understanding https://issues.apache.org/jira/browse/SPARK-20657 is proposing to not using the rest api and force more on the backing store which exists only for history server but not a running application. Running application already have this data in memory anyway. Or are you proposing changing the rest api to use the new backing store and the stage page to the rest api? If not then you are splitting the ui pages to be more specific to running vs history or at least having a different backend store to fetch them from. If its specific to the ui pages then it doesn't help the rest api. If it helps the rest api then why not use that for the web pages too. Either way the data has to get to the web browser, I would think not using the rest api would be memory efficient if it can read directly from existing objects (or the backing store) without having to create a new objects in the rest api to send out. Maybe that doesn't matter so much if we make it more server side to send out small bits at a time. The rest api would be a nice way to abstract the backend out from the UI pages but if it doesn't perform well enough we shouldn't do it. Either one is doable I think we should just choose a direction. I think the server side stuff makes a lot of sense the question is do we want to do it now with this pr, or perhaps a pre-jira to this one, or do we do it later. To me the current pages are frustrating enough I don't mind doing it later but it probably depends on how often you load one of the pages with 100,000+ tasks. I don't think its any slower then loading the history server pages now (which I'm glad you are working on) Let me know your thoughts.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org