[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084265#comment-16084265 ] Marcelo Vanzin commented on SPARK-18085: Sure, if it's just a matter of adding the label to the

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083612#comment-16083612 ] Reynold Xin commented on SPARK-18085: - That sounds good to me. I don't actually think the SPIP

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083607#comment-16083607 ] Sean Owen commented on SPARK-18085: --- Just call the existing doc a SPIP? > Better History Server

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083453#comment-16083453 ] Reynold Xin commented on SPARK-18085: - This is just large enough to warrant / deserve a SPIP to be

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078517#comment-16078517 ] Marcelo Vanzin commented on SPARK-18085: [~rxin] this has been open with a specification document

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077606#comment-16077606 ] Reynold Xin commented on SPARK-18085: - [~vanzin] seems like this should have a SPIP? Looks super

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041498#comment-16041498 ] Marcelo Vanzin commented on SPARK-18085: BTW for the last issue I had already filed SPARK-20656

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041153#comment-16041153 ] Marcelo Vanzin commented on SPARK-18085: bq. the procedure of loading the history summary page is

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040633#comment-16040633 ] DjvuLee commented on SPARK-18085: - [~vanzin] the procedure of loading the history summary page is still a

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039223#comment-16039223 ] Marcelo Vanzin commented on SPARK-18085: Yes, https://github.com/vanzin/spark/commits/shs-ng/HEAD

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16038210#comment-16038210 ] DjvuLee commented on SPARK-18085: - [~vanzin] I want to try your branch. Does all the information is the

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001192#comment-16001192 ] Marcelo Vanzin commented on SPARK-18085: Alright, I have the code in a state that I like (seems

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971594#comment-15971594 ] Marcelo Vanzin commented on SPARK-18085: I'm getting close to a point where I think the code can

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939191#comment-15939191 ] Marcelo Vanzin commented on SPARK-18085: I'm also keeping the current activity at

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937377#comment-15937377 ] Marcelo Vanzin commented on SPARK-18085: Hi all, an update. I started working on this again and I

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897730#comment-15897730 ] Marcelo Vanzin commented on SPARK-18085: bq. "A separate jar file" means we generate a new jar

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896652#comment-15896652 ] DjvuLee commented on SPARK-18085: - "A separate jar file" means we generate a new jar file for the history

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896619#comment-15896619 ] Marcelo Vanzin commented on SPARK-18085: Not sure what you mean by "a separate jar file". There's

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896070#comment-15896070 ] DjvuLee commented on SPARK-18085: - [~vanzin] Thanks for your reply! Does your new solution will generate

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894768#comment-15894768 ] Marcelo Vanzin commented on SPARK-18085: bq. does this local db will delete the data as

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894074#comment-15894074 ] DjvuLee commented on SPARK-18085: - [~vanzin] This is a nice design. There is not much information about

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850205#comment-15850205 ] Marcelo Vanzin commented on SPARK-18085: They're only being changed to use the new data store.

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849965#comment-15849965 ] Thomas Graves commented on SPARK-18085: --- Are the ui updates just updates to use new api or are you

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849151#comment-15849151 ] Marcelo Vanzin commented on SPARK-18085: Jobs tab code at:

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828980#comment-15828980 ] Marcelo Vanzin commented on SPARK-18085: I uploaded branch shs-ng/M4.1 to my repo which contains

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828866#comment-15828866 ] Alex Bozarth commented on SPARK-18085: -- I'm interested in helping with M4, but I am currently bogged

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828860#comment-15828860 ] Marcelo Vanzin commented on SPARK-18085: BTW if anyone is still following this, I updated the

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749660#comment-15749660 ] Marcelo Vanzin commented on SPARK-18085: Yes. The REST API returns JSON. That JSON is not read

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749652#comment-15749652 ] Dmitry Buzolin commented on SPARK-18085: I don't think you read my response with enough attention

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749610#comment-15749610 ] Marcelo Vanzin commented on SPARK-18085: bq. while I pressed a link on SHS application page That

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749592#comment-15749592 ] Dmitry Buzolin commented on SPARK-18085: I meant to say the discussion was becoming not

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727475#comment-15727475 ] Marcelo Vanzin commented on SPARK-18085: I'm not trying to flame you. I'm trying to point out

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727446#comment-15727446 ] Dmitry Buzolin commented on SPARK-18085: I posted my comments not to start the endless flame on

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726053#comment-15726053 ] Marcelo Vanzin commented on SPARK-18085: bq. Since SHS keeps this data in memory I don't see how

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725880#comment-15725880 ] Dmitry Buzolin commented on SPARK-18085: Spark log size is directly depending on few things: -

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723547#comment-15723547 ] Marcelo Vanzin commented on SPARK-18085: While the issues you raise are valid, they're also

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722425#comment-15722425 ] Dmitry Buzolin commented on SPARK-18085: I would like add my observations after working with SHS:

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709943#comment-15709943 ] Marcelo Vanzin commented on SPARK-18085: I uploaded code for milestone 3 from the document:

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667820#comment-15667820 ] Yin Huai commented on SPARK-18085: -- [~vanzin] Sorry for jumping in late. I'd like to look at the doc

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667722#comment-15667722 ] Marcelo Vanzin commented on SPARK-18085: {{/v2}} is both a way to try out my changes without

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667226#comment-15667226 ] Thomas Graves commented on SPARK-18085: --- I think at a high level its fine. You mention using a /v2

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665618#comment-15665618 ] Marcelo Vanzin commented on SPARK-18085: Hey all, any more feedback here? I've made some