Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
Thank you everybody for their valuable comments. I do agree with Tom that
optimizations need to be done on this issue. We need to move sorting to server
side and try reducing the number of loops
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
After reading through Tom and Marcello's comments I'll defer to them on
whether this should be merged in or not. I will note that moving from a
hard-coded, server-side, scala generated web ui to
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
ok trying this out it is taking a lot longer then I expected. retrieving
the data is relatively quick (2 seconds for 4 tasks) but then is taking a
lot longer to parse and then display.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
seems to be issues with the pr still. Getting errors trying to run on
larger jobs:
project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
disclaimer is I personally haven't tried this out on larger stages so I'll
try to do that tomorrow to really see how the user experience is with larger #
of tasks. If its really that bad then
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19270
> from my understanding SPARK-20657 is proposing to not using the rest api
Not really; I just don't know enough JS to actually make the change to use
the REST api, but I do believe using it
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
Right so from my understanding
https://issues.apache.org/jira/browse/SPARK-20657 is proposing to not using the
rest api and force more on the backing store which exists only for history
server
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82693/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82693 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82693/testReport)**
for PR 19270 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19270
The main change I'm talking about is
https://issues.apache.org/jira/browse/SPARK-20657 (code at
https://github.com/vanzin/spark/pull/41). I did not change the format of the
tables, but how data is
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
@vanzin I haven't followed the prs from SPARK-18085 closely, you keep
mentioning it here but don't give anymore context as to what that has to do
with this change. Did you change the format of
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19270
> sort over all the entries rather than the 20 entries in the current page
Sort is over all entries, isn't it? If it isn't, the work I have on
SPARK-18085 sorts over all entries and is
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
> Does datatables support requesting pages dynamically from the server side
instead of having to load the full data set from the get go?
If that is the case, then my change serves no purpose.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19270
Also, there's a second question of how much memory does it take to generate
the JSON for such a large stage. While working on SPARK-18085 I remember
measuring this and one time requesting the stage
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19270
> the initial page loading time may be slightly high
But how high is high?
Does datatables support requesting pages dynamically from the server side
instead of having to load the
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82693 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82693/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@vanzin Actually the JIRA was reported by somebody else and I copied the
description in the first para. so I just knew what needed to be done but missed
out on the specifics of the description.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19270
I'm confused about the PR description. It mentions "sorting was disabled if
there is any application that has more than one attempt" which as far as I know
has nothing to do with the stage page.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82646/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82646 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82646/testReport)**
for PR 19270 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82646 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82646/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@ajbozarth I do not quite understand what you are saying. Everything seems
to be working fine on my test setup. Can you please let me know how do I
replicate the issue? Thank you.
---
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
So I think I know why the appId was handled the way it was, the live app ui
no longer works because the appId var is "undefined" in all the api calls
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82563/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82563 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82563/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
Yep I did that! Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
Also could you update the description with new (and more) screen shots?
---
-
To unsubscribe, e-mail:
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
Thanks, I'll take a look later today
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@ajbozarth Fixed the issue with the accumulators table.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82563 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82563/testReport)**
for PR 19270 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82522/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82522 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82522/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
Ok Yes, I just noticed that comment. Will get back to you on this one !
---
-
To unsubscribe, e-mail:
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
Thanks, I must've missed that in the description. You've take care of all
but my last comment, I'm not seeing the accumulators table, have you checked it
shows up when theres accumulators?
---
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@ajbozarth Regarding your earlier comment on the missing functionality of
show additional metrics, I have mentioned that in the description of the PR. I
restate it below:
Because of the
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82522 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82522/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
No problem, @ajbozarth , and thank you for your valuable feedback. I really
appreciate it.
---
-
To unsubscribe, e-mail:
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
Thanks, I'll pull the latest changes and keep testing. And thanks for your
quick responses, I understand large changes like this take forever to review
and can get frustrating for the submitter.
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@ajbozarth I have fixed the issue of my changes not working in the web ui
for local, standalone and yarn. Let me know if you are still facing issues with
the testing.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82321/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82321 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82321/testReport)**
for PR 19270 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82321 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82321/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@ajbozarth There were two unit tests in StagePageSuite.scala that were
failing as they are no longer valid for the modified ui that generate
datatables dynamically from Javascript. I have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82252/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82252 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82252/testReport)**
for PR 19270 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82252 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82252/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
Ok, I will look into it. I am currently fixing ui bugs and unit tests, so
will commit those changes first, then will look into the above issue. Thank you.
---
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
Ok so I'm still doing more testing but I've narrowed the above problem. The
above error is occurring when using either local or standalone, the error
doesn't appear when using yarn. I'll continue
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
I just tried this out and it appears to be working for me for a running
application, haven't tried the history UI yet.@ajbozarth What browser are
you using and what are you running (a
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
---
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
For me it's definitely the UI that doesn't work and the SHS that does, I''
see if I can recreate and screenshot the js error I'm getting for you
---
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
I believe you mean the opposite of what you wrote. My changes are visible
in the web ui(while the app is running) and not in the SHS(once the job is
done). Yep I see that and am working on the
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
I'm still going through the code but I also checked out, built and ran you
changes and found that the page doesn't work in the web UI only in the SHS. Did
you test this on both the Web UI and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82003/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82003 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82003/testReport)**
for PR 19270 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #82003 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82003/testReport)**
for PR 19270 at commit
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
No problem. Thank you for your valuable comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
On a second look I think I figured out my misunderstanding, and I've
realized a through review will take quite a bit of time, I'll do my best to
finish by the end of the week but no promises. As
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
The error logs for test build #81683 state that method
this(Long,Int,Int,Long,Long,Long,Long,Long,Long)Unit in class
org.apache.spark.status.api.v1.ExecutorStageSummary does not have a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81936/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #81936 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81936/testReport)**
for PR 19270 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19270
**[Test build #81936 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81936/testReport)**
for PR 19270 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19270
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
All of the code is part of the same task. Can you please be more specific
about the code that you have doubts about, and I can elaborate further on it.
---
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
I'll look at the html/js code tomorrow, but it looks like there still
unrelated code that adds new fields, is that code supposed to be there or is it
for another task?
---
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/19270
Thanks, I'll try to review this by EOD tomorrow
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19270
@ajbozarth Thank you for your comment on the previous PR. I have closed
that one. Apologies for the confusion caused in the previous PR!
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19270
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
76 matches
Mail list logo
