Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
hi @jerryshao good points. First, we should probably move this discussion
to jira so its more visible -- feel free to open two issues for these if you
want, or first discuss on dev@. (Sorry its my
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
@squito , by revising this code, I found there're some places which are
misleading and could be improved:
* All the memory usage referred here about on-heap memory and off-heap
memory
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
I see. The current code leverages `SparkListenerBlockUpdated` event to
calculate memory usage, let me try to investigate the feasibility of using
`taskEnd.taskMetrics.updatedBlocks`, to see if it
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
yeah, we definitely don't want to start logging more events. But it seems
like this info is already available -- taskEnd.taskMetrics.updatedBlocks
already has everything, doesn't it?
---
If your
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Thanks @squito .
Regarding showing memory usage in history server. My major concern is that
putting so many block update event into event log will significantly increase
the file size
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
btw, anybody interested in looking at getting the memory to show up in the
history server as well? this issue we were discussing earlier:
>> AFAIK we don't record block update events in
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
merged to master. Thanks @jerryshao
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/14617
This looks good to me. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
lgtm, thanks for the updates
any more comments @tgravescs @CodingCat @ajbozarth @jsoltren?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75539/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75539 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75539/testReport)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Thanks @squito , thanks so much for your review, just address the comments
you mentioned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75539 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75539/testReport)**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75444/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75444 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75444/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75444 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75444/testReport)**
for PR 14617 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
> AFAIK we don't record block update events in history server, so we could
not calculate the used memory from event log.
good point, sorry I had totally forgotten about. Seems like this
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
@squito , AFAIK we don't record block update events in history server. So
we could not calculate the used memory from event log.
---
If your project is set up for it, you can reply to this email
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
@jerryshao sorry to be very particular about this, but can you add a test
case where the off heap and onheap memory *used* is non-zero? It looks like
you only changed the max memory. (You should
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Hi @squito , would you please review the code again? Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75261/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75261 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75261/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75261 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75261/testReport)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75259/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75259 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75259/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75259 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75259/testReport)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Thanks @squito , I will change some event logs files to test non zero the
off heap memory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75160/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75160 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75160/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75160 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75160/testReport)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Jenkins test is abruptly with signal -9.
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75148/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75148 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75148/testReport)**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75088/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75088 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75088/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75088 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75088/testReport)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75085/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75085 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75085/testReport)**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75023/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75023 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75023/testReport)**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75022/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75022 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75022/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75023 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75023/testReport)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75022 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75022/testReport)**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75021/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #75021 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75021/testReport)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Hi All, thanks a lot for your comments.
Here is the UI after changed:
**
for PR 14617 at commit
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/14617
After reading through the previous comments I agree adding checkboxes to
this page is a good idea, I would even suggest that we look at making
checkboxes for a few of the current columns (default
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
thanks @jerryshao . Two other points:
1. We should have the same treatment in the executor summary table
2. as tom mentioned, there is also the storage page. you can do that
separately
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14617
Checkbox sounds good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/14617
I agree with the checkpoint based solution , thanks for asking
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Thanks @tgravescs and @squito for your comments. Based on @tgravescs 's
point, looks like making them as a table column is more valid.
So I will revert back to use column and combine
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
good points @tgravescs . What about making them additional metrics, turned
on by a checkbox, like the extra task metrics?
---
If your project is set up for it, you can reply to this email and have
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14617
while I kind of like the hover because it doesn't clutter the page, it does
bring up a couple concerns:
- user can't sort by them
- user might not know to hover (none of the other pages
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
Looks like the failures are real (you probably just need to regenerate the
expectations for the new blacklisting tests)
---
If your project is set up for it, you can reply to this email and have
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #74884 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74884/testReport)**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74884/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Change the UI according to @CodingCat 's comment.
**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
That's a good idea, thanks @CodingCat .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/14617
ah...somehow I missed this email.
I like this idea, the comments here would be that `if the last screenshot
is the latest design, will the user have to sum up the total amount of used
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Thanks @squito , finally get someone's attention :). I will update the code
soon.
Before this change off-heap on on-heap memory are added together
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
hi @jerryshao sorry this went unnoticed for so long, if you bring this up
to date I'll keep an eye on it.
Before this change, is off-heap storage completely ignored in the UI? Or
does the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64450/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #64450 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64450/consoleFull)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #64450 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64450/consoleFull)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/14617
@jerryshao The UI changes look great. I have not had a chance to scrutinize
the source changes. Hopefully we can get someone else to help review.
---
If your project is set up for it, you can
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64406/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #64406 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64406/consoleFull)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #64406 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64406/consoleFull)**
for PR 14617 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
@mallman I changed the UI based on your comment, here is the new one
(separate the on heap and off heap memory usage in two columns):
**
for PR 14617 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63657/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63656/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #63656 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63656/consoleFull)**
for PR 14617 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14617
**[Test build #63656 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63656/consoleFull)**
for PR 14617 at commit
99 matches
Mail list logo
