Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
merged to master. Thanks @attilapiros !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21068
Here is the new task for the metrics:
https://issues.apache.org/jira/browse/SPARK-24594.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92040/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #92040 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92040/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #92040 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92040/testReport)**
for PR 21068 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92031/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #92031 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92031/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #92031 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92031/testReport)**
for PR 21068 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
Looks like it was modified to kill if all nodes blacklisted so I'm good
with this approach.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91920/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91920 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91920/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91920 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91920/testReport)**
for PR 21068 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
lgtm
will leave open for a couple of days to let @tgravescs take a look
---
-
To unsubscribe, e-mail:
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91907/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91907 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91907/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91905/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91905 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91905/testReport)**
for PR 21068 at commit
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21068
Retested manually on a cluster with the result the PR's description is
updated.
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91907 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91907/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91905 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91905/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91860/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91860 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91860/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91860 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91860/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91779/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91779 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91779/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91779 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91779/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91764/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91764 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91764/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91764 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91764/testReport)**
for PR 21068 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
Tom and I had a chance to discuss this in person, and after some back and
forth I think we decided that maybe its best to remove the limit but have the
application fail if the entire cluster is
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
hey sorry I have been meaning to respond to this but keep getting
sidetracked. As Tom and I are going to meet in person next week anyway, I
figure at this point it makes sense to just wait till we
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91307/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91307 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91307/testReport)**
for PR 21068 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
well the downside to that and not just failing the application is similar
to what @squito was mentioning, if the cluster is just busy and you can't get
containers on those last few nodes, it
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #91307 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91307/testReport)**
for PR 21068 at commit
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21068
@tgravescs what about removing YARN_BLACKLIST_MAX_NODE_BLACKLIST_RATIO
config and when the set of backlisted nodes reaches numClusterNodes I stop
synchronising the backlisted nodes toward YARN
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
so specifically on the limit, I'm ok with removing it as long as we have
the basic check to fail. I guess perhaps you are saying the limit and that
check are essentially the same thing? I was
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
I mean when `YarnAllocatorBlacklistTracker` decides to blacklist because of
allocation failures, it doesn't send any message back to the driver -- so the
driver doesn't have a msg in the logs, nor
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
What do you mean by adding notification to the driver? Like I mentioned I'm
fine with removing the limit for now but I think we have to do something here
if the entire cluster gets blacklisted,
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
I totally understand your motivation for wanting the limit. But I'm trying
to balance that against behavior which might not really achieve the desired
effect and be even more confusing in some
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
Ah, sorry haven't had time to get to back to this. Yeah the driver
interaction could be an issue. But whether its the limit or just the yarn side
blacklisting I think you would need some
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
ping @tgravescs . honestly I still don't love the blacklist limit,
especially since it makes reporting back to the driver pretty confusing, and I
don't think it buys us much. But I can live with
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90125/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #90125 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90125/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #90125 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90125/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90062/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #90062 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90062/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #90062 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90062/testReport)**
for PR 21068 at commit
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21068
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21068
I assume it is just a flaky R test.
Jenkins retest this please
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90043/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #90043 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90043/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #90043 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90043/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89903/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89903 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89903/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89898/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89898 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89898/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89903 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89903/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89898 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89898/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89889/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89889 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89889/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89889 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89889/testReport)**
for PR 21068 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
A couple more high-level thoughts:
1) Do we want to have a event posted about the node getting blacklisted? I
think it would be useful. But then there needs to be a msg from the
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
ok sounds fine to me, so we should review as is then
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
@tgravescs on the blacklist ratio for task-based blacklisting -- there is
nothing, but there are some related jiras:
[SPARK-22148](https://issues.apache.org/jira/browse/SPARK-22148) &
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89514/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89514 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89514/testReport)**
for PR 21068 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
thanks for filing that jira @squito, I agree we should have blacklisting
work with dynamic allocation disabled as well. (A bit of a tangent from this
jira) I'm actually wondering now about the
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
I think Tom makes a good case for why this should live in the YarnAllocator
as you have it.
I also don't think you need to worry about creating an abstract class yet,
that refactoring can
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89514 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89514/testReport)**
for PR 21068 at commit
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21068
Yes we can create an abstract class from `YarnAllocatorBlacklistTracker`
(like `AbstractAllocatorBlacklistTracker`) where the method
`synchronizeBlacklistedNodes` can have different
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21068
> actually the only other thing I need to make sure is there aren't any
delays if we now send the information from yarn allocator back to scheduler and
then I assume it would need to get it back
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21068
actually the only other thing I need to make sure is there aren't any
delays if we now send the information from yarn allocator back to scheduler and
then I assume it would need to get it back
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89373/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89373 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89373/testReport)**
for PR 21068 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89373 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89373/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89355/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21068
**[Test build #89355 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89355/testReport)**
for PR 21068 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21068
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89350/
Test FAILed.
---
1 - 100 of 115 matches
Mail list logo
