Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21758
Thanks! Merged to master.
@jiangxb1987 Please submit the other PRs that are blocked by this ASAP.
---
-
To
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21758
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93590/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93590 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93590/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93590 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93590/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21758
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93580/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93580 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93580/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93580 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93580/testReport)**
for PR 21758 at commit
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21758
> What's the failure mode if there are not enough slots for the barrier
mode? We should throw an exception right?
Yes, as mentioned in
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21758
What's the failure mode if there are not enough slots for the barrier mode?
We should throw an exception right?
---
-
To
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21758
LGTM
@squito Has @jiangxb1987 address your comments?
We want to merge this PR ASAP and a few to-be-submitted PRs depend on this
PR. This feature is targeting to Spark 2.4
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93452/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93452 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93452/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93452 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93452/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21758
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93446/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93446 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93446/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93445/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93445 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93445/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93446 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93446/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93445 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93445/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93412/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93412 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93412/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93411/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93411 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93411/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93412 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93412/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93411 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93411/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93388/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93388 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93388/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93388 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93388/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93356 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93356/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93356/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93354/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93354 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93354/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93356 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93356/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93354 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93354/testReport)**
for PR 21758 at commit
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21758
Thanks, I opened https://issues.apache.org/jira/browse/SPARK-24874 to track
the issue.
---
-
To unsubscribe, e-mail:
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21758
I see, thanks for explaining. Maybe it is worth to mark as a TODO.
---
-
To unsubscribe, e-mail:
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21758
For now we force all the tasks to be barrier. We may be able to allow
hybrid of both barrier tasks and regular tasks in a stage, but that shall make
it more complex to do failure handling.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21758
I mean rddC's partitions are derived from rddA and rddB, here assuming
partitions in rddA requires barrier, but not required in rddB. So rddC's
partitions are the half barrier and half not
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21758
@jerryshao In your case we just make sure that `rddC.isBarrier() == true`,
coz there is no ShuffleDependency in `union()` operation, the tasks are
launched based on rddC's partitions.
---
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21758
Hi @jiangxb1987 one question about barrier task.
For example, `rddA` is marked as barrier, and `rddB` is a normal RDD, if
`rddC = rddA.union(rddB)`, seems it contains both normal task and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93287/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93287 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93287/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93287 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93287/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93284/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93284 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93284/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93284 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93284/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93282/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93282 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93282/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93282 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93282/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21758
@mridulm Sorry I missed that message, now I've updated the comment, we can
continue the discussion on that thread.
---
-
To
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/21758
I had left a few comments on SPARK-24375 @jiangxb1987 ... unfortunately the
jira's have moved around a bit.
If this is active PR for introducing the feature, would be great to get
clarity on
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93180/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93180 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93180/testReport)**
for PR 21758 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21758
**[Test build #93180 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93180/testReport)**
for PR 21758 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21758
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
1 - 100 of 118 matches
Mail list logo
