Github user rezasafi commented on the issue:
https://github.com/apache/spark/pull/19388
The new PR is here:
https://github.com/apache/spark/pull/19848
---
-
To unsubscribe, e-mail:
Github user rezasafi commented on the issue:
https://github.com/apache/spark/pull/19388
I am closing this PR in favor of another one with a simpler approach that I
will open shortly. Thank you everyone for the comments, here.
---
Github user rezasafi commented on the issue:
https://github.com/apache/spark/pull/19388
@mridulm thank you very much for the comment. I was thinking about
something similar, but more like sending both stageID and a monotonically
increasing jobId to the commit method and then let try
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19388
Internally, spark can continue to use stageId for its commit protocol -
which is independent of what we expose as jobId to mapred/mapreduce committer.
As an implementation detail, we were making
Github user rezasafi commented on the issue:
https://github.com/apache/spark/pull/19388
Thank you very much, @mridulm for the comment. Yeah, I agree that the
provided solution is complex. The complexity came form the fact that in this
solution both executors and the driver use a
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19388
I agree with @vanzin - this looks very complicated for a enforcing a fairly
simple constraint.
It would be easier to depend on a AtomicInteger in driver for the id - and
propagate that to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84033/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #84033 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84033/testReport)**
for PR 19388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #84033 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84033/testReport)**
for PR 19388 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19388
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84006/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #84006 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84006/testReport)**
for PR 19388 at commit
Github user rezasafi commented on the issue:
https://github.com/apache/spark/pull/19388
Thank you very much, @dongjoon-hyun. I forgot to pushed the fix for
streaming failure.
@vanzin My understanding based on reading the code before this change is
that the logic behind it avoids
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #84006 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84006/testReport)**
for PR 19388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84002/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #84002 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84002/testReport)**
for PR 19388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #84002 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84002/testReport)**
for PR 19388 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19388
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83976/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #83976 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83976/testReport)**
for PR 19388 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19388
High level question: do you need to create the commit id based on the rdd +
stage ids, or could it be something else (like a monotonically increasing
value, or a uuid)? That would probably simplify
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #83976 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83976/testReport)**
for PR 19388 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19388
also @mridulm I think you worked on the committer protocol before, both in
spark and MR, so would be great to get your thoughts.
---
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19388
I haven't seen any issues with speculation, most people aren't on 2.2 yet
though either. Unfortunately won't have time to look at this today in detail,
will try to early next week.
---
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19388
@rezasafi and I talked about this for a while offline. Its a lot trickier
than this seems at first. Here's whats going on:
1. `SparkHadoopWriter` is pretending `rddId` is `stageId`, but
Github user rezasafi commented on the issue:
https://github.com/apache/spark/pull/19388
Sorry for the delay. It seems that to be able to commit the same rdd in
different stages we need to use stageId. So the jobId and other configurations
in the write method of SparkHadoopWriter
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82331/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82331 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82331/testReport)**
for PR 19388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82325/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82325 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82325/testReport)**
for PR 19388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82331 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82331/testReport)**
for PR 19388 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19388
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19388
cc @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82325 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82325/testReport)**
for PR 19388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82304 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82304/testReport)**
for PR 19388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82304/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82304 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82304/testReport)**
for PR 19388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82300 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82300/testReport)**
for PR 19388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82300/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19388
**[Test build #82300 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82300/testReport)**
for PR 19388 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19388
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
49 matches
Mail list logo
