Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21577
Thanks for fixing this, @vanzin!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
Yeah sorry about that, my fault. I merged the fix - SPARK-22897
---
-
To unsubscribe, e-mail:
Github user zzcclp commented on the issue:
https://github.com/apache/spark/pull/21577
@vanzin @tgravescs , after merge this pr into branch-2.2, there is an error
"stageAttemptNumber is not a member of org.apache.spark.TaskContext" in
SparkHadoopMapRedUtil, I think it needs to merge
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
merged to master, 2.3, and 2.2
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21577
> I pushed the change for that in: vanzin/spark@e6a862e
I like it, it's simpler to use task id to replace stage attempt id and task
attempt id. For safety we should do it in master only
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
I will
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
So anyone wants to do the actual merging?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
+1
this is a bit of a side while looking through the scenarios I filed:
https://issues.apache.org/jira/browse/SPARK-24622 . shouldn't be a problem here
though with this fix.
---
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
the code here lgtm, I was trying to make one more pass through all the
scenarios but got stuck in meetings, will try to do it later tonight or
tomorrow morning but we can always have another
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92138/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92138 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92138/testReport)**
for PR 21577 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
I pushed the change for that in:
https://github.com/vanzin/spark/commit/e6a862ecb83c64a0ea2f5bd469bc0febe25e15ba
In case anyone wants to take a look.
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
I filed SPARK-24611 to track some enhancements to this part of the code
that have been discussed here. Of those, I'd consider the "use task IDs instead
of TaskIdentifier" as something we could
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
Sounds good to me (although I'm trying the change locally and unit tests
are so far happy).
---
-
To unsubscribe, e-mail:
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
> Ah, right, d'oh. I just checked about whether stages register with the
coordinator, and saw the duplicate registration for the resubmitted map stage.
Yeah I noticed that to but I think
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/21577
> * t2 finishes before that kill message arrives, is allowed to commit.
> If that can happen it would generate a duplicate map output; but my guess
(hope?) is that the map output tracker would
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
> if its a map stage then I don't expect it to be asking to commit.
Ah, right, d'oh. I just checked about whether stages register with the
coordinator, and saw the duplicate registration for
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
> The test I added can sort of illustrate that if you look at what happens.
There are two stages (map stage 2, result stage 3), and the fetch failure
causes a retry of stage 3 plus a resubmission
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
I was referring to a race caused by asynchronously killing speculative
tasks. Granted it's incredibly unlikely to occur in real life:
- in s1a1 1, t1 and t2 are started for the same
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
> * fixed the issue Mridul brought up, but I think the race that Tom
describes still exists. I'm just not sure it would cause problems, since as far
as I can tell it can only happen in a map
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4252/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92138 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92138/testReport)**
for PR 21577 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/356/
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92109/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92109 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92109/testReport)**
for PR 21577 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92106/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92106 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92106/testReport)**
for PR 21577 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92104/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92104 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92104/testReport)**
for PR 21577 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92109 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92109/testReport)**
for PR 21577 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4233/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/337/
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/21577
Thanks for the changes @vanzin, looks good to me !
Ideally would have been great to test the speculative execution part as
well; but that would be fairly nasty to reliably reproduce I guess.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4230/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92106 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92106/testReport)**
for PR 21577 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/334/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92096/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92096 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92096/testReport)**
for PR 21577 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
> I think the commit/delete thing is also an issue for existing v1 and
hadoop committers as well.
I took a look at that code and I agree with you. It's actually quite
annoying how "task id"
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4226/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/331/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/332/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4225/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92104 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92104/testReport)**
for PR 21577 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
Also, one idea to try to fix the remaining race would be to move the
commit-related state to the `org.apache.spark.scheduler.Stage` class, which is
reused across attempts (even in the resubmission
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/325/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4219/
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
A few notes about the latest updates:
- I reverted the `TaskCommitDenied` changes so that this patch can be
backported more easily. I'm not against the change but I think it'd be better
if
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21577
**[Test build #92096 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92096/testReport)**
for PR 21577 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
I'm fine with separating them but we need a jira or need to update the v2
jira to handle all cases
---
-
To unsubscribe,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21577
This in general looks good, IMO we shall focus on fixing the output commit
coordinator issue in this PR, and discuss the data source issue in a separated
thread.
I'm OOO this week but will
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21577
So I think the commit/delete thing is also an issue for existing v1 and
hadoop committers as well. So this doesn't fully solve the problem. spark
uses a file format like
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21577
FYI I plan to fix the mima issue later. Haven't decided whether to revert
the change or just add excludes... probably the latter since it's a developer
api.
---
71 matches
Mail list logo
