Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/20704
@megaserg : if you are writing to GCS, Azure, algorithm 2 is fine. If S3 is
the target, then it's only safe to use with a consistent store (Hadoop 3.0
+S3Guard, Amazon Consistent EMR); you
Github user megaserg commented on the issue:
https://github.com/apache/spark/pull/20704
Thank you @dongjoon-hyun! This was also affecting our Spark job performance!
We're using `mapreduce.fileoutputcommitter.algorithm.version=2` in our
Spark job config, as recommended e.g.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/20704
kicks in downstream depending on the order of imports; maven is
closest-first in the graph. If you explicitly add hadoop-client in your deps at
the top then everything gets reconciled
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20704
Yeah, I'm just wondering why that didn't happen in the dependency:tree
output in your description. Anyway, not really important to figure that out.
---
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20704
Thank you for review and merging, @vanzin .
We generated both `spark-deps-hadoop-2.6` and `spark-deps-hadoop-2.7` with
the following.
```
./dev/test-dependencies.sh
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20704
Hmm, I guess it was just luck that this didn't trigger the deps check,
since that jar is checked for a specific version (2.7.3 in the case of
hadoop2.7).
LGTM, merging to master / 2.3.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87854/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20704
**[Test build #87854 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87854/testReport)**
for PR 20704 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20704
Thank you for review, @jerryshao !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20704
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20704
**[Test build #87854 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87854/testReport)**
for PR 20704 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1205/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20704
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20704
The failure is due to flaky test.
```
org.apache.spark.sql.execution.streaming.RateSourceV2Suite.basic
microbatch execution
```
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87848/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20704
**[Test build #87848 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87848/testReport)**
for PR 20704 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1199/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20704
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20704
**[Test build #87848 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87848/testReport)**
for PR 20704 at commit
22 matches
Mail list logo
