GitHub user FavioVazquez reopened a pull request:
https://github.com/apache/spark/pull/5786
[SPARK-7249] Updated Hadoop dependencies due to inconsistency in the
versions
Updated Hadoop dependencies due to inconsistency in the versions. Now the
global properties are the ones used by the hadoop-2.2 profile, and the profile
was set to empty but kept for backwards compatibility reasons.
Changes proposed by @vanzin resulting from previous pull-request
https://github.com/apache/spark/pull/5783 that did not fixed the problem
correctly.
Please let me know if this is the correct way of doing this, the comments
of @vanzin are in the pull-request mentioned.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/FavioVazquez/spark update-hadoop-dependencies
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5786.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5786
----
commit ec91ce3c405123818a4c56ef361d9cc82951677d
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-29T17:58:09Z
- Updated protobuf-java version of com.google.protobuf dependancy to fix
blocking error when connecting to HDFS via the Hadoop Cloudera HDFS CDH5 (fix
for 2.5.0-cdh5.3.3 version)
commit 660decce9d3c2300aee493b605da0da8a74b3ea6
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-29T19:16:04Z
- Updated Hadoop dependencies due to inconsistency in the versions. Now the
global properties are the ones used by the hadoop-2.2 profile, and the profile
was set to empty but kept for backwards compatibility reasons
commit 7e9955df29b5d5c9cda950636d51da753e6d17ea
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-29T19:35:08Z
- Updated Hadoop dependencies due to inconsistency in the versions. Now the
global properties are the ones used by the hadoop-2.2 profile, and the profile
was set to empty but kept for backwards compatibility reasons
commit 6b4bfafbe4f98c92ac2fe7aeb5f36a37d27a9678
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-30T21:41:08Z
- Cleanup in hadoop-2.x profiles since they contained mostly redundant
stuff.
commit 13542929c9cb3ddfec31bbb794e490b44c273df4
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-30T22:13:50Z
- Fixed hadoop-1 version to match jenkins build profile in hadoop1.0 tests
and documentation
commit 287fa2ffc31bb0c9eaf5daf80825ff0093f3f20d
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-30T22:17:44Z
- Updated documentation about specifying the hadoop version in
building-spark. Now is clear that Spark will build against Hadoop 2.2.0 by
default.
- Added Cloudera CDH 5.3.3 without MapReduce example in the building-spark
doc.
commit 70b8344dcad8f6de71bd6356cd6eec375211fdb3
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-04-30T22:57:16Z
- Fixed typo in the make-distribution.sh file and added hadoop-1 in the
Related profiles
commit 88a8b88a13a02cbde04792cb63e3c6a81407d915
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T16:48:27Z
- Simplified Hadoop profiles due to new setting of global properties in the
pom.xml file
- Added comment to specify that the hadoop-2.2 profile is now the default
hadoop profile in the pom.xml file
- Erased hadoop-2.2 from related hadoop profiles now that is a no-op in the
make-distribution.sh file
commit 199f40b1733015a414eb928b2090f3bf4d0b7a7e
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T20:44:30Z
- Erased unnecessary CDH5-specific note in docs/building-spark.md
- Remove example of instance -Phadoop-2.2 -Dhadoop.version=2.2.0 in
docs/building-spark.md
- Enabled hadoop-2.2 profile when the Hadoop version is 2.2.0, which is now
the default .Added comment in the yarn/pom.xml to specify that.
commit a6507792cc12fc03139be825357f22329773c823
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T20:50:46Z
- Default value of avro.mapred.classifier has been set to hadoop2 in pom.xml
- Cleaned up hadoop-2.3 and 2.4 profiles due to change in the default set
in avro.mapred.classifier in pom.xml
commit 0470587ad7af93041e25dcb07954b835d9508a10
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T21:06:52Z
- Erased unnecessary instance of -Phadoop-2.2 -Dhadoop.version=2.2.0 in
create-release.sh
- Updated how the releases are made in the create-release.sh no that the
default hadoop version is the 2.2.0
- Erased unnecessary instance of -Phadoop-2.2 -Dhadoop.version=2.2.0 in
scalastyle
- Erased unnecessary instance of -Phadoop-2.2 -Dhadoop.version=2.2.0 in
run-tests
- Better example given in the hadoop-third-party-distributions.md now that
the default hadoop version is 2.2.0
commit fda6a51986ed4a656d37539502cd4684c46c8cfe
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T21:42:04Z
- Updated hadoop1 releases in create-release.sh due to changes in the
default hadoop version set
- Erased unnecessary instance of -Dyarn.version=2.2.0 in create-release.sh
- Prettify comment in yarn/pom.xml
commit 668d126bc86e5a71f4a724fd36617e7e5a7ff8bc
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T23:19:14Z
- Moved <dependencies> <activation> and <properties> sections of the
hadoop-2.2 profile in the YARN POM to the YARN profile in the root POM
- Erased unnecessary hadoop-2.2 profile from the YARN POM
commit 93f7624224aed23697eb4bad589dcc256d391c19
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-01T23:32:37Z
- Deleted unnecessary comments and <activation> tag on the YARN profile in
the main POM
commit 83dc332aea0cfa95f79cb76d46c6d8e15d6b8377
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-02T00:23:28Z
- Cleaned up the main POM concerning the yarn profile
- Erased hadoop-2.2 profile from yarn/pom.xml and its content was
integrated into yarn/pom.xml
commit cbb93e80dc4a79a5219ec6856634b2ac7e9a91a3
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-02T01:25:58Z
- Added comment related to SPARK-3710 about hadoop-yarn-server-tests in
Hadoop 2.2 that fails to pull some needed dependencies
commit 31bdafad21674fe5bc582fa678753454b04026ad
Author: FavioVazquez <favio.vazqu...@gmail.com>
Date: 2015-05-02T15:27:41Z
- Added missing instances in -Phadoop-1 in create-release.sh, run-tests and
in the building-spark documentation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
