I figured it out.
I tried to build Spark configured to access OpenStack Swift and
hadoop-openstack.jar has the same issue as were described here
https://github.com/apache/spark/pull/7090/commits
So for those who wants to build Spark 1.5.- master with OpenStack Swift
support, just remove mockito-all dependency from the core/pom.xml:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-openstack</artifactId>
<exclusions>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
</exclusion>
</exclusions>
</dependency>
I guess this is needed for Hadoop version 2.6.0, but perhaps latest Hadoop
versions has the same mockito versions as Spark uses.
Gil Vernik.
From: Gil Vernik/Haifa/IBM@IBMIL
To: Dev <[email protected]>
Date: 14/07/2015 12:23
Subject: problems with build of latest the master
I just did checkout of the master and tried to build it with
mvn -Dhadoop.version=2.6.0 -DskipTests clean package
Got:
[ERROR]
/Users/gilv/Dev/Spark/spark/core/src/test/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java:117:
error: cannot find symbol
[ERROR]
when(shuffleMemoryManager.tryToAcquire(anyLong())).then(returnsFirstArg());
[ERROR] ^
[ERROR] symbol: method then(Answer<Object>)
[ERROR] location: interface OngoingStubbing<Long>
[ERROR]
/Users/gilv/Dev/Spark/spark/core/src/test/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java:408:
error: cannot find symbol
[ERROR] .then(returnsFirstArg()) // Allocate initial sort buffer
[ERROR] ^
[ERROR] symbol: method then(Answer<Object>)
[ERROR] location: interface OngoingStubbing<Long>
[ERROR]
/Users/gilv/Dev/Spark/spark/core/src/test/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java:435:
error: cannot find symbol
[ERROR] .then(returnsFirstArg()) // Allocate initial sort buffer
[ERROR] ^
[ERROR] symbol: method then(Answer<Object>)
[ERROR] location: interface OngoingStubbing<Long>
[ERROR]
/Users/gilv/Dev/Spark/spark/core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java:98:
error: cannot find symbol
[ERROR]
when(shuffleMemoryManager.tryToAcquire(anyLong())).then(returnsFirstArg());
[ERROR] ^
[ERROR] symbol: method then(Answer<Object>)
[ERROR] location: interface OngoingStubbing<Long>
[ERROR]
/Users/gilv/Dev/Spark/spark/core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java:130:
error: cannot find symbol
[ERROR] .then(returnsSecondArg());
[ERROR] ^
[ERROR] symbol: method then(Answer<Object>)
[ERROR] location: interface OngoingStubbing<InputStream>
[ERROR] Note:
/Users/gilv/Dev/Spark/spark/core/src/test/java/org/apache/spark/JavaAPISuite.java
uses or overrides a deprecated API.
[ERROR] Note: Recompile with -Xlint:deprecation for details.
[ERROR] 5 errors
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [
3.183 s]
[INFO] Spark Project Launcher ............................. SUCCESS [
7.681 s]
[INFO] Spark Project Networking ........................... SUCCESS [
7.178 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
4.125 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [
3.734 s]
[INFO] Spark Project Core ................................. FAILURE [02:16
min]
[INFO] Spark Project Bagel ................................ SKIPPED
