[
https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159384#comment-14159384
]
cocoatomo commented on SPARK-3794:
----------------------------------
Thank you for the comment.
Building from the root directory results in a same error.
{noformat}
$ mvn -Dhadoop.version=1.1.0 -DskipTests clean compile
...
[ERROR]
/Users/<user>/MyRepos/Scala/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:720:
value listFilesAndDirs is not a member of object
org.apache.commons.io.FileUtils
[ERROR] val files = FileUtils.listFilesAndDirs(dir, TrueFileFilter.TRUE,
TrueFileFilter.TRUE)
[ERROR] ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.147 s]
[INFO] Spark Project Core ................................. FAILURE [ 42.550 s]
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.365 s
[INFO] Finished at: 2014-10-05T10:29:48+09:00
[INFO] Final Memory: 34M/1017M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on
project spark-core_2.10: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed. CompileFailed ->
[Help 1]
{noformat}
> Building spark core fails with specific hadoop version
> ------------------------------------------------------
>
> Key: SPARK-3794
> URL: https://issues.apache.org/jira/browse/SPARK-3794
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Environment: Mac OS X 10.9.5
> Reporter: cocoatomo
> Labels: spark
>
> At the commit cf1d32e3e1071829b152d4b597bf0a0d7a5629a2, building spark core
> result in compilation error when we specify some hadoop versions.
> To reproduce this issue, we should execute following command with
> <hadoop.version>=1.1.0, 1.1.1, 1.1.2, 1.2.0, 1.2.1, or 2.2.0.
> {noformat}
> $ cd ./core
> $ mvn -Dhadoop.version=<hadoop.version> -DskipTests clean compile
> ...
> [ERROR]
> /Users/tomohiko/MyRepos/Scala/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:720:
> value listFilesAndDirs is not a member of object
> org.apache.commons.io.FileUtils
> [ERROR] val files = FileUtils.listFilesAndDirs(dir,
> TrueFileFilter.TRUE, TrueFileFilter.TRUE)
> [ERROR] ^
> {noformat}
> Because that compilation uses commons-io version 2.1 and
> FileUtils#listFilesAndDirs method was added at commons-io version 2.2, this
> compilation always fails.
> FileUtils#listFilesAndDirs →
> http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/FileUtils.html#listFilesAndDirs%28java.io.File,%20org.apache.commons.io.filefilter.IOFileFilter,%20org.apache.commons.io.filefilter.IOFileFilter%29
> Because a hadoop-client in those problematic version depends on commons-io
> 2.1 not 2.4, we should have assumption that commons-io is version 2.1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
