[
https://issues.apache.org/jira/browse/SPARK-37994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481507#comment-17481507
]
Vu Tan commented on SPARK-37994:
--------------------------------
[~xkrogen] Thank you for your comment !
[~csun]
Thank you for the instruction.
I have tested on my PC (twice) and it all showed the below error (a new one)
{code:java}
[INFO] --- scala-maven-plugin:4.3.0:testCompile (scala-test-compile-first) @
spark-yarn_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file:
/Users/JP28431/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.15__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin:
BasicArtifact(com.github.ghik,silencer-plugin_2.12.15,1.7.6,null)
[INFO] Compiling 22 Scala sources and 3 Java sources to
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/target/scala-2.12/test-classes
...
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala:30:
object MiniYARNCluster is not a member of package org.apache.hadoop.yarn.server
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala:61:
not found: type MiniYARNCluster
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala:104:
not found: type MiniYARNCluster
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:37:
object resourcemanager is not a member of package org.apache.hadoop.yarn.server
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:38:
object resourcemanager is not a member of package org.apache.hadoop.yarn.server
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:39:
object resourcemanager is not a member of package org.apache.hadoop.yarn.server
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:40:
object resourcemanager is not a member of package org.apache.hadoop.yarn.server
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:250:
not found: type RMContext
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:252:
not found: type RMApp
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:261:
not found: type RMApplicationHistoryWriter
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:263:
not found: type SystemMetricsPublisher
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:267:
not found: type RMAppManager
[ERROR] [Error]
/Users/JP28431/Downloads/spark-3.2.0-github/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala:272:
not found: type ClientRMService
[ERROR] 13 errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.2.0:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 19.531 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 13.606 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 9.357 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 6.603 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 10.494 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.382 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 12.624 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 11.122 s]
[INFO] Spark Project Core ................................. SUCCESS [04:11 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 45.036 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 38.573 s]
[INFO] Spark Project Streaming ............................ SUCCESS [01:16 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [04:12 min]
[INFO] Spark Project SQL .................................. SUCCESS [05:41 min]
[INFO] Spark Project ML Library ........................... SUCCESS [03:53 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 12.873 s]
[INFO] Spark Project Hive ................................. SUCCESS [02:46 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 36.992 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 24.638 s]
[INFO] Spark Project YARN ................................. FAILURE [ 53.060 s]
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27:01 min
[INFO] Finished at: 2022-01-25T11:33:02+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile
(scala-test-compile-first) on project spark-yarn_2.12: Execution
scala-test-compile-first of goal
net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile failed.: CompileFailed
-> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :spark-yarn_2.12 {code}
Do you have any ideas about this ?
> Unable to build spark3.2 with -Dhadoop.version=3.1.4
> ----------------------------------------------------
>
> Key: SPARK-37994
> URL: https://issues.apache.org/jira/browse/SPARK-37994
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 3.2.0
> Reporter: Vu Tan
> Priority: Minor
>
> I downloaded Spark 3.2 sourcecode from
> [https://github.com/apache/spark/archive/refs/tags/v3.2.0.zip]
> and try building with the below command
> {code:java}
> ./dev/make-distribution.sh --name without-hadoop --pip --r --tgz -Psparkr
> -Phive -Phive-thriftserver -Phadoop-provided -Pyarn -Dhadoop.version=3.1.4
> -Pkubernetes {code}
> Then it gives the below error
> {code:java}
> [INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @
> spark-core_2.12 ---
> [INFO] Using incremental compilation using Mixed compile order
> [INFO] Compiler bridge file:
> /Users/JP28431/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.15__52.0-1.3.1_20191012T045515.jar
> [INFO] compiler plugin:
> BasicArtifact(com.github.ghik,silencer-plugin_2.12.15,1.7.6,null)
> [INFO] Compiling 567 Scala sources and 104 Java sources to
> /Users/JP28431/Downloads/spark-3.2.0-github/core/target/scala-2.12/classes ...
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:38:
> object io is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:2778:
> not found: type ArrayWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:2777:
> not found: type Writable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SSLOptions.scala:24:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SSLOptions.scala:174:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:25:
> object io is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:26:
> object security is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:33:
> object fs is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:32:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:121:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:284:
> not found: value UserGroupInformation
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:41:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:40:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:39:
> object mapred is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:37:
> object fs is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:36:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:348:
> not found: type Credentials
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:350:
> not found: value UserGroupInformation
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:402:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:22:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:23:
> object io is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:24:
> object io is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:30:
> not found: type Writable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:38:
> not found: type ObjectWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:43:
> not found: type ObjectWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:20:
> object fs is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:21:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:210:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:306:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:26:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:55:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1188:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1161:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1120:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925:
> not found: type TextInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925:
> not found: type LongWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:968:
> not found: value NewHadoopJob
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:971:
> not found: value NewFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:31:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:21:
> object io is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:39:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:39:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:25:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:34:
> not found: type CombineFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala:21:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala:32:
> not found: type HConfigurable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:35:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:36:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:20:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:37:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:976:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:977:
> not found: type Text
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1020:
> not found: value NewHadoopJob
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1023:
> not found: value NewFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:22:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/PortableDataStream.scala:28:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/PortableDataStream.scala:39:
> not found: type CombineFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:20:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:34:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1050:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1237:
> not found: type NewInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:23:
> object mapreduce is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:43:
> not found: type FileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1210:
> not found: type NewInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1053:
> not found: type LongWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1053:
> not found: type BytesWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1055:
> not found: type LongWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1056:
> not found: type BytesWritable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1087:
> not found: type JobConf
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1088:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1096:
> class FileSystem in package io cannot be accessed in package java.io
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:34:
> object mapred is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:131:
> not found: type JobConf
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:29:
> object mapred is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:112:
> not found: type JobConf
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:113:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:99:
> not found: type JobConf
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:100:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1128:
> not found: value FileSystem
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/util/SerializableConfiguration.scala:21:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/util/SerializableConfiguration.scala:31:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1132:
> not found: type JobConf
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1132:
> not found: value FileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1242:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1247:
> not found: value FileSystem
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1251:
> not found: value NewHadoopJob
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1254:
> not found: value NewFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:73:
> not found: type InputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:27:
> object conf is not a member of package org.apache.hadoop
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:76:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1278:
> not found: type Configuration
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1277:
> not found: type NewInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1286:
> not found: value FileSystem
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1289:
> not found: type JobConf
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1315:
> not found: type SequenceFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1375:
> not found: type SequenceFileInputFormat
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:3045:
> not found: type Writable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1377:
> not found: type Writable
> [ERROR] [Error]
> /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1378:
> not found: type Writable
> [INFO] [Info] : Any <: String?
> [INFO] [Info] : false
> [ERROR] 100 errors found
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary for Spark Project Parent POM 3.2.0:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [ 4.964
> s]
> [INFO] Spark Project Tags ................................. SUCCESS [ 8.381
> s]
> [INFO] Spark Project Sketch ............................... SUCCESS [ 7.871
> s]
> [INFO] Spark Project Local DB ............................. SUCCESS [ 2.191
> s]
> [INFO] Spark Project Networking ........................... SUCCESS [ 4.879
> s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 2.505
> s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [ 8.310
> s]
> [INFO] Spark Project Launcher ............................. SUCCESS [ 1.990
> s]
> [INFO] Spark Project Core ................................. FAILURE [ 45.343
> s]
> [INFO] Spark Project ML Local Library ..................... SKIPPED
> [INFO] Spark Project GraphX ............................... SKIPPED
> [INFO] Spark Project Streaming ............................ SKIPPED
> [INFO] Spark Project Catalyst ............................. SKIPPED
> [INFO] Spark Project SQL .................................. SKIPPED
> [INFO] Spark Project ML Library ........................... SKIPPED
> [INFO] Spark Project Tools ................................ SKIPPED
> [INFO] Spark Project Hive ................................. SKIPPED
> [INFO] Spark Project REPL ................................. SKIPPED
> [INFO] Spark Project YARN Shuffle Service ................. SKIPPED
> [INFO] Spark Project YARN ................................. SKIPPED
> [INFO] Spark Project Kubernetes ........................... SKIPPED
> [INFO] Spark Project Hive Thrift Server ................... SKIPPED
> [INFO] Spark Project Assembly ............................. SKIPPED
> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
> [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
> [INFO] Spark Project Examples ............................. SKIPPED
> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
> [INFO] Spark Avro ......................................... SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 01:27 min
> [INFO] Finished at: 2022-01-24T11:07:30+09:00
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on
> project spark-core_2.12: Execution scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed.: CompileFailed ->
> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn <args> -rf :spark-core_2.12 {code}
> is this a expected behavior or should be considered a bug ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
