Thanks Deepak! Yes, I want to try it with Docker. But my AWS account ran
out of free period. Is there a shared EC2 for Spark that we can use for
free?
Ping
On Monday, December 9, 2019, Deepak Vohra <dvohr...@yahoo.com> wrote:
> Haven't tested but the general procedure is to exclude all guava
dependencies that are not needed. The hadoop-common depedency does not have
a dependency on guava according to Maven Repository: org.apache.hadoop »
hadoop-common
>
> Maven Repository: org.apache.hadoop » hadoop-common
>
> Apache Spark 2.4 has dependency on guava 14.
> If a Docker image for Cloudera Hadoop is used Spark is may be installed
on Docker for Windows.
> For Docker on Windows on EC2 refer Getting Started with Docker for
Windows - Developer.com
>
> Getting Started with Docker for Windows - Developer.com
>
> Docker for Windows makes it feasible to run a Docker daemon on Windows
Server 2016. Learn to harness its power.
>
>
> Conflicting versions is not an issue if Docker is used.
> "Apache Spark applications usually have a complex set of required
software dependencies. Spark applications may require specific versions of
these dependencies (such as Pyspark and R) on the Spark executor hosts,
sometimes with conflicting versions."
> Running Spark in Docker Containers on YARN
>
> Running Spark in Docker Containers on YARN
>
>
>
>
>
> On Monday, December 9, 2019, 08:37:47 p.m. UTC, Ping Liu <
pingpinga...@gmail.com> wrote:
>
> Hi Deepak,
> I tried it. Unfortunately, it still doesn't work. 28.1-jre isn't
downloaded for somehow. I'll try something else. Thank you very much for
your help!
> Ping
>
> On Fri, Dec 6, 2019 at 5:28 PM Deepak Vohra <dvohr...@yahoo.com> wrote:
>
> As multiple guava versions are found exclude guava from all the
dependecies it could have been downloaded with. And explicitly add a recent
guava version.
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>3.2.1</version>
> <exclusions>
> <exclusion>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>28.1-jre</version>
> </dependency>
> </dependencies>
> </dependencyManagement>
>
> On Friday, December 6, 2019, 10:12:55 p.m. UTC, Ping Liu <
pingpinga...@gmail.com> wrote:
>
> Hi Deepak,
> Following your suggestion, I put exclusion of guava in topmost POM (under
Spark home directly) as follows.
> 2227- </dependency>
> 2228- <dependency>
> 2229- <groupId>org.apache.hadoop</groupId>
> 2230: <artifactId>hadoop-common</artifactId>
> 2231- <version>3.2.1</version>
> 2232- <exclusions>
> 2233- <exclusion>
> 2234- <groupId>com.google.guava</groupId>
> 2235- <artifactId>guava</artifactId>
> 2236- </exclusion>
> 2237- </exclusions>
> 2238- </dependency>
> 2239- </dependencies>
> 2240- </dependencyManagement>
> I also set properties for spark.executor.userClassPathFirst=true and
spark.driver.userClassPathFirst=true
> D:\apache\spark>mvn -Pyarn -Phadoop-3.2 -Dhadoop-version=3.2.1
-Dspark.executor.userClassPathFirst=true
-Dspark.driver.userClassPathFirst=true -DskipTests clean package
> and rebuilt spark.
> But I got the same error when running spark-shell.
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.0.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [
25.092 s]
> [INFO] Spark Project Tags ................................. SUCCESS [
22.093 s]
> [INFO] Spark Project Sketch ............................... SUCCESS [
19.546 s]
> [INFO] Spark Project Local DB ............................. SUCCESS [
10.468 s]
> [INFO] Spark Project Networking ........................... SUCCESS [
17.733 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
6.531 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [
25.327 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [
27.264 s]
> [INFO] Spark Project Core ................................. SUCCESS
[07:59 min]
> [INFO] Spark Project ML Local Library ..................... SUCCESS
[01:39 min]
> [INFO] Spark Project GraphX ............................... SUCCESS
[02:08 min]
> [INFO] Spark Project Streaming ............................ SUCCESS
[02:56 min]
> [INFO] Spark Project Catalyst ............................. SUCCESS
[08:55 min]
> [INFO] Spark Project SQL .................................. SUCCESS
[12:33 min]
> [INFO] Spark Project ML Library ........................... SUCCESS
[08:49 min]
> [INFO] Spark Project Tools ................................ SUCCESS [
16.967 s]
> [INFO] Spark Project Hive ................................. SUCCESS
[06:15 min]
> [INFO] Spark Project Graph API ............................ SUCCESS [
10.219 s]
> [INFO] Spark Project Cypher ............................... SUCCESS [
11.952 s]
> [INFO] Spark Project Graph ................................ SUCCESS [
11.171 s]
> [INFO] Spark Project REPL ................................. SUCCESS [
55.029 s]
> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
[01:07 min]
> [INFO] Spark Project YARN ................................. SUCCESS
[02:22 min]
> [INFO] Spark Project Assembly ............................. SUCCESS [
21.483 s]
> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
56.450 s]
> [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS
[01:21 min]
> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS
[02:33 min]
> [INFO] Spark Project Examples ............................. SUCCESS
[02:05 min]
> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
30.780 s]
> [INFO] Spark Avro ......................................... SUCCESS
[01:43 min]
> [INFO]
------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
------------------------------------------------------------------------
> [INFO] Total time: 01:08 h
> [INFO] Finished at: 2019-12-06T11:43:08-08:00
> [INFO]
------------------------------------------------------------------------
>
> D:\apache\spark>spark-shell
> 'spark-shell' is not recognized as an internal or external command,
> operable program or batch file.
>
> D:\apache\spark>cd bin
>
> D:\apache\spark\bin>spark-shell
> Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
> at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
> at
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
> at
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
> at
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
> at
org.apache.spark.deploy.SparkSubmit$$Lambda$132/1985836631.apply(Unknown
Source)
> at scala.Option.getOrElse(Option.scala:189)
> at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
> at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
> at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
> at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Before building spark, I went to my local Maven repo and removed guava at
all. But after building, I found the same versions of guava have been
downloaded.
> D:\mavenrepo\com\google\guava\guava>ls
> 14.0.1 16.0.1 18.0 19.0
> On Thu, Dec 5, 2019 at 5:12 PM Deepak Vohra <dvohr...@yahoo.com> wrote:
>
> Just to clarify, excluding Hadoop provided guava in pom.xml is an
alternative to using an Uber jar, which is a more involved process.
>
> On Thursday, December 5, 2019, 10:37:39 p.m. UTC, Ping Liu <
pingpinga...@gmail.com> wrote:
>
> Hi Sean,
> Thanks for your response!
> Sorry, I didn't mention that "build/mvn ..." doesn't work. So I did go
to Spark home directory and ran mvn from there. Following is my build and
running result. The source code was just updated yesterday. I guess the
POM should specify newer Guava library somehow.
>
> Thanks Sean.
> Ping
> [INFO] Reactor Summary for Spark Project Parent POM 3.0.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [
14.794 s]
> [INFO] Spark Project Tags ................................. SUCCESS [
18.233 s]
> [INFO] Spark Project Sketch ............................... SUCCESS [
20.077 s]
> [INFO] Spark Project Local DB ............................. SUCCESS [
7.846 s]
> [INFO] Spark Project Networking ........................... SUCCESS [
14.906 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
6.267 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [
31.710 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [
10.227 s]
> [INFO] Spark Project Core ................................. SUCCESS
[08:03 min]
> [INFO] Spark Project ML Local Library ..................... SUCCESS
[01:51 min]
> [INFO] Spark Project GraphX ............................... SUCCESS
[02:20 min]
> [INFO] Spark Project Streaming ............................ SUCCESS
[03:16 min]
> [INFO] Spark Project Catalyst ............................. SUCCESS
[08:45 min]
> [INFO] Spark Project SQL .................................. SUCCESS
[12:12 min]
> [INFO] Spark Project ML Library ........................... SUCCESS [
16:28 h]
> [INFO] Spark Project Tools ................................ SUCCESS [
23.602 s]
> [INFO] Spark Project Hive ................................. SUCCESS
[07:50 min]
> [INFO] Spark Project Graph API ............................ SUCCESS [
8.734 s]
> [INFO] Spark Project Cypher ............................... SUCCESS [
12.420 s]
> [INFO] Spark Project Graph ................................ SUCCESS [
10.186 s]
> [INFO] Spark Project REPL ................................. SUCCESS
[01:03 min]
> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
[01:19 min]
> [INFO] Spark Project YARN ................................. SUCCESS
[02:19 min]
> [INFO] Spark Project Assembly ............................. SUCCESS [
18.912 s]
> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
57.925 s]
> [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS
[01:20 min]
> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS
[02:26 min]
> [INFO] Spark Project Examples ............................. SUCCESS
[02:00 min]
> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
28.354 s]
> [INFO] Spark Avro ......................................... SUCCESS
[01:44 min]
> [INFO]
------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
------------------------------------------------------------------------
> [INFO] Total time: 17:30 h
> [INFO] Finished at: 2019-12-05T12:20:01-08:00
> [INFO]
------------------------------------------------------------------------
>
> D:\apache\spark>cd bin
>
> D:\apache\spark\bin>ls
> beeline load-spark-env.cmd run-example spark-shell
spark-sql2.cmd sparkR.cmd
> beeline.cmd load-spark-env.sh run-example.cmd
spark-shell.cmd spark-submit sparkR2.cmd
> docker-image-tool.sh pyspark spark-class
spark-shell2.cmd spark-submit.cmd
> find-spark-home pyspark.cmd spark-class.cmd spark-sql
spark-submit2.cmd
> find-spark-home.cmd pyspark2.cmd spark-class2.cmd spark-sql.cmd
sparkR
>
> D:\apache\spark\bin>spark-shell
> Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
> at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
> at
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
> at
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
> at
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
> at
org.apache.spark.deploy.SparkSubmit$$Lambda$132/817978763.apply(Unknown
Source)
> at scala.Option.getOrElse(Option.scala:189)
> at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
> at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
> at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
> at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> D:\apache\spark\bin>
> On Thu, Dec 5, 2019 at 1:33 PM Sean Owen <sro...@gmail.com> wrote:
>
> What was the build error? you didn't say. Are you sure it succeeded?
> Try running from the Spark home dir, not bin.
> I know we do run Windows tests and it appears to pass tests, etc.
>
> On Thu, Dec 5, 2019 at 3:28 PM Ping Liu <pingpinga...@gmail.com> wrote:
>>
>> Hello,
>>
>> I understand Spark is preferably built on Linux. But I have a Windows
machine with a slow Virtual Box for Linux. So I wish I am able to build
and run Spark code on Windows environment.
>>
>> Unfortunately,
>>
>> # Apache Hadoop 2.6.X
>> ./build/mvn -Pyarn -DskipTests clean package
>>
>> # Apache Hadoop 2.7.X and later
>> ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean
package
>>
>>
>> Both are listed on
http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn
>>
>> But neither works for me (I stay directly under spark root directory and
run "mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean
package"
>>
>> and
>>
>> Then I tried "mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.2.1 -DskipTests
clean package"
>>
>> Now build works. But when I run spark-shell. I got the following error.
>>
>> D:\apache\spark\bin>spark-shell
>> Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>> at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
>> at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
>> at
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
>> at
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
>> at
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
>> at
org.apache.spark.deploy.SparkSubmit$$Lambda$132/817978763.apply(Unknown
Source)
>> at scala.Option.getOrElse(Option.scala:189)
>> at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
>> at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
>> at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
>> at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>> at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>> at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
>> at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>> Has anyone experienced building and running Spark source code
successfully on Windows? Could you please share your experience?
>>
>> Thanks a lot!
>>
>> Ping
>>
>
