Re: Is it feasible to build and run Spark on Windows?

pingpinga...@gmail.com> wrote: > > > Super. Thanks Deepak! > > On Mon, Dec 9, 2019 at 6:58 PM Deepak Vohra wrote: > > Please install Apache Spark on Windows as discussed in Apache Spark on > Windows - DZone Open Source > <https://dzone.com/articles/working-on-apac

Re: Is it feasible to build and run Spark on Windows?

: Please install Apache Spark on Windows as discussed in Apache Spark on Windows - DZone Open Source | | | | | | | | | | | Apache Spark on Windows - DZone Open Source This article explains and provides solutions for some of the most common errors developers come across when inst

Re: Is it feasible to build and run Spark on Windows?

Super. Thanks Deepak! On Mon, Dec 9, 2019 at 6:58 PM Deepak Vohra wrote: > Please install Apache Spark on Windows as discussed in Apache Spark on > Windows - DZone Open Source > <https://dzone.com/articles/working-on-apache-spark-on-windows> > > Apache Spark on Window

Re: Is it feasible to build and run Spark on Windows?

Please install Apache Spark on Windows as discussed in Apache Spark on Windows - DZone Open Source | | | | | | | | | | | Apache Spark on Windows - DZone Open Source This article explains and provides solutions for some of the most common errors developers come across when inst

Re: Is it feasible to build and run Spark on Windows?

Thanks Deepak! Yes, I want to try it with Docker. But my AWS account ran out of free period. Is there a shared EC2 for Spark that we can use for free? Ping On Monday, December 9, 2019, Deepak Vohra wrote: > Haven't tested but the general procedure is to exclude all guava dependencies that ar

Re: Is it feasible to build and run Spark on Windows?

Haven't tested but the general procedure is to exclude all guava dependencies that are not needed. The hadoop-common depedency does not have a dependency on guava according to Maven Repository: org.apache.hadoop » hadoop-common | | | | | | | | | | | Maven Repository: org.apache.hado

Re: Is it feasible to build and run Spark on Windows?

Hi Deepak, I tried it. Unfortunately, it still doesn't work. 28.1-jre isn't downloaded for somehow. I'll try something else. Thank you very much for your help! Ping On Fri, Dec 6, 2019 at 5:28 PM Deepak Vohra wrote: > As multiple guava versions are found exclude guava from all the > depe

Re: Is it feasible to build and run Spark on Windows?

 As multiple guava versions are found exclude guava from all the dependecies it could have been downloaded with. And explicitly add a recent guava version.         org.apache.hadoop        hadoop-common         3.2.1                                com.google.guava             guava            

Re: Is it feasible to build and run Spark on Windows?

Hi Deepak, Following your suggestion, I put exclusion of guava in topmost POM (under Spark home directly) as follows. 2227- 2228- 2229-org.apache.hadoop 2230:hadoop-common 2231-3.2.1 2232- 2233- 2234-com.google.guava 2235-

Re: Is it feasible to build and run Spark on Windows?

Just to clarify, excluding Hadoop provided guava in pom.xml is an alternative to using an Uber jar, which is a more involved process. On Thursday, December 5, 2019, 10:37:39 p.m. UTC, Ping Liu wrote: Hi Sean, Thanks for your response! Sorry, I didn't mention that "build/mvn ..." doesn

Re: Is it feasible to build and run Spark on Windows?

ker to do major work if possible. Thanks! Ping On Thu, Dec 5, 2019 at 2:24 PM Deepak Vohra wrote: Several alternatives are available: - Use Maven to build Spark on Windows.  http://spark.apache.org/docs/latest/building-spark.html#apache-maven - Use Docker image for  CDH on Wi

Re: Is it feasible to build and run Spark on Windows?

my Windows doesn't have Microsoft Hyper-V.  So I want to avoid using Docker to do major work if possible. Thanks! Ping On Thu, Dec 5, 2019 at 2:24 PM Deepak Vohra wrote: Several alternatives are available: - Use Maven to build Spark on Windows.  http://spark.apache.org/docs/latest/building-sp

Re: Is it feasible to build and run Spark on Windows?

gt; pingpinga...@gmail.com> wrote: > > > Hi Deepak, > > Yes, I did use Maven. I even have the build pass successfully when setting > Hadoop version to 3.2. Please see my response to Sean's email. > > Unfortunately, I only have Docker Toolbox as my Windows doesn'

Re: Is it feasible to build and run Spark on Windows?

epak Vohra wrote: Several alternatives are available: - Use Maven to build Spark on Windows.  http://spark.apache.org/docs/latest/building-spark.html#apache-maven - Use Docker image for  CDH on WindowsDocker Hub | | | | Docker Hub | | | On Thursday, December 5, 2019, 09:33:4

Re: Is it feasible to build and run Spark on Windows?

Windows doesn't have > Microsoft Hyper-V. So I want to avoid using Docker to do major work if > possible. > > Thanks! > > Ping > > > On Thu, Dec 5, 2019 at 2:24 PM Deepak Vohra wrote: > > Several alternatives are available: > > - Use Maven to build Spark o

Re: Is it feasible to build and run Spark on Windows?

Multiple Guava versions could be in the classpath inherited from Hadoop. Use the Guava version supported by Spark, and exclude other Guava. Also add spark.executor.userClassPathFirst=true and spark.driver.userClassPathFirst=true in properties. On Thursday, December 5, 2019, 11:35:27 PM UT

Re: Is it feasible to build and run Spark on Windows?

Hi Sean, Oh, sorry. I just came back to Spark home. However, the same error came out. D:\apache\spark\bin>cd .. D:\apache\spark>bin\spark-shell Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

Re: Is it feasible to build and run Spark on Windows?

ndows doesn't have Microsoft Hyper-V.  So I want to avoid using Docker to do major work if possible. Thanks! Ping On Thu, Dec 5, 2019 at 2:24 PM Deepak Vohra wrote: Several alternatives are available: - Use Maven to build Spark on Windows.  http://spark.apache.org/docs/latest/building-spark.h

Re: Is it feasible to build and run Spark on Windows?

or work if possible. Thanks! Ping On Thu, Dec 5, 2019 at 2:24 PM Deepak Vohra wrote: > Several alternatives are available: > > - Use Maven to build Spark on Windows. > http://spark.apache.org/docs/latest/building-spark.html#apache-maven > > - Use Docker image for CDH on

Re: Is it feasible to build and run Spark on Windows?

No, the build works fine, at least certainly on test machines. As I say, try running from the actual Spark home, not bin/. You are still running spark-shell there. On Thu, Dec 5, 2019 at 4:37 PM Ping Liu wrote: > > Hi Sean, > > Thanks for your response! > > Sorry, I didn't mention that "build/mvn

Re: Is it feasible to build and run Spark on Windows?

Hi Sean, Thanks for your response! Sorry, I didn't mention that "build/mvn ..." doesn't work. So I did go to Spark home directory and ran mvn from there. Following is my build and running result. The source code was just updated yesterday. I guess the POM should specify newer Guava library so

Re: Is it feasible to build and run Spark on Windows?

Several alternatives are available: - Use Maven to build Spark on Windows.  http://spark.apache.org/docs/latest/building-spark.html#apache-maven - Use Docker image for  CDH on WindowsDocker Hub | | | | Docker Hub | | | On Thursday, December 5, 2019, 09:33:43 p.m. UTC, Sean

Re: Is it feasible to build and run Spark on Windows?

What was the build error? you didn't say. Are you sure it succeeded? Try running from the Spark home dir, not bin. I know we do run Windows tests and it appears to pass tests, etc. On Thu, Dec 5, 2019 at 3:28 PM Ping Liu wrote: > > Hello, > > I understand Spark is preferably built on Linux. But

Is it feasible to build and run Spark on Windows?

Hello, I understand Spark is preferably built on Linux. But I have a Windows machine with a slow Virtual Box for Linux. So I wish I am able to build and run Spark code on Windows environment. Unfortunately, # Apache Hadoop 2.6.X ./build/mvn -Pyarn -DskipTests clean package # Apache Hadoop 2.7

Apache spark on windows without shortnames enabled

Hi, We use Apache Spark 2.2.0 in our stack. Our software by default like other softwares gets installed under "C:\Program Files\". We have a restriction that we cannot ask our customers to enable short names on their machines. From our experience, spark does not handle the absolute paths well

Re: Spark (on Windows) not picking up HADOOP_CONF_DIR

Hi, How did you set it? How do you run the app? Use sys.env to know whether it was set or not. Jacek On 17 Jul 2016 11:33 a.m., "Daniel Haviv" wrote: > Hi, > I'm running Spark using IntelliJ on Windows and even though I set > HADOOP_CONF_DIR it does not affect the contents of sc.hadoopConfigur

Spark (on Windows) not picking up HADOOP_CONF_DIR

Hi, I'm running Spark using IntelliJ on Windows and even though I set HADOOP_CONF_DIR it does not affect the contents of sc.hadoopConfiguration. Anybody encountered it ? Thanks, Daniel

Re: Spark on Windows platform

58 pm, "gaurav pathak" wrote: > Can someone guide me the steps and information regarding, installation of > SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will be > great to read your experiences in using SPARK on Windows platform. > > > Thanks & Regards, > Gaurav Pathak >

Re: Spark on Windows platform

On 29 Feb 2016, at 13:40, gaurav pathak mailto:gauravpathak...@gmail.com>> wrote: Thanks Jorn. Any guidance on how to get started with getting SPARK on Windows, is highly appreciated. Thanks & Regards Gaurav Pathak you are at risk of seeing stack traces when you try to talk to

Re: Spark on Windows platform

> Hi > I am running spark on windows but a standalone one. > > Use this code > > SparkConf conf = new SparkConf().setMaster("local[1]").seatAppName("spark").setSparkHome("c:/spark/bin/spark-submit.cmd"); > > Where sparkhome is the path wher

Re: Spark on Windows platform

Thanks Jorn. Any guidance on how to get started with getting SPARK on Windows, is highly appreciated. Thanks & Regards Gaurav Pathak ~ sent from handheld device On Feb 29, 2016 5:34 AM, "Jörn Franke" wrote: > I think Hortonworks has a Windows Spark distribution. May

Re: Spark on Windows platform

I think Hortonworks has a Windows Spark distribution. Maybe Bigtop as well? > On 29 Feb 2016, at 14:27, gaurav pathak wrote: > > Can someone guide me the steps and information regarding, installation of > SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will be

Spark on Windows platform

Can someone guide me the steps and information regarding, installation of SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will be great to read your experiences in using SPARK on Windows platform. Thanks & Regards, Gaurav Pathak

Re: Spark on Windows

You can check "spark.master" property in conf/spark-defaults.conf and try to give IP of the VM in place of "localhost". On Tue, Feb 16, 2016 at 7:48 AM, KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi, > > I am new to spark and starting working on it by writing small programs. I > am

Spark on Windows

Hi, I am new to spark and starting working on it by writing small programs. I am able to run those in cloudera quickstart VM but not able to run in the eclipse when giving master URL *Steps I perfromed:* Started Master and can access it through http://localhost:8080 Started worker and access it

RE: Error building Spark on Windows with sbt

I have not had any success building using sbt/sbt on windows. However, I have been able to binary by using maven command directly. From: Richard Eggert [mailto:richard.egg...@gmail.com] Sent: Sunday, October 25, 2015 12:51 PM To: Ted Yu Cc: User Subject: Re: Error building Spark on Windows with

Re: Error building Spark on Windows with sbt

Yes, I know, but it would be nice to be able to test things myself before I push commits. On Sun, Oct 25, 2015 at 3:50 PM, Ted Yu wrote: > If you have a pull request, Jenkins can test your change for you. > > FYI > > On Oct 25, 2015, at 12:43 PM, Richard Eggert > wrote: > > Also, if I run the M

Re: Error building Spark on Windows with sbt

If you have a pull request, Jenkins can test your change for you. FYI > On Oct 25, 2015, at 12:43 PM, Richard Eggert wrote: > > Also, if I run the Maven build on Windows or Linux without setting > -DskipTests=true, it hangs indefinitely when it gets to > org.apache.spark.JavaAPISuite. > >

Re: Error building Spark on Windows with sbt

Also, if I run the Maven build on Windows or Linux without setting -DskipTests=true, it hangs indefinitely when it gets to org.apache.spark.JavaAPISuite. It's hard to test patches when the build doesn't work. :-/ On Sun, Oct 25, 2015 at 3:41 PM, Richard Eggert wrote: > By "it works", I mean, "I

Re: Error building Spark on Windows with sbt

By "it works", I mean, "It gets past that particular error". It still fails several minutes later with a different error: java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.scala-lang#scala-library;2.10.3 On Sun, Oct 25, 2015 at 3:38 PM, Ric

Error building Spark on Windows with sbt

When I try to start up sbt for the Spark build, or if I try to import it in IntelliJ IDEA as an sbt project, it fails with a "No such file or directory" error when it attempts to "git clone" sbt-pom-reader into .sbt/0.13/staging/some-sha1-hash. If I manually create the expected directory before r

Re: Download Apache Spark on Windows 7 for a Proof of Concept installation

message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Download-Apache-Spark-on-Windows-7-for-a-Proof-of-Concept-installation-tp23992p23998.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Download Apache Spark on Windows 7 for a Proof of Concept installation

ver to download for a Proof of Concept installation of Apache Spark on > Windows 7. I have spent quite some time following a number of different > recipes to no avail. I have tried about 10 different permutations to date. > > I prefer the easiest approach, e.g. download Pre-build

Download Apache Spark on Windows 7 for a Proof of Concept installation

I just wanted an easy step by step guide as to exactly what version of what ever to download for a Proof of Concept installation of Apache Spark on Windows 7. I have spent quite some time following a number of different recipes to no avail. I have tried about 10 different permutations to date. I

RE: spark on Windows 2008 failed to save RDD to windows shared folder

: Friday, May 22, 2015 5:02 PM To: Wang, Ningjun (LNG-NPV) Cc: user@spark.apache.org Subject: Re: spark on Windows 2008 failed to save RDD to windows shared folder The stack trace is related to hdfs. Can you tell us which hadoop release you are using ? Is this a secure cluster ? Thanks On Fri

Re: spark on Windows 2008 failed to save RDD to windows shared folder

The stack trace is related to hdfs. Can you tell us which hadoop release you are using ? Is this a secure cluster ? Thanks On Fri, May 22, 2015 at 1:55 PM, Wang, Ningjun (LNG-NPV) < ningjun.w...@lexisnexis.com> wrote: > I used spark standalone cluster on Windows 2008. I kept on getting the >

spark on Windows 2008 failed to save RDD to windows shared folder

I used spark standalone cluster on Windows 2008. I kept on getting the following error when trying to save an RDD to a windows shared folder rdd.saveAsObjectFile("file:///T:/lab4-win02/IndexRoot01/tobacco-07/myrdd.obj") 15/05/22 16:49:05 ERROR Executor: Exception in task 0.0 in stage 12.0 (TID 1

Fwd: Change ivy cache for spark on Windows

+user -- Forwarded message -- From: Burak Yavuz Date: Mon, Apr 27, 2015 at 1:59 PM Subject: Re: Change ivy cache for spark on Windows To: mj Hi, In your conf file (SPARK_HOME\conf\spark-defaults.conf) you can set: `spark.jars.ivy \your\path` Best, Burak On Mon, Apr 27

Change ivy cache for spark on Windows

, but spark doesn't seem to pick it up. Does anyone have any ideas of how to get around this issue? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Change-ivy-cache-for-spark-on-Windows-tp22675.html Sent from the Apache Spark User

Re: Spark on Windows

Thanks, Sree! Are you able to run your applications using spark-submit? Even after we were able to build successfully, we ran into problems with running the spark-submit script. If everything worked correctly for you, we can hope that things will be smoother when 1.4.0 is made generally available.

Re: Spark on Windows

spark 'master' branch (i.e. v1.4.0) builds successfully on windows 8.1 intel i7 64-bit with oracle jdk8_45.with maven opts without the flag "-XX:ReservedCodeCacheSize=1g". takes about 33 minutes. Thanking you. With Regards Sree  On Thursday, April 16, 2015 9:07 PM, Arun Lists wrote:

Re: Spark on Windows

The hadoop "support" from HortonWorks only *actually *works with Windows Server - well at least as of Spark Summit last year : and AFAIK that has not changed since 2015-04-16 15:18 GMT-07:00 Dean Wampler : > If you're running Hadoop, too, now that Hortonworks supports Spark, you > might be able

Re: Spark on Windows

If you're running Hadoop, too, now that Hortonworks supports Spark, you might be able to use their distribution. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition (O'Reilly) Typesafe @deanwampler

Re: Spark on Windows

Thanks, Matei! We'll try that and let you know if it works. You are correct in inferring that some of the problems we had were with dependencies. We also had problems with the spark-submit scripts. I will get the details from the engineer who worked on the Windows builds and provide them to you.

Re: Spark on Windows

You could build Spark with Scala 2.11 on Mac / Linux and transfer it over to Windows. AFAIK it should build on Windows too, the only problem is that Maven might take a long time to download dependencies. What errors are you seeing? Matei > On Apr 16, 2015, at 9:23 AM, Arun Lists wrote: > > We

Spark on Windows

We run Spark on Mac and Linux but also need to run it on Windows 8.1 and Windows Server. We ran into problems with the Scala 2.10 binary bundle for Spark 1.3.0 but managed to get it working. However, on Mac/Linux, we are on Scala 2.11.6 (we built Spark from the sources). On Windows, however despit

Error when running Spark on Windows 8.1

Hi, We are trying to run a Spark application using spark-submit on Windows 8.1. The application runs successfully to completion on MacOS 10.10 and on Ubuntu Linux. On Windows, we get the following error messages (see below). It appears that Spark is trying to delete some temporary directory that i

Re: Building Spark on Windows WAS: Any IRC channel on Spark?

Dear Yu, Are you mean "scalastyle-output.xml"? i coped its content below On Tue, Mar 17, 2015 at 4:11 PM, Ted Yu wrote: > Can you look in build output for scalastyle warning in mllib module ? > > Cheers > > > > On Mar 17, 2015, at 3:00 AM, Ahmed Nawar wrote: > > Dear Yu, > >

Re: Building Spark on Windows WAS: Any IRC channel on Spark?

Can you look in build output for scalastyle warning in mllib module ? Cheers > On Mar 17, 2015, at 3:00 AM, Ahmed Nawar wrote: > > Dear Yu, > >With -X i got below error. > > > [INFO] > > [INFO] Reactor Summary: >

Re: Building Spark on Windows WAS: Any IRC channel on Spark?

Dear Yu, With -X i got below error. [INFO] [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ... SUCCESS [ 7.418 s] [INFO] Spark Project Networking ... SUCCES

Building Spark on Windows WAS: Any IRC channel on Spark?

Have you tried with -X switch ? Thanks > On Mar 17, 2015, at 1:47 AM, Ahmed Nawar wrote: > > Dears, > > Is there any instructions to build spark 1.3.0 on windows 7. > > I tried "mvn -Phive -Phive-thriftserver -DskipTests clean package" but i > got below errors > > > [INFO] Spark

Re: Stand-alone Spark on windows

Hi Can you share the complete stack trace Thanks Arush On Thu, Feb 26, 2015 at 6:09 PM, Sergey Gerasimov wrote: > Hi! > > I downloaded Spark binaries unpacked and could successfully run pyspark > shell and write and execute some code here > > BUT > > I failed with submitting stand-alone python

Stand-alone Spark on windows

Hi! I downloaded Spark binaries unpacked and could successfully run pyspark shell and write and execute some code here BUT I failed with submitting stand-alone python scripts or jar files via spark-submit: spark-submit pi.py I always get exception stack trace with NullPointerException in jav

RE: Spark on Windows 2008 R2 serv er does not work

(LNG-NPV) [mailto:ningjun.w...@lexisnexis.com] > Sent: Tuesday, January 27, 2015 10:28 PM > To: user@spark.apache.org > Subject: Spark on Windows 2008 R2 serv er does not work > > > > I download and install spark-1.2.0-bin-hadoop2.4.tgz pre-built > versio

Re: Spark on Windows 2008 R2 serv er does not work

Sent: Tuesday, January 27, 2015 10:28 PM > To: user@spark.apache.org > Subject: Spark on Windows 2008 R2 serv er does not work > > > > I download and install spark-1.2.0-bin-hadoop2.4.tgz pre-built version on > Windows 2008 R2 server. When I submit a job using spark-submit, I got

RE: Spark on Windows 2008 R2 serv er does not work

: Tuesday, January 27, 2015 10:28 PM To: user@spark.apache.org Subject: Spark on Windows 2008 R2 serv er does not work I download and install spark-1.2.0-bin-hadoop2.4.tgz pre-built version on Windows 2008 R2 server. When I submit a job using spark-submit, I got the following error WARN

Spark on Windows 2008 R2 serv er does not work

I download and install spark-1.2.0-bin-hadoop2.4.tgz pre-built version on Windows 2008 R2 server. When I submit a job using spark-submit, I got the following error WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform ... using builtin-java classe