I solved this problem following this article http://qnalist.com/questions/4994960/run-spark-unit-test-on-windows-7
1) download compiled winutils.exe from http://social.msdn.microsoft.com/Forums/windowsazure/en-US/28a57efb-082b-424b-8d9e-731b1fe135de/please-read-if-experiencing-job-failures?forum=hdinsight 2) put this file into d:\winutil\bin 3) add in my test: System.setProperty("hadoop.home.dir", "d:\\winutil\\") It solved my original problem. But then I got a new error lang.NullPointerException org.apache.hadoop.util.Shell.runCommand at java.lang.ProcessBuilder.start(Unknown Source) at org.apache.hadoop.util.Shell.runCommand(Shell.java:445) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873) at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:411) at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDep endencies$6.apply(Executor.scala:350) at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDep endencies$6.apply(Executor.scala:347) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scal a:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies (Executor.scala:347) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Very frustrated. Does anybody successfully get spark running on Windows? Regards, Ningjun Wang Consulting Software Engineer LexisNexis 121 Chanlon Road New Providence, NJ 07974-1541 -----Original Message----- From: Marcelo Vanzin [mailto:van...@cloudera.com] Sent: Wednesday, January 28, 2015 5:15 PM To: Wang, Ningjun (LNG-NPV) Cc: user@spark.apache.org Subject: Re: Spark on Windows 2008 R2 serv er does not work https://issues.apache.org/jira/browse/SPARK-2356 Take a look through the comments, there are some workarounds listed there. On Wed, Jan 28, 2015 at 1:40 PM, Wang, Ningjun (LNG-NPV) <ningjun.w...@lexisnexis.com> wrote: > Has anybody successfully install and run spark-1.2.0 on windows 2008 > R2 or windows 7? How do you get that works? > > > > Regards, > > > > Ningjun Wang > > Consulting Software Engineer > > LexisNexis > > 121 Chanlon Road > > New Providence, NJ 07974-1541 > > > > From: Wang, Ningjun (LNG-NPV) [mailto:ningjun.w...@lexisnexis.com] > Sent: Tuesday, January 27, 2015 10:28 PM > To: user@spark.apache.org > Subject: Spark on Windows 2008 R2 serv er does not work > > > > I download and install spark-1.2.0-bin-hadoop2.4.tgz pre-built > version on Windows 2008 R2 server. When I submit a job using > spark-submit, I got the following error > > > > WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load > native-hadoop library for your platform > > ... using builtin-java classes where applicable > > ERROR org.apache.hadoop.util.Shell: Failed to locate the winutils > binary in the hadoop binary path > > java.io.IOException: Could not locate executable null\bin\winutils.exe > in the Hadoop binaries. > > at > org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318) > > at > org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333) > > at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326) > > at > org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) > > at > org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:77) > > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups > .java:240) > > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupIn > formation.java:255) > > > > > > Please advise. Thanks. > > > > > > Ningjun > > > > -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
