[jira] [Comment Edited] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[ https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055633#comment-15055633 ] Michael Han edited comment on SPARK-2356 at 12/14/15 10:03 AM: --- Hello Everyone, I encounter this issue today again when I tried to create a cluster using two windows 7 (64) desktop. This errors happens when I register the second worker to the master using the following command: spark-class org.apache.spark.deploy.worker.Worker spark://masternode:7077 Strange it works fine when I register the first worker to the master. anyone knows some work around to fix this issue? The above work around works fine when I using local mode. Since I registered one worker successfully in the cluster, but when run spark-submit in the successfully worker, it also throw this exception. I google the entire internet and never seen any body has the experience to deploy a windows spark cluster successfully without hadoop, I have a demo in later days so hope anyone can help me on this ;) thank you. Otherwise I have to run vmwares I tried to set the HADOOP_HOME = C:\winutil in the env variables, but it doesn't work. The error is: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/12/14 16:49:22 WARN NativeCodeLoader: Unable to load native-hadoop library fo r your platform... using builtin-java classes where applicable 15/12/14 16:49:22 ERROR Shell: Failed to locate the winutils binary in the hadoo p binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha doop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370) at org.apache.hadoop.util.Shell.(Shell.java:363) at org.apache.hadoop.util.StringUtils.(StringUtils.java:79) at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104) at org.apache.hadoop.security.Groups.(Groups.java:86) at org.apache.hadoop.security.Groups.(Groups.java:66) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group s.java:280) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI nformation.java:271) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use rGroupInformation.java:248) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject( UserGroupInformation.java:763) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou pInformation.java:748) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr oupInformation.java:621) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2091) at org.apache.spark.SecurityManager.(SecurityManager.scala:212) at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker. scala:692) at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:674) at org.apache.spark.deploy.worker.Worker.main(Worker.scala) 15/12/14 16:49:22 INFO SecurityManager: Changing view acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: Changing modify acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: SecurityManager: authentication disabled ; ui acls disabled; users with view permissions: Set(mh6); users with modify per missions: Set(mh6) 15/12/14 16:49:23 INFO Slf4jLogger: Slf4jLogger started 15/12/14 16:49:23 INFO Remoting: Starting remoting 15/12/14 16:49:24 INFO Remoting: Remoting started; listening on addresses :[akka .tcp://sparkWorker@167.3.129.160:46862] 15/12/14 16:49:24 INFO Utils: Successfully started service 'sparkWorker' on port 46862. 15/12/14 16:49:24 INFO Worker: Starting Spark worker 167.3.129.160:46862 with 4 cores, 2.9 GB RAM 15/12/14 16:49:24 INFO Worker: Running Spark version 1.5.2 15/12/14 16:49:24 INFO Worker: Spark home: C:\spark-1.5.2-bin-hadoop2.6\bin\.. 15/12/14 16:49:24 INFO Utils: Successfully started service 'WorkerUI' on port 80 81. 15/12/14 16:49:24 INFO WorkerWebUI: Started WorkerWebUI at http://167.3.129.160: 8081 15/12/14 16:49:24 INFO Worker: Connecting to master 192.168.79.1:7077... 15/12/14 16:49:39 INFO Worker: Retrying connection to master (attempt # 1) 15/12/14 16:49:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thr ead Thread[sparkWorker-akka.actor.default-dispatcher-2,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.Futur eTask@3ef5e68c rejected from java.util.concurrent.ThreadPoolExecutor@741cb720[Ru nning, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] at
[jira] [Comment Edited] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[ https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055633#comment-15055633 ] Michael Han edited comment on SPARK-2356 at 12/14/15 9:09 AM: -- Hello Everyone, I encounter this issue today again when I tried to create a cluster using two windows 7 (64) desktop. This errors happens when I register the second worker to the master using the following command: spark-class org.apache.spark.deploy.worker.Worker spark://masternode:7077 Strange it works fine when I register the first worker to the master. anyone knows some work around to fix this issue? The above work around works fine when I using local mode. I tried to set the HADOOP_HOME = C:\winutil in the env variables, but it doesn't work. The error is: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/12/14 16:49:22 WARN NativeCodeLoader: Unable to load native-hadoop library fo r your platform... using builtin-java classes where applicable 15/12/14 16:49:22 ERROR Shell: Failed to locate the winutils binary in the hadoo p binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha doop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370) at org.apache.hadoop.util.Shell.(Shell.java:363) at org.apache.hadoop.util.StringUtils.(StringUtils.java:79) at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104) at org.apache.hadoop.security.Groups.(Groups.java:86) at org.apache.hadoop.security.Groups.(Groups.java:66) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group s.java:280) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI nformation.java:271) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use rGroupInformation.java:248) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject( UserGroupInformation.java:763) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou pInformation.java:748) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr oupInformation.java:621) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2091) at org.apache.spark.SecurityManager.(SecurityManager.scala:212) at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker. scala:692) at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:674) at org.apache.spark.deploy.worker.Worker.main(Worker.scala) 15/12/14 16:49:22 INFO SecurityManager: Changing view acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: Changing modify acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: SecurityManager: authentication disabled ; ui acls disabled; users with view permissions: Set(mh6); users with modify per missions: Set(mh6) 15/12/14 16:49:23 INFO Slf4jLogger: Slf4jLogger started 15/12/14 16:49:23 INFO Remoting: Starting remoting 15/12/14 16:49:24 INFO Remoting: Remoting started; listening on addresses :[akka .tcp://sparkWorker@167.3.129.160:46862] 15/12/14 16:49:24 INFO Utils: Successfully started service 'sparkWorker' on port 46862. 15/12/14 16:49:24 INFO Worker: Starting Spark worker 167.3.129.160:46862 with 4 cores, 2.9 GB RAM 15/12/14 16:49:24 INFO Worker: Running Spark version 1.5.2 15/12/14 16:49:24 INFO Worker: Spark home: C:\spark-1.5.2-bin-hadoop2.6\bin\.. 15/12/14 16:49:24 INFO Utils: Successfully started service 'WorkerUI' on port 80 81. 15/12/14 16:49:24 INFO WorkerWebUI: Started WorkerWebUI at http://167.3.129.160: 8081 15/12/14 16:49:24 INFO Worker: Connecting to master 192.168.79.1:7077... 15/12/14 16:49:39 INFO Worker: Retrying connection to master (attempt # 1) 15/12/14 16:49:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thr ead Thread[sparkWorker-akka.actor.default-dispatcher-2,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.Futur eTask@3ef5e68c rejected from java.util.concurrent.ThreadPoolExecutor@741cb720[Ru nning, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution (ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.jav a:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.ja va:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorS ervice.java:112) at
[jira] [Comment Edited] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[ https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055633#comment-15055633 ] Michael Han edited comment on SPARK-2356 at 12/14/15 9:05 AM: -- Hello Everyone, I encounter this issue today again when I tried to create a cluster using two windows 7 (64) desktop. This errors happens when I register the second worker to the master using the following command: spark-class org.apache.spark.deploy.worker.Worker spark://masternode:7077 Strange it works fine when I register the first worker to the master. anyone knows some work around to fix this issue? The above work around works fine when I using local mode. The error is: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/12/14 16:49:22 WARN NativeCodeLoader: Unable to load native-hadoop library fo r your platform... using builtin-java classes where applicable 15/12/14 16:49:22 ERROR Shell: Failed to locate the winutils binary in the hadoo p binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha doop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370) at org.apache.hadoop.util.Shell.(Shell.java:363) at org.apache.hadoop.util.StringUtils.(StringUtils.java:79) at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104) at org.apache.hadoop.security.Groups.(Groups.java:86) at org.apache.hadoop.security.Groups.(Groups.java:66) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group s.java:280) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI nformation.java:271) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use rGroupInformation.java:248) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject( UserGroupInformation.java:763) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou pInformation.java:748) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr oupInformation.java:621) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2091) at org.apache.spark.SecurityManager.(SecurityManager.scala:212) at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker. scala:692) at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:674) at org.apache.spark.deploy.worker.Worker.main(Worker.scala) 15/12/14 16:49:22 INFO SecurityManager: Changing view acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: Changing modify acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: SecurityManager: authentication disabled ; ui acls disabled; users with view permissions: Set(mh6); users with modify per missions: Set(mh6) 15/12/14 16:49:23 INFO Slf4jLogger: Slf4jLogger started 15/12/14 16:49:23 INFO Remoting: Starting remoting 15/12/14 16:49:24 INFO Remoting: Remoting started; listening on addresses :[akka .tcp://sparkWorker@167.3.129.160:46862] 15/12/14 16:49:24 INFO Utils: Successfully started service 'sparkWorker' on port 46862. 15/12/14 16:49:24 INFO Worker: Starting Spark worker 167.3.129.160:46862 with 4 cores, 2.9 GB RAM 15/12/14 16:49:24 INFO Worker: Running Spark version 1.5.2 15/12/14 16:49:24 INFO Worker: Spark home: C:\spark-1.5.2-bin-hadoop2.6\bin\.. 15/12/14 16:49:24 INFO Utils: Successfully started service 'WorkerUI' on port 80 81. 15/12/14 16:49:24 INFO WorkerWebUI: Started WorkerWebUI at http://167.3.129.160: 8081 15/12/14 16:49:24 INFO Worker: Connecting to master 192.168.79.1:7077... 15/12/14 16:49:39 INFO Worker: Retrying connection to master (attempt # 1) 15/12/14 16:49:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thr ead Thread[sparkWorker-akka.actor.default-dispatcher-2,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.Futur eTask@3ef5e68c rejected from java.util.concurrent.ThreadPoolExecutor@741cb720[Ru nning, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution (ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.jav a:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.ja va:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorS ervice.java:112) at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deplo
[jira] [Comment Edited] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[ https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055633#comment-15055633 ] Michael Han edited comment on SPARK-2356 at 12/14/15 9:21 AM: -- Hello Everyone, I encounter this issue today again when I tried to create a cluster using two windows 7 (64) desktop. This errors happens when I register the second worker to the master using the following command: spark-class org.apache.spark.deploy.worker.Worker spark://masternode:7077 Strange it works fine when I register the first worker to the master. anyone knows some work around to fix this issue? The above work around works fine when I using local mode. Since I registered one worker successfully in the cluster, but when run spark-submit in the successfully worker, it also throw this exception. I tried to set the HADOOP_HOME = C:\winutil in the env variables, but it doesn't work. The error is: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/12/14 16:49:22 WARN NativeCodeLoader: Unable to load native-hadoop library fo r your platform... using builtin-java classes where applicable 15/12/14 16:49:22 ERROR Shell: Failed to locate the winutils binary in the hadoo p binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha doop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370) at org.apache.hadoop.util.Shell.(Shell.java:363) at org.apache.hadoop.util.StringUtils.(StringUtils.java:79) at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104) at org.apache.hadoop.security.Groups.(Groups.java:86) at org.apache.hadoop.security.Groups.(Groups.java:66) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group s.java:280) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI nformation.java:271) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use rGroupInformation.java:248) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject( UserGroupInformation.java:763) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou pInformation.java:748) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr oupInformation.java:621) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils .scala:2091) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2091) at org.apache.spark.SecurityManager.(SecurityManager.scala:212) at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker. scala:692) at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:674) at org.apache.spark.deploy.worker.Worker.main(Worker.scala) 15/12/14 16:49:22 INFO SecurityManager: Changing view acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: Changing modify acls to: mh6 15/12/14 16:49:22 INFO SecurityManager: SecurityManager: authentication disabled ; ui acls disabled; users with view permissions: Set(mh6); users with modify per missions: Set(mh6) 15/12/14 16:49:23 INFO Slf4jLogger: Slf4jLogger started 15/12/14 16:49:23 INFO Remoting: Starting remoting 15/12/14 16:49:24 INFO Remoting: Remoting started; listening on addresses :[akka .tcp://sparkWorker@167.3.129.160:46862] 15/12/14 16:49:24 INFO Utils: Successfully started service 'sparkWorker' on port 46862. 15/12/14 16:49:24 INFO Worker: Starting Spark worker 167.3.129.160:46862 with 4 cores, 2.9 GB RAM 15/12/14 16:49:24 INFO Worker: Running Spark version 1.5.2 15/12/14 16:49:24 INFO Worker: Spark home: C:\spark-1.5.2-bin-hadoop2.6\bin\.. 15/12/14 16:49:24 INFO Utils: Successfully started service 'WorkerUI' on port 80 81. 15/12/14 16:49:24 INFO WorkerWebUI: Started WorkerWebUI at http://167.3.129.160: 8081 15/12/14 16:49:24 INFO Worker: Connecting to master 192.168.79.1:7077... 15/12/14 16:49:39 INFO Worker: Retrying connection to master (attempt # 1) 15/12/14 16:49:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thr ead Thread[sparkWorker-akka.actor.default-dispatcher-2,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.Futur eTask@3ef5e68c rejected from java.util.concurrent.ThreadPoolExecutor@741cb720[Ru nning, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution (ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.jav a:823) at
[jira] [Comment Edited] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[
https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504723#comment-14504723
]
Lucas Partridge edited comment on SPARK-2356 at 4/21/15 10:23 AM:
--
Neither HADOOP_CONF nor HADOOP_CONF_DIR worked for me. I had to do this instead
(I'm using Spark 1.3.0 on Windows 7):
set HADOOP_HOME=DISK:\FOLDERS
was (Author: asflucas):
Neither HADOOP_CONF nor HADOOP_CONF_DIR worked for me. I had to do this instead:
set HADOOP_HOME=DISK:\FOLDERS
Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
---
Key: SPARK-2356
URL: https://issues.apache.org/jira/browse/SPARK-2356
Project: Spark
Issue Type: Bug
Components: Windows
Affects Versions: 1.0.0
Reporter: Kostiantyn Kudriavtsev
Priority: Critical
I'm trying to run some transformation on Spark, it works fine on cluster
(YARN, linux machines). However, when I'm trying to run it on local machine
(Windows 7) under unit test, I got errors (I don't use Hadoop, I'm read file
from local filesystem):
{code}
14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the
hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.clinit(Shell.java:326)
at org.apache.hadoop.util.StringUtils.clinit(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
at org.apache.hadoop.security.Groups.init(Groups.java:77)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
at
org.apache.spark.deploy.SparkHadoopUtil.init(SparkHadoopUtil.scala:36)
at
org.apache.spark.deploy.SparkHadoopUtil$.init(SparkHadoopUtil.scala:109)
at
org.apache.spark.deploy.SparkHadoopUtil$.clinit(SparkHadoopUtil.scala)
at org.apache.spark.SparkContext.init(SparkContext.scala:228)
at org.apache.spark.SparkContext.init(SparkContext.scala:97)
{code}
It's happened because Hadoop config is initialized each time when spark
context is created regardless is hadoop required or not.
I propose to add some special flag to indicate if hadoop config is required
(or start this configuration manually)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[
https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116536#comment-14116536
]
Remus Rusanu edited comment on SPARK-2356 at 8/30/14 7:50 PM:
--
HADOOP-11003 is requesting hadoop-common to reduce the severity of the error
logged in this case. The error is raised, but getWinUtilsPath() catches it and
logs the stack with error severity. Your code should not see the exception.
was (Author: rusanu):
HADOOP-11003 is requesting hadoop-common to reduce the severity of the error
logged in this case.
Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
---
Key: SPARK-2356
URL: https://issues.apache.org/jira/browse/SPARK-2356
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Kostiantyn Kudriavtsev
Priority: Critical
I'm trying to run some transformation on Spark, it works fine on cluster
(YARN, linux machines). However, when I'm trying to run it on local machine
(Windows 7) under unit test, I got errors (I don't use Hadoop, I'm read file
from local filesystem):
{code}
14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the
hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.clinit(Shell.java:326)
at org.apache.hadoop.util.StringUtils.clinit(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
at org.apache.hadoop.security.Groups.init(Groups.java:77)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
at
org.apache.spark.deploy.SparkHadoopUtil.init(SparkHadoopUtil.scala:36)
at
org.apache.spark.deploy.SparkHadoopUtil$.init(SparkHadoopUtil.scala:109)
at
org.apache.spark.deploy.SparkHadoopUtil$.clinit(SparkHadoopUtil.scala)
at org.apache.spark.SparkContext.init(SparkContext.scala:228)
at org.apache.spark.SparkContext.init(SparkContext.scala:97)
{code}
It's happened because Hadoop config is initialized each time when spark
context is created regardless is hadoop required or not.
I propose to add some special flag to indicate if hadoop config is required
(or start this configuration manually)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
