Hello and happy holiday to those who are already enjoying it!
I am still having trouble running Hive with Spark. I downloaded Spark 1.5.2 and
built it like this (my Hadoop is version 2.7.1):
./make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided”
When trying to run it with Hive 1.2.1 (a simple command that creates a Spark
job like ‘Select count(*) from userstweetsdailystatistics;') get the following
error
15/12/24 17:12:54 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:54 INFO log.PerfLogger: <PERFLOG method=SparkBuildPlan
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
15/12/24 17:12:54 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:54 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Map 1
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
15/12/24 17:12:54 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:54 INFO Configuration.deprecation: mapred.task.is.map is deprecated.
Instead, use mapreduce.task.ismap
15/12/24 17:12:54 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:54 INFO exec.Utilities: Processing alias userstweetsdailystatistics
15/12/24 17:12:54 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:54 INFO exec.Utilities: Adding input file
hdfs://hadoop-master:8020/user/ubuntu/hive/warehouse/userstweetsdailystatistics
15/12/24 17:12:55 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:55 INFO log.PerfLogger: <PERFLOG method=serializePlan
from=org.apache.hadoop.hive.ql.exec.Utilities>
15/12/24 17:12:55 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:55 INFO exec.Utilities: Serializing MapWork via kryo
15/12/24 17:12:56 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:56 INFO log.PerfLogger: </PERFLOG method=serializePlan
start=1450973575887 end=1450973576279 duration=392
from=org.apache.hadoop.hive.ql.exec.Utilities>
15/12/24 17:12:57 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:57 INFO storage.MemoryStore: ensureFreeSpace(572800) called with
curMem=0, maxMem=556038881
15/12/24 17:12:57 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:57 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 559.4 KB, free 529.7 MB)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO storage.MemoryStore: ensureFreeSpace(43075) called with
curMem=572800, maxMem=556038881
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 42.1 KB, free 529.7 MB)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on
192.168.1.64:49690 (size: 42.1 KB, free: 530.2 MB)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 ERROR util.Utils: uncaught error in thread SparkListenerBus, stopping
SparkContext
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl:
java.lang.AbstractMethodError
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/api,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/static,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs,null}
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO spark.SparkContext: Created broadcast 0 from hadoopRDD at
SparkPlanGenerator.java:188
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.64:4040
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut
down
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Map 1
start=1450973574712 end=1450973578874 duration=4162
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Reducer 2
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO log.PerfLogger: <PERFLOG method=serializePlan
from=org.apache.hadoop.hive.ql.exec.Utilities>
15/12/24 17:12:58 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:58 INFO exec.Utilities: Serializing ReduceWork via kryo
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 INFO log.PerfLogger: </PERFLOG method=serializePlan
start=1450973578926 end=1450973579000 duration=74
from=org.apache.hadoop.hive.ql.exec.Utilities>
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Reducer 2
start=1450973578874 end=1450973579073 duration=199
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 INFO log.PerfLogger: </PERFLOG method=SparkBuildPlan
start=1450973574707 end=1450973579074 duration=4367
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 INFO log.PerfLogger: <PERFLOG method=SparkBuildRDDGraph
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 WARN remote.ReliableDeliverySupervisor: Association with remote system
[akka.tcp://sparkExecutor@192.168.1.64:35089] has failed, address is now gated
for [5000] ms. Reason: [Disassociated]
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 INFO log.PerfLogger: </PERFLOG method=SparkBuildRDDGraph
start=1450973579074 end=1450973579273 duration=199
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/24
17:12:59 INFO client.RemoteDriver: Failed to run job
d3746d11-eac8-4bf9-9897-bef27fd0423e
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl:
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.SparkContext.submitJob(SparkContext.scala:1981)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:118)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:116)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions.foreachAsync(AsyncRDDActions.scala:116)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.api.java.JavaRDDLike$class.foreachAsync(JavaRDDLike.scala:690)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.api.java.AbstractJavaRDDLike.foreachAsync(JavaRDDLike.scala:47)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:257)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
15/12/24 17:12:59 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.lang.Thread.run(Thread.java:745)
15/12/24 17:12:59 [RPC-Handler-3]: INFO client.SparkClientImpl: Received result
for d3746d11-eac8-4bf9-9897-bef27fd0423e
Status: Failed
15/12/24 17:12:59 [Thread-8]: ERROR status.SparkJobMonitor: Status: Failed
15/12/24 17:12:59 [Thread-8]: INFO log.PerfLogger: </PERFLOG method=SparkRunJob
start=1450973569576 end=1450973579584 duration=10008
from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor>
FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
15/12/24 17:13:01 [main]: ERROR ql.Driver: FAILED: Execution Error, return code
3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
15/12/24 17:13:01 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1450973565261 end=1450973581307 duration=16046
from=org.apache.hadoop.hive.ql.Driver>
15/12/24 17:13:01 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/12/24 17:13:01 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1450973581308 end=1450973581308 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/12/24 17:13:01 [main]: INFO exec.ListSinkOperator: 7 finished. closing...
15/12/24 17:13:01 [main]: INFO exec.ListSinkOperator: 7 Close done
15/12/24 17:13:01 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/12/24 17:13:01 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1450973581362 end=1450973581362 duration=0
from=org.apache.hadoop.hive.ql.Driver>
The only useful thing I can find at the Spark side is in the worker log:
15/12/24 17:12:53 INFO worker.Worker: Asked to launch executor
app-20151224171253-0000/0 for Hive on Spark
15/12/24 17:12:53 INFO spark.SecurityManager: Changing view acls to: ubuntu
15/12/24 17:12:53 INFO spark.SecurityManager: Changing modify acls to: ubuntu
15/12/24 17:12:53 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(ubuntu); users
with modify permissions: Set(ubuntu)
15/12/24 17:12:53 INFO worker.ExecutorRunner: Launch command:
"/usr/lib/jvm/java-7-openjdk-amd64/bin/java" "-cp"
"/usr/local/hadoop/etc/hadoop/:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs/:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/lib/spark-assembly-1.5.2-hadoop2.4.0.jar:/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/lib/spark-assembly-1.5.2-hadoop2.4.0.jar:/usr/local/hadoop/etc/hadoop/:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs/:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/lib/spark-assembly-1.5.2-hadoop2.4.0.jar:/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/lib/spark-assembly-1.5.2-hadoop2.4.0.jar:/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/sbin/../conf/:/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/lib/spark-assembly-1.5.2-hadoop2.4.0.jar:/usr/local/hadoop/etc/hadoop/:/usr/local/hadoop/etc/hadoop/:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs/:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar"
"-Xms1024M" "-Xmx1024M" "-Dspark.driver.port=44858"
"-Dhive.spark.log.dir=/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/logs/"
"-XX:MaxPermSize=256m"
"org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url"
"akka.tcp://sparkDriver@192.168.1.64:44858/user/CoarseGrainedScheduler"
"--executor-id" "0" "--hostname" "192.168.1.64" "--cores" "3" "--app-id"
"app-20151224171253-0000" "--worker-url"
"akka.tcp://sparkWorker@192.168.1.64:54209/user/Worker"
15/12/24 17:12:58 INFO worker.Worker: Asked to kill executor
app-20151224171253-0000/0
15/12/24 17:12:58 INFO worker.ExecutorRunner: Runner thread for executor
app-20151224171253-0000/0 interrupted
15/12/24 17:12:58 INFO worker.ExecutorRunner: Killing process!
15/12/24 17:12:58 ERROR logging.FileAppender: Error writing stream to file
/home/ubuntu/Downloads/spark-1.5.2-bin-hadoop2-without-hive/work/app-20151224171253-0000/0/stderr
java.io.IOException: Stream closed
at
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
15/12/24 17:12:59 INFO worker.Worker: Executor app-20151224171253-0000/0
finished with state KILLED exitStatus 143
15/12/24 17:12:59 INFO worker.Worker: Cleaning up local directories for
application app-20151224171253-0000
15/12/24 17:12:59 INFO shuffle.ExternalShuffleBlockResolver: Application
app-20151224171253-0000 removed, cleanupLocalDirs = true
Here is my Spark configuration
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export SPARK_DIST_CLASSPATH=`hadoop class path`
Any hints as to what could be going wrong? Why is the executor getting killed?
Have I built Spark wrongly? I have tried building it in several different ways
and I keep failing.
I must admit I am confused with the information I find online on how to
use/build Spark on Hive and which version goes with what.
Can I download a pre-built version of Spark that would be suitable with my
existing Hadoop 2.7.1 and my Hive 1.2.1?
This error has been baffling me for weeks..
More than grateful for any help!
Sofia
