Hi All,
Thank you all very much for the information. After make the
correction on the spark.master parameter (spark://hoatname:port) and others
parameters as listed below. Restart the cluster environment everything comes up
ok but error out when doing query.
Some of the spark-defaults.conf parameters
spark.executor.instances 6
spark.driver.memory 4G
spark.yarn.executor.memoryOverhead 400m
spark.executor.memory 1136m
spark.executor.cores 2
beeline execution:
0: jdbc:hive2://localhost:10000> select count(*) from master_request_tab where
date_key='22-Jul-2015';
+----------+--+
| _c0 |
+----------+--+
| 1423538 |
+----------+--+
1 row selected (0.761 seconds)
0: jdbc:hive2://localhost:10000> select ip, count(*) as total from
master_request_tab where date_key='22-Jul-2015' group by ip order by ip asc;
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=1)
0: jdbc:hive2://localhost:10000>
Hive.log
2015-09-29 13:31:46,161 WARN [HiveServer2-Background-Pool: Thread-84]:
client.SparkClientImpl (SparkClientImpl.java:<init>(101)) - Error while waiting
for client to connect.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException:
Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client
connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at
io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
... 1 more
2015-09-29 13:31:46,163 WARN [Driver]: client.SparkClientImpl
(SparkClientImpl.java:run(430)) - Waiting thread interrupted, killing child
process.
2015-09-29 13:31:46,165 ERROR [HiveServer2-Background-Pool: Thread-84]:
spark.SparkTask (SessionState.java:printError(960)) - Failed to execute spark
task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed
to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 22 more
Caused by: java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
... 26 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client
connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at
io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
... 1 more
2015-09-29 13:31:46,165 ERROR [HiveServer2-Background-Pool: Thread-84]:
spark.SparkTask (SparkTask.java:execute(131)) - Failed to execute spark task,
with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to
create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 22 more
Caused by: java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
... 26 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client
connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at
io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
... 1 more
2015-09-29 13:31:46,165 ERROR [HiveServer2-Background-Pool: Thread-84]:
ql.Driver (SessionState.java:printError(960)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
2015-09-29 13:31:46,166 INFO [HiveServer2-Background-Pool: Thread-84]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG
method=Driver.execute start=1443547816118 end=1443547906166 duration=90048
from=org.apache.hadoop.hive.ql.Driver>
2015-09-29 13:31:46,166 INFO [HiveServer2-Background-Pool: Thread-84]:
log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG
method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2015-09-29 13:31:46,166 WARN [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(572)) - Error in redirector thread.
java.io.IOException: Stream closed
at
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at
org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:568)
at java.lang.Thread.run(Thread.java:745)
2015-09-29 13:31:46,188 INFO [HiveServer2-Background-Pool: Thread-84]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG
method=releaseLocks start=1443547906166 end=1443547906188 duration=22
from=org.apache.hadoop.hive.ql.Driver>
2015-09-29 13:31:46,189 ERROR [HiveServer2-Background-Pool: Thread-84]:
operation.Operation (SQLOperation.java:run(209)) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-09-29 13:31:46,191 INFO [HiveServer2-Handler-Pool: Thread-49]:
exec.ListSinkOperator (Operator.java:close(612)) - 23 finished. closing...
2015-09-29 13:31:46,191 INFO [HiveServer2-Handler-Pool: Thread-49]:
exec.ListSinkOperator (Operator.java:close(634)) - 23 Close done
From: Jeetendra G [mailto:[email protected]]
Sent: Tuesday, September 29, 2015 8:11 AM
To: [email protected]
Subject: Re: hive on spark query error
) - Error: Master must start with yarn, spark, mesos, or local please set the
spark.master to either local[*] or yarn-cluster
On Fri, Sep 25, 2015 at 10:45 PM, Xuefu Zhang
<[email protected]<mailto:[email protected]>> wrote:
What's the value of spark.master in your case? The error specifically says
something wrong with it.
--Xuefu
On Fri, Sep 25, 2015 at 9:18 AM, Garry Chen
<[email protected]<mailto:[email protected]>> wrote:
Hi All,
I am following
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started?
To setup hive on spark. After setup/configuration everything startup I am
able to show tables but when executing sql statement within beeline I got
error. Please help and thank you very much.
Cluster Environment (3 nodes) as following
hadoop-2.7.1
spark-1.4.1-bin-hadoop2.6
zookeeper-3.4.6
apache-hive-1.2.1-bin
Error from hive log:
2015-09-25 11:51:03,123 INFO [HiveServer2-Handler-Pool: Thread-50]:
client.SparkClientImpl (SparkClientImpl.java:startDriver(375)) - Attempting
impersonation of oracle
2015-09-25 11:51:03,133 INFO [HiveServer2-Handler-Pool: Thread-50]:
client.SparkClientImpl (SparkClientImpl.java:startDriver(409)) - Running client
driver with argv: /u01/app/spark-1.4.1-bin-hadoop2.6/bin/spark-submit
--proxy-user oracle --properties-file
/tmp/spark-submit.840692098393819749.properties --class
org.apache.hive.spark.client.RemoteDriver
/u01/app/apache-hive-1.2.1-bin/lib/hive-exec-1.2.1.jar --remote-host
ip-10-92-82-229.ec2.internal --remote-port 40476 --conf
hive.spark.client.connect.timeout=1000 --conf
hive.spark.client.server.connect.timeout=90000 --conf
hive.spark.client.channel.log.level=null --conf
hive.spark.client.rpc.max.size=52428800 --conf hive.spark.client.rpc.threads=8
--conf hive.spark.client.secret.bits=256
2015-09-25 11:51:03,867 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Warning: Ignoring non-spark config property:
hive.spark.client.server.connect.timeout=90000
2015-09-25 11:51:03,868 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Warning: Ignoring non-spark config property:
hive.spark.client.rpc.threads=8
2015-09-25 11:51:03,868 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Warning: Ignoring non-spark config property:
hive.spark.client.connect.timeout=1000
2015-09-25 11:51:03,868 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Warning: Ignoring non-spark config property:
hive.spark.client.secret.bits=256
2015-09-25 11:51:03,868 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Warning: Ignoring non-spark config property:
hive.spark.client.rpc.max.size=52428800
2015-09-25 11:51:03,876 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Error: Master must start with yarn, spark,
mesos, or local
2015-09-25 11:51:03,876 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - Run with --help for usage help or --verbose
for debug output
2015-09-25 11:51:03,885 INFO [stderr-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(569)) - 15/09/25 11:51:03 INFO util.Utils: Shutdown
hook called
2015-09-25 11:51:03,889 WARN [Driver]: client.SparkClientImpl
(SparkClientImpl.java:run(427)) - Child process exited with code 1.
