hi Xuefu i am running spark on local mode and my spark master log is
15/03/20 10:52:52 INFO AbstractConnector: Started SelectChannelConnector@nn01:6066 15/03/20 10:52:52 INFO Utils: Successfully started service on port 6066. 15/03/20 10:52:52 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066 15/03/20 10:52:52 INFO Master: Starting Spark master at spark://nn01:7077 15/03/20 10:52:52 INFO Master: Running Spark version 1.3.0 15/03/20 10:52:52 INFO Server: jetty-8.y.z-SNAPSHOT 15/03/20 10:52:52 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080 15/03/20 10:52:52 INFO Utils: Successfully started service 'MasterUI' on port 8080. 15/03/20 10:52:52 INFO MasterWebUI: Started MasterWebUI at http://nn01:8080 15/03/20 10:52:53 INFO Master: I have been elected leader! New state: ALIVE And my Hive log 2015-03-20 10:53:12,871 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2015-03-20 10:53:12,871 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 2015-03-20 10:53:12,871 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2015-03-20 10:53:12,872 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at java.lang.reflect.Method.invoke(Method.java:606) 2015-03-20 10:53:12,872 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) 2015-03-20 10:53:12,872 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) 2015-03-20 10:53:12,872 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) 2015-03-20 10:53:12,873 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) 2015-03-20 10:53:12,873 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(530)) - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2015-03-20 10:53:13,412 WARN [Driver]: client.SparkClientImpl (SparkClientImpl.java:run(388)) - Child process exited with code 1. 2015-03-20 10:53:28,202 WARN [Thread-42]: client.SparkClientImpl (SparkClientImpl.java:<init>(96)) - Error while waiting for client to connect. java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94) at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88) at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:134) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:744) 2015-03-20 10:53:28,207 ERROR [Thread-42]: exec.Task (SessionState.java:printError(861)) - Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:104) at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88) at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) ... 6 more Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94) ... 10 more Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:134) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:744) 2015-03-20 10:53:28,207 ERROR [Thread-42]: exec.Task (SparkTask.java:execute(124)) - Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:104) at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88) at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) ... 6 more Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94) ... 10 more Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:134) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:744) Thanks & Regards Amithsha On Mon, Mar 16, 2015 at 9:32 PM, Jimmy Xiang <jxi...@cloudera.com> wrote: > One more thing, "java.lang.NoSuchFieldError: > SPARK_RPC_CLIENT_CONNECT_TIMEOUT", are your jar files consistent? > > On Mon, Mar 16, 2015 at 6:47 AM, Xuefu Zhang <xzh...@cloudera.com> wrote: >> >> It seems that your remote driver failed to start. I suggest #1: try >> spark.master=local first; #2: check spark.log to find out why the remote >> driver fails. >> >> --Xuefu >> >> On Sun, Mar 15, 2015 at 10:17 PM, Amith sha <amithsh...@gmail.com> wrote: >>> >>> Hi, >>> >>> I have already added the spark-assembly jar in hive lib & here is my hive >>> log >>> >>> >>> 2015-03-16 10:40:08,299 INFO [main]: SessionState >>> (SessionState.java:printInfo(852)) - Added >>> >>> [/opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar] >>> to class path >>> 2015-03-16 10:40:08,300 INFO [main]: SessionState >>> (SessionState.java:printInfo(852)) - Added resources: >>> >>> [/opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar] >>> 2015-03-16 10:40:36,914 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=Driver.run >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:36,915 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=TimeToSubmit >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:36,915 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=compile >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:36,916 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=parse >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:36,916 INFO [main]: parse.ParseDriver >>> (ParseDriver.java:parse(185)) - Parsing command: insert into table >>> test values(5,8900) >>> 2015-03-16 10:40:36,917 INFO [main]: parse.ParseDriver >>> (ParseDriver.java:parse(206)) - Parse Completed >>> 2015-03-16 10:40:36,925 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=parse >>> start=1426482636916 end=1426482636925 duration=9 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:36,929 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=semanticAnalyze >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:37,034 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:analyzeInternal(10146)) - Starting Semantic >>> Analysis >>> 2015-03-16 10:40:37,212 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:genResolvedParseTree(10129)) - Completed phase >>> 1 of Semantic Analysis >>> 2015-03-16 10:40:37,212 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:getMetaData(1434)) - Get metadata for source >>> tables >>> 2015-03-16 10:40:37,213 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:getMetaData(1582)) - Get metadata for >>> subqueries >>> 2015-03-16 10:40:37,213 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:getMetaData(1606)) - Get metadata for >>> destination tables >>> 2015-03-16 10:40:37,214 INFO [pool-3-thread-2]: >>> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(732)) - 2: >>> source:10.10.10.25 get_table : db=test tbl=test >>> 2015-03-16 10:40:37,214 INFO [pool-3-thread-2]: HiveMetaStore.audit >>> (HiveMetaStore.java:logAuditEvent(358)) - ugi=hadoop2 >>> ip=10.10.10.25 cmd=source:10.10.10.25 get_table : db=test tbl=test >>> 2015-03-16 10:40:37,316 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:genResolvedParseTree(10133)) - Completed >>> getting MetaData in Semantic Analysis >>> 2015-03-16 10:40:37,318 INFO [main]: parse.BaseSemanticAnalyzer >>> (CalcitePlanner.java:canHandleAstForCbo(349)) - Not invoking CBO >>> because the statement has too few joins >>> 2015-03-16 10:40:37,320 INFO [main]: common.FileUtils >>> (FileUtils.java:mkdir(501)) - Creating directory if it doesn't exist: >>> >>> hdfs://nn01:9000/user/hive/warehouse/test.db/test/.hive-staging_hive_2015-03-16_10-40-36_915_4571608652542611567-1 >>> 2015-03-16 10:40:37,429 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:genFileSinkPlan(6474)) - Set stats collection >>> dir : >>> hdfs://nn01:9000/user/hive/warehouse/test.db/test/.hive-staging_hive_2015-03-16_10-40-36_915_4571608652542611567-1/-ext-10001 >>> 2015-03-16 10:40:37,450 INFO [main]: ppd.OpProcFactory >>> (OpProcFactory.java:process(657)) - Processing for FS(3) >>> 2015-03-16 10:40:37,455 INFO [main]: ppd.OpProcFactory >>> (OpProcFactory.java:process(657)) - Processing for SEL(2) >>> 2015-03-16 10:40:37,455 INFO [main]: ppd.OpProcFactory >>> (OpProcFactory.java:process(657)) - Processing for SEL(1) >>> 2015-03-16 10:40:37,455 INFO [main]: ppd.OpProcFactory >>> (OpProcFactory.java:process(384)) - Processing for TS(0) >>> 2015-03-16 10:40:37,507 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG >>> method=partition-retrieving >>> from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner> >>> 2015-03-16 10:40:37,510 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG >>> method=partition-retrieving start=1426482637507 end=1426482637510 >>> duration=3 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner> >>> 2015-03-16 10:40:37,583 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG >>> method=SparkOptimizeOperatorTree >>> from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler> >>> 2015-03-16 10:40:37,638 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG >>> method=SparkOptimizeOperatorTree start=1426482637583 end=1426482637638 >>> duration=55 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler> >>> 2015-03-16 10:40:37,660 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG >>> method=SparkGenerateTaskTree >>> from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler> >>> 2015-03-16 10:40:37,711 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG >>> method=SparkGenerateTaskTree start=1426482637640 end=1426482637711 >>> duration=71 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler> >>> 2015-03-16 10:40:37,715 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG >>> method=SparkOptimizeTaskTree >>> from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler> >>> 2015-03-16 10:40:37,737 INFO [main]: physical.NullScanTaskDispatcher >>> (NullScanTaskDispatcher.java:dispatch(175)) - Looking for table scans >>> where optimization is applicable >>> 2015-03-16 10:40:37,741 INFO [main]: physical.NullScanTaskDispatcher >>> (NullScanTaskDispatcher.java:dispatch(199)) - Found 0 null table scans >>> 2015-03-16 10:40:37,742 INFO [main]: physical.NullScanTaskDispatcher >>> (NullScanTaskDispatcher.java:dispatch(175)) - Looking for table scans >>> where optimization is applicable >>> 2015-03-16 10:40:37,744 INFO [main]: physical.NullScanTaskDispatcher >>> (NullScanTaskDispatcher.java:dispatch(199)) - Found 0 null table scans >>> 2015-03-16 10:40:37,747 INFO [main]: physical.NullScanTaskDispatcher >>> (NullScanTaskDispatcher.java:dispatch(175)) - Looking for table scans >>> where optimization is applicable >>> 2015-03-16 10:40:37,754 INFO [main]: physical.NullScanTaskDispatcher >>> (NullScanTaskDispatcher.java:dispatch(199)) - Found 0 null table scans >>> 2015-03-16 10:40:37,756 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG >>> method=SparkOptimizeTaskTree start=1426482637715 end=1426482637756 >>> duration=41 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler> >>> 2015-03-16 10:40:37,762 INFO [main]: parse.CalcitePlanner >>> (SemanticAnalyzer.java:analyzeInternal(10231)) - Completed plan >>> generation >>> 2015-03-16 10:40:37,762 INFO [main]: ql.Driver >>> (Driver.java:compile(433)) - Semantic Analysis Completed >>> 2015-03-16 10:40:37,762 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=semanticAnalyze >>> start=1426482636929 end=1426482637762 duration=833 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:37,763 INFO [main]: ql.Driver >>> (Driver.java:getSchema(239)) - Returning Hive schema: >>> Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), >>> FieldSchema(name:_col1, type:int, comment:null)], properties:null) >>> 2015-03-16 10:40:37,765 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=compile >>> start=1426482636915 end=1426482637765 duration=850 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:37,765 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG >>> method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:38,193 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG >>> method=acquireReadWriteLocks start=1426482637765 end=1426482638193 >>> duration=428 from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:38,193 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=Driver.execute >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:38,193 INFO [main]: ql.Driver >>> (Driver.java:execute(1317)) - Starting command: insert into table test >>> values(5,8900) >>> 2015-03-16 10:40:38,194 INFO [main]: ql.Driver >>> (SessionState.java:printInfo(852)) - Query ID = >>> hadoop2_20150316104040_c19975af-9dc4-4af2-bfab-d44619224679 >>> 2015-03-16 10:40:38,194 INFO [main]: ql.Driver >>> (SessionState.java:printInfo(852)) - Total jobs = 1 >>> 2015-03-16 10:40:38,194 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=TimeToSubmit >>> start=1426482636914 end=1426482638194 duration=1280 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:38,194 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=runTasks >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:38,194 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG >>> method=task.SPARK.Stage-1 from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:40:38,195 INFO [main]: ql.Driver >>> (SessionState.java:printInfo(852)) - Launching Job 1 out of 1 >>> 2015-03-16 10:40:38,195 INFO [main]: ql.Driver >>> (Driver.java:launchTask(1630)) - Starting task [Stage-1:MAPRED] in >>> parallel >>> 2015-03-16 10:40:38,200 INFO [Thread-49]: hive.metastore >>> (HiveMetaStoreClient.java:open(365)) - Trying to connect to metastore >>> with URI thrift://nn01:7099 >>> 2015-03-16 10:40:38,208 INFO [Thread-49]: hive.metastore >>> (HiveMetaStoreClient.java:open(461)) - Connected to metastore. >>> 2015-03-16 10:40:38,233 INFO [Thread-49]: session.SessionState >>> (SessionState.java:start(488)) - No Tez session required at this >>> point. hive.execution.engine=mr. >>> 2015-03-16 10:40:38,234 INFO [Thread-49]: exec.Task >>> (SessionState.java:printInfo(852)) - In order to change the average >>> load for a reducer (in bytes): >>> 2015-03-16 10:40:38,234 INFO [Thread-49]: exec.Task >>> (SessionState.java:printInfo(852)) - set >>> hive.exec.reducers.bytes.per.reducer=<number> >>> 2015-03-16 10:40:38,234 INFO [Thread-49]: exec.Task >>> (SessionState.java:printInfo(852)) - In order to limit the maximum >>> number of reducers: >>> 2015-03-16 10:40:38,234 INFO [Thread-49]: exec.Task >>> (SessionState.java:printInfo(852)) - set >>> hive.exec.reducers.max=<number> >>> 2015-03-16 10:40:38,234 INFO [Thread-49]: exec.Task >>> (SessionState.java:printInfo(852)) - In order to set a constant number >>> of reducers: >>> 2015-03-16 10:40:38,235 INFO [Thread-49]: exec.Task >>> (SessionState.java:printInfo(852)) - set >>> mapreduce.job.reduces=<number> >>> 2015-03-16 10:40:38,243 INFO [Thread-49]: >>> session.SparkSessionManagerImpl >>> (SparkSessionManagerImpl.java:setup(82)) - Setting up the session >>> manager. >>> 2015-03-16 10:40:38,502 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.connect.timeout -> >>> 1000). >>> 2015-03-16 10:40:38,504 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.serializer -> >>> org.apache.spark.serializer.KryoSerializer). >>> 2015-03-16 10:40:38,505 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.eventLog.enabled -> true). >>> 2015-03-16 10:40:38,508 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.rpc.threads -> 8). >>> 2015-03-16 10:40:38,508 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.secret.bits -> >>> 256). >>> 2015-03-16 10:40:38,508 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.home -> /opt/spark-1.2.1/). >>> 2015-03-16 10:40:38,509 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.rpc.max.size -> >>> 52428800). >>> 2015-03-16 10:40:38,509 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.master -> >>> spark://10.10.10.25:7077). >>> 2015-03-16 10:40:38,509 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.executor.memory -> 512m). >>> 2015-03-16 10:40:38,510 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration >>> (hive.spark.client.server.connect.timeout -> 90000). >>> 2015-03-16 10:40:39,100 WARN [Thread-49]: rpc.RpcConfiguration >>> (RpcConfiguration.java:getServerAddress(123)) - Your hostname, nn01, >>> resolves to a loopback address, but we couldn't find any external IP >>> address! >>> 2015-03-16 10:40:39,101 WARN [Thread-49]: rpc.RpcConfiguration >>> (RpcConfiguration.java:getServerAddress(125)) - Set >>> hive.spark.client.server.address if you need to bind to another >>> address. >>> 2015-03-16 10:40:39,113 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.connect.timeout -> >>> 1000). >>> 2015-03-16 10:40:39,115 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.serializer -> >>> org.apache.spark.serializer.KryoSerializer). >>> 2015-03-16 10:40:39,116 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.eventLog.enabled -> true). >>> 2015-03-16 10:40:39,116 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.rpc.threads -> 8). >>> 2015-03-16 10:40:39,117 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.secret.bits -> >>> 256). >>> 2015-03-16 10:40:39,118 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.home -> /opt/spark-1.2.1/). >>> 2015-03-16 10:40:39,119 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration (hive.spark.client.rpc.max.size -> >>> 52428800). >>> 2015-03-16 10:40:39,120 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.master -> >>> spark://10.10.10.25:7077). >>> 2015-03-16 10:40:39,121 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(113)) - load spark >>> property from hive configuration (spark.executor.memory -> 512m). >>> 2015-03-16 10:40:39,122 INFO [Thread-49]: >>> spark.HiveSparkClientFactory >>> (HiveSparkClientFactory.java:initiateSparkConf(130)) - load RPC >>> property from hive configuration >>> (hive.spark.client.server.connect.timeout -> 90000). >>> 2015-03-16 10:40:43,081 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Spark assembly has been built with >>> Hive, including Datanucleus jars on classpath >>> 2015-03-16 10:40:47,867 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Warning: Ignoring non-spark config >>> property: hive.spark.client.connect.timeout=1000 >>> 2015-03-16 10:40:47,869 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Warning: Ignoring non-spark config >>> property: hive.spark.client.rpc.threads=8 >>> 2015-03-16 10:40:47,869 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Warning: Ignoring non-spark config >>> property: hive.spark.client.rpc.max.size=52428800 >>> 2015-03-16 10:40:47,870 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Warning: Ignoring non-spark config >>> property: hive.spark.client.secret.bits=256 >>> 2015-03-16 10:40:47,872 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Warning: Ignoring non-spark config >>> property: hive.spark.client.server.connect.timeout=90000 >>> 2015-03-16 10:40:48,329 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - 15/03/16 10:40:48 INFO >>> client.RemoteDriver: Connecting to: nn01:53098 >>> 2015-03-16 10:40:48,379 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - Exception in thread "main" >>> java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT >>> 2015-03-16 10:40:48,379 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> >>> org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:46) >>> 2015-03-16 10:40:48,380 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:137) >>> 2015-03-16 10:40:48,380 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:528) >>> 2015-03-16 10:40:48,380 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> 2015-03-16 10:40:48,384 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> 2015-03-16 10:40:48,384 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> 2015-03-16 10:40:48,384 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> java.lang.reflect.Method.invoke(Method.java:606) >>> 2015-03-16 10:40:48,387 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) >>> 2015-03-16 10:40:48,387 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) >>> 2015-03-16 10:40:48,387 INFO [stderr-redir-1]: client.SparkClientImpl >>> (SparkClientImpl.java:run(530)) - at >>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> 2015-03-16 10:40:49,055 WARN [Driver]: client.SparkClientImpl >>> (SparkClientImpl.java:run(388)) - Child process exited with code 1. >>> 2015-03-16 10:42:10,399 WARN [Thread-49]: client.SparkClientImpl >>> (SparkClientImpl.java:<init>(96)) - Error while waiting for client to >>> connect. >>> java.util.concurrent.ExecutionException: >>> java.util.concurrent.TimeoutException: Timed out waiting for client >>> connection. >>> at >>> io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) >>> at >>> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94) >>> at >>> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) >>> at >>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) >>> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) >>> Caused by: java.util.concurrent.TimeoutException: Timed out waiting >>> for client connection. >>> at >>> org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:134) >>> at >>> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) >>> at >>> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) >>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) >>> at java.lang.Thread.run(Thread.java:744) >>> 2015-03-16 10:42:10,413 ERROR [Thread-49]: exec.Task >>> (SessionState.java:printError(861)) - Failed to execute spark task, >>> with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed >>> to create spark client.)' >>> org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark >>> client. >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) >>> at >>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) >>> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) >>> Caused by: java.lang.RuntimeException: >>> java.util.concurrent.ExecutionException: >>> java.util.concurrent.TimeoutException: Timed out waiting for client >>> connection. >>> at com.google.common.base.Throwables.propagate(Throwables.java:156) >>> at >>> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:104) >>> at >>> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) >>> ... 6 more >>> Caused by: java.util.concurrent.ExecutionException: >>> java.util.concurrent.TimeoutException: Timed out waiting for client >>> connection. >>> at >>> io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) >>> at >>> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94) >>> ... 10 more >>> Caused by: java.util.concurrent.TimeoutException: Timed out waiting >>> for client connection. >>> at >>> org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:134) >>> at >>> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) >>> at >>> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) >>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) >>> at java.lang.Thread.run(Thread.java:744) >>> >>> 2015-03-16 10:42:10,413 ERROR [Thread-49]: exec.Task >>> (SparkTask.java:execute(124)) - Failed to execute spark task, with >>> exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to >>> create spark client.)' >>> org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark >>> client. >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) >>> at >>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) >>> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) >>> Caused by: java.lang.RuntimeException: >>> java.util.concurrent.ExecutionException: >>> java.util.concurrent.TimeoutException: Timed out waiting for client >>> connection. >>> at com.google.common.base.Throwables.propagate(Throwables.java:156) >>> at >>> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:104) >>> at >>> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58) >>> at >>> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) >>> ... 6 more >>> Caused by: java.util.concurrent.ExecutionException: >>> java.util.concurrent.TimeoutException: Timed out waiting for client >>> connection. >>> at >>> io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) >>> at >>> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94) >>> ... 10 more >>> Caused by: java.util.concurrent.TimeoutException: Timed out waiting >>> for client connection. >>> at >>> org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:134) >>> at >>> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) >>> at >>> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) >>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) >>> at java.lang.Thread.run(Thread.java:744) >>> 2015-03-16 10:42:12,204 ERROR [main]: ql.Driver >>> (SessionState.java:printError(861)) - FAILED: Execution Error, return >>> code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask >>> 2015-03-16 10:42:12,205 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=Driver.execute >>> start=1426482638193 end=1426482732205 duration=94012 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:42:12,205 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:42:12,544 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks >>> start=1426482732205 end=1426482732544 duration=339 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:42:12,583 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:42:12,583 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks >>> start=1426482732583 end=1426482732583 duration=0 >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:44:30,939 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=Driver.run >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:44:30,939 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=TimeToSubmit >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:44:30,939 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=compile >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:44:30,940 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=parse >>> from=org.apache.hadoop.hive.ql.Driver> >>> 2015-03-16 10:44:30,941 INFO [main]: parse.ParseDriver >>> (ParseDriver.java:parse(185)) - Parsing command: insert into table >>> test values(5,8900) >>> 2015-03-16 10:44:30,942 INFO [main]: parse.ParseDriver >>> (ParseDriver.java:parse(206)) - Parse Completed >>> 2015-03-16 10:44:30,942 INFO [main]: log.PerfLogger >>> (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=parse >>> start=1426482870940 end=1426482870942 duration=2 >>> from=org.apache.hadoop.hive.ql.Driver> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Thanks & Regards >>> Amithsha >>> >>> >>> On Fri, Mar 13, 2015 at 7:36 PM, Xuefu Zhang <xzh...@cloudera.com> wrote: >>> > You need to copy the spark-assembly.jar to your hive/lib. >>> > >>> > Also, you can check hive.log to get more messages. >>> > >>> > On Fri, Mar 13, 2015 at 4:51 AM, Amith sha <amithsh...@gmail.com> >>> > wrote: >>> >> >>> >> Hi all, >>> >> >>> >> >>> >> Recently i have configured Spark 1.2.0 and my environment is hadoop >>> >> 2.6.0 hive 1.1.0 Here i have tried hive on Spark while executing >>> >> insert into i am getting the following g error. >>> >> >>> >> Query ID = hadoop2_20150313162828_8764adad-a8e4-49da-9ef5-35e4ebd6bc63 >>> >> Total jobs = 1 >>> >> Launching Job 1 out of 1 >>> >> In order to change the average load for a reducer (in bytes): >>> >> set hive.exec.reducers.bytes.per.reducer=<number> >>> >> In order to limit the maximum number of reducers: >>> >> set hive.exec.reducers.max=<number> >>> >> In order to set a constant number of reducers: >>> >> set mapreduce.job.reduces=<number> >>> >> Failed to execute spark task, with exception >>> >> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create >>> >> spark client.)' >>> >> FAILED: Execution Error, return code 1 from >>> >> org.apache.hadoop.hive.ql.exec.spark.SparkTask >>> >> >>> >> >>> >> >>> >> Have added the spark-assembly jar in hive lib >>> >> And also in hive console using the command add jar followed by the >>> >> steps >>> >> >>> >> set spark.home=/opt/spark-1.2.1/; >>> >> >>> >> >>> >> add jar >>> >> >>> >> /opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar; >>> >> >>> >> >>> >> >>> >> set hive.execution.engine=spark; >>> >> >>> >> >>> >> set spark.master=spark://xxxxxxx:7077; >>> >> >>> >> >>> >> set spark.eventLog.enabled=true; >>> >> >>> >> >>> >> set spark.executor.memory=512m; >>> >> >>> >> >>> >> set spark.serializer=org.apache.spark.serializer.KryoSerializer; >>> >> >>> >> Can anyone suggest!!!! >>> >> >>> >> >>> >> >>> >> Thanks & Regards >>> >> Amithsha >>> > >>> > >> >> >
