Stana,
I'm not sure if I fully understand the problem. spark-submit is launched in
the same host as your application, which should be able to access
hive-exec.jar. Yarn cluster needs the jar also, but HS2 or Hive CLI will
take care of that. Since you are not using either of which, then, it's your
application's responsibility to make that happen.
Did I missed anything else?
Thanks,
Xuefu
On Sun, Mar 20, 2016 at 11:18 PM, Stana <st...@is-land.com.tw> wrote:
> Does anyone have suggestions in setting property of hive-exec-2.0.0.jar
> path in application?
> Something like
>
> 'hiveConf.set("hive.remote.driver.jar","hdfs://storm0:9000/tmp/spark-assembly-1.4.1-hadoop2.6.0.jar")'.
>
>
>
> 2016-03-11 10:53 GMT+08:00 Stana <st...@is-land.com.tw>:
>
> > Thanks for reply
> >
> > I have set the property spark.home in my application. Otherwise the
> > application threw 'SPARK_HOME not found exception'.
> >
> > I found hive source code in SparkClientImpl.java:
> >
> > private Thread startDriver(final RpcServer rpcServer, final String
> > clientId, final String secret)
> > throws IOException {
> > ...
> >
> > List<String> argv = Lists.newArrayList();
> >
> > ...
> >
> > argv.add("--class");
> > argv.add(RemoteDriver.class.getName());
> >
> > String jar = "spark-internal";
> > if (SparkContext.jarOfClass(this.getClass()).isDefined()) {
> > jar = SparkContext.jarOfClass(this.getClass()).get();
> > }
> > argv.add(jar);
> >
> > ...
> >
> > }
> >
> > When hive executed spark-submit , it generate the shell command with
> > --class org.apache.hive.spark.client.RemoteDriver ,and set jar path with
> > SparkContext.jarOfClass(this.getClass()).get(). It will get the local
> path
> > of hive-exec-2.0.0.jar.
> >
> > In my situation, the application and yarn cluster are in different
> cluster.
> > When application executed spark-submit with local path of
> > hive-exec-2.0.0.jar to yarn cluster, there 's no hive-exec-2.0.0.jar in
> > yarn cluster. Then application threw the exception: "hive-exec-2.0.0.jar
> > does not exist ...".
> >
> > Can it be set property of hive-exec-2.0.0.jar path in application ?
> > Something like 'hiveConf.set("hive.remote.driver.jar",
> > "hdfs://storm0:9000/tmp/spark-assembly-1.4.1-hadoop2.6.0.jar")'.
> > If not, is it possible to achieve in the future version?
> >
> >
> >
> >
> > 2016-03-10 23:51 GMT+08:00 Xuefu Zhang <xu...@uber.com>:
> >
> >> You can probably avoid the problem by set environment variable
> SPARK_HOME
> >> or JVM property spark.home that points to your spark installation.
> >>
> >> --Xuefu
> >>
> >> On Thu, Mar 10, 2016 at 3:11 AM, Stana <st...@is-land.com.tw> wrote:
> >>
> >> > I am trying out Hive on Spark with hive 2.0.0 and spark 1.4.1, and
> >> > executing org.apache.hadoop.hive.ql.Driver with java application.
> >> >
> >> > Following are my situations:
> >> > 1.Building spark 1.4.1 assembly jar without Hive .
> >> > 2.Uploading the spark assembly jar to the hadoop cluster.
> >> > 3.Executing the java application with eclipse IDE in my client
> computer.
> >> >
> >> > The application went well and it submitted mr job to the yarn cluster
> >> > successfully when using " hiveConf.set("hive.execution.engine", "mr")
> >> > ",but it threw exceptions in spark-engine.
> >> >
> >> > Finally, i traced Hive source code and came to the conclusion：
> >> >
> >> > In my situation, SparkClientImpl class will generate the spark-submit
> >> > shell and executed it.
> >> > The shell command allocated --class with RemoteDriver.class.getName()
> >> > and jar with SparkContext.jarOfClass(this.getClass()).get(), so that
> >> > my application threw the exception.
> >> >
> >> > Is it right? And how can I do to execute the application with
> >> > spark-engine successfully in my client computer ? Thanks a lot!
> >> >
> >> >
> >> > Java application code:
> >> >
> >> > public class TestHiveDriver {
> >> >
> >> > private static HiveConf hiveConf;
> >> > private static Driver driver;
> >> > private static CliSessionState ss;
> >> > public static void main(String[] args){
> >> >
> >> > String sql = "select * from hadoop0263_0 as a join
> >> > hadoop0263_0 as b
> >> > on (a.key = b.key)";
> >> > ss = new CliSessionState(new
> >> HiveConf(SessionState.class));
> >> > hiveConf = new HiveConf(Driver.class);
> >> > hiveConf.set("fs.default.name",
> "hdfs://storm0:9000");
> >> > hiveConf.set("yarn.resourcemanager.address",
> >> > "storm0:8032");
> >> > hiveConf.set("yarn.resourcemanager.scheduler.address",
> >> > "storm0:8030");
> >> >
> >> >
> >>
> hiveConf.set("yarn.resourcemanager.resource-tracker.address","storm0:8031");
> >> > hiveConf.set("yarn.resourcemanager.admin.address",
> >> > "storm0:8033");
> >> > hiveConf.set("mapreduce.framework.name", "yarn");
> >> > hiveConf.set("mapreduce.johistory.address",
> >> > "storm0:10020");
> >> >
> >> >
> >>
> hiveConf.set("javax.jdo.option.ConnectionURL","jdbc:mysql://storm0:3306/stana_metastore");
> >> >
> >> >
> >>
> hiveConf.set("javax.jdo.option.ConnectionDriverName","com.mysql.jdbc.Driver");
> >> > hiveConf.set("javax.jdo.option.ConnectionUserName",
> >> > "root");
> >> > hiveConf.set("javax.jdo.option.ConnectionPassword",
> >> > "123456");
> >> > hiveConf.setBoolean("hive.auto.convert.join",false);
> >> > hiveConf.set("spark.yarn.jar",
> >> > "hdfs://storm0:9000/tmp/spark-assembly-1.4.1-hadoop2.6.0.jar");
> >> > hiveConf.set("spark.home","target/spark");
> >> > hiveConf.set("hive.execution.engine", "spark");
> >> > hiveConf.set("hive.dbname", "default");
> >> >
> >> >
> >> > driver = new Driver(hiveConf);
> >> > SessionState.start(hiveConf);
> >> >
> >> > CommandProcessorResponse res = null;
> >> > try {
> >> > res = driver.run(sql);
> >> > } catch (CommandNeedRetryException e) {
> >> > // TODO Auto-generated catch block
> >> > e.printStackTrace();
> >> > }
> >> >
> >> > System.out.println("Response Code:" +
> >> > res.getResponseCode());
> >> > System.out.println("Error Message:" +
> >> > res.getErrorMessage());
> >> > System.out.println("SQL State:" + res.getSQLState());
> >> >
> >> > }
> >> > }
> >> >
> >> >
> >> >
> >> >
> >> > Exception of spark-engine:
> >> >
> >> > 16/03/10 18:32:58 INFO SparkClientImpl: Running client driver with
> >> > argv:
> >> >
> >>
> /Volumes/Sdhd/Documents/project/island/java/apache/hive-200-test/hive-release-2.0.0/itests/hive-unit/target/spark/bin/spark-submit
> >> > --properties-file
> >> >
> >> >
> >>
> /var/folders/vt/cjcdhms903x7brn1kbh558s40000gn/T/spark-submit.7697089826296920539.properties
> >> > --class org.apache.hive.spark.client.RemoteDriver
> >> >
> >> >
> >>
> /Users/stana/.m2/repository/org/apache/hive/hive-exec/2.0.0/hive-exec-2.0.0.jar
> >> > --remote-host MacBook-Pro.local --remote-port 51331 --conf
> >> > hive.spark.client.connect.timeout=1000 --conf
> >> > hive.spark.client.server.connect.timeout=90000 --conf
> >> > hive.spark.client.channel.log.level=null --conf
> >> > hive.spark.client.rpc.max.size=52428800 --conf
> >> > hive.spark.client.rpc.threads=8 --conf
> >> > hive.spark.client.secret.bits=256
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: 16/03/10 18:33:09 INFO Client:
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: client token: N/A
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: diagnostics: N/A
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: ApplicationMaster
> host:
> >> > N/A
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: ApplicationMaster RPC
> >> > port: -1
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: queue: default
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: start time:
> >> 1457180833494
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: final status:
> UNDEFINED
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: tracking URL:
> >> > http://storm0:8088/proxy/application_1457002628102_0043/
> >> > 16/03/10 18:33:09 INFO SparkClientImpl: user: stana
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: 16/03/10 18:33:10 INFO Client:
> >> > Application report for application_1457002628102_0043 (state: FAILED)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: 16/03/10 18:33:10 INFO Client:
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: client token: N/A
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: diagnostics:
> >> Application
> >> > application_1457002628102_0043 failed 1 times due to AM Container for
> >> > appattempt_1457002628102_0043_000001 exited with exitCode: -1000
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: For more detailed output,
> >> > check application tracking
> >> > page:http://storm0:8088/proxy/application_1457002628102_0043/Then,
> >> > click on links to logs of each attempt.
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: Diagnostics:
> >> > java.io.FileNotFoundException: File
> >> >
> >> >
> >>
> file:/Users/stana/.m2/repository/org/apache/hive/hive-exec/2.0.0/hive-exec-2.0.0.jar
> >> > does not exist
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: Failing this attempt. Failing
> >> > the application.
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: ApplicationMaster
> host:
> >> > N/A
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: ApplicationMaster RPC
> >> > port: -1
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: queue: default
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: start time:
> >> 1457180833494
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: final status: FAILED
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: tracking URL:
> >> > http://storm0:8088/cluster/app/application_1457002628102_0043
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: user: stana
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: Exception in thread "main"
> >> > org.apache.spark.SparkException: Application
> >> > application_1457002628102_0043 finished with failed status
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > org.apache.spark.deploy.yarn.Client.run(Client.scala:920)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > org.apache.spark.deploy.yarn.Client$.main(Client.scala:966)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > org.apache.spark.deploy.yarn.Client.main(Client.scala)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> >
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> >
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > java.lang.reflect.Method.invoke(Method.java:606)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> >
> >> >
> >>
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> >
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: at
> >> > org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: 16/03/10 18:33:10 INFO
> >> > ShutdownHookManager: Shutdown hook called
> >> > 16/03/10 18:33:10 INFO SparkClientImpl: 16/03/10 18:33:10 INFO
> >> > ShutdownHookManager: Deleting directory
> >> >
> >> >
> >>
> /private/var/folders/vt/cjcdhms903x7brn1kbh558s40000gn/T/spark-5b92ce20-b6f8-4832-8b15-5e98bd0e0705
> >> > 16/03/10 18:33:10 WARN SparkClientImpl: Error while waiting for client
> >> > to connect.
> >> > java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> >> > Cancel client '5bda93c0-865b-48a8-b368-c2fcc30e81e8'. Error: Child
> >> > process exited before connecting back
> >> > at
> >> > io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> >> > ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> >> > at
> >> >
> >>
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:101)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:98)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:94)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:63)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:114)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:131)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:117)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:181)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:119)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:102)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10195)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> >> > [hive-exec-2.0.0.jar:2.0.0]
> >> > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> >> > [hive-exec-2.0.0.jar:?]
> >> > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:319)
> >> > [hive-exec-2.0.0.jar:?]
> >> > at
> >> > org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1255)
> >> > [hive-exec-2.0.0.jar:?]
> >> > at
> >> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1301)
> >> > [hive-exec-2.0.0.jar:?]
> >> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> >> > [hive-exec-2.0.0.jar:?]
> >> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172)
> >> > [hive-exec-2.0.0.jar:?]
> >> > at
> >> > org.apache.hadoop.hive.ql.TestHiveDriver.main(TestHiveDriver.java:41)
> >> > [test-classes/:?]
> >> > Caused by: java.lang.RuntimeException: Cancel client
> >> > '5bda93c0-865b-48a8-b368-c2fcc30e81e8'. Error: Child process exited
> >> > before connecting back
> >> > at
> >> >
> >>
> org.apache.hive.spark.client.rpc.RpcServer.cancelClient(RpcServer.java:179)
> >> > ~[hive-exec-2.0.0.jar:2.0.0]
> >> > at
> >> >
> >>
> org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:450)
> >> > ~[hive-exec-2.0.0.jar:2.0.0]
> >> > at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_67]
> >> > 16/03/10 18:33:10 WARN SparkClientImpl: Child process exited with code
> >> 1.
> >> > FAILED: SemanticException Failed to get a spark session:
> >> > org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> >> > spark client.
> >> > 16/03/10 18:33:10 ERROR Driver: FAILED: SemanticException Failed to
> >> > get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException:
> >> > Failed to create spark client.
> >> > org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a
> >> > spark session: org.apache.hadoop.hive.ql.metadata.HiveException:
> >> > Failed to create spark client.
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:121)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:181)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:119)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:102)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10195)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229)
> >> > at
> >> >
> >>
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> >> > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> >> > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:319)
> >> > at
> >> > org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1255)
> >> > at
> >> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1301)
> >> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
> >> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172)
> >> > at
> >> > org.apache.hadoop.hive.ql.TestHiveDriver.main(TestHiveDriver.java:41)
> >> >
> >>
> >
> >
>
