hi,
I have just tested the same command, it works here, can you please provide your
create table command?
regards
Arthur
scala> hiveContext.hql("show tables")
warning: there were 1 deprecation warning(s); re-run with -deprecation for
details
2014-10-03 17:14:33,575 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 17:14:33,575 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 17:14:33,600 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is
deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-10-03 17:14:33,600 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.run>
2014-10-03 17:14:33,600 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=TimeToSubmit>
2014-10-03 17:14:33,600 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=compile>
2014-10-03 17:14:33,600 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=parse>
2014-10-03 17:14:33,600 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 17:14:33,600 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 17:14:33,601 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1412327673600
end=1412327673601 duration=1>
2014-10-03 17:14:33,601 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze>
2014-10-03 17:14:33,603 INFO [main] ql.Driver (Driver.java:compile(450)) -
Semantic Analysis Completed
2014-10-03 17:14:33,603 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=semanticAnalyze
start=1412327673601 end=1412327673603 duration=2>
2014-10-03 17:14:33,604 INFO [main] exec.ListSinkOperator
(Operator.java:initialize(338)) - Initializing Self 0 OP
2014-10-03 17:14:33,604 INFO [main] exec.ListSinkOperator
(Operator.java:initializeChildren(403)) - Operator 0 OP initialized
2014-10-03 17:14:33,604 INFO [main] exec.ListSinkOperator
(Operator.java:initialize(378)) - Initialization Done 0 OP
2014-10-03 17:14:33,604 INFO [main] ql.Driver (Driver.java:getSchema(264)) -
Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
2014-10-03 17:14:33,605 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=compile
start=1412327673600 end=1412327673605 duration=5>
2014-10-03 17:14:33,605 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.execute>
2014-10-03 17:14:33,605 INFO [main] ql.Driver (Driver.java:execute(1117)) -
Starting command: show tables
2014-10-03 17:14:33,605 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=TimeToSubmit
start=1412327673600 end=1412327673605 duration=5>
2014-10-03 17:14:33,605 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=runTasks>
2014-10-03 17:14:33,605 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=task.DDL.Stage-0>
2014-10-03 17:14:33,606 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(454)) - 0: get_database: default
2014-10-03 17:14:33,606 INFO [main] HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr
cmd=get_database: default
2014-10-03 17:14:33,609 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(454)) - 0: get_tables: db=default pat=.*
2014-10-03 17:14:33,609 INFO [main] HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr
cmd=get_tables: db=default pat=.*
2014-10-03 17:14:33,612 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=task.DDL.Stage-0
start=1412327673605 end=1412327673612 duration=7>
2014-10-03 17:14:33,613 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=runTasks
start=1412327673605 end=1412327673613 duration=8>
2014-10-03 17:14:33,613 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.execute
start=1412327673605 end=1412327673613 duration=8>
2014-10-03 17:14:33,613 INFO [main] ql.Driver
(SessionState.java:printInfo(410)) - OK
2014-10-03 17:14:33,613 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 17:14:33,613 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks
start=1412327673613 end=1412327673613 duration=0>
2014-10-03 17:14:33,613 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.run
start=1412327673600 end=1412327673613 duration=13>
2014-10-03 17:14:33,614 INFO [main] mapred.FileInputFormat
(FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2014-10-03 17:14:33,615 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 17:14:33,615 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks
start=1412327673615 end=1412327673615 duration=0>
res4: org.apache.spark.sql.SchemaRDD =
SchemaRDD[3] at RDD at SchemaRDD.scala:103
== Query Plan ==
<Native command: executed by Hive>
scala>
On 3 Oct, 2014, at 4:52 pm, Michael Armbrust <[email protected]> wrote:
> Are you running master? There was briefly a regression here that is
> hopefully fixed by spark#2635.
>
> On Fri, Oct 3, 2014 at 1:43 AM, Kevin Paul <[email protected]> wrote:
> Hi all, I tried to launch my application with spark-submit, the command I use
> is:
>
> bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master local
> myApplicationJar.jar
>
> I've buillt spark with SPARK_HIVE=true, and was able to start HiveContext,
> and was able to run command like,
> hiveContext.sql("select * from myTable")
> or hiveContext.sql("select count(*) from myTable")
> myTable is a table on my hive database. However, when I run the command:
> hiveContext.sql("show tables"), I got the following error:
>
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.Driver.validateConfVariables(Driver.java:1057)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:948)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:298)
> at
> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272)
> at
> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
> at
> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
> at
> org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
> at
> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360)
> at
> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360)
>
> Thanks,
> Kelvin Paul
>
