[ https://issues.apache.org/jira/browse/SPARK-17373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15458431#comment-15458431 ]
prasannaP commented on SPARK-17373: ----------------------------------- I am new to spark.Now I am able to query Hive managed tables through SparkSQL.But I don't know how to query HbaseStorage Handler tables through SparkSQL.Can you please guide me. Thank you Sean Owen. > spark+hive+hbase+hbaseIntegration not working > --------------------------------------------- > > Key: SPARK-17373 > URL: https://issues.apache.org/jira/browse/SPARK-17373 > Project: Spark > Issue Type: Bug > Components: Spark Shell > Reporter: prasannaP > Labels: soon > > SparkSQL+Hive+Hbase+HbaseIntegration doesn't work > Hi, > I am getting error when I am trying to connect hive table (which is being > created through HbaseIntegration) in spark > Steps I followed : > *Hive Table creation code *: > CREATE TABLE test.sample(id string,name string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH > SERDEPROPERTIES ("hbase.columns.mapping" = ":key,details:name") > TBLPROPERTIES ("hbase.table.name" = "sample"); > *DESCRIBE TEST ;* > col_name data_type comment > id string from deserializer > name string from deserializer > *Starting Spark shell* > spark-shell --master local[2] --driver-class-path > /usr/local/hive/lib/hive-hbase-handler-1.2.1.jar: > /usr/local/hbase/lib/hbase-server-0.98.9-hadoop2.jar:/usr/local/hbase/lib/hbase-protocol-0.98.9-hadoo2.jar: > /usr/local/hbase/lib/hbase-hadoop2-compat-0.98.9-hadoop2.jar:/usr/local/hbase/lib/hbase-hadoop-compat-0.98.9-hadoop2.jar: > /usr/local/hbase/lib/hbase-client-0.98.9-hadoop2.jar:/usr/local/hbase/lib/hbase-common-0.98.9-hadoop2.jar: > /usr/local/hbase/lib/htrace-core-2.04.jar:/usr/local/hbase/lib/hbase-common-0.98.9-hadoop2-tests.jar: > /usr/local/hbase/lib/hbase-server-0.98.9-hadoop2-tests.jar:/usr/local/hive/lib/zookeeper-3.4.6.jar:/usr/local/hive/lib/guava-14.0.1.jar > In spark-shell: > val sqlContext=new org.apache.spark.sql.hive.HiveContext(sc) > sqlContext.sql(“select count(*) from test.sample”).collect() > I added this setting in hadoop-env.sh as > export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/lib/* > *Stack Trace* : > Stack SQL context available as sqlContext. > scala> sqlContext.sql("select count(*) from test.sample").collect() > 16/09/02 04:49:28 INFO parse.ParseDriver: Parsing command: select count(*) > from test.sample > 16/09/02 04:49:35 INFO parse.ParseDriver: Parse Completed > 16/09/02 04:49:40 INFO metastore.HiveMetaStore: 0: get_table : db=test > tbl=sample > 16/09/02 04:49:40 INFO HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr > cmd=get_table : db=test tbl=sample > java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes > at > org.apache.hadoop.hive.hbase.HBaseSerDe.parseColumnsMapping(HBaseSerDe.java:184) > at > org.apache.hadoop.hive.hbase.HBaseSerDeParameters.<init>(HBaseSerDeParameters.java:73) > at > org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:117) > at > org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:53) > at > org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:521) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:391) > at > org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276) > at > org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258) > at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605) > at > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1$$anonfun$3.apply(ClientWrapper.scala:331) > at > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1$$anonfun$3.apply(ClientWrapper.scala:326) > at scala.Option.map(Option.scala:145) > at > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1.apply(ClientWrapper.scala:326) > at > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1.apply(ClientWrapper.scala:321) > at > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:279) > at > org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:226) > at > org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:225) > at > org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:268) > at > org.apache.spark.sql.hive.client.ClientWrapper.getTableOption(ClientWrapper.scala:321) > at > org.apache.spark.sql.hive.client.ClientInterface$class.getTable(ClientInterface.scala:122) > at > org.apache.spark.sql.hive.client.ClientWrapper.getTable(ClientWrapper.scala:60) > at > org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:384) > at > org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:457) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:161) > at > org.apache.spark.sql.hive.HiveContext$$anon$2.lookupRelation(HiveContext.scala:457) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:303) > > Could somebody help me in resolving the error. > Would really appreciate the help . > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org