I’m trying to create a DF for an external Hive table that is in HBase. I get the a NoSuchMethodError org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initSerdeParams(Lorg/apache/hadoop/conf/Configuration;Ljava/util/Properties;Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe$SerDeParameters;
I’m running Spark 1.6.0 on HDP 2.2.4-12-1 (Hive 0.14 and HBase 0.98.4) in secure mode. Anybody see this before ? Below is a stack trace and the hive table’s info. scala> sqlContext.table("item_data_lib.pcn_item") java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initSerdeParams(Lorg/apache/hadoop/conf/Configuration;Ljava/util/Properties;Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe$SerDeParameters; at org.apache.hadoop.hive.hbase.HBaseSerDeParameters.<init>(HBaseSerDeParameters.java:93) at org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:92) at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:53) at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:521) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:391) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258) at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1$$anonfun$3.apply(ClientWrapper.scala:331) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1$$anonfun$3.apply(ClientWrapper.scala:326) at scala.Option.map(Option.scala:145) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1.apply(ClientWrapper.scala:326) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getTableOption$1.apply(ClientWrapper.scala:321) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:279) at org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:226) at org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:225) at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:268) at org.apache.spark.sql.hive.client.ClientWrapper.getTableOption(ClientWrapper.scala:321) at org.apache.spark.sql.hive.client.ClientInterface$class.getTable(ClientInterface.scala:122) at org.apache.spark.sql.hive.client.ClientWrapper.getTable(ClientWrapper.scala:60) at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:384) at org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:457) at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:161) at org.apache.spark.sql.hive.HiveContext$$anon$2.lookupRelation(HiveContext.scala:457) at org.apache.spark.sql.SQLContext.table(SQLContext.scala:831) at org.apache.spark.sql.SQLContext.table(SQLContext.scala:827) hive> show create table item_data_lib.pcn_item; OK CREATE EXTERNAL TABLE `item_data_lib.pcn_item`( `key` string COMMENT 'from deserializer', `p1` string COMMENT 'from deserializer', `p2` string COMMENT 'from deserializer', `p3` string COMMENT 'from deserializer', `p4` string COMMENT 'from deserializer', `p5` string COMMENT 'from deserializer', `p6` string COMMENT 'from deserializer', `p7` string COMMENT 'from deserializer', `p8` string COMMENT 'from deserializer', `p9` string COMMENT 'from deserializer', `p10` string COMMENT 'from deserializer', `p11` string COMMENT 'from deserializer', `p12` string COMMENT 'from deserializer', `p13` string COMMENT 'from deserializer', `d1` string COMMENT 'from deserializer', `d2` string COMMENT 'from deserializer', `d3` string COMMENT 'from deserializer', `d4` string COMMENT 'from deserializer', `d5` string COMMENT 'from deserializer', `d6` string COMMENT 'from deserializer', `d7` string COMMENT 'from deserializer', `d8` string COMMENT 'from deserializer', `d9` string COMMENT 'from deserializer', `d10` string COMMENT 'from deserializer', `d11` string COMMENT 'from deserializer', `d12` string COMMENT 'from deserializer', `d13` string COMMENT 'from deserializer', `d14` string COMMENT 'from deserializer', `d15` string COMMENT 'from deserializer', `d16` string COMMENT 'from deserializer', `d17` string COMMENT 'from deserializer') ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'hbase.columns.mapping'=':key,p:p1,p:p2,p:p3,p:p4,p:p5,p:p6,p:p7,p:p8,p:p9,p:p10,p:p11,p:p12,p:p13,d:d1,d:d2,d:d3,d:d4,d:d5,d:d6,d:d7,d:d8,d:d9,d:d10,d:d11,d:d12,d:d13,d:d14,d:d15,d:d16,d:d17', 'serialization.format'='1') TBLPROPERTIES ( 'hbase.table.name'='item_data_lib:pcn_item', 'transient_lastDdlTime'='1454038459’) Thanks, Doug --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org