Spark 1.6.0 + Hive + HBase

Hi, I'm trying to run SQL query on Hive table which is stored on HBase. I'm using: - Spark 1.6.0 - HDP 2.2 - Hive 0.14.0 - HBase 0.98.4 I managed to configure working classpath, but I have following problems: 1) I have UDF defined in Hive Metastore (FUNCS table). Spark cannot use it.. File "/op

?????? Spark 1.6.0 + Hive + HBase

?) 8:09 ??: "Maciej Bry??ski"; : "Ted Yu"; "dev"; : Re: Spark 1.6.0 + Hive + HBase Hi, Indeed, Hive is not able to perform predicate pushdown through a HBase table. Nor Hive or Impala can. Broadly speaking, if you need to query your HBase table th

Re: Spark 1.6.0 + Hive + HBase

For the last two problems, hbase-site.xml seems not to be on classpath. Once hbase-site.xml is put on classpath, you should be able to make progress. Cheers > On Jan 28, 2016, at 1:14 AM, Maciej Bryński wrote: > > Hi, > I'm trying to run SQL query on Hive table which is stored on HBase. > I'

Re: Spark 1.6.0 + Hive + HBase

Ted, You're right. hbase-site.xml resolved problems 2 and 3, but... Problem 4) Spark don't push down predicates for HiveTableScan, which means that every query is full scan. == Physical Plan == TungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[count#144L]) +- T

Re: Spark 1.6.0 + Hive + HBase

Hi, Indeed, Hive is not able to perform predicate pushdown through a HBase table. Nor Hive or Impala can. Broadly speaking, if you need to query your HBase table through a field other than de rowkey: A) Try to "encode" as much info as possible in the rowkey field and use it as your predicate

Re: Spark 1.6.0 + Hive + HBase

anyone took a look at this issue: https://issues.apache.org/jira/browse/HIVE-11166 i got same exception by inserting into hbase table -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-6-0-Hive-HBase-tp16128p16332.html Sent from the Apache Spark

Re: 回复： Spark 1.6.0 + Hive + HBase

"Julio Antonio Soto de Vicente";; > 发送时间: 2016年1月28日(星期四) 晚上8:09 > 收件人: "Maciej Bryński"; > 抄送: "Ted Yu"; "dev"; > 主题: Re: Spark 1.6.0 + Hive + HBase > > Hi, > > Indeed, Hive is not able to perform predicate pushdown through a HBase

Re: 回复： Spark 1.6.0 + Hive + HBase

alsIgnoreCase("string"); > > String tsColName = null; > if (iTimestamp >= 0) { > tsColName = > jobConf.get(serdeConstants.LIST_COLUMNS).split(",")[iTimestamp]; > } > > > > -- 原始邮件 ------ > *发件人:* "Jörn Franke";;

回复： 回复： Spark 1.6.0 + Hive + HBase

{ tsColName = jobConf.get(serdeConstants.LIST_COLUMNS).split(",")[iTimestamp]; } -- 原始邮件 -- 发件人: "Jörn Franke";; 发送时间: 2016年1月28日(星期四) 晚上9:09 收件人: "开心延年"; 抄送: "Julio Antonio Soto de Vicente"; "Maciej Bryński"; "Ted Yu&

回复： 回复： Spark 1.6.0 + Hive + HBase

Thanks Ted ,I will try on this version. -- 原始邮件 -- 发件人: "Ted Yu";; 发送时间: 2016年1月28日(星期四) 晚上11:35 收件人: "开心延年"; 抄送: "Jörn Franke"; "Julio Antonio Soto de Vicente"; "Maciej Bryński"; "dev"; 主题: Re: 回复：