Hi,
I'm trying to run SQL query on Hive table which is stored on HBase.
I'm using:
- Spark 1.6.0
- HDP 2.2
- Hive 0.14.0
- HBase 0.98.4
I managed to configure working classpath, but I have following problems:
1) I have UDF defined in Hive Metastore (FUNCS table).
Spark cannot use it..
File "/op
?) 8:09
??: "Maciej Bry??ski";
: "Ted Yu"; "dev";
: Re: Spark 1.6.0 + Hive + HBase
Hi,
Indeed, Hive is not able to perform predicate pushdown through a HBase table.
Nor Hive or Impala can.
Broadly speaking, if you need to query your HBase table th
For the last two problems, hbase-site.xml seems not to be on classpath.
Once hbase-site.xml is put on classpath, you should be able to make progress.
Cheers
> On Jan 28, 2016, at 1:14 AM, Maciej Bryński wrote:
>
> Hi,
> I'm trying to run SQL query on Hive table which is stored on HBase.
> I'
Ted,
You're right.
hbase-site.xml resolved problems 2 and 3, but...
Problem 4)
Spark don't push down predicates for HiveTableScan, which means that every
query is full scan.
== Physical Plan ==
TungstenAggregate(key=[],
functions=[(count(1),mode=Final,isDistinct=false)],
output=[count#144L])
+- T
Hi,
Indeed, Hive is not able to perform predicate pushdown through a HBase table.
Nor Hive or Impala can.
Broadly speaking, if you need to query your HBase table through a field other
than de rowkey:
A) Try to "encode" as much info as possible in the rowkey field and use it as
your predicate
anyone took a look at this issue:
https://issues.apache.org/jira/browse/HIVE-11166
i got same exception by inserting into hbase table
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-6-0-Hive-HBase-tp16128p16332.html
Sent from the Apache Spark
"Julio Antonio Soto de Vicente";;
> 发送时间: 2016年1月28日(星期四) 晚上8:09
> 收件人: "Maciej Bryński";
> 抄送: "Ted Yu"; "dev";
> 主题: Re: Spark 1.6.0 + Hive + HBase
>
> Hi,
>
> Indeed, Hive is not able to perform predicate pushdown through a HBase
alsIgnoreCase("string");
>
> String tsColName = null;
> if (iTimestamp >= 0) {
> tsColName =
> jobConf.get(serdeConstants.LIST_COLUMNS).split(",")[iTimestamp];
> }
>
>
>
> -- 原始邮件 ------
> *发件人:* "Jörn Franke";;
{
tsColName =
jobConf.get(serdeConstants.LIST_COLUMNS).split(",")[iTimestamp];
}
-- 原始邮件 --
发件人: "Jörn Franke";;
发送时间: 2016年1月28日(星期四) 晚上9:09
收件人: "开心延年";
抄送: "Julio Antonio Soto de Vicente"; "Maciej
Bryński"; "Ted Yu&
Thanks Ted ,I will try on this version.
-- 原始邮件 --
发件人: "Ted Yu";;
发送时间: 2016年1月28日(星期四) 晚上11:35
收件人: "开心延年";
抄送: "Jörn Franke"; "Julio Antonio Soto de
Vicente"; "Maciej Bryński";
"dev";
主题: Re: 回复:
10 matches
Mail list logo