Re: hive.query.string not reflecting the current query
Hi, thank you all for your replies. I switched to using 'hive.io.filter.text' inline with Peters reply. I also applied the filter negotiation mechanism (HiveStoragePredicateHandler) in my storage handler. It works very well (so far) even though the filter negotiation mechanism is a bit limited in the allowed expression. I'll bring up that question in a separate thread. Br, Petter 2013/12/5 Peter Marron peter.mar...@trilliumsoftware.com Hi, Sorry for the late reply. Maybe the property ‘hive.io.filter.expr.serialized’ is something that can help? It works for me, and it certainly works in the case where the query does not result in a Map/Reduce (which is something that I rely on). (If you google you should be able to find out about it.) Regards, *Peter Marron* Senior Developer, Research Development Office: +44 *(0) 118-940-7609* peter.mar...@trilliumsoftware.com Theale Court First Floor, 11-13 High Street, Theale, RG7 5AH, UK https://www.facebook.com/pages/Trillium-Software/109184815778307 https://twitter.com/TrilliumSW http://www.linkedin.com/company/17710 *www.trilliumsoftware.com http://www.trilliumsoftware.com/* Be Certain About Your Data. Be Trillium Certain. *From:* Petter von Dolwitz (Hem) [mailto:petter.von.dolw...@gmail.com] *Sent:* 03 December 2013 12:46 *To:* user@hive.apache.org *Subject:* hive.query.string not reflecting the current query Hi, I use hive 0.11 with a five machine cluster. I am reading the property hive.query.string from a custom RecordReader (used for reading external tables). If I first invoke a query like select * from mytable where mycolumn='myvalue'; I get the correct query string in this property. If I then invoke select * from mytable limit 100; the property hive.query.string still contains the first query. Seems like hive uses local mode for the second query. Don't know if it is related. Anybody knows why the query string is not updated in the second case? Thanks, Petter image001.pngimage002.pngimage004.pngimage003.png
Limitations in the IndexPredicateAnalyzer
Hi, I use the HiveStoragePredicateHandler interface on my storage handler to be able to push down filters to my record reader for early pruning. It seems like the IndexPredicateAnalyzer is a bit limited on what expression that could be pushed down. From a comment in IndexPredicateAnalyzer: We can only push down stuff which appears as part of a pure conjunction: reject OR, CASE, etc. As such I can push down simple AND expressions in my where clause (and also IN RLIKE and LIKE) but I cannot use OR. Does anybody know why this limitation exists? A guess from my side is that it was originally developed for HBASE integration where OR expressions might not translate well. Having other data sources with custom indexes (or relational databases for that matter) would benefit from being able to handle a wider spectrum of expression. Thanks, Petter
Re: Hive + Secure Hbase
Did you see this JIRA? https://issues.apache.org/jira/browse/HBASE-8662 -- Lefty On Mon, Dec 9, 2013 at 8:35 AM, Justin Workman justinjwork...@gmail.comwrote: I have just started to experiment with Hbase in our cluster. I have an secure Hbase cluster setup and would like to create an external Hive table around several of our Hbase tables. This works fine until we enabled secure (kerberos) client access to Hbase. Now we cannot query any of our Hive external Hbase tables. Is there anyway to make this work, or plans to fix what appears to be an impersonation issue with Hbase? I wasn't able to find and specific Jira's. I am using CDH4.3.0 with the standard components that shipped with CDH4.3.0. Thanks Justin
Re: Versioninfo and platformName issue.
Adam, Here is what i get when run $ hadoop version Hadoop 2.0.0-cdh4.4.0 Subversion file:///var/lib/jenkins/workspace/generic-package-ubuntu64-12-04/CDH4.4.0-Packaging-Hadoop-2013-09-03_18-48-35/hadoop-2.0.0+1475-1.cdh4.4.0.p0.23~precise/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79 Compiled by jenkins on Tue Sep 3 19:33:54 PDT 2013 From source with checksum ac7e170aa709b3ace13dc5f775487180 This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar Do you have specific idea what could have gone wrong with Hadoop Classpath? Thank You, Manish. On Wednesday 11 December 2013 04:51 AM, Adam Kawa wrote: $ hadoop version
Re: Hive + Secure Hbase
Yes, thanks for the reply. I found that one yesterday. I was trying to patch 0.94 yesterday without success. Many unit tests failed during build. Any pointers? Sent from my iPhone On Dec 11, 2013, at 3:13 AM, Lefty Leverenz leftylever...@gmail.com wrote: Did you see this JIRA? https://issues.apache.org/jira/browse/HBASE-8662 -- Lefty On Mon, Dec 9, 2013 at 8:35 AM, Justin Workman justinjwork...@gmail.comwrote: I have just started to experiment with Hbase in our cluster. I have an secure Hbase cluster setup and would like to create an external Hive table around several of our Hbase tables. This works fine until we enabled secure (kerberos) client access to Hbase. Now we cannot query any of our Hive external Hbase tables. Is there anyway to make this work, or plans to fix what appears to be an impersonation issue with Hbase? I wasn't able to find and specific Jira's. I am using CDH4.3.0 with the standard components that shipped with CDH4.3.0. Thanks Justin
Re: Versioninfo and platformName issue.
I think you need to explicitly set HADOOP_VERSION to 2.0.0. Can you try? I have seen hive script reading this variable and falling back to running hadoop version, which in your case it cannot parse due to the cdh suffix. Thanks Szehon On Wed, Dec 11, 2013 at 6:37 AM, Manish manishbh...@rocketmail.com wrote: Adam, Here is what i get when run $ hadoop version Hadoop 2.0.0-cdh4.4.0 Subversion file:///var/lib/jenkins/workspace/generic-package- ubuntu64-12-04/CDH4.4.0-Packaging-Hadoop-2013-09-03_ 18-48-35/hadoop-2.0.0+1475-1.cdh4.4.0.p0.23~precise/src/ hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356 b7de835e79 Compiled by jenkins on Tue Sep 3 19:33:54 PDT 2013 From source with checksum ac7e170aa709b3ace13dc5f775487180 This command was run using /usr/lib/hadoop/hadoop-common- 2.0.0-cdh4.4.0.jar Do you have specific idea what could have gone wrong with Hadoop Classpath? Thank You, Manish. On Wednesday 11 December 2013 04:51 AM, Adam Kawa wrote: $ hadoop version
Re: requesting access to hive confluence wiki
Thanks:) Best, Xiao On Dec 10, 2013, at 6:36 PM, Ashutosh Chauhan hashut...@apache.orgmailto:hashut...@apache.org wrote: Sorry, I missed this one. Xiao you should now have perms to edit cwiki. Thanks, Ashutosh On Tue, Dec 10, 2013 at 6:26 PM, Navis??? navis@nexr.commailto:navis@nexr.com wrote: Is there someone who knows how to do this? 2013/11/30 Xiao Meng xi...@simba.commailto:xi...@simba.com Hi, I would like update/fix some contents on the performance test part. My user name is xiaom. Thanks, Xiao
Question about Hive on Tez
Excuse me. May I ask a question about Hive on Tez? We just started evaluating Hive on Tez. Would like to know, is Hive on Tez development done yet? Is there any documentation we could reference to run benchmarks? Also, is there any known limitations in Hive on Tez? Is it in Hive0.12? Thanks, Zhenxiao
Re: requesting access to hive confluence wiki
Sent from my iPhone On Dec 10, 2013, at 6:35 PM, Ashutosh Chauhan hashut...@apache.org wrote: Sorry
