Re: Programmatically determine version of Hive running on server

Hi Bohdan, Check *HiveVersionInfo* class You may use *HiveVersionInfo.getVersion()* to get the hive version from Java code. Regards, Amey On Mon, 13 Aug 2018 at 16:14, Bohdan Kazydub wrote: > No. Sorry, forgot to mention that this should be done from Java code. > > Best regards, > Bohdan > >

Any hooks to invoke the custom database's statistics for aggregate hive queries

Hi All, We have developed a custom storgeHandler implementing *HiveStorageHandler*. We also have Api's/statistics for totalCount, max, min etc. for the data stored in our database. See below example queries: 1. select count(*) from my_table; 2. select max(id_column) from my_table; So for above q

Re: Pro and Cons of using HBase table as an external table in HIVE

Hi Michael, "If there is predicate pushdown, then you will be faster, assuming that the query triggers an implied range scan" ---> Does this bring results faster than plain hive querying over ORC / Text file formats In other words Is querying over plain hive (ORC or Text) *always* faster than thr

Re: Hive orc use case

Thanks Alan, Your comment answers my question :) I will start looking into HiveEndPoint api's. Regards, Amey Barve On 26 September 2016 at 23:50, Alan Gates wrote: > As long as there is a spare worker thread this should be picked up within > a few seconds. It’s true you can’t

Hive orc use case

Hi All, I have an use case where I need to append either 1 or many rows to orcFile as well as read 1 or many rows from it. I observed that I cannot read rows from OrcFile unless I close the OrcFile's writer, is this correct? Why doesn't write actually flush the rows to the orcFile, is there any

Re: Predicates for 'like' and 'between' operators to custom storage handler.

are doing this through Java? > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpr

Re: Predicates for 'like' and 'between' operators to custom storage handler.

gt; > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 5 May 2016

Predicates for 'like' and 'between' operators to custom storage handler.

Hi All, I have implemented custom storage-handler and able to get predicates from hive for '=', '!=', '<', '<=', '>' and '>=' operators. But I cannot get predicates from hive for 'like', 'between' operators. Here's my code: *final String expression = conf.get(TableScanDesc.FILTER_EXPR_CONF_STR);

Re: Hive footprint

m > isn't busy handling compaction and lots of new write then your read > performance in all cases will improve. Always do your own POC and use your > own data to test. > > Thanks, > Peyman > > > > On Tue, Apr 19, 2016 at 2:26 AM, Amey Barve wrote: > >> Hi

Re: Hive footprint

Hi Peyman, You say: "you can use Hive storage handler to read data from HBase the performance would be lower than reading from HDFS directly for analytic." Why is it so? Is it slow as compared to ORC, Parquet, and even Text file format? Regards, Amey On Tue, Apr 19, 2016 at 4:32 AM, Peyman Mohaj

Re: select count(*) from table;

Hi All, Can custom storage handlers get information for queries like count, max, min etc. from hive directly so that for each of such queries RecordReader need not fetch all the records? Regards, Amey On Tue, Mar 22, 2016 at 1:46 PM, Amey Barve wrote: > Thanks Nitin, Mich, > > if

Re: select count(*) from table;

> > > File statistics, Stripe statistics and row group statistics are kept. So > ORC table will rely on those if needed > > > HTH > > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCC

select count(*) from table;

select count(*) from table; How does hive evaluate count(*) on a table? Does it return count by actually querying table, or directly return count by consulting some statistics locally. For Hive's Text format it takes few seconds while Hive's Orc format takes fraction of seconds. Regards, Amey

predicate push down with custom StorageHandler

Hi All, I have implemented HiveStoragePredicateHandler for my StorageHandler. I am able to push down predicates for following query and this works fine. Example: select o_custkey,c_custkey from customer,orders where c_mktsegment = 'BUILDING'; (*column-ref = c_mktsegment comparison-op = op constan

Re: Is Hive Index officially not recommended?

Hi Gopal, As you suggested in your email above that *Part #1 of using hive indexes effectively is to write your ownHiveIndexHandler, with usesIndexTable=false;* *And then write a IndexPredicateAnalyzer, which lets you map arbitrarylookups into other range conditions.* Is anybody storing there

Re: How to set hive.aux.jars.path in hive1.1.0?

Hi Heng, Following steps worked for me: 1. sudo mkdir $HIVE_HOME/auxlib on machine running hive server 2 2. Copy all your jars in this folder 3. restart hive server 2 4. Now you can run hive queries through beeline Regards, Amey On Wed, Dec 30, 2015 at 4:41 AM, Heng Chen wrote: > Yeah, this is

Re: unique-id for the mapper task with tez execution engine

Ok Thanks. Can I get this *, ***from some conf to be absolutely sure that I get unique id ? Regards, Amey On Wed, Dec 23, 2015 at 12:06 PM, Gopal Vijayaraghavan wrote: > Hi, > > > So what do you suggest to get unique-id for mapper task with tez > >execution engine? > > > > conf.get("mapreduce.

Re: unique-id for the mapper task with tez execution engine

Thanks Gopal! So what do you suggest to get unique-id for mapper task with tez execution engine? conf.get("*mapreduce.task.**partition*"); Is this correct? Regards, Amey On Wed, Dec 23, 2015 at 10:58 AM, Gopal Vijayaraghavan wrote: > Hi, > > (x-posts to bcc:) > > On

unique-id for the mapper task with tez execution engine

Hi All, Thanks in advance! I am running hive queries with mr engine and I wanted to get unique-id from the mapper task so I used following property from configuration conf.get("mapreduce.task.id"); Now I want to run same hive queries with tez engine and I want to know what should be my unique-id

Test framework for hive

Hello All, I want to test a hive storage handler, What is the best way to do so? Any test framework that I can use to test with all of hive versions. I also have another question: Can anybody point me to hive test/s that supports all hive data types and then insertion and retrieval of data on to

Re: override log4j level

Hi Patcharee, Use *--hiveconf hive.root.logger=ERROR,console* Regards, Amey On Mon, Nov 16, 2015 at 10:57 PM, pth001 wrote: > Hi, > > How can I override log4j level by using --hiveconf? I want to use ERROR > level for some tasks. > > Thanks, > Patcharee >

Hive Storage Handler to replace HDFS with custom store

Hi All, Hive by default queries onto HDFS. If I implement Hive Storage Handler for my store with 1. input format 2. output format 3. AbstractSerDe Hive queries will then store and retrieve from my store instead of HDFS, Is this understanding correct? or any other changes needed in Hive code? If