Re: mapreduce on hive with HCatInputFormat and skip.header.line.count=1

2017-06-07 Thread vinay gupta
Hello hive-users,  I am reading a hive table with skip.header.line.count  set to 1 in TBLPROPERTIES In the driver code I do this.     val hiveMetaStoreClient = new HiveMetaStoreClient(new HiveConf(job.getConfiguration, HiveIngestDriver.getClass)) val hiveTable:Table = hiveMetaStoreClient.ge

Re: Pro and Cons of using HBase table as an external table in HIVE

2017-06-07 Thread Uli Bethke
Why are you thinking of using HBase? Just store the SCD versions in a normal Hive dimension table. In case you are worried about updates to columns such as 'valid to' and 'latest record indicator' you can calculate these on the fly using window functions. No need to create and update them phys

Re: Pro and Cons of using HBase table as an external table in HIVE

2017-06-07 Thread Mich Talebzadeh
As I know using Hive on Hbase can only be done through Hive Example hive> create external table MARKETDATAHBASE (key STRING, TICKER STRING, TIMECREATED STRING, PRICE STRING) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,PRI

Re: Hive 2.2

2017-06-07 Thread Barna Zsombor Klara
Hi Boris, you can build from the branches branch-2.3, branch-2 or master. All of these should have spark 2.0 based on the git commit history. However these are non-released versions, so I'm not sure what you can expect in terms of stability. On Wed, Jun 7, 2017 at 4:05 PM, Boris Lublinsky < boris

Re: Hive 2.2

2017-06-07 Thread Boris Lublinsky
Thanks Vergil I do not see version 2.4 The Highest branch is 2.3.RC Is 2.4 master? Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Jun 6, 2017, at 10:07 PM, vergil wrote: > > Hi, > You can build distribution from source code,github > URL:https://git

Pro and Cons of using HBase table as an external table in HIVE

2017-06-07 Thread Ramasubramanian Narayanan
Hi, Can you please let us know Pro and Cons of using HBase table as an external table in HIVE. Will there be any performance degrade when using Hive over HBase instead of using direct HIVE table. The table that I am planning to use in HBase will be master table like account, customer. Wanting to

Re: meet error when building hive-2.4.x from source

2017-06-07 Thread Bing Li
Hi, Please try to build hive-storage-api module in local ahead. e.g. cd storage-api mvn clean install -DskipTests And then build the whole hive project. 2017-06-05 17:20 GMT+08:00 赵伟 : > hi! > First of all,Thank you for your reading my letter. > I meet a problem when I build 2.4.x branch from so

deleting duplicates from large table

2017-06-07 Thread Tousif
Hi Users, I want to know if it is possible to delete duplicates from large non partitioned table. How does ACID perform with large table with billions of rows. -- Regards Tousif Khazi