RE: [Hive 0.13.1] - Explanation/confusion over "Fatal error occurred when node tried to create too many dynamic partitions" on small dataset with dynamic partitions

Hi Lefty, I took a look at the documentation link and I noticed that it can be improved. For example the paragraph below: “How does Hive distribute the rows across the buckets? In general, the bucket number is determined by the expression hash_function(bucketing_column) mod num_buckets.

UDF cannot be found when the query is submitted via templeton

Hi experts I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following. select count(FindPattern(s_sitename)) AS testcol from weblogs; However, when I tried to use the UDF from WebHCat (i.e. submit the above command via WebHCat), the job always fails

Transactional table read lifecycle

Hi, I'm working on a Cascading Tap that reads the data that backs a transactional Hive table. I've successfully utilised the in-built OrcInputFormat functionality to read and merge the deltas with the base and optionally pull in the RecordIdentifiers. However, I'm now considering what other steps I

test from Ahmed Mosaad

test 17/4/2015

Hive Stats shows numRows=0 when data imported through Sqoop

Hi All, I'm using CDH 5.3.2 and i use sqoop to load data into hive tables. last statement of the sqoop import shows stats log which always shows [numFiles=,,numRows=0], am i missing any conffiguration here? Though I have set MySQL as stats db, auto stats doesn't seem to be working, but when I ru

RE: Hive Stats shows numRows=0 when data imported through Sqoop

Hi Suresh, I guess you are also using MySQL as your Hive metastore? What configuration have you set for stats collection? HTH Mich Talebzadeh http://talebzadehmich.wordpress.com Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0

Re: Hive Stats shows numRows=0 when data imported through Sqoop

Hi Mich Yes MySQL is my meta store db, I have set 1. Hive.stats.dbclause 2.hive.stats.jdbcdriver 3.hive.stats.dbconnectionstring The above configure for stats On Friday, April 17, 2015, Mich Talebzadeh wrote: > Hi Suresh, > > > > I guess you are also using MySQL as your Hive metastore? > > > >

Error when connecting Hue to HiveServer2

Hey Guys, I am see the following error when attempting to connect Hue to the hive metatstore: From hive-site.log 015-04-17 05:12:48,857 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers_core(679)) - No user is added in admin role, since config is empty 2015-04-17 05:12:4

RE: Error when connecting Hue to HiveServer2

Hi Gary, Is your hiverserver2 running OK. How did you start it? $HIVE_HOME/bin/hiveserver2 & What do you when you run the command below? I assume that your hiveserver is running on port 1? netstat -alnp|egrep 'Local|1|9083' HTH Mich Talebzadeh http://taleb

Table bucketing on structured fields

Hi there, in Hive 0.13.0, I am trying to create a table that should be bucketed by a structured field: CREATE TABLE foo (bar struct) CLUSTERED BY (bar.a) INTO 32 buckets; Unfortunately, I am getting an error that dots are not allowed in the buckets specification: Error occurred executing hive

RE: Error when connecting Hue to HiveServer2

Hi Mich, tcp0 0 0.0.0.0:1 0.0.0.0:* LISTEN 16561/java tcp0 0 0.0.0.0:90830.0.0.0:* LISTEN 16654/java I am starting to think this is firewall issue not sure what is going on. Yep HiveServer

RE: Error when connecting Hue to HiveServer2

OK so both are running and up (metastore on 9083 and hiveserver on 1 Try doing telnet hostname 1 Can you connect like below (mine runs on 10010 J) hduser@rhes564::/home/hduser> telnet rhes564 10010 Trying 50.140.197.217... Connected to rhes564. Escape character is '^]'.

RE: Table bucketing on structured fields

Hi Michael, I would be curious to know what advantage you are going to get by hashing a structured field. Has that structured field got very high selectivity so you end up with equally sized buckets (files) spread? How about the following hive> CREATE TABLE foo (id bigint, bar struct) C

Re: UDF cannot be found when the query is submitted via templeton

Does fully qualifying the function name (HiveUDF.FindPattern()) in the query help here? On Apr 17, 2015, at 6:44 AM, Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Hi experts I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following. select

RE: Error when connecting Hue to HiveServer2

Thanks Mich good test. [root@localhost root]# telnet 127.0.0.1 1 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. Yes all good it seems. Cheers, Gazza From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: Friday, April 17, 2015 11:44 AM To: user@hive.apache.org Subje

RE: Error when connecting Hue to HiveServer2

2015-04-17 08:24:12,858 DEBUG [HiveServer2-Handler-Pool: Thread-23]: transport.TSaslTransport (TSaslTransport.java:open(243)) - opening transport org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$HiveSaslServerTransportFactory$1@12d391f8 2015-04-17 08:24:12,858 DEBUG [HiveServer2-Handler-Pool:

RE: Error when connecting Hue to HiveServer2

The below I had to set to avoid on a derby database to get past this problem. hive.server2.authentication NOSASL Expects one of [nosasl, none, ldap, kerberos, pam, custom]. Client authentication types. NONE: no authentication check LDAP: LDAP/AD based au

RE: Error when connecting Hue to HiveServer2

It always surprises how many parameters are around for different metastores J I use Oracle as my Metastore the parameter is set to hive.server2.authentication NONE Expects one of [nosasl, none, ldap, kerberos, pam, custom]. Client authentication types.

RE: Error when connecting Hue to HiveServer2

Yep a true gem ☺ From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: Friday, April 17, 2015 2:03 PM To: user@hive.apache.org Subject: RE: Error when connecting Hue to HiveServer2 It always surprises how many parameters are around for different metastores ☺ I use Oracle as my Metastore the

Re: Table bucketing on structured fields

Hi Mich, yes, the structured field has very good selectivity. I would not achieve perfectly equally sized buckets, but I don't expect any skew problems. Of course, moving the structured field to top-level would allow bucketing. But I would prefer to not change the schema, as many queries have a

Re: [Hive] Slow Loading Data Process with Parquet over 30k Partitions

Hi Slava, We would be interested in reviewing your patch. Can you please provide more details? Is there any other way to disable the partition creation step? Thanks, Chris On 4/13/15 10:59 PM, Slava Markeyev wrote: This is something I've encountered when doing ETL with hive and having it c

Table Lock Manager: ZooKeeper cluster

Hi there, I read on the Hive installation documentation that we need to have a ZooKeeper cluster setup to support Table Lock Manager (Cloudera docs link below). As we have HBase with a ZooKeeper cluster already, my question is if we can use the same ZK cluster for Hive. Is that recommended? What

Hive documentation update for isNull, isNotNull etc.

I'm having major trouble finding documentation on hive functions isNull and isNotNull. At first I was assuming the function just wasn't available, now I believe these functions are not documented. I believe that the LanguageManual+UDF#LanguageManualUDF-Built-inFunctions

Re: Hive documentation update for isNull, isNotNull etc.

I also recently realized that NVL function is available, but not documented :( Dmitry Tolpeko -- PL/HQL - Procedural SQL-on-Hadoop - www.plhql.org On Sat, Apr 18, 2015 at 12:22 AM, Moore, Douglas < douglas.mo...@thinkbiganalytics.com> wrote: > I'm having major trouble finding documentation o

Re: [Hive] Slow Loading Data Process with Parquet over 30k Partitions

I've created HIVE-10385 and attached a patch. Unit tests to come. -Slava On Fri, Apr 17, 2015 at 1:34 PM, Chris Roblee wrote: > Hi Slava, > > We would be interested in reviewing your patch. Can you please provide > more details? > > Is there any other way to disable the partition creation step

Re: Hive documentation update for isNull, isNotNull etc.

Hooray for updating the docs! Douglas, if you tell me your Confluence username I'll grant you write access to the wiki (see About This Wiki ). Thanks. -- Lefty On Fri, Apr 17, 2015 at 6:11 PM