Re: Hive Metastore Bottleneck

Is the MySQL database virtualized? Bottlenecks to storage of the MySQL database? Network could be a bottleneck? Firewalls blocking new connections in case of a sudden connection increase? > On 30 Mar 2016, at 23:28, Udit Mehta wrote: > > Hi all, > > We are currently running Hive in productio

?????? Hive2.0+Hadoop-2.3.0-cdh5.0 encounted KryoException "Encounteredunregistered class ID: 21"

Thank you for reply! I use default execution engine "mr" (hive.execution.engine=mr) to run HQL, and Hive 0.13 works fine, it also deserializing MapWork plan via Kryo. -- -- ??: "Mich Talebzadeh";; : 2016??3??31??(??) 0:32 ??: "u

Re: Hive Metastore Bottleneck

Mich, Thanks for clarifying. My words were probably misleading :-) Udit, Fallback (or HA) is different from load balancing. Having load issues need not mean it is unreachable, so your second service. You want equal load on both metastore services so a simple Round Robin load balancer sho

Re: Hive Metastore Bottleneck

But dont the clients always pick the first URI for multiple instances mentioned in "*hive.metastore.uris" *config and fallback to the others only if the first is unreachable? This way, we would still have a bottleneck, right? Can you give a little more information on your setup and how you enable l

Re: Hive Metastore Bottleneck

Hi Gautam When you stated "Have you tried putting multiple metastores behind a load balancer" Should read "Have you tried putting multiple *metastore services* Basically there is only one backend database AKA metastore. Unless you have set up bi-directional replication on the database, t

Re: Hive Metastore Bottleneck

The metastore service is a java process that is a thrift server .. so you can point multiple such hive metastore instances with "javax.jdo.option.ConnectionURL" poitning to the same mysql db. On Wed, Mar 30, 2016 at 3:11 PM, Mich Talebzadeh wrote: > > > Can you clarify this please > > "Have you

Re: Hive Metastore Bottleneck

Can you clarify this please "Have you tried putting multiple metastores behind a load balancer" Are you implying that metastore and backend DB are different entities here. As far as I know $HIVE_HOME/bin/hive --service metastore & starts Hive threads to the backend database/metastore and Hive se

Re: Hive Metastore Bottleneck

I was looking at : *hive**.metastore.max.server.threads *but reading more into it tells me its a config for the thrift server and not the metastore. Most of our applications accessing the metastore are Spark Sql applications which do INSERT operations on multiple partitions on a hourly basis. This

Re: Hive Metastore Bottleneck

Can you elaborate on where you see the bottleneck? A general overview of your access path would be useful. For instance if you'r accessing Hive metastore via HiveServer2 or from webhcat using embedded cli or something else. Have you tried putting multiple metastores behind a load balancer? It's

Re: Hive Metastore Bottleneck

Are you talking about increase in number of threads from Hive server2 connection to your database (MySQL)? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Hive Metastore Bottleneck

Hi all, We are currently running Hive in production and staging with the metastore connecting to a MySql database in the backend. The traffic in production accessing the metastore is more than staging which is expected. We have had a sudden increase in traffic which has led to the metastore operat

Re: Best way of Unpivoting of hiva table data. Any Analytic function for unpivoting

From mytable Select id, 'mycol' as name, col1 as value Union Select id, 'mycol2' as name, col2 as value Something like this might work for you? Cheers, Andrew On Mon, Mar 28, 2016 at 7:53 PM, Ryan Harris < ryan.har...@zionsbancorp.com [ryan.har...@zionsbancorp.com] > wrote: collect_list(col)

Re: Escaped Quotes

That's what I meant - I apologize for the poor wording. I am using JDBC to programmatically interact with Hive, and then also using beeline on the command line for quick testing, and seeing the behavior in both cases. I could have sworn that earlier testing that I had done using double-quotes arou

Re: Escaped Quotes

Curious to know why use JDBC and beeline to insert into Hive table rather than using Hive sql through beeline create table testme(col1 varchar(30)); INSERT INTO testme VALUES ("Dougs' Diner"); 0: jdbc:hive2://rhes564:10010/default> select * from testme; +---+--+ | testme.col1 | +

Re: Hive_primary_key

Thank you for the quick reply. I will track it there. On Wednesday, March 30, 2016, Hari Sivarama Subramaniyan < hsubramani...@hortonworks.com> wrote: > This is an on-going work, can be tracked under the tasks in > https://issues.apache.org/jira/browse/HIVE-13076​ > > > Thanks > > Hari >

Re: Hive_primary_key

This is an on-going work, can be tracked under the tasks in https://issues.apache.org/jira/browse/HIVE-13076? Thanks Hari From: Ajay Chander Sent: Wednesday, March 30, 2016 11:33 AM To: user@hive.apache.org Subject: Hive_primary_key Hi Users, Just wanted to

Hive_primary_key

Hi Users, Just wanted to check if the support for defining primary keys(constraints) while creating a hive external tables is available yet? Thank you!

Escaped Quotes

Hi, I use JDBC and the beeline client to interact with a Hive 1.2.1 store. I have some string data that contains single quotes (as well as other special characters) that requires being escaped in order to be inserted into my tables. For example, the string "Doug's Diner": INSERT INTO my_table (s

Re: Hive2.0+Hadoop-2.3.0-cdh5.0 encounted KryoException "Encountered unregistered class ID: 21"

Hive 2 works fine. I don't use any vendor specific build but it should work pretty easy. What engine (mr, spark, TZ) have you specified in hive-site.xml. Kind of reminds me some error I saw in the past. Try to set hive.execution.engine=mr and run it again. HTH Dr Mich Talebzadeh LinkedIn *

unsubscribe

Unsubscribe

Extra ZEROS getting generated in Target table

Hey folks, I am executing INSERT INTO SELECT hive query through oozie workflow. Below are the details. Extra zeros are getting generated in target table(parquet_table). Can anyone help me to identify the issue. Ex. Value of COLUMN_SAMPLE in source table(tab1) : 1 Value of COLUMN_SAMPLE in dest

Hive2.0+Hadoop-2.3.0-cdh5.0 encounted KryoException "Encountered unregistered class ID: 21"

Hi,experts: When I use Hive 2.0 and Hadoop-2.3.0-cdh5.0 to run HQL like "SELECT COUNT(1) FROM myTable", and encounted this exception below??(by the way,the configuration hive.plan.serialization.format was removed in Hive2.0) 2016-03-30 16:30:42,799 INFO [main] org.apache.hadoop.hive.ql.exec.Ut

Re: Mechanism when doing a select *

Hello guys ! Just a quick thank you again for your answer on this topic ! I noticed that when a job is performed (if the table is bigger than hive.fetch.task.conversion.threshold), then it seems that temporary files are created in HDFS (in /tmp). If I understood well, the select * is divided into

High-Availbility of HiveServer2 and different transport modes

Hello ! I send you this mail because I have a cluster with multiple servers, with among them four edge servers hosting each one of them an hiveserver2 : On two of them, there are HiveServer2 using HTTP as transport mode. On the other two, there are HiveServer2 using Thrift as transport mode. My