hive on spark

HI All, I am trying to rum hive on spark means from Hive terminal setting up execution engine as spark. I have copied the hive-default.xml to spark conf directory. Hive is not able to find the table.giving me error "table_name" not found? Can you help me with exact steps how to make spark as a e

Re: HiveMetaStoreClient

Hi Subroto, thanks for the suggestion, so I've read the api and looks like setVar is the better way to go for me so I did this HiveConf conf = new HiveConf(); conf.setVar(HiveConf.ConfVars.METASTOREURIS, "some-uri"); return new HiveMetaStoreClient(conf); but I keep getting ` error: cannot access

Re: UDF Configure method not getting called

I don't think I understand your use case very well. Would you be able to give a bit more detail on what you are trying to do here, and how your UDF was accomplishing that? Maybe someone might have a suggestion. From: Rahul Sharma Sent: Wednesday, August 26, 20

Re: HiveServer2 & Kerberos

I think the best way would be to parse the URL and see if its Kerberos enabled and then not ask for username and password. However, I doubt it is that trivial. Case in point - lets say SQLBasedAuthorization is being used, then which user will the authorization requests be checked against? - Use

Re: HiveServer2 & Kerberos

On Wed, Aug 26, 2015 at 4:53 PM, kulkarni.swar...@gmail.com < kulkarni.swar...@gmail.com> wrote: > > my understanding is that after using kerberos authentication, you > probably don’t need the password. > > That is not an accurate statement. Beeline is a JDBC client as compared to > Hive CLI which

RE: HiveServer2 & Kerberos

The prompts are a waste of time when kerberised - you can just hit enter twice at them once you have a ticket, so what's the point? I think the JIRA is valid (but if I recall correctly it is also a duplicate of an existing one). Sent from my Windows Phone From: R

Re: HiveMetaStoreClient

Try something like: HiveConf hiveConf; hiveConf=new HiveConf(); hiveConf.addResource("/etc/hive/conf/hive-default.xml"); HiveMetaStoreClient hiveClient; try { hiveClient = new HiveMetaStoreClient(hiveConf); // use the client; } catch (Exception exp) { exp.printStackTrace(); } On Wed, Aug 26, 2015

RE: Loading multiple file format in hive

You'll want to keep an eye on HIVE-9490 [Parquet] Support Alter Table/Partition Concatenate This will be the "correct" way of merging the small files. From: Jeetendra G [mailto:jeetendr...@housing.com] Sent: Wednesday, August 26, 2015 4:48 AM To: user@hive.apache.org Subject: Re: Loading multipl

Re: HiveMetaStoreClient

Thanks all, that was super quick and helpful! Can someone give me a quick code example of how to create a HiveMetaStoreClient object from configurations in hive-site.xml, or how would I create a HiveMetaStoreClient object given the uri and the credentials to talk to the thrift server? On Wed, Aug

Re: HiveMetaStoreClient

Yes, sorry, I wasn't clear. As others have stated, HiveMetaStoreClient uses the thrift API. So using that should be fine. Alan. Subroto Sanyal August 26, 2015 at 10:07 Hi Jerrick, HiveMetastoreClient will be using thrift communication internally via the connect

Re: HiveMetaStoreClient

Hi Jerrick, I'm sure someone more knowledgeable will correct me if I'm wrong, but I believe that class operates over the thrift api. Nathan From: Jerrick Hoang Sent: Wednesday, August 26, 2015 9:44 AM To: user@hive.apache.org Subject: Re: HiveMetaStoreClien

Re: HiveMetaStoreClient

Hi Jerrick, HiveMetastoreClient will be using thrift communication internally via the connection. IMO It is no different from using thrift api. Cheers, Subroto Sanyal On Wed, Aug 26, 2015 at 6:44 PM, Jerrick Hoang wrote: > Ok, I'm super confused now. The hive metastore is a RDBMS database. I >

Re: HiveMetaStoreClient

Why not use hcatalog web service api? Le mer. 26 août 2015 à 18:44, Jerrick Hoang a écrit : > Ok, I'm super confused now. The hive metastore is a RDBMS database. I > totally agree that I shouldn't access it directly via jdbc. So what about > using this class > http://hive.apache.org/javadocs/r0.

Re: HiveServer2 & Kerberos

Even I (and a few others I know in different orgs) have been confused by the password prompts. So looking at the multiple users using their own credentials to authenticate, would that mean Kerberos is not used for authentication? Only for Authorization? In which case what will the authorization be

Re: HiveMetaStoreClient

Ok, I'm super confused now. The hive metastore is a RDBMS database. I totally agree that I shouldn't access it directly via jdbc. So what about using this class http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html ? Is it something that can

Re: UDF Configure method not getting called

Thanks again Jason. I tried hive.fetch.task.conversion=minimal/none and it ran a map-reduce task and the UDF ran fine. The problem with this approach is that the property needs to be changed in conf.server's hive-site and thus affects every query. A workaround would be to add "hive.fetch.task.conve

Re: HiveMetaStoreClient

Accessing the RDBMS directly isn't recommended. We don't view the table layout as part of the public API and sometimes change things between releases. With the thrift API we at least try to keep it backwards compatible. Alan. Dr Mich Talebzadeh August 26, 2015

Re: HiveServer2 & Kerberos

Nope. Because the credentials are different. You might have multiple users using there own credentials to authenticate themselves but there are only single defined credentials to be used by the metastore server. On Wed, Aug 26, 2015 at 10:58 AM, Loïc Chanel wrote: > I understand the behavior, bu

Re: HiveServer2 & Kerberos

I understand the behavior, but when Kerberos is enabled, isn't that a bit redundant ? Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-08-26 17:53 GMT+02:00 kulkarni.swar...@gmail.com < kulkarni.swar...@gmail.com>: > > my understanding is that after using

Re: HiveServer2 & Kerberos

> my understanding is that after using kerberos authentication, you probably don’t need the password. That is not an accurate statement. Beeline is a JDBC client as compared to Hive CLI which is a thrift client to talk to HIveServer2. So it would need the password to establish that JDBC connection

Re: HiveServer2 & Kerberos

Here it is : https://issues.apache.org/jira/browse/HIVE-11653 Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-08-25 23:10 GMT+02:00 Sergey Shelukhin : > Sure! > > From: Loïc Chanel > Reply-To: "user@hive.apache.org" > Date: Tuesday, August 25, 2015 at

Re: Loading multiple file format in hive

Thanks Nitin and Ryan,this will really help. @Ryan spark streaming can write in the desired format, but since these are basically clickstream events ,and we are hitting with many numbers of small files in HDFC in parquet form. for merging these files and create bigger files in necessary. On

Re: HiveMetaStoreClient

What about using the hcatalog apis? Le mer. 26 août 2015 à 8:27, Jerrick Hoang a écrit : > Hi all, > > I want to interact with HiveMetaStore table from code and was looking at > http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html > , wa

Re: HiveMetaStoreClient

Personally from my experience having your hive metastore on Oracle offers the best concurrency. Oracle locking and concurrency is best suited for metastore to avoid deadlocks and excessive serialisation. On 26/8/2015, "Subroto Sanyal" wrote: >hi Jerrick > >Yes, thats the correct way to interac

Re: HiveMetaStoreClient

hi Jerrick Yes, thats the correct way to interact with HiveMetastore(provided you are only interested to play around metadata and Hive Metastore Service is accessible from the java client). Further, you can try the Hive JDBC driver or Hive Server2 thrift API (couldn't find the documentation for th

Re: HiveMetaStoreClient

What is your hive metastore is on? If it is on an RDBMS, then you can simply use client/JDBC tools to access the metastore. Pretty simple. HTH, On 26/8/2015, "Jerrick Hoang" wrote: >Hi all, > >I want to interact with HiveMetaStore table from code and was looking at >http://hive.apache.org/jav