Regarding configuration of Storage based authorization in Hive 0.12

2015-01-16 Thread Jim Green 
Hi Team,

Per the documentation for Hive Storage based authorization in Hive, it has
2 uses:
1. Metastore side authorization
2. Client side authorization

I have some questions because after enabling client side authorization by
adding below configurations in hive-site.xml on client side, I
got NullPointerException when querying talbes.However I can run “show
tables” without issue.

hive.security.authorization.enabled
true


hive.security.authorization.manager

org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider


hive> select * from testorc limit 1;
FAILED: NullPointerException null

>From hive.log:
2015-01-16 14:29:55,157 ERROR ql.Driver (SessionState.java:printError(419))
- FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:173)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:699)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:507)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:351)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:977)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:913)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:903)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)

1. Can we setup SBAP on both client side and also metastore side?
2. If so, which configuration needed on each side?
Per
https://cwiki.apache.org/confluence/display/Hive/Storage+Based+Authorization+in+the+Metastore+Server
,
My understanding is:
Put below 2 parameters in hive-site.xml on metastore side:
hive.metastore.pre.event.listeners
hive.security.metastore.authorization.manager

Per
https://cwiki.apache.org/confluence/display/HCATALOG/Storage+Based+Authorization
, put below 2 parameters in hive-site.xml on client side:
hive.security.authorization.enabled
hive.security.authorization.manager

Am I right?

3. Can I use only client side SBAP or only metastore side SBAP?

4. Any ideas on above error message?

Thanks!


-- 
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)

Re: Writing Hive Query Output to local system

2015-01-16 Thread Sanjay Subramanian 
I am using Cloudera Manager and I know how to add a param thru Safety Valve I 
don't know how to ignore a param thru Safety Valve :-) 
Actually I will put the question on the CDH groups as well...
Any thoughts guys ?
thanks

sanjay
  From: "grimaldi.vince...@gmail.com" 
 To: user@hive.apache.org; Sanjay Subramanian  
 Sent: Friday, January 16, 2015 11:27 AM
 Subject: Re: Writing Hive Query Output to local system
   
In your hive-site.xml remove the  block corresponding to the 
parametereç

hive.metastore.local

That's hit. Is a deprecated parameter and is not necessary anymore.

Regards,



  
||   Vincenzo Grimaldi
 Senior Business Intelligence Consultant   p:00 39 380 52 22 218 | m:00 353 851 
69 84 58 | e:grimaldi.vince...@gmail.com  | a: Aprt. 148A, Smithfield Market, 
Smithfield, Dublin 7, Ireland     |

Get a signature like this:   Click here!  
2015-01-16 19:05 GMT+00:00 Sanjay Subramanian :



hey guys
I recall this did not happen in the days of 0.9.x version
But I use 0.13.x now and when I run a hive query 
hive -e "select * from tablename" > ./myfile.txt
The first line in myfile.txt is as follows
2015-01-16 10:48:13,091 WARN  [main] conf.HiveConf 
(HiveConf.java:initialize(1491)) - DEPRECATED: Configuration property 
hive.metastore.local no longer has any effect. Make sure to provide a valid 
value for hive.metastore.uris if you are connecting to a remote metastore. 
How do I avoid this ?
thanks
sanjay



-- 

||   Vincenzo Grimaldi
 Senior Business Intelligence Consultant   p:00 39 380 52 22 218 | m:00 353 851 
69 84 58 | e:grimaldi.vince...@gmail.com  | a: Aprt. 148A, Smithfield Market, 
Smithfield, Dublin 7, Ireland      |

Re: Writing Hive Query Output to local system

2015-01-16 Thread grimaldi.vince...@gmail.com 
In your hive-site.xml remove the  block corresponding to the
parametereç

hive.metastore.local

That's hit. Is a deprecated parameter and is not necessary anymore.

Regards,



  [image: photo]
*Vincenzo Grimaldi*
Senior Business Intelligence Consultant
 p:00 39 380 52 22 218 | m:00 353 851 69 84 58 | e:
grimaldi.vince...@gmail.com | a: Aprt. 148A, Smithfield Market, Smithfield,
Dublin 7, Ireland
 

   Get a signature like this:

Click
here!


2015-01-16 19:05 GMT+00:00 Sanjay Subramanian :

> hey guys
>
> I recall this did not happen in the days of 0.9.x version
>
> But I use 0.13.x now and when I run a hive query
>
> hive -e "select * from tablename" > ./myfile.txt
>
> The first line in myfile.txt is as follows
>
> 2015-01-16 10:48:13,091 WARN  [main] conf.HiveConf
> (HiveConf.java:initialize(1491)) - DEPRECATED: Configuration property
> hive.metastore.local no longer has any effect. Make sure to provide a valid
> value for hive.metastore.uris if you are connecting to a remote metastore.
>
> How do I avoid this ?
>
> thanks
>
> sanjay
>



-- 
[image: photo]
*Vincenzo Grimaldi*
Senior Business Intelligence Consultant
 p:00 39 380 52 22 218 | m:00 353 851 69 84 58 | e:
grimaldi.vince...@gmail.com | a: Aprt. 148A, Smithfield Market, Smithfield,
Dublin 7, Ireland

Writing Hive Query Output to local system

2015-01-16 Thread Sanjay Subramanian 
hey guys
I recall this did not happen in the days of 0.9.x version
But I use 0.13.x now and when I run a hive query 
hive -e "select * from tablename" > ./myfile.txt
The first line in myfile.txt is as follows
2015-01-16 10:48:13,091 WARN  [main] conf.HiveConf 
(HiveConf.java:initialize(1491)) - DEPRECATED: Configuration property 
hive.metastore.local no longer has any effect. Make sure to provide a valid 
value for hive.metastore.uris if you are connecting to a remote metastore. 
How do I avoid this ?
thanks
sanjay

Hive UPDATE : query around usage

2015-01-16 Thread Devopam Mittra 
hi All,
I wanted to UPDATE multiple rows in a table by joining it with another
table for identification of matching records (otherwise in the RDBMS world
also commonly known as SET based UPDATE).

The idea is to update multiple records in one go , to downsize the lead
time for each MR job to kick off.

Is it supported or I am being greedy here with my wishlist ?

Also, at an atomic level update as well, is there a way I can update a
table without providing explicit variables (aka expression)  in the WHERE
clause.
e.g.: I want to update one row in a table and in the where clause i wish to
use a SELECT with other table to identify / match the record.

Kindly drop a pointer to the syntax manual to look it up.

Happy weekend !
-- 
Devopam Mittra
Life and Relations are not binary

Re: date_add statement problem

2015-01-16 Thread Jason Dere 
Devopam is right, your date format looks wrong (should be -mm-dd rather 
than /mm/dd.

Casting should work, trying this with Hive trunk:

hive> select c1, date_add('1985-01-01', c1) from short1;
FAILED: SemanticException [Error 10014]: Line 1:11 Wrong arguments 'c1':  
DATE_ADD() only takes INT types as second  argument, got SHORT
hive> select c1, date_add('1985-01-01', cast(c1 as int)) from short1;
OK
484 1986-04-30
98  1985-04-09
278 1985-10-06
255 1985-09-13
409 1986-02-14
165 1985-06-15
27  1985-01-28
311 1985-11-08
86  1985-03-28
238 1985-08-27
Time taken: 0.044 seconds, Fetched: 10 row(s)

But the UDF is a bit restrictive here, we could definitely make 
date_add()/date_sub() accept tinyint/smallint/bigint here. I've created 
https://issues.apache.org/jira/browse/HIVE-9396 for this.

On Jan 16, 2015, at 1:09 AM, Devopam Mittra  wrote:

> hi Yashwanth,
> Can you please check once if  DATE_ADD('1985-01-01', day_key) works for you 
> 
> regards
> Devopam
> 
> 
> On Fri, Jan 16, 2015 at 1:12 PM, Yashwanth M  wrote:
> Hi,
> 
>  
> 
> I’ve been facing some issues in using date_add()
> 
>  
> 
> I have column ‘day_key’  that is short integer and stores day keys with 
> respect to 1985/01/01.
> 
>  
> 
> I need to extract actual date from this column.
> 
>  
> 
> Date_add(‘1985/01/01’,day_key)
> 
> However I get an error saying the second argument needs integer.
> 
>  
> 
> I also did
> 
> Date_add(‘1985/01/01’,cast(day_key as int))
> 
> Resulted in an error saying the column is short.
> 
>  
> 
> Please help me with this.
> 
>  
> 
> Thanks
> 
> Regards,
> 
> Yashwanth M | +91 8884 177 884 |www.mu-sigma.com |
> 
>  
> 
> Disclaimer: http://www.mu-sigma.com/disclaimer.html
> 
> 
> 
> -- 
> Devopam Mittra
> Life and Relations are not binary


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: date_add statement problem

2015-01-16 Thread Devopam Mittra 
hi Yashwanth,
Can you please check once if  DATE_ADD('1985-01-01', day_key) works for you

regards
Devopam


On Fri, Jan 16, 2015 at 1:12 PM, Yashwanth M 
wrote:

>  Hi,
>
>
>
> I’ve been facing some issues in using date_add()
>
>
>
> I have column ‘*day_key*’  that is short integer and stores day keys with
> respect to 1985/01/01.
>
>
>
> I need to extract actual date from this column.
>
>
>
> *Date_add(‘1985/01/01’,day_key)*
>
> However I get an error saying the second argument needs integer.
>
>
>
> I also did
>
> *Date_add(‘1985/01/01’,cast(day_key as int))*
>
> Resulted in an error saying the column is short.
>
>
>
> Please help me with this.
>
>
>
> Thanks
>
> Regards,
>
> Yashwanth M *|* +91 8884 177 884 *|*www.mu-sigma.com *|*
>
>
>  Disclaimer: http://www.mu-sigma.com/disclaimer.html
>



-- 
Devopam Mittra
Life and Relations are not binary