[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916624#action_12916624
 ] 

Ashutosh Chauhan commented on HIVE-1264:


@Todd,

Any progress on this? Some of the utility methods introduced in this patch will 
be useful for HIVE-1667

> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264-fb-mirror.txt, hive-1264.txt, 
> HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914573#action_12914573
 ] 

John Sichi commented on HIVE-1264:
--

Todd, I'm getting one test failure when running ant test.  Probably 
hbase-handler/build.xml needs a fix.  You should be able to repro it with

ant test -Dtestcase=TestHBaseMinimrCliDriver


testCliDriver_hbase_bulkError   org/apache/hadoop/hdfs/MiniDFSCluster 

java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/MiniDFSCluster
at org.apache.hadoop.hive.shims.Hadoop20Shims.getMiniDfs(Hadoop20Shims.java:90)
at org.apache.hadoop.hive.ql.QTestUtil.(QTestUtil.java:224)
at org.apache.hadoop.hive.hbase.HBaseQTestUtil.(HBaseQTestUtil.java:30)
at 
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.setUp(TestHBaseMinimrCliDriver.java:43)
at junit.framework.TestCase.runBare(TestCase.java:125)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.extensions.TestSetup.run(TestSetup.java:23)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hdfs.MiniDFSCluster
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)

> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264-fb-mirror.txt, hive-1264.txt, 
> HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914306#action_12914306
 ] 

John Sichi commented on HIVE-1264:
--

+1.  Will commit when tests pass.


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264-fb-mirror.txt, hive-1264.txt, 
> HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913886#action_12913886
 ] 

John Sichi commented on HIVE-1264:
--

Mirror is up; Todd, could you test it and then update the patch with the 
location?

http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hadoop-0.20.3-CDH3-SNAPSHOT


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912686#action_12912686
 ] 

John Sichi commented on HIVE-1264:
--

Todd has added the .md5/.asc files; I'm working on getting the mirror set up.


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[HBase Review Board (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910389#action_12910389
 ] 

HBase Review Board commented on HIVE-1264:
--

Message from: "Carl Steinbach" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/860/#review1251
---

Ship it!


+1 Looks good to me.


build.properties


If this is the convention going forward then it's probably more appropriate 
to rename the old style as "oldstyle-name" instead of introducing a 
"newstyle-name".


- Carl





> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[HBase Review Board (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910381#action_12910381
 ] 

HBase Review Board commented on HIVE-1264:
--

Message from: "Todd Lipcon" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/860/
---

Review request for Hive Developers and John Sichi.


Summary
---

Adds a shim layer for secure Hadoop, currently pulling a secure CDH3b3 
prerelease snapshot


This addresses bug HIVE-1264.
http://issues.apache.org/jira/browse/HIVE-1264


Diffs
-

  build-common.xml 0b76688 
  build.properties 3e392f7 
  build.xml 4b345b5 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2c7123 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java b2966de 
  shims/build.xml b339871 
  shims/ivy.xml de56e4f 
  shims/src/0.17/java/org/apache/hadoop/hive/shims/Hadoop17Shims.java 17110ab 
  shims/src/0.18/java/org/apache/hadoop/hive/shims/Hadoop18Shims.java 9cc8d56 
  shims/src/0.19/java/org/apache/hadoop/hive/shims/Hadoop19Shims.java c643108 
  shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 0675a79 
  shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 
PRE-CREATION 
  shims/src/0.20S/java/org/apache/hadoop/hive/shims/Jetty20SShims.java 
PRE-CREATION 
  shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 4310942 
  shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java c847d69 

Diff: http://review.cloudera.org/r/860/diff


Testing
---

Able to run MR jobs on our secure cluster with standalone (ie no separate 
metastore, etc)


Thanks,

Todd




> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910377#action_12910377
 ] 

John Sichi commented on HIVE-1264:
--

See here for existing checksum file conventions:

http://mirror.facebook.net/facebook/hive-deps/hadoop/core/


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910376#action_12910376
 ] 

John Sichi commented on HIVE-1264:
--

OK, can you add a checksum file to your directory, and then I'll ask our ops to 
create the mirror?  Once that's done, we'll need one more patch which 
references the FB location as default.


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910373#action_12910373
 ] 

Todd Lipcon commented on HIVE-1264:
---

Submitted to RB: https://review.cloudera.org/r/860/

Regarding the snapshot - it's fine by me to pull from there, I think the 
people.apache.org web server is reasonably stable. If it turns out to be flaky 
it's also cool if you want to mirror it - FB is probably more reliable than ASF.

> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910346#action_12910346
 ] 

John Sichi commented on HIVE-1264:
--

Hey Todd,

I think you mentioned at the contributor meetup the other day that this is 
ready for review.  If so, click the Submit Patch button and create a 
reviewboard entry.

I just did a quick check to verify that I could apply the patch and run ant 
package+test (without changing hadoop.version) and ivy was able to fetch the 
dependency from your snapshot successfully.

If we commit it as is, every hive developer is going to automatically start 
pulling from that snapshot by default.  Is that OK, or should I try to get a 
copy hosted at http://mirror.facebook.net/facebook/hive-deps?


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905156#action_12905156
 ] 

Todd Lipcon commented on HIVE-1264:
---

(btw, if you want to test this against a different tarball, you can use 
{{-Dhadoop.security.url=http://url/of/your/tarball 
-Dhadoop.security.version=0.20.104 or whatever.}})

> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Jeff Hammerbacher
>Assignee: Todd Lipcon
> Attachments: hive-1264.txt, HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[Ashish Thusoo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892517#action_12892517
 ] 

Ashish Thusoo commented on HIVE-1264:
-

Can these changes be packed in the shims layer. So all the calls can be 
replaced with a call to shims with the shim for 20.1xx doing the right thing. 

> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Jeff Hammerbacher
>Assignee: Venkatesh S
> Attachments: HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[Venkatesh S (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891002#action_12891002
 ] 

Venkatesh S commented on HIVE-1264:
---

This will work against 20.1xx branch. You need to include the 20.1xx hadoop 
dependency and it does compile and run. The interface contract does not change 
and hence not sure if I need to change the shim. UGI has changed in 20S and 
UnixUGI class is no more.

Please suggest how to proceed with this incompatible change.

> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Jeff Hammerbacher
> Attachments: HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1264) Make Hive work with Hadoop security

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890877#action_12890877
 ] 

Carl Steinbach commented on HIVE-1264:
--

@Venkatesh: I looked at your patch. Your changes break the build when compiled 
against any version of Hadoop that does not include the new security APIs. Hive 
is currently designed to maintain backward compatibility with Hadoop 0.17, 
0.18, 0.19 and 0.20 using a shim layer. In order to get this patch committed 
you will need to modify the shim layer and ensure that you have not broken 
compatibility with any of these older versions of Hadoop.


> Make Hive work with Hadoop security
> ---
>
> Key: HIVE-1264
> URL: https://issues.apache.org/jira/browse/HIVE-1264
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Jeff Hammerbacher
> Attachments: HiveHadoop20S_patch.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.