[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918689#action_12918689 ] Pradeep Kamath commented on HIVE-842: - Hey Todd, I did the changes you mentioned and got it to compile. While trying to test it out I had to run the metastore as user whose keytab file only had a user principal and not a service principal - so I hacked the code in the patch a little to not check if the principal had the service/h...@realm structure and I hardcoded the host name into the calls. With all these machinations I got the server to run and tried running show tables and got the following with loglevel DEBUG (on the client side): javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:95) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:254) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38) Do you think this is because I don't have a service principal in the keytab used by the metastore? Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: hive-842.txt, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918745#action_12918745 ] Todd Lipcon commented on HIVE-842: -- Hey Pradeep. It sounds like it might be - I haven't seen that error before, but I also have only been testing with actual service principals (ie principals of the type metastore/hostname). You can try running both sides with HADOOP_OPTS=-Dsun.security.krb5.debug=true and it should give you some extra details. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: hive-842.txt, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[
https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918072#action_12918072
]
Pradeep Kamath commented on HIVE-842:
-
I tried applying this patch after applying the patch for HIVE-1264 and got the
following compile errors which seem to suggest I am missing some jar (seems
thrift related) - any pointers on how to resolve these errors?
{noformat}
build_shims:
[echo] Compiling shims against hadoop 0.20.104.3.1007202301
(/tmp/hive-svn/build/hadoopcore/hadoop-0.20.104.3.1007202301)
[javac] Compiling 8 source files to /tmp/hive-svn/build/shims/classes
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/thrift/transport/TSaslTransport.java:109:
cannot find symbol
[javac] symbol : class TMemoryInputTransport
[javac] location: class org.apache.thrift.transport.TSaslTransport
[javac] private TMemoryInputTransport readBuffer = new
TMemoryInputTransport();
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:354:
cannot find symbol
[javac] symbol : method getBuffer()
[javac] location: class org.apache.thrift.transport.TTransport
[javac] return wrapped.getBuffer();
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:352:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:359:
cannot find symbol
[javac] symbol : method getBufferPosition()
[javac] location: class org.apache.thrift.transport.TTransport
[javac] return wrapped.getBufferPosition();
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:357:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:364:
cannot find symbol
[javac] symbol : method getBytesRemainingInBuffer()
[javac] location: class org.apache.thrift.transport.TTransport
[javac] return wrapped.getBytesRemainingInBuffer();
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:362:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:369:
cannot find symbol
[javac] symbol : method consumeBuffer(int)
[javac] location: class org.apache.thrift.transport.TTransport
[javac] wrapped.consumeBuffer(len);
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java:367:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/thrift/transport/TSaslTransport.java:109:
cannot find symbol
[javac] symbol : class TMemoryInputTransport
[javac] location: class org.apache.thrift.transport.TSaslTransport
[javac] private TMemoryInputTransport readBuffer = new
TMemoryInputTransport();
[javac] ^
[javac]
/tmp/hive-svn/shims/src/0.20S/java/org/apache/thrift/transport/TSaslTransport.java:352:
cannot find symbol
[javac] symbol : method encodeFrameSize(int,byte[])
[javac] location: class org.apache.thrift.transport.TFramedTransport
[javac] TFramedTransport.encodeFrameSize(length, lenBuf);
[javac] ^
[javac] Note:
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note:
/tmp/hive-svn/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 11 errors
{noformat}
Authentication Infrastructure for Hive
--
Key: HIVE-842
URL: https://issues.apache.org/jira/browse/HIVE-842
Project: Hadoop Hive
Issue Type: New Feature
Components: Server Infrastructure
Reporter: Edward Capriolo
Assignee: Todd Lipcon
Attachments: hive-842.txt, HiveSecurityThoughts.pdf
This issue deals with the authentication (user name,password) infrastructure.
Not the authorization
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918076#action_12918076 ] Todd Lipcon commented on HIVE-842: -- Hey Pradeep. You also need HIVE-1526 which updates Hive to use Thrift 0.4.0. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: hive-842.txt, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[
https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918085#action_12918085
]
Pradeep Kamath commented on HIVE-842:
-
Hey Todd, I applied the patches in the following sequence on current hive
trunk:
hive-1264.txt, hive-842.txt and then HIVE-1526.2.patch.txt. The last one didn't
apply cleanly for
ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java - so I
manually edited it based on the reject file. After that, I get the following
compile error:
[javac] Compiling 607 source files to /tmp/hive-svn/build/ql/classes
[javac]
/tmp/hive-svn/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java:384:
cannot find symbol
[javac] symbol : class StageType
[javac] location: class org.apache.hadoop.hive.ql.exec.MapRedTask
[javac] public StageType getType() {
[javac] ^
[javac]
/tmp/hive-svn/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java:385:
cannot find symbol
[javac] symbol : variable StageType
[javac] location: class org.apache.hadoop.hive.ql.exec.MapRedTask
[javac] return StageType.MAPREDLOCAL;
[javac]^
[javac]
/tmp/hive-svn/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java:214:
getType() in org.apache.hadoop.hive.ql.exec.StatsTask cannot override getType()
in org.apache.hadoop.hive.ql.exec.Task; attempting to use incompatible return
type
[javac] found : int
[javac] required: org.apache.hadoop.hive.ql.plan.api.StageType
[javac] public int getType() {
[javac] ^
[javac]
/tmp/hive-svn/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java:215:
cannot find symbol
[javac] symbol : variable STATS
[javac] location: class org.apache.hadoop.hive.ql.plan.api.StageType
[javac] return StageType.STATS;
[javac] ^
[javac]
/tmp/hive-svn/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java:213:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
Authentication Infrastructure for Hive
--
Key: HIVE-842
URL: https://issues.apache.org/jira/browse/HIVE-842
Project: Hadoop Hive
Issue Type: New Feature
Components: Server Infrastructure
Reporter: Edward Capriolo
Assignee: Todd Lipcon
Attachments: hive-842.txt, HiveSecurityThoughts.pdf
This issue deals with the authentication (user name,password) infrastructure.
Not the authorization components that specify what a user should be able to
do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918239#action_12918239 ] Todd Lipcon commented on HIVE-842: -- Seems like the patch that updates Thrift has fallen out of date with trunk. I'll try to regenerate it ASAP. You can probably fix the above issues by (a) importing StageType in MapRedTask, and (b) replacing StatsTask.getType's return with the StageType enum. (the new version of Thrift uses java enums instead of ints to represent thrift enums) Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: hive-842.txt, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12916687#action_12916687 ] Todd Lipcon commented on HIVE-842: -- bq. should there be an option whereby the metastore uses a keytab to authenticate to HDFS, but doesn't require users to authenticate to it? bq. Wouldn't this leave a hole as it currently exists? Yea - I think the use case is that you may have some old Thrift clients that haven't yet been updated to work with the SASL implementation (eg PHP). For those clients, perhaps you can provide security based on firewall rules, etc. But you would still like to run Hive on top of a secured HDFS. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12914408#action_12914408 ] Venkatesh S commented on HIVE-842: -- Should the metastore always take HDFS actions as the user making the RPC? Yes, metastore will run as a super-user (Hadoop proxy user) enabling DO AS operations and impersonate the target user while accessing data on HDFS. If we see that Hadoop Security is enabled, should we enable SASL on the metastore thrift server by default? I'd think so. should there be an option whereby the metastore uses a keytab to authenticate to HDFS, but doesn't require users to authenticate to it? Wouldn't this leave a hole as it currently exists? Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913439#action_12913439 ] Todd Lipcon commented on HIVE-842: -- As discussed at the last contributor meeting, I am working on authenticating access to the metastore by kerberizing the Thrift interface. Plan is currently: 1) Update the version of Thrift in Hive to 0.4.0 2) Temporarily check in the SASL support from Thrift trunk (this will be in 0.5.0 release, due out in October some time) 3) Build a bridge between Thrift's SASL support and Hadoop's UserGroupInformation classes. Thus, if a user has a current UGI on the client side, it will get propagated to the JAAS context on the handler side. 4) In places where the metastore accesses the file system, use the proxy user functionality to act on behalf of the authenticated user. 5) When we detect that we are running on secure hadoop with security enabled, enable the above functionality. I'd like to attack the Hive Web UI separately. One open question: - Do Hive *tasks* ever need to authenticate to the metastore? If so, we will have to build a delegation token system into Hive. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913691#action_12913691 ] Todd Lipcon commented on HIVE-842: -- OK. The code in Hadoop Common is somewhat reusable for this, so it shouldn't be too hard to implement. If I recall correctly, though, the delegation tokens rely on a secret key that the master daemon periodically rotates. We need to add some kind of persistent token storage for this to work - I guess in the metastore's DB? To make this easier to review, I'd like to do the straight kerberos first, and then add delegation tokens in a second patch/JIRA. Sound good? Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913706#action_12913706 ] Venkatesh S commented on HIVE-842: -- Sounds good to me. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913741#action_12913741 ] Edward Capriolo commented on HIVE-842: -- By attack the Web UI separately what is meant? Will it be broken or non-functional at any phase here? That is what I find happens often, some of it is really the WUI's fault for using JSP and not servlets, but there is no simple way to code cover the wui and all the different ways its gets broken. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913787#action_12913787 ] Todd Lipcon commented on HIVE-842: -- I don't anticipate breaking the web UI (or anything) on non-secure Hadoop versions. But it will probably be insecure to run the web UI, which currently trusts users to say who they want to be - i.e I don't plan in the short term to integrate an auth layer for the web UI itself. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12765362#action_12765362 ] Min Zhou commented on HIVE-842: --- @Edward Kerberos for authethication is a good way I think, user/password is no need here. This issue would be implemented in the future. btw, we've finished the development of authorization infrastructure for Hive. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hadoop Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Edward Capriolo This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
