[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547890#comment-13547890 ] Hudson commented on HIVE-2764: -- Integrated in Hive-trunk-hadoop2 #54 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/54/]) HIVE-2764: Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311418 Files : * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.9.0 Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2764.D2205.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2764.D2205.2.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2764.D2205.3.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2764.D2205.4.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2764.D2205.5.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250410#comment-13250410 ] Hudson commented on HIVE-2764: -- Integrated in Hive-trunk-h0.21 #1364 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1364/]) HIVE-2764: Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311418 Files : * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.9.0 Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764.D2205.5.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247075#comment-13247075 ] Ashutosh Chauhan commented on HIVE-2764: Those tests passed. So, thats a progress. But following new ones failed. * rcfile_merge3.q * rcfile_createas1.q Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246816#comment-13246816 ] Phabricator commented on HIVE-2764: --- ashutoshc has commented on the revision HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup. Changes look good. But, I see plenty of failures while running tests. REVISION DETAIL https://reviews.facebook.net/D2205 Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246820#comment-13246820 ] Phabricator commented on HIVE-2764: --- enis has commented on the revision HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup. Strange, all of the tests passed for me except for one. I rerun that and it passed as well. Can you rerun the tests or post the logs ? REVISION DETAIL https://reviews.facebook.net/D2205 Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246830#comment-13246830 ] Ashutosh Chauhan commented on HIVE-2764: In TestCliDriver, following tests failed: * alter_concatenate_indexed_table.q * alter_merge.q * alter_merge_2.q * alter_merge_stats.q * concatenate_inherit_table_location.q * create_merge_compressed.q * escape2.q I re-ran them again individually, and they again failed. Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245921#comment-13245921 ] Phabricator commented on HIVE-2764: --- ashutoshc has commented on the revision HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 Sorry, I wasn't clear. What I meant was we should remove setting of OF class inside setNullOF() method, because its redundant as we set it again later in this method. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 Yes, please use reflection to check if method is there and then only call. REVISION DETAIL https://reviews.facebook.net/D2205 BRANCH hbase-dt-of Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245964#comment-13245964 ] Phabricator commented on HIVE-2764: --- enis has commented on the revision HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 Hadoop framework calls the getRR() although we do not emit any records. So we need some kind of an empty implementation. REVISION DETAIL https://reviews.facebook.net/D2205 BRANCH hbase-dt-of Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225683#comment-13225683 ] Phabricator commented on HIVE-2764: --- enis has commented on the revision HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup. Some comments about the patch: Changing the StorageHandler.configureJobProperties() to be called with the actual job is kind of intrusive, so I have given up on that approach, and go with the plan of obtaining the delegation tokens from the IF.getSplits() and OF.checkOutputSpecs() time. However, one problem with this approach is that although on the input side it works fine, on the output side, Hive does not use OF's properly. So, this patch adds a HiveOutputFormatImpl class, and changes the HOF to extend OF. The actual usage of OFs are not changed as a part of this patch, since it is out of the scope of this issue. However, not the HOFImpl class delgates the checkOutputSpecs() call to the underlying OutputFormat, so that if the OF wants to do any initialization, it can. REVISION DETAIL https://reviews.facebook.net/D2205 Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222862#comment-13222862 ] Enis Soztutar commented on HIVE-2764: - We had some discussion with Ashutosh about how to best approach this. As a part of this issue, we have to obtain delegation tokens from the storage handler (hbase in this case), and then pass this to the submitted job. However, HiveStorageHandler.configureTableJobProperties() does not expose the JobConf object, or an interface to configure the job's credentials. There seems a couple of options we can go for: 1. At HiveStorageHandler.configureTableJobProperties(), create and save the job credentials somewhere, and at the time of job submit, merge all the credentials before submitting the job. 2. Change the HiveStorageHandler.configureTableJobProperties() to get the actual Job to be submitted, so that Storage handlers have a real shot at configuring the job. 3. Add a method like HiveStorageHandler.configureJobCredentials(), and call it from HIF.getSplits() and HiveHBaseTableOutputFormat.checkOutputSpecs(). Number 1 and 3 seems hackish, and we will want to give the storagehandlers ability to configure the actual job. So we are leaning towards that. But this will also be merged with HIVE-2773. I am still trying to make this work, so any comments / suggestions are more than welcome. Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222994#comment-13222994 ] Francis Liu commented on HIVE-2764: --- Having making getCredentials() exposed as a TableDesc member sounds ok to me as it really is a special property of JobConf. Though it breaks the expectation of tableDesc being readonly and jobProperties being the only one mutable for configureTableProperties(). I wonder if we can remove jobProperties altogether and use tableDesc.jobProperties to set properties. As tableDesc.jobProperties is always a new HashMap when configureTableProperties() is called? Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
