[jira] [Commented] (HIVE-7890) SessionState creates HMS Client while not impersonating

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119997#comment-14119997
 ] 

Brock Noland commented on HIVE-7890:


Thank you Dong and Prasad! I have committed this to trunk.

 SessionState creates HMS Client while not impersonating
 ---

 Key: HIVE-7890
 URL: https://issues.apache.org/jira/browse/HIVE-7890
 Project: Hive
  Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
 Fix For: 0.14.0

 Attachments: HIVE-7890.2.patch


 In SessionState.start [an instance of the the HMSClient is 
 created|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L367].
  When impersonation is enabled, this call does not occur within a doas call 
 and thus the HMSClient is created as the server user, not the impersonated 
 user.
 Thus calls to the HMS are made by the hive user as opposed to the end user. 
 This causes file ownership such as a database directory owner to be 
 incorrect. While debugging this, I got stack trace below. As you can see we 
 are calling getMSC without a doas.
 {noformat}
   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2474)
   at 
 org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:367)
   at 
 org.apache.hive.service.cli.session.HiveSessionImpl.init(HiveSessionImpl.java:121)
   at 
 org.apache.hive.service.cli.session.HiveSessionImplwithUGI.init(HiveSessionImplwithUGI.java:49)
   at 
 org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:130)
   at 
 org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:163)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:290)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:208)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1313)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1298)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
   at 
 org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7890) SessionState creates HMS Client while not impersonating

[Dong Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118044#comment-14118044
 ] 

Dong Chen commented on HIVE-7890:
-

The patch looks good! Thanks. It ensure MSC is created as an impersonated user.

 SessionState creates HMS Client while not impersonating
 ---

 Key: HIVE-7890
 URL: https://issues.apache.org/jira/browse/HIVE-7890
 Project: Hive
  Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-7890.2.patch


 In SessionState.start [an instance of the the HMSClient is 
 created|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L367].
  When impersonation is enabled, this call does not occur within a doas call 
 and thus the HMSClient is created as the server user, not the impersonated 
 user.
 Thus calls to the HMS are made by the hive user as opposed to the end user. 
 This causes file ownership such as a database directory owner to be 
 incorrect. While debugging this, I got stack trace below. As you can see we 
 are calling getMSC without a doas.
 {noformat}
   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2474)
   at 
 org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:367)
   at 
 org.apache.hive.service.cli.session.HiveSessionImpl.init(HiveSessionImpl.java:121)
   at 
 org.apache.hive.service.cli.session.HiveSessionImplwithUGI.init(HiveSessionImplwithUGI.java:49)
   at 
 org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:130)
   at 
 org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:163)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:290)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:208)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1313)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1298)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
   at 
 org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7890) SessionState creates HMS Client while not impersonating

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118314#comment-14118314
 ] 

Brock Noland commented on HIVE-7890:


Thank you [~dongc]!!

 SessionState creates HMS Client while not impersonating
 ---

 Key: HIVE-7890
 URL: https://issues.apache.org/jira/browse/HIVE-7890
 Project: Hive
  Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-7890.2.patch


 In SessionState.start [an instance of the the HMSClient is 
 created|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L367].
  When impersonation is enabled, this call does not occur within a doas call 
 and thus the HMSClient is created as the server user, not the impersonated 
 user.
 Thus calls to the HMS are made by the hive user as opposed to the end user. 
 This causes file ownership such as a database directory owner to be 
 incorrect. While debugging this, I got stack trace below. As you can see we 
 are calling getMSC without a doas.
 {noformat}
   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2474)
   at 
 org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:367)
   at 
 org.apache.hive.service.cli.session.HiveSessionImpl.init(HiveSessionImpl.java:121)
   at 
 org.apache.hive.service.cli.session.HiveSessionImplwithUGI.init(HiveSessionImplwithUGI.java:49)
   at 
 org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:130)
   at 
 org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:163)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:290)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:208)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1313)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1298)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
   at 
 org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7890) SessionState creates HMS Client while not impersonating

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118350#comment-14118350
 ] 

Prasad Mujumdar commented on HIVE-7890:
---

Looks fine to me. 
HS2 handles the thrift connection to HMS by recreating a new connection for new 
session in case of impersonation. The HiveDB object was not handled in the same 
way. This patch is handling that more efficiently.

+1


 SessionState creates HMS Client while not impersonating
 ---

 Key: HIVE-7890
 URL: https://issues.apache.org/jira/browse/HIVE-7890
 Project: Hive
  Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-7890.2.patch


 In SessionState.start [an instance of the the HMSClient is 
 created|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L367].
  When impersonation is enabled, this call does not occur within a doas call 
 and thus the HMSClient is created as the server user, not the impersonated 
 user.
 Thus calls to the HMS are made by the hive user as opposed to the end user. 
 This causes file ownership such as a database directory owner to be 
 incorrect. While debugging this, I got stack trace below. As you can see we 
 are calling getMSC without a doas.
 {noformat}
   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2474)
   at 
 org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:367)
   at 
 org.apache.hive.service.cli.session.HiveSessionImpl.init(HiveSessionImpl.java:121)
   at 
 org.apache.hive.service.cli.session.HiveSessionImplwithUGI.init(HiveSessionImplwithUGI.java:49)
   at 
 org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:130)
   at 
 org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:163)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:290)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:208)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1313)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1298)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
   at 
 org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7890) SessionState creates HMS Client while not impersonating

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112123#comment-14112123
 ] 

Hive QA commented on HIVE-7890:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664580/HIVE-7890.2.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6115 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/523/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/523/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-523/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12664580

 SessionState creates HMS Client while not impersonating
 ---

 Key: HIVE-7890
 URL: https://issues.apache.org/jira/browse/HIVE-7890
 Project: Hive
  Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-7890.2.patch


 In SessionState.start [an instance of the the HMSClient is 
 created|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L367].
  When impersonation is enabled, this call does not occur within a doas call 
 and thus the HMSClient is created as the server user, not the impersonated 
 user.
 Thus calls to the HMS are made by the hive user as opposed to the end user. 
 This causes file ownership such as a database directory owner to be 
 incorrect. While debugging this, I got stack trace below. As you can see we 
 are calling getMSC without a doas.
 {noformat}
   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2474)
   at 
 org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:367)
   at 
 org.apache.hive.service.cli.session.HiveSessionImpl.init(HiveSessionImpl.java:121)
   at 
 org.apache.hive.service.cli.session.HiveSessionImplwithUGI.init(HiveSessionImplwithUGI.java:49)
   at 
 org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:130)
   at 
 org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:163)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:290)
   at 
 org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:208)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1313)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1298)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
   at 
 org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)