[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286161#comment-14286161 ] Chinna Rao Lalam commented on HIVE-9289: I have verified this code, reusing the session is not happening because as [~chengxiang li] explained Hive Client-SessionHandler(session id inside)- HiveSessionImpl-SessionState-SparkSession this linear mapping is maintained. Updated the patch by removing that code. TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch, HIVE-9289.2-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286176#comment-14286176 ] Xuefu Zhang commented on HIVE-9289: --- +1 pending on test. TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch, HIVE-9289.2-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286876#comment-14286876
]
Hive QA commented on HIVE-9289:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12693680/HIVE-9289.2-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7309 tests executed
*Failed tests:*
{noformat}
TestPigHBaseStorageHandler - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/662/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/662/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12693680 - PreCommit-HIVE-SPARK-Build
TODO : Store user name in session [Spark Branch]
Key: HIVE-9289
URL: https://issues.apache.org/jira/browse/HIVE-9289
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Attachments: HIVE-9289.1-spark.patch, HIVE-9289.2-spark.patch
TODO : this we need to store the session username somewhere else as
getUGIForConf never used the conf SparkSessionManagerImpl.java
/hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145
Java Task
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268928#comment-14268928 ] Chengxiang Li commented on HIVE-9289: - We does reuse session for queries from same client, [~xuefuz], I verified before and i just did again, just may not work in the way we thought before. HiveServer2 introduce a new approach to manager session, every RPC call references a session ID which the server then maps to persistent session state,the linear mapping is like: Hive Client-SessionHandler(session id inside)\-HiveSessionImpl-SessionState. For Hive on Spark, as we would like to share the singleton SparkContext for a user session, we may expand the linear mapping like: Hive Client-SessionHandler(session id inside)\- HiveSessionImpl-SessionState-SparkSession-SparkClient-RemoteDriver-SparkContext, with only exception: create new SparkSession while spark configuration update. Hive Client-SessionHandler(session id inside)-HiveSessionImpl mapping should already ensure that whether Hive session should reused, I think we do not need to check again by user name in SparkSessionManager unless we have other reasons to create new SparkSession beside spark configuration update. [~chinnalalam], I just read related code today, not sure if i fully understand it, do you have any idea about it? TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268105#comment-14268105 ] Xuefu Zhang commented on HIVE-9289: --- Maybe I don't fully understand the code, but it seems a little concerning that session is reused purely based on user name. Hive supports multiple sessions for the same user, and we don't want such session is reused. I also have a feeling that we don't have any session reuse (though we have code here and there for that). If that's the case, we'd rather just get rid of the code. [~chengxiang li], could you comment on this? TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267772#comment-14267772
]
Hive QA commented on HIVE-9289:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690535/HIVE-9289.1-spark.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7283 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/612/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/612/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-612/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690535 - PreCommit-HIVE-SPARK-Build
TODO : Store user name in session [Spark Branch]
Key: HIVE-9289
URL: https://issues.apache.org/jira/browse/HIVE-9289
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Attachments: HIVE-9289.1-spark.patch
TODO : this we need to store the session username somewhere else as
getUGIForConf never used the conf SparkSessionManagerImpl.java
/hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145
Java Task
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268072#comment-14268072 ] Szehon Ho commented on HIVE-9289: - +1 it looks ok to me as it seems this is used just to determine whether session can be reused, but I admit I'm not too familiar with this code. TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
