[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681262#comment-17681262
]
AjaykumarDev commented on HIVE-17502:
-
Hi [~zabetak] ,
I am so happy that you replied. Please find my comments below ,
# Code is not merged into branch-4.0.0-alpha-2 branch hence, It will still
throw the error.
https://github.com/apache/hive/blob/branch-4.0.0-alpha-2/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java
return (queueName == null) ? confQueueName == null :
queueName.equals(confQueueName);
} else {
// this session should never be a default session unless something has
messed up.
throw new HiveException("The pool session " + session + " should have
been returned to the pool");
}
Could you please check if the patch is already merged ?
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Priority: Blocker
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is
[jira] [Updated] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AjaykumarDev updated HIVE-17502:
Priority: Blocker (was: Major)
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Priority: Blocker
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[ https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855 ] AjaykumarDev edited comment on HIVE-17502 at 1/27/23 8:41 AM: -- [~vgarg] [~thejas] [~sershe] [~thai.bui] h4. [Stamatis Zampetakis|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=zabetak] h4. We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. was (Author: JIRAUSER298812): [~vgarg] [~thejas] [~sershe] [~thai.bui] Stamatis Zampetakis h4. We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please
[jira] [Comment Edited] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[ https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855 ] AjaykumarDev edited comment on HIVE-17502 at 1/27/23 8:40 AM: -- [~vgarg] [~thejas] [~sershe] [~thai.bui] Stamatis Zampetakis h4. We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. was (Author: JIRAUSER298812): [~vgarg] [~thejas] [~sershe] [~thai.bui] h4. [Vineet Garg|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vgarg] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if
[jira] [Assigned] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AjaykumarDev reassigned HIVE-17502:
---
Assignee: AjaykumarDev (was: Thai Bui)
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: AjaykumarDev
>Priority: Major
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AjaykumarDev reassigned HIVE-17502:
---
Assignee: (was: AjaykumarDev)
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Priority: Major
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[ https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855 ] AjaykumarDev edited comment on HIVE-17502 at 1/27/23 6:36 AM: -- [~vgarg] [~thejas] [~sershe] [~thai.bui] h4. [Vineet Garg|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vgarg] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. was (Author: JIRAUSER298812): [~vgarg] [~thejas] [~sershe] [~thai.bui] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information
[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681170#comment-17681170
]
AjaykumarDev commented on HIVE-17502:
-
Any update on above.
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
>Priority: Major
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AjaykumarDev updated HIVE-17502:
Status: In Progress (was: Patch Available)
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.2.0, 2.1.1
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
>Priority: Major
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[ https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855 ] AjaykumarDev edited comment on HIVE-17502 at 1/26/23 3:46 AM: -- [~vgarg] [~thejas] [~sershe] [~thai.bui] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. was (Author: JIRAUSER298812): [~vgarg] [~thejas] [~sershe] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. > Reuse of default session should not throw an exception in LLAP w/ Tez >
[jira] [Comment Edited] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[ https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855 ] AjaykumarDev edited comment on HIVE-17502 at 1/26/23 3:45 AM: -- [~vgarg] [~thejas] [~sershe] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. was (Author: JIRAUSER298812): [~vgarg] [~thejas] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. > Reuse of default session should not throw an exception in LLAP w/ Tez >
[jira] [Comment Edited] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[ https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855 ] AjaykumarDev edited comment on HIVE-17502 at 1/26/23 3:44 AM: -- [~vgarg] [~thejas] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. was (Author: JIRAUSER298812): [~vgarg] [~vgarg] We are also facing similar issue in out Hive LLap Application as below , # user1 executes the query and user2 also executes the query both users are using the same userid. As user1 executed the query first, he gets the session from thread pool however, in case of other user he gets the below error. Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received pre-hook notification for :hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a 2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228] exec.Task: Failed to execute tez graph. org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have been returned to the pool at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667) [hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] This Jira was created to address the same issue and the patch was also available. So, when can we expect this patch to be available in any future hive release or any alternative is available ? Please let me know because the solution for this issue is very critical for success of my company project, the company has made big investment for hive llap project and we want to resolve this issue as soon as possible. Please let us know if you need more information about the same. > Reuse of default session should not throw an exception in LLAP w/ Tez >
[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680855#comment-17680855
]
AjaykumarDev commented on HIVE-17502:
-
[~vgarg] [~vgarg]
We are also facing similar issue in out Hive LLap Application as below ,
# user1 executes the query and user2 also executes the query both users are
using the same userid. As user1 executed the query first, he gets the session
from thread pool however, in case of other user he gets the below error.
Error : 2023-01-25T15:40:54,852 INFO [ATS Logger 0] hooks.ATSHook: Received
pre-hook notification for
:hive_20230125154054_3e24f02f-c0e5-4ad2-87a0-758d025cb02a
2023-01-25T15:40:54,863 ERROR [HiveServer2-Background-Pool: Thread-228]
exec.Task: Failed to execute tez graph.
org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
sessionId=59675216-4e35-4872-8e22-835c50b8c23d, queueName=llap, user=hive,
doAs=false, isOpen=true, isDefault=true, expires in 593431620ms should have
been returned to the pool at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:546)
~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:556)
~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:150)
[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
This Jira was created to address the same issue and the patch was also
available. So, when can we expect this patch to be available in any future hive
release or any alternative is available ? Please let me know because the
solution for this issue is very critical for success of my company project, the
company has made big investment for hive llap project and we want to resolve
this issue as soon as possible. Please let us know if you need more information
about the same.
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
>Priority: Major
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
>
