[
https://issues.apache.org/jira/browse/HIVE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gaozhan ding updated HIVE-23802:
--------------------------------
Attachment: (was: 企业微信截图_15940042679272.png)
> “merge files” job was submited to default queue when set hive.merge.tezfiles
> to true
> ------------------------------------------------------------------------------------
>
> Key: HIVE-23802
> URL: https://issues.apache.org/jira/browse/HIVE-23802
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 3.1.0
> Reporter: gaozhan ding
> Assignee: gaozhan ding
> Priority: Major
>
> We use tez as the query engine. When hive.merge.tezfiles set to true，merge
> files task, which followed by orginal task, will be submit to default queue
> rather then the queue same with orginal task.
> I study this issue for days and found that, every time starting a container,
> "tez,queue.name" whill be unset in current session. Code are as below:
> {code:java}
> // TezSessionState.startSessionAndContainers()
> // sessionState.getQueueName() comes from cluster wide configured queue names.
> // sessionState.getConf().get("tez.queue.name") is explicitly set by user in
> a session.
> // TezSessionPoolManager sets tez.queue.name if user has specified one or
> use the one from
> // cluster wide queue names.
> // There is no way to differentiate how this was set (user vs system).
> // Unset this after opening the session so that reopening of session uses
> the correct queue
> // names i.e, if client has not died and if the user has explicitly set a
> queue name
> // then reopened session will use user specified queue name else default
> cluster queue names.
> conf.unset(TezConfiguration.TEZ_QUEUE_NAME);
> {code}
> So after the orgin tast was submited to yarn, "tez,queue.name" will be unset.
> While starting merge file task, it will try use the same session with orgin
> job, but get false due to tez,queue.name was unset.
> {code:java}
> // TezSessionPoolManager.canWorkWithSameSession()
> if (!session.isDefault()) {
> String queueName = session.getQueueName();
> String confQueueName = conf.get(TezConfiguration.TEZ_QUEUE_NAME);
> LOG.info("Current queue name is " + queueName + " incoming queue name is "
> + confQueueName);
> return (queueName == null) ? confQueueName == null :
> queueName.equals(confQueueName);
> } else {
> // this session should never be a default session unless something has
> messed up.
> throw new HiveException("The pool session " + session + " should have been
> returned to the pool");
> }
> {code}
> !企业微信截图_15940042679272.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
