[
https://issues.apache.org/jira/browse/HIVE-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277959#comment-16277959
]
Sergey Shelukhin commented on HIVE-18153:
-----------------------------------------
This basically moves Hive resources out of the Tez scratch dir, and also
removes the resource logic that is split between TezTask (that had most but not
all of conf resources' logic) and TezSessionState (that had most by not all of
the non-conf resources logic). The session now localizes all the resources and
does all the checks, AM calls, and whatnot.
Additionally, resources are tracked in a separate POJO that can be reused on
reopen. It keeps track both of the directory and of what has been localized
into it; that can potentially reduce number of FS calls for conf-based
resources that are currently just refreshed blindly on every use of the same
session.
> refactor reopen and file management in TezTask
> ----------------------------------------------
>
> Key: HIVE-18153
> URL: https://issues.apache.org/jira/browse/HIVE-18153
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Attachments: HIVE-18153.patch
>
>
> TezTask reopen relies on getting the same session object in terms of setup;
> WM reopen returns a new session from the pool.
> The former has the advantage of not having to reupload files and stuff... but
> the object reuse results in a lot of ugly code, and also reopen might be
> slower on average with the session pool than just getting a session from the
> pool. Either WM needs to do the object-preserving reopen, or TezTask needs to
> be refactored. It looks like DAG would have to be rebuilt to do the latter
> because of some paths tied to a directory of the old session. Let me see if I
> can get around that; if not we can do the former; and then if the former
> results in too much ugly code in WM to account for object reuse for different
> Tez client I'd do the latter anyway since it's a failure path :)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
