[
https://issues.apache.org/jira/browse/HIVE-26564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610790#comment-17610790
]
Yi Zhang commented on HIVE-26564:
---------------------------------
[~zabetak] updated the description with example layouts before and after the
change.
> Separate query live operation log and historical operation log
> --------------------------------------------------------------
>
> Key: HIVE-26564
> URL: https://issues.apache.org/jira/browse/HIVE-26564
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Affects Versions: 4.0.0-alpha-2
> Reporter: Yi Zhang
> Assignee: Yi Zhang
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> HIVE-24802 added OperationLogManager to support historical operation logs.
> OperationLogManager.createOperationLog creates operation log inside
> historical operation log dir if
> HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on
> session level, SessionManager and HiveSession are using original operation
> log session directory.
> Proposed change is to separate live query's operation log and historical
> operation log. Upon operation close, OperationLogManager.closeOperation is
> called to move the operation log from session directory to historical log
> dir. OperationLogManager is only responsible to clean up historical operation
> logs.
> This change also makes it easier to manage historical logs, for example, user
> may want to persist historical logs, it is easier to differentiate live and
> historical operation logs.
>
> before this change, if HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true, the
> operation logs lay out is as following. in operation_logs_historic has both
> live queries and historic queries's operational logs
> /tmp/hive/
> ├── operation_logs
> └── operation_logs_historic
> └── hs2hostname_startupTimestamp
> ├── session_id_1
> │ ├── hive_query_id_1
> │ ├── hive_query_id_2
> │ └── hive_query_id_3
> ├── session_id_2
> │ ├── hive_query_id_4
> │ └── hive_query_id_5
> ├── session_id_3
> │ └── hive_query_id_6
> └── session_id_4
> ├── hive_query_id_7
> └── hive_query_id_8
>
> after this change, the live queries operation logs are under <operation_logs>
> and historical ones under <operation_logs_historic>
> /tmp/hive
> ├── operation_logs
> │ ├── session_id_1
> │ │ ├── hive_query_id_2
> │ │ └── hive_query_id_3
> │ └── session_id_4
> │ └── hive_query_id_8
> └── operation_logs_historic
> └── hs2hostname_startupTimestamp
> ├── session_id_1
> │ └── hive_query_id_1
> ├── session_id_2
> │ ├── hive_query_id_4
> │ └── hive_query_id_5
> ├── session_id_3
> │ └── hive_query_id_6
> └── session_id_4
> └── hive_query_id_7
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
