[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=814507&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814507 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 06/Oct/22 22:57 Start Date: 06/Oct/22 22:57 Worklog Time Spent: 10m Work Description: sunchao merged PR #3621: URL: https://github.com/apache/hive/pull/3621 Issue Time Tracking --- Worklog Id: (was: 814507) Time Spent: 2h 10m (was: 2h) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. > > before this change, if HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true, the > operation logs lay out is as following. in operation_logs_historic has both > live queries and historic queries's operational logs > ``` > /tmp/hive/ > ├── operation_logs > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ ├── hive_query_id_1 > │ ├── hive_query_id_2 > │ └── hive_query_id_3 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > ├── hive_query_id_7 > └── hive_query_id_8 > ``` > after this change, the live queries operation logs are under > and historical ones under > /tmp/hive > ├── operation_logs > │ ├── session_id_1 > │ │ ├── hive_query_id_2 > │ │ └── hive_query_id_3 > │ └── session_id_4 > │ └── hive_query_id_8 > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ └── hive_query_id_1 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > └── hive_query_id_7 > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=814508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814508 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 06/Oct/22 22:57 Start Date: 06/Oct/22 22:57 Worklog Time Spent: 10m Work Description: sunchao commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1270806065 Merged to master. Thanks @yigress Issue Time Tracking --- Worklog Id: (was: 814508) Time Spent: 2h 20m (was: 2h 10m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. > > before this change, if HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true, the > operation logs lay out is as following. in operation_logs_historic has both > live queries and historic queries's operational logs > ``` > /tmp/hive/ > ├── operation_logs > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ ├── hive_query_id_1 > │ ├── hive_query_id_2 > │ └── hive_query_id_3 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > ├── hive_query_id_7 > └── hive_query_id_8 > ``` > after this change, the live queries operation logs are under > and historical ones under > /tmp/hive > ├── operation_logs > │ ├── session_id_1 > │ │ ├── hive_query_id_2 > │ │ └── hive_query_id_3 > │ └── session_id_4 > │ └── hive_query_id_8 > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ └── hive_query_id_1 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > └── hive_query_id_7 > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=813510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813510 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 29/Sep/22 21:52 Start Date: 29/Sep/22 21:52 Worklog Time Spent: 10m Work Description: zabetak commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1262853833 Hey @yigress , I'm still busy with other reviews. Will get back to this ASAP. Issue Time Tracking --- Worklog Id: (was: 813510) Time Spent: 2h (was: 1h 50m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. > > before this change, if HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true, the > operation logs lay out is as following. in operation_logs_historic has both > live queries and historic queries's operational logs > ``` > /tmp/hive/ > ├── operation_logs > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ ├── hive_query_id_1 > │ ├── hive_query_id_2 > │ └── hive_query_id_3 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > ├── hive_query_id_7 > └── hive_query_id_8 > ``` > after this change, the live queries operation logs are under > and historical ones under > /tmp/hive > ├── operation_logs > │ ├── session_id_1 > │ │ ├── hive_query_id_2 > │ │ └── hive_query_id_3 > │ └── session_id_4 > │ └── hive_query_id_8 > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ └── hive_query_id_1 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > └── hive_query_id_7 > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=813452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813452 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 29/Sep/22 19:02 Start Date: 29/Sep/22 19:02 Worklog Time Spent: 10m Work Description: yigress commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1262694824 @zabetak ping ping, latest tests failures are unrelated. Can you help proceed? Thank you! Issue Time Tracking --- Worklog Id: (was: 813452) Time Spent: 1h 50m (was: 1h 40m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. > > before this change, if HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true, the > operation logs lay out is as following. in operation_logs_historic has both > live queries and historic queries's operational logs > ``` > /tmp/hive/ > ├── operation_logs > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ ├── hive_query_id_1 > │ ├── hive_query_id_2 > │ └── hive_query_id_3 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > ├── hive_query_id_7 > └── hive_query_id_8 > ``` > after this change, the live queries operation logs are under > and historical ones under > /tmp/hive > ├── operation_logs > │ ├── session_id_1 > │ │ ├── hive_query_id_2 > │ │ └── hive_query_id_3 > │ └── session_id_4 > │ └── hive_query_id_8 > └── operation_logs_historic > └── hs2hostname_startupTimestamp > ├── session_id_1 > │ └── hive_query_id_1 > ├── session_id_2 > │ ├── hive_query_id_4 > │ └── hive_query_id_5 > ├── session_id_3 > │ └── hive_query_id_6 > └── session_id_4 > └── hive_query_id_7 > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=813060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813060 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 28/Sep/22 16:48 Start Date: 28/Sep/22 16:48 Worklog Time Spent: 10m Work Description: yigress commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1261180092 Thank you @dengzhhu653 for the review! The current test failures are unrelated. @zabetak as pending reviewer, can you help review? Thank you! Issue Time Tracking --- Worklog Id: (was: 813060) Time Spent: 1h 40m (was: 1.5h) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812707&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812707 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 28/Sep/22 01:19 Start Date: 28/Sep/22 01:19 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1260270179 Hi @yigress, the failed tests seem related, cloud you please fix them? Issue Time Tracking --- Worklog Id: (was: 812707) Time Spent: 1.5h (was: 1h 20m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812572&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812572 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 27/Sep/22 15:54 Start Date: 27/Sep/22 15:54 Worklog Time Spent: 10m Work Description: sfc-gh-aixu commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1259707761 > @dengzhhu653 can you help review this? Thank you! + @pvary and @nrg4878 can you also help review this change? Issue Time Tracking --- Worklog Id: (was: 812572) Time Spent: 1h 20m (was: 1h 10m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812343&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812343 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 27/Sep/22 03:49 Start Date: 27/Sep/22 03:49 Worklog Time Spent: 10m Work Description: yigress commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1258933010 Thank you @dengzhhu653 so much for your review! The reason for this change is that we want to add a feature to persist the hive QueryInfo and OperationLog at first, for example, persist the information on HDFS or other storage, for ephemeral clusters. Right now only a limited number of historical queries are available in hs2 UI. So OperationLogManager seems an ideal place to do that, or some other ways outside hive that look at the historical operation log location, but current mixing live queries inside historical dir makes it harder to differentiate. Issue Time Tracking --- Worklog Id: (was: 812343) Time Spent: 1h 10m (was: 1h) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[
https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812341&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812341
]
ASF GitHub Bot logged work on HIVE-26564:
-
Author: ASF GitHub Bot
Created on: 27/Sep/22 03:43
Start Date: 27/Sep/22 03:43
Worklog Time Spent: 10m
Work Description: yigress commented on code in PR #3621:
URL: https://github.com/apache/hive/pull/3621#discussion_r980701496
##
service/src/test/org/apache/hive/service/cli/operation/TestOperationLogManager.java:
##
@@ -61,6 +60,7 @@ public void setUp() throws Exception {
HiveConf.setIntVar(hiveConf,
HiveConf.ConfVars.HIVE_SERVER2_WEBUI_MAX_HISTORIC_QUERIES, 1);
HiveConf.setIntVar(hiveConf, HiveConf.ConfVars.HIVE_SERVER2_WEBUI_PORT,
8080);
HiveConf.setBoolVar(hiveConf, HiveConf.ConfVars.HIVE_IN_TEST, true);
+HiveConf.setBoolVar(hiveConf, HiveConf.ConfVars.HIVE_TESTING_REMOVE_LOGS,
false);
Review Comment:
This is because
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java#L80
if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST)) {
isRemoveLogs =
hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_TESTING_REMOVE_LOGS);
Instead of changing the code there, I thought it may be better set it to
false for this test case. In production no need to change this setting.
##
service/src/java/org/apache/hive/service/cli/operation/Operation.java:
##
@@ -314,12 +318,13 @@ protected synchronized void cleanupOperationLog(final
long operationLogCleanupDe
} else {
if (operationLogCleanupDelayMs > 0) {
ScheduledExecutorService scheduledExecutorService =
Executors.newScheduledThreadPool(1);
- scheduledExecutorService.schedule(new
OperationLogCleaner(operationLog), operationLogCleanupDelayMs,
+ scheduledExecutorService.schedule(new OperationLogCleaner(this,
operationLog), operationLogCleanupDelayMs,
TimeUnit.MILLISECONDS);
scheduledExecutorService.shutdown();
} else {
log.info("Closing operation log {} without delay", operationLog);
operationLog.close();
+ OperationLogManager.closeOperation(this);
Review Comment:
The reason I did not set inside operationLog.close is that OperationLog is a
wrapper of the log file itself and has no knowledge of the operation.
Issue Time Tracking
---
Worklog Id: (was: 812341)
Time Spent: 1h (was: 50m)
> Separate query live operation log and historical operation log
> --
>
> Key: HIVE-26564
> URL: https://issues.apache.org/jira/browse/HIVE-26564
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: Yi Zhang
>Assignee: Yi Zhang
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HIVE-24802 added OperationLogManager to support historical operation logs.
> OperationLogManager.createOperationLog creates operation log inside
> historical operation log dir if
> HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on
> session level, SessionManager and HiveSession are using original operation
> log session directory.
> Proposed change is to separate live query's operation log and historical
> operation log. Upon operation close, OperationLogManager.closeOperation is
> called to move the operation log from session directory to historical log
> dir. OperationLogManager is only responsible to clean up historical operation
> logs.
> This change also makes it easier to manage historical logs, for example, user
> may want to persist historical logs, it is easier to differentiate live and
> historical operation logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812315&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812315 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 27/Sep/22 02:27 Start Date: 27/Sep/22 02:27 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1258882698 > @dengzhhu653 can you help review this? Thank you! The idea overall makes sense to me, thank you for the contribution! By the way, which kinds of problem do you meet regarding the original design? Issue Time Tracking --- Worklog Id: (was: 812315) Time Spent: 50m (was: 40m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[
https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812313&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812313
]
ASF GitHub Bot logged work on HIVE-26564:
-
Author: ASF GitHub Bot
Created on: 27/Sep/22 02:20
Start Date: 27/Sep/22 02:20
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3621:
URL: https://github.com/apache/hive/pull/3621#discussion_r980668220
##
service/src/java/org/apache/hive/service/cli/operation/Operation.java:
##
@@ -314,12 +318,13 @@ protected synchronized void cleanupOperationLog(final
long operationLogCleanupDe
} else {
if (operationLogCleanupDelayMs > 0) {
ScheduledExecutorService scheduledExecutorService =
Executors.newScheduledThreadPool(1);
- scheduledExecutorService.schedule(new
OperationLogCleaner(operationLog), operationLogCleanupDelayMs,
+ scheduledExecutorService.schedule(new OperationLogCleaner(this,
operationLog), operationLogCleanupDelayMs,
TimeUnit.MILLISECONDS);
scheduledExecutorService.shutdown();
} else {
log.info("Closing operation log {} without delay", operationLog);
operationLog.close();
+ OperationLogManager.closeOperation(this);
Review Comment:
nit: can we put `OperationLogManager.closeOperation(this)` into
`operationLog.close()`? so we can only take care of SQLOperation's log
Issue Time Tracking
---
Worklog Id: (was: 812313)
Time Spent: 40m (was: 0.5h)
> Separate query live operation log and historical operation log
> --
>
> Key: HIVE-26564
> URL: https://issues.apache.org/jira/browse/HIVE-26564
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: Yi Zhang
>Assignee: Yi Zhang
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> HIVE-24802 added OperationLogManager to support historical operation logs.
> OperationLogManager.createOperationLog creates operation log inside
> historical operation log dir if
> HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on
> session level, SessionManager and HiveSession are using original operation
> log session directory.
> Proposed change is to separate live query's operation log and historical
> operation log. Upon operation close, OperationLogManager.closeOperation is
> called to move the operation log from session directory to historical log
> dir. OperationLogManager is only responsible to clean up historical operation
> logs.
> This change also makes it easier to manage historical logs, for example, user
> may want to persist historical logs, it is easier to differentiate live and
> historical operation logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[
https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812312&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812312
]
ASF GitHub Bot logged work on HIVE-26564:
-
Author: ASF GitHub Bot
Created on: 27/Sep/22 02:15
Start Date: 27/Sep/22 02:15
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3621:
URL: https://github.com/apache/hive/pull/3621#discussion_r980666461
##
service/src/test/org/apache/hive/service/cli/operation/TestOperationLogManager.java:
##
@@ -61,6 +60,7 @@ public void setUp() throws Exception {
HiveConf.setIntVar(hiveConf,
HiveConf.ConfVars.HIVE_SERVER2_WEBUI_MAX_HISTORIC_QUERIES, 1);
HiveConf.setIntVar(hiveConf, HiveConf.ConfVars.HIVE_SERVER2_WEBUI_PORT,
8080);
HiveConf.setBoolVar(hiveConf, HiveConf.ConfVars.HIVE_IN_TEST, true);
+HiveConf.setBoolVar(hiveConf, HiveConf.ConfVars.HIVE_TESTING_REMOVE_LOGS,
false);
Review Comment:
why this property should set to false when
HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED is enabled? we do not want to set
this in production.
Issue Time Tracking
---
Worklog Id: (was: 812312)
Time Spent: 0.5h (was: 20m)
> Separate query live operation log and historical operation log
> --
>
> Key: HIVE-26564
> URL: https://issues.apache.org/jira/browse/HIVE-26564
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: Yi Zhang
>Assignee: Yi Zhang
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-24802 added OperationLogManager to support historical operation logs.
> OperationLogManager.createOperationLog creates operation log inside
> historical operation log dir if
> HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on
> session level, SessionManager and HiveSession are using original operation
> log session directory.
> Proposed change is to separate live query's operation log and historical
> operation log. Upon operation close, OperationLogManager.closeOperation is
> called to move the operation log from session directory to historical log
> dir. OperationLogManager is only responsible to clean up historical operation
> logs.
> This change also makes it easier to manage historical logs, for example, user
> may want to persist historical logs, it is easier to differentiate live and
> historical operation logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812296 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 26/Sep/22 23:37 Start Date: 26/Sep/22 23:37 Worklog Time Spent: 10m Work Description: yigress commented on PR #3621: URL: https://github.com/apache/hive/pull/3621#issuecomment-1258768746 @dengzhhu653 can you help review this? Thank you! Issue Time Tracking --- Worklog Id: (was: 812296) Time Spent: 20m (was: 10m) > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26564) Separate query live operation log and historical operation log
[ https://issues.apache.org/jira/browse/HIVE-26564?focusedWorklogId=812191&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812191 ] ASF GitHub Bot logged work on HIVE-26564: - Author: ASF GitHub Bot Created on: 26/Sep/22 18:12 Start Date: 26/Sep/22 18:12 Worklog Time Spent: 10m Work Description: yigress opened a new pull request, #3621: URL: https://github.com/apache/hive/pull/3621 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 812191) Remaining Estimate: 0h Time Spent: 10m > Separate query live operation log and historical operation log > -- > > Key: HIVE-26564 > URL: https://issues.apache.org/jira/browse/HIVE-26564 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-24802 added OperationLogManager to support historical operation logs. > OperationLogManager.createOperationLog creates operation log inside > historical operation log dir if > HIVE_SERVER2_HISTORIC_OPERATION_LOG_ENABLED=true. This is confusing, since on > session level, SessionManager and HiveSession are using original operation > log session directory. > Proposed change is to separate live query's operation log and historical > operation log. Upon operation close, OperationLogManager.closeOperation is > called to move the operation log from session directory to historical log > dir. OperationLogManager is only responsible to clean up historical operation > logs. > This change also makes it easier to manage historical logs, for example, user > may want to persist historical logs, it is easier to differentiate live and > historical operation logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
