[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815705 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 11/Oct/22 14:46 Start Date: 11/Oct/22 14:46 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #3645: URL: https://github.com/apache/hive/pull/3645 Issue Time Tracking --- Worklog Id: (was: 815705) Time Spent: 4h 50m (was: 4h 40m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815565 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 11/Oct/22 09:13 Start Date: 11/Oct/22 09:13 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3645: URL: https://github.com/apache/hive/pull/3645#issuecomment-1274373345 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3645) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3645=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3645=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 815565) Time Spent: 4h 40m (was: 4.5h) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815511
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 08:07
Start Date: 11/Oct/22 08:07
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991965326
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -67,34 +70,53 @@ static CompactionMetricData
of(List compacts) {
private void init() {
final Map lastElements = new
HashMap<>();
-poolCount = new HashMap<>();
+initiatedPoolCount = new HashMap<>();
+workingPoolCount = new HashMap<>();
+oldestEnqueueTimePerPool = new HashMap<>();
+oldestWorkingTimePerPool = new HashMap<>();
oldestEnqueueTime = OLDEST_TIME_NO_VALUE;
oldestWorkingTime = OLDEST_TIME_NO_VALUE;
oldestCleaningTime = OLDEST_TIME_NO_VALUE;
+
+long currentTime = System.currentTimeMillis();
for (ShowCompactResponseElement element : compacts) {
final String key = element.getDbname() + "/" + element.getTablename() +
(element.getPartitionname() != null ? "/" +
element.getPartitionname() : "");
// If new key, add the element, if there is an existing one, change to
the element if the element.id is greater than old.id
lastElements.compute(key, (k, old) -> (old == null) ? element :
(element.getId() > old.getId() ? element : old));
- // find the oldest elements with initiated and working states
String state = element.getState();
- if (TxnStore.INITIATED_RESPONSE.equals(state) && (oldestEnqueueTime >
element.getEnqueueTime())) {
-oldestEnqueueTime = element.getEnqueueTime();
-poolCount.compute(element.getPoolName(), (k, old) -> (old == null) ? 1
: old + 1);
+ if (TxnStore.INITIATED_RESPONSE.equals(state)) {
+final int enqueueSeconds = (int) ((currentTime -
element.getEnqueueTime()) / 1000);
+oldestWorkingTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? enqueueSeconds : Math.max(enqueueSeconds, old));
Review Comment:
good catch, I accidentally swapped the variables. I also changed the names
to longestEnqueueDurationPerPool and longestWorkingDurationPerPool (+ getters).
Issue Time Tracking
---
Worklog Id: (was: 815511)
Time Spent: 4.5h (was: 4h 20m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815495=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815495
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 07:37
Start Date: 11/Oct/22 07:37
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991919119
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -67,34 +70,53 @@ static CompactionMetricData
of(List compacts) {
private void init() {
final Map lastElements = new
HashMap<>();
-poolCount = new HashMap<>();
+initiatedPoolCount = new HashMap<>();
+workingPoolCount = new HashMap<>();
+oldestEnqueueTimePerPool = new HashMap<>();
+oldestWorkingTimePerPool = new HashMap<>();
oldestEnqueueTime = OLDEST_TIME_NO_VALUE;
oldestWorkingTime = OLDEST_TIME_NO_VALUE;
oldestCleaningTime = OLDEST_TIME_NO_VALUE;
+
+long currentTime = System.currentTimeMillis();
for (ShowCompactResponseElement element : compacts) {
final String key = element.getDbname() + "/" + element.getTablename() +
(element.getPartitionname() != null ? "/" +
element.getPartitionname() : "");
// If new key, add the element, if there is an existing one, change to
the element if the element.id is greater than old.id
lastElements.compute(key, (k, old) -> (old == null) ? element :
(element.getId() > old.getId() ? element : old));
- // find the oldest elements with initiated and working states
String state = element.getState();
- if (TxnStore.INITIATED_RESPONSE.equals(state) && (oldestEnqueueTime >
element.getEnqueueTime())) {
-oldestEnqueueTime = element.getEnqueueTime();
-poolCount.compute(element.getPoolName(), (k, old) -> (old == null) ? 1
: old + 1);
+ if (TxnStore.INITIATED_RESPONSE.equals(state)) {
+final int enqueueSeconds = (int) ((currentTime -
element.getEnqueueTime()) / 1000);
+oldestWorkingTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? enqueueSeconds : Math.max(enqueueSeconds, old));
+
+initiatedPoolCount.compute(element.getPoolName(), (k, old) -> (old ==
null) ? 1 : old + 1);
+// find the oldest element
+if (oldestEnqueueTime > element.getEnqueueTime()) {
+ oldestEnqueueTime = element.getEnqueueTime();
+}
}
if (element.isSetStart()) {
-if (TxnStore.WORKING_RESPONSE.equals(state) && (oldestWorkingTime >
element.getStart())) {
- oldestWorkingTime = element.getStart();
+if (TxnStore.WORKING_RESPONSE.equals(state)) {
+ final int startSeconds = (int) ((currentTime - element.getStart()) /
1000);
+ oldestEnqueueTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? startSeconds : Math.max(startSeconds, old));
+
+ workingPoolCount.compute(element.getPoolName(), (k, old) -> (old ==
null) ? 1 : old + 1);
+ // find the oldest element
+ if (oldestWorkingTime > element.getStart()) {
+oldestWorkingTime = element.getStart();
+ }
}
}
if (element.isSetCleanerStart()) {
-if (TxnStore.CLEANING_RESPONSE.equals(state) && (oldestCleaningTime >
element.getCleanerStart())) {
- oldestCleaningTime = element.getCleanerStart();
+if (TxnStore.CLEANING_RESPONSE.equals(state)) {
Review Comment:
yes, please
Issue Time Tracking
---
Worklog Id: (was: 815495)
Time Spent: 4h 20m (was: 4h 10m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815493
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 07:36
Start Date: 11/Oct/22 07:36
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991917809
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -152,8 +174,20 @@ Map getStateCount() {
return new HashMap<>(stateCount);
}
- public Map getPoolCount() {
-return new HashMap<>(poolCount);
+ public Map getInitiatedPoolCount() {
+return new HashMap<>(initiatedPoolCount);
Review Comment:
should it be unmodifiable map?
Issue Time Tracking
---
Worklog Id: (was: 815493)
Time Spent: 4h 10m (was: 4h)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815491
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 07:34
Start Date: 11/Oct/22 07:34
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991915282
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -67,34 +70,53 @@ static CompactionMetricData
of(List compacts) {
private void init() {
final Map lastElements = new
HashMap<>();
-poolCount = new HashMap<>();
+initiatedPoolCount = new HashMap<>();
+workingPoolCount = new HashMap<>();
+oldestEnqueueTimePerPool = new HashMap<>();
+oldestWorkingTimePerPool = new HashMap<>();
oldestEnqueueTime = OLDEST_TIME_NO_VALUE;
oldestWorkingTime = OLDEST_TIME_NO_VALUE;
oldestCleaningTime = OLDEST_TIME_NO_VALUE;
+
+long currentTime = System.currentTimeMillis();
for (ShowCompactResponseElement element : compacts) {
final String key = element.getDbname() + "/" + element.getTablename() +
(element.getPartitionname() != null ? "/" +
element.getPartitionname() : "");
// If new key, add the element, if there is an existing one, change to
the element if the element.id is greater than old.id
lastElements.compute(key, (k, old) -> (old == null) ? element :
(element.getId() > old.getId() ? element : old));
- // find the oldest elements with initiated and working states
String state = element.getState();
- if (TxnStore.INITIATED_RESPONSE.equals(state) && (oldestEnqueueTime >
element.getEnqueueTime())) {
-oldestEnqueueTime = element.getEnqueueTime();
-poolCount.compute(element.getPoolName(), (k, old) -> (old == null) ? 1
: old + 1);
+ if (TxnStore.INITIATED_RESPONSE.equals(state)) {
+final int enqueueSeconds = (int) ((currentTime -
element.getEnqueueTime()) / 1000);
+oldestWorkingTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? enqueueSeconds : Math.max(enqueueSeconds, old));
Review Comment:
shouldn't it be vice versa? when INITIATED_RESPONSE ->
oldestEnqueueTimePerPool, when WORKING_RESPONSE -> oldestWorkingTimePerPool
btw, the name is misleading, you are calculating duration here , not the
time, is that expected?
Issue Time Tracking
---
Worklog Id: (was: 815491)
Time Spent: 4h (was: 3h 50m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815490
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 07:33
Start Date: 11/Oct/22 07:33
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991915282
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -67,34 +70,53 @@ static CompactionMetricData
of(List compacts) {
private void init() {
final Map lastElements = new
HashMap<>();
-poolCount = new HashMap<>();
+initiatedPoolCount = new HashMap<>();
+workingPoolCount = new HashMap<>();
+oldestEnqueueTimePerPool = new HashMap<>();
+oldestWorkingTimePerPool = new HashMap<>();
oldestEnqueueTime = OLDEST_TIME_NO_VALUE;
oldestWorkingTime = OLDEST_TIME_NO_VALUE;
oldestCleaningTime = OLDEST_TIME_NO_VALUE;
+
+long currentTime = System.currentTimeMillis();
for (ShowCompactResponseElement element : compacts) {
final String key = element.getDbname() + "/" + element.getTablename() +
(element.getPartitionname() != null ? "/" +
element.getPartitionname() : "");
// If new key, add the element, if there is an existing one, change to
the element if the element.id is greater than old.id
lastElements.compute(key, (k, old) -> (old == null) ? element :
(element.getId() > old.getId() ? element : old));
- // find the oldest elements with initiated and working states
String state = element.getState();
- if (TxnStore.INITIATED_RESPONSE.equals(state) && (oldestEnqueueTime >
element.getEnqueueTime())) {
-oldestEnqueueTime = element.getEnqueueTime();
-poolCount.compute(element.getPoolName(), (k, old) -> (old == null) ? 1
: old + 1);
+ if (TxnStore.INITIATED_RESPONSE.equals(state)) {
+final int enqueueSeconds = (int) ((currentTime -
element.getEnqueueTime()) / 1000);
+oldestWorkingTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? enqueueSeconds : Math.max(enqueueSeconds, old));
Review Comment:
shouldn't it be vice versa? when INITIATED_RESPONSE ->
oldestEnqueueTimePerPool, when WORKING_RESPONSE -> oldestWorkingTimePerPool
Issue Time Tracking
---
Worklog Id: (was: 815490)
Time Spent: 3h 50m (was: 3h 40m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815488
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 07:32
Start Date: 11/Oct/22 07:32
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991914034
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -67,34 +70,53 @@ static CompactionMetricData
of(List compacts) {
private void init() {
final Map lastElements = new
HashMap<>();
-poolCount = new HashMap<>();
+initiatedPoolCount = new HashMap<>();
+workingPoolCount = new HashMap<>();
+oldestEnqueueTimePerPool = new HashMap<>();
+oldestWorkingTimePerPool = new HashMap<>();
oldestEnqueueTime = OLDEST_TIME_NO_VALUE;
oldestWorkingTime = OLDEST_TIME_NO_VALUE;
oldestCleaningTime = OLDEST_TIME_NO_VALUE;
+
+long currentTime = System.currentTimeMillis();
for (ShowCompactResponseElement element : compacts) {
final String key = element.getDbname() + "/" + element.getTablename() +
(element.getPartitionname() != null ? "/" +
element.getPartitionname() : "");
// If new key, add the element, if there is an existing one, change to
the element if the element.id is greater than old.id
lastElements.compute(key, (k, old) -> (old == null) ? element :
(element.getId() > old.getId() ? element : old));
- // find the oldest elements with initiated and working states
String state = element.getState();
- if (TxnStore.INITIATED_RESPONSE.equals(state) && (oldestEnqueueTime >
element.getEnqueueTime())) {
-oldestEnqueueTime = element.getEnqueueTime();
-poolCount.compute(element.getPoolName(), (k, old) -> (old == null) ? 1
: old + 1);
+ if (TxnStore.INITIATED_RESPONSE.equals(state)) {
+final int enqueueSeconds = (int) ((currentTime -
element.getEnqueueTime()) / 1000);
+oldestWorkingTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? enqueueSeconds : Math.max(enqueueSeconds, old));
+
+initiatedPoolCount.compute(element.getPoolName(), (k, old) -> (old ==
null) ? 1 : old + 1);
+// find the oldest element
+if (oldestEnqueueTime > element.getEnqueueTime()) {
+ oldestEnqueueTime = element.getEnqueueTime();
+}
}
if (element.isSetStart()) {
-if (TxnStore.WORKING_RESPONSE.equals(state) && (oldestWorkingTime >
element.getStart())) {
- oldestWorkingTime = element.getStart();
+if (TxnStore.WORKING_RESPONSE.equals(state)) {
+ final int startSeconds = (int) ((currentTime - element.getStart()) /
1000);
+ oldestEnqueueTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? startSeconds : Math.max(startSeconds, old));
+
+ workingPoolCount.compute(element.getPoolName(), (k, old) -> (old ==
null) ? 1 : old + 1);
+ // find the oldest element
+ if (oldestWorkingTime > element.getStart()) {
+oldestWorkingTime = element.getStart();
+ }
}
}
if (element.isSetCleanerStart()) {
-if (TxnStore.CLEANING_RESPONSE.equals(state) && (oldestCleaningTime >
element.getCleanerStart())) {
- oldestCleaningTime = element.getCleanerStart();
+if (TxnStore.CLEANING_RESPONSE.equals(state)) {
Review Comment:
Just to have the same structure as above. I can revert it if you prefer
Issue Time Tracking
---
Worklog Id: (was: 815488)
Time Spent: 3h 40m (was: 3.5h)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815486
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 11/Oct/22 07:29
Start Date: 11/Oct/22 07:29
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3645:
URL: https://github.com/apache/hive/pull/3645#discussion_r991911089
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -67,34 +70,53 @@ static CompactionMetricData
of(List compacts) {
private void init() {
final Map lastElements = new
HashMap<>();
-poolCount = new HashMap<>();
+initiatedPoolCount = new HashMap<>();
+workingPoolCount = new HashMap<>();
+oldestEnqueueTimePerPool = new HashMap<>();
+oldestWorkingTimePerPool = new HashMap<>();
oldestEnqueueTime = OLDEST_TIME_NO_VALUE;
oldestWorkingTime = OLDEST_TIME_NO_VALUE;
oldestCleaningTime = OLDEST_TIME_NO_VALUE;
+
+long currentTime = System.currentTimeMillis();
for (ShowCompactResponseElement element : compacts) {
final String key = element.getDbname() + "/" + element.getTablename() +
(element.getPartitionname() != null ? "/" +
element.getPartitionname() : "");
// If new key, add the element, if there is an existing one, change to
the element if the element.id is greater than old.id
lastElements.compute(key, (k, old) -> (old == null) ? element :
(element.getId() > old.getId() ? element : old));
- // find the oldest elements with initiated and working states
String state = element.getState();
- if (TxnStore.INITIATED_RESPONSE.equals(state) && (oldestEnqueueTime >
element.getEnqueueTime())) {
-oldestEnqueueTime = element.getEnqueueTime();
-poolCount.compute(element.getPoolName(), (k, old) -> (old == null) ? 1
: old + 1);
+ if (TxnStore.INITIATED_RESPONSE.equals(state)) {
+final int enqueueSeconds = (int) ((currentTime -
element.getEnqueueTime()) / 1000);
+oldestWorkingTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? enqueueSeconds : Math.max(enqueueSeconds, old));
+
+initiatedPoolCount.compute(element.getPoolName(), (k, old) -> (old ==
null) ? 1 : old + 1);
+// find the oldest element
+if (oldestEnqueueTime > element.getEnqueueTime()) {
+ oldestEnqueueTime = element.getEnqueueTime();
+}
}
if (element.isSetStart()) {
-if (TxnStore.WORKING_RESPONSE.equals(state) && (oldestWorkingTime >
element.getStart())) {
- oldestWorkingTime = element.getStart();
+if (TxnStore.WORKING_RESPONSE.equals(state)) {
+ final int startSeconds = (int) ((currentTime - element.getStart()) /
1000);
+ oldestEnqueueTimePerPool.compute(element.getPoolName(), (k, old) ->
(old == null) ? startSeconds : Math.max(startSeconds, old));
+
+ workingPoolCount.compute(element.getPoolName(), (k, old) -> (old ==
null) ? 1 : old + 1);
+ // find the oldest element
+ if (oldestWorkingTime > element.getStart()) {
+oldestWorkingTime = element.getStart();
+ }
}
}
if (element.isSetCleanerStart()) {
-if (TxnStore.CLEANING_RESPONSE.equals(state) && (oldestCleaningTime >
element.getCleanerStart())) {
- oldestCleaningTime = element.getCleanerStart();
+if (TxnStore.CLEANING_RESPONSE.equals(state)) {
Review Comment:
why is this change?
Issue Time Tracking
---
Worklog Id: (was: 815486)
Time Spent: 3.5h (was: 3h 20m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815479 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 11/Oct/22 07:17 Start Date: 11/Oct/22 07:17 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3645: URL: https://github.com/apache/hive/pull/3645#discussion_r991899820 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/AcidMetricService.java: ## @@ -333,7 +336,10 @@ public static void updateMetricsFromShowCompact(ShowCompactResponse showCompactR } } - Metrics.getOrCreateMapMetrics(COMPACTION_POOLS_ITEM_COUNT).update(metricData.getPoolCount()); + Metrics.getOrCreateMapMetrics(COMPACTION_POOLS_INITIATED_ITEM_COUNT).update(metricData.getInitiatedPoolCount()); + Metrics.getOrCreateMapMetrics(COMPACTION_POOLS_WORKING_ITEM_COUNT).update(metricData.getWorkingPoolCount()); Review Comment: maybe, getWorkingCountPerPool? Issue Time Tracking --- Worklog Id: (was: 815479) Time Spent: 3h 20m (was: 3h 10m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=815478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-815478 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 11/Oct/22 07:16 Start Date: 11/Oct/22 07:16 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3645: URL: https://github.com/apache/hive/pull/3645#discussion_r991899417 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/AcidMetricService.java: ## @@ -333,7 +336,10 @@ public static void updateMetricsFromShowCompact(ShowCompactResponse showCompactR } } - Metrics.getOrCreateMapMetrics(COMPACTION_POOLS_ITEM_COUNT).update(metricData.getPoolCount()); + Metrics.getOrCreateMapMetrics(COMPACTION_POOLS_INITIATED_ITEM_COUNT).update(metricData.getInitiatedPoolCount()); Review Comment: maybe, getInitiatedCountPerPool to be consistent? Issue Time Tracking --- Worklog Id: (was: 815478) Time Spent: 3h 10m (was: 3h) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=814489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814489 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 06/Oct/22 20:13 Start Date: 06/Oct/22 20:13 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3645: URL: https://github.com/apache/hive/pull/3645#issuecomment-1270626099 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3645) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [66 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3645=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3645=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 814489) Time Spent: 3h (was: 2h 50m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 3h > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=814446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814446 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 06/Oct/22 16:54 Start Date: 06/Oct/22 16:54 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3645: URL: https://github.com/apache/hive/pull/3645#issuecomment-1270406568 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3645) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3645=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [66 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3645=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3645=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3645=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 814446) Time Spent: 2h 50m (was: 2h 40m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=814334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814334 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 06/Oct/22 11:24 Start Date: 06/Oct/22 11:24 Worklog Time Spent: 10m Work Description: veghlaci05 opened a new pull request, #3645: URL: https://github.com/apache/hive/pull/3645 ### What changes were proposed in this pull request? Fix the metric computing by counting all the INITIATED queue items. ### Why are the changes needed? The metric computing introduced was bad because only elements matching the oldestEnqueueTime > element.getEnqueueTime() were counted. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually Issue Time Tracking --- Worklog Id: (was: 814334) Time Spent: 2h 40m (was: 2.5h) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809957 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 19/Sep/22 07:45 Start Date: 19/Sep/22 07:45 Worklog Time Spent: 10m Work Description: lcspinter merged PR #3521: URL: https://github.com/apache/hive/pull/3521 Issue Time Tracking --- Worklog Id: (was: 809957) Time Spent: 2.5h (was: 2h 20m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809540
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 14:04
Start Date: 16/Sep/22 14:04
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r973062171
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -148,6 +152,10 @@ Map getStateCount() {
return new HashMap<>(stateCount);
}
+ public Map getPoolCount() {
+return new HashMap<>(poolCount);
Review Comment:
I think it is enough if only `CompactionMetricData` is immutable. The
returned shallow copy allows the caller to modify these collections without
modifying CompactionMetricData itself, which is fine in my opinion. However, in
the current case both methods are used only one time making this discussion
rather theoretical. I simply copied the way the already existing method works.
Issue Time Tracking
---
Worklog Id: (was: 809540)
Time Spent: 2h 20m (was: 2h 10m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809535 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 16/Sep/22 13:33 Start Date: 16/Sep/22 13:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3521: URL: https://github.com/apache/hive/pull/3521#issuecomment-1249371926 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3521) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3521=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3521=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 809535) Time Spent: 2h 10m (was: 2h) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809517
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 12:53
Start Date: 16/Sep/22 12:53
Worklog Time Spent: 10m
Work Description: lcspinter commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r972989266
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -148,6 +152,10 @@ Map getStateCount() {
return new HashMap<>(stateCount);
}
+ public Map getPoolCount() {
+return new HashMap<>(poolCount);
Review Comment:
If the statecount and poolcount are meant to be immutable, why don't we
return an immutable hashmap?
Issue Time Tracking
---
Worklog Id: (was: 809517)
Time Spent: 2h (was: 1h 50m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809506=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809506
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 12:01
Start Date: 16/Sep/22 12:01
Worklog Time Spent: 10m
Work Description: zsmiskolczi commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r972949807
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -148,6 +152,10 @@ Map getStateCount() {
return new HashMap<>(stateCount);
}
+ public Map getPoolCount() {
+return new HashMap<>(poolCount);
Review Comment:
Good to know, thank you.
Issue Time Tracking
---
Worklog Id: (was: 809506)
Time Spent: 1h 50m (was: 1h 40m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809505
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 12:01
Start Date: 16/Sep/22 12:01
Worklog Time Spent: 10m
Work Description: zsmiskolczi commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r972949618
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -45,6 +45,8 @@ final class CompactionMetricData {
private Map stateCount;
+ private Map poolCount;
Review Comment:
Great, thank you.
Issue Time Tracking
---
Worklog Id: (was: 809505)
Time Spent: 1h 40m (was: 1.5h)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809504=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809504
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 11:56
Start Date: 16/Sep/22 11:56
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r972945999
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -148,6 +152,10 @@ Map getStateCount() {
return new HashMap<>(stateCount);
}
+ public Map getPoolCount() {
+return new HashMap<>(poolCount);
Review Comment:
AS far as I can see `CompactionMetricData` is designed to be immutable,
stateCount returned the same way
`return new HashMap<>(stateCount);`
I did not want to break immutability.
Issue Time Tracking
---
Worklog Id: (was: 809504)
Time Spent: 1.5h (was: 1h 20m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809503=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809503
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 11:51
Start Date: 16/Sep/22 11:51
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r972942476
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -45,6 +45,8 @@ final class CompactionMetricData {
private Map stateCount;
+ private Map poolCount;
Review Comment:
That is totally impossible, compaction related queries would start to fail
much-much earlier. Even a few thousand unprocessed compaction entries mean a
serious issue and start affecting performance.
Issue Time Tracking
---
Worklog Id: (was: 809503)
Time Spent: 1h 20m (was: 1h 10m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809499
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 16/Sep/22 11:42
Start Date: 16/Sep/22 11:42
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r972929518
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -148,6 +152,10 @@ Map getStateCount() {
return new HashMap<>(stateCount);
}
+ public Map getPoolCount() {
+return new HashMap<>(poolCount);
Review Comment:
I wonder, what is the reason of copying the poolCount map instead of passing
it to the caller?
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/CompactionMetricData.java:
##
@@ -45,6 +45,8 @@ final class CompactionMetricData {
private Map stateCount;
+ private Map poolCount;
Review Comment:
I wonder if the counter can be Long or not. I mean, the `CC_ID` column in
the meta-store database is big int. I don't know our usage statistics but I
wonder: it is possible to have so many compaction requests in pool that Integer
is not enough?
Issue Time Tracking
---
Worklog Id: (was: 809499)
Time Spent: 1h 10m (was: 1h)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809085 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 15/Sep/22 13:09 Start Date: 15/Sep/22 13:09 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3521: URL: https://github.com/apache/hive/pull/3521#issuecomment-1248080505 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3521) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3521=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3521=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 809085) Time Spent: 1h (was: 50m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=809003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-809003 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 15/Sep/22 08:25 Start Date: 15/Sep/22 08:25 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3521: URL: https://github.com/apache/hive/pull/3521#issuecomment-1247758611 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3521) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3521=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3521=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 809003) Time Spent: 50m (was: 40m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=807386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807386 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 09/Sep/22 12:42 Start Date: 09/Sep/22 12:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3521: URL: https://github.com/apache/hive/pull/3521#issuecomment-1241930687 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3521) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3521=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3521=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 807386) Time Spent: 40m (was: 0.5h) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=807125=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807125 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 08/Sep/22 17:33 Start Date: 08/Sep/22 17:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3521: URL: https://github.com/apache/hive/pull/3521#issuecomment-1241024059 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3521) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3521=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3521=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3521=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3521=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 807125) Time Spent: 0.5h (was: 20m) > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[
https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=807028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-807028
]
ASF GitHub Bot logged work on HIVE-26471:
-
Author: ASF GitHub Bot
Created on: 08/Sep/22 13:11
Start Date: 08/Sep/22 13:11
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on code in PR #3521:
URL: https://github.com/apache/hive/pull/3521#discussion_r965933976
##
common/src/java/org/apache/hadoop/hive/conf/RegexMatchingConfEntry.java:
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.conf;
+
+import java.util.regex.Matcher;
+
+public class RegexMatchingConfEntry {
Review Comment:
Unused class
Issue Time Tracking
---
Worklog Id: (was: 807028)
Time Spent: 20m (was: 10m)
> New metric for Compaction pooling
> -
>
> Key: HIVE-26471
> URL: https://issues.apache.org/jira/browse/HIVE-26471
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> To be able to properly supervise the pool based compaction, a new metric is
> required:
> Number of 'Initiated' compaction requests per compaction pool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26471) New metric for Compaction pooling
[ https://issues.apache.org/jira/browse/HIVE-26471?focusedWorklogId=800597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-800597 ] ASF GitHub Bot logged work on HIVE-26471: - Author: ASF GitHub Bot Created on: 15/Aug/22 13:37 Start Date: 15/Aug/22 13:37 Worklog Time Spent: 10m Work Description: veghlaci05 opened a new pull request, #3521: URL: https://github.com/apache/hive/pull/3521 ### What changes were proposed in this pull request? A new metric is added: Number of 'Initiated' compaction requests per compaction pool. ### Why are the changes needed? To be able to properly supervise the pool based compaction. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually and through unit tests Issue Time Tracking --- Worklog Id: (was: 800597) Remaining Estimate: 0h Time Spent: 10m > New metric for Compaction pooling > - > > Key: HIVE-26471 > URL: https://issues.apache.org/jira/browse/HIVE-26471 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > To be able to properly supervise the pool based compaction, a new metric is > required: > Number of 'Initiated' compaction requests per compaction pool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
