[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=439911&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439911 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 02/Jun/20 06:11 Start Date: 02/Jun/20 06:11 Worklog Time Spent: 10m Work Description: pvary closed pull request #633: URL: https://github.com/apache/hive/pull/633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439911) Time Spent: 6h (was: 5h 50m) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch, > HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.6.patch, HIVE-21740.patch > > Time Spent: 6h > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251657
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 09:33
Start Date: 31/May/19 09:33
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289320836
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3160,22 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for successful map tasks
+if (!TaskAttemptState.SUCCEEDED.equals(taskAttempt.getState()) ||
!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
Review comment:
I am not yet entirely familiar with this part of the code.
I thought that the vertex can tell us more but getVertex is package private
method for TaskAttemptImp, and getting the Vertex from would need something
like this (found in getTransitiveVertexOutputs):
--
DagInfo info = getContext().getCurrentDagInfo();
if (!(info instanceof DAG)) {
LOG.warn("DAG info is not a DAG");
return;
}
DAG dag = (DAG) info;
Vertex vertex = dag.getVertex(taskAttempt.getVertexID());
--
I found casting DagInfo to DAG more shady than relying on counters, but feel
free to disagree.
Also open to any suggestions where should I dig around more to find a better
solution!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251657)
Time Spent: 5h 40m (was: 5.5h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251659
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 09:33
Start Date: 31/May/19 09:33
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289320836
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3160,22 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for successful map tasks
+if (!TaskAttemptState.SUCCEEDED.equals(taskAttempt.getState()) ||
!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
Review comment:
I am not yet entirely familiar with this part of the code.
I thought that the vertex can tell us more but getVertex is package private
method for TaskAttemptImp, and getting the Vertex from would need something
like this (found in getTransitiveVertexOutputs):
--
DagInfo info = getContext().getCurrentDagInfo();
if (!(info instanceof DAG)) {
LOG.warn("DAG info is not a DAG");
return;
}
DAG dag = (DAG) info;
Vertex vertex = dag.getVertex(taskAttempt.getVertexID());
--
I found casting DagInfo to DAG more shady than relying on counters, but feel
free to disagree.
Also open to any suggestions where should I dig around more to find a better
solution!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251659)
Time Spent: 5h 50m (was: 5h 40m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251656
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 09:33
Start Date: 31/May/19 09:33
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289320836
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3160,22 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for successful map tasks
+if (!TaskAttemptState.SUCCEEDED.equals(taskAttempt.getState()) ||
!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
Review comment:
I am not yet entirely familiar with this part of the code.
I thought that the vertex can tell us more but getVertex is package private
method for TaskAttemptImp, and getting the Vertex from would need something
like this (found in getTransitiveVertexOutputs):
DagInfo info = getContext().getCurrentDagInfo();
if (!(info instanceof DAG)) {
LOG.warn("DAG info is not a DAG");
return;
}
DAG dag = (DAG) info;
Vertex vertex = dag.getVertex(taskAttempt.getVertexID());
--
I found casting DagInfo to DAG more shady than relying on counters, but feel
free to disagree.
Also open to any suggestions where should I dig around more to find a better
solution!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251656)
Time Spent: 5.5h (was: 5h 20m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251649&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251649
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 09:22
Start Date: 31/May/19 09:22
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289317467
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -1175,6 +1178,9 @@ public boolean deallocateTask(
LOG.debug("Processing deallocateTask for task={}, taskSucceeded={},
endReason={}", task,
taskSucceeded, endReason);
}
+if (task instanceof TaskAttemptImpl && metrics != null) {
+ updateMetrics((TaskAttemptImpl)task);
Review comment:
The metrics can be null in case of
conf.getBoolean(ConfVars.HIVE_IN_TEST.varname, false)
I tried to find the style problem, but not sure which part should be written
differently
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251649)
Time Spent: 5h 20m (was: 5h 10m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251607
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 08:00
Start Date: 31/May/19 08:00
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289292223
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -254,6 +266,14 @@ public void setWmUnusedGuaranteed(int unusedGuaranteed) {
wmUnusedGuaranteedCount.set(unusedGuaranteed);
}
+ public void addTaskLatency(String daemonId, long value) {
Review comment:
Done.
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251607)
Time Spent: 5h (was: 4h 50m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251608&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251608
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 08:00
Start Date: 31/May/19 08:00
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289292314
##
File path: hive-site.xml
##
@@ -0,0 +1,358 @@
+
+
+
+
+
+ hive.server2.thrift.max.worker.threads
+ 1
+
+
+ hive.server2.thrift.min.worker.threads
+ 1
+
+
+ datanucleus.deletionPolicy
+ DataNucleus
+
+
+
+
+ hive.jar.path
+
/Users/petervary/dev/upstream/hive/ql/target/hive-exec-4.0.0-SNAPSHOT.jar
+ The location of hive_cli.jar that is used when submitting jobs
in a separate jvm.
+
+
+ hive.hadoop.classpath
+
/Users/petervary/dev/upstream/hive/ql/target/hive-exec-4.0.0-SNAPSHOT.jar
+
+
+ hive.metastore.local
+ false
+
+
+ metastore.thrift.uris
+ thrift://localhost:9084
+
+
+ metastore.thrift.port
+ 9084
+
+
+ hive.server2.thrift.port
+ 10003
+
+
+ hive.server2.webui.port
+ 10004
+
+
+ metastore.warehouse.dir
+ pfile:///Users/petervary/data/apache/hive/warehouse
+
+
+ fs.pfile.impl
+ org.apache.hadoop.fs.ProxyLocalFileSystem
+ A proxy for local file system used for cross file system
testing
+
+
+ metastore.expression.proxy
+
org.apache.hadoop.hive.metastore.DefaultPartitionExpressionProxy
+
+
+ metastore.task.threads.always
+ org.apache.hadoop.hive.metastore.events.EventCleanerTask
+
+
+
+
+
+
+
+
+ hive.metastore.schema.verification
+ false
+
+
+
+datanucleus.autoCreateTables
+true
+
+
+
+datanucleus.schema.autoCreateAll
+true
+
+
+
+
+metastore.metastore.event.db.notification.api.auth
+false
+
+
+
+
+
+
+
+ hive.exec.scratchdir
+ /tmp/hive-${user.name}
+
+
+
+
+
+
+
+
+
+
+
+ javax.jdo.option.ConnectionURL
+
+ jdbc:mysql://localhost/upstream_hive
+ JDBC connect string for a JDBC metastore
+
+
+
+ hive.metastore.try.direct.sql
+ true
+
+
+
+ javax.jdo.option.ConnectionDriverName
+ com.mysql.jdbc.Driver
+
+
+
+ javax.jdo.option.ConnectionUserName
+ upstream_hive
+
+
+
+ javax.jdo.option.ConnectionPassword
+ hive
+
+
+
+ hive.server2.enable.doAs
+ false
+
+
+
+
+ hive.server2.enable.impersonation
+ false
+
+
+
+
+ dfs.namenode.acls.enabled
+ false
+
+
+
+
+
+
+
+
+
+
+
+
+
Review comment:
Removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251608)
Time Spent: 5h 10m (was: 5h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251604 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 31/May/19 07:54 Start Date: 31/May/19 07:54 Worklog Time Spent: 10m Work Description: pvary commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r289290466 ## File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java ## @@ -426,9 +428,10 @@ public LlapTaskSchedulerService(TaskSchedulerContext taskSchedulerContext, Clock this.pauseMonitor = new JvmPauseMonitor(conf); pauseMonitor.start(); String displayName = "LlapTaskSchedulerMetrics-" + MetricsUtils.getHostName(); + int latencyMetricWindowSize = HiveConf.getIntVar(conf, ConfVars.LLAP_LATENCY_METRIC_WINDOW_SIZE); Review comment: Added a new feature to turn off this metrics if this value is 0, or negative number. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251604) Time Spent: 4h 50m (was: 4h 40m) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch, > HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251587&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251587 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 31/May/19 07:32 Start Date: 31/May/19 07:32 Worklog Time Spent: 10m Work Description: pvary commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r289284589 ## File path: llap-common/src/test/org/apache/hadoop/hive/llap/metrics/MockMetricsCollector.java ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.llap.metrics; + +import com.google.common.collect.Lists; +import org.apache.hadoop.metrics2.AbstractMetric; +import org.apache.hadoop.metrics2.MetricsCollector; +import org.apache.hadoop.metrics2.MetricsInfo; +import org.apache.hadoop.metrics2.MetricsRecordBuilder; +import org.apache.hadoop.metrics2.MetricsTag; +import org.apache.hadoop.metrics2.lib.Interns; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Mock metrics collector for this test only. + * This MetricsCollector implementation is used to get the actual + * MetricsSource data, collected by the Review comment: I guess it does not matter this case, but done it anyway since I doing a new patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251587) Time Spent: 4h 40m (was: 4.5h) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch, > HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251578&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251578
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 07:25
Start Date: 31/May/19 07:25
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289282970
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4438,6 +4438,8 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_COLLECT_LOCK_METRICS("hive.llap.lockmetrics.collect", false,
"Whether lock metrics (wait times, counts) are collected for LLAP "
+ "related locks"),
+LLAP_LATENCY_METRIC_WINDOW_SIZE("hive.llap.metrics.latency.window.size",
2048,
Review comment:
By severeal TCPDS tests it seems that we require minimally around 1000
measurements to have a steady result for the average. Set it to 2k to be sure
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251578)
Time Spent: 4.5h (was: 4h 20m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251577 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 31/May/19 07:24 Start Date: 31/May/19 07:24 Worklog Time Spent: 10m Work Description: pvary commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r289282673 ## File path: hive-site.xml ## @@ -0,0 +1,358 @@ + Review comment: Added by mistake. Removed. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251577) Time Spent: 4h 20m (was: 4h 10m) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch, > HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251575
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 31/May/19 07:22
Start Date: 31/May/19 07:22
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289282184
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -254,6 +272,14 @@ public void setWmUnusedGuaranteed(int unusedGuaranteed) {
wmUnusedGuaranteedCount.set(unusedGuaranteed);
}
+ public void addTaskLatency(String key, long value) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251575)
Time Spent: 4h 10m (was: 4h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251279 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 30/May/19 20:16 Start Date: 30/May/19 20:16 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r289142935 ## File path: hive-site.xml ## @@ -0,0 +1,358 @@ + Review comment: what's this file needed for ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251279) Time Spent: 3h 10m (was: 3h) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch, > HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251286&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251286
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 30/May/19 20:16
Start Date: 30/May/19 20:16
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289145945
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -1175,6 +1178,9 @@ public boolean deallocateTask(
LOG.debug("Processing deallocateTask for task={}, taskSucceeded={},
endReason={}", task,
taskSucceeded, endReason);
}
+if (task instanceof TaskAttemptImpl && metrics != null) {
+ updateMetrics((TaskAttemptImpl)task);
Review comment:
style is off and not sure what case metrics is null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251286)
Time Spent: 4h (was: 3h 50m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251285
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 30/May/19 20:16
Start Date: 30/May/19 20:16
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289146624
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3160,22 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for successful map tasks
+if (!TaskAttemptState.SUCCEEDED.equals(taskAttempt.getState()) ||
!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
Review comment:
is this really the best way to find if this is a map task ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251285)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251283&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251283
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 30/May/19 20:16
Start Date: 30/May/19 20:16
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289153854
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -254,6 +266,14 @@ public void setWmUnusedGuaranteed(int unusedGuaranteed) {
wmUnusedGuaranteedCount.set(unusedGuaranteed);
}
+ public void addTaskLatency(String daemonId, long value) {
Review comment:
nit. this can be much simpler
```java
concurrentHashMap.computeIfAbsent("key", k -> new
DaemonLatencyMetric()).add(val);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251283)
Time Spent: 3h 40m (was: 3.5h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251284
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 30/May/19 20:16
Start Date: 30/May/19 20:16
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289154247
##
File path: hive-site.xml
##
@@ -0,0 +1,358 @@
+
+
+
+
+
+ hive.server2.thrift.max.worker.threads
+ 1
+
+
+ hive.server2.thrift.min.worker.threads
+ 1
+
+
+ datanucleus.deletionPolicy
+ DataNucleus
+
+
+
+
+ hive.jar.path
+
/Users/petervary/dev/upstream/hive/ql/target/hive-exec-4.0.0-SNAPSHOT.jar
+ The location of hive_cli.jar that is used when submitting jobs
in a separate jvm.
+
+
+ hive.hadoop.classpath
+
/Users/petervary/dev/upstream/hive/ql/target/hive-exec-4.0.0-SNAPSHOT.jar
+
+
+ hive.metastore.local
+ false
+
+
+ metastore.thrift.uris
+ thrift://localhost:9084
+
+
+ metastore.thrift.port
+ 9084
+
+
+ hive.server2.thrift.port
+ 10003
+
+
+ hive.server2.webui.port
+ 10004
+
+
+ metastore.warehouse.dir
+ pfile:///Users/petervary/data/apache/hive/warehouse
+
+
+ fs.pfile.impl
+ org.apache.hadoop.fs.ProxyLocalFileSystem
+ A proxy for local file system used for cross file system
testing
+
+
+ metastore.expression.proxy
+
org.apache.hadoop.hive.metastore.DefaultPartitionExpressionProxy
+
+
+ metastore.task.threads.always
+ org.apache.hadoop.hive.metastore.events.EventCleanerTask
+
+
+
+
+
+
+
+
+ hive.metastore.schema.verification
+ false
+
+
+
+datanucleus.autoCreateTables
+true
+
+
+
+datanucleus.schema.autoCreateAll
+true
+
+
+
+
+metastore.metastore.event.db.notification.api.auth
+false
+
+
+
+
+
+
+
+ hive.exec.scratchdir
+ /tmp/hive-${user.name}
+
+
+
+
+
+
+
+
+
+
+
+ javax.jdo.option.ConnectionURL
+
+ jdbc:mysql://localhost/upstream_hive
+ JDBC connect string for a JDBC metastore
+
+
+
+ hive.metastore.try.direct.sql
+ true
+
+
+
+ javax.jdo.option.ConnectionDriverName
+ com.mysql.jdbc.Driver
+
+
+
+ javax.jdo.option.ConnectionUserName
+ upstream_hive
+
+
+
+ javax.jdo.option.ConnectionPassword
+ hive
+
+
+
+ hive.server2.enable.doAs
+ false
+
+
+
+
+ hive.server2.enable.impersonation
+ false
+
+
+
+
+ dfs.namenode.acls.enabled
+ false
+
+
+
+
+
+
+
+
+
+
+
+
+
Review comment:
if this is for testing please added as a resource.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251284)
Time Spent: 3h 50m (was: 3h 40m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251282&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251282
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 30/May/19 20:16
Start Date: 30/May/19 20:16
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289144370
##
File path:
llap-common/src/test/org/apache/hadoop/hive/llap/metrics/MockMetricsCollector.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.metrics;
+
+import com.google.common.collect.Lists;
+import org.apache.hadoop.metrics2.AbstractMetric;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsTag;
+import org.apache.hadoop.metrics2.lib.Interns;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Mock metrics collector for this test only.
+ * This MetricsCollector implementation is used to get the actual
+ * MetricsSource data, collected by the
Review comment:
nit. minor recommend using `{@code text}` as per the [Java
Docs](https://docs.oracle.com/javase/1.5.0/docs/tooldocs/windows/javadoc.html#%7B@code%7D)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251282)
Time Spent: 3h 40m (was: 3.5h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251281
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 30/May/19 20:16
Start Date: 30/May/19 20:16
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r289143060
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4438,6 +4438,8 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_COLLECT_LOCK_METRICS("hive.llap.lockmetrics.collect", false,
"Whether lock metrics (wait times, counts) are collected for LLAP "
+ "related locks"),
+LLAP_LATENCY_METRIC_WINDOW_SIZE("hive.llap.metrics.latency.window.size",
2048,
Review comment:
any clue on why 2k is the default ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251281)
Time Spent: 3.5h (was: 3h 20m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch,
> HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=251280&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251280 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 30/May/19 20:16 Start Date: 30/May/19 20:16 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r289145568 ## File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java ## @@ -426,9 +428,10 @@ public LlapTaskSchedulerService(TaskSchedulerContext taskSchedulerContext, Clock this.pauseMonitor = new JvmPauseMonitor(conf); pauseMonitor.start(); String displayName = "LlapTaskSchedulerMetrics-" + MetricsUtils.getHostName(); + int latencyMetricWindowSize = HiveConf.getIntVar(conf, ConfVars.LLAP_LATENCY_METRIC_WINDOW_SIZE); Review comment: what happen if this is set to `< 0` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251280) Time Spent: 3h 20m (was: 3h 10m) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.3.patch, > HIVE-21740.4.patch, HIVE-21740.5.patch, HIVE-21740.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248813&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248813
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:50
Start Date: 27/May/19 12:50
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r287778469
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -276,6 +302,43 @@ private void getTaskSchedulerStats(MetricsRecordBuilder
rb) {
.addCounter(SchedulerPendingPreemptionTaskCount,
pendingPreemptionTasksCount.value())
.addCounter(SchedulerPreemptedTaskCount, preemptedTasksCount.value())
.addCounter(SchedulerCompletedDagCount, completedDagcount.value());
+daemonTaskLatency.forEach((k, v) -> rb.addGauge(v, v.getMean()));
+ }
+
+ static class DaemonLatencyMetric implements MetricsInfo {
+private String name;
+private ExponentiallyDecayingReservoir reservoir;
Review comment:
Do not like Guava, and Stats appeared in guava 20. We are on 19 now and
upgrading can cause problems with other projects like hadoop.
Moved to using apache math DescriptiveStatistics instead.
Also moved to simple windowing average instead of DecayMetrics
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248813)
Time Spent: 3h (was: 2h 50m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248812
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:50
Start Date: 27/May/19 12:50
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r287778469
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -276,6 +302,43 @@ private void getTaskSchedulerStats(MetricsRecordBuilder
rb) {
.addCounter(SchedulerPendingPreemptionTaskCount,
pendingPreemptionTasksCount.value())
.addCounter(SchedulerPreemptedTaskCount, preemptedTasksCount.value())
.addCounter(SchedulerCompletedDagCount, completedDagcount.value());
+daemonTaskLatency.forEach((k, v) -> rb.addGauge(v, v.getMean()));
+ }
+
+ static class DaemonLatencyMetric implements MetricsInfo {
+private String name;
+private ExponentiallyDecayingReservoir reservoir;
Review comment:
Do not like Guava, and Stats appeared in guava 20. We are on 19 now and
upgrading can cause problems with other projects like hadoop.
Moved to using apache math DescriptiveStatistics instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248812)
Time Spent: 2h 50m (was: 2h 40m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248810&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248810
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:48
Start Date: 27/May/19 12:48
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r28851
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -276,6 +302,43 @@ private void getTaskSchedulerStats(MetricsRecordBuilder
rb) {
.addCounter(SchedulerPendingPreemptionTaskCount,
pendingPreemptionTasksCount.value())
.addCounter(SchedulerPreemptedTaskCount, preemptedTasksCount.value())
.addCounter(SchedulerCompletedDagCount, completedDagcount.value());
+daemonTaskLatency.forEach((k, v) -> rb.addGauge(v, v.getMean()));
Review comment:
We need to add them as a gauge so it will show up on the counter /JMX list
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248810)
Time Spent: 2h 40m (was: 2.5h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248809 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 27/May/19 12:47 Start Date: 27/May/19 12:47 Worklog Time Spent: 10m Work Description: pvary commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r28583 ## File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java ## @@ -55,6 +61,9 @@ private final JvmMetrics jvmMetrics; private final String sessionId; private final MetricsRegistry registry; + private final int decayMetricSampleSize; + private final double decayMetricAlphaFactor; + private Map daemonTaskLatency = new ConcurrentHashMap<>(); Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248809) Time Spent: 2.5h (was: 2h 20m) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248808&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248808
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:46
Start Date: 27/May/19 12:46
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r28380
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3165,28 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for map tasks
+if (!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
+boolean isMapTask = false;
+for(TezCounter counter : taskAttempt.getCounters().getGroup("HIVE")) {
+ if(counter.getName().startsWith("RECORDS_IN_Map")) {
+isMapTask = true;
Review comment:
Optimized the counter based solution as suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248808)
Time Spent: 2h 20m (was: 2h 10m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248807
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:46
Start Date: 27/May/19 12:46
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r28340
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3165,28 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for map tasks
+if (!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
+boolean isMapTask = false;
+for(TezCounter counter : taskAttempt.getCounters().getGroup("HIVE")) {
Review comment:
Accessing vertex data seems even more complicated.
Optimized the counter based solution as suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248807)
Time Spent: 2h 10m (was: 2h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248805 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 27/May/19 12:45 Start Date: 27/May/19 12:45 Worklog Time Spent: 10m Work Description: pvary commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r28099 ## File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java ## @@ -426,9 +428,12 @@ public LlapTaskSchedulerService(TaskSchedulerContext taskSchedulerContext, Clock this.pauseMonitor = new JvmPauseMonitor(conf); pauseMonitor.start(); String displayName = "LlapTaskSchedulerMetrics-" + MetricsUtils.getHostName(); + int decayMetricSampleSize = HiveConf.getIntVar(conf, ConfVars.LLAP_DECAY_METRIC_SIZE); + double decayMetricAlphaFactor = (double) HiveConf.getFloatVar(conf, ConfVars.LLAP_DECAY_METRIC_ALPHA); Review comment: Not needed anymore This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248805) Time Spent: 2h (was: 1h 50m) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.patch > > Time Spent: 2h > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248804&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248804
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:45
Start Date: 27/May/19 12:45
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r28002
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4438,6 +4438,10 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_COLLECT_LOCK_METRICS("hive.llap.lockmetrics.collect", false,
"Whether lock metrics (wait times, counts) are collected for LLAP "
+ "related locks"),
+LLAP_DECAY_METRIC_SIZE("hive.llap.metrics.decay.size", 1028,
+"The number of samples to keep in the sampling reservoir"),
+LLAP_DECAY_METRIC_ALPHA("hive.llap.metrics.decay.alpha", 0.015f,
+"Exponential decay factor; higher is more biased towards newer
values"),
Review comment:
Removed the alpha since we moved from Decaying to simple window
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248804)
Time Spent: 1h 50m (was: 1h 40m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=248803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248803
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 27/May/19 12:42
Start Date: 27/May/19 12:42
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r287776346
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4438,6 +4438,10 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_COLLECT_LOCK_METRICS("hive.llap.lockmetrics.collect", false,
"Whether lock metrics (wait times, counts) are collected for LLAP "
+ "related locks"),
+LLAP_DECAY_METRIC_SIZE("hive.llap.metrics.decay.size", 1028,
Review comment:
Updated to 2048. Based on my tests on TCPDS it should be ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248803)
Time Spent: 1h 40m (was: 1.5h)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245426&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245426
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285701027
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4438,6 +4438,10 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_COLLECT_LOCK_METRICS("hive.llap.lockmetrics.collect", false,
"Whether lock metrics (wait times, counts) are collected for LLAP "
+ "related locks"),
+LLAP_DECAY_METRIC_SIZE("hive.llap.metrics.decay.size", 1028,
Review comment:
We need to test/verify is 1028 samples delivers us a stable enough average.
Also, why this 1028?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245426)
Time Spent: 0.5h (was: 20m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245431
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285709384
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3165,28 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for map tasks
+if (!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
+boolean isMapTask = false;
+for(TezCounter counter : taskAttempt.getCounters().getGroup("HIVE")) {
Review comment:
I am sure, there are more efficient ways to figure out, if this is a Map
task. The CounterGroupBase for example has a findCounter, therefore iterating
all counters of a group to find a specific one, is not necessary. Maybe,
instead of looking at the counters, there are other indicators (i.e. at the
attached Vertex), giving away if this a Map task.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245431)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245433&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245433
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285714898
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -276,6 +302,43 @@ private void getTaskSchedulerStats(MetricsRecordBuilder
rb) {
.addCounter(SchedulerPendingPreemptionTaskCount,
pendingPreemptionTasksCount.value())
.addCounter(SchedulerPreemptedTaskCount, preemptedTasksCount.value())
.addCounter(SchedulerCompletedDagCount, completedDagcount.value());
+daemonTaskLatency.forEach((k, v) -> rb.addGauge(v, v.getMean()));
+ }
+
+ static class DaemonLatencyMetric implements MetricsInfo {
+private String name;
+private ExponentiallyDecayingReservoir reservoir;
Review comment:
What is the benefit of using am exponential decay here, vs. simple sliding
window (i.e. guava stats)? Specifically with the low amount of entries (right
now 1028), a few more recent task execution times can have a huge impact on
what we believe is the node's average over a longer period of time. The whole
idea of using the average here is to smooth out the task execution times across
a larger amount of tasks. I have the fear, that the exponential decay is
contradicting this by giving few, very recent values, a high impact.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245433)
Time Spent: 1.5h (was: 1h 20m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245425&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245425
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285701292
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4438,6 +4438,10 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
LLAP_COLLECT_LOCK_METRICS("hive.llap.lockmetrics.collect", false,
"Whether lock metrics (wait times, counts) are collected for LLAP "
+ "related locks"),
+LLAP_DECAY_METRIC_SIZE("hive.llap.metrics.decay.size", 1028,
+"The number of samples to keep in the sampling reservoir"),
+LLAP_DECAY_METRIC_ALPHA("hive.llap.metrics.decay.alpha", 0.015f,
+"Exponential decay factor; higher is more biased towards newer
values"),
Review comment:
Listing valid ranges for alpha would help.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245425)
Time Spent: 20m (was: 10m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245432&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245432
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285712289
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -254,6 +272,14 @@ public void setWmUnusedGuaranteed(int unusedGuaranteed) {
wmUnusedGuaranteedCount.set(unusedGuaranteed);
}
+ public void addTaskLatency(String key, long value) {
Review comment:
Rename key to daemonID and/or add comment, describing the parameters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245432)
Time Spent: 1h 20m (was: 1h 10m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245427&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245427 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 20/May/19 18:34 Start Date: 20/May/19 18:34 Worklog Time Spent: 10m Work Description: odraese commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r285704254 ## File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java ## @@ -426,9 +428,12 @@ public LlapTaskSchedulerService(TaskSchedulerContext taskSchedulerContext, Clock this.pauseMonitor = new JvmPauseMonitor(conf); pauseMonitor.start(); String displayName = "LlapTaskSchedulerMetrics-" + MetricsUtils.getHostName(); + int decayMetricSampleSize = HiveConf.getIntVar(conf, ConfVars.LLAP_DECAY_METRIC_SIZE); + double decayMetricAlphaFactor = (double) HiveConf.getFloatVar(conf, ConfVars.LLAP_DECAY_METRIC_ALPHA); Review comment: Suggestion: Could be replaced with conf.getDouble(ConfVars.LLAP_DECAY_METRIC_ALPHA.varname); This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245427) Time Spent: 40m (was: 0.5h) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245428
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285709679
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -3154,4 +3165,28 @@ public void taskInfoUpdated(TezTaskAttemptID attemptId,
boolean isGuaranteed) {
+ attemptId + ", " + newState);
sendUpdateMessageAsync(ti, newState);
}
+
+ private void updateMetrics(TaskAttemptImpl taskAttempt) {
+// Only do it for map tasks
+if (!isMapTask(taskAttempt)) {
+ return;
+}
+// Check if this task was already assigned to a node
+NodeInfo nodeInfo = knownTasks.get(taskAttempt).assignedNode;
+if (nodeInfo == null) {
+ return;
+}
+
+metrics.addTaskLatency(nodeInfo.shortStringBase,
taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
+ }
+
+ private boolean isMapTask(TaskAttemptImpl taskAttempt) {
+boolean isMapTask = false;
+for(TezCounter counter : taskAttempt.getCounters().getGroup("HIVE")) {
+ if(counter.getName().startsWith("RECORDS_IN_Map")) {
+isMapTask = true;
Review comment:
and if you prefer to iterate the group, you should break here instead of
comparing all counters to the end.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245428)
Time Spent: 50m (was: 40m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245430&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245430 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 20/May/19 18:34 Start Date: 20/May/19 18:34 Worklog Time Spent: 10m Work Description: odraese commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633#discussion_r285711920 ## File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java ## @@ -55,6 +61,9 @@ private final JvmMetrics jvmMetrics; private final String sessionId; private final MetricsRegistry registry; + private final int decayMetricSampleSize; + private final double decayMetricAlphaFactor; + private Map daemonTaskLatency = new ConcurrentHashMap<>(); Review comment: a comment, telling what the key is, would help. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245430) Time Spent: 1h 10m (was: 1h) > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[
https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=245429&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245429
]
ASF GitHub Bot logged work on HIVE-21740:
-
Author: ASF GitHub Bot
Created on: 20/May/19 18:34
Start Date: 20/May/19 18:34
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #633: HIVE-21740:
Collect LLAP execution latency metrics
URL: https://github.com/apache/hive/pull/633#discussion_r285713028
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapTaskSchedulerMetrics.java
##
@@ -276,6 +302,43 @@ private void getTaskSchedulerStats(MetricsRecordBuilder
rb) {
.addCounter(SchedulerPendingPreemptionTaskCount,
pendingPreemptionTasksCount.value())
.addCounter(SchedulerPreemptedTaskCount, preemptedTasksCount.value())
.addCounter(SchedulerCompletedDagCount, completedDagcount.value());
+daemonTaskLatency.forEach((k, v) -> rb.addGauge(v, v.getMean()));
Review comment:
Not sure that I understand the purpose of this line (storing the
DaemonLatencyMetric as an own gauge upon reset)...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245429)
Time Spent: 1h (was: 50m)
> Collect LLAP execution latency metrics
> --
>
> Key: HIVE-21740
> URL: https://issues.apache.org/jira/browse/HIVE-21740
> Project: Hive
> Issue Type: New Feature
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21740.2.patch, HIVE-21740.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Collect metrics for LLAP task execution times
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21740) Collect LLAP execution latency metrics
[ https://issues.apache.org/jira/browse/HIVE-21740?focusedWorklogId=243890&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243890 ] ASF GitHub Bot logged work on HIVE-21740: - Author: ASF GitHub Bot Created on: 17/May/19 09:04 Start Date: 17/May/19 09:04 Worklog Time Spent: 10m Work Description: pvary commented on pull request #633: HIVE-21740: Collect LLAP execution latency metrics URL: https://github.com/apache/hive/pull/633 Add a decaying metrics which measures the task latency. Refactored the MockMetricsCollector to reuse that in the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243890) Time Spent: 10m Remaining Estimate: 0h > Collect LLAP execution latency metrics > -- > > Key: HIVE-21740 > URL: https://issues.apache.org/jira/browse/HIVE-21740 > Project: Hive > Issue Type: New Feature >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21740.2.patch, HIVE-21740.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Collect metrics for LLAP task execution times -- This message was sent by Atlassian JIRA (v7.6.3#76005)
