Gopal V created HIVE-20103:
------------------------------
Summary: WM: Only Aggregate DAG counters if at least one is used
Key: HIVE-20103
URL: https://issues.apache.org/jira/browse/HIVE-20103
Project: Hive
Issue Type: Bug
Components: llap
Affects Versions: 4.0.0
Reporter: Gopal V
{code}
status = dagClient.getDAGStatus(EnumSet.of(StatusGetOpts.GET_COUNTERS),
checkInterval);
TezCounters dagCounters = status.getDAGCounters();
...
if (dagCounters != null && wmContext != null) {
Set<String> desiredCounters = wmContext.getSubscribedCounters();
if (desiredCounters != null && !desiredCounters.isEmpty()) {
Map<String, Long> currentCounters = getCounterValues(dagCounters,
vertexNames, vertexProgressMap,
desiredCounters, done);
{code}
Skip collecting DAG counters unless there at least one desired counter in
wmContext.
The AM has a hard-lock around the counters, so the current jstacks are full of
{code}
java.lang.Thread.State: RUNNABLE
at java.lang.String.intern(Native Method)
at
org.apache.hadoop.util.StringInterner.weakIntern(StringInterner.java:71)
at
org.apache.tez.common.counters.GenericCounter.<init>(GenericCounter.java:50)
at
org.apache.tez.common.counters.TezCounters$GenericGroup.newCounter(TezCounters.java:65)
at
org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
at
org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
- locked <0x00007efb3ac7af38> (a
org.apache.tez.common.counters.TezCounters$GenericGroup)
at
org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
at
org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
- eliminated <0x00007efb3ac64ee8> (a
org.apache.tez.common.counters.TezCounters)
at
org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:357)
- locked <0x00007efb3ac64ee8> (a
org.apache.tez.common.counters.TezCounters)
at
org.apache.tez.dag.app.dag.impl.TaskImpl.getCounters(TaskImpl.java:462)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.aggrTaskCounters(VertexImpl.java:1342)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.getAllCounters(VertexImpl.java:1202)
at
org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:755)
at
org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:704)
at
org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
at
org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
at
org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
