[
https://issues.apache.org/jira/browse/HIVE-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25429:
---------------------------------
Description:
There's a limit to the number of tez counters allowed (tez.counters.max). Delta
metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for
each partition touched by a given query, which can result in a huge number of
counters, which is unnecessary because we're only interested in n number of
partitions with the most deltas. This change limits the number of counters
created to hive.txn.acid.metrics.max.cache.size*3.
Also when tez.counters.max is reached a LimitExceededException is thrown but
isn't caught on the Hive side and causes the query to fail. We should catch
this and skip delta metrics collection in this case.
Also make sure that metrics are only collected if
hive.metastore.acidmetrics.ext.on=true
was:
There's a limit to the number of tez counters allowed (tez.counters.max). Delta
metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for
each partition touched by a given query, which can result in a huge number of
counters, which is unnecessary because we're only interested in n number of
partitions with the most deltas. This change limits the number of counters
created to hive.txn.acid.metrics.max.cache.size*3.
Also when tez.counters.max is reached a LimitExceededException is thrown but
isn't caught on the Hive side and causes the query to fail. We should catch
this and skip delta metrics collection in this case.
> Delta metrics collection may cause number of tez counters to exceed
> tez.counters.max limit
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-25429
> URL: https://issues.apache.org/jira/browse/HIVE-25429
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Affects Versions: 4.0.0
> Reporter: Karen Coppage
> Assignee: Karen Coppage
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There's a limit to the number of tez counters allowed (tez.counters.max).
> Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3
> counters for each partition touched by a given query, which can result in a
> huge number of counters, which is unnecessary because we're only interested
> in n number of partitions with the most deltas. This change limits the number
> of counters created to hive.txn.acid.metrics.max.cache.size*3.
> Also when tez.counters.max is reached a LimitExceededException is thrown but
> isn't caught on the Hive side and causes the query to fail. We should catch
> this and skip delta metrics collection in this case.
> Also make sure that metrics are only collected if
> hive.metastore.acidmetrics.ext.on=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
