[
https://issues.apache.org/jira/browse/HIVE-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944389#comment-13944389
]
Lefty Leverenz commented on HIVE-6500:
--------------------------------------
I updated the wiki for *hive.stats.dbclass* -- please review:
{quote}
Hive 0.13 and later: The storage that stores temporary Hive statistics. In FS
based statistics collection, each task writes statistics it has collected in a
file on the filesystem, which will be aggregated after the job has finished.
Supported values are fs (filesystem), jdbc(:.*), hbase, counter and custom
(HIVE-6500).
{quote}
* [Configuration Properties: hive.stats.dbclass
|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.dbclass]
> Stats collection via filesystem
> -------------------------------
>
> Key: HIVE-6500
> URL: https://issues.apache.org/jira/browse/HIVE-6500
> Project: Hive
> Issue Type: New Feature
> Components: Statistics
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.13.0
>
> Attachments: HIVE-6500.2.patch, HIVE-6500.3.patch, HIVE-6500.patch
>
>
> Recently, support for stats gathering via counter was [added |
> https://issues.apache.org/jira/browse/HIVE-4632] Although, its useful it has
> following issues:
> * [Length of counter group name is limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L340]
> * [Length of counter name is limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L337]
> * [Number of distinct counter groups are limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L343]
> * [Number of distinct counters are limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L334]
> Although, these limits are configurable, but setting them to higher value
> implies increased memory load on AM and job history server.
> Now, whether these limits makes sense or not is [debatable |
> https://issues.apache.org/jira/browse/MAPREDUCE-5680] it is desirable that
> Hive doesn't make use of counters features of framework so that it we can
> evolve this feature without relying on support from framework. Filesystem
> based counter collection is a step in that direction.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
