[
https://issues.apache.org/jira/browse/SPARK-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070717#comment-15070717
]
Saisai Shao commented on SPARK-12514:
-------------------------------------
Since we need to differentiate the metrics between applications, so the
application prefix is unavoidable, otherwise the metrics between different
applications will be mixed together. For Ganglia that will be a problem, but
you could filter out application specific metrics to not push into Ganglia,
since Sink can be configured with different destinations.
Also you could choose other tools like openTSDB, which is backed by HBase.
> Spark MetricsSystem can fill disks/cause OOMs when using GangliaSink
> --------------------------------------------------------------------
>
> Key: SPARK-12514
> URL: https://issues.apache.org/jira/browse/SPARK-12514
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.5.2
> Reporter: Aaron Tokhy
> Priority: Minor
>
> The MetricsSystem implementation in Spark generates unique metric names for
> each spark application that has been submitted (to a YARN cluster, for
> example). This can be problematic for certain metrics environments, like
> Ganglia.
> This creates metric names that look like the following (for each submitted
> application):
> application_1450753701508_0001.driver.ExecutorAllocationManager.executors.numberAllExecutors
>
> On Spark clusters where thousands of applications are submitted, some metrics
> will eventually cause Ganglia daemons to reach their memory limits (gmond),
> or to run out of disk space (gmetad). This is due to the fact that some
> existing metrics systems do not expect new metric names to be generated in
> the lifetime of a cluster.
> Ganglia as a spark metrics sink is one example of where the current
> implementation can run into problems. Each new set of metrics per
> application introduces a new set of RRD files that are never deleted (round
> robin databases) and metrics in gmetad/gmond, which can cause the gmond
> aggregator's memory usage to bloat over time, and gmetad to generate new
> round robin databases for every new set of metrics, per application. These
> round robin databases are permanent, so each new set of metrics will
> introduce files that would never be cleaned up.
> So the MetricsSystem may need to account for metrics sinks that have problems
> with the introduction of new metrics, and buildRegistryName would have to
> behave differently in this case.
> https://github.com/apache/spark/blob/d83c2f9f0b08d6d5d369d9fae04cdb15448e7f0d/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L126
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
