Blake Livingston created SPARK-11192:
----------------------------------------
Summary: When graphite metric sink is enabled, spark sql leaks
org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time
Key: SPARK-11192
URL: https://issues.apache.org/jira/browse/SPARK-11192
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.1
Environment: java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
org.apache.spark/spark-sql_2.10 "1.5.1"
Embedded, in-process spark. Have not tested on standalone or yarn clusters.
Reporter: Blake Livingston
Priority: Minor
Noticed that slowly, over the course of a day or two, heap memory usage on a
long running spark process increased monotonically.
After doing a heap dump and examining in jvisualvm, saw there were over 15M
org.apache.spark.sql.execution.ui.SQLTaskMetrics objects allocated, taking over
500MB.
Accumulation does not occur when I removed metrics.properties.
metrics.properties content:
# Enable Graphite
*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
*.sink.graphite.host=xxxxx
*.sink.graphite.port=2003
*.sink.graphite.period=10
# Enable jvm source for instance master, worker, driver and executor
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
