Blake Livingston created SPARK-11192: ----------------------------------------
Summary: When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time Key: SPARK-11192 URL: https://issues.apache.org/jira/browse/SPARK-11192 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.5.1 Environment: java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) org.apache.spark/spark-sql_2.10 "1.5.1" Embedded, in-process spark. Have not tested on standalone or yarn clusters. Reporter: Blake Livingston Priority: Minor Noticed that slowly, over the course of a day or two, heap memory usage on a long running spark process increased monotonically. After doing a heap dump and examining in jvisualvm, saw there were over 15M org.apache.spark.sql.execution.ui.SQLTaskMetrics objects allocated, taking over 500MB. Accumulation does not occur when I removed metrics.properties. metrics.properties content: # Enable Graphite *.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink *.sink.graphite.host=xxxxx *.sink.graphite.port=2003 *.sink.graphite.period=10 # Enable jvm source for instance master, worker, driver and executor master.source.jvm.class=org.apache.spark.metrics.source.JvmSource worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org