[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29055:
--------------------------------
Description:
I used Spark 2.1.1 and I upgraded into the latest version 2.4.4. I observed
from Spark UI that the driver memory is{color:#ff0000} increasing
continuously{color} and after of long running I had the following error :
{color:#ff0000}java.lang.OutOfMemoryError: GC overhead limit exceeded{color}
In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was extremely
low and after the run of ContextCleaner and BlockManager the memory was
decreasing.
Also, I tested the Spark versions 2.3.3, 2.4.3 and I had the same behavior.
*HOW TO REPRODUCE THIS BEHAVIOR:*
Create a very simple application(streaming count_file.py) in order to reproduce
this behavior. This application reads CSV files from a directory, count the
rows and then remove the processed files.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
Submit code:
{code:java}
spark-submit
--master spark://xxx.xxx.xx.xxx
--deploy-mode client
--executor-memory 4g
--executor-cores 3
streaming count_file.py
{code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
was:
I used Spark 2.1.1 and I upgraded into the latest version 2.4.4. I observed
from Spark UI that the driver memory is{color:#ff0000} increasing
continuously{color} and after of long running I had the following error :
{color:#ff0000}java.lang.OutOfMemoryError: GC overhead limit exceeded{color}
In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was extremely
low and after the run of ContextCleaner and BlockManager the memory was
decreasing.
Also, I tested the Spark versions 2.3.3, 2.4.3 and I had the same behavior.
*HOW TO REPRODUCE THIS BEHAVIOR:*
Create a very simple application(streaming count_file.py) in order to reproduce
this behavior. This application reads CSV files from a directory, count the
rows and then remove the processed files.
{code:java}
import time
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
Submit code:
{code:java}
spark-submit
--master spark://xxx.xxx.xx.xxx
--deploy-mode client
--executor-memory 4g
--executor-cores 3
streaming count_file.py
{code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
> Memory leak in Spark Driver
> ---------------------------
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
> Affects Versions: 2.3.3, 2.4.3, 2.4.4
> Reporter: George Papa
> Priority: Major
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into the latest version 2.4.4. I observed
> from Spark UI that the driver memory is{color:#ff0000} increasing
> continuously{color} and after of long running I had the following error :
> {color:#ff0000}java.lang.OutOfMemoryError: GC overhead limit exceeded{color}
> In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
> Also, I tested the Spark versions 2.3.3, 2.4.3 and I had the same behavior.
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
