[ https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942279#comment-16942279 ]
Jungtaek Lim commented on SPARK-29321: -------------------------------------- [~Geopap] Appreciate if you could migrate the last comment on SPARK-29055. I would like to unassign myself but I can't do it. Appreciate if any committer could adjust this. > CLONE - Possible memory leak in Spark > ------------------------------------- > > Key: SPARK-29321 > URL: https://issues.apache.org/jira/browse/SPARK-29321 > Project: Spark > Issue Type: Bug > Components: Block Manager, Spark Core > Affects Versions: 2.3.3 > Reporter: George Papa > Assignee: Jungtaek Lim > Priority: Major > > I used Spark 2.1.1 and I upgraded into new versions. After Spark version > 2.3.3, I observed from Spark UI that the driver memory is{color:#ff0000} > increasing continuously.{color} > In more detail, the driver memory and executors memory have the same used > memory storage and after each iteration the storage memory is increasing. You > can reproduce this behavior by running the following snippet code. The > following example, is very simple, without any dataframe persistence, but the > memory consumption is not stable as it was in former Spark versions > (Specifically until Spark 2.3.2). > Also, I tested with Spark streaming and structured streaming API and I had > the same behavior. I tested with an existing application which reads from > Kafka source and do some aggregations, persist dataframes and then unpersist > them. The persist and unpersist it works correct, I see the dataframes in the > storage tab in Spark UI and after the unpersist, all dataframe have removed. > But, after the unpersist the executors memory is not zero, BUT has the same > value with the driver memory. This behavior also affects the application > performance because the memory of the executors is increasing as the driver > increasing and after a while the persisted dataframes are not fit in the > executors memory and I have spill to disk. > Another error which I had after a long running, was > {color:#ff0000}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I > don't know if its relevant with the above behavior or not.{color} > > *HOW TO REPRODUCE THIS BEHAVIOR:* > Create a very simple application(streaming count_file.py) in order to > reproduce this behavior. This application reads CSV files from a directory, > count the rows and then remove the processed files. > {code:java} > import time > import os > from pyspark.sql import SparkSession > from pyspark.sql import functions as F > from pyspark.sql import types as T > target_dir = "..." > spark=SparkSession.builder.appName("DataframeCount").getOrCreate() > while True: > for f in os.listdir(target_dir): > df = spark.read.load(target_dir + f, format="csv") > print("Number of records: {0}".format(df.count())) > time.sleep(15){code} > Submit code: > {code:java} > spark-submit > --master spark://xxx.xxx.xx.xxx > --deploy-mode client > --executor-memory 4g > --executor-cores 3 > streaming count_file.py > {code} > > *TESTED CASES WITH THE SAME BEHAVIOUR:* > * I tested with default settings (spark-defaults.conf) > * Add spark.cleaner.periodicGC.interval 1min (or less) > * {{Turn spark.cleaner.referenceTracking.blocking}}=false > * Run the application in cluster mode > * Increase/decrease the resources of the executors and driver > * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC > -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12 > > *DEPENDENCIES* > * Operation system: Ubuntu 16.04.3 LTS > * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221) > * Python: Python 2.7.12 > > *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was > extremely low and after the run of ContextCleaner and BlockManager the memory > was decreasing. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org