[ https://issues.apache.org/jira/browse/SPARK-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357797#comment-15357797 ]
Peter Liu commented on SPARK-16333: ----------------------------------- please see if this answers your question: the history data file is the one whose location is defined in the spark-defaults.conf, such as: spark.eventLog.dir /hdd10/spark-event-log spark.history.fs.logDirectory /hdd10/spark-event-log "app-20160622195029-0000" would be one file example. the size becomes excessive in Spark2.0.0-preview (5.3G) as seen below, compared to 51M in spark1.6.1: (a) after - using spark2.0.0-preview (May-24): rwxrwx-- 1 root root 5.3G Jun 30 09:39 app-20160630091959-0000 rwxrwx-- 1 root root 5.3G Jun 30 09:56 app-20160630094213-0000 (b) before - using Spark 1.6.1 -rwxrwx--- 1 root root 51M Jun 22 20:00 app-20160622195029-0000 -rwxrwx--- 1 root root 51M Jun 22 20:13 app-20160622200258-0000 IMPACT: it can't be loaded into the Spark UI from the 18080 port (you got OOM error); No spark analysis on UI is possible after the run is over. > Excessive Spark history event/json data size (5GB each) > ------------------------------------------------------- > > Key: SPARK-16333 > URL: https://issues.apache.org/jira/browse/SPARK-16333 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.0.0 > Environment: this is seen on both x86 (Intel(R) Xeon(R), E5-2699 ) > and ppc platform (Habanero, Model: 8348-21C), Red Hat Enterprise Linux Server > release 7.2 (Maipo)., Spark2.0.0-preview (May-24, 2016 build) > Reporter: Peter Liu > Labels: performance, spark2.0.0 > > With Spark2.0.0-preview (May-24 build), the history event data (the json > file), that is generated for each Spark application run (see below), can be > as big as 5GB (instead of 14 MB for exactly the same application run and the > same input data of 1TB under Spark1.6.1) > -rwxrwx--- 1 root root 5.3G Jun 30 09:39 app-20160630091959-0000 > -rwxrwx--- 1 root root 5.3G Jun 30 09:56 app-20160630094213-0000 > -rwxrwx--- 1 root root 5.3G Jun 30 10:13 app-20160630095856-0000 > -rwxrwx--- 1 root root 5.3G Jun 30 10:30 app-20160630101556-0000 > The test is done with Sparkbench V2, SQL RDD (see github: > https://github.com/SparkTC/spark-bench) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org