[
https://issues.apache.org/jira/browse/SPARK-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367506#comment-15367506
]
Wenchen Fan commented on SPARK-16333:
-------------------------------------
I'm not quite sure about it, but SPARK-11206 seems add more data to history
server event log, [~carsonwang] can you double check it? thanks!
> Excessive Spark history event/json data size (5GB each)
> -------------------------------------------------------
>
> Key: SPARK-16333
> URL: https://issues.apache.org/jira/browse/SPARK-16333
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.0
> Environment: this is seen on both x86 (Intel(R) Xeon(R), E5-2699 )
> and ppc platform (Habanero, Model: 8348-21C), Red Hat Enterprise Linux Server
> release 7.2 (Maipo)., Spark2.0.0-preview (May-24, 2016 build)
> Reporter: Peter Liu
> Labels: performance, spark2.0.0
>
> With Spark2.0.0-preview (May-24 build), the history event data (the json
> file), that is generated for each Spark application run (see below), can be
> as big as 5GB (instead of 14 MB for exactly the same application run and the
> same input data of 1TB under Spark1.6.1)
> -rwxrwx--- 1 root root 5.3G Jun 30 09:39 app-20160630091959-0000
> -rwxrwx--- 1 root root 5.3G Jun 30 09:56 app-20160630094213-0000
> -rwxrwx--- 1 root root 5.3G Jun 30 10:13 app-20160630095856-0000
> -rwxrwx--- 1 root root 5.3G Jun 30 10:30 app-20160630101556-0000
> The test is done with Sparkbench V2, SQL RDD (see github:
> https://github.com/SparkTC/spark-bench)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
