[
https://issues.apache.org/jira/browse/SPARK-16332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357811#comment-15357811
]
Peter Liu commented on SPARK-16332:
-----------------------------------
I think this is likely related to issue "SPARK-16333" due the huge amount of
data per each run (5Gb per run). when the cluster is idle, it does the
initialization and consumes 1000% cpu; when under load, the cpu consumption
goes above 2000% (per linux top command);
the net is: for the same scenario (exactly the same amount of data and the
source code), when running spark 1.6.1, the jvm of the spark history server
only consumes ~30% under load (where is data amount is also much smaller).
the fix of this issue can be made dependent of the fix of "SPARK-16333", I
think.
thanks ...
Peter
> the history server of spark2.0-preview (may-24 build) consumes more than
> 1000% cpu
> ----------------------------------------------------------------------------------
>
> Key: SPARK-16332
> URL: https://issues.apache.org/jira/browse/SPARK-16332
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.0
> Environment: this is seen on both x86 (Intel(R) Xeon(R), E5-2699 )
> and ppc platform IBM Power8 Habanero (Model: 8348-21C), Red Hat Enterprise
> Linux Server release 7.2 (Maipo), Spark2.0.0-preview (May-24, 2016)
> Reporter: Peter Liu
>
> the JVM instance of the Spark history server of spark2.0-preview (may-24
> build) consumes more than 1000% cpu without the spark standalone cluster (of
> 6 nodes) being under load.
> When under load (1TB input data for a SQL query scenario), the JVM instance
> of the Spark history server of spark2.0-preview consumes 2000% cpu (as seen
> with "top" on linux 3.10)
> Note: can't see a proper Component selection here, surely not a Web GUI issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
