[
https://issues.apache.org/jira/browse/SPARK-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435767#comment-15435767
]
Michael Allman edited comment on SPARK-17231 at 8/24/16 11:29 PM:
------------------------------------------------------------------
I've attached screenshots from four independent test runs on four different EC2
clusters with the same configuration. The only differences are that the master
files are from test runs without the logging patches and the
logging_perf_improvements files are from test runs with the logging patches.
was (Author: michael):
Note that in the attached screenshots, all stats are the same except task and
gc time.
> Avoid building debug or trace log messages unless the respective log level is
> enabled
> -------------------------------------------------------------------------------------
>
> Key: SPARK-17231
> URL: https://issues.apache.org/jira/browse/SPARK-17231
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.0
> Environment: Spark cluster with 8 r3.8xl EC2 worker instances
> Reporter: Michael Allman
> Priority: Minor
> Attachments: logging_perf_improvements 2.jpg,
> logging_perf_improvements.jpg, master 2.jpg, master.jpg
>
>
> While debugging the performance of a large GraphX connected components
> computation, I found several places in the {{network-common}} and
> {{network-shuffle}} code bases where trace or debug log messages are
> constructed even if the respective log level is disabled. Refactoring the
> respective code to avoid these constructions except where necessary led to a
> modest but measurable reduction in task time, GC time and the ratio thereof.
> (PR to come.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
