[
https://issues.apache.org/jira/browse/BEAM-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Moravek updated BEAM-5987:
--------------------------------
Description:
We did some profiling of a spark job and 90% of the application time was spent
on side input deserialization.
For spark, an easy fix is to cache materialized side inputs per bundle. This
improved running time of the profiled job from 3 hours to 30 minutes.
!image-2018-11-06-18-45-06-045.png!
was:
We did some profiling of a spark job and 90% of the application time was spent
on side input deserialization.
!image-2018-11-06-13-06-04-350.png!
For spark, an easy fix is to cache materialized side inputs per bundle. This
improved running time of the profiled job from 3 hours to 30 minutes.
> Spark SideInputReader performance
> ---------------------------------
>
> Key: BEAM-5987
> URL: https://issues.apache.org/jira/browse/BEAM-5987
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Affects Versions: 2.8.0
> Reporter: David Moravek
> Assignee: David Moravek
> Priority: Major
> Fix For: 2.9.0
>
> Attachments: Screen Shot 2018-11-06 at 13.05.36.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We did some profiling of a spark job and 90% of the application time was
> spent on side input deserialization.
> For spark, an easy fix is to cache materialized side inputs per bundle. This
> improved running time of the profiled job from 3 hours to 30 minutes.
> !image-2018-11-06-18-45-06-045.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
