[ https://issues.apache.org/jira/browse/BEAM-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052240#comment-17052240 ]
Ismaël Mejía edited comment on BEAM-9440 at 3/5/20, 3:15 PM: ------------------------------------------------------------- Pinging [~kenn] [~robertwb] and [~lcwik] for awareness on the publication and in case they have something else to add to the discussion. was (Author: iemejia): Pinging [~kenn] [~robertwb] and [~lcwik] for awareness on the publication and in case they have something else to add to the discussoin. > Performance Issue with Spark Runner compared with Native Spark > -------------------------------------------------------------- > > Key: BEAM-9440 > URL: https://issues.apache.org/jira/browse/BEAM-9440 > Project: Beam > Issue Type: Bug > Components: runner-spark > Reporter: Soumabrata Chakraborty > Priority: Major > > While doing a performance evaluation of Apache Beam with Spark Runner - I > found that even for a simple word count problem on a text file – Beam with > Spark runner was slower by a factor of 5 times as compared to Spark for a > dataset as small as 14 GB. > You will find more details on this evaluation here - > [https://github.com/soumabrata-chakraborty/spark-vs-beam/blob/master/README.md] > I also came across this analysis called _**Quantitative Impact Evaluation of > an Abstraction Layer for Data Stream Processing Systems_ > ([https://arxiv.org/pdf/1907.08302.pdf]) > According to it, the observation was that for most scenarios the slowdown was > at least a factor of 3 with the worse case being a factor of 58! > While it is understood that an abstraction layer would come with some > performance cost - the current performance cost seems to be very high. -- This message was sent by Atlassian Jira (v8.3.4#803005)