Hi Gerard, Really nice guide!
I'm particularly interested in the Mesos scheduling side to more evenly distribute cores across cluster. I wonder if you are using coarse grain mode or fine grain mode? I'm making changes to the spark mesos scheduler and I think we can propose a best way to achieve what you mentioned. Tim Sent from my iPhone > On Dec 22, 2014, at 8:33 AM, Gerard Maas <gerard.m...@gmail.com> wrote: > > Hi, > > After facing issues with the performance of some of our Spark Streaming > jobs, we invested quite some effort figuring out the factors that affect > the performance characteristics of a Streaming job. We defined an > empirical model that helps us reason about Streaming jobs and applied it to > tune the jobs in order to maximize throughput. > > We have summarized our findings in a blog post with the intention of > collecting feedback and hoping that it is useful to other Spark Streaming > users facing similar issues. > > http://www.virdata.com/tuning-spark/ > > Your feedback is welcome. > > With kind regards, > > Gerard. > Data Processing Team Lead > Virdata.com > @maasg --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
