Hi everyone, I want to share what helped us resolve the issue short term and also our concerns longer term.
*Some Background*: Many of our jobs that look at a few weeks of data have task counts around 3500+. We went with a small cluster of 4 EC2 larges for Mesos+Spark in production for now because our data sizes are small (3 files hourly on the order of 10-20mb). The idea is to get this steady and scale horizontally as the data flow continues to go up. *Our Problem*: In short, the time to execute simple tasks with this data of this size for the past 2 weeks or so was not reasonable for us. The jobs were taking well over an hour to complete. We were very concerned that as the hourly data grows in size this will become completely unacceptable. *The (Short Term) Solution*: Thanks to Paco and Evan's input, we decided to switch Spark to run in coarse grained mode for the time-being. We saw an immediate 6-8x improvement on our job runtime and we were able to complete our short term tasks. The downside we see to this is that it takes away many of the benefits of running on Mesos - namely, the fine-grained resource management. *The Longer Term Solution*: We are looking at ways of tuning to reduce the number of Spark tasks being generated overall. We think if we can get this down enough, it justifies taking the performance hit in fine-grained mode since the cluster resources will now be allocated dynamically for multiple jobs. Hope this is helpful to people, Gary On Thu, Nov 14, 2013 at 11:47 AM, Gary Malouf <malouf.g...@gmail.com> wrote: > I bring this up because the performance we are seeing is dreadful. From > cpu usage, it appears the issue is the spark shell cpu power. We have > increased this node from a EC2 medium to an xl, we are seeing slightly > better performance but still not great. > > My understanding of Spark was that most of the work should be done on the > slaves with just the results going back to the shell at the end if we do a > take. It appears from what we see that the client is doing much more work > than expected. > > > > > On Wed, Nov 13, 2013 at 10:40 PM, Gary Malouf <malouf.g...@gmail.com>wrote: > >> Hi, >> >> We have an HDFS set up of a namenode and three datanodes all on EC2 >> larges. One of our data partitions basically has files that are fed from a >> few Flume instances rolling *hourly*. This equates to around 3 4-8mb >> files per hour right now >> >> Our Mesos cluster consists of a Master and the three slave nodes >> colocated on these EC2 larges as well (slaves -> datanodes, mesos master -> >> namenode). Spark scheduled jobs are launched from spark shell ad-hoc today. >> >> The data is serialized protobuf messages in sequence files. Our >> operations typically consist of deserializing the data, grabbing a few >> primitive fields out of the message and doing some maps/reduces. >> >> For grabbing on the order of 2 days of data this size, what would the >> expected Spark performance be? We are seeing simple maps and 'takes' on >> this data taking on the order of 15 minutes. >> >> Thanks, >> >> Gary >> > >
