Re: Spark + Kafka processing trouble

on't use print(), > since they may have non-obvious performance implications. > > If that works, add on further operations one step at a time and see > when issues arise. > > On Mon, May 30, 2016 at 8:45 PM, Malcolm Lockyer > <malcolm.lock...@hapara.com> wrote: >> Ho

Re: Spark + Kafka processing trouble

On Tue, May 31, 2016 at 3:14 PM, Darren Govoni wrote: > Well that could be the problem. A SQL database is essential a big > synchronizer. If you have a lot of spark tasks all bottlenecking on a single > database socket (is the database clustered or colocated with spark

Re: Spark + Kafka processing trouble

On Tue, May 31, 2016 at 1:56 PM, Darren Govoni wrote: > So you are calling a SQL query (to a single database) within a spark > operation distributed across your workers? Yes, but currently with very small sets of data (1-10,000) and on a single (dev) machine right now.

Spark + Kafka processing trouble

Thanks, -- Malcolm Lockyer - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org