How are threads created in SQL Executor?

2016-07-01 Thread emiretsk
It seems like threads are created by SQLExecution.withExecutionId, which is called inside BroadcastExchangeExec.scala. When does the plan executor execute a BroadcaseExchange, and is there a way to control the number of threads? We have a job that writes DataFrames to an external DB, and it seems

Spark Streaming: Doing operation in Receiver vs RDD

2015-10-07 Thread emiretsk
Hi, I have a Spark Streaming program that is consuming message from Kafka and has to decrypt and deserialize each message. I can implement it either as Kafka deserializer (that will run in a receiver or the new receiver-less Kafka consumer) or as RDD operations. What are the pros/cons of each?