Re: Number of CPU cores for a Spark Streaming app in Standalone mode

I am adding an answer from SO: http://stackoverflow.com/questions/34861947/read-more-kafka-topics-than-number-of-cpu-cores -- View this message in context:

Number of CPU cores for a Spark Streaming app in Standalone mode

I somehow don't want to believe this waste of resources. Is it really true that if I have 20 input streams I must have at least 21 CPU cores? Even if I read only once per minute and only a few messages? I still hope that I miss an important information. Thanks a lot -- View this message in

Re: Number of CPU cores for a Spark Streaming app in Standalone mode

If you are using receiver-based input streams, then you have to dedicate 1 core to each receiver. If you read only once per minute on each receiver, than consider consolidating the data reading pipeline such that you can use fewer receivers. On Mon, Jan 18, 2016 at 12:13 PM, radoburansky