Hi, I am running spark streaming with Yarn with -
*spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-memory 8g --driver-memory 2g --executor-cores 8 ..* I am consuming Kafka through DireactStream approach (No receiver). I have 2 topics (each with 3 partitions). I reparation RDD (i have one DStream) into 16 parts (assuming no of executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do foreachPartition and writes each partition to local file and then send it to other server (not spark) through http (Using apache http client post with multi-part). *When i checked details of this step (or JOB is it correct naming?) through Spark UI, it showed that total 16 task executed on single executor with 8 task at a time.* This is Spark UI details - *Details for Stage 717 (Attempt 0)* Index ID Attempt Status Locality Level Executor ID / Host Launch Time Duration GC Time Shuffle Read Size / Records Errors 0 5080 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137 1 5081 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452 2 5082 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364 3 5083 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306 4 5084 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364 5 5085 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307 6 5086 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356 7 5087 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207 8 5088 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 317.7 KB / 6245 9 5089 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 320.4 KB / 6280 10 5090 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 323.0 KB / 6334 11 5091 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 323.7 KB / 6371 12 5092 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 316.7 KB / 6218 13 5093 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 321.0 KB / 6301 14 5094 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 321.4 KB / 6304 I was expecting it to execute 16 parallel task (2 executor * 8 core) on either one or more executor. I think i am missing something. Please help. -- *Nishant Kumar* Senior Software Engineer Phone: +91 80088 42030 Skype: nishant.kumar_applift *AppLift India* 107/3, 80 Feet Main Road, Koramangala 4th Block, Bangalore - 560034 www.AppLift.com <http://www.applift.com/> -- Meet us at: @ *PG Connects*, London, Jan 16-17 @ *Mobile Games Forum*, London, Jan 17-18 @ *GMASA*, Jakarta, Jan 26 @* Casual Connect*, Berlin, Feb 7-9 @* Mobile World Congress*, Barcelona, Feb 27-Mar 2 @* GDC*, San Francisco, Feb 27-Mar 3 Click here <http://www.applift.com> to see all the events we will be attending.
