I have updated my question: http://stackoverflow.com/questions/41345552/spark-streaming-with-yarn-executors-not-fully-utilized
On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <nishant.ku...@applift.com> wrote: > Hi, > > I am running spark streaming with Yarn with - > > *spark-submit --master yarn --deploy-mode cluster --num-executors 2 > --executor-memory 8g --driver-memory 2g --executor-cores 8 ..* > > I am consuming Kafka through DireactStream approach (No receiver). I have > 2 topics (each with 3 partitions). > > I reparation RDD (i have one DStream) into 16 parts (assuming no of > executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do > foreachPartition and writes each partition to local file and then send it > to other server (not spark) through http (Using apache http client post > with multi-part). > > *When i checked details of this step (or JOB is it correct naming?) > through Spark UI, it showed that total 16 task executed on single executor > with 8 task at a time.* > > This is Spark UI details - > > *Details for Stage 717 (Attempt 0)* > > Index ID Attempt Status Locality Level Executor ID / Host Launch Time > Duration GC Time Shuffle Read Size / Records Errors > 0 5080 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 313.3 KB / 6137 > 1 5081 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 328.5 KB / 6452 > 2 5082 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 324.3 KB / 6364 > 3 5083 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 321.5 KB / 6306 > 4 5084 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 324.8 KB / 6364 > 5 5085 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 320.8 KB / 6307 > 6 5086 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 2 s 11 ms 323.4 KB / 6356 > 7 5087 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:46 3 s 11 ms 316.8 KB / 6207 > 8 5088 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 317.7 KB / 6245 > 9 5089 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 320.4 KB / 6280 > 10 5090 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 323.0 KB / 6334 > 11 5091 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 323.7 KB / 6371 > 12 5092 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 316.7 KB / 6218 > 13 5093 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 321.0 KB / 6301 > 14 5094 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 > 12:11:48 2 s 321.4 KB / 6304 > > I was expecting it to execute 16 parallel task (2 executor * 8 core) on > either one or more executor. I think i am missing something. Please help. > > > -- > *Nishant Kumar* > Senior Software Engineer > > Phone: +91 80088 42030 > Skype: nishant.kumar_applift > > > *AppLift India* > 107/3, 80 Feet Main Road, > Koramangala 4th Block, > Bangalore - 560034 > www.AppLift.com <http://www.applift.com/> > -- *Nishant Kumar* Senior Software Engineer Phone: +91 80088 42030 Skype: nishant.kumar_applift *AppLift India* 107/3, 80 Feet Main Road, Koramangala 4th Block, Bangalore - 560034 www.AppLift.com <http://www.applift.com/> -- Meet us at: @ *PG Connects*, London, Jan 16-17 @ *Mobile Games Forum*, London, Jan 17-18 @ *GMASA*, Jakarta, Jan 26 @* Casual Connect*, Berlin, Feb 7-9 @* Mobile World Congress*, Barcelona, Feb 27-Mar 2 @* GDC*, San Francisco, Feb 27-Mar 3 Click here <http://www.applift.com> to see all the events we will be attending.
