Hi, in 1.6, how many executors do you see for each node? in standalone mode how are you increasing the number of worker instances. Are you starting another slave on each node?
HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 10 April 2016 at 08:26, Vikash Pareek <vikash.par...@infoobjects.com> wrote: > Hi, > > I have upgraded 5 node spark cluster from spark-1.5 to spark-1.6 (to use > mapWithState function). > After using spark-1.6, I am getting a strange behaviour of spark, jobs are > not using multiple executors of different nodes at a time means there is no > parallel processing if each node having single worker and executor. > I am running jobs in spark standalone mode. > > I observed following points related to this issue. > 1. If I run same job with spark-1.5 then this will use multiple executors > across different nodes at a time. > 2. In Spark-1.6, If I increase no of cores(spark.cores.max) then jobs are > running in parallel thread but within same executor. > 3. In Spark-1.6, If I increase no of worker instances on each node then > jobs > are running in parallel as no of workers but within same executor. > > Can anyone suggest, why spark 1.6 can not use multiple executors across > different node at a time for parallel processing. > Your suggestion will be highly appreciated. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-in-spark-1-6-and-spark-1-5-tp26733.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
