Thanks for the reply. Forgot to mention that, our Batch ETL Jobs are in Core-Spark.
> On Sep 22, 2017, at 3:13 PM, Vadim Semenov <vadim.seme...@datadoghq.com> > wrote: > > 1. 40s is pretty negligible unless you run your job very frequently, there > can be many factors that influence that. > > 2. Try to compare the CPU time instead of the wall-clock time > > 3. Check the stages that got slower and compare the DAGs > > 4. Test with dynamic allocation disabled > > On Fri, Sep 22, 2017 at 2:39 PM, Gokula Krishnan D <email2...@gmail.com > <mailto:email2...@gmail.com>> wrote: > Hello All, > > Currently our Batch ETL Jobs are in Spark 1.6.0 and planning to upgrade into > Spark 2.1.0. > > With minor code changes (like configuration and Spark Session.sc) able to > execute the existing JOB into Spark 2.1.0. > > But noticed that JOB completion timings are much better in Spark 1.6.0 but no > in Spark 2.1.0. > > For the instance, JOB A completed in 50s in Spark 1.6.0. > > And with the same input and JOB A completed in 1.5 mins in Spark 2.1.0. > > Is there any specific factor needs to be considered when switching to Spark > 2.1.0 from Spark 1.6.0. > > > > Thanks & Regards, > Gokula Krishnan (Gokul) >
