Hi Shyam, It will be good if you mention what are you using the --master url as? Is it running on YARN, Mesos or Spark cluster?
However, I faced such an issue in my earlier trials with spark, in which I created connections with a lot of external databases like Cassandra within the Driver (or main program of my app). After the job completed, my Main program/driver task never finished, after debugging I found out to be the reason as open sessions with Cassandra. Closing out those connections at the end of my main program helped resolve the problem. As you can guess, this issue was then irrespective of the Cluster manager used. Akshay Bhardwaj +91-97111-33849 On Tue, Jun 11, 2019 at 7:41 PM Shyam P <shyamabigd...@gmail.com> wrote: > Hi, > Any clue why spark job goes into UNDEFINED state ? > > More detail are in the url. > > https://stackoverflow.com/questions/56545644/why-my-spark-sql-job-stays-in-state-runningfinalstatus-undefined > > > Appreciate your help. > > Regards, > Shyam >
