Hi, Airflow is nothing but a new version of cron on linux with dag dependency. What operator in airflow are you using to submit your spark-submit for example BashOperator?
Can you actually run the command outside of airflow by submitting spark-submit to K8s cluster? Is that GKE cluster or something else? HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Thu, 14 Oct 2021 at 14:02, Nick Shivhare <nickshivhare...@gmail.com> wrote: > Hi All, > > We are facing an issue and would be thankful if anyone can help us on > this issue. > Environment: Spark, Kubernetes and Airflow. > Airflow is used to schedule job spark job over kubernetes. > We are using bash script which is using spark submit command to submit > spark jobs. > Issue: > We are submitting spark job through airflow in a *cluster mode*. However, > when the job is completed and executors are closed, airflow is not able to > schedule another job. > > As per our investigation, we found that jobs are completed but spark > submit command in our script is not able to exit and continuously running > with following logs: > 21/10/12 08:54:26 INFO LoggingPodStatusWatcherImpl: Application status for > spark-3f914f93ad684743b1a7b17aa26b4329 (phase: Running) > > In order to confirm this issue is not from the airflow side, we tried to > kill spark submit command and it was able to schedule another job so our > observation is that somehow after completion of job spark-submit script is > not able to exit and still running. > > FYI we have closed the spark session already in our code. > One of the weird observations was that it is running completely fine in > local mode and we are checking for client mode presently. > > > Would be thankful if you can guide us on this? > > > Thanks, > Shishir >
