adityamohan93 opened a new issue, #54365: URL: https://github.com/apache/airflow/issues/54365
### Description In Airflow, for EMR on EKS virtual cluster, EmrContainerOperator is advised to start a new Spark job. From doc, seems it needs `sparkSubmitJobDriver` as input[1] where we provide an S3 location for script. However, EKS or Kubernetes uses only Docker images, not a script file. It also support Spark Operator to submit jobs[2]. Are these supported in Airflow because from documentation seems it doesn't. 1. https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/emr/emr_eks.html#submit-a-job-to-an-amazon-emr-virtual-cluster 2. https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator-gs.html ### Use case/motivation Run Spark job on AWS EMR on EKS. This question is relevant to anyone doing the same. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
