gbloisi-openaire commented on issue #38017: URL: https://github.com/apache/airflow/issues/38017#issuecomment-1987928257
Plese find it below. It is an adaption from [tests/system/providers/cncf/kubernetes/example_spark_kubernetes_spark_pi.yaml](https://github.com/apache/airflow/blob/main/tests/system/providers/cncf/kubernetes/example_spark_kubernetes_spark_pi.yaml). I removed license's comments after a few tries to have confirmation that the error was reporting the content of this file. Also my DAG is an adaption from the provided [tests/system/providers/cncf/kubernetes/example_spark_kubernetes.py](https://github.com/apache/airflow/blob/main/tests/system/providers/cncf/kubernetes/example_spark_kubernetes.py). So far I got it working successfully by rewriting the file content as a template_spec object and by removing the do_xcom_push=True and the SparkKubernetesSensor task. ``` apiVersion: "sparkoperator.k8s.io/v1beta2" kind: SparkApplication metadata: name: spark-pi namespace: lot1-spark-jobs spec: type: Scala mode: cluster image: "apache/spark:v3.1.3" imagePullPolicy: Always mainClass: org.apache.spark.examples.SparkPi mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.3.jar" sparkVersion: "3.1.3" restartPolicy: type: Never volumes: - name: "test-volume" hostPath: path: "/tmp" type: Directory driver: cores: 1 coreLimit: "1200m" memory: "512m" labels: version: 3.1.3 serviceAccount: spark volumeMounts: - name: "test-volume" mountPath: "/tmp" executor: cores: 1 instances: 1 memory: "512m" labels: version: 3.1.3 volumeMounts: - name: "test-volume" mountPath: "/tmp" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
