Hello, I am trying to run spark jobs using Spark Kubernetes Operator. But when I try to bundle a conda python environment using the following resource description the python interpreter is only unpack to the driver and not to the executors.
apiVersion: "sparkoperator.k8s.io/v1beta2" kind: SparkApplication metadata: name: ... spec: type: Python pythonVersion: "3" mode: cluster mainApplicationFile: local:///path/script.py .. sparkConf: "spark.archives": "local:///path/conda-env.tar.gz#environment" "spark.pyspark.python": "./environment/bin/python" "spark.pyspark.driver.python": "./environment/bin/python" The driver is unpacking the archive and the python scripts gets executed. On executors there is no log message indicating that the archive gets unpacked. Executors then fail as they cant find the python executable at the given location "./environment/bin/python". Any hint? Best, Meikel