hong-yangzhao opened a new issue, #1492:
URL: https://github.com/apache/sedona/issues/1492
## Expected behavior
run spark-pi example successfully
## Actual behavior
```sh
Error: failed to create containerd task: failed to create shim task: OCI
runtime create failed: runc create failed:
unable to start container process: exec: "driver": executable file not found
in $PATH: unknown
```
## Steps to reproduce the problem
I plan to run a Sedona cluster on Kubernetes. Prior to this, I have
successfully used the spark-operator to run a Spark cluster and obtained
results from the spark-pi example:
```yaml
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: spark-pi-test
namespace: spark-tasks
spec:
type: Scala
mode: cluster
image: docker.io/spark:3.5.1
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile:
"local:///opt/spark/examples/jars/spark-examples_2.12-3.5.1.jar"
sparkVersion: "3.5.1"
restartPolicy:
type: Never
driver:
cores: 4
memory: "4G"
labels:
version: 3.5.1
serviceAccount: spark-task-account
env:
- name: SPARK_MODE
value: "driver"
executor:
cores: 4
instances: 2
memory: "4G"
labels:
version: 3.5.1
arguments:
- "1000"
```
However, when I change the image to Sedona's image:
```yaml
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: sedona-test
namespace: spark-tasks
spec:
type: Scala
mode: cluster
image: docker.io/apache/sedona:1.6.0
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile:
"local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar"
sparkVersion: "3.4.1"
restartPolicy:
type: Never
driver:
cores: 4
memory: "4G"
labels:
version: 3.4.1
serviceAccount: spark-task-account
env:
- name: SPARK_MODE
value: "driver"
executor:
cores: 4
instances: 2
memory: "4G"
labels:
version: 3.4.1
arguments:
- "1000"
```
Submitting this YAML file results in an error:
The configMap (spark-drv-fba0809048193c11-conf-map) mentioned in the error
message does exist:
```sh
kubectl get configmap -n spark-tasks
NAME DATA AGE
kube-root-ca.crt 1 11d
spark-drv-fba0809048193c11-conf-map 2 25m
```
My question is: Does the Sedona image not support running on Kubernetes? Or
is there a problem with my configuration? Thank you.
## Settings
Sedona version = 1.6.0
Apache Spark version = 3.4.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
