I'm trying to run a sample code that reads a file from s3 so I need the aws sdk and aws hadoop dependencies. If I assemble these deps into the main jar everything works fine. But when I try using --packages, the deps are not seen by the pods.
This is my submit command: spark-submit --master k8s://https://xx.xx.xx.xx --class "SimpleApp" --deploy-mode cluster --conf spark.kubernetes.container.image=docker.io/pacuna/spark:0.2 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-test-user --packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3 --conf spark.hadoop.fs.s3a.access.key=... --conf spark.hadoop.fs.s3a.secret.key=... https://xxxxx/simple-project_2.11-1.0.jar And the error I'm getting in the driver pod is: 19/06/05 20:13:50 ERROR SparkContext: Failed to add file:///home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar to Spark environment java.io.FileNotFoundException: Jar /home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar not found I'm getting that error for all the deps jars needed. Any ideas? Thanks. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
