Thanks! I will have a look. Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited London United Kingdom
view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 12 Apr 2023 at 18:26, Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > Yes, it looks inside the docker containers folder. It will work if you are > using s3 og gs. > > ons. 12. apr. 2023, 18:02 skrev Mich Talebzadeh <mich.talebza...@gmail.com > >: > >> Hi, >> >> In my spark-submit to eks cluster, I use the standard code to submit to >> the cluster as below: >> >> spark-submit --verbose \ >> --master k8s://$KUBERNETES_MASTER_IP:443 \ >> --deploy-mode cluster \ >> --name sparkOnEks \ >> --py-files local://$CODE_DIRECTORY/spark_on_eks.zip \ >> local:///home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py >> >> In Google Kubernetes Engine (GKE) I simply load them from gs:// storage >> bucket.and it works fine. >> >> I am getting the following error in driver pod >> >> + CMD=("$SPARK_HOME/bin/spark-submit" --conf >> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client >> "$@") >> + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf >> spark.driver.bindAddress=192.168.39.251 --deploy-mode client >> --properties-file /opt/spark/conf/spark.properties --class >> org.apache.spark.deploy.PythonRunner >> local:///home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py >> 23/04/11 23:07:23 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> /usr/bin/python3: can't open file >> '/home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py': [Errno >> 2] No such file or directory >> log4j:WARN No appenders could be found for logger >> (org.apache.spark.util.ShutdownHookManager). >> It says can't open file >> '/home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py': >> >> >> [Errno 2] No such file or directory but it is there! >> >> ls -l /home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py >> -rw-rw-rw- 1 hduser hadoop 5060 Mar 18 14:16 >> /home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py >> So not sure what is going on. I have suspicion that it is looking inside the >> docker itself for this file? >> >> >> Is that a correct assumption? >> >> >> Thanks >> >> >> Mich Talebzadeh, >> Lead Solutions Architect/Engineering Lead >> Palantir Technologies Limited >> London >> United Kingdom >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >
