I am not sure about that. However, with Kubernetes and docker image for PySpark, I build the packages into the image itself as below in the dockerfile
RUN pip install pyyaml numpy cx_Oracle and that will add those packages that you can reference in your py script import yaml import cx_Oracle HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 24 Nov 2021 at 17:44, Bode, Meikel, NMA-CFD < meikel.b...@bertelsmann.de> wrote: > Can we add Python dependencies as we can do for mvn coordinates? So that > we run sth like pip install <dep> or download from pypi index? > > > > *From:* Mich Talebzadeh <mich.talebza...@gmail.com> > *Sent:* Mittwoch, 24. November 2021 18:28 > *Cc:* user@spark.apache.org > *Subject:* Re: [issue] not able to add external libs to pyspark job while > using spark-submit > > > > The easiest way to set this up is to create dependencies.zip file. > > > > Assuming that you have a virtual environment already set-up, where there > is directory called site-packages, go to that directory and just create a > minimal a shell script say package_and_zip_dependencies.sh to do it for > you > > > > Example: > > > > cat package_and_zip_dependencies.sh > > > > #!/bin/bash > > # https://blog.danielcorin.com/posts/2015-11-09-pyspark/ > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.danielcorin.com%2Fposts%2F2015-11-09-pyspark%2F&data=04%7C01%7CMeikel.Bode%40bertelsmann.de%7Cbdadcaa955124c44178808d9af6fcf46%7C1ca8bd943c974fc68955bad266b43f0b%7C0%7C0%7C637733717018773969%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WyMHGm1PfvLfcoyUfu0mQRewFxJ6%2FSLz1Q6hCjnySnM%3D&reserved=0> > > zip -r ../dependencies.zip . > > ls -l ../dependencies.zip > > exit 0 > > > > One created, create an environment variable called DEPENDENCIES > > > > export DEPENDENCIES="export > DEPENDENCIES="/usr/src/Python-3.7.3/airflow_virtualenv/lib/python3.7/dependencies.zip" > > > > Then in spark-submit you can do this > > > > spark-submit --master yarn --deploy-mode client --driver-memory xG > --executor-memory yG --num-executors m --executor-cores n --py-files > $DEPENDENCIES --jars $HOME/jars/spark-sql-kafka-0-10_2.12-3.1.0.jar > > > > Also check this link as well > https://blog.danielcorin.com/posts/2015-11-09-pyspark/ > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.danielcorin.com%2Fposts%2F2015-11-09-pyspark%2F&data=04%7C01%7CMeikel.Bode%40bertelsmann.de%7Cbdadcaa955124c44178808d9af6fcf46%7C1ca8bd943c974fc68955bad266b43f0b%7C0%7C0%7C637733717018783923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=OSzimB4rV0vksIgvoEdQedI47NNxi5EH6XmucYGT%2Bpo%3D&reserved=0> > > > > HTH > > > > > > view my Linkedin profile > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=04%7C01%7CMeikel.Bode%40bertelsmann.de%7Cbdadcaa955124c44178808d9af6fcf46%7C1ca8bd943c974fc68955bad266b43f0b%7C0%7C0%7C637733717018783923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=drsA5Ywhxbav%2Bj2E255t4I14lS4wEXAQ5gEtsdIpbZo%3D&reserved=0> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > > > > On Wed, 24 Nov 2021 at 14:03, Atheer Alabdullatif <a.alabdulla...@lean.sa> > wrote: > > Dear Spark team, > > hope my email finds you well > > > > > > I am using pyspark 3.0 and facing an issue with adding external library > [configparser] while running the job using [spark-submit] & [yarn] > > issue: > > > > import configparser > > ImportError: No module named configparser > > 21/11/24 08:54:38 INFO util.ShutdownHookManager: Shutdown hook called > > solutions I tried: > > 1- installing library src files and adding it to the session using > [addPyFile]: > > - files structure: > > -- main dir > > -- subdir > > -- libs > > -- configparser-5.1.0 > > -- src > > -- configparser.py > > -- configparser.zip > > -- sparkjob.py > > 1.a zip file: > > spark = SparkSession.builder.appName(jobname + '_' + table).config( > > "spark.mongodb.input.uri", uri + > > "." + > > table + > > "").config( > > "spark.mongodb.input.sampleSize", > > 9900000).getOrCreate() > > > > spark.sparkContext.addPyFile('/maindir/subdir/libs/configparser.zip') > > df = spark.read.format("mongo").load() > > 1.b python file > > spark = SparkSession.builder.appName(jobname + '_' + table).config( > > "spark.mongodb.input.uri", uri + > > "." + > > table + > > "").config( > > "spark.mongodb.input.sampleSize", > > 9900000).getOrCreate() > > > > spark.sparkContext.addPyFile('maindir/subdir/libs/configparser-5.1.0/src/configparser.py') > > df = spark.read.format("mongo").load() > > > > 2- using os library > > def install_libs(): > > ''' > > this function used to install external python libs in yarn > > ''' > > os.system("pip3 install configparser") > > > > if __name__ == "__main__": > > > > # install libs > > install_libs() > > > > we value your support > > best, > > Atheer Alabdullatif > > > > > > > > ****إشعار السرية وإخلاء المسؤولية**** > هذه الرسالة ومرفقاتها معدة لاستخدام المُرسل إليه المقصود بالرسالة فقط وقد > تحتوي على معلومات سرية أو محمية قانونياً، إن لم تكن الشخص المقصود فنرجو > إخطار المُرسل فوراً عن طريق الرد على هذا البريد الإلكتروني وحذف الرسالة من > البريد الإلكتروني، وعدم إبقاء نسخ منه، لا يجوز استخدام أو عرض أو نشر > المحتوى سواء بشكل مباشر أو غير مباشر دون موافقة خطية مسبقة، لا تتحمل شركة > لين مسؤولية الأضرار الناتجة عن أي فيروسات قد تحملها هذه الرسالة. > > > > **Confidentiality & Disclaimer Notice** > This e-mail message, including any attachments, is for the sole use of the > intended recipient(s) and may contain confidential and privileged > information or otherwise protected by law. If you are not the intended > recipient, please immediately notify the sender, delete the e-mail, and do > not retain any copies of it. It is prohibited to use, disseminate or > distribute the content of this e-mail, directly or indirectly, without > prior written consent. Lean accepts no liability for damage caused by any > virus that may be transmitted by this Email. > > > > > >
