Re: update LD_LIBRARY_PATH when running apache job in a YARN cluster

2018-01-17 Thread Keith Chapman
Hi Manuel, You could use the following to add a path to the library search path, --conf spark.driver.extraLibraryPath=PathToLibFolder --conf spark.executor.extraLibraryPath=PathToLibFolder Thanks, Keith. Regards, Keith. http://keith-chapman.com On Wed, Jan 17, 2018 at 5:39 PM, Manuel Sopena

update LD_LIBRARY_PATH when running apache job in a YARN cluster

2018-01-17 Thread Manuel Sopena Ballesteros
Dear Spark community, I have a spark running in a yarn cluster and I am getting some error when trying to run my python application. /home/mansop/virtenv/bin/python2.7: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory Is