Re: Updating spark-env.sh per application

Hi Renu, SPARK_DIST_CLASSPATH is specific to hadoop API and it is not part of conf/spark-env.sh, although you can add to this file as per doc here However, two things will throw light into this 1. Version of Spark you are using?

Re: Updating spark-env.sh per application

Hi, The easiest (but perhaps not necessarily the most flexible) is simply to use two different versions of spark-submit script with the env var set to two different values. Have you tried it yet? Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books

Re: Updating spark-env.sh per application

Hi Mich, In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this variable during runtime as wanted to exclude one lib class from it. On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, wrote: > Hi, > > Environment variables Re read in when spark-submit kicks off. What exactly > you

Re: Updating spark-env.sh per application

Hi, Environment variables Re read in when spark-submit kicks off. What exactly you need to refresh at the application level? HTH On Fri, 7 May 2021 at 11:34, Renu Yadav wrote: > Hi Team, > > Is it possible to override the variable of spark-env.sh on application > level ? > > Thanks &

Re: Updating spark-env.sh per application

Hi Team, Is it possible to override the variable of spark-env.sh on application level ? Thanks & Regards, Renu Yadav On Fri, May 7, 2021 at 12:16 PM Renu Yadav wrote: > Hi Team, > > Is it possible to override the variable of spark-env.sh on application > level ? > > Thanks & Regards, >