Re: Updating spark-env.sh per application

[Mich Talebzadeh]

Hi Renu,

SPARK_DIST_CLASSPATH is specific to hadoop API and it is not part of
conf/spark-env.sh, although you can add to this file as per doc here


However, two things will throw light into this


   1. Version of Spark you are using?
   2. What is the drive to modify this classpath within the app itself, can
   you add the jar file to spark-submit with classpath etc? It is not common
   to modify the env variables after spark job kicks in


HTH



   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sun, 9 May 2021 at 04:32, Renu Yadav  wrote:

> Hi Mich,
>
> In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this
> variable during runtime as wanted to exclude one lib class from it.
>
>
>
> On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, 
> wrote:
>
>> Hi,
>>
>> Environment variables Re read in when spark-submit kicks off. What
>> exactly you need to refresh at the application level?
>>
>> HTH
>>
>> On Fri, 7 May 2021 at 11:34, Renu Yadav  wrote:
>>
>>>   Hi Team,
>>>
>>> Is it possible to override the variable of spark-env.sh on application
>>> level ?
>>>
>>> Thanks & Regards,
>>> Renu Yadav
>>>
>>>
>>> On Fri, May 7, 2021 at 12:16 PM Renu Yadav  wrote:
>>>
 Hi Team,

 Is it possible to override the variable of spark-env.sh on application
 level ?

 Thanks & Regards,
 Renu Yadav

 --
>>
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>

Re: Updating spark-env.sh per application

[Jacek Laskowski]

Hi,

The easiest (but perhaps not necessarily the most flexible) is simply to
use two different versions of spark-submit script with the env var set to
two different values. Have you tried it yet?

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Sun, May 9, 2021 at 5:32 AM Renu Yadav  wrote:

> Hi Mich,
>
> In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this
> variable during runtime as wanted to exclude one lib class from it.
>
>
>
> On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, 
> wrote:
>
>> Hi,
>>
>> Environment variables Re read in when spark-submit kicks off. What
>> exactly you need to refresh at the application level?
>>
>> HTH
>>
>> On Fri, 7 May 2021 at 11:34, Renu Yadav  wrote:
>>
>>>   Hi Team,
>>>
>>> Is it possible to override the variable of spark-env.sh on application
>>> level ?
>>>
>>> Thanks & Regards,
>>> Renu Yadav
>>>
>>>
>>> On Fri, May 7, 2021 at 12:16 PM Renu Yadav  wrote:
>>>
 Hi Team,

 Is it possible to override the variable of spark-env.sh on application
 level ?

 Thanks & Regards,
 Renu Yadav

 --
>>
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>

Re: Updating spark-env.sh per application

[Renu Yadav]

Hi Mich,

In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this
variable during runtime as wanted to exclude one lib class from it.



On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, 
wrote:

> Hi,
>
> Environment variables Re read in when spark-submit kicks off. What exactly
> you need to refresh at the application level?
>
> HTH
>
> On Fri, 7 May 2021 at 11:34, Renu Yadav  wrote:
>
>>   Hi Team,
>>
>> Is it possible to override the variable of spark-env.sh on application
>> level ?
>>
>> Thanks & Regards,
>> Renu Yadav
>>
>>
>> On Fri, May 7, 2021 at 12:16 PM Renu Yadav  wrote:
>>
>>> Hi Team,
>>>
>>> Is it possible to override the variable of spark-env.sh on application
>>> level ?
>>>
>>> Thanks & Regards,
>>> Renu Yadav
>>>
>>> --
>
>
>
>view my Linkedin profile
> 
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>

Re: Updating spark-env.sh per application

[Mich Talebzadeh]

Hi,

Environment variables Re read in when spark-submit kicks off. What exactly
you need to refresh at the application level?

HTH

On Fri, 7 May 2021 at 11:34, Renu Yadav  wrote:

>   Hi Team,
>
> Is it possible to override the variable of spark-env.sh on application
> level ?
>
> Thanks & Regards,
> Renu Yadav
>
>
> On Fri, May 7, 2021 at 12:16 PM Renu Yadav  wrote:
>
>> Hi Team,
>>
>> Is it possible to override the variable of spark-env.sh on application
>> level ?
>>
>> Thanks & Regards,
>> Renu Yadav
>>
>> --



   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: Updating spark-env.sh per application

[Renu Yadav]

  Hi Team,

Is it possible to override the variable of spark-env.sh on application
level ?

Thanks & Regards,
Renu Yadav


On Fri, May 7, 2021 at 12:16 PM Renu Yadav  wrote:

> Hi Team,
>
> Is it possible to override the variable of spark-env.sh on application
> level ?
>
> Thanks & Regards,
> Renu Yadav
>
>