Re: ClassLoader resources on executor

[Marcelo Vanzin]

On Tue, Dec 1, 2015 at 12:45 PM, Charles Allen
 wrote:
> Is there a way to pass configuration file resources to be resolvable through
> the classloader?

Not in general. If you're using YARN, you can cheat and use
"spark.yarn.dist.files" which will place those files in the classpath;
the same for "--files" in yarn cluster mode (but *not* client mode!).

-- 
Marcelo

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: ClassLoader resources on executor

[Charles Allen]

I still have to propagate the file into the directory somehow, and also
that's marked as only for legacy jobs (deprecated?), so no, I have not
experimented with it yet.

On Wed, Dec 2, 2015 at 12:53 AM Rishi Mishra  wrote:

> Did you try to use *spark.executor.extraClassPath*. The classpath
> resources will be accessible through the executors class loader which
> executes your job.
>
> On Wed, Dec 2, 2015 at 2:15 AM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> Is there a way to pass configuration file resources to be resolvable
>> through the classloader?
>>
>> For example, if I'm using a library (non-spark) that can use a
>> some-lib.properties file in the classpath/classLoader, can I pass that file
>> so that when it tries to get the resource from the classloader it is able
>> to find it?
>>
>> One potential solution is to take the files and package them as resources
>> in a jar, and include the jar as part of the spark job, but that feels like
>> a hack instead of an actual solution.
>>
>> Is there any support or planned support for such a thing?
>>
>> https://github.com/apache/spark/pull/9118 seems to tackle a similar
>> problem in a hard-coded kind of way.
>>
>> Thank you,
>> Charles Allen
>>
>
>
>
> --
> Regards,
> Rishitesh Mishra,
> SnappyData . (http://www.snappydata.io/)
>
> https://in.linkedin.com/in/rishiteshmishra
>