Putting files on the classpath doesn't make them accessible to JVM's
resource loader. If you have dir/foo.properties, then "dir" needs to
be on the classpath, not "dir/foo.properties". Since the working dir
of the task is on the classpath, then -files works since it gets the
properties file into a directory on the classpath.
-Todd
On Mon, Aug 15, 2011 at 8:09 PM, Anty <anty....@gmail.com> wrote:
> thanks very much for you reply, todd.
> I am at a complete loss. I want to ship a configuration file to the
> cluster to run my mapreduce job.
>
> if I use -libjars option to ship the configuration file, the launched
> child JVM created by task tracker
> can't find the configuration file,curiously, the configuration file
> is already on the classpath of the child JVM.
>
> if I use -files option to ship the configuration file, the child JVM
> can find the file.
> IMO, what's the difference between -libjars and -files is that -files
> will create a symbol sink to the configuration file
> in current workding directory of child JVM.
>
> I dig into the source code,but it's so complicated, i can't figure out
> the root cause of this.
> So my question is :
> with -libjars option ,the configuration file is already on the
> classpath, why classload can't the configuration file ,
> but why JVM classload CAN find the shipped jar with -libjars option?
>
> any help will be appreciated.
>
> On Tue, Aug 16, 2011 at 1:06 AM, Todd Lipcon <t...@cloudera.com> wrote:
>> Your "driver" is the program that submits the job. The task is the
>> thing that runs on the cluster. They have separate classpaths.
>>
>> Better to ask on the public lists if you want a more indepth explanation
>>
>> -Todd
>>
>> On Mon, Aug 15, 2011 at 9:02 AM, Anty <anty....@gmail.com> wrote:
>>> Hi:Todd
>>> Would you please explain a litter more?
>>>
>>> On Sat, Dec 11, 2010 at 2:08 AM, Todd Lipcon <t...@cloudera.com> wrote:
>>>>
>>>> You need to put the library jar on your classpath (eg using
>>>> HADOOP_CLASSPATH) as well. The -libjars will ship it to the cluster
>>>> and put it on the classpath of your task, but not the classpath of
>>>> your "driver" code.
>>>>
>>> I still can't understand you mean by " but not the classpath of
>>> your "driver" code."
>>>
>>> THX advance.
>>>
>>>
>>>> -Todd
>>>>
>>>> On Thu, Dec 9, 2010 at 10:29 PM, Vipul Pandey <vipan...@gmail.com> wrote:
>>>> > disclaimer : a newbie!!!
>>>> > Howdy?
>>>> > Got a quick question. -libjars option doesn't seem to work for me in -
>>>> > prettymuch - my first (or mayby second) mapreduce job.
>>>> > Here's what i'm doing :
>>>> > $bin/hadoop jar sherlock.jar somepkg.FindSchoolsJob -libjars
>>>> > HStats-1A18.jar input output
>>>> >
>>>> > sherlock.jar has my main class (ofcourse) FindSchoolsJob, which runs
>>>> > just
>>>> > fine by itself till I add a dependency on a class in HStats-1A18.jar.
>>>> > When I run the above command with -libjars specified - it fails to find
>>>> > my
>>>> > classes that 'are' inside HStats jar file.
>>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> > com/*****/HAgent
>>>> > at com.*****.FindSchoolsJob.run(FindSchoolsJob.java:46)
>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>> > at com.******.FindSchoolsJob.main(FindSchoolsJob.java:101)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>> > Caused by: java.lang.ClassNotFoundException:com/*****/HAgent
>>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> > at java.security.AccessController.doPrivileged(Native Method)
>>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> > ... 8 more
>>>> >
>>>> > My main class is defined as below :
>>>> > public class FindSchoolsJob extends Configured implements Tool {
>>>> > :
>>>> > public int run(String[] args) throws Exception {
>>>> > :
>>>> > :
>>>> > }
>>>> > :
>>>> > public static void main(String[] args) throws Exception {
>>>> > int res = ToolRunner.run(new Configuration(), new FindSchoolsJob(),
>>>> > args);
>>>> > System.exit(res);
>>>> > }
>>>> > }
>>>> > Any hint would be highly appreciated.
>>>> > Thank You!
>>>> > ~V
>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>
>>>
>>>
>>> --
>>> Best Regards
>>> Anty Rao
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Best Regards
> Anty Rao
>
--
Todd Lipcon
Software Engineer, Cloudera
