2.0.0 The problem was I was creating a new Configuration and giving it to the Job ctor (which I believe is demonstrated in some tutorials) whereas the correct behavior was to retrieve the preexisting Configuration and use that instead. This may be a distinction between writing a bare driver and one that overrides Configured and Tool.
On Jan 17, 2014, at 09:46 , Vinod Kumar Vavilapalli wrote: > What is the version of Hadoop that you are using? > > +Vinod > > On Jan 16, 2014, at 2:41 PM, Keith Wiley <kwi...@keithwiley.com> wrote: > >> My driver is implemented around Tool and so should be wrapping >> GenericOptionsParser internally. Nevertheless, neither -files nor >> DistributedCache methods seem to work. Usage on the command line is >> straight forward, I simply add "-files foo.py,bar.py" right after the class >> name (where those files are in the current directory I'm running hadoop >> from, i.e., the local nonHDFS filesystem). The mapper then inspects the >> file list via >> DistributedCache.getLocalCacheFiles(context.getConfiguration()) and doesn't >> see the files, there's nothing there. Likewise, if I attempt to run those >> python scripts from the mapper using hadoop.util.Shell, the files obviously >> can't be found. >> >> That should have worked, so I shouldn't have to rely on the DC methods, but >> nevertheless, I tried anyway, so in the driver I create a new Configuration, >> then call DistributedCache.addCacheFile(new URI("./foo.py"), conf), thus >> referencing the local nonHDFS file in the current working directory. I then >> add conf to the job ctor, seems straight forward. Still no dice, the mapper >> can't see the files, they simply aren't there. >> >> What on Earth am I doing wrong here? >> >> ________________________________________________________________________________ >> Keith Wiley kwi...@keithwiley.com keithwiley.com >> music.keithwiley.com >> >> "Luminous beings are we, not this crude matter." >> -- Yoda >> ________________________________________________________________________________ >> > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. ________________________________________________________________________________ Keith Wiley kwi...@keithwiley.com keithwiley.com music.keithwiley.com "It's a fine line between meticulous and obsessive-compulsive and a slippery rope between obsessive-compulsive and debilitatingly slow." -- Keith Wiley ________________________________________________________________________________