I've been steadily adding more and more shared libraries to the -files option of my pipes command and have had moderate success in that each time I add a new library the app no longer fails on that library, but rather on the next one.
However, I've hit a snag. I'm getting the following error even though this file is being added to the distributed cache: stderr logs [CRAZY_LONG_PATH]/MyPipesApp: error while loading shared libraries: libboost_system-gcc41-mt-1_37.so.1.37.0: cannot open shared object file: No such file or directory Despite the fact that the file clearly does exist: $ hadoop fs -ls /uwphysics/kwiley/mosaic/c++_bin Found 14 items ... -rw-r--r-- 1 kwiley uwphysics 12144 2010-04-12 13:38 /uwphysics/kwiley/mosaic/c++_bin/libboost_system-gcc41-mt-1_37.so.1.37.0 ... Here's my run command: $ hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -files \ ...[SEVERAL .so FILES TO DISTRIBUTED CACHE] hdfs://[HDFS SITE]/uwphysics/kwiley/mosaic/c++_bin/libboost_system-gcc41-mt-1_37.so.1.37.0, \ ...[SEVERAL .so FILES TO DISTRIBUTED CACHE] -input /uwphysics/kwiley/mosaic/input -output /uwphysics/kwiley/mosaic/output -program /uwphysics/kwiley/mosaic/c++_bin/MyPipesApp Note that this is working for a number of other shared libraries all stored in the same directory on HDFS, so it isn't simply a minor path error. It's more serious than that. I'm open to suggestions as to how to go about fixing this. Thanks. ________________________________________________________________________________ Keith Wiley kwi...@keithwiley.com www.keithwiley.com "The easy confidence with which I know another man's religion is folly teaches me to suspect that my own is also." -- Mark Twain ________________________________________________________________________________
