Hi Jason,
when will the full version of your book be available??
On Thu, Apr 30, 2009 at 8:51 AM, jason hadoop <jason.had...@gmail.com>wrote:
> You need to make sure that the shared library is available on the
> tasktracker nodes, either by installing it, or by pushing it around via the
> distributed cache
>
>
>
> On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson <jonhson....@gmail.com>
> wrote:
>
> > Dear all,
> >
> > I wrote a plugin codes for Hadoop, which calls the interfaces
> > in Cpp-built .so library. The plugin codes are written in java,
> > so I prepared a JNI class to encapsulate the C interfaces.
> >
> > The java codes can be executed successfully when I compiled
> > it and run it standalone. However, it does not work when I embedded
> > in Hadoop. The exception shown out is (found in Hadoop logs):
> >
> >
> > ------------ screen dump ---------------------
> >
> > # grep myClass logs/* -r
> > logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
> > thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
> > java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
> > logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
> > org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
> > logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
> > thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
> > java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
> > logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
> > org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
> > ...
> >
> > --------------------------------------------------------
> >
> > It seems the library can not be loaded in Hadoop. My codes
> > (myClass.java) is like:
> >
> >
> > --------------- myClass.java ------------------
> >
> > public class myClass
> > {
> >
> > public static final Log LOG =
> > LogFactory.getLog("org.apache.hadoop.mapred.myClass");
> >
> >
> > public myClass() {
> >
> > try {
> > //System.setProperty("java.library.path",
> > "/usr/local/lib");
> >
> > /* The above line does not work, so I have to
> > do something
> > * like following line.
> > */
> > addDir(new String("/usr/local/lib"));
> > System.loadLibrary("myclass");
> > }
> > catch(UnsatisfiedLinkError e) {
> > LOG.info( "Cannot load library:\n " +
> > e.toString() );
> > }
> > catch(IOException ioe) {
> > LOG.info( "IO error:\n " +
> > ioe.toString() );
> > }
> >
> > }
> >
> > /* Since the System.setProperty() does not work, I have to add the
> > following
> > * function to force the path is added in java.library.path
> > */
> > public static void addDir(String s) throws IOException {
> >
> > try {
> > Field field =
> > ClassLoader.class.getDeclaredField("usr_paths");
> > field.setAccessible(true);
> > String[] paths = (String[])field.get(null);
> > for (int i = 0; i < paths.length; i++) {
> > if (s.equals(paths[i])) {
> > return;
> > }
> > }
> > String[] tmp = new String[paths.length+1];
> > System.arraycopy(paths,0,tmp,0,paths.length);
> > tmp[paths.length] = s;
> >
> > field.set(null,tmp);
> > } catch (IllegalAccessException e) {
> > throw new IOException("Failed to get
> > permissions to set library path");
> > } catch (NoSuchFieldException e) {
> > throw new IOException("Failed to get field
> > handle to set library path");
> > }
> > }
> >
> > public native int myClassfsMount(String subsys);
> > public native int myClassfsUmount(String subsys);
> >
> >
> > }
> >
> > --------------------------------------------------------
> >
> >
> > I don't know what missed in my codes and am wondering whether there are
> any
> > rules in Hadoop I should obey if I want to achieve my target.
> >
> > FYI, the myClassfsMount() and myClassfsUmount() will open a socket to
> call
> > services from a daemon. I would better if this design did not cause the
> > fail in
> > my codes.
> >
> >
> > Any comments?
> >
> >
> > Thanks in advance,
> >
> > Ian
> >
>
>
>
> --
> Alpha Chapters of my book on Hadoop are available
> http://www.apress.com/book/view/9781430219422
>
