We are having difficulties running a Hadoop program making calls to external
libraries - but this occurs only when we run the program on our cluster and not
from within Eclipse where we are apparently running in Hadoop's standalone
mode. This program invokes the Open Computer Vision libraries (
I'm only guessing here and might be grossly wrong about my hunch.
Are you reusing your JVMs across tasks? Could you see if this goes
away without reuse?
Would be good if you can monitor your launched Tasks
(JConsole/VisualVM/etc.) to affirm that there's either a code-based
memory leak or some oth
I don't know if putting native-code .so files inside a jar works. A
native-code .so is not "classloaded" in the same way .class files are.
So the correct .so files probably need to exist in some physical directory
on the worker machines. You may want to doublecheck that the correct
directory on th
Actually, I just misread your email and missed the difference between your
2nd and 3rd attempts.
Are you enforcing min/max JVM heap sizes on your tasks? Are you enforcing a
ulimit (either through your shell configuration, or through Hadoop itself)?
I don't know where these "cannot allocate memory"
ge processing
algorithms.
-Original Message-
From: Ratner, Alan S (IS) [mailto:alan.rat...@ngc.com]
Sent: Friday, March 04, 2011 3:53 PM
To: common-user@hadoop.apache.org
Subject: EXT :Problem running a Hadoop program with external libraries
We are having difficulties running a Hadoop program
value to 1.28 GB.
Alan
-Original Message-
From: Ratner, Alan S (IS) [mailto:alan.rat...@ngc.com]
Sent: Tuesday, March 08, 2011 4:22 PM
To: common-user@hadoop.apache.org
Cc: Gerlach, Hannah L (IS); Andrew Levine
Subject: EXT :RE: Problem running a Hadoop program with external libraries
On Mar 8, 2011, at 1:21 PM, Ratner, Alan S (IS) wrote:
> We had tried putting all the libraries directly in HDFS with a pointer in
> mapred-site.xml:
> mapred.child.envLD_LIBRARY_PATH=/user/ngc/lib
> as described in https://issues.apache.org/jira/browse/HADOOP-2838 but this
> did not work for us
(410) 707-8605 (cell)
From: Aaron Kimball [mailto:akimbal...@gmail.com]
Sent: Friday, March 04, 2011 4:30 PM
To: common-user@hadoop.apache.org
Cc: Ratner, Alan S (IS)
Subject: EXT :Re: Problem running a Hadoop program with external libraries
Actually, I just misread your email and missed the differen
ve
> Annapolis Junction, MD 20701
> (410) 707-8605 (cell)
>
> From: Aaron Kimball [mailto:akimbal...@gmail.com]
> Sent: Friday, March 04, 2011 4:30 PM
> To: common-user@hadoop.apache.org
> Cc: Ratner, Alan S (IS)
> Subject: EXT :Re: Problem running a Hadoop program with external l
is -Xmx1536m
>> -Djava.library.path=/path/to/native/libs
>>
>>
>> Alan Ratner
>> Northrop Grumman Information Systems
>> Manager of Large-Scale Computing
>> 9020 Junction Drive
>> Annapolis Junction, MD 20701
>> (410) 707-8605 (cell)
>>
;>
> >> I take it you are suggesting that I change the java.opts command to:
> >>
> >> Mapred.child.java.opts is -Xmx1536m
> -Djava.library.path=/path/to/native/libs
> >>
> >>
> >> Alan Ratner
> >> Northrop Grumman Information Systems
11 matches
Mail list logo
