Re: Re: Re: Re: Re: Re: Re: Re: Doubt in hama

Wed, 04 Nov 2009 00:17:40 -0800 

>> Caused by: java.lang.ClassNotFoundException: 
>> org.apache.commons.httpclient.HttpMethod

Firstly, you have a ClassNotFoundException. Please add dependencies to
your project. I would recommend to add all of {$HAMA_HOME/lib}.

> 03/04/27 08:02:57 INFO mapred.MapTask: io.sort.mb = 100
> 03/04/27 08:02:57 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.OutOfMemoryError: Java heap space

Regarding OutOfMemoryError, It's hard to tell. Do you run the program
via eclipse?

On Wed, Nov 4, 2009 at 3:59 PM,  <[email protected]> wrote:
> Hey,
>    In my program I am creating 2 matrices, putting values in them and then I
> call the function
>
> DenseMatrix resultMatrix= covMatrix.mult(retMatrix);
>
> I am not setting the number of maps/ reduces to be done, the block thing. Is
> there any problem because of that ?Also the full error I am getting is as 
> under.
> I noticed these 2 sentences mapred.MapTask: io.sort.mb = 100 and
> mapred.LocalJobRunner: job_local_0001. Can you tell what is the significance 
> of
> these 2 sentences ?
>
> 03/04/27 08:02:56 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 03/04/27 08:02:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing 
> the
> arguments. Applications should implement Tool for the same.
> 03/04/27 08:02:56 WARN mapred.JobClient: No job jar file set.  User classes 
> may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 03/04/27 08:02:56 INFO mapred.HTableInputFormatBase: split: 0->aphrodite:,
> 03/04/27 08:02:56 INFO mapred.JobClient: Running job: job_local_0001
> 03/04/27 08:02:57 INFO mapred.HTableInputFormatBase: split: 0->aphrodite:,
> 03/04/27 08:02:57 INFO mapred.MapTask: numReduceTasks: 1
> 03/04/27 08:02:57 INFO mapred.MapTask: io.sort.mb = 100
> 03/04/27 08:02:57 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.OutOfMemoryError: Java heap space
>        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:498)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
> Exception in thread "Thread-14" java.lang.NoClassDefFoundError:
> org/apache/commons/httpclient/HttpMethod
>        at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:196)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.commons.httpclient.HttpMethod
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>        ... 1 more
> Exception in thread "main" java.io.IOException: Job failed!
>        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>        at org.apache.hama.util.JobManager.execute(JobManager.java:52)
>        at org.apache.hama.DenseMatrix.mult(DenseMatrix.java:554)
>        at Markowitz.MarkowitzMain.main(MarkowitzMain.java:33)
> had...@zeus:~/HAMA$
>
> Thank You
>
> Abhishek Agrawal
>
> SUNY- Buffalo
> (716-435-7122)
>
> On Wed 11/04/09 12:19 AM , "Edward J. Yoon" [email protected] sent:
>> I don't see any error during perform matrix multiplication of a matrix
>> with itself.
>>
>> How large your matrix is? (and, let me know your cluster information)
>>
>> On Wed, Nov 4, 2009 at 11:40 AM, Edward J. Yoon  [email protected]> wrote:> OK, 
>> That error appears *only when* perform the
>> matrix multiplication> of a matrix with itself?
>> >
>> > I'll check it.
>> >
>> > On Wed, Nov 4, 2009 at 5:38 AM,
>>   lo.edu> wrote:>> Hi,
>> >>   As I was telling you yesterday
>> I am getting an error when I call the matrix>> multiplication function from 
>> my
> program. The
>> exact error I am getting is as>> under. The call to the function
>> is>>
>> >>     Â
>>       Â
>>  DenseMatrix resultMatrix=
>> covMatrix.mult(covMatrix);>>
>> >> I am multiplying the matrix covMatrix by
>> itself just to check if the>> configuration is done correctly. But then I
>> am getting an exception. I tried to>> run the default matrix multiplication
>> program bin/hama examples mult matA matA>> but that worked perfectly well. 
>> SO I
> think
>> its not that the heap is full rather I>> have not done some setting
>> properly.>>
>> >> had...@zeus:~
>> /HAMA$ java Markowitz.MarkowitzMain>> In matrix Multiplication
>> >> 03/04/26 20:29:05 INFO jvm.JvmMetrics:
>> Initializing JVM Metrics with>> processName=JobTracker,
>> sessionId=>> 03/04/26 20:29:05 WARN mapred.JobClient: Use
>> GenericOptionsParser for parsing the>> arguments. Applications should 
>> implement
>> Tool for the same.>> 03/04/26 20:29:05 WARN mapred.JobClient: No
>> job jar file set.  User classes may>> not be found. See JobConf(Class) or
>> JobConf#setJar(String).>> 03/04/26 20:29:05 INFO
>> mapred.HTableInputFormatBase: split: 0->aphrodite:,>> 03/04/26 20:29:05 INFO
> mapred.JobClient:
>> Running job: job_local_0001>> 03/04/26 20:29:06 INFO
>> mapred.HTableInputFormatBase: split: 0->aphrodite:,>> 03/04/26 20:29:06 INFO
> mapred.MapTask:
>> numReduceTasks: 1>> 03/04/26 20:29:06 INFO mapred.MapTask:
>> io.sort.mb = 100>> 03/04/26 20:29:06 WARN
>> mapred.LocalJobRunner: job_local_0001>> java.lang.OutOfMemoryError: Java heap
>> space>>     Â
>>  at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:498)>>     Â
>>  at
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)>>     Â
>>  at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)>>
> Exception in thread "Thread-14"
>> java.lang.NoClassDefFoundError:>>
>> org/apache/commons/httpclient/HttpMethod>>     Â
>>  at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:196)>>
> Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.httpclient.HttpMethod>>     Â
>>  at
>> java.net.URLClassLoader$1.run(URLClassLoader.java:200)>>     Â
>>  at java.security.AccessController.doPrivileged(Native
>> Method)>>     Â
>>  at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)>>     Â
>>  at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)>>     Â
>>  at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)>>     Â
>>  at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:251)>>     Â
>>  at
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)>>     Â
>>  ... 1 more>> Exception in thread "main"
>> java.io.IOException: Job failed!>>     Â
>>  at
>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)>>     Â
>>  at
>> org.apache.hama.util.JobManager.execute(JobManager.java:52)>>     Â
>>  at
>> org.apache.hama.DenseMatrix.mult(DenseMatrix.java:554)>>     Â
>>  at
>> Markowitz.MarkowitzMain.main(MarkowitzMain.java:32)>> had...@zeus:~
>> /HAMA$>>
>> >> Thank You
>> >>
>> >>
>> >> Abhishek Agrawal
>> >>
>> >> SUNY- Buffalo
>> >> (716-435-7122)
>> >>
>> >> On Tue 11/03/09  2:58 AM ,
>> "Edward J. Yoon" edwardy
>> [email protected] sent:>>> Hi,
>> >>>
>> >>> # The maximum amount of heap to use, in
>> MB. Default is 1000.>>> # export HBASE_HEAPSIZE=1000
>> >>>
>> >>> Check above option and regarding
>> temporary tables, there is no method>>> yet. But, You can delete them using
>> HbaseShell's disable and delete>>> command If you have no important data in
>> HDFS, re-format HDFS is good>>> idea. It's just simple tip.
>> >>>
>> >>> On Tue, Nov 3, 2009 at 4:39 PM,
>>   lo.edu> wrote:> Hi,>>> > �  I have
>> another doubt. I am running>>> a program of mine which results in>
>> creation of quite a few tables in Hbase(397 to>>> be exact). Now when I am
> running a>
>> matrix-matrix multiplication job I am getting a>>> heap size not enough
> exception.> What
>> should I do to delete these 397 table>>> easily ?>
>> >>> > Thank You
>> >>> >
>> >>> > Abhishek Agrawal
>> >>> >
>> >>> > SUNY- Buffalo
>> >>> > (716-435-7122)
>> >>> >
>> >>> > On Sat 10/31/09
>> � 3:25 AM ,>>> "Edward J. Yoon"
>> edwardy>>> [email protected]
>>  sent:>> Sounds good. Thank you, I'll update>>> wiki.>>
>> >>> >> (you can also change the wiki
>> pages, please>>> feel free to
>> contribute!)>>>>> >> On Sat, Oct 31, 2009 at 2:28
>> PM,>>> � > lo.edu>
>> wrote:> Hey,>>> >> >
>> ���>>>
>> ��� I just ran a test.
>> I>> put a println statement and it worked.>>> Thanks a lot for> your help. 
>> From
>> my> experience, I feel addition>>> >> of the following statements to
>> the> wiki>>> page would make it easier for new users
>> to>> install/ get started with>>> hama.>>> > 1. Absolute paths
>> have to be given in>>> the config>> file> 2. To see
>> the O/P, changes have to>>> be made to>> the source code, the
>> code has to be>>>> compiled again and the necessary
>> files> copied to>>> >> $HAMA_HOME>
>> >>> >> > Thank You
>> >>> >> >
>> >>> >> > Abhishek
>> Agrawal>>> >> >
>> >>> >> > SUNY- Buffalo
>> >>> >> > (716-435-7122)
>> >>> >> >
>> >>> >>
>> >>> >> --
>> >>> >> Best Regards, Edward J. Yoon @
>> NHN,>>> corp.>> edwardy
>> >>> >> [email protected]http://blog.udanax.org>>>>> >>
>> >>> >>
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon @ NHN,
>> corp.>>> edwardy
>> >>> [email protected]http://blog.udanax.org>>>
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon @ NHN,
>> corp.> edwardy
>> [email protected]> http://blog.udanax.org>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> edwardy
>> [email protected]http://blog.udanax.org
>>
>>
>>
>
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
[email protected]
http://blog.udanax.org

Reply via email to