Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Doubt in hama

Wed, 11 Nov 2009 18:44:37 -0800 

Hi all,
       If I run my map reduce program in the cluster then my map reduce programs
does not start. On the other hand if I run the program in the debug mode I get a
java heap space error. I don't know what could be the problem. Any help would be
appreciated.

Thank You

Abhishek Agrawal

SUNY- Buffalo
(716-435-7122)

On Wed 11/11/09  9:33 PM , [email protected] sent:
> Hi all,
> I having a couple of problems in Hama. I am trying to call the matrix
> multiply function from my own program but I am having lots and lots of
> problems.Has any one ever done such a thing ? Any help would be appreciated.
> 
> 
> 
> Matrix resultMatrix= ((DenseMatrix) covMatrix).mult(retMatrix);
> 
> 
> Thank You
> 
> Abhishek Agrawal
> 
> SUNY- Buffalo
> (716-435-7122)
> 
> On Wed 11/04/09  3:17 AM , "Edward J. Yoon" edwardy
> [email protected] sent:> >> Caused by:
> java.lang.ClassNotFoundException:>
> org.apache.commons.httpclient.HttpMethod> Firstly, you have a
ClassNotFoundException.
> Please add dependencies to> your project. I would recommend to add all of
> {$HAMA_HOME/lib}.> 
> > > 03/04/27 08:02:57 INFO
> mapred.MapTask:> io.sort.mb = 100> 03/04/27 08:02:57 WARN
> mapred.LocalJobRunner:> job_local_0001> java.lang.OutOfMemoryError:
> Java heap> space
> > Regarding OutOfMemoryError, It's hard to tell.
> Do you run the program> via eclipse?
> > 
> > On Wed, Nov 4, 2009 at 3:59 PM,  <aa...@buffa> lo.edu> wrote:> Hey,
> > > � 
> � In my program I am> creating 2 matrices, putting values in them and
> then I> call the function> >
> > > DenseMatrix resultMatrix=
> > covMatrix.mult(retMatrix);>
> > > I am not setting the number of maps/
> reduces to> be done, the block thing. Is> there any
> problem because of that ?Also the full> error I am getting is as under.> I 
> noticed
> these 2 sentences mapred.MapTask:> io.sort.mb = 100 and> 
> mapred.LocalJobRunner:
> job_local_0001. Can you> tell what is the significance of> these 2
> sentences ?> >
> > > 03/04/27 08:02:56 INFO
> jvm.JvmMetrics:> Initializing JVM Metrics with>
> processName=JobTracker, sessionId=> > 03/04/27 08:02:56 WARN mapred.JobClient:
> Use> GenericOptionsParser for parsing the>
> arguments. Applications should implement Tool> for the same.> 03/04/27 
> 08:02:56
WARN
> mapred.JobClient: No job> jar file set. � User classes
> may> not be found. See JobConf(Class) or> JobConf#setJar(String).> 03/04/27
08:02:56
> INFO> mapred.HTableInputFormatBase: split:
> 0->aphrodite:,> 03/04/27 08:02:56 INFOmapred.JobClient: Running
> > job: job_local_0001> 03/04/27 08:02:57
> INFO> mapred.HTableInputFormatBase: split:
> 0->aphrodite:,> 03/04/27 08:02:57 INFOmapred.MapTask:
> > numReduceTasks: 1> 03/04/27 08:02:57 INFO
> mapred.MapTask:> io.sort.mb = 100> 03/04/27 08:02:57 WARN
> mapred.LocalJobRunner:> job_local_0001> java.lang.OutOfMemoryError:
> Java heap> space> � 
> �  � > � at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:498)>
> �  � 
> � > � at
> >
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)>
> �  � 
> � > � at
> >
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)>
> ;Exception in thread "Thread-14"
> > java.lang.NoClassDefFoundError:>
> > org/apache/commons/httpclient/HttpMethod>
> �  � 
> � > � at
> >
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:196)>
> ;Caused by: java.lang.ClassNotFoundException:
> > org.apache.commons.httpclient.HttpMethod>
> �  � 
> � > � at
> >
> java.net.URLClassLoader$1.run(URLClassLoader.java:200)>
> �  � 
> � > � at
> java.security.AccessController.doPrivileged(Native> Method)> � 
> �  � > � at
> >
> java.net.URLClassLoader.findClass(URLClassLoader.java:188)>
> �  � 
> � > � at
> >
> java.lang.ClassLoader.loadClass(ClassLoader.java:306)>
> �  � 
> � > � at
> >
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)>
> �  � 
> � > � at
> >
> java.lang.ClassLoader.loadClass(ClassLoader.java:251)>
> �  � 
> � > � at
> >
> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)>
> �  � 
> � > � ... 1 more> Exception
> in thread "main"> java.io.IOException: Job failed!>
> �  � 
> � > � at
> >
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)>
> �  � 
> � > � at
> >
> org.apache.hama.util.JobManager.execute(JobManager.java:52)>
> �  � 
> � > � at
> >
> org.apache.hama.DenseMatrix.mult(DenseMatrix.java:554)>
> �  � 
> � > � at
> >
> Markowitz.MarkowitzMain.main(MarkowitzMain.java:33)> had...@zeus:~
> > /HAMA$>
> > > Thank You
> > >
> > > Abhishek Agrawal
> > >
> > > SUNY- Buffalo
> > > (716-435-7122)
> > >
> > > On Wed 11/04/09 12:19 AM , "Edward
> J.> Yoon" edwardy
> > [email protected]
>  sent:>> I don't see any error during perform matrix> multiplication of a
matrix>> with
> itself.> >>
> > >> How large your matrix is? (and, let me
> know> your cluster information)>>
> > >> On Wed, Nov 4, 2009 at 11:40 AM, Edward
> J.> Yoon Ã� [email protected]
> > > wrote:> OK, That error appears *only
> when* perform> the>> matrix multiplication> of a
> matrix with> itself?>> >
> > >> > I'll check it.
> > >> >
> > >> > On Wed, Nov 4, 2009 at
> 5:38> AM,>>
> ���  lo.edu>> wrote:>> Hi,>> >>
> ���  As I was> telling you yesterday>> I am getting an
> error when I call the> matrix>> multiplication function from
> my> program. The> >> exact error I am getting is
> as>>> under. The call to the function>>
> is>>> >> >>
> ��� >
> ��� 
> ��>>
> ��� >
> ��� 
> ��� 
> ��>>
> ��� DenseMatrix> resultMatrix=>>
> > covMatrix.mult(covMatrix);>>>>
> >> I am multiplying the matrix> covMatrix by>> itself just to check if
> the>>> configuration is done correctly. But then
> I>> am getting an exception. I tried to>>> run the default matrix 
> multiplication>>
> program bin/hama examples mult matA> matA>> but that worked perfectly well. SO
> I> think> >> its not that the heap is full
> rather> I>> have not done some setting>>
> properly.>>> >> >> had...@zeus:~
> > >> /HAMA$ java
> Markowitz.MarkowitzMain>>> In matrix Multiplication>> >>
> 03/04/26 20:29:05 INFO> jvm.JvmMetrics:>> Initializing JVM Metrics
> with>>> processName=JobTracker,>>
> sessionId=>> 03/04/26 20:29:05 WARN> mapred.JobClient: Use>>
> GenericOptionsParser for parsing the>>> arguments. Applications should 
> implement>>
> Tool for the same.>> 03/04/26 20:29:05> WARN mapred.JobClient: No>> job jar 
> file
> set. ��� User> classes may>> not be found. See
> JobConf(Class) or>> JobConf#setJar(String).>>03/04/26
> > 20:29:05 INFO>>
> mapred.HTableInputFormatBase: split:> 0->aphrodite:,>> 03/04/26 20:29:05
> INFO> mapred.JobClient:> >> Running job: job_local_0001>>
> 03/04/26> 20:29:06 INFO>>
> mapred.HTableInputFormatBase: split:> 0->aphrodite:,>> 03/04/26 20:29:06
> INFO> mapred.MapTask:> >> numReduceTasks: 1>> 03/04/26
> 20:29:06> INFO mapred.MapTask:>> io.sort.mb =
> 100>> 03/04/26 20:29:06> WARN>> mapred.LocalJobRunner:
> > job_local_0001>>
> java.lang.OutOfMemoryError: Java heap>> space>>
> ��� >
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:498)>>
> >
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)>
> ;> ;>> Exception in thread
> > "Thread-14">>
> >
> java.lang.NoClassDefFoundError:>>>>> 
> org/apache/commons/httpclient/HttpMethod>>
> ��� >
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:196)>
> ;> ;>> Caused by:
> >
> java.lang.ClassNotFoundException:>>> 
> org.apache.commons.httpclient.HttpMethod>>
> ��� >
> ��� 
> ��>>
> ��� at> >>
> >
> java.net.URLClassLoader$1.run(URLClassLoader.java:200)>>>
> ��� 
> ��� 
> ��>>
> ��� at>
> java.security.AccessController.doPrivileged(Native>> Method)>>
> ��� >
> ��� 
> ��>>
> ��� at> >>
> >
> java.net.URLClassLoader.findClass(URLClassLoader.java:188)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> java.lang.ClassLoader.loadClass(ClassLoader.java:306)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> java.lang.ClassLoader.loadClass(ClassLoader.java:251)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)>>>
> ��� 
> ��� 
> ��>>
> ��� ... 1
> more>>> Exception in thread "main">>
> java.io.IOException: Job failed!>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hama.util.JobManager.execute(JobManager.java:52)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> org.apache.hama.DenseMatrix.mult(DenseMatrix.java:554)>>>
> ��� 
> ��� 
> ��>>
> ��� at> >>
> >
> Markowitz.MarkowitzMain.main(MarkowitzMain.java:32)>> had...@zeus:~
> > >> /HAMA$>>
> > >> >> Thank You
> > >> >>
> > >> >>
> > >> >> Abhishek Agrawal
> > >> >>
> > >> >> SUNY- Buffalo
> > >> >> (716-435-7122)
> > >> >>
> > >> >> On Tue 11/03/09
> >
> ��� 2:58 AM ,>>
> "Edward J. Yoon"> edwardy>> [email protected]
> >  sent:>>> Hi,>>
> >>>> >> >>> # The maximum amount of
> heap to> use, in>> MB. Default is 1000.>>>
> # export> HBASE_HEAPSIZE=1000>>
> >>>> >> >>> Check above option
> and> regarding>> temporary tables, there is
> no> method>>> yet. But, You can delete them
> using>> HbaseShell's disable and delete>>>> command If you have no important
data in>>
> HDFS, re-format HDFS is good>>>> idea. It's just simple tip.>>
> >>>> >> >>> On Tue, Nov 3, 2009 at
> 4:39> PM,>>
> ���  lo.edu>
> wrote:>> Hi,>>> >
> ���
> ��� > I have>> another doubt. I am
> running>>> a> program of mine which results in>>>
> creation of quite a few tables in Hbase(397> to>>> be exact). Now when I am>
> running a>> >> matrix-matrix multiplication job I
> am> getting a>>> heap size not enough>
> exception.> What> >> should I do to delete these
> 397> table>>> easily ?>>>
> >>> > Thank You> >> >>> >
> > >> >>> > Abhishek
> > Agrawal>> >>> >
> > >> >>> > SUNY-
> Buffalo> >> >>> >
> (716-435-7122)> >> >>> >
> > >> >>> > On Sat
> 10/31/09> >>
> >
> ���
> ��� 3:25 AM
> ,>>>> "Edward J. Yoon">>
> edwardy>>> [email protected]
> > >> � sent:>>
> Sounds good.> Thank you, I'll update>>>
> wiki.>>>> >>> >> (you can also change> the wiki>> pages, please>>> feel
> free> to>> contribute!)>>>>>
> >> On> Sat, Oct 31, 2009 at 2:28>>
> PM,>>>>
> ���
> ��� >> lo.edu>>> wrote:> Hey,>>>
> >>> >>>
> >
> ���
> �¯�¿�½>
> ���
> >>>>>>
> ���
> �¯�¿�½>
> ���
> ��� I just ran
> a> test.>> I>> put a println statement
> and it> worked.>>> Thanks a lot for> your
> help. From>> my> experience, I feel> addition>>> >> of the following
> statements to>> the> wiki>>> page would make
> it> easier for new users>> to>> install/
> get started> with>>> hama.>>> > 1.
> Absolute paths>> have to be given in>>> the> config>> file> 2. To see>> the
> O/P, changes have to>>> be made> to>> the source code, the>> code has
> to be>>>> compiled> again and the necessary>> files> copied
> to>>> >>> $HAMA_HOME>>> >>> >>
> > Thank> You>> >>> >>
> >> >> >>> >> >
> > Abhishek>> Agrawal>>> >>
> >> >> >>> >> >
> SUNY-> Buffalo>> >>> >>
> >> (716-435-7122)>> >>> >>
> >> >> >>> >>
> > >> >>> >> --
> > >> >>> >> Best Regards,
> Edward> J. Yoon @>> NHN,>>>
> corp.>>> edwardy>> >>> >> [email protected]http://blog.udanax.org>>>>>
> > >>>> >>>
> >>> >> >>> >
> > >> >>> >
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> --
> > >> >>> Best Regards, Edward J.
> Yoon @> NHN,>> corp.>>>
> edwardy> >> >>> [email protected]http://blog.udanax.org>>>>>
> >>>> >> >>>
> > >> >>
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Best Regards, Edward J. Yoon
> @> NHN,>> corp.> edwardy
> > >> [email protected]
> > > http://blog.udanax.org>>>> >>
> > >>
> > >> --
> > >> Best Regards, Edward J. Yoon @
> NHN,> corp.>> edwardy
> > >> [email protected]http://blog.udanax.org>>> >>
> > >>
> > >
> > >
> > 
> > 
> > 
> > -- 
> > Best Regards, Edward J. Yoon @ NHN,
> corp.> edwardy
> > [email protected]http://blog.udanax.org> 
> > 
> > 
> 
> 
>

Reply via email to