Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Doubt in hama

Edward J. Yoon Sat, 07 Nov 2009 03:03:44 -0800

Hi,

One very simple way is the copy jars (hbase, hama) to
{$HADOOP_HOME}/lib folder of all machines and re-boot hadoop cluster.


On Sat, Nov 7, 2009 at 11:49 AM,  <[email protected]> wrote:
> Hey,
>    The above mentioned error has got solved by adding the files 
> hbase-0.19.3.jar
> to the export classpath in $HADOOP_HOME/conf/hadoop-env.sh. But now I am 
> getting
> a new bug. This time the program cannot find files of hama-0.1.0-dev.jar. 
> Inspite
> of the fact that I have added it to the hadoop-env.sh. Do I need to do some 
> thing
> else as well ? Any other ideas ?
>
> Thank You
>
> Abhishek Agrawal
>
> SUNY- Buffalo
> (716-435-7122)
>
> On Fri 11/06/09  8:30 PM , [email protected] sent:
>> Hi,
>> I have been working for last 2-3 days trying to solve the bugs that I
>> hadmentioned previously. I solved some of them. My job is no longer running
>> locallyand I am no more getting the java heap space exception. But now I am
>> gettinganother exception. The full stack trace is as under. Once again this
>> exceptiondoes not occur when I run the matrix multiplication by using 
>> bin/hbase
>> multcommand. Appreciate any thoughts in this matter
>>
>> 03/04/30 02:34:06 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing thearguments. Applications should implement Tool for the same.
>> 03/04/30 02:34:07 INFO mapred.HTableInputFormatBase: split:
>> 0->aphrodite:,03/04/30 02:34:07 INFO mapred.JobClient: Running job:
>> job_200304291805_000403/04/30 02:34:08 INFO mapred.JobClient:  map 0% reduce 
>> 0%
>> 03/04/30 02:34:14 INFO mapred.JobClient: Task Id :
>> attempt_200304291805_0004_m_000000_0, Status : FAILED
>> java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
>> not foundat org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
>> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.mapred.TableSplit
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:247)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>> ... 1 more
>>
>>
>> Thank You
>>
>> Abhishek Agrawal
>>
>> SUNY- Buffalo
>> (716-435-7122)
>>
>> On Wed 11/04/09  3:17 AM , "Edward J. Yoon" edwardy
>> [email protected] sent:> >> Caused by:
>> java.lang.ClassNotFoundException:>
>> org.apache.commons.httpclient.HttpMethod> Firstly, you have a
> ClassNotFoundException.
>> Please add dependencies to> your project. I would recommend to add all of
>> {$HAMA_HOME/lib}.>
>> > > 03/04/27 08:02:57 INFO
>> mapred.MapTask:> io.sort.mb = 100> 03/04/27 08:02:57 WARN
>> mapred.LocalJobRunner:> job_local_0001> java.lang.OutOfMemoryError:
>> Java heap> space
>> > Regarding OutOfMemoryError, It's hard to tell.
>> Do you run the program> via eclipse?
>> >
>> > On Wed, Nov 4, 2009 at 3:59 PM,  <aa...@buffa> lo.edu> wrote:> Hey,
>> > > Ã�Â
>> Ã�Â In my program I am> creating 2 matrices, putting values in them and
>> then I> call the function> >
>> > > DenseMatrix resultMatrix=
>> > covMatrix.mult(retMatrix);>
>> > > I am not setting the number of maps/
>> reduces to> be done, the block thing. Is> there any
>> problem because of that ?Also the full> error I am getting is as under.> I 
>> noticed
>> these 2 sentences mapred.MapTask:> io.sort.mb = 100 and> 
>> mapred.LocalJobRunner:
>> job_local_0001. Can you> tell what is the significance of> these 2
>> sentences ?> >
>> > > 03/04/27 08:02:56 INFO
>> jvm.JvmMetrics:> Initializing JVM Metrics with>
>> processName=JobTracker, sessionId=> > 03/04/27 08:02:56 WARN 
>> mapred.JobClient:
>> Use> GenericOptionsParser for parsing the>
>> arguments. Applications should implement Tool> for the same.> 03/04/27 
>> 08:02:56
> WARN
>> mapred.JobClient: No job> jar file set. Ã�Â User classes
>> may> not be found. See JobConf(Class) or> JobConf#setJar(String).> 03/04/27
> 08:02:56
>> INFO> mapred.HTableInputFormatBase: split:
>> 0->aphrodite:,> 03/04/27 08:02:56 INFOmapred.JobClient: Running
>> > job: job_local_0001> 03/04/27 08:02:57
>> INFO> mapred.HTableInputFormatBase: split:
>> 0->aphrodite:,> 03/04/27 08:02:57 INFOmapred.MapTask:
>> > numReduceTasks: 1> 03/04/27 08:02:57 INFO
>> mapred.MapTask:> io.sort.mb = 100> 03/04/27 08:02:57 WARN
>> mapred.LocalJobRunner:> job_local_0001> java.lang.OutOfMemoryError:
>> Java heap> space> Ã�Â
>> Ã�Â  Ã�Â > Ã�Â at
>> >
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:498)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)>
>> ;Exception in thread "Thread-14"
>> > java.lang.NoClassDefFoundError:>
>> > org/apache/commons/httpclient/HttpMethod>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:196)>
>> ;Caused by: java.lang.ClassNotFoundException:
>> > org.apache.commons.httpclient.HttpMethod>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> java.net.URLClassLoader$1.run(URLClassLoader.java:200)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> java.security.AccessController.doPrivileged(Native> Method)> Ã�Â
>> Ã�Â  Ã�Â > Ã�Â at
>> >
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> java.lang.ClassLoader.loadClass(ClassLoader.java:251)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â ... 1 more> Exception
>> in thread "main"> java.io.IOException: Job failed!>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> org.apache.hama.util.JobManager.execute(JobManager.java:52)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> org.apache.hama.DenseMatrix.mult(DenseMatrix.java:554)>
>> Ã�Â  Ã�Â
>> Ã�Â > Ã�Â at
>> >
>> Markowitz.MarkowitzMain.main(MarkowitzMain.java:33)> had...@zeus:~
>> > /HAMA$>
>> > > Thank You
>> > >
>> > > Abhishek Agrawal
>> > >
>> > > SUNY- Buffalo
>> > > (716-435-7122)
>> > >
>> > > On Wed 11/04/09 12:19 AM , "Edward
>> J.> Yoon" edwardy
>> > [email protected]
>>  sent:>> I don't see any error during perform matrix> multiplication of a
> matrix>> with
>> itself.> >>
>> > >> How large your matrix is? (and, let me
>> know> your cluster information)>>
>> > >> On Wed, Nov 4, 2009 at 11:40 AM, Edward
>> J.> Yoon Ã�Â [email protected]
>> > > wrote:> OK, That error appears *only
>> when* perform> the>> matrix multiplication> of a
>> matrix with> itself?>> >
>> > >> > I'll check it.
>> > >> >
>> > >> > On Wed, Nov 4, 2009 at
>> 5:38> AM,>>
>> Ã�ï¿½Ã�Â  lo.edu>> wrote:>> Hi,>> >>
>> Ã�ï¿½Ã�Â  As I was> telling you yesterday>> I am getting an
>> error when I call the> matrix>> multiplication function from
>> my> program. The> >> exact error I am getting is
>> as>>> under. The call to the function>>
>> is>>> >> >>
>> Ã�ï¿½Ã�Â >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â DenseMatrix> resultMatrix=>>
>> > covMatrix.mult(covMatrix);>>>>
>> >> I am multiplying the matrix> covMatrix by>> itself just to check if
>> the>>> configuration is done correctly. But then
>> I>> am getting an exception. I tried to>>> run the default matrix 
>> multiplication>>
>> program bin/hama examples mult matA> matA>> but that worked perfectly well. 
>> SO
>> I> think> >> its not that the heap is full
>> rather> I>> have not done some setting>>
>> properly.>>> >> >> had...@zeus:~
>> > >> /HAMA$ java
>> Markowitz.MarkowitzMain>>> In matrix Multiplication>> >>
>> 03/04/26 20:29:05 INFO> jvm.JvmMetrics:>> Initializing JVM Metrics
>> with>>> processName=JobTracker,>>
>> sessionId=>> 03/04/26 20:29:05 WARN> mapred.JobClient: Use>>
>> GenericOptionsParser for parsing the>>> arguments. Applications should 
>> implement>>
>> Tool for the same.>> 03/04/26 20:29:05> WARN mapred.JobClient: No>> job jar 
>> file
>> set. Ã�ï¿½Ã�Â User> classes may>> not be found. See
>> JobConf(Class) or>> JobConf#setJar(String).>>03/04/26
>> > 20:29:05 INFO>>
>> mapred.HTableInputFormatBase: split:> 0->aphrodite:,>> 03/04/26 20:29:05
>> INFO> mapred.JobClient:> >> Running job: job_local_0001>>
>> 03/04/26> 20:29:06 INFO>>
>> mapred.HTableInputFormatBase: split:> 0->aphrodite:,>> 03/04/26 20:29:06
>> INFO> mapred.MapTask:> >> numReduceTasks: 1>> 03/04/26
>> 20:29:06> INFO mapred.MapTask:>> io.sort.mb =
>> 100>> 03/04/26 20:29:06> WARN>> mapred.LocalJobRunner:
>> > job_local_0001>>
>> java.lang.OutOfMemoryError: Java heap>> space>>
>> Ã�ï¿½Ã�Â >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:498)>>
>> >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)>
>> ;> ;>> Exception in thread
>> > "Thread-14">>
>> >
>> java.lang.NoClassDefFoundError:>>>>> 
>> org/apache/commons/httpclient/HttpMethod>>
>> Ã�ï¿½Ã�Â >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:196)>
>> ;> ;>> Caused by:
>> >
>> java.lang.ClassNotFoundException:>>> 
>> org.apache.commons.httpclient.HttpMethod>>
>> Ã�ï¿½Ã�Â >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> java.net.URLClassLoader$1.run(URLClassLoader.java:200)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at>
>> java.security.AccessController.doPrivileged(Native>> Method)>>
>> Ã�ï¿½Ã�Â >
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> java.lang.ClassLoader.loadClass(ClassLoader.java:251)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â ... 1
>> more>>> Exception in thread "main">>
>> java.io.IOException: Job failed!>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hama.util.JobManager.execute(JobManager.java:52)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> org.apache.hama.DenseMatrix.mult(DenseMatrix.java:554)>>>
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½Ã�Â
>> Ã�ï¿½>>
>> Ã�ï¿½Ã�Â at> >>
>> >
>> Markowitz.MarkowitzMain.main(MarkowitzMain.java:32)>> had...@zeus:~
>> > >> /HAMA$>>
>> > >> >> Thank You
>> > >> >>
>> > >> >>
>> > >> >> Abhishek Agrawal
>> > >> >>
>> > >> >> SUNY- Buffalo
>> > >> >> (716-435-7122)
>> > >> >>
>> > >> >> On Tue 11/03/09
>> >
>> Ã�ï¿½Ã�Â 2:58 AM ,>>
>> "Edward J. Yoon"> edwardy>> [email protected]
>> >  sent:>>> Hi,>>
>> >>>> >> >>> # The maximum amount of
>> heap to> use, in>> MB. Default is 1000.>>>
>> # export> HBASE_HEAPSIZE=1000>>
>> >>>> >> >>> Check above option
>> and> regarding>> temporary tables, there is
>> no> method>>> yet. But, You can delete them
>> using>> HbaseShell's disable and delete>>>> command If you have no important
> data in>>
>> HDFS, re-format HDFS is good>>>> idea. It's just simple tip.>>
>> >>>> >> >>> On Tue, Nov 3, 2009 at
>> 4:39> PM,>>
>> Ã�ï¿½Ã�Â  lo.edu>
>> wrote:>> Hi,>>> >
>> Ã�ï¿½Ã¯Â¿Â½
>> Ã�ï¿½Ã�Â > I have>> another doubt. I am
>> running>>> a> program of mine which results in>>>
>> creation of quite a few tables in Hbase(397> to>>> be exact). Now when I am>
>> running a>> >> matrix-matrix multiplication job I
>> am> getting a>>> heap size not enough>
>> exception.> What> >> should I do to delete these
>> 397> table>>> easily ?>>>
>> >>> > Thank You> >> >>> >
>> > >> >>> > Abhishek
>> > Agrawal>> >>> >
>> > >> >>> > SUNY-
>> Buffalo> >> >>> >
>> (716-435-7122)> >> >>> >
>> > >> >>> > On Sat
>> 10/31/09> >>
>> >
>> Ã�ï¿½Ã¯Â¿Â½
>> Ã�ï¿½Ã�Â 3:25 AM
>> ,>>>> "Edward J. Yoon">>
>> edwardy>>> [email protected]
>> > >> Ã�Â sent:>>
>> Sounds good.> Thank you, I'll update>>>
>> wiki.>>>> >>> >> (you can also change> the wiki>> pages, please>>> feel
>> free> to>> contribute!)>>>>>
>> >> On> Sat, Oct 31, 2009 at 2:28>>
>> PM,>>>>
>> Ã�ï¿½Ã¯Â¿Â½
>> Ã�ï¿½Ã�Â >> lo.edu>>> wrote:> Hey,>>>
>> >>> >>>
>> >
>> Ã�ï¿½Ã¯Â¿Â½
>> Ã�Â¯Ã�Â¿Ã�Â½>
>> Ã�ï¿½Ã¯Â¿Â½
>> >>>>>>
>> Ã�ï¿½Ã¯Â¿Â½
>> Ã�Â¯Ã�Â¿Ã�Â½>
>> Ã�ï¿½Ã¯Â¿Â½
>> Ã�ï¿½Ã�Â I just ran
>> a> test.>> I>> put a println statement
>> and it> worked.>>> Thanks a lot for> your
>> help. From>> my> experience, I feel> addition>>> >> of the following
>> statements to>> the> wiki>>> page would make
>> it> easier for new users>> to>> install/
>> get started> with>>> hama.>>> > 1.
>> Absolute paths>> have to be given in>>> the> config>> file> 2. To see>> the
>> O/P, changes have to>>> be made> to>> the source code, the>> code has
>> to be>>>> compiled> again and the necessary>> files> copied
>> to>>> >>> $HAMA_HOME>>> >>> >>
>> > Thank> You>> >>> >>
>> >> >> >>> >> >
>> > Abhishek>> Agrawal>>> >>
>> >> >> >>> >> >
>> SUNY-> Buffalo>> >>> >>
>> >> (716-435-7122)>> >>> >>
>> >> >> >>> >>
>> > >> >>> >> --
>> > >> >>> >> Best Regards,
>> Edward> J. Yoon @>> NHN,>>>
>> corp.>>> edwardy>> >>> >> [email protected]http://blog.udanax.org>>>>>
>> > >>>> >>>
>> >>> >> >>> >
>> > >> >>> >
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>> --
>> > >> >>> Best Regards, Edward J.
>> Yoon @> NHN,>> corp.>>>
>> edwardy> >> >>> [email protected]http://blog.udanax.org>>>>>
>> >>>> >> >>>
>> > >> >>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Best Regards, Edward J. Yoon
>> @> NHN,>> corp.> edwardy
>> > >> [email protected]
>> > > http://blog.udanax.org>>>> >>
>> > >>
>> > >> --
>> > >> Best Regards, Edward J. Yoon @
>> NHN,> corp.>> edwardy
>> > >> [email protected]http://blog.udanax.org>>> >>
>> > >>
>> > >
>> > >
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon @ NHN,
>> corp.> edwardy
>> > [email protected]http://blog.udanax.org>
>> >
>> >
>>
>>
>>
>
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
[email protected]
http://blog.udanax.org

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Doubt in hama

Reply via email to