I have set "c.setOutputFormat(NullOutputFormat.class);" otherwise its showing the error "Output directory not set in JobConf."
I think this is causing troubles ... any idea? On Thu, Jul 23, 2009 at 10:12 PM, bharath vissapragada < bharathvissapragada1...@gmail.com> wrote: > I have tried apache -commons logging ... > > instead of printing the row ... i have written log.error(row) ... > even then i got the same output as follows ... > > 09/07/24 03:41:38 INFO jvm.JvmMetrics: Initializing JVM Metrics with > processName=JobTracker, sessionId= > 09/07/24 03:41:38 WARN mapred.JobClient: No job jar file set. User classes > may not be found. See JobConf(Class) or JobConf#setJar(String). > 09/07/24 03:41:39 INFO mapred.TableInputFormatBase: split: > 0->localhost.localdomain:, > 09/07/24 03:41:39 INFO mapred.JobClient: Running job: job_local_0001 > 09/07/24 03:41:39 INFO mapred.TableInputFormatBase: split: > 0->localhost.localdomain:, > 09/07/24 03:41:40 INFO mapred.MapTask: numReduceTasks: 1 > 09/07/24 03:41:40 INFO mapred.MapTask: io.sort.mb = 100 > 09/07/24 03:41:40 INFO mapred.MapTask: data buffer = 79691776/99614720 > 09/07/24 03:41:40 INFO mapred.MapTask: record buffer = 262144/327680 > 09/07/24 03:41:40 INFO mapred.MapTask: Starting flush of map output > 09/07/24 03:41:40 INFO mapred.MapTask: Finished spill 0 > 09/07/24 03:41:40 INFO mapred.TaskRunner: > Task:attempt_local_0001_m_000000_0 is done. And is in the process of > commiting > 09/07/24 03:41:40 INFO mapred.LocalJobRunner: > 09/07/24 03:41:40 INFO mapred.TaskRunner: Task > 'attempt_local_0001_m_000000_0' done. > 09/07/24 03:41:40 INFO mapred.LocalJobRunner: > 09/07/24 03:41:40 INFO mapred.Merger: Merging 1 sorted segments > 09/07/24 03:41:40 INFO mapred.Merger: Down to the last merge-pass, with 1 > segments left of total size: 333 bytes > 09/07/24 03:41:40 INFO mapred.LocalJobRunner: > 09/07/24 03:41:40 INFO mapred.TaskRunner: > Task:attempt_local_0001_r_000000_0 is done. And is in the process of > commiting > 09/07/24 03:41:40 INFO mapred.LocalJobRunner: reduce > reduce > 09/07/24 03:41:40 INFO mapred.TaskRunner: Task > 'attempt_local_0001_r_000000_0' done. > 09/07/24 03:41:40 INFO mapred.JobClient: Job complete: job_local_0001 > 09/07/24 03:41:40 INFO mapred.JobClient: Counters: 11 > 09/07/24 03:41:40 INFO mapred.JobClient: File Systems > 09/07/24 03:41:40 INFO mapred.JobClient: Local bytes read=38933 > 09/07/24 03:41:40 INFO mapred.JobClient: Local bytes written=78346 > 09/07/24 03:41:40 INFO mapred.JobClient: Map-Reduce Framework > 09/07/24 03:41:40 INFO mapred.JobClient: Reduce input groups=8 > 09/07/24 03:41:40 INFO mapred.JobClient: Combine output records=0 > 09/07/24 03:41:40 INFO mapred.JobClient: Map input records=8 > 09/07/24 03:41:40 INFO mapred.JobClient: Reduce output records=8 > 09/07/24 03:41:40 INFO mapred.JobClient: Map output bytes=315 > 09/07/24 03:41:40 INFO mapred.JobClient: Map input bytes=0 > 09/07/24 03:41:40 INFO mapred.JobClient: Combine input records=0 > 09/07/24 03:41:40 INFO mapred.JobClient: Map output records=8 > 09/07/24 03:41:40 INFO mapred.JobClient: Reduce input records=8 > > > > On Thu, Jul 23, 2009 at 9:32 PM, Jean-Daniel Cryans > <jdcry...@apache.org>wrote: > >> And you don't need any more config to run local MR jobs on HBase. But >> you do need Hadoop when running MR jobs on HBase on a cluster. >> >> Also your code is running fine as you could see, the real question is >> where is the stdout going when in local mode. When you ran your other >> MR jobs, it was on a working Hadoop setup right? So you were looking >> at the logs in the web UI? One simple thing to do is to do your >> debugging with a logger so you are sure to see your output as I >> already proposed. Another simple thing is to get a pseudo-distributed >> setup and run you HBase MR jobs with Hadoop and get your logs like I'm >> sure you did before. >> >> J-D >> >> On Thu, Jul 23, 2009 at 11:54 AM, bharath >> vissapragada<bharathvissapragada1...@gmail.com> wrote: >> > I am really thankful to you J-D for replying me inspite of ur busy >> schedule. >> > I am still in a learning stage and there are no good guides on HBase >> other >> > than Its own one .. So please spare me and I really appreciate ur help . >> > >> > Now i got ur point that there is no need of hadoop while running Hbase >> MR >> > programs .... But iam confused abt the config . I have only set the >> > JAVA_HOME in the "hbase-env.sh" and other than that i didn't do anything >> .. >> > so i wonder if my conf was wrong or some error in that simple code ... >> > because stdout worked for me while writing mapreduce programs ... >> > >> > Thanks once again! >> > >> > On Thu, Jul 23, 2009 at 9:14 PM, Jean-Daniel Cryans < >> jdcry...@apache.org>wrote: >> > >> >> The code itself is very simple, I was referring to your own >> >> description of your situation. You say you use standalone HBase yet >> >> you talk about Hadoop configuration. You also talk about the >> >> JobTracker web UI which is in no use since you run local jobs directly >> >> on HBase. >> >> >> >> J-D >> >> >> >> On Thu, Jul 23, 2009 at 11:41 AM, bharath >> >> vissapragada<bharathvissapragada1...@gmail.com> wrote: >> >> > I used stdout for debugging while writing codes in hadoop MR programs >> and >> >> it >> >> > worked fine ... >> >> > Can you please tell me wch part of the code u found confusing so that >> i >> >> can >> >> > explain it a bit clearly ... >> >> > >> >> > >> >> > On Thu, Jul 23, 2009 at 9:06 PM, Jean-Daniel Cryans < >> jdcry...@apache.org >> >> >wrote: >> >> > >> >> >> What you wrote is a bit confusing to me, sorry. >> >> >> >> >> >> The usual way to debug MR jobs is to define a logger and post with >> >> >> either info or debug level, not sysout like you did. I'm not even >> sure >> >> >> where the standard output is logged when using a local job. Also >> since >> >> >> this is local you won't see anything in your host:50030 web UI. So >> use >> >> >> apache common logging and you should see your output. >> >> >> >> >> >> J-D >> >> >> >> >> >> On Thu, Jul 23, 2009 at 11:13 AM, bharath >> >> >> vissapragada<bharathvissapragada1...@gmail.com> wrote: >> >> >> > Thanks for ur reply J-D ... Im pasting some part of the code ... >> >> >> > >> >> >> > Im doing it frm the command line .. Iam pasting some part of the >> code >> >> >> here >> >> >> > .... >> >> >> > >> >> >> > public void mapp(ImmutableBytesWritable row, RowResult value, >> >> >> > OutputCollector<Text, Text> output, Reporter reporter) throws >> >> IOException >> >> >> { >> >> >> > System.out.println(row); >> >> >> > } >> >> >> > >> >> >> > public JobConf createSubmittableJob(String[] args) throws >> IOException >> >> { >> >> >> > JobConf c = new JobConf(getConf(), >> >> >> MR_DS_Scan_Case1.class); >> >> >> > c.set("col.name", args[1]); >> >> >> > c.set("operator.name",args[2]); >> >> >> > c.set("val.name",args[3]); >> >> >> > IdentityTableMap.initJob(args[0], args[1], >> >> >> this.getClass(), >> >> >> > c); >> >> >> > c.setOutputFormat(NullOutputFormat.class); >> >> >> > return c >> >> >> > } >> >> >> > >> >> >> > As u can see ... im just printing the value of row in the map .. i >> >> can't >> >> >> see >> >> >> > in the terminal ..... >> >> >> > I only wan't the map phase ... so i didn't write any reduce phase >> .. >> >> is >> >> >> my >> >> >> > jobConf correct?? >> >> >> > >> >> >> > Also as i have already asked how to check the job logs and web >> >> interface >> >> >> > like "localhost:<port>/jobTracker.jsp"... since im running in >> local >> >> mode >> >> >> ... >> >> >> > >> >> >> > On Thu, Jul 23, 2009 at 6:32 PM, Jean-Daniel Cryans < >> >> jdcry...@apache.org >> >> >> >wrote: >> >> >> > >> >> >> >> What output do you need exactly? I see that you have 8 output >> records >> >> >> >> in your reduce task so if you take a look in your output folder >> or >> >> >> >> table (I don't know which sink you used) you should see them. >> >> >> >> >> >> >> >> Also did you run your MR inside Eclipse or in command line? >> >> >> >> >> >> >> >> Thx, >> >> >> >> >> >> >> >> J-D >> >> >> >> >> >> >> >> On Thu, Jul 23, 2009 at 8:30 AM, bharath >> >> >> >> vissapragada<bhara...@students.iiit.ac.in> wrote: >> >> >> >> > This is the output i go t.. seems everything is fine ..but no >> >> output!! >> >> >> >> > >> >> >> >> > 09/07/23 23:25:36 INFO jvm.JvmMetrics: Initializing JVM Metrics >> >> with >> >> >> >> > processName=JobTracker, sessionId= >> >> >> >> > 09/07/23 23:25:36 WARN mapred.JobClient: No job jar file set. >> User >> >> >> >> classes >> >> >> >> > may not be found. See JobConf(Class) or JobConf#setJar(String). >> >> >> >> > 09/07/23 23:25:36 INFO mapred.TableInputFormatBase: split: >> >> >> >> > 0->localhost.localdomain:, >> >> >> >> > 09/07/23 23:25:37 INFO mapred.JobClient: Running job: >> >> job_local_0001 >> >> >> >> > 09/07/23 23:25:37 INFO mapred.TableInputFormatBase: split: >> >> >> >> > 0->localhost.localdomain:, >> >> >> >> > 09/07/23 23:25:37 INFO mapred.MapTask: numReduceTasks: 1 >> >> >> >> > 09/07/23 23:25:37 INFO mapred.MapTask: io.sort.mb = 100 >> >> >> >> > 09/07/23 23:25:37 INFO mapred.MapTask: data buffer = >> >> 79691776/99614720 >> >> >> >> > 09/07/23 23:25:37 INFO mapred.MapTask: record buffer = >> >> 262144/327680 >> >> >> >> > 09/07/23 23:25:37 INFO mapred.MapTask: Starting flush of map >> output >> >> >> >> > 09/07/23 23:25:37 INFO mapred.MapTask: Finished spill 0 >> >> >> >> > 09/07/23 23:25:37 INFO mapred.TaskRunner: >> >> >> >> Task:attempt_local_0001_m_000000_0 >> >> >> >> > is done. And is in the process of commiting >> >> >> >> > 09/07/23 23:25:37 INFO mapred.LocalJobRunner: >> >> >> >> > 09/07/23 23:25:37 INFO mapred.TaskRunner: Task >> >> >> >> > 'attempt_local_0001_m_000000_0' done. >> >> >> >> > 09/07/23 23:25:37 INFO mapred.LocalJobRunner: >> >> >> >> > 09/07/23 23:25:37 INFO mapred.Merger: Merging 1 sorted segments >> >> >> >> > 09/07/23 23:25:37 INFO mapred.Merger: Down to the last >> merge-pass, >> >> >> with 1 >> >> >> >> > segments left of total size: 333 bytes >> >> >> >> > 09/07/23 23:25:37 INFO mapred.LocalJobRunner: >> >> >> >> > 09/07/23 23:25:37 INFO mapred.TaskRunner: >> >> >> >> Task:attempt_local_0001_r_000000_0 >> >> >> >> > is done. And is in the process of commiting >> >> >> >> > 09/07/23 23:25:37 INFO mapred.LocalJobRunner: reduce > reduce >> >> >> >> > 09/07/23 23:25:37 INFO mapred.TaskRunner: Task >> >> >> >> > 'attempt_local_0001_r_000000_0' done. >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Job complete: >> >> job_local_0001 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Counters: 11 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: File Systems >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Local bytes >> read=38949 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Local bytes >> >> written=78378 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Map-Reduce Framework >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Reduce input >> groups=8 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Combine output >> >> records=0 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Map input >> records=8 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Reduce output >> >> records=8 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Map output >> bytes=315 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Map input bytes=0 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Combine input >> >> records=0 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Map output >> records=8 >> >> >> >> > 09/07/23 23:25:38 INFO mapred.JobClient: Reduce input >> records=8 >> >> >> >> > >> >> >> >> > >> >> >> >> > On Thu, Jul 23, 2009 at 12:17 PM, bharath vissapragada < >> >> >> >> > bhara...@students.iiit.ac.in> wrote: >> >> >> >> > >> >> >> >> >> since i haven;t started the cluster .. i can even see the >> details >> >> in >> >> >> >> >> "localhost:<port>/jobTracker.jsp" .. i didn't even add >> anything >> >> to >> >> >> >> >> hadoop/conf/hadoop-site.xml >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Jul 23, 2009 at 12:16 PM, bharath vissapragada < >> >> >> >> >> bhara...@students.iiit.ac.in> wrote: >> >> >> >> >> >> >> >> >> >>> Hi all , >> >> >> >> >>> >> >> >> >> >>> I wanted to run HBase in standalone mode to check my Hbase MR >> >> >> programs >> >> >> >> ... >> >> >> >> >>> I have dl a built version of hbase-0.20. and i have hadoop >> 0.19.3 >> >> >> >> >>> >> >> >> >> >>> "I have set JAVA_HOME in both of them" .. then i started >> hbase >> >> and >> >> >> >> >>> inserted some tables using JAVA API .. Now i have written >> some MR >> >> >> >> programs >> >> >> >> >>> onHBase and when i run them on Hbase it runs perfectly >> without >> >> any >> >> >> >> errors >> >> >> >> >>> and all the Map -reduce statistics are displayed correctly >> but i >> >> >> get >> >> >> >> no >> >> >> >> >>> output . >> >> >> >> >>> >> >> >> >> >>> I have one doubt now .. how do HBase recognize hadoop in >> stand >> >> alone >> >> >> >> >>> mode(i haven;t started my hadoop even) .. Even simple print >> >> >> statements >> >> >> >> donot >> >> >> >> >>> work .. no output is displayed on the screen ... I doubt my >> >> config >> >> >> .... >> >> >> >> >>> >> >> >> >> >>> Do i need to add some config to run them ... Please reply ... >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> > >> > >