[jira] Commented: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode

2009-12-16 Thread Jeff Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791706#action_12791706
 ] 

Jeff Zhang commented on MAPREDUCE-1214:
---

Hi Ankit, what's problem with this patch ? Actually it works fine on my machine.

> Add support for counters in Hadoop Local Mode
> -
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in 
> Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode

2009-12-14 Thread Jeff Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790166#action_12790166
 ] 

Jeff Zhang commented on MAPREDUCE-1214:
---

Hi Ankit,

I checked the code in pig, and find that you use the JobClient the 
HExecutionEngine, but actually the JobClient in 
org.apache.hadoop.mapred.jobcontrol.Job submit the job. And each JobClient will 
create a LocalJobRunner, so that means you use the wrong 
JobClient/LocalJobRunner to get RunningJob. This won't affect the mapreduce 
mode, because you only have one master node as the JobTracker.

Forgive me to attach the Pig patch here.
Ankit, do you have the JIRA for this issue in Pig, I can move the patch there.



> Add support for counters in Hadoop Local Mode
> -
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in 
> Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode

2009-12-14 Thread Jeff Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Zhang updated MAPREDUCE-1214:
--

Attachment: MAPREDUCE-1214.patch

> Add support for counters in Hadoop Local Mode
> -
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in 
> Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode

2009-12-13 Thread Jeff Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789973#action_12789973
 ] 

Jeff Zhang commented on MAPREDUCE-1214:
---

Ankit,
I have tried to use counters in hadoop local mode, and it works. Also custom 
counters is supported in hadoop local mode. Hope I did not misunderstand your 
meaning.
Here's my code snippet and output
{code}
public class MapReduceExample extends Configured implements Tool {

static class MyMapper extends Mapper {
public MyMapper(){

}

protected void map(
LongWritable key,
Text value,

org.apache.hadoop.mapreduce.Mapper.Context context)
throws java.io.IOException, 
InterruptedException {
context.getCounter("mygroup", "jeff").increment(1);
context.write(key, value);
};
}

@Override
public int run(String[] args) throws Exception {
Job job = new Job();
job.setMapperClass(MyMapper.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

job.waitForCompletion(true);
return 0;
}

public static void main(String[] args) throws Exception {
FileUtils.deleteDirectory(new File("data/output"));
args = new String[] { "data/input", "data/output" };
ToolRunner.run(new MapReduceExample(), args);
}
}
{code}

{code}
2009-12-14 11:38:41,843 INFO  mapred.JobClient 
(JobClient.java:monitorAndPrintJob(1288)) -  map 100% reduce 100%
2009-12-14 11:38:41,843 INFO  mapred.JobClient 
(JobClient.java:monitorAndPrintJob(1343)) - Job complete: job_local_0001
2009-12-14 11:38:41,843 INFO  mapred.JobClient (Counters.java:log(514)) - 
Counters: 13
2009-12-14 11:38:41,843 INFO  mapred.JobClient (Counters.java:log(516)) -   
FileSystemCounters
2009-12-14 11:38:41,843 INFO  mapred.JobClient (Counters.java:log(518)) - 
FILE_BYTES_READ=26096
2009-12-14 11:38:41,843 INFO  mapred.JobClient (Counters.java:log(518)) - 
FILE_BYTES_WRITTEN=52792
2009-12-14 11:38:41,843 INFO  mapred.JobClient (Counters.java:log(516)) -   
mygroup
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
jeff=2
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(516)) -   
Map-Reduce Framework
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Reduce input groups=0
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Combine output records=0
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Map input records=2
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Reduce shuffle bytes=0
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Reduce output records=0
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Spilled Records=4
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Map output bytes=28
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Combine input records=0
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Map output records=2
2009-12-14 11:38:41,858 INFO  mapred.JobClient (Counters.java:log(518)) - 
Reduce input records=2
{code}


> Add support for counters in Hadoop Local Mode
> -
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Ankit Modi
>
> Currently there is no support for counters ( Records and Bytes written ) in 
> Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.