Out of curiosity, how reliable are the counters from the perspective of the
JobClient while the job is in progress? While hitting 'refresh' on the
status web page for a job, I notice that my counters bounce all over the
place, showing wildly different figures second-to-second. Is that using a
different (less well-synchronized?) mechanism to access the counters than
the user has available in the JobClient? (If so, is this something we can
easily patch to make more consistent?)
- Aaron
On Fri, Nov 7, 2008 at 12:21 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
> On Nov 7, 2008, at 12:12 PM, Brian MacKay wrote:
>
>
>> Looking for a way to dynamically terminate a job once Reporter in a Map
>> job hits a threshold,
>>
>> Example:
>>
>> public void map(WritableComparable key, Text values, Output
>> Collector<Text, Text> output, Reporter reporter) throws IOException {
>>
>> if( reporter.getCount() > SomeConfigValue) {
>> return;
>> }
>> .... map job code
>> }
>>
>> Obviously, reporter.getCount() doesn't exist. Open to other ideas, and
>> any advice would be appreciated.
>>
>
> If you _really_ need this, you could do this from your JobClient... use
> JobClient.submitJob (rather than runJob:
> http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring),
> manually fetch the Counters you need and terminate the Job via
> JobClient.getJob(jobId).killJob().
>
> Arun
>
>