Out of curiosity, how reliable are the counters from the perspective of the
JobClient while the job is in progress? While hitting 'refresh' on the
status web page for a job, I notice that my counters bounce all over the
place, showing wildly different figures second-to-second. Is that using a
different (less well-synchronized?) mechanism to access the counters than
the user has available in the JobClient? (If so, is this something we can
easily patch to make more consistent?)

- Aaron

On Fri, Nov 7, 2008 at 12:21 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

>
> On Nov 7, 2008, at 12:12 PM, Brian MacKay wrote:
>
>
>> Looking for a way to dynamically terminate a job once Reporter in a Map
>> job hits a threshold,
>>
>> Example:
>>
>> public void map(WritableComparable key, Text values, Output
>> Collector<Text, Text> output, Reporter reporter) throws IOException {
>>
>>    if( reporter.getCount()  > SomeConfigValue) {
>>               return;
>>           }
>>         ....  map job code
>>       }
>>
>> Obviously,  reporter.getCount() doesn't exist.  Open to other ideas, and
>> any advice would be appreciated.
>>
>
> If you _really_ need this, you could do this from your JobClient... use
> JobClient.submitJob (rather than runJob:
> http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring),
> manually fetch the Counters you need and terminate the Job via
> JobClient.getJob(jobId).killJob().
>
> Arun
>
>

Reply via email to