ers, and will not published (aggregated) externally until
>>>>> job completed successfully?
>>>>>
>>>>> I meant that if a Task created and updated a counter, a different Task
>>>>> has access to that counter.
>>>>>
>>>>&
want to count the number of quality errors
>>>> and then fail after X number of errors, I can't use Global counters to do
>>>> this.
>>>>
>>>>> regards,
>>>>> Lin
>>>>>
>>>>> On Fri, Oct 19, 2012 a
gt;> Lin
>>>>
>>>> On Fri, Oct 19, 2012 at 10:35 PM, Michael Segel <
>>>> michael_se...@hotmail.com> wrote:
>>>>
>>>>> As I understand it... each Task has its own counters and are
>>>>> independently updated. As the
s... I guess that will depend on the amount of memory on the JT
>>>> machine, the size of the cluster (Number of TT) and the number of
>>>> counters.
>>>>
>>>> In terms of global accessibility... Maybe.
>>>>
>>>> The rea
;>>> JT machine, the size of the cluster (Number of TT) and the number of
>>>> counters.
>>>>
>>>> In terms of global accessibility... Maybe.
>>>>
>>>> The reason I say maybe is that I'm not sure by what you mean by
&g
e reason I say maybe is that I'm not sure by what you mean by globally
>>> accessible.
>>> If a task creates and implements a dynamic counter... I know that it will
>>> eventually be reflected in the JT. However, I do not believe that a
>>> separate Task could conn
r, I do not believe that a
>>> separate Task could connect with the JT and see if the counter exists or if
>>> it could get a value or even an accurate value since the updates are
>>> asynchronous. Not to mention that I don't believe that the counters are
>>
r if it could
>> get a value or even an accurate value since the updates are asynchronous.
>> Not to mention that I don't believe that the counters are aggregated until
>> the job ends. It would make sense that the JT maintains a unique counter for
>> each task until th
are
>> aggregated until the job ends. It would make sense that the JT maintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I
o delete
> the counters so that when the task is restarted the correct count is
> maintained. ) Note, I haven't looked at the source code so I am probably
> wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma wrote:
>
>> Hi guys,
>>
>>
Fri, Oct 19, 2012 at 8:51 PM, Bejoy KS
>> >> wrote:
>> >> > Hi Jay
>> >> >
>> >> > Counters are reported at the end of a task to JT. So if a task fails
>> >> > the
>> >> > counters from that task are not send
ote:
> >> > Hi Jay
> >> >
> >> > Counters are reported at the end of a task to JT. So if a task fails
> the
> >> > counters from that task are not send to JT and hence won't be included
> >> > in
> >> > the final value of counters
ce code
> so I am probably wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma wrote:
>
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
>
>- Hadoop counter (customer defined) is global accessible (for both
>read and writ
e
>> > counters from that task are not send to JT and hence won't be included
>> > in
>> > the final value of counters from that Job.
>> > Regards
>> > Bejoy KS
>> >
>> > Sent from handheld, please excuse typos.
>> > ___
x27;t be included in
> > the final value of counters from that Job.
> > Regards
> > Bejoy KS
> >
> > Sent from handheld, please excuse typos.
> >
> > From: Jay Vyas
> > Date: Fri, 19 Oct 2012 10:18:42 -0500
> > T
That clarifies why counters can go down during a job. Very important
clarification because being able to rely on such "empemeral" counters is a
really important tool for realtime monitoring of failures: thanks guys
Thanks Harsh. Great learning from you as always. :)
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-Original Message-
From: Harsh J
Date: Fri, 19 Oct 2012 21:20:07
To: ;
Reply-To: user@hadoop.apache.org
Subject: Re: Hadoop counter
Bejoy is almost right, except that
om handheld, please excuse typos.
>
> From: Jay Vyas
> Date: Fri, 19 Oct 2012 10:18:42 -0500
> To:
> ReplyTo: user@hadoop.apache.org
> Subject: Re: Hadoop counter
>
> Ah this answers alot about why some of my dynamic counters never show up and
> i have t
: Jay Vyas
Date: Fri, 19 Oct 2012 10:18:42
To:
Reply-To: user@hadoop.apache.org
Subject: Re: Hadoop counter
Ah this answers alot about why some of my dynamic counters never show up
and i have to bite my nails waiting to see whats going on until the end of
the job- thanks.
Another question:
aintains a
>> unique counter for each task until the tasks complete. (If a task fails, it
>> would have to delete the counters so that when the task is restarted the
>> correct count is maintained. ) Note, I haven't looked at the source code
>> so I am probably wro
ly wrong.
>
> HTH
> Mike
> On Oct 19, 2012, at 5:50 AM, Lin Ma wrote:
>
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
>
>- Hadoop counter (customer defined) is global accessible (for both
>read and write) for all Mappers and Reducer
sk is restarted the correct count is maintained. )
Note, I haven't looked at the source code so I am probably wrong.
HTH
Mike
On Oct 19, 2012, at 5:50 AM, Lin Ma wrote:
> Hi guys,
>
> I have some quick questions regarding to Hadoop counter,
>
> Hadoop counter (customer
Hi guys,
I have some quick questions regarding to Hadoop counter,
- Hadoop counter (customer defined) is global accessible (for both read
and write) for all Mappers and Reducers in a job?
- What is the performance and best practices of using Hadoop counters? I
am not sure if using
23 matches
Mail list logo
