Re: Counters across all jobs
Counters are per-job in Hadoop MapReduce. You need an external aggregator for such cross-job counters - for e.g. a node in Zookeeper. Also, is it just for display or your job-logic depends on this? If it is the earlier, and if you don't have a problem with waiting till jobs finish, you can do a post-process on the counters of all jobs and calculate the aggregates. Thanks, +Vinod Kumar Vavilapalli Hortonworks Inc. http://hortonworks.com/ On Aug 28, 2012, at 1:20 AM, Kasi Subrahmanyam wrote: > Hi, > > I have around 4 jobs running in a controller. > How can i have a single unique counter present in all the jobs and > incremented where ever used in a job? > > For example:Consider a counter ACount. > If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6. > Can i have the counter displayed output in the jobtracker as > job1:2 > job2:2 > job3:7 > job4:13 > > Thanks, > Subbu > signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Counters across all jobs
Hi Subbu, You're probably looking for something called "Distributed counters". Take a look at this question at StackOverflow: http://stackoverflow.com/questions/2671858/distributed-sequence-number-generation Best regards, Robin Verlangen *Software engineer* * * W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/8/28 Kasi Subrahmanyam > Hi, > > I have around 4 jobs running in a controller. > How can i have a single unique counter present in all the jobs and > incremented where ever used in a job? > > For example:Consider a counter ACount. > If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6. > Can i have the counter displayed output in the jobtracker as > job1:2 > job2:2 > job3:7 > job4:13 > > Thanks, > Subbu > >
Counters across all jobs
Hi, I have around 4 jobs running in a controller. How can i have a single unique counter present in all the jobs and incremented where ever used in a job? For example:Consider a counter ACount. If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6. Can i have the counter displayed output in the jobtracker as job1:2 job2:2 job3:7 job4:13 Thanks, Subbu
