Re: map task execution time
Hi, Am 05.04.2012 um 00:20 schrieb bikash sharma: > Is it possible to get the execution time of the constituent map/reduce > tasks of a MapReduce job (say sort) at the end of a job run? > Preferably, can we obtain this programatically? you can access the JobTracker's web UI and see the start and stop timestamps for every individual task. Since the JobTracker Java API is exposed, you can write your own application to fetch that data through your own code. Also, "hadoop job" on the command line can be used to read job statistics. Kai -- Kai Voigt k...@123.org
Re: map task execution time
Thanks Kai, I will try those. On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt wrote: > Hi, > > Am 05.04.2012 um 00:20 schrieb bikash sharma: > > > Is it possible to get the execution time of the constituent map/reduce > > tasks of a MapReduce job (say sort) at the end of a job run? > > Preferably, can we obtain this programatically? > > > you can access the JobTracker's web UI and see the start and stop > timestamps for every individual task. > > Since the JobTracker Java API is exposed, you can write your own > application to fetch that data through your own code. > > Also, "hadoop job" on the command line can be used to read job statistics. > > Kai > > > -- > Kai Voigt > k...@123.org > > > > >
Re: map task execution time
How can "hadoop job" be used to read m/r statistics ? On Thu, Apr 5, 2012 at 7:30 AM, bikash sharma wrote: > Thanks Kai, I will try those. > > On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt wrote: > > > Hi, > > > > Am 05.04.2012 um 00:20 schrieb bikash sharma: > > > > > Is it possible to get the execution time of the constituent map/reduce > > > tasks of a MapReduce job (say sort) at the end of a job run? > > > Preferably, can we obtain this programatically? > > > > > > you can access the JobTracker's web UI and see the start and stop > > timestamps for every individual task. > > > > Since the JobTracker Java API is exposed, you can write your own > > application to fetch that data through your own code. > > > > Also, "hadoop job" on the command line can be used to read job > statistics. > > > > Kai > > > > > > -- > > Kai Voigt > > k...@123.org > > > > > > > > > > > -- Jay Vyas MMSB/UCHC
Re: map task execution time
(excuse the typo in the last email : I meant "I've been playing with Cinch" , not "I've been with Cinch") On Thu, Apr 5, 2012 at 7:54 AM, Jay Vyas wrote: > How can "hadoop job" be used to read m/r statistics ? > > On Thu, Apr 5, 2012 at 7:30 AM, bikash sharma wrote: > >> Thanks Kai, I will try those. >> >> On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt wrote: >> >> > Hi, >> > >> > Am 05.04.2012 um 00:20 schrieb bikash sharma: >> > >> > > Is it possible to get the execution time of the constituent map/reduce >> > > tasks of a MapReduce job (say sort) at the end of a job run? >> > > Preferably, can we obtain this programatically? >> > >> > >> > you can access the JobTracker's web UI and see the start and stop >> > timestamps for every individual task. >> > >> > Since the JobTracker Java API is exposed, you can write your own >> > application to fetch that data through your own code. >> > >> > Also, "hadoop job" on the command line can be used to read job >> statistics. >> > >> > Kai >> > >> > >> > -- >> > Kai Voigt >> > k...@123.org >> > >> > >> > >> > >> > >> > > > > -- > Jay Vyas > MMSB/UCHC > -- Jay Vyas MMSB/UCHC
Re: map task execution time
Yes, how can we use "hadoop job" to get MR job stats, especially constituent task finish times? On Thu, Apr 5, 2012 at 9:02 AM, Jay Vyas wrote: > (excuse the typo in the last email : I meant "I've been playing with Cinch" > , not "I've been with Cinch") > > On Thu, Apr 5, 2012 at 7:54 AM, Jay Vyas wrote: > > > How can "hadoop job" be used to read m/r statistics ? > > > > On Thu, Apr 5, 2012 at 7:30 AM, bikash sharma >wrote: > > > >> Thanks Kai, I will try those. > >> > >> On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt wrote: > >> > >> > Hi, > >> > > >> > Am 05.04.2012 um 00:20 schrieb bikash sharma: > >> > > >> > > Is it possible to get the execution time of the constituent > map/reduce > >> > > tasks of a MapReduce job (say sort) at the end of a job run? > >> > > Preferably, can we obtain this programatically? > >> > > >> > > >> > you can access the JobTracker's web UI and see the start and stop > >> > timestamps for every individual task. > >> > > >> > Since the JobTracker Java API is exposed, you can write your own > >> > application to fetch that data through your own code. > >> > > >> > Also, "hadoop job" on the command line can be used to read job > >> statistics. > >> > > >> > Kai > >> > > >> > > >> > -- > >> > Kai Voigt > >> > k...@123.org > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > -- > > Jay Vyas > > MMSB/UCHC > > > > > > -- > Jay Vyas > MMSB/UCHC >
Re: map task execution time
On 04/04/2012 11:20 PM, bikash sharma wrote: Hi, Is it possible to get the execution time of the constituent map/reduce tasks of a MapReduce job (say sort) at the end of a job run? Preferably, can we obtain this programatically? Thanks, Bikash You could parse the logs inside the job's output directory (dir>/_logs) You may also find this tool to be interesting: http://www.cs.duke.edu/starfish/ It can extract all that task info for you, after doing a profiling run. -- Luca Pireddu CRS4 - Distributed Computing Group Loc. Pixina Manna Edificio 1 09010 Pula (CA), Italy Tel: +39 0709250452
