RE: M/R Staticstics

[Kevin Burton]

Answers below.

 

From: Omkar Joshi [mailto:ojo...@hortonworks.com] 
Sent: Friday, April 26, 2013 7:15 PM
To: user@hadoop.apache.org
Subject: Re: M/R Staticstics

 

Have you enabled security?

No

 

can you share the output for your hdfs?

 

bin/hadoop fs -ls /

 

kevin@devUbuntu05:~$ hadoop fs -ls /

Found 2 items

drwxrwxrwx   - hdfs supergroup  0 2013-04-26 13:33 /tmp

drwxr-xr-x   - hdfs supergroup  0 2013-04-19 16:40 /user

 

and is /tmp/hadoop-yarn/staging/history/done directory present in hdfs ? if
so then what permissions?

 

kevin@devUbuntu05:~$ hadoop fs -ls -R /tmp

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn

ls: Permission denied: user=kevin, access=READ_EXECUTE,
inode="/tmp/hadoop-yarn":mapred:supergroup:drwxrwx---

 

 

kevin@devUbuntu05:~$ sudo -u hdfs hadoop fs -ls -R /tmp

[sudo] password for kevin:

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging/history

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging/history/done

drwxrwxrwt   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging/history/done_intermediate

kevin@devUbuntu05:~$

 

also please share exception stack trace...

 

There is no exception now that I created /tmp on HDFS. But I still cannot
see the logs via port 50030 on the master. In other words nothing seems to
be listening on http:devubuntu05:50030. The log for map reduce looks like:

 

2013-04-26 13:35:26,107 INFO
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService: Instantiated
MRClientService at devUbuntu05/172.16.26.68:10020

2013-04-26 13:35:26,107 INFO org.apache.hadoop.yarn.service.AbstractService:
Service:HistoryClientService is started.

2013-04-26 13:35:26,107 INFO org.apache.hadoop.yarn.service.AbstractService:
Service:org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer is started.

2013-04-26 13:35:55,290 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
History Cleaner started

2013-04-26 13:35:55,295 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
History Cleaner complete

2013-04-26 13:38:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:41:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:44:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:47:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:50:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

 

 

Thanks,

Omkar Joshi

Hortonworks Inc

 

On Fri, Apr 26, 2013 at 3:05 PM,  wrote:

  

I was able to overcome the permission exception in the log by creating an
HDFS tmp folder (hadoop fs -mkdir /tmp) and opening it up to the world
(hadoop fs -chmod a+rwx /tmp). That got rid of the exception put I still am
able to connect to port 50030 to see M/R status. More ideas?

 

Even though the exception was missing from the logs of one server in the
cluster, l looked on another server and found essentially the same
permission problem:

 

2013-04-26 13:34:56,462 FATAL
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting
JobHistoryServer

org.apache.hadoop.yarn.YarnException: Error creating done directory:
[hdfs://devubuntu05:9000/tmp/hadoop-yarn/staging/history/done]

at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.init(HistoryFileManager
.java:424)

at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.init(JobHistory.java:87)

at
org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:5
8)

 

. . . . .

 

On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote: 

 

  do you see "retired jobs" on job tracker page. There is also "job tracker
history" on the bottom of page.  

 

something like this  http://nn.zettabyte.com:50030/jobtracker.jsp

Thanks and Regards, 

Rishi Yadav 







On Fri, Apr 26, 2013 at 7:36 AM, < rkevinbur...@charter.net> wrote: 

When I submit a simple "Hello World" M/R job like WordCount it takes less
than 5 seconds. The texts show numerous methods for monitoring M/R jobs as
they are happening but I have yet to see any that show statistics about a
job after it has completed. Obviously simple jobs that take a short amount
of time don't allow time to fire up any web mage or monitoring tool to see
how it progresses through the JobTracker and TaskTracker as well as which
node it is processed on. Any suggestions on how could see this kind of data
*after* a job has completed? 

 

 

Re: M/R Staticstics

[Omkar Joshi]

Have you enabled security?

can you share the output for your hdfs?

bin/hadoop fs -ls /

and is /tmp/hadoop-yarn/staging/history/done directory present in hdfs ? if
so then what permissions?

also please share exception stack trace...

Thanks,
Omkar Joshi
Hortonworks Inc


On Fri, Apr 26, 2013 at 3:05 PM,  wrote:

>
> I was able to overcome the permission exception in the log by creating an
> HDFS tmp folder (hadoop fs -mkdir /tmp) and opening it up to the world
> (hadoop fs -chmod a+rwx /tmp). That got rid of the exception put I still am
> able to connect to port 50030 to see M/R status. More ideas?
>
> Even though the exception was missing from the logs of one server in the
> cluster, l looked on another server and found essentially the same
> permission problem:
>
> 2013-04-26 13:34:56,462 FATAL
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting
> JobHistoryServer
> org.apache.hadoop.yarn.YarnException: Error creating done directory:
> [hdfs://devubuntu05:9000/tmp/hadoop-yarn/staging/history/done]
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.init(HistoryFileManager.java:424)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.init(JobHistory.java:87)
> at
> org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58)
>
> . . . . .
>
> On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote:
>
>   do you see "retired jobs" on job tracker page. There is also "job
> tracker history" on the bottom of page.
>
> something like this  
> *http://nn.zettabyte.com:50030/jobtracker.jsp*
> Thanks and Regards,
> Rishi Yadav
>
>
>
>
>
> On Fri, Apr 26, 2013 at 7:36 AM, < *rkevinbur...@charter.net*> wrote:
> When I submit a simple "Hello World" M/R job like WordCount it takes less
> than 5 seconds. The texts show numerous methods for monitoring M/R jobs as
> they are happening but I have yet to see any that show statistics about a
> job after it has completed. Obviously simple jobs that take a short amount
> of time don't allow time to fire up any web mage or monitoring tool to see
> how it progresses through the JobTracker and TaskTracker as well as which
> node it is processed on. Any suggestions on how could see this kind of data
> *after* a job has completed?
>
>

Re: M/R Staticstics

[rkevinburton]

I was able to overcome the permission exception in the log by creating 
an HDFS tmp folder (hadoop fs -mkdir /tmp) and opening it up to the 
world (hadoop fs -chmod a+rwx /tmp). That got rid of the exception put I 
still am able to connect to port 50030 to see M/R status. More ideas?


Even though the exception was missing from the logs of one server in the 
cluster, l looked on another server and found essentially the same 
permission problem:


2013-04-26 13:34:56,462 FATAL 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting 
JobHistoryServer
org.apache.hadoop.yarn.YarnException: Error creating done directory: 
[hdfs://devubuntu05:9000/tmp/hadoop-yarn/staging/history/done]
at 
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.init(HistoryFileManager.java:424)
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistory.init(JobHistory.java:87)
at 
org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58)


. . . . .

On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote:

  do you see "retired jobs" on job tracker page. There is also "job 
tracker history" on the bottom of page. 


something like this  http://nn.zettabyte.com:50030/jobtracker.jsp 


Thanks and Regards,
Rishi Yadav



On Fri, Apr 26, 2013 at 7:36 AM, < rkevinbur...@charter.net 
 

wrote:
When I submit a simple "Hello World" M/R job like WordCount it takes 
less than 5 seconds. The texts show numerous methods for monitoring M/R 
jobs as they are happening but I have yet to see any that show 
statistics about a job after it has completed. Obviously simple jobs 
that take a short amount of time don't allow time to fire up any web 
mage or monitoring tool to see how it progresses through the JobTracker 
and TaskTracker as well as which node it is processed on. Any 
suggestions on how could see this kind of data *after* a job has 
completed?

Re: M/R Staticstics

[rkevinburton]

It seems to be related to some permission problem but I am not sure how 
to over come it:



2013-04-26 12:35:08,235 INFO 
org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init
2013-04-26 12:35:08,886 FATAL 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting 
JobHistoryServer
org.apache.hadoop.yarn.YarnException: Error creating done directory: 
[hdfs://devubuntu05:9000/tmp/hadoop-yarn/staging/history/done]
at 
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.init(HistoryFileManager.java:424)
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistory.init(JobHistory.java:87)
at 
org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58)
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.init(JobHistoryServer.java:87)
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:145)

. . . . .


On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote:

 do you see "retired jobs" on job tracker page. There is also "job 
tracker history" on the bottom of page. 


something like this  http://nn.zettabyte.com:50030/jobtracker.jsp 


Thanks and Regards,
Rishi Yadav



On Fri, Apr 26, 2013 at 7:36 AM, < rkevinbur...@charter.net 
 

wrote:
When I submit a simple "Hello World" M/R job like WordCount it takes 
less than 5 seconds. The texts show numerous methods for monitoring M/R 
jobs as they are happening but I have yet to see any that show 
statistics about a job after it has completed. Obviously simple jobs 
that take a short amount of time don't allow time to fire up any web 
mage or monitoring tool to see how it progresses through the JobTracker 
and TaskTracker as well as which node it is processed on. Any 
suggestions on how could see this kind of data *after* a job has 
completed?

Re: M/R Staticstics

[rkevinburton]

As an addendum I looked to see what was installed with  apt-cache and 
got the following outout


kevin@devUbuntu05:~$ apt-cache search hadoop
python-mrjob - MapReduce framework for writing and running Hadoop 
Streaming jobs
ubuntu-orchestra-modules-hadoop - Modules mainly used by 
orchestra-management-server
flume-ng - reliable, scalable, and manageable distributed data 
collection application

hadoop - A software platform for processing vast amounts of data
hadoop-0.20-conf-pseudo - Hadoop installation in pseudo-distributed mode 
with MRv1
hadoop-0.20-mapreduce - A software platform for processing vast amounts 
of data

hadoop-0.20-mapreduce-jobtracker - JobTracker for Hadoop
hadoop-0.20-mapreduce-jobtrackerha - High Availability JobTracker for 
Hadoop

hadoop-0.20-mapreduce-tasktracker - Task Tracker for Hadoop
hadoop-0.20-mapreduce-zkfc - Hadoop MapReduce failover controller
hadoop-client - Hadoop client side dependencies
hadoop-conf-pseudo - Pseudo-distributed Hadoop configuration
hadoop-doc - Documentation for Hadoop
hadoop-hdfs - The Hadoop Distributed File System
hadoop-hdfs-datanode - Data Node for Hadoop
hadoop-hdfs-fuse - HDFS exposed over a Filesystem in Userspace
hadoop-hdfs-journalnode - Hadoop HDFS JournalNode
hadoop-hdfs-namenode - Name Node for Hadoop
hadoop-hdfs-secondarynamenode - Secondary Name Node for Hadoop
hadoop-hdfs-zkfc - Hadoop HDFS failover controller
hadoop-httpfs - HTTPFS for Hadoop
hadoop-mapreduce - The Hadoop MapReduce (MRv2)
hadoop-mapreduce-historyserver - MapReduce History Server
hadoop-yarn - The Hadoop NextGen MapReduce (YARN)
hadoop-yarn-nodemanager - Node manager for Hadoop
hadoop-yarn-proxyserver - Web proxy for YARN
hadoop-yarn-resourcemanager - Resource manager for Hadoop
hbase - HBase is the Hadoop database
hcatalog - Apache HCatalog is a table and storage management service.
hive - A data warehouse infrastructure built on top of Hadoop
hue-common - A browser-based desktop interface for Hadoop
hue-filebrowser - A UI for the Hadoop Distributed File System (HDFS)
hue-jobbrowser - A UI for viewing Hadoop map-reduce jobs
hue-jobsub - A UI for designing and submitting map-reduce jobs to Hadoop
hue-plugins - Plug-ins for Hadoop to enable integration with Hue
hue-shell - A shell for console based Hadoop applications
libhdfs0 - JNI Bindings to access Hadoop HDFS from C
mahout - A set of Java libraries for scalable machine learning.
oozie - A workflow and coordinator sytem for Hadoop jobs.
pig - A platform for analyzing large data sets using Hadoop
pig-udf-datafu - A collection of user-defined functions for Hadoop and 
Pig.
sqoop - Tool for easy imports and exports of data sets between databases 
and HDFS
sqoop2 - Tool for easy imports and exports of data sets between 
databases and HDFS
webhcat - WEBHcat provides a REST-like web API for HCatalog and related 
Hadoop components.

cdh4-repository - Cloudera's Distribution including Apache Hadoop

So it seems that MapReduce is installed but I don't see anything in 
/etc/init.d to start it up. Ideas?


On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote:

 do you see "retired jobs" on job tracker page. There is also "job 
tracker history" on the bottom of page. 


something like this  http://nn.zettabyte.com:50030/jobtracker.jsp 


Thanks and Regards,
Rishi Yadav



On Fri, Apr 26, 2013 at 7:36 AM, < rkevinbur...@charter.net 
 

wrote:
When I submit a simple "Hello World" M/R job like WordCount it takes 
less than 5 seconds. The texts show numerous methods for monitoring M/R 
jobs as they are happening but I have yet to see any that show 
statistics about a job after it has completed. Obviously simple jobs 
that take a short amount of time don't allow time to fire up any web 
mage or monitoring tool to see how it progresses through the JobTracker 
and TaskTracker as well as which node it is processed on. Any 
suggestions on how could see this kind of data *after* a job has 
completed?

Re: M/R Staticstics

[rkevinburton]

I get a message like:

Oops! Google Chrome could not connect to devubuntu05:50030

Where devubuntu05 is the machine (JobTracker, NameNode) running hadoop.

I know it is running because when I do ps I get something like:

kevin@devUbuntu05:~$ ps aux | grep hadoop

hdfs  1095  0.0  2.7 1983656 113048 ?  Sl   Apr23   3:18 
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Dproc_datanode -Xmx1000m 
-Dhadoop.log.dir=/var/log/hadoop-hdfs 
-Dhadoop.log.file=hadoop-hdfs-datanode-devUbuntu05.log 
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs 
-Dhadoop.root.logger=INFO,RFA 
-Djava.library.path=/usr/lib/hadoop/lib/native 
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-server -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote -Dhadoop.security.logger=INFO,RFAS 
org.apache.hadoop.hdfs.server.datanode.DataNode


hdfs  1415  0.1  3.1 1972424 126388 ?  Sl   Apr23   5:52 
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Dproc_namenode -Xmx1000m 
-Dhadoop.log.dir=/var/log/hadoop-hdfs 
-Dhadoop.log.file=hadoop-hdfs-namenode-devUbuntu05.log 
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs 
-Dhadoop.root.logger=INFO,RFA 
-Djava.library.path=/usr/lib/hadoop/lib/native 
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote -Dhadoop.security.logger=INFO,RFAS 
org.apache.hadoop.hdfs.server.namenode.NameNode


hdfs  1529  0.0  2.7 1961128 111680 ?  Sl   Apr23   1:49 
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Dproc_secondarynamenode 
-Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-hdfs 
-Dhadoop.log.file=hadoop-hdfs-secondarynamenode-devUbuntu05.log 
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs 
-Dhadoop.root.logger=INFO,RFA 
-Djava.library.path=/usr/lib/hadoop/lib/native 
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote -Dhadoop.security.logger=INFO,RFAS 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode


yarn  1701  0.0  5.3 1998140 216820 ?  Sl   Apr23   2:36 
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Dproc_nodemanager -Xmx1000m 
-server -Dhadoop.log.dir=/var/log/hadoop-yarn 
-Dyarn.log.dir=/var/log/hadoop-yarn 
-Dhadoop.log.file=yarn-yarn-nodemanager-devUbuntu05.log 
-Dyarn.log.file=yarn-yarn-nodemanager-devUbuntu05.log 
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dhadoop.home.dir=/usr/lib/hadoo 
-yarn -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA 
-Djava.library.path=/usr/lib/hadoop/lib/native -classpath 
/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*:/etc/hadoop/conf/nm-config/log4j.properties 
org.apache.hadoop.yarn.server.nodemanager.NodeManager


yarn  1828  0.0  5.3 2128812 217740 ?  Sl   Apr23   2:38 
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Dproc_resourcemanager 
-Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-yarn 
-Dyarn.log.dir=/var/log/hadoop-yarn 
-Dhadoop.log.file=yarn-yarn-resourcemanager-devUbuntu05.log 
-Dyarn.log.file=yarn-yarn-resourcemanager-devUbuntu05.log 
-Dyarn.home.dir=/usr/lib/hadoop-yarn 
-Dhadoop.home.dir=/usr/lib/hadoop-yarn -Dhadoop.root.logger=INFO,RFA 
-Dyarn.root.logger=INFO,RFA 
-Djava.library.path=/usr/lib/hadoop/lib/native -classpath 
/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*:/etc/hadoop/conf/rm-config/log4j.properties 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager


So why isn't the JobTracker running?

On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote:

 do you see "retired jobs" on job tracker page. There is also "job 
tracker history" on the bottom of page. 


something like this  http://nn.zettabyte.com:50030/jobtracker.jsp 


Thanks and Regards,
Rishi Yadav



On Fri, Apr 26, 2013 at 7:36 AM, < rkevinbur...@charter.net 
 

wrote:
When I submit a simple "Hello World" M/R job like WordCount it takes 
less than 5 seconds. The texts show numerous methods for monitoring M/R 
jobs as they are happening but I have yet to see any that show 
statistics about a job after it has completed. Obviously simple jobs 
that take a short amount of time don't allow time to fire up any web 
mage or monitoring tool to see how it progresses through the JobTracker 
and TaskTracker as well as which node it

Re: M/R Staticstics

[Rishi Yadav]

do you see "retired jobs" on job tracker page. There is also "job tracker
history" on the bottom of page.

something like this http://nn.zettabyte.com:50030/jobtracker.jsp

Thanks and Regards,

Rishi Yadav




On Fri, Apr 26, 2013 at 7:36 AM,  wrote:

> When I submit a simple "Hello World" M/R job like WordCount it takes less
> than 5 seconds. The texts show numerous methods for monitoring M/R jobs as
> they are happening but I have yet to see any that show statistics about a
> job after it has completed. Obviously simple jobs that take a short amount
> of time don't allow time to fire up any web mage or monitoring tool to see
> how it progresses through the JobTracker and TaskTracker as well as which
> node it is processed on. Any suggestions on how could see this kind of data
> *after* a job has completed?
>

M/R Staticstics

[rkevinburton]

When I submit a simple "Hello World" M/R job like WordCount it takes 
less than 5 seconds. The texts show numerous methods for monitoring M/R 
jobs as they are happening but I have yet to see any that show 
statistics about a job after it has completed. Obviously simple jobs 
that take a short amount of time don't allow time to fire up any web 
mage or monitoring tool to see how it progresses through the JobTracker 
and TaskTracker as well as which node it is processed on. Any 
suggestions on how could see this kind of data *after* a job has 
completed?