Hi Shrinivas,
Yes, this is the behavior of the task logs when using JVM Reuse. You should
notice in the log directories for the other tasks a log index file which
specifies the byte offsets into the log files where the task starts and
stops. When viewing logs through the web UI, it will use these index files
to show you the right portion of the logs.
-Todd
On Wed, Mar 30, 2011 at 1:17 PM, Shrinivas Joshi jshrini...@gmail.comwrote:
It seems like when JVM reuse is enabled map task log data is not getting
written to their corresponding log files; log data from certain map tasks
gets appended to log files corresponding to some other map task.
For example, I have a case here where 8 map JVMs are running simultaneously
and all syslog data from map task 9, 17 and 25 gets appended in to log file
for map task 0. Whereas no syslog file gets generated in
attempt_*m_09_0/ , attempt_*m_17_0/ and attempt_*m_25_0/
folders. This job creates 32 map tasks.
This behavior might also be applicable to reduce log files, however, in our
case total # of reduce tasks is not more than max reduce JVMs running at
the
same time and hence it might not be manifesting.
BTW, this is on Apache distro 0.21.0.
-Shrinivas
--
Todd Lipcon
Software Engineer, Cloudera
