Nathan,
On Oct 3, 2008, at 5:18 PM, Nathan Marz wrote:
Hello,
We have been doing some profiling of our MapReduce jobs, and we are
seeing about 20% of the time of our jobs is spent calling "FileSystem
$Statistics.incrementBytesRead" when we interact with the
FileSystem. Is there a way to turn this stats-collection off?
This is interesting... could you provide more details? Are you seeing
this on Maps or Reduces? Which FileSystem exhibited this i.e. HDFS or
LocalFS? Any details on about your application?
To answer your original question - no, there isn't a way to disable
this. However, if this turns out to be a systemic problem we
definitely should consider having an option to allow users to switch
it off.
So any information you can provide helps - thanks!
Arun
Thanks,
Nathan Marz
Rapleaf
