I'm seeing some really bizarre behavior from Hadoop 0.19.1. I have a
fairly large job with about 29000 map tasks and 72 reducers. there are
304 map task slots in the cluster. When the job starts, it runs 304
map tasks at a time. As time goes on the number of map tasks run
concurrently drops. For at least half of the execution exactly 152
mappers were run at a time. Towards, the end , when there were only
100 or so tasks remaining, the number of concurrent mappers quickly
fell to 2 at a time, bringing the end of the map phase to a crawl.
This was the only job running on the cluster. Has anyone else seen
behavior like this?
- Mappers become less utilized as time goes on? Nathan Marz
- Re: Mappers become less utilized as time goes on? Runping Qi
- Re: Mappers become less utilized as time goes on? Nathan Marz
- Re: Mappers become less utilized as time goes ... Pedro Vivancos