I'm seeing some really bizarre behavior from Hadoop 0.19.1. I have a fairly large job with about 29000 map tasks and 72 reducers. there are 304 map task slots in the cluster. When the job starts, it runs 304 map tasks at a time. As time goes on the number of map tasks run concurrently drops. For at least half of the execution exactly 152 mappers were run at a time. Towards, the end , when there were only 100 or so tasks remaining, the number of concurrent mappers quickly fell to 2 at a time, bringing the end of the map phase to a crawl. This was the only job running on the cluster. Has anyone else seen behavior like this?

Reply via email to