Doug Cutting wrote:
Apache Wiki wrote:
+ Sort performances on 1400 nodes and 2000 nodes are pretty good too -
sorting 14TB of data on a 1400-node cluster takes 2.2 hours; sorting
20TB on a 2000-node cluster takes 2.5 hours. The updates to the above
configuration being: + * `mapred.job.tracker.handler.count = 60`
+ * `mapred.reduce.parallel.copies = 50`
+ * `tasktracker.http.threads = 50`
This is a pretty good indication of stuff that we might better specify
as proportional to cluster size. For example, we might replace the
first with something like mapred.jobtracker.tasks.per.handler=30. To
determine the number of handlers we'd determine the number of task slots
(#nodes * mapred.tasktracker.tasks.maximum) and divide that by
tasks.per.handler to determine the number of handlers. Then folks
wouldn't need to alter these settings as their cluster grows.
It's best if folks don't have to change defaults for good performance.
Not only does that simplify configuration, but it means we can more
easily change implementations. For example, if we switch to async RPC
responses, then the handler count may change significantly, and we'll
probably change the default, and it would be nice if most folks were not
overriding the default.
Thoughts? Should we file an issue?
I don't think there is an explanation of why increasing the handlers
proportionally helps (I does help, but it might be a big hammer
approach). I think ipc Q-length and q management also matters a lot. I
will open a Jira with couple thoughts/explanation/improvements regd Q
mangagement in our IPC.
Raghu.
