Hadoop has been used in quite varying cluster sizes (in the range
1-2000), so am strongly in favor of as much automatic configuration as
possible.
Doug Cutting wrote:
Raghu Angadi wrote:
Right now Namenode does not know about the cluster size before
starting IPC server.
Sounds like perhaps we should make the handler count, queue size, etc.
dynamically adjustable, e.g., by adding Server methods for
setHandlerCount(), setQueueSize(), etc. There's been talk of trying
to automatically adjust these within Server.java, based on load, and
that would be better yet, but short of that, we might adjust them
heuristically based on cluster size.
The urgent thing, since we expect the best settings for large clusters
to change, is to make it so that folks don't need to adjust these
manually, even if the automation is an ill-understood heuristic. I
think we can easily get some workable heuristics into 0.15, but we
might not get be able to implement async responses or figure out how
to adjust it automatically in Server.java or whatever in that
timeframe. Perhaps we should just change the defaults to be big
enough for 2000 nodes, but that seems like too big of a hammer.
Doug
- Re: [Lucene-hadoop Wiki] Update of "FAQ" by ... Enis Soztutar
-
