Doğacan Güney wrote:
On 3/2/07, Dan Creswell <[EMAIL PROTECTED]> wrote:
Nigel Daley wrote:
> One more difference...
> ,,,
> OTOH, Hadoop servers and clients are currently pre-configured with
> necessary host/ports.
>
Indeed - I was toying with doing something about removing this
pre-configuration - worthwhile?
...
In a data center there are typically 5 chubby instances and
their locations are stored in DNS. A client reads their location from
DNS and then traverses them to locate the Chubby master. After
[snipped the rest]
This manual configuration in Hadoop was a limitation I noticed first
thing and have suggested Zeroconf (dynamic multicast
central-server-optional service-discovering DNS) as a solution.
http://www.zeroconf.org/
I've haven't looked into the details yet, but it is clear to me that DNS
Service Discovery would work dandy for implementing auto configuration
for Hadoop.
http://www.dns-sd.org/
Multicast also has good potential to enhance Hadoop networking
performance, but that is a separate issue.
And for those concerned that Zeroconf is something exotic, it turns out
Amazon EC2 is already using it for the parameterized launch. That's
where the funny "169.254.169.254" address comes from.
http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/TechnicalFAQ.html#d0e14061
A few notes I've collected on Zeroconf:
http://www.ifcx.org/wiki/LocalNetworking.html
Jim
