Hi, On 3/2/07, Dan Creswell <[EMAIL PROTECTED]> wrote:
Nigel Daley wrote: > One more difference... > > Being that JavaSpaces is a Jini service, its host/port can be > dynamically discovered (and rediscovered else where if it fails) at > run time by clients. > OTOH, Hadoop servers and clients are currently pre-configured with > necessary host/ports. > Indeed - I was toying with doing something about removing this pre-configuration - worthwhile?
Here is a description of how Google does this (As far as I understand it) (Everyone here probably knows this but anyway). Google uses Chubby to locate GFS and Bigtable masters (also probably jobtracker master too). In a data center there are typically 5 chubby instances and their locations are stored in DNS. A client reads their location from DNS and then traverses them to locate the Chubby master. After locating it, the client reads a predefined file in Chubby to locate the GFS/Bigtable/MR master. If, say, a GFS namenode dies, a GFS datanode becomes the namenode and updates the file in Chubby. If the Chubby master dies, a Chubby slave becomes the master. If a client tries to connect to the old Chubby master, the connection will timeout and client will retraverse all Chubby instances to find out which is the master. [snipped the rest] -- Doğacan Güney
