[EMAIL PROTECTED] wrote on Tue, 12 Sep 2006 17:07 -0500:
> Sam and I discussed the option of overhauling the configuration file and
> server setup options today and here is a summary of what we had come up
> with.. (Sam, please correct me if I missed out anything or misunderstood
> something)
>
> Well-known problems with the current setup are
> a) the need for 2 config files (a global one and a per-server config file)
> and the implicit reliance on a savvy admin to keep these consistent and
> synchronized.
> b) Not having a clean way to be able to do a SIGHUP or equivalent to get
> restart functionality on servers.
These are great goals. I like your approach.
> We were thinking of going about solving this in a 3 step process.
> 1) Eliminate the need for a per-server config file and pass the
> server host id as a command line parameter.
Then there is exactly one process that every reads any configuration
file from disk. It would likely be the one mentioned in fstabs.
All the other servers are started by passing a BMI address to this
primary server process on the command line. (This is maybe not what
you had in mind.)
You'll also need a command-line argument to specify each server's
address. A good default _could_ be chosen by the primary server
by looking at the peer name given by BMI and seeing if the hostname
part matches something in the config file. Some setups, like for
failover or for machines serving through a non-primary interface,
will need to be able to specify their own server address on the
command-line; it can't be guessed from hostname.
> The fs.conf file will have a
> new Defaults tag for the storage space in case it is the same on all
> servers (which is usually the case) like the log file pathname.
> Since the fs.conf aliases section maps aliases to host ids, servers can
> use that to obtain the aliases and thence the metahandle, datahandle
> ranges.
> If each server has a different storage space path, we could have a new tag
> <StorageSpace>
> localhost1 /tmp/path1
> localhost2 /tmp/path2
> ...
> </StorageSpace>
> and so on..
I'd rather see StorageSpace and LogFile as normal tags in the
Filesystem section, then add <Server ib01> ... </Server> as a
way to override any particular setting, like in ssh_config, e.g.
Most central setups will use similar pathnames on each machine.
But this is only a slight preference.
(And your "localhost1" etc. must be server aliases, not hostnames.)
> Question is, how does this server talk to the remaining servers to push
> the config requests? We could just simply use tcp for this communication
> and have the putconfig act like an implicit SIGHUP which will cause all
> servers to restart and listen on the appropriate interfaces subsequent to
> the putconfig.
Please use the existing BMI infrastructure if possible. It would
be preferable to allow any transport for these server-to-server
operations, and not to require TCP specifically.
> As regards the port numbers to do the putconfig, we could query the
> portmapper or dedicate a range of port numbers on each host for running
> the remaining servers initially.
Not crazy about portmapper or listening on random ports. There are
two cases here:
1. Initial server startup. Acts as a client in connecting to
the root server to figure its config via "getconfig". Does
not open any listening ports until this is resolved.
2. Config file modification. Here the root server acts as a
client and connects to each server on its given address with a
usual sendunexpected "putconfig" message.
I don't see the need to have a temporary ephemeral listening port
for each server.
> NOTE: each server maintains a global in-memory session identifier that is
> propagated by the putconfig() from the root-server and this identifier
> is part of every client servreq structure. If it matches with the server
> id, the requests are processed, else the clients are forced to do a
> getconfig().
That's a nice addition. Can you use the hash value of the config
file itself?
Sam said:
> To continue the discussion here that I had offline with Murali, what
> about having servers send an initial hello message to the master
> server, so that they don't have to figure out what port to listen
> on? The master can send that server's HostID, wait for a ready
> response, and then push the config file to that endpoint.
Totally.
Julian's ideas about zeroconf are a natural extension of this
centralized config management, but I think they are a little out
of scope for most current PVFS usage models (centralized well-known
setup). Maybe good for the future though.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
