On Sep 12, 2006, at 5:07 PM, Murali Vilayannur wrote:
Hey guys,
Sam and I discussed the option of overhauling the configuration
file and
server setup options today and here is a summary of what we had
come up
with.. (Sam, please correct me if I missed out anything or
misunderstood
something)
Well-known problems with the current setup are
a) the need for 2 config files (a global one and a per-server
config file)
and the implicit reliance on a savvy admin to keep these consistent
and
synchronized.
b) Not having a clean way to be able to do a SIGHUP or equivalent
to get
restart functionality on servers.
We were thinking of going about solving this in a 3 step process.
1) Eliminate the need for a per-server config file and pass the
server host id as a command line parameter. The fs.conf file will
have a
new Defaults tag for the storage space in case it is the same on all
servers (which is usually the case) like the log file pathname.
Since the fs.conf aliases section maps aliases to host ids, servers
can
use that to obtain the aliases and thence the metahandle, datahandle
ranges.
If each server has a different storage space path, we could have a
new tag
<StorageSpace>
localhost1 /tmp/path1
localhost2 /tmp/path2
...
</StorageSpace>
and so on..
NOTE: If people feel that passing an address to the servers is in some
sense similar to having the good old config file and does not
really solve
the config problems, we could probably
just listen on the 0.0.0.0 address and any ephemeral port, and
register
the ephemeral port with the portmapper. More on this issue a little
later.
2) Now, we eliminate the need for having a synchronized fs.conf
file by
designating a single metadata server (similar to what Julian had
written
earlier today) as the sole authority of the config files for a
particular
file system and having it push the config files to all the servers.
When this metadata server starts up, it parses through all the
hostid's
with the exception of itself and then connects to all the remaining
servers associated with that fsid and does a putconfig request (a new
request type).
Question is, how does this server talk to the remaining servers to
push
the config requests? We could just simply use tcp for this
communication
and have the putconfig act like an implicit SIGHUP which will cause
all
servers to restart and listen on the appropriate interfaces
subsequent to
the putconfig.
As regards the port numbers to do the putconfig, we could query the
portmapper or dedicate a range of port numbers on each host for
running
the remaining servers initially.
So what we have is as follows
<Root Server> <Other servers>
start by providing either the actual
bmi url
hostid or any ephemeral port number as
cmdline
parameter.
setup the sm's etc and wait for a putconfig()
All other requests will be Nack'ed
parse the fs.conf
files and issue a putconfig()
to all the remaining servers.
And until all the remaining
servers acknowledge, continue
doing this.
Start waiting for regular
reqs to show up
It is now enough for
SIGHUP to be sent to
the RootServer. All other
servers will ignore
SIGHUP. The root server
will issue a putconfig()
to all servers.
as soon as we get a putconfig, act like
it was
a SIGHUP, parse the config files and
restart
implicitly.
Now start servicing new requests..
Any subsequent putconfig() requests will again
act like a SIGHUP (wait for current
reqs to be
drained, new reqs NACked) and restart.
NOTE: each server maintains a global in-memory session identifier
that is
propagated by the putconfig() from the root-server and this identifier
is part of every client servreq structure. If it matches with the
server
id, the requests are processed, else the clients are forced to do a
getconfig().
I still dont think we have addressed all issues here.. For instance
server
(non root) restarts should cause putconfig's implicitly or
getconfig's?
Should the session id be on db on the root servers (and non root
servers)
or is it sufficient to be in memory?
I felt this topic requires some discussion and brain storming before
prototyping and hence this longish email.
To continue the discussion here that I had offline with Murali, what
about having servers send an initial hello message to the master
server, so that they don't have to figure out what port to listen
on? The master can send that server's HostID, wait for a ready
response, and then push the config file to that endpoint.
-sam
Comments and suggestions welcome!
thanks,
Murali
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
