Re: [Pvfs2-developers] RFC: Config file overhaul
I feel like the HUP should be a full server restart? Clients don't have a way to know that they need to reload the config file, currently. Likewise, since someone could just kill the server and then restart with a totally different config file, I don't think that we want to try to enforce any rules on what parameters may be changed. This is just a convenient way to reload, right? Rob Phil Carns wrote: I have a few questions about some details of the implementation: - What exactly will a HUP do on the server side? For example, is this pretty much a full server restart, a restart of the I/O components (job/trove/bmi/etc.), or just selectively reseting parameters that can be changed dynamically? - How will a reload of the configuration affect clients that already have the file system mounted? They typically have already read and parsed the configuration locally. Do they continue with the previous configuration or do they get the updates as well? - This sort of depends on the answers to the previous questions, but is there any restriction on what configuration parameters can actually be changed on the fly? There are some obvious ones that cannot be changed (fsid, root handle, range ownership, etc.). Are there any others? Do we need anything to protect against this or just document what parameters are off limits? FWIW, I agree with the direction that you guys are now heading with this stuff for the near future. The dependence on a particular servers being around for startup and use of things like portmapper would cause problems for our environment. We have to worry a good bit about firewalls, and asymmetric servers would make it harder to play nice with failover software. -Phil Rob Ross wrote: Hi all, Since I'm being discussed, I thought that I should chime in :). I agree that what we're talking about here is just a couple of steps in the right general direction, not where we would really like to be in the long run. The zero-conf concept sounds great to me, conceptually. However, I don't want to have an implementation of that that forces us into a dependence on a particular server for startup. More discussion might change my mind? I dunno. Server-to-server (s2s) seems like the right way to do the config file verification in the long term. However, the code we're going to use to do that is just now shaping up. So we're not quite ready to do that yet. So, I proposed (off-line) that we do this interim thing for now (which cleans things up quite a bit) and continue thinking and talking about how we might like the system to work in the longer term. Regards, Rob Murali Vilayannur wrote: Hi pete, I thought with your discussion of getconfig/putconfig that you were implicitly planning to get server-to-server communication to work too. I was planning on doing that. But it fizzled out because of RobR's concern about scalability and dependence on a single root server.. What you describe here sounds like fine stuff. And if later somebody gets ambitious and wants to do more "zeroconf"-ish work, they can build on this. Thanks for the explanations. Yep. Exactly. I hope so too :) thanks! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
Ok- just kinda checking to make sure of what you guys had in mind. If the HUP is a full restart of the server, then it will complain on its own if the config file doesn't match what's in the storage space, so that isn't really a danger. It also means that pretty much any tuning parameter is legal to change since all of the I/O components will be starting from scratch. Just a heads up on one detail- we need to make sure that the server keeps its pid on the HUP or else updates its pidfile so that monitoring scripts don't get confused. I think that leaving the client configuration alone is fine. -Phil Rob Ross wrote: I feel like the HUP should be a full server restart? Clients don't have a way to know that they need to reload the config file, currently. Likewise, since someone could just kill the server and then restart with a totally different config file, I don't think that we want to try to enforce any rules on what parameters may be changed. This is just a convenient way to reload, right? Rob Phil Carns wrote: I have a few questions about some details of the implementation: - What exactly will a HUP do on the server side? For example, is this pretty much a full server restart, a restart of the I/O components (job/trove/bmi/etc.), or just selectively reseting parameters that can be changed dynamically? - How will a reload of the configuration affect clients that already have the file system mounted? They typically have already read and parsed the configuration locally. Do they continue with the previous configuration or do they get the updates as well? - This sort of depends on the answers to the previous questions, but is there any restriction on what configuration parameters can actually be changed on the fly? There are some obvious ones that cannot be changed (fsid, root handle, range ownership, etc.). Are there any others? Do we need anything to protect against this or just document what parameters are off limits? FWIW, I agree with the direction that you guys are now heading with this stuff for the near future. The dependence on a particular servers being around for startup and use of things like portmapper would cause problems for our environment. We have to worry a good bit about firewalls, and asymmetric servers would make it harder to play nice with failover software. -Phil Rob Ross wrote: Hi all, Since I'm being discussed, I thought that I should chime in :). I agree that what we're talking about here is just a couple of steps in the right general direction, not where we would really like to be in the long run. The zero-conf concept sounds great to me, conceptually. However, I don't want to have an implementation of that that forces us into a dependence on a particular server for startup. More discussion might change my mind? I dunno. Server-to-server (s2s) seems like the right way to do the config file verification in the long term. However, the code we're going to use to do that is just now shaping up. So we're not quite ready to do that yet. So, I proposed (off-line) that we do this interim thing for now (which cleans things up quite a bit) and continue thinking and talking about how we might like the system to work in the longer term. Regards, Rob Murali Vilayannur wrote: Hi pete, I thought with your discussion of getconfig/putconfig that you were implicitly planning to get server-to-server communication to work too. I was planning on doing that. But it fizzled out because of RobR's concern about scalability and dependence on a single root server.. What you describe here sounds like fine stuff. And if later somebody gets ambitious and wants to do more "zeroconf"-ish work, they can build on this. Thanks for the explanations. Yep. Exactly. I hope so too :) thanks! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
I have a few questions about some details of the implementation: - What exactly will a HUP do on the server side? For example, is this pretty much a full server restart, a restart of the I/O components (job/trove/bmi/etc.), or just selectively reseting parameters that can be changed dynamically? - How will a reload of the configuration affect clients that already have the file system mounted? They typically have already read and parsed the configuration locally. Do they continue with the previous configuration or do they get the updates as well? - This sort of depends on the answers to the previous questions, but is there any restriction on what configuration parameters can actually be changed on the fly? There are some obvious ones that cannot be changed (fsid, root handle, range ownership, etc.). Are there any others? Do we need anything to protect against this or just document what parameters are off limits? FWIW, I agree with the direction that you guys are now heading with this stuff for the near future. The dependence on a particular servers being around for startup and use of things like portmapper would cause problems for our environment. We have to worry a good bit about firewalls, and asymmetric servers would make it harder to play nice with failover software. -Phil Rob Ross wrote: Hi all, Since I'm being discussed, I thought that I should chime in :). I agree that what we're talking about here is just a couple of steps in the right general direction, not where we would really like to be in the long run. The zero-conf concept sounds great to me, conceptually. However, I don't want to have an implementation of that that forces us into a dependence on a particular server for startup. More discussion might change my mind? I dunno. Server-to-server (s2s) seems like the right way to do the config file verification in the long term. However, the code we're going to use to do that is just now shaping up. So we're not quite ready to do that yet. So, I proposed (off-line) that we do this interim thing for now (which cleans things up quite a bit) and continue thinking and talking about how we might like the system to work in the longer term. Regards, Rob Murali Vilayannur wrote: Hi pete, I thought with your discussion of getconfig/putconfig that you were implicitly planning to get server-to-server communication to work too. I was planning on doing that. But it fizzled out because of RobR's concern about scalability and dependence on a single root server.. What you describe here sounds like fine stuff. And if later somebody gets ambitious and wants to do more "zeroconf"-ish work, they can build on this. Thanks for the explanations. Yep. Exactly. I hope so too :) thanks! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
Hi all, Since I'm being discussed, I thought that I should chime in :). I agree that what we're talking about here is just a couple of steps in the right general direction, not where we would really like to be in the long run. The zero-conf concept sounds great to me, conceptually. However, I don't want to have an implementation of that that forces us into a dependence on a particular server for startup. More discussion might change my mind? I dunno. Server-to-server (s2s) seems like the right way to do the config file verification in the long term. However, the code we're going to use to do that is just now shaping up. So we're not quite ready to do that yet. So, I proposed (off-line) that we do this interim thing for now (which cleans things up quite a bit) and continue thinking and talking about how we might like the system to work in the longer term. Regards, Rob Murali Vilayannur wrote: Hi pete, I thought with your discussion of getconfig/putconfig that you were implicitly planning to get server-to-server communication to work too. I was planning on doing that. But it fizzled out because of RobR's concern about scalability and dependence on a single root server.. What you describe here sounds like fine stuff. And if later somebody gets ambitious and wants to do more "zeroconf"-ish work, they can build on this. Thanks for the explanations. Yep. Exactly. I hope so too :) thanks! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
Hi pete, > I thought with your discussion of getconfig/putconfig that you were > implicitly planning to get server-to-server communication to work > too. I was planning on doing that. But it fizzled out because of RobR's concern about scalability and dependence on a single root server.. > What you describe here sounds like fine stuff. And if later > somebody gets ambitious and wants to do more "zeroconf"-ish work, > they can build on this. Thanks for the explanations. Yep. Exactly. I hope so too :) thanks! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
[EMAIL PROTECTED] wrote on Tue, 19 Sep 2006 10:41 -0500: > Without some form of server-to-server communication, verification of this > config file is probably not doable. I was not envisioning adding the > verification protocols at the server. I have vastly simplified the > problem by relying on admins to ensure that the files are kept uptodate, > which admittedly is quite unrealistic. > > Instead, the clients verify whether all servers have the same config files > which we recently added to the fs-add state machine and optionally > verify at the time of PVFS_isys_fs_add. Alright, totally makes sense. The client verification is just a sanity check that the admin has not messed anything up, then. Valid. > Admins are expected to edit the fs.conf files, copy it to all servers (or > if it was on an NFS volume edit just once), and have a new mgmt tool which > causes all servers to re-read the config files and have clients do the > verification.. Okay, that's fine, and a vast improvement on our current scenario. I thought with your discussion of getconfig/putconfig that you were implicitly planning to get server-to-server communication to work too. What you describe here sounds like fine stuff. And if later somebody gets ambitious and wants to do more "zeroconf"-ish work, they can build on this. Thanks for the explanations. -- Pete ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
Hi Pete, > Then on top of this you add some sort of verification protocol to > make sure that file is up-to-date, right? A server with an old > config file exits with error? Without some form of server-to-server communication, verification of this config file is probably not doable. I was not envisioning adding the verification protocols at the server. I have vastly simplified the problem by relying on admins to ensure that the files are kept uptodate, which admittedly is quite unrealistic. Instead, the clients verify whether all servers have the same config files which we recently added to the fs-add state machine and optionally verify at the time of PVFS_isys_fs_add. > Regarding your idea about SIGHUP, and forcing a config file re-read. > This would be implemented by sending a message to all servers that > they should locally read their fs.conf again? And verify against > the one provided by the root server? > Admins are expected to edit one of the fs.conf files, copy it to all > the server machines, then SIGHUP the root server? Admins are expected to edit the fs.conf files, copy it to all servers (or if it was on an NFS volume edit just once), and have a new mgmt tool which causes all servers to re-read the config files and have clients do the verification.. > Sorry to be tedious, but reading the earlier mails on the topic, I > obviously did not understand some fundamental aspects of your > design. Nope, my bad. I should have described things in more detail. Does this sound ok to you guys? If it is too simplistic a view of the problem, then please let me know what you would like to see to mitigate the configuration problems :) thanks! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
[EMAIL PROTECTED] wrote on Tue, 19 Sep 2006 09:57 -0500: > > Then there is exactly one process that every reads any configuration > > file from disk. It would likely be the one mentioned in fstabs. > > All the other servers are started by passing a BMI address to this > > primary server process on the command line. (This is maybe not what > > you had in mind.) > > Sam and I had discussed this possibility and I think RobR does not wish to > go down this path because every server would depend on this one particular > server to get started. > In the short term, it is easiest to remove the dependence on the second > config file by passing in the server alias as the command line parameter > and having the parsing code figure out BMI addresses, storage spaces, log > file pathname etc from the fs.conf file. Oh, I see. Your scope for overhauling the config setups is not what I had imagined. Now instead of every server reading two files from disk, every server will read one file from disk. That's a good first step. Then on top of this you add some sort of verification protocol to make sure that file is up-to-date, right? A server with an old config file exits with error? Regarding your idea about SIGHUP, and forcing a config file re-read. This would be implemented by sending a message to all servers that they should locally read their fs.conf again? And verify against the one provided by the root server? Admins are expected to edit one of the fs.conf files, copy it to all the server machines, then SIGHUP the root server? Sorry to be tedious, but reading the earlier mails on the topic, I obviously did not understand some fundamental aspects of your design. -- Pete ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
Hi pete, > Then there is exactly one process that every reads any configuration > file from disk. It would likely be the one mentioned in fstabs. > All the other servers are started by passing a BMI address to this > primary server process on the command line. (This is maybe not what > you had in mind.) Sam and I had discussed this possibility and I think RobR does not wish to go down this path because every server would depend on this one particular server to get started. In the short term, it is easiest to remove the dependence on the second config file by passing in the server alias as the command line parameter and having the parsing code figure out BMI addresses, storage spaces, log file pathname etc from the fs.conf file. > You'll also need a command-line argument to specify each server's > address. A good default _could_ be chosen by the primary server > by looking at the peer name given by BMI and seeing if the hostname > part matches something in the config file. Some setups, like for > failover or for machines serving through a non-primary interface, > will need to be able to specify their own server address on the > command-line; it can't be guessed from hostname. Right; except that since we do not wish to add that initial handshaking, we will probably just pass the server's alias as command line argument. Does that sound ok to you? > I'd rather see StorageSpace and LogFile as normal tags in the > Filesystem section, then add ... as a > way to override any particular setting, like in ssh_config, e.g. > Most central setups will use similar pathnames on each machine. > But this is only a slight preference. Okay. Makes sense. > > (And your "localhost1" etc. must be server aliases, not hostnames.) Right.. My bad. > > > NOTE: each server maintains a global in-memory session identifier that is > > propagated by the putconfig() from the root-server and this identifier > > is part of every client servreq structure. If it matches with the server > > id, the requests are processed, else the clients are forced to do a > > getconfig(). > > That's a nice addition. Can you use the hash value of the config > file itself? That's true. We could just pass that around instead of maintaining a separate session identifier. I will try to put together a patch for eliminating the second config file and support for SIGHUP handling. thanks for the comments! Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
[EMAIL PROTECTED] wrote on Tue, 12 Sep 2006 17:07 -0500: > Sam and I discussed the option of overhauling the configuration file and > server setup options today and here is a summary of what we had come up > with.. (Sam, please correct me if I missed out anything or misunderstood > something) > > Well-known problems with the current setup are > a) the need for 2 config files (a global one and a per-server config file) > and the implicit reliance on a savvy admin to keep these consistent and > synchronized. > b) Not having a clean way to be able to do a SIGHUP or equivalent to get > restart functionality on servers. These are great goals. I like your approach. > We were thinking of going about solving this in a 3 step process. > 1) Eliminate the need for a per-server config file and pass the > server host id as a command line parameter. Then there is exactly one process that every reads any configuration file from disk. It would likely be the one mentioned in fstabs. All the other servers are started by passing a BMI address to this primary server process on the command line. (This is maybe not what you had in mind.) You'll also need a command-line argument to specify each server's address. A good default _could_ be chosen by the primary server by looking at the peer name given by BMI and seeing if the hostname part matches something in the config file. Some setups, like for failover or for machines serving through a non-primary interface, will need to be able to specify their own server address on the command-line; it can't be guessed from hostname. > The fs.conf file will have a > new Defaults tag for the storage space in case it is the same on all > servers (which is usually the case) like the log file pathname. > Since the fs.conf aliases section maps aliases to host ids, servers can > use that to obtain the aliases and thence the metahandle, datahandle > ranges. > If each server has a different storage space path, we could have a new tag > > localhost1 /tmp/path1 > localhost2 /tmp/path2 > ... > > and so on.. I'd rather see StorageSpace and LogFile as normal tags in the Filesystem section, then add ... as a way to override any particular setting, like in ssh_config, e.g. Most central setups will use similar pathnames on each machine. But this is only a slight preference. (And your "localhost1" etc. must be server aliases, not hostnames.) > Question is, how does this server talk to the remaining servers to push > the config requests? We could just simply use tcp for this communication > and have the putconfig act like an implicit SIGHUP which will cause all > servers to restart and listen on the appropriate interfaces subsequent to > the putconfig. Please use the existing BMI infrastructure if possible. It would be preferable to allow any transport for these server-to-server operations, and not to require TCP specifically. > As regards the port numbers to do the putconfig, we could query the > portmapper or dedicate a range of port numbers on each host for running > the remaining servers initially. Not crazy about portmapper or listening on random ports. There are two cases here: 1. Initial server startup. Acts as a client in connecting to the root server to figure its config via "getconfig". Does not open any listening ports until this is resolved. 2. Config file modification. Here the root server acts as a client and connects to each server on its given address with a usual sendunexpected "putconfig" message. I don't see the need to have a temporary ephemeral listening port for each server. > NOTE: each server maintains a global in-memory session identifier that is > propagated by the putconfig() from the root-server and this identifier > is part of every client servreq structure. If it matches with the server > id, the requests are processed, else the clients are forced to do a > getconfig(). That's a nice addition. Can you use the hash value of the config file itself? Sam said: > To continue the discussion here that I had offline with Murali, what > about having servers send an initial hello message to the master > server, so that they don't have to figure out what port to listen > on? The master can send that server's HostID, wait for a ready > response, and then push the config file to that endpoint. Totally. Julian's ideas about zeroconf are a natural extension of this centralized config management, but I think they are a little out of scope for most current PVFS usage models (centralized well-known setup). Maybe good for the future though. -- Pete ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] RFC: Config file overhaul
Hi, > We were thinking of going about solving this in a 3 step process. This is really cool ;) I think it might be a good idea to thing about the zero conf issue and about extending the server farm during this process. This actually might not be much more efford to think about it and make plans for it so that you can later add another step to get to a zeroconf (extending) environment... Concearning zeroconf a "" tag in the normal fs.conf I think is no good idea... The information of handle ranges and available servers should be maintained separately from the input of the sysadmin. For simplicity it could be generated on the fly by the rootserver, though... I mean so you could do something like giving each new server 1 million handles the first time it is added and later give another amount once the million is used up... (Once it is assigned it has to be stored of course...) >To continue the discussion here that I had offline with Murali, what >about having servers send an initial hello message to the master >server, so that they don't have to figure out what port to listen >on? The master can send that server's HostID, wait for a ready >response, and then push the config file to that endpoint. That sounds good. You still have to give a address of the rootserver to the non-rootservers, right ? Then it figures out which network it has to use for the first-contact :) >>For instance server >>(non root) restarts should cause putconfig's implicitly or getconfig's? >To continue the discussion here that I had offline with Murali, what >about having servers send an initial hello message to the master >server, so that they don't have to figure out what port to listen >on? The master can send that server's HostID, wait for a ready >response, and then push the config file to that endpoint Cool, this seems to be a good idea. The other servers once they come up contact the rootserver for a new config the server implicitly gets their current used network address, the rootserver then could generate a new entry with that address... Once the admin changes global parameters a putconfig to all servers is necessary, of course. Also, if a new metaserver is added the client config is invalid, but not necessarily if a new dataserver is added... I really like all the stuff ;) Have a nice day, Julian ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
