Refreshing stored data at administrator's signal
Greetings. I have an application that accesses some relatively static database tables to create drop-down lists. As an example, one of these tables is a list of common commercial aircraft. At the moment (and not in a production environment), every time the drop-down list is generated for a web page, the script queries the database to retrieve the entire list of aircraft. I would prefer to retrieve the list of aircraft when each Perl interpreter starts and then not retrieve it again until the administrator sends a signal. For this particular table, the signal would only occur when new aircraft hit the market, like the Boeing 787 will (hopefully) in December of this year. The most UNIX-ish way to do this, I guess, would be to send SIGHUP to each running perl process, causing it to reload its configuration, update its stored lists, and so forth. I'd rather do this in a more Perl-ish or Apache-ish way, though, and I'd also rather be specific about which list it should update. At the moment, there are about 10 such lists, and I can see that number growing to about 20 before the site goes live. At a guess, the lists average about 300 elements each (with the list of aircraft being one of the shorter and less-frequently-updated lists). My ideal solution would be to have an external application (the administrator app or whatever it ends up being) update some flag inside each mod_perl process every time they need to update a list, and then each mod_perl application would see the flag and perform the update. I could do this relatively easily with a combination of threads and file hooks or "update sockets" or something, but I don't plan on adding threads or sockets to the application, and I think adding that much complexity and overhead for this "simple" feature would be overkill. Any thoughts? Thanks. Colin
Re: Refreshing stored data at administrator's signal
Colin Wetherbee wrote: At the moment (and not in a production environment), every time the drop-down list is generated for a web page, the script queries the database to retrieve the entire list of aircraft. I would prefer to retrieve the list of aircraft when each Perl interpreter starts and then not retrieve it again until the administrator sends a signal. For this particular table, the signal would only occur when new aircraft hit the market, like the Boeing 787 will (hopefully) in December of this year. The most UNIX-ish way to do this, I guess, would be to send SIGHUP to each running perl process, causing it to reload its configuration, update its stored lists, and so forth. I'd rather do this in a more Perl-ish or Apache-ish way, though, and I'd also rather be specific about which list it should update. Wouldn't a simpler approach be to just restart Apache when you want to update the lists? You could even have the 'add to list' function send SIGUSR1 to the parent Apache, causing a graceful restart. Having said that, if running 20 DB queries returning a few hundred records is causing you a speed problem, are you sure the DB is running efficiently? Is this a very high traffic site? Is there a requirement for ultra-fast page generation? I've got pages that make dozens and dozens of DB queries returning hundreds of records and do lots of post-processing, and I can generate pages in under a second much of the time. cheers John
Re: Refreshing stored data at administrator's signal
John ORourke wrote: Colin Wetherbee wrote: At the moment (and not in a production environment), every time the drop-down list is generated for a web page, the script queries the database to retrieve the entire list of aircraft. I would prefer to retrieve the list of aircraft when each Perl interpreter starts and then not retrieve it again until the administrator sends a signal. For this particular table, the signal would only occur when new aircraft hit the market, like the Boeing 787 will (hopefully) in December of this year. The most UNIX-ish way to do this, I guess, would be to send SIGHUP to each running perl process, causing it to reload its configuration, update its stored lists, and so forth. I'd rather do this in a more Perl-ish or Apache-ish way, though, and I'd also rather be specific about which list it should update. Wouldn't a simpler approach be to just restart Apache when you want to update the lists? You could even have the 'add to list' function send SIGUSR1 to the parent Apache, causing a graceful restart. I'm trying to avoid restarting Apache altogether, although I admit it would be a pretty simple solution. Having said that, if running 20 DB queries returning a few hundred records is causing you a speed problem, are you sure the DB is running efficiently? Is this a very high traffic site? Is there a requirement for ultra-fast page generation? I've got pages that make dozens and dozens of DB queries returning hundreds of records and do lots of post-processing, and I can generate pages in under a second much of the time. The point is more like "well, this isn't really super-dynamic data, so running a query every time I need it seems like a waste of processor time and disk activity." It's not causing any slow-down right now, though when the site goes live, it certainly could. Colin
Re: Refreshing stored data at administrator's signal
Colin Wetherbee wrote: John ORourke wrote: Colin Wetherbee wrote: Wouldn't a simpler approach be to just restart Apache when you want to update the lists? You could even have the 'add to list' function send SIGUSR1 to the parent Apache, causing a graceful restart. I'm trying to avoid restarting Apache altogether, although I admit it would be a pretty simple solution. I'd seriously consider it - it's simple and clean and only takes a few seconds, and it happens every night when you rotate your logs anyway. If you really really don't want to restart Apache, you could get your 'add data' function to create a file called 'need_restart' somewhere on the disk, and after processing each request your mod_perl handler could check for the file and call $r->child_terminate if it finds it. You'd have to have some method of stopping it from constantly restarting could get complicated. The cynic in me suspects you'll spend too many hours on this not-really-a-problem, when there may be other parts of the system that would benefit from more attention! cheers John
Re: Refreshing stored data at administrator's signal
John ORourke wrote: Colin Wetherbee wrote: John ORourke wrote: Colin Wetherbee wrote: Wouldn't a simpler approach be to just restart Apache when you want to update the lists? You could even have the 'add to list' function send SIGUSR1 to the parent Apache, causing a graceful restart. I'm trying to avoid restarting Apache altogether, although I admit it would be a pretty simple solution. I'd seriously consider it - it's simple and clean and only takes a few seconds, and it happens every night when you rotate your logs anyway. If you really really don't want to restart Apache, you could get your 'add data' function to create a file called 'need_restart' somewhere on the disk, and after processing each request your mod_erl handler could check for the file and call $r->child_terminate if it finds it. You'd have to have some method of stopping it from constantly restarting could get complicated. I thought about the file thing... if the file exists, check its last modified timestamp; if that timestamp is greater than the stored timestamp, then update the data from the database. It seems like unnecessary disk access, though. Then again, this whole problem is riddled with unnecessary disk access. :) The cynic in me suspects you'll spend too many hours on this not-really-a-problem, when there may be other parts of the system that would benefit from more attention! Well, you're probably right about that. ;) Perhaps I'll set up a restart-based system and then worry about it later if it becomes an "actual" problem. Thanks for your input. Colin
Re: Refreshing stored data at administrator's signal
On Jan 13, 2008 4:19 PM, Colin Wetherbee <[EMAIL PROTECTED]> wrote: > I thought about the file thing... if the file exists, check its last > modified timestamp; if that timestamp is greater than the stored > timestamp, then update the data from the database. It seems like > unnecessary disk access, though. Then again, this whole problem is > riddled with unnecessary disk access. :) Using a "touch file" is the classic solution to this problem. You check the mod time on a file (it's okay for it always be there -- it's just the mod time we care about) and compare that to the last update time that you keep in a global. It's dirt simple, avoids messy problems with signals, and it should end up in your operating system's disk cache so it really won't do any physical disk reads. - Perrin
