Hi, I have the following requirement, and I am seeking your input.
1. web-based users make requests for data which are put in a queue 2. these requests and their status need to be in a database so that users can watch the status of the queue and their requests in the queue 3. a set of servers process the requests for data from the queue and put the results in a results table so that users can view their data when their requests are done QUESTIONS: * What queue mechanism would you use, assuming all of the writers and readers are on the same system? (IPC::Msg? MsgQ?) * How about if the queue writers were distributed, but the queue readers were all on one machine? (RPC to insert into the above-mentioned local queues?) * How about if the queue writers and queue readers were all distributed around the network? (Spread::Queue::FIFO? Parallel::PVM? Parallel::MPI? MQSeries::Queue?) * What Perl server-building software on CPAN do you recommend, and why? (Net::Server? Net::Daemon? POE?) I started working with Net::Server, but it seems focused on being a network request multi-server, not a queue working multi-server. * Would you implement it as many peer-level servers waiting on a single queue? or a single parent server waiting on the queue, dispatching queued work units to waiting child servers? QUICK AND DIRTY SINGLE-SERVER SOLUTION I implemented a quick-and-dirty single-server solution, where I use a single server to process requests. I simply poll the request table in the database once a minute for new requests, and if they exist, I process them. Now I am looking to upgrade this for higher throughput (multiple parallel servers), lower background load (no polling during quiet periods), and lower latency (immediate response to queue insertion rather than waiting for the next poll interval). MY HUNCHES I think I'll use IPC::Msg as the queue because the queue readers will all be on one machine. I'll also have to implement a simple RPC server (using Net::Server) to perform remote insertions into the local queue. If this seems too rough, I'll probably install the Spread Toolkit and use Spread::Queue. I currently think I'll keep working with Net::Server to see if I can use it to process a queue rather than listen on a network port, but I'm not sure that this is the right use of the module. I may end up ditching this effort and just have a set of parallel servers all waiting on the queue. The queue mechanism itself will work out who gets to work on which request. Any input? Stephen