Re: Implementing a Queue in a process
On 22 November 2011 07:16, Shantanu Bhadoria shant...@cpan.org wrote: Hey people, Here is a interesting problem that I am facing right now. I need to implement a process(in perl) that would sit and wait for someone to push tasks into its queue, each task consists of a set of information in its data structure w.r.t the task details. Search back in the list archives. We discussed this at length in the past year. http://cpanratings.perl.org/dist/AnyMQ Good generic message queue interface, extensible by traits beyond the default in-memory only model. Unfortunately, despite the numerous message-queue modules and bindings (for beanstalk, stomp, rabbitmq, zeromq, amqp, various hand-rolled modules) on CPAN, there are no interfaces between them and AnyMQ on the CPAN. bindings for message queue products https://metacpan.org/module/Net::Stomp https://metacpan.org/module/Net::RabbitMQ https://metacpan.org/module/ZeroMQ https://metacpan.org/release/Beanstalk-Client queue backed by database https://metacpan.org/module/TheSchwartz multi-worker batch processing https://metacpan.org/module/MooseX::Workers https://metacpan.org/module/Schedule::Pluggable Regards, Peter http://perl.dragonstaff.co.uk
Re: Implementing a Queue in a process
What about: search.cpan.org/perldoc?AnyEvent::RabbitMQ -- Ciao Richard Foley http://www.rfi.net/books.html On Tue, Nov 22, 2011 at 07:48:22AM +, Peter Edwards wrote: On 22 November 2011 07:16, Shantanu Bhadoria shant...@cpan.org wrote: Hey people, Here is a interesting problem that I am facing right now. I need to implement a process(in perl) that would sit and wait for someone to push tasks into its queue, each task consists of a set of information in its data structure w.r.t the task details. Search back in the list archives. We discussed this at length in the past year. http://cpanratings.perl.org/dist/AnyMQ Good generic message queue interface, extensible by traits beyond the default in-memory only model. Unfortunately, despite the numerous message-queue modules and bindings (for beanstalk, stomp, rabbitmq, zeromq, amqp, various hand-rolled modules) on CPAN, there are no interfaces between them and AnyMQ on the CPAN. bindings for message queue products https://metacpan.org/module/Net::Stomp https://metacpan.org/module/Net::RabbitMQ https://metacpan.org/module/ZeroMQ https://metacpan.org/release/Beanstalk-Client queue backed by database https://metacpan.org/module/TheSchwartz multi-worker batch processing https://metacpan.org/module/MooseX::Workers https://metacpan.org/module/Schedule::Pluggable Regards, Peter http://perl.dragonstaff.co.uk
Re: Implementing a Queue in a process
On Tue, 2011-11-22 at 08:23 +0100, Abigail wrote: My first idea is named piped, but the devil is in the details. A named pipe is a good solution. I've been using one for a specific task for almost 11 years without any particular problems. There is a serious drawback to this approach however - once you take something off the pipe it is gone forever. If the process reading from it fails for whatever reason you cannot replay the entries unless you have them logged elsewhere. It's never been a problem for me as my purpose for using this isn't too critical but for something important you either need a queuing method that logs or another solution. You don't even write which best you are looking for. Presumably the universal best - one in which someone else provides the solution and it's already implemented in an easy to use manner at little or no cost.
Re: Implementing a Queue in a process
On 22/11/11 07:16, Shantanu Bhadoria wrote: - How to implement this queue in a perl process. (remote shared Database:mysql?, internal data structure, local SQLite?) - How do I push new tasks into this queue ( i.e. interprocess communication, Simpler(uglier) for a shared database, but I would prefer a process like that to sit in isolation and accept requests and manage them on its own so whats the ideal inter-process communication format for it?named pipes?). I would really love to know your opinions on the best way to do it. The straightforward but ugly way to do this would probably be putting tasks into a database and have the other process ping the database for new tasks in queue every second. But I believe there has gotta be a better way to do it and what better place to figure it out then ask you folks. :) So what do you guys think is the best implementation for such a requirement? Now is your chance to use AnyEvent and Json. That is what I do these days. I used to use my own message handling stuff, but these are better and much faster. Dirk
Re: london.pm Digest, Vol 73, Issue 15
Presumably the universal best - one in which someone else provides the solution and it's already implemented in an easy to use manner at little or no cost. Yep thats the one I am looking for but I should have specified that. so here is the details on what I am trying to do. I am controlling a couple of devices via a process that runs continuously and wait for tasks to be performed on those devices, there is a UI that runs on another system that is supposed to push tasks to this process(interface could be restful or something else, but we are going with the idea that the process chould be running on a remote machine away from the system(s) that tell it to do things. Now when this process gets a message(task) it needs to put the task in some sort of a internal queue and run it based on priority(I am thinking multiple queues, one for each priority) so my process would then take this queue and process it. second part is the tricky part. The process need to have some way of telling a querying system if it has worked on a task in the queue or not. or if it working on it at the moment. One other trouble spot is if a lower priority task is going on it can be over ridden if a higher priority task is pushed in the high priority queue. I know there is no system to do all of this but I think a good MQ solution would form part of my final solution regardless. so Jason nailed it down closest on that I am looking for something(s) that is/are prebuilt and I can plug into my solution to make it work. thanks, -Shantanu
Re: london.pm Digest, Vol 73, Issue 15
On Tue, 2011-11-22 at 21:15 +0800, Shantanu Bhadoria wrote: I am controlling a couple of devices via a process that runs continuously and wait for tasks to be performed on those devices, Is that device able to multitask? Could you run multiple processes on that device to run the jobs as and when they arrive?
Re: Implementing a Queue in a process
You might give TheSchwartz a try: https://metacpan.org/module/TheSchwartz
