Re: Implementing a Queue in a process
On 22 November 2011 07:16, Shantanu Bhadoria shant...@cpan.org wrote: Hey people, Here is a interesting problem that I am facing right now. I need to implement a process(in perl) that would sit and wait for someone to push tasks into its queue, each task consists of a set of information in its data structure w.r.t the task details. Search back in the list archives. We discussed this at length in the past year. http://cpanratings.perl.org/dist/AnyMQ Good generic message queue interface, extensible by traits beyond the default in-memory only model. Unfortunately, despite the numerous message-queue modules and bindings (for beanstalk, stomp, rabbitmq, zeromq, amqp, various hand-rolled modules) on CPAN, there are no interfaces between them and AnyMQ on the CPAN. bindings for message queue products https://metacpan.org/module/Net::Stomp https://metacpan.org/module/Net::RabbitMQ https://metacpan.org/module/ZeroMQ https://metacpan.org/release/Beanstalk-Client queue backed by database https://metacpan.org/module/TheSchwartz multi-worker batch processing https://metacpan.org/module/MooseX::Workers https://metacpan.org/module/Schedule::Pluggable Regards, Peter http://perl.dragonstaff.co.uk
Re: Implementing a Queue in a process
What about: search.cpan.org/perldoc?AnyEvent::RabbitMQ -- Ciao Richard Foley http://www.rfi.net/books.html On Tue, Nov 22, 2011 at 07:48:22AM +, Peter Edwards wrote: On 22 November 2011 07:16, Shantanu Bhadoria shant...@cpan.org wrote: Hey people, Here is a interesting problem that I am facing right now. I need to implement a process(in perl) that would sit and wait for someone to push tasks into its queue, each task consists of a set of information in its data structure w.r.t the task details. Search back in the list archives. We discussed this at length in the past year. http://cpanratings.perl.org/dist/AnyMQ Good generic message queue interface, extensible by traits beyond the default in-memory only model. Unfortunately, despite the numerous message-queue modules and bindings (for beanstalk, stomp, rabbitmq, zeromq, amqp, various hand-rolled modules) on CPAN, there are no interfaces between them and AnyMQ on the CPAN. bindings for message queue products https://metacpan.org/module/Net::Stomp https://metacpan.org/module/Net::RabbitMQ https://metacpan.org/module/ZeroMQ https://metacpan.org/release/Beanstalk-Client queue backed by database https://metacpan.org/module/TheSchwartz multi-worker batch processing https://metacpan.org/module/MooseX::Workers https://metacpan.org/module/Schedule::Pluggable Regards, Peter http://perl.dragonstaff.co.uk
Re: Implementing a Queue in a process
On Tue, 2011-11-22 at 08:23 +0100, Abigail wrote: My first idea is named piped, but the devil is in the details. A named pipe is a good solution. I've been using one for a specific task for almost 11 years without any particular problems. There is a serious drawback to this approach however - once you take something off the pipe it is gone forever. If the process reading from it fails for whatever reason you cannot replay the entries unless you have them logged elsewhere. It's never been a problem for me as my purpose for using this isn't too critical but for something important you either need a queuing method that logs or another solution. You don't even write which best you are looking for. Presumably the universal best - one in which someone else provides the solution and it's already implemented in an easy to use manner at little or no cost.
Re: Implementing a Queue in a process
On 22/11/11 07:16, Shantanu Bhadoria wrote: - How to implement this queue in a perl process. (remote shared Database:mysql?, internal data structure, local SQLite?) - How do I push new tasks into this queue ( i.e. interprocess communication, Simpler(uglier) for a shared database, but I would prefer a process like that to sit in isolation and accept requests and manage them on its own so whats the ideal inter-process communication format for it?named pipes?). I would really love to know your opinions on the best way to do it. The straightforward but ugly way to do this would probably be putting tasks into a database and have the other process ping the database for new tasks in queue every second. But I believe there has gotta be a better way to do it and what better place to figure it out then ask you folks. :) So what do you guys think is the best implementation for such a requirement? Now is your chance to use AnyEvent and Json. That is what I do these days. I used to use my own message handling stuff, but these are better and much faster. Dirk
Re: Implementing a Queue in a process
You might give TheSchwartz a try: https://metacpan.org/module/TheSchwartz
Implementing a Queue in a process
Hey people, Here is a interesting problem that I am facing right now. I need to implement a process(in perl) that would sit and wait for someone to push tasks into its queue, each task consists of a set of information in its data structure w.r.t the task details. Now I need to figure out two things: - How to implement this queue in a perl process. (remote shared Database:mysql?, internal data structure, local SQLite?) - How do I push new tasks into this queue ( i.e. interprocess communication, Simpler(uglier) for a shared database, but I would prefer a process like that to sit in isolation and accept requests and manage them on its own so whats the ideal inter-process communication format for it?named pipes?). I would really love to know your opinions on the best way to do it. The straightforward but ugly way to do this would probably be putting tasks into a database and have the other process ping the database for new tasks in queue every second. But I believe there has gotta be a better way to do it and what better place to figure it out then ask you folks. :) So what do you guys think is the best implementation for such a requirement? cheers, -Shantanu
Re: Implementing a Queue in a process
On Tue, Nov 22, 2011 at 03:16:13PM +0800, Shantanu Bhadoria wrote: Hey people, Here is a interesting problem that I am facing right now. I need to implement a process(in perl) that would sit and wait for someone to push tasks into its queue, each task consists of a set of information in its data structure w.r.t the task details. Now I need to figure out two things: - How to implement this queue in a perl process. (remote shared Database:mysql?, internal data structure, local SQLite?) - How do I push new tasks into this queue ( i.e. interprocess communication, Simpler(uglier) for a shared database, but I would prefer a process like that to sit in isolation and accept requests and manage them on its own so whats the ideal inter-process communication format for it?named pipes?). I would really love to know your opinions on the best way to do it. The straightforward but ugly way to do this would probably be putting tasks into a database and have the other process ping the database for new tasks in queue every second. But I believe there has gotta be a better way to do it and what better place to figure it out then ask you folks. :) So what do you guys think is the best implementation for such a requirement? My first idea is named piped, but the devil is in the details. There's way too little information in this post to even have an inkling of a suggestion for the best implementation. You don't even write which best you are looking for. Abigail
