To keep it simple, try threads and Thread::Queue module. That will keep it all 'in-house'.
On Wed, Apr 3, 2013 at 11:28 AM, David Larochelle <da...@larochelle.name>wrote: > Thanks, > > I think that Redis would work but the system currently uses > Postgresql, I'm looking for something simpler than having to maintain > another service. > (We're actively looking at NoSQL solutions but that would be a huge > rearchitecting of the system.) > > Is there a clean way to do this with pipes or tmp files or something > similar? > > -- > > David > > On Wed, Apr 3, 2013 at 10:55 AM, Anthony Caravello <t...@caravello.us>wrote: > >> I've generally had luck with threads in Perl 5.10 and beyond, though if >> you're sharing variables containing large amounts of data they can be >> inefficient. Storing your data in Redis is often handy and fast, and >> Redis::List works nicely as a queuing system, even if you use threads. >> >> Hope that helps, >> Tony Caravello >> >> >> On Wed, Apr 3, 2013 at 10:47 AM, Ricker, William >> <william.ric...@fmr.com>wrote: >> >>> This would be a great topic for a meeting, either a report later on what >>> you found and how you used it, or as a workshop to evaluate your options. >>> >>> [FYI, Schedule for next Tuesday is Federico will update us on his >>> embedded Perl hardware hacking project.] >>> >>> bill@$dayjob >>> #include <not_speaking_for_the_firm> >>> >>> -----Original Message----- >>> From: Boston-pm [mailto:boston-pm-bounces+william.ricker= >>> fmr....@mail.pm.org] On Behalf Of David Larochelle >>> Sent: Wednesday, April 03, 2013 10:34 AM >>> To: Boston Perl Mongers >>> Subject: [Boston.pm] Passing large complex data structures between >>> process >>> >>> I'm trying to optimize a database driven web crawler and I was wondering >>> if >>> anyone could offer any recommendations for interprocess communications. >>> >>> Currently, the driver process periodically queries a database to get a >>> list of URLs to crawler. It then stores these url's to be downloaded in a >>> complex in memory and pipes them to separate processes that do the actual >>> downloading. The problem is that the database queries are slow and block >>> the driver process. >>> >>> I'd like to rearchitect the system so that the database queries occur in >>> the background. However, I'm unsure what mechanism to use. In theory, >>> threads would be ideally suited for this case but I'm not sure how stable >>> they are (I'm running Perl 5.14.1 with 200+ cpan modules). >>> >>> Does anyone have any recommendations? >>> >>> Thanks, >>> >>> David >>> >>> _______________________________________________ >>> Boston-pm mailing list >>> Boston-pm@mail.pm.org >>> http://mail.pm.org/mailman/listinfo/boston-pm >>> >>> _______________________________________________ >>> Boston-pm mailing list >>> Boston-pm@mail.pm.org >>> http://mail.pm.org/mailman/listinfo/boston-pm >>> >> >> > _______________________________________________ Boston-pm mailing list Boston-pm@mail.pm.org http://mail.pm.org/mailman/listinfo/boston-pm
