Thanks, I think that Redis would work but the system currently uses Postgresql, I'm looking for something simpler than having to maintain another service. (We're actively looking at NoSQL solutions but that would be a huge rearchitecting of the system.)
Is there a clean way to do this with pipes or tmp files or something similar? -- David On Wed, Apr 3, 2013 at 10:55 AM, Anthony Caravello <t...@caravello.us>wrote: > I've generally had luck with threads in Perl 5.10 and beyond, though if > you're sharing variables containing large amounts of data they can be > inefficient. Storing your data in Redis is often handy and fast, and > Redis::List works nicely as a queuing system, even if you use threads. > > Hope that helps, > Tony Caravello > > > On Wed, Apr 3, 2013 at 10:47 AM, Ricker, William > <william.ric...@fmr.com>wrote: > >> This would be a great topic for a meeting, either a report later on what >> you found and how you used it, or as a workshop to evaluate your options. >> >> [FYI, Schedule for next Tuesday is Federico will update us on his >> embedded Perl hardware hacking project.] >> >> bill@$dayjob >> #include <not_speaking_for_the_firm> >> >> -----Original Message----- >> From: Boston-pm [mailto:boston-pm-bounces+william.ricker= >> fmr....@mail.pm.org] On Behalf Of David Larochelle >> Sent: Wednesday, April 03, 2013 10:34 AM >> To: Boston Perl Mongers >> Subject: [Boston.pm] Passing large complex data structures between process >> >> I'm trying to optimize a database driven web crawler and I was wondering >> if >> anyone could offer any recommendations for interprocess communications. >> >> Currently, the driver process periodically queries a database to get a >> list of URLs to crawler. It then stores these url's to be downloaded in a >> complex in memory and pipes them to separate processes that do the actual >> downloading. The problem is that the database queries are slow and block >> the driver process. >> >> I'd like to rearchitect the system so that the database queries occur in >> the background. However, I'm unsure what mechanism to use. In theory, >> threads would be ideally suited for this case but I'm not sure how stable >> they are (I'm running Perl 5.14.1 with 200+ cpan modules). >> >> Does anyone have any recommendations? >> >> Thanks, >> >> David >> >> _______________________________________________ >> Boston-pm mailing list >> Boston-pm@mail.pm.org >> http://mail.pm.org/mailman/listinfo/boston-pm >> >> _______________________________________________ >> Boston-pm mailing list >> Boston-pm@mail.pm.org >> http://mail.pm.org/mailman/listinfo/boston-pm >> > > _______________________________________________ Boston-pm mailing list Boston-pm@mail.pm.org http://mail.pm.org/mailman/listinfo/boston-pm
