Hello, On Sun, May 3, 2009 at 1:27 AM, Rocco Caputo <rcap...@pobox.com> wrote: > On May 2, 2009, at 05:37, Roy M. wrote: > Time constraint = 1 hour. > 50 servers * 300 files = 15,000 files. > 15,000 files * 5 minutes = 75,000 minutes. >
sorry for the misleading: I mean to download 300 files from a single ftp server is less than a few minutes, e.g. 5 minutes for safe. (files are just very small, most are a few KB, e.g. <10KB) so currently I already have a sequential Net::FTP perl script which can handle the job for ONE ftp server every hours very nicely. The problem arise when I add more ftp servers to handle by the script, e.g. 50 ftp servers x 5 minutes = 250 mins = 4 hours (worst case) total bandwidth needed for all ftp sessions = 10KB x 300 files x 50 = 146MB as you can see, bandwidth (giga), disk, cpu should not be a problem to handle 146MB of data in 1 hour of time. so I am thinking 2 solutions: 1. Use thread (e.g. 8 concurrent process as my CPU is quad core SMP) to parallel the worker to ftp servers. 2. Investigate other solutions such as POE and that's why I am asking here: So far the POE::FTP components only allow me to connect to one ftp server at a time, how can I get the same/similar speed as 8 concurrent process?). In fact, this is a quite common multiple-producer and multiple-consumer pattern in muilt-threading theory. But so far the POE example only allow me to do thing in a event-driven way, but not a way to handle this pattern, or something I have missed? Thanks.
