Poe and FTP
Hi I have been looking at Poe for some time but now I believe I have an application for it. I would welcome some general feedback on my comments. Currently we have (a number of) FTP scripts. They each work by sending files sequentially to a remote FTP servers via one of a number of methods. Net::FTP Net::SFTP Net::SFTP::Foreign::Compat I am trying to rationalise these scripts since they are rather in a mess! Each script is called by a different process so in theory we can have multiple copies of the same script running at the same time sending to the same or different FTP servers. I can't work out if Poe would be a suitable alternative. On the one hand, sending files sequentially to the same FTP server would seem to be limited by the throughput of the pipe to the server. Running Poe with multiple processes all trying to use the same pipe should be no faster since we are presumably limited by the pipe bandwidth. On the other hand, if one of the files had a problem and timed out, then other files may not be delayed and could in theory still be transferred (depending upon the reason for the problem). I can see that if I was sending files to different FTP servers then Poe would be of use since then I would be sending files down different pipes, but this is no better that having several processes, one for each server. I have also looked at the FTP components. Poe::Component::Net::FTP I understand that if I were to use the 'put' command then it would block other Poe components until the file was sent. For this reason I would have to use the lower level write command which adds complexity. Has anyone implemented a none-blocking put for this module? Poe::Component::Client::FTP I think this has the same problem, there is no high level put command. I am not yet convinced that for my application Poe is a suitable advantage, but I am willing to be convinced :-) Regards Ian This email is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of RedBee Media Metadata. If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please notify the sender. Red Bee Media Metadata is a trading name of Broadcasting Dataservices Limited. Registered in England and Wales No.: 2554733. Registered Office: 201 Wood Lane, London W12 7TP, UK. Broadcasting Dataservices Limited is a wholly owned subsidiary of Red Bee Media Limited.
RE: Poe and FTP
Typical. Having posted a question, I then find similar points in the archive. Sorry. However, I think some of my points may still be relevant and are not covered in the thread 'How does POE speed up operation in reality?'. Chris Fedde. Mentioned the use of a single host process (presumably a daemon) that could fork to manage a number of transfer processes. I assume this would only be a benefit if it connected to a number of different hosts, rather than a single host so that transfers down one pipe would not be blocked by the bandwidth of the slowest pipe. No benefit that I can see if there are multiple files to just one FTP server. Rocco Caputo. Made the excellent point that Poe is appropriate where something would otherwise block other work being done. In my example, all I want to do is to send files to a single server via FTP. There is nothing else to do (other than perhaps a log file output) which could be blocked making Poe unnecessary. Regards Ian -Original Message- From: Ian Docherty [mailto:ian.doche...@bds.tv] Sent: 07 May 2009 08:05 To: poe@perl.org Subject: Poe and FTP Hi I have been looking at Poe for some time but now I believe I have an application for it. I would welcome some general feedback on my comments. Currently we have (a number of) FTP scripts. They each work by sending files sequentially to a remote FTP servers via one of a number of methods. Net::FTP Net::SFTP Net::SFTP::Foreign::Compat I am trying to rationalise these scripts since they are rather in a mess! Each script is called by a different process so in theory we can have multiple copies of the same script running at the same time sending to the same or different FTP servers. I can't work out if Poe would be a suitable alternative. On the one hand, sending files sequentially to the same FTP server would seem to be limited by the throughput of the pipe to the server. Running Poe with multiple processes all trying to use the same pipe should be no faster since we are presumably limited by the pipe bandwidth. On the other hand, if one of the files had a problem and timed out, then other files may not be delayed and could in theory still be transferred (depending upon the reason for the problem). I can see that if I was sending files to different FTP servers then Poe would be of use since then I would be sending files down different pipes, but this is no better that having several processes, one for each server. I have also looked at the FTP components. Poe::Component::Net::FTP I understand that if I were to use the 'put' command then it would block other Poe components until the file was sent. For this reason I would have to use the lower level write command which adds complexity. Has anyone implemented a none-blocking put for this module? Poe::Component::Client::FTP I think this has the same problem, there is no high level put command. I am not yet convinced that for my application Poe is a suitable advantage, but I am willing to be convinced :-) Regards Ian This email is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of RedBee Media Metadata. If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please notify the sender. Red Bee Media Metadata is a trading name of Broadcasting Dataservices Limited. Registered in England and Wales No.: 2554733. Registered Office: 201 Wood Lane, London W12 7TP, UK. Broadcasting Dataservices Limited is a wholly owned subsidiary of Red Bee Media Limited.
RE: using POE and ftp to build a win32 - *nix job system?
I don't know your reasons for FTP, but are you stuck on it? My preferred method for doing this has been to just mount a windows share via SMB/CIFS, and use it like just-another-unix-filesystem. IMHO There's no good reason to write networking code where file IO will suffice. Just another point of failure. You did mention that it's all inside the firewall, and your windows Server probably already has the directory in question shared. (C$ at the least!) All you need then is a mount statement, either manually in your script, or automounted in your /etc/fstab. A note though, If you use win 2k3 - you will need CIFS, not samba, to mount with proper permissions. This is easier on a 2.6 series kernel, as the 2.4 series, (on slackware at least,) has to be patched for cifs support. (Though it really is quite easy to do.) Regards, Gavin McDonald EVI Logistic Enterprises email: [EMAIL PROTECTED] phone: (604) 313-3845 -Original Message- From: Jeff McCarrell [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 22, 2006 4:01 PM To: poe@perl.org Subject: using POE and ftp to build a win32 - *nix job system? Hi folks. I need to create a system that - moves some files generated on a *nix system to a win32 system - does some processing on those files - moves the results back to the *nix system The win32 system is the server side of the connection, and it needs to be able to support multiple clients, be highly available, etc. All of the usual daemon reqs, except for security. This is running inside the firewall, so I don't have to worry about that class of problems. My thought is to use POE and some of the FTP modules to handle the file transfers for a couple of reasons: - I don't want to deal with \r\n - \n line ending conversions - the result files are not text files My first thought was that POE and, in particular, POE::Component::Client::FTP and POE::Component::Server::FTP, were perfect matches, but the comments at http://poe.perl.org/?POE_Components_List are not inspiring to say the least. I could fall back to using Net::FTP and Net::FTPServer, I think, but that would make my client and server pretty chunky, and not very cooperative-multitasking-friendly. I've been through most all of the POE tutorials and docs, and I have the control protocol that I want the client and server to talk worked out pretty well. But I'm unclear on how best to leverage the POE and FTP combination. So I seek your wisdom: if it were your decision, which would you choose? Thanks in advance... -- jeff
