Re: Monitor a FTP site for arrival of new/updated files
On 25 Gen, 21:11, Steve Holden wrote: > pyt...@bdurham.com wrote: > > Any suggestions on a best practice way to monitor a remote FTP site for > > the arrival of new/updated files? I don't need specific code, just some > > coaching on technique based on your real-world experience including > > suggestions for a utility vs. code based solution. > > > My goal is to maintain a local collection of files synced with a remote > > FTP site and when I download a new/updated file locally, run a script to > > process it. The arrival and format of the files that I need to sync with > > are beyond my control (eliminating a rsync solution) ... all I have is a > > generic FTP connection to a specific FTP address. Note: The remote site > > I'm monitoring may have multiple uploads occuring at the same time. > > > My basic strategy is to poll the remote directory on a regular basis and > > compare the new directory listing to a previous snapshot of the > > directory listing. If a file timestamp or size has changed (or a new > > file has appeared), then track this file as a changed file. Once a file > > has been marked as changed, wait polling cycles for the file > > timestamp and size to remain stable, then download it, and trigger a > > local script to process the file. In addition to detecting new or > > changed files, I would compare remote directory listings to my local > > sync folder and delete local files that no longer exist on the remote site. > > > My concern about using a utility is the utility's ability to detect when > > a remote file has finished being updated. I don't want to download files > > that are still in the process of being updated - I only want to download > > new/updated files after they've been closed on the remote site. > > > Any ideas appreciated! > > Well, the ftpmirror will cope with most of what you want to do as it is, > but I am unsure how you can determine whether a file is in the process > of being written on the server. > > regards > Steve > -- > Steve Holden +1 571 484 6266 +1 800 494 3119 > Holden Web LLC http://www.holdenweb.com/- Nascondi testo citato > > - Mostra testo citato - If you're going to do that manually through a script, and the server supports it, use MLSD instead of LIST command for listing files and determine their last modification time. My 2 cents --- Giampaolo http://code.google.com/p/pyftpdlib -- http://mail.python.org/mailman/listinfo/python-list
Re: Monitor a FTP site for arrival of new/updated files
When the date was Sunday 25 January 2009, pyt...@bdurham.com wrote: > Any suggestions on a best practice way to monitor a remote FTP > site for the arrival of new/updated files? For a custom solution, you may want to take a look to pyinotify: http://pyinotify.sourceforge.net/ -- Michael Iatrou (idyz) -- http://mail.python.org/mailman/listinfo/python-list
Re: Monitor a FTP site for arrival of new/updated files
> Well, the ftpmirror will cope with most of what you want to do as it is, but > I am unsure how you can determine whether a file is in the process of being written on the server. Looks like that may be a fit. Thank you Steve! Malcolm -- http://mail.python.org/mailman/listinfo/python-list
Re: Monitor a FTP site for arrival of new/updated files
pyt...@bdurham.com wrote: > Any suggestions on a best practice way to monitor a remote FTP site for > the arrival of new/updated files? I don't need specific code, just some > coaching on technique based on your real-world experience including > suggestions for a utility vs. code based solution. > > My goal is to maintain a local collection of files synced with a remote > FTP site and when I download a new/updated file locally, run a script to > process it. The arrival and format of the files that I need to sync with > are beyond my control (eliminating a rsync solution) ... all I have is a > generic FTP connection to a specific FTP address. Note: The remote site > I'm monitoring may have multiple uploads occuring at the same time. > > My basic strategy is to poll the remote directory on a regular basis and > compare the new directory listing to a previous snapshot of the > directory listing. If a file timestamp or size has changed (or a new > file has appeared), then track this file as a changed file. Once a file > has been marked as changed, wait polling cycles for the file > timestamp and size to remain stable, then download it, and trigger a > local script to process the file. In addition to detecting new or > changed files, I would compare remote directory listings to my local > sync folder and delete local files that no longer exist on the remote site. > > My concern about using a utility is the utility's ability to detect when > a remote file has finished being updated. I don't want to download files > that are still in the process of being updated - I only want to download > new/updated files after they've been closed on the remote site. > > Any ideas appreciated! > Well, the ftpmirror will cope with most of what you want to do as it is, but I am unsure how you can determine whether a file is in the process of being written on the server. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Monitor a FTP site for arrival of new/updated files
Any suggestions on a best practice way to monitor a remote FTP site for the arrival of new/updated files? I don't need specific code, just some coaching on technique based on your real-world experience including suggestions for a utility vs. code based solution. My goal is to maintain a local collection of files synced with a remote FTP site and when I download a new/updated file locally, run a script to process it. The arrival and format of the files that I need to sync with are beyond my control (eliminating a rsync solution) ... all I have is a generic FTP connection to a specific FTP address. Note: The remote site I'm monitoring may have multiple uploads occuring at the same time. My basic strategy is to poll the remote directory on a regular basis and compare the new directory listing to a previous snapshot of the directory listing. If a file timestamp or size has changed (or a new file has appeared), then track this file as a changed file. Once a file has been marked as changed, wait polling cycles for the file timestamp and size to remain stable, then download it, and trigger a local script to process the file. In addition to detecting new or changed files, I would compare remote directory listings to my local sync folder and delete local files that no longer exist on the remote site. My concern about using a utility is the utility's ability to detect when a remote file has finished being updated. I don't want to download files that are still in the process of being updated - I only want to download new/updated files after they've been closed on the remote site. Any ideas appreciated! Thanks, Malcolm -- http://mail.python.org/mailman/listinfo/python-list
