Hello everyone,
I posted about this a while back but am still having trouble.
I need to download thousands of files from the Security and Exchange
Commission's website. Access is through anonymous FTP with "anonymous" as the
username and my email address as the password. I've been using Put in a Repeat
With loop as
repeat with each line remoteFilePath in listOfFilePaths
— set new localFileName is set before the download request is made
put url ("ftp://anonymous:myemailaddr...@ftp.sec.gov/"; & remoteFilePath)
into url ("file:/" & exportFolderPath & "/" & localFileName )
end repeat
but my script dies (the stack is lifeless and unresponsive) after a few dozen,
and sometimes a few hundred downloads. I used similar scripts in Mathematica
and confirmed that the problem is session-timed-out and
cannot-connect-to-server types of errors. The SEC's webmaster tells me, "There
is no load/rate limiting on FTP, but if you are running a fast process, it is
possible you are temporarily overwhelming the server." So, I'm thinking that I
need to throttle my requests, and maybe should be using libURLDownloadToFile to
check the status of the current file being downloaded and not request another
file until the current download is complete. I also wonder whether I should be
connecting to the FTP site only once with the username and password, loop my
requests, and then close the connection. Not sure how to do either of these and
would greatly appreciate any suggestions or tips.
Gregory
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
