FTP has been called the misbehaving child of networking, and I'm being kind. While other protocols play nicely on a network, not grabbing all the bandwidth they can and refusing to throttle down when needed, FTP generally does the opposite. FTP will try to commandeer all the bandwidth your infrastructure allows, and won't let go once it has it.
I'm sure modern FTP servers are better behaved than their cave-man-days predecessors, but the protocol itself is still what it is. If you have a router or switch with built in QOS, you may be able to do it there. Barring that you may want to use HTTP file transfers instead. Bob S On Sep 21, 2015, at 14:33 , Gregory Lypny <gregory.ly...@videotron.ca<mailto:gregory.ly...@videotron.ca>> wrote: Hello everyone, I posted about this a while back but am still having trouble. I need to download thousands of files from the Security and Exchange Commission's website. Access is through anonymous FTP with "anonymous" as the username and my email address as the password. I've been using Put in a Repeat With loop as repeat with each line remoteFilePath in listOfFilePaths — set new localFileName is set before the download request is made put url ("ftp://anonymous:myemailaddr...@ftp.sec.gov/"; & remoteFilePath) into url ("file:/" & exportFolderPath & "/" & localFileName ) end repeat but my script dies (the stack is lifeless and unresponsive) after a few dozen, and sometimes a few hundred downloads. I used similar scripts in Mathematica and confirmed that the problem is session-timed-out and cannot-connect-to-server types of errors. The SEC's webmaster tells me, "There is no load/rate limiting on FTP, but if you are running a fast process, it is possible you are temporarily overwhelming the server." So, I'm thinking that I need to throttle my requests, and maybe should be using libURLDownloadToFile to check the status of the current file being downloaded and not request another file until the current download is complete. I also wonder whether I should be connecting to the FTP site only once with the username and password, loop my requests, and then close the connection. Not sure how to do either of these and would greatly appreciate any suggestions or tips. Gregory _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
