Re: Need Help Throttling Downloads From an FTP Site

2015-09-23 Thread Gregory Lypny
Hi Bob Sneidar, Scott Rossi, Mike Bonner, and Jim Lambert, Thanks for your suggestions. I going to experiment with all of them and share my results with the list. Regards, Gregory ___ use-livecode mailing list use-livecode@lists.runrev.com Please vis

Re: Need Help Throttling Downloads From an FTP Site

2015-09-21 Thread Jim Lambert
Whoops! That should be: on getNextFile if lListOfFilePaths = empty then exit getNextFile put line 1 of lListOfFilePaths into remoteFilePath delete line 1 of lListOfFilePaths —SET THE LOCAL FILE’S NAME HOWEVER YOU NORMALLY WOULD put whatever into l

Re: Need Help Throttling Downloads From an FTP Site

2015-09-21 Thread Jim Lambert
Gregory, Try this (untested): local lListOfFilePaths on downloadAll put theListofFiles into lListOfFilePaths getnextFile end repeat on getNextFile if lListOfFilePaths = empty then exit getNextFile put line 1 of lListOfFilePaths into remoteFilePath

Re: Need Help Throttling Downloads From an FTP Site

2015-09-21 Thread Mike Bonner
The problem doesn't seem to be a local network issue. When I try to grab files from the sec site, too many connections too fast make it choke. (There end, not mine, most likely anti-bot code) As scott rossi said, using a delay should help. I've noticed, the magic number seems to be 5, so I used

Re: Need Help Throttling Downloads From an FTP Site

2015-09-21 Thread Scott Rossi
How large are the files you're retrieving? If the script below is your actual script, you might try allowing some execution time in the loop: repeat with each line remoteFilePath in listOfFilePaths -- set new localFileName is set before the download request is made put url ("ftp://anonymo

Re: Need Help Throttling Downloads From an FTP Site

2015-09-21 Thread Bob Sneidar
FTP has been called the misbehaving child of networking, and I'm being kind. While other protocols play nicely on a network, not grabbing all the bandwidth they can and refusing to throttle down when needed, FTP generally does the opposite. FTP will try to commandeer all the bandwidth your infra

Need Help Throttling Downloads From an FTP Site

2015-09-21 Thread Gregory Lypny
Hello everyone, I posted about this a while back but am still having trouble. I need to download thousands of files from the Security and Exchange Commission's website. Access is through anonymous FTP with "anonymous" as the username and my email address as the password. I've been using Put in