-----Original Message-----
From: PLUG <plug-boun...@lists.pdxlinux.org> On Behalf Of Keith Lofstrom
Sent: Friday, November 17, 2023 7:20 PM
>There may be ways to rate-limit your bulk data request, so it doesn't
trigger their rate-limits, and looks more like an obsessed human user. >I
hypothesize; there are web provider process management experts reading this,
who know how incoming 15 GB requests are handled, >throttled, or thriftily
ignored. Please educate us!
Yes. You simply do your data-slurping late at night. Top bandwidth usage
on the Internet is between 10am-4pm in any given time zone, so if you are
hitting sites in the USA then starting at 6am PST it gets busy (east coast
is why) and by noon the North American network is extremely busy.
If you are a doofus who decides to do your slurping at lunchtime then maybe
you can understand when the various ISPs take a dim view of your idiocy and
shut you down.
But after 11PM it's quiet. From 3AM PST to 6AM PST you have a window where
activity is very low and you will only be competing with the likes of The
Internet Archive and it's Wayback machine, and the various corporate
"buffered cloud backup" schemes.
Of course, if you are hitting foreign sites then their networks will not
appreciate you but since their networks aren't owned by your ISP your ISP
doesn't give a tinker's damn.
Ted
