On Fri, Nov 17, 2023 at 08:26:21AM -0800, Rich Shepard wrote: > I need to download ~15G of data from a web site. Using a PLUG mail list > thread from 2008 I tried this syntax: > wget -r --accept *.* http://ph-public-data.com/
A related question is "how much will the Portland Harbor Superfund Site need to pay to upload 15 GB to you? How much upload bandwidth do they have? I've been in this situation before. I "wget-ed" a website, and was soon contacted by a panicked/angry sysadmin watching their website brought to a crawl because their 5 mbps upload bandwidth was clobbered for hours by my scrape of their site. My bad. Perhaps the Right Thing To Do is to contact the project manager (see site's contact page) and ask them how you can help them with their important mission. I can imagine showing up in person with a flash drive, copying the 15 GB to that drive on site, then making a few dozen copies of that flash drive to give to them, which they can mail to other researchers like you. Bonus points if you can figure out some way for them to design access for "supplemental uploads" to go with those flash drives, so that other researchers only need to upload recent data to add to older data on the drives. I can also imagine this process leading to new research and consulting contracts for the individual researcher who provides the "free" flash drives. Paraphrasing Dwight Eisenhower, "If a problem can't be solved, enlarge it." Keith L. P.S. to other pluggers reading this - what is the best source of reliable 32GB flash drives, quantity 50, with printed/embossed logos? -- Keith Lofstrom [email protected]
