Hi Thomas, as you say, wget stores the links to traverse in memory. Also the links already traversed. That sums up after a while... but what are talking about here ? One million links with ~100 bytes sum to ~100MB. That's not very much for a modern machine.
To get a better idea: - what version of wget do you use (wget --version) - how long is wget running resp. how many pages are referred (rough estimate) ? - did you run a top (with pressing 'm' to sort for memory usage) ? if not, please do and report memory usage and speed of eating up memory. You could send me the real URL you are using via PM so I can try to reproduce the behavior here. Tim On Monday 14 March 2016 10:59:25 thomas hood wrote: > Hey, > > Using wget recently to handle large amounts of data. Finding it frequently > eats into memory quite a bit.Mar 12 06:32:08 ubuntu kernel: > [1439386.493486] Out of memory: Kill process 63125 (wget) score 661 or > sacrifice childMar 12 06:32:08 ubuntu kernel: [1439386.494595] Killed > process 63125 (wget) total-vm:4206452kB, anon-rss:3161380kB, > file-rss:0kBThese are the parameters I'm using; wget -c -nd -b -r -e > robots=off --header="Accept: text/html" --user-agent="Mozilla/5.0 (Windows > NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" http://$URL/$3 2>&1 | > grep '^--' 2>&1 | awk '{ print $3 }' | grep -v > '\.\(css\|js\|png\|gif\|jpg\|JPG\)$' &Doing a bit of research on Google > I've found other people have experienced this in the past one or two years. > Can anyone shed light on the cause? Is it just wget storing the relative > links it has to traverse in main memory? Obviously there's ways to manage > that but I'm unsure if it's an issue of memory management. > > Regards,Thomas Hood
signature.asc
Description: This is a digitally signed message part.
