Memory Problem
Hi all, What has taken up all my swap space. Is it wget ? I am mirroring several websites on Sunday evening. If this is normal, then I have to put some more RAM in the Machine. Thank you for your answers. Mettavihari CPU states: 0.0% user, 3.1% system, 0.0% nice, 96.8% idle Mem:62872K av, 61260K used,1612K free,8140K shrd, 1308K buff Swap: 104380K av, 104380K used, 0K free 9136K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 2268 root 11 0 123M 43M 6968 D 1.1 70.7 27:37 wget 14581 admin 11 0 568 568 336 R 1.1 0.9 0:01 top 5 root 1 0 00 0 SW0.3 0.0 3:36 kswapd 3 root 1 0 00 0 SW0.1 0.0 6:16 kupdate 14589 qmaild 5 0 328 304 256 S 0.1 0.4 0:00 tcpserver 1 root 0 0 104 6044 S 0.0 0.0 0:06 init 2 root 0 0 00 0 SW0.0 0.0 2:54 kflushd 4 root 0 0 00 0 SW0.0 0.0 0:00 kpiod 6 root -20 -20 00 0 SW< 0.0 0.0 0:00 mdrecoveryd 61 root 0 0 00 0 SW0.0 0.0 0:00 khubd 340 root 0 0 132 6844 S 0.0 0.1 1:09 syslogd Sort by CPU seize - Mem: 62872K av, 61868K used, 1004K free, 7212K shrd, 1308K buff Swap: 104380K av, 104380K used, 0K free 9772K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 2268 root 12 0 123M 42M 5576 R 0.9 68.5 27:39 wget 5966 apache 1 0 2984 820 712 D 0.3 1.3 0:02 httpd 14581 admin 15 0 584 584 344 R 1.7 0.9 0:02 top 14556 admin 0 0 344 344 0 S 0.0 0.5 0:00 bash 14542 root 0 0 232 232 0 S 0.0 0.3 0:00 login 12520 apache 0 0 2524 200 0 S 0.0 0.3 0:00 httpd 5967 apache 0 0 2524 196 0 S 0.0 0.3 0:03 httpd 5970 apache 0 0 2532 168 0 S 0.0 0.2 0:02 httpd 6206 apache 0 0 2524 168 0 S 0.0 0.2 0:02 httpd 14532 root 2 0 168 16828 S 0.5 0.2 0:00 in.telnetd 6208 apache 0 0 2532 156 0 S 0.0 0.2 0:01 httpd 5969 apache 0 0 2556 14032 S 0.0 0.2 0:02 httpd 5968 apache 0 0 2524 120 0 S 0.0 0.1 0:04 httpd 765 root 0 0 176 11680 S 0.0 0.1 0:04 crond 5972 apache 0 0 2524 104 0 S 0.0 0.1 0:01 httpd This is what it looks like after a reboot. -- 7:43am up 34 min, 1 user, load average: 0.00, 0.00, 0.00 61 processes: 60 sleeping, 1 running, 0 zombie, 0 stopped CPU states: 0.0% user, 0.5% system, 0.0% nice, 99.4% idle Mem:62872K av, 45164K used, 17708K free, 61524K shrd,4664K buff Swap: 104380K av, 0K used, 104380K free 20224K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 1025 admin 14 0 1036 1036 816 R 0.5 1.6 0:00 top 1 root 0 0 532 532 468 S 0.0 0.8 0:05 init 2 root 0 0 00 0 SW0.0 0.0 0:00 kflushd 3 root 2 0 00 0 SW0.0 0.0 0:00 kupdate 4 root 0 0 00 0 SW0.0 0.0 0:00 kpiod 5 root 0 0 00 0 SW0.0 0.0 0:00 kswapd 6 root -20 -20 00 0 SW< 0.0 0.0 0:00 mdrecoveryd 61 root 0 0 00 0 SW0.0 0.0 0:00 khubd Sort by Memory - 7:43am up 35 min, 1 user, load average: 0.00, 0.00, 0.00 61 processes: 60 sleeping, 1 running, 0 zombie, 0 stopped CPU states: 0.1% user, 0.1% system, 0.0% nice, 99.6% idle Mem:62872K av, 45300K used, 17572K free, 61552K shrd,4672K buff Swap: 104380K av, 0K used, 104380K free 20244K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 585 root 0 0 4936 4936 668 S 0.0 7.8 0:01 ravmd 766 apache 0 0 4600 4600 4392 S 0.0 7.3 0:00 httpd 769 apache 0 0 4596 4596 4364 S 0.0 7.3 0:00 httpd 767 apache 0 0 4584 4584 4392 S 0.0 7.2 0:00 httpd 768 apache 0 0 4584 4584 4392 S 0.0 7.2 0:00 httpd 773 apache 0 0 4584 4584 4392 S 0.0 7.2 0:00 httpd 975 apache 0 0 4488 4488 4360 S 0.0 7.1 0:00 httpd 770 apache 0 0 4464 4464 4336 S 0.0 7.1 0:00 httpd 771 apache 0 0 4464 4464 4336 S 0.0 7.1 0:00 httpd 772 apache 0 0 4464 4464 4336 S 0.0 7.1 0:00 httpd A saying of the Buddha from http://metta.lk/ >From lust arises grief, from lust arises fear; for him who is free from lust there is >no grief, whence fear? Random Dhammapada Verse 215
Re: robots.txt
On Sun, Jun 09, 2002 at 09:10:48PM +0200, Jens Rösner wrote: Hi all of you. Thank you for the lively discussion. I learned a lot from this tread. robots=offdownloaded the site and I can now coolly have a look at the text without having to be on line. Once again thanks Mettavihari > Hi! > > > >>> Why not just put "robots=off" in your .wgetrc? > > hey hey > > the "robots.txt" didn't just appear in the website; someone's > > put it there and thought about it. what's in there has a good reason. > Wll, from my own experience, the #1 reason is that webmasters > do not want webgrabbers of any kind to download the site in order to > force > the visitor to interactively browse the site and thus click > advertisement banners. > > > The only reason is > > you might be indexing old, doubled or invalid data, > That is cute, someone who believes that all people in the > internet do what they do to make life easier for everyone. > If you said "one reason is" or even "one reason might be", > I would not be that cynical, sorry. > > > or your indexing mech might loop on it, or crash the server. who knows. > I have yet to find a site which forces wGet into a "loop" as you said. > Others on the list probably can estimate the theoretical likelyhood of > such events. > > > ask the webmaster or sysadmin before you 'hack' the site. > LOL! > hack! Please provide a serious definition of "to hack" that includes > "automatically downloading pages that could be downloaded with any > interactive web-browser" > If the robots.txt said that no user-agent may access the page, you would > be right. > But then: How would anyone know of the existence of this page then? > [rant] > Then again, maybe the page has a high percentage of cgi, JavaScript, > iFrames and thus only allows > IE 6.0.123b to access the site. Then wget could maybe slow down the > server, especially as it is > probably a w-ows box :> But I ask: Is this a bad thing? > Whuahaha! > [/rant] > > Ok, sorry vor my sarcasm, but I think you overestimate the benefits of > robots.txt for mankind. > > CU > Jens A saying of the Buddha from http://metta.lk/ Verily, misers go not to the celestial realms. Fools do not indeed praise liberality. The wise man rejoices in giving and thereby becomes happy thereafter. Random Dhammapada Verse 177
robots.txt
Hi All, I use wget often. On some sites I cannot download files. On one such site I found this file "robots.txt". Is this file the cause for wget not downloading the files. Any way I could circumvent this and download this website. best regards Mettavihari -- robots.txt -- User-agent: Googlebot Disallow: User-agent: Atomz Disallow: User-agent: WebCrawler Disallow: User-agent: Lycos Disallow: User-agent: AltaVista Disallow: User-agent: HotBot Disallow: User-agent: Infoseek Disallow: User-agent: Pathfinder Disallow: User-agent: Yahoo Disallow: User-agent: * Disallow: / A saying of the Buddha from http://metta.lk/ Delightful are the forests where worldlings delight not; the passionless will rejoice (therein), (for) they seek no sensual pleasures. Random Dhammapada Verse 99
memory problem
Hi all, I run wget 1.6 and RH 7.0 on a small machine. 64Mb Ram and 100 MB swap I find that wget is taking all my memory. Please let me know what is happening and if I have to configure wget to settle this in some way by telling wget "use Max mem xxMB" Thanks in advance for your advice. Mettavihari top before killing wget --- 10:01pm up 5 days, 4:41, 0 users, load average: 2.11, 1.37, 1.22 66 processes: 65 sleeping, 1 running, 0 zombie, 0 stopped CPU states: 1.1% user, 0.7% system, 0.0% nice, 2.3% idle Mem: 62816K av, 61800K used, 1016K free, 41016K shrd, 1516K buff Swap: 104380K av, 104312K used, 68K free 48760K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 4045 root 16 0 944 944 740 R 5.8 1.5 0:00 top 5 root 1 0 00 0 SW0.6 0.0 5:13 kswapd 768 qmailr15 0 120 8872 S 0.6 0.1 0:00 qmail-rspawn 27442 root 5 0 92204 34M 3 D 0.6 55.4 26:09 wget 4057 qmailr11 0 508 508 428 S 0.6 0.8 0:00 qmail-remote 1 root 0 0 104 6048 S 0.0 0.0 0:07 init After killing the processes and restarting It looks like the below. - 6503 root 16 0 1028 1028 816 R 0.9 1.6 0:00 top 6492 root 7 0 1116 1116 844 S 0.3 1.7 0:00 wget 6420 root 1 0 168 6028 S 0.1 0.0 0:00 in.telnetd 1 root 0 0 116 7660 S 0.0 0.1 0:08 init 2 root 0 0 00 0 SW0.0 0.0 3:57 kflushd A saying of the Buddha from http://metta.lk/ Though little he recites the Sacred Texts, but acts in accordance with the teaching, forsaking lust, hatred and ignorance, truly knowing, with mind well freed, clinging to naught here and hereafter, he shares the fruits of the Holy life. Random Dhammapada Verse 20
wget only draw 317 MB of the 600 MB
Hi, I am trying out wget 1.5.3 just for my testing to draw my website of 650 MB on 2 linux machines from machine 1 on the lan to machine 2. The command I use is: wget -r -b -l40 http://narada.col7.metta.lk/ I am drawing from the machine http://harsha.col7.metta.lk None of the machines are on the Net. It works fine and draws a lot of data, but finally I end up having only 317 MB on machine 2 I tried on another machine to draw from an NT server to a Linux machine and had a similar result. In that case I found wget had skipped almost all the Directories beginning with underline such as "_some-name" Help is much appriciated I thank you for the suggestion to compile the version 1.7.beta and shall try that out in a few days time. best regards Mettavihari (Jacob) Sri Lanka. A saying of the Buddha from http://metta.lk/ He who in every way knows the death and rebirth of beings, who is non-attached, well-gone, and enlightened, - him I call a Brahmana. Random Dhammapada Verse 419
