Karen, thanks for the figures. I see that you have 2Gb of RAM, and
more than half of that amount (about 1.1Gb) is used by kernel's
disk cache. This is normal situation - linux always use free
RAM for cache.
You haven't shown MySQL RAM usage, but by looking at your my.cnf
I estimate it should be something near 300 Mb, which is OK.
By the way, to further speed up indexing and search, you can
increase the value of MySQL's key_buffer twice, to 512M.
The index process itself occupies 309M, which is OK too.
> By the end of the indexing process the system will be
> thrashing the disk cached RAM and will never be released.
Kernel releases memory used by disk cache only if some programs
ask for more memory, otherwise it is still occupied by disk cache.
This situation is _normal_, you should not worry about it, you
should not reboot the server to return "free RAM".
Also, read my previous letter about this.
PS please move the discussion to aseek-users list.
Karen Barnes wrote:
> Hi John,
>
> I've experienced the exact same thing. When running the indexer I can
> see memory continually climb until all memory is consumed. I only have
> about 2,000,000 URLs (many just robots.txt) and when I do an index
> that's what happens. When index is done the memory is not released and
> other programs are swaping to disk which ultimately starts thrashing the
> hard drive. I too have 2GB ram, raid 5 and 205GB of free space. Never
> had this problem using the same mysql setup as I have always used
> (my.cnf) before, but since I installed aspseek that's what happens. Here
> are my memory stats:
>
> r b w swpd free buff cache si so bi bo in cs
> 0 0 0 1044 16356 42120 1186860 0 0 29 109 111 102
>
> I run the index like this "./index -N 80"
>
> Top shows this:
>
> 9:20am up 22:45, 2 users, load average: 0.28, 0.24, 0.19
> 288 processes: 287 sleeping, 1 running, 0 zombie, 0 stopped
> CPU0 states: 0.1% user, 0.0% system, 0.0% nice, 99.4% idle
> CPU1 states: 0.4% user, 0.2% system, 0.0% nice, 98.5% idle
> CPU2 states: 0.4% user, 16.3% system, 0.0% nice, 82.4% idle
> CPU3 states: 8.2% user, 19.1% system, 0.0% nice, 72.1% idle
> Mem: 2065152K av, 2044500K used, 20652K free, 0K shrd, 42584K
> buff
> Swap: 2096472K av, 1044K used, 2095428K free 1181836K
> cached
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 13679 scrubweb 25 0 309M 309M 1792 S 0.0 15.3 0:17 index
> and of course many many more index to follow
>
> The my.cnf looks like this:
>
> # The MySQL server
> [mysqld]
> user = mysql
> set-variable = max_connections=512
> port = 3306
> socket = /tmp/mysql.sock
> skip-locking
> set-variable = key_buffer=256M
> set-variable = max_allowed_packet=3M
> set-variable = table_cache=256
> set-variable = sort_buffer=1M
> set-variable = net_buffer_length=8K
> set-variable = myisam_sort_buffer_size=64M
> log-bin
> server-id = 1
>
> [mysqldump]
> quick
> set-variable = max_allowed_packet=16M
>
> [mysql]
> no-auto-rehash
>
> [isamchk]
> set-variable = key_buffer=128M
> set-variable = sort_buffer=128M
> set-variable = read_buffer=2M
> set-variable = write_buffer=2M
>
> [myisamchk]
> set-variable = key_buffer=128M
> set-variable = sort_buffer=128M
> set-variable = read_buffer=2M
> set-variable = write_buffer=2M
>
> [mysqlhotcopy]
> interactive-timeout
>
> Doing a "free"
>
> total used free shared buffers cached
> Mem: 2065152 2047720 17432 0 43224 1184380
> -/+ buffers/cache: 820116 1245036
> Swap: 2096472 1044 2095428
>
> That's a lot of memory being consumed. This is a standalone box so the
> only thing running right now is index. I have searchd running, but no
> access. Yesterday I rebooted my system to recover the memory. Waited
> throughout the night and when I checked RAM this morning it only showed
> that 175,345 was used. All other programs were running like mysql and
> even searchd. As soon as I started index I could just watch the memory
> trickle away. By the end of the indexing process the system will be
> thrashing the disk cached RAM and will never be released.
>
> Just thought I wold let you know my experience. Seems like this is just
> how things work with index. If I don't spawn off 80 threads it will take
> all day to index 200,000 URLs.
>
> Karen
>
>> Today I started with an empty index and inserted 175,000 URLs:
>>
>> ./index -i -f myurls.txt
>>
>> Then I ran the indexer:
>>
>> ./index -N 100 -s 0
>>
>> Consumed a bunch of memory once all was resolved and running. In fact
>> I'm running 2GB DDR 200MHZ RAM on a dual 2.2GHZ Zeon box with Linux
>> running the latest kernel. While running index it will not only
>> consume the entire 2GB, but also about 2,000k of disk swap. Once index
>> is complete none of the RAM was released. The only way I can get it
>> back so other programs were not swaping memory to disk is to do a
>> reboot. This process took 7 hours by the way.
>>
>> Has anyone experienced the same problem? I could easily index
>> 1,000,000 URLs all at once using mysql and a multi-threaded perl
>> indexer with minimum RAM consumption and when done (about 20 hours
>> later) all RAM is released. Maybe it's because I wrote the Perl
>> script? All other programs I have used release the RAM back, but not
>> aspseek's indexer for some reason.
--
[EMAIL PROTECTED] 7551596@ICQ [EMAIL PROTECTED]
Dream like you'll live forever...Love like you've never been hurt...
Work like you don't need the money...and Dance like nobody is watching!
-- Satchel Paige
