Re: higher memory requirements in 2.4.0 ? [isolated]
Note, my system configuration is different from yours. I am running Perl 5.005_03 on RH 5.2 w/2.2.9 kernel. There is a possibility that Perl 5.004 has some memory leaks exposed by MHonArc v2.4.0. The perl version that is recommended in RedHat 5.2's errata notes is: # rpm -q perl MHonArc perl-5.004m7-1 Running under this perl, all memory is consumed every time I try asking MHonArc 2.4.0 to add to the big database. So, I tried upgrading perl. Scrounging the net for a slightly newer version of perl, I found this one in ftp://contrib.redhat.com: # rpm -q perl perl-5.005_02-1 # perl -v This is perl, version 5.005_02 built for i386-linux-thread [...] MHonArc 2.4.0 worked fine under this perl and contained itself to about 143 meg. (More or less, I get the number from watching top during the run.) Switching back to the old version of perl caused the problem to reappear. I'm not going to draw any broad sweeping conclusions from this experiment, but it's safe to say I will be sticking with the newer version of perl. Jeff
Re: higher memory requirements in 2.4.0 ? [isolated]
On June 29, 1999 at 01:40, Jeff Breidenbach wrote: Note, my system configuration is different from yours. I am running Perl 5.005_03 on RH 5.2 w/2.2.9 kernel. There is a possibility that Perl 5.004 has some memory leaks exposed by MHonArc v2.4.0. The perl version that is recommended in RedHat 5.2's errata notes is: # rpm -q perl MHonArc perl-5.004m7-1 RH 5.2 is somewhat old. I've made it a habit of just building from source obtained from language.perl.com. Running under this perl, all memory is consumed every time I try asking MHonArc 2.4.0 to add to the big database. So, I tried upgrading perl. Scrounging the net for a slightly newer version of perl, I found this one in ftp://contrib.redhat.com: # rpm -q perl perl-5.005_02-1 # perl -v This is perl, version 5.005_02 built for i386-linux-thread --^^ Threading is currently very experimental in Perl. MHonArc 2.4.0 worked fine under this perl and contained itself to about 143 meg. (More or less, I get the number from watching top during the run.) Switching back to the old version of perl caused the problem to reappear. I'm not going to draw any broad sweeping conclusions from this experiment, but it's safe to say I will be sticking with the newer version of perl. Hey, if it works, go with it. --ewh
Re: higher memory requirements in 2.4.0 ?
On June 26, 1999 at 23:59, Jeff Breidenbach wrote:
Upgrading to 2.4.0 caused out of memory errors when adding to a 60,000
document archive. In version 2.3.3, similar operations only required
about 130 meg, or half of available memory. Running out memory is
serious, because the lack of memory effectively stops the kernel from
starting new processes (causing commands like 'ps' to
coredump). This makes the machine unusable until MHonArc finally exits
with return code 137. Smaller archives appear to work correctly.
Here are some runs of MHonArc v2.4.0 and v2.3.3 on my system
(Note, "mha-work" = v2.4.0, and "mha" = v2.3.3):
119 messages (default settings):
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15711 ehood 0 0 4320 4320 1036 T 0 0.0 3.3 0:06 mha-work
15712 ehood 10 0 4196 4196 1036 T 0 0.0 3.2 0:06 mha
Ratio: 1.029552
121 messages (default settings):[1]
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15696 ehood 0 0 4060 4060 1040 T 0 0.0 3.1 0:05 mha-work
15697 ehood 13 0 3920 3920 1028 T 0 0.0 3.0 0:05 mha
Ratio: 1.0357143
477 messages (default settings):[1]
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15689 ehood 18 0 5840 5840 1048 T 0 0.0 4.5 0:23 mha-work
15686 ehood 0 0 5640 5640 1036 T 0 0.0 4.3 0:21 mha
Ratio: 1.035461
2963 messages (default settings):
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15577 ehood 0 0 16600 16M 1048 T 0 0.0 12.9 2:37 mha-work
15575 ehood 0 0 16360 15M 1036 T 0 0.0 12.7 2:24 mha
Ratio: 1.0146699
2963 messages (Jeff's rcfile):
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15766 ehood 0 0 14788 14M 1052 T 0 0.0 11.5 2:35 mha-work
15769 ehood 9 0 14576 14M 1040 T 0 0.0 11.3 2:32 mha
Ratio: 1.0145445
[1] Digest::MD5 got loaded for mha-work, causing an increase in memory
usage to hold the module.
From my observations, the memory increase is small, and as the number
of messages increases, the ratio of v2.4.0/2.3.3 approached 1.0. I
would guess that with 60,000 messages, the extra memory would amount
to ~1-1.5 MBs. It is interesting that your rcfile actually reduced the
total memory usage of both versions from the default settings.
Note, my system configuration is different from yours. I am running
Perl 5.005_03 on RH 5.2 w/2.2.9 kernel. There is a possibility that
Perl 5.004 has some memory leaks exposed by MHonArc v2.4.0. The way to
check is to run both versions of MHonArc on smaller data sets and check
the process sizes. The way I caught the size info for above is that I
created a variant of the "mhonarc" source file to suspend itself after
processing, but before exiting. Then I can use top(1) (or ps(1) can be
used) to get its size.
Here is source of the variant version:
MAIN: {
unshift(@INC, 'lib'); # Should I leave this line in?
require 'mhamain.pl' || die qq/ERROR: Unable to require "mhamain.pl"\n/;
mhonarc::initialize();
mhonarc::process_input();
kill "STOP", $$; # to suspend ourself (SIGSTOP = 23)
}
Make sure that the mhamain.pl required is getting the proper version
you are testing. You can verify by the following:
shell mhonarc.variant -v
And check what version is printed out. I had to be careful since I
could be grabbing different libraries depending on what my current
working directory is.
--ewh
Re: higher memory requirements in 2.4.0 ?
On June 26, 1999 at 23:59, Jeff Breidenbach wrote: No changes were made to the (complicated) rcfile, except the removal of a timezone resource. No changes were made to the (complicated) command line arguments. I'd be happy to supply the core file, database file, rcfile, command line options, run profiling or debugging tools (need instructions) or provide remote access to a machine that demonstrates the problem. Here's some system information. Can you send me your rcfile and command-line options you specify? The only thing I can think of off-hand that would consume more memory is author and subject based sorting since there is a pre-extraction phase to make comparisons quicker. I do not know if there are any memory profilers for perl. I'll check the Devel::* modules, but I doubt there is anything. If anyone knows of any memory profile tools for perl, please drop me a note. --ewh
higher memory requirements in 2.4.0 ?
I've been putting 2.4.0 through some paces. Upgrading to 2.4.0 caused out of memory errors when adding to a 60,000 document archive. In version 2.3.3, similar operations only required about 130 meg, or half of available memory. Running out memory is serious, because the lack of memory effectively stops the kernel from starting new processes (causing commands like 'ps' to coredump). This makes the machine unusable until MHonArc finally exits with return code 137. Smaller archives appear to work correctly. No changes were made to the (complicated) rcfile, except the removal of a timezone resource. No changes were made to the (complicated) command line arguments. I'd be happy to supply the core file, database file, rcfile, command line options, run profiling or debugging tools (need instructions) or provide remote access to a machine that demonstrates the problem. Here's some system information. Jeff --- # mhonarc -V MHonArc v2.4.0 (Perl 5.00405) Copyright (C) 1995-1999 Earl Hood, [EMAIL PROTECTED] MHonArc comes with ABSOLUTELY NO WARRANTY and MHonArc may be copied only under the terms of the GNU General Public License, which may be found in the MHonArc distribution. # rpm -q perl MHonArc perl-5.004m7-1 MHonArc-2.4.0-1 # uname -a Linux marmot.jab.org 2.0.36 #1 Tue Dec 29 13:11:13 EST 1998 i586 unknown # free total used free sharedbuffers cached Mem:257048 41856 215192 16804 14348 14736 -/+ buffers/cache: 12772 244276 Swap:72256 7832 64424 From system logs, MHonArc output is reported as 'mailme'. The interpreter messages are from me trying to run programs as root and then as a regular user during the problem time. Jun 26 20:57:54 marmot mailme: Warning: Database (2.3.3) != program (2.4.0) version. Jun 26 20:59:27 marmot mailme: Out of memory! Jun 26 21:00:20 marmot kernel: Unable to load interpreter Jun 26 21:00:52 marmot kernel: Unable to load interpreter Jun 26 21:01:58 marmot last message repeated 3 times Jun 26 21:02:41 marmot last message repeated 4 times Jun 26 21:03:20 marmot last message repeated 3 times Jun 26 21:03:21 marmot PAM_pwdb[18495]: (su) session closed for user root Jun 26 21:03:24 marmot kernel: Unable to load interpreter Jun 26 21:03:36 marmot last message repeated 2 times Jun 26 21:04:09 marmot mailme: MHonArc returned exit code 137 for [EMAIL PROTECTED]
