http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4859
Summary: memory bloat in resolver
Product: Spamassassin
Version: SVN Trunk (Latest Devel Version)
Platform: Other
OS/Version: other
Status: NEW
Severity: critical
Priority: P1
Component: spamassassin
AssignedTo: [email protected]
ReportedBy: [EMAIL PROTECTED]
i had slow memory leak that i've been fighting for a while now... so today i
tackled it.
this effects all versions of SA that use URIBLDNS via plugin...
initial testing of looping through a corpus and running each file through M::SA
see memory creep from 23MB starting out to over 50MB after 6000 messages unless
i create a new SA instance after so many messages processed. spamd's
max-conn-per-child is why many of you dont notice this problem.
So I ran some testing, and did some DataDump's and here are the results. As you
can see, the number of items returned from a Dumper \$sa increases with each
scan.
# perl samemtest.pl
Run #1, $sa length = 84881
Run #2, $sa length = 1383448
Run #3, $sa length = 1386779
Run #4, $sa length = 1390087
Run #5, $sa length = 1393439
Run #6, $sa length = 1396768
Run #7, $sa length = 1400074
Run #8, $sa length = 1403419
Run #9, $sa length = 1406740
Run #10, $sa length = 1410038
Run #11, $sa length = 1413383
Run #12, $sa length = 1416706
Run #13, $sa length = 1420009
Run #14, $sa length = 1423363
Run #15, $sa length = 1426694
Run #16, $sa length = 1430002
Run #17, $sa length = 1433351
Run #18, $sa length = 1436677
Run #19, $sa length = 1439979
Run #20, $sa length = 1443323
Run #21, $sa length = 1446644
Run #22, $sa length = 1449965
Run #23, $sa length = 1453312
Run #24, $sa length = 1456639
create new instance...
Run #25, $sa length = 84881
Run #26, $sa length = 1382241
Run #27, $sa length = 1385571
Run #28, $sa length = 1388878
Run #29, $sa length = 1392224
Run #30, $sa length = 1395546
Run #31, $sa length = 1398844
Run #32, $sa length = 1402188
Run #33, $sa length = 1405510
Run #34, $sa length = 1408810
Run #35, $sa length = 1412161
Run #36, $sa length = 1415492
So, i decided to write Dumper \$sa to dump$count.txt for each iteration and
compare the changes. What I have found is that the bloat comes from the
resolver. seeing that the $self->{resolver} is created M::SA->new(), the
memory bloat (not a leak) comes from there.
the problem exists in the storage of $self->{resolver}->{id_to_callback} area.
in dump2 we have 27 instances in id_to_callback. in dump19, we have 486
instances. if i let it loop through a few thousand messages, it just keeps on
growing. poll_responses() is supposed to delete the instances with
my $cb = delete $self->{id_to_callback}->{$id};
but, its failing to get to that point on messages that abort the dns queries due
to deadline being reached (aka uridnsbl_timeout).
i added the following 3 lines to the bottom of
M::SA::Plugin::URIDNSBL::abort_remaining_lookups(); and now i have no more
memory bloat (thank god, those process restarts every 2 days sucked), datadump
per run as shown....
-rw-r--r-- 1 root root 84995 Apr 7 19:16 dump1.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump2.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump3.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump4.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump5.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump6.txt
-rw-r--r-- 1 root root 1324687 Apr 7 19:16 dump7.txt
attaching patch shortly...
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
