Re: [Mimedefang] Embedded Perl (continued)
On 2015-9-22 17:16 , Steffen Kaiser wrote:
> I had SpamAssassin rules allocating about 100MB, the forked children
> only shared the C libraries after some time. That's a problem of
> Perl's way to handle rereferences to data.
It might help not to integrate SpamAssassin, but to use spamc to communicate
with spamd. However, that only saves memory if you have other stuff in your
filter rules that take time processing messages, like DNS blacklist checks,
virus scanners, SPF/DKIM/DMARC processing, SMTP forward lookups, etc.
That way, perl processes doing those other things do not have a big lump of
SpamAssassin rules sitting in memory (which is usually quite a lot of memory
due to the way spamassassin works). You'd generally need fewer spamassassin
slaves than mimedefang slaves (if you don't, then this won't save memory but
eat a bit more memory instead because of the extra perl processes involved).
On the other hand, it does make things a bit more complex because you have to
manage another daemon, monitor it, restart when rules change, maintain configs,
etc. Per-recipient rules might be somewhat harder.
Oh, and stock mimedefang doesn't support it. I've attached the SpamC.pm that we
use for spamd communication. Also make sure that you set
$Features{"SpamAssassin"} = 0 in your filter, to prevent Mail::SpamAssassin
from loading (otherwise your mimedefang slaves would still eat memory for
spamassassin).
You will need to modify this SpamC.pm as it uses a modular Mimedefang.pm, but
the changes should be trivial.
--
Jan-Pieter Cornet
"Any sufficiently advanced incompetence is indistinguishable from malice."
- Grey's Law
package MailFilter::SpamC;
# provide spamc interface to spamassassin, call-compatible with mimedefang
# API
# ... mostly. It actually only provides spam_assassin_check().
use Mimedefang qw(gen_msgid_header synthesize_received_header
:global :logging :config);
use IPC::Open2;
use base Exporter;
our @SpamAssassinExtraHeaders;
our @EXPORT_OK = qw(
spam_assassin_check
@SpamAssassinExtraHeaders
);
my $spamc = "/usr/bin/spamc";
my @spamc_opts = qw(-F /etc/spamd/spamc.conf);
sub spam_assassin_check {
### open communications to spamc
my $in;
unless ( open $in, "<", "./INPUTMSG" ) {
md_syslog('err', "$MsgID: Spamc error: Cannot read INPUTMSG: $!");
return;
}
my($sprd, $spwr);
my $sp_pid = open2($sprd, $spwr, $spamc, @spamc_opts);
unless ( $sp_pid ) {
md_syslog('err', "$MsgID: Spamc error: Cannot fork $spamc: $!");
return;
}
### note: the lines below duplicate the effect in the real
### spam_assassin_check somewhat
### build complete headers
my $hdrs = "Return-Path: $Sender\n" .
synthesize_received_header();
$hdrs .= gen_msgid_header() if ($MessageID eq "NOQUEUE");
### get message headers, remember if we had a "To:" header
my($seen_to, $seen_eoh);
while ( <$in> ) {
if ( /^$/ ) {
$seen_eoh++;
last;
}
$seen_to++ if /^To:/i;
$hdrs .= $_;
}
$hdrs .= "To: undisclosed-recipients:;\n" if !$seen_to;
if ( $AddApparentlyToForSpamAssassin and @Recipients ) {
$hdrs .= "Apparently-To: " . join(", ", @Recipients) . "\n";
}
$hdrs .= join("", @SpamAssassinExtraHeaders);
### add header-body separation line that we ate in the loop above
$hdrs .= "\n";
### $hdrs now contains the complete headers as sent to spamc
### send headers to spamc
print $spwr $hdrs;
### send rest of message (if there was any left)
if ( $seen_eoh ) {
print $spwr $_ while <$in>;
}
close $spwr;
### wait for result
my $output = join("", <$sprd>);
close $sprd;
waitpid($sp_pid, 0);
if ( $? ) {
md_syslog('err', "$MsgID: spamc returned non-zero exit code: $?\n");
return;
}
my($hits, $req, $names, $report, %sa_tags);
### first line is hits/req
if ( $output =~ s!\A(-?\d+(?:\.\d+)?)/(-?\d+(?:\.\d+)?)\r?\n!! ) {
($hits, $req) = ($1, $2);
} else {
my $sample = $output;
if ( length($sample) > 80 ) {
$sample = substr($sample, 0, 80) . "...";
}
$sample =~ s{[^ -~]}{sprintf("\\x%02x", ord $1)}ge;
md_syslog('err',
"$MsgID: Error: spamc returned invalid output: $sample");
return;
}
### process rest of output
while ( $output =~ s/\A(\w+):\s+(.*)\r?\n// ) {
my($k,$v) = ($1,$2);
$hits = $v, next if $k eq "Score";
$req = $v, next if $k eq "Required";
$names = $v, next if $k eq "Tests";
$sa_tags{$k} = $v;
}
### anything that is left now is the full report
$output =~ s/^\s+//;
$report = $output;
return($hits, $req, $names, $report, \%sa_tags);
}
1;
signature.asc
Description: OpenPGP digital signature
___
NOTE: If there is a disclaim
Re: [Mimedefang] Embedded Perl (continued)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 22 Sep 2015, Amit Gupta wrote: My situation is that the number of mimedefang.pl processes jumps to about 70 during peak loads (we are processing a couple hundres messages per minute on average). Our filter file is in need of some optimizations(since each mimedefang.pl is taking about 125mb of ^^^ resident memory), but I'm wondering if using embedded perl will help in this situation. I see you mentioned using embedded perl prevents forking entire processes.. So does this mean each request is handled by a thread within the main process instead? So would my RAM requirements be reduced drastically? Read Dianne's response about the garbage collector. Unless the script use very view different values of your loaded data or use weak references, you will not notice any reduction in long run. I had SpamAssassin rules allocating about 100MB, the forked children only shared the C libraries after some time. That's a problem of Perl's way to handle rereferences to data. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBVgFwzFGgR0+MU/4GAQJ5gQf7B/MqyaeU97R22AxFCsT2+/se7Aqy8yFK oMcjXfsyIKG0sUVLbR5fGNALHtw/jpxDFiiikm2z7QzFIhingTUS04/zAwjuqVF2 LhvQ/RgZeGUyq8MHDd4z6sFLH8znbOINpnoIJBhrrE0ewq77gONwi8XRU+F/382z VW3a0k8t9A2QRLqa2JgE1lsVF+mRM/R7/YCASf2CazscwdUtgd0bFUDbzYhGZvO3 Xm1hajxMjdm+xCMBN5WxsjO/iQ1Q9XF083oQy8A/1GGXJR9R91psU4q+Bsu7V5N8 LFLHKGLZayCms1Eh4qshEPtUJde8AX1CicVvr0u3q6DivQHTeQ08Zw== =yqjd -END PGP SIGNATURE- ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl (continued)
On Tue, 22 Sep 2015 08:20:16 -0700 Amit Gupta wrote: > We have 16GB of ram, though there are other processes running on this > machine such as DB that will be segmented later. I'm curious how much > resident memory each of your mimedefang.pl processes uses? About 110MB, but not sure how much of that is shared. > Also, Am I right in thinking the forking issue is not such a big deal > because the processes are pre-forked and stay running for some amount > of time and eventually get cleared down to your minimum setting. Forking is not a big deal at all. execing may be more of a big deal, but still probably not a major performance factor. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl (continued)
We have 16GB of ram, though there are other processes running on this machine such as DB that will be segmented later. I'm curious how much resident memory each of your mimedefang.pl processes uses? I haven't been tracking my mimedefang.pl memory usage over time, so I was a little surprised to see it at 125Mb. Before I go down a rabbit hole of minimizing it, i want to make sure it's actually significantly higher than your situation. Also, Am I right in thinking the forking issue is not such a big deal because the processes are pre-forked and stay running for some amount of time and eventually get cleared down to your minimum setting. I have my min processes set to 10, and max to 100.. And my monitoring system shows that I have about 20 running mimedefang.pl processes on average. On Tue, Sep 22, 2015 at 8:12 AM, Dianne Skoll wrote: > On Tue, 22 Sep 2015 07:57:18 -0700 > Amit Gupta wrote: > >> My situation is that the number of mimedefang.pl processes jumps to >> about 70 during peak loads (we are processing a couple hundres >> messages per minute on average). > > How much RAM do you have? 70 parallel scanners is not outlandish on > busy machines. Our biggest scanning machine is configured to allow > up to 400 scanners. It's a pretty powerful machine with 48GB of RAM, > though, and our volume is 5-10x yours. > >> I see you mentioned using embedded perl prevents >> forking entire processes. > > No... it still forks each time, but it doesn't exec a new program. > >> So would my RAM requirements be reduced drastically? > > Probably not. As I said, embedded Perl helps a little bit, but not > dramatically. > > Regards, > > Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl (continued)
On Tue, 22 Sep 2015 07:57:18 -0700 Amit Gupta wrote: > My situation is that the number of mimedefang.pl processes jumps to > about 70 during peak loads (we are processing a couple hundres > messages per minute on average). How much RAM do you have? 70 parallel scanners is not outlandish on busy machines. Our biggest scanning machine is configured to allow up to 400 scanners. It's a pretty powerful machine with 48GB of RAM, though, and our volume is 5-10x yours. > I see you mentioned using embedded perl prevents > forking entire processes. No... it still forks each time, but it doesn't exec a new program. > So would my RAM requirements be reduced drastically? Probably not. As I said, embedded Perl helps a little bit, but not dramatically. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl
On Mon, 21 Sep 2015 18:53:04 -0400 "Bill Cole" wrote: > As MD & Perl have matured and systems have gotten much bigger and > much faster it has become less of a burden to just fork and exec > fully independent slaves and let them live a very long time, so it > is somewhat questionable whether embedding Perl is valuable to very > many users. At a scale of hundreds of messages per minute on a > single slow CPU it could be indispensable, but few systems have that > sort of challenge. Well... in theory, embedded Perl can also save memory because by not execing an external program, the child processes can share memory pages as long as no-one writes to them. In practice, this isn't very effective because Perl uses reference-counting to implment its garbage collector, so even so-called "read-only" data gets touched as references to it are added and deleted when variables come into and go out of scope. My personal opinion is similar to Bill Cole's, except I'd phrase it as: "It can't hurt. It might help a little bit. But it probably won't make all that much difference." FWIW, we do run a fairly busy scanning cluster that peaks around 11 million messages/day (~127/second or so) and we do use embedded Perl. A given machine in the cluster might peak at 3 million/day. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl
On 21 Sep 2015, at 17:04, Amit Gupta wrote: Hi, I'm using Centos7.1 and mimedefang 2.75 (still need to upgrade to latest). The documentation indicates that if you start mimedefang with the -E parameter, it uses embedded perl which is supposedly faster. I was wondering if someone could give me insight into why it's actually faster under the hood? Because the multiplexor only loads a single Perl interpreter, mimedefang.pl, and mimedefang-filter once at launch (and whenever it gets a 'reread' command) and forks off slaves using the embedded interpreter and pre-loaded scripts as needed, rather than launching new independent slave processes that have to load the interpreter and load the scripts themselves. And maybe any anecdotal or real statistics on how much faster. Oh, you want real data... Picky picky! I can't say how much faster it is in any real world sense, as it was dysfunctional a decade ago on the platforms I tried it on then and I haven't bothered with it since. In principle it isn't going to do much for you if you don't use a lot of slaves and/or don't recycle them often.(The specific definitions of "a lot" and "often" are left as an exercise for you to determine...) As MD & Perl have matured and systems have gotten much bigger and much faster it has become less of a burden to just fork and exec fully independent slaves and let them live a very long time, so it is somewhat questionable whether embedding Perl is valuable to very many users. At a scale of hundreds of messages per minute on a single slow CPU it could be indispensable, but few systems have that sort of challenge. Also, should embedded perl be enabled by default? That depends on your system. I'd expect not for yours. A basic EL7 installation doesn't include the perl-ExtUtils-Embed package (although it is in RedHat's "base" collection so you needn't use CPAN or hunt down a 3rd-party RPM.) I see a --disable-embedded-perl option in the configure script. I did not configure my system to disable embedded perl, however when I try to experiment with the -E option, mimedefang complains that I did not enable embedded perl.. Since ExtUtils::Embed is needed for building an embedded Perl interpreter, the MD configure script won't enable the feature for you automatically if the module's not present, which it isn't by default on CentOS 7.1: [root@cloud119-65 ~]# cat /etc/centos-release CentOS Linux release 7.1.1503 (Core) [root@cloud119-65 ~]# yum list installed perl-ExtUtils-Embed Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.advancedhosters.com * extras: centos.mirror.constant.com * updates: mirror.us.leaseweb.net Error: No matching Packages to list [root@cloud119-65 ~]# yum info perl-ExtUtils-Embed Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.advancedhosters.com * extras: centos.mirror.constant.com * updates: mirror.us.leaseweb.net Available Packages Name: perl-ExtUtils-Embed Arch: noarch Version : 1.30 Release : 285.el7 Size: 49 k Repo: base/7/x86_64 Summary : Utilities for embedding Perl in C/C++ applications URL : http://www.perl.org/ License : GPL+ or Artistic Description : Utilities for embedding Perl in C/C++ applications. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl
On Mon, 21 Sep 2015 14:04:04 -0700 Amit Gupta wrote: > Hi, I'm using Centos7.1 and mimedefang 2.75 (still need to upgrade to > latest). The documentation indicates that if you start mimedefang with > the -E parameter, it uses embedded perl which is supposedly faster. I > was wondering if someone could give me insight into why it's actually > faster under the hood? The speedup comes because with embedded Perl, the multiplexor does not have to fork/exec a Perl process. I don't think it's significantly faster, but it probably helps a little bit. > Also, should embedded perl be enabled by default? Yes. It can't hurt. > I see a --disable-embedded-perl option in the configure script. I did > not configure my system to disable embedded perl, however when I try > to experiment with the -E option, mimedefang complains that I did not > enable embedded perl.. If you didn't have the Perl development libraries installed when you compiled MIMEDefang, it won't have support for embedded Perl. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl
If embedded perl is the same as compiled perl then I would guess the speed-up comes from not having to interpret a perl script with the perl interpreter. Or am I off-base here? -Original Message- From: mimedefang-boun...@lists.roaringpenguin.com [mailto:mimedefang-boun...@lists.roaringpenguin.com] On Behalf Of Amit Gupta Sent: Monday, September 21, 2015 2:04 PM To: mimedefang@lists.roaringpenguin.com Subject: [Mimedefang] Embedded Perl Hi, I'm using Centos7.1 and mimedefang 2.75 (still need to upgrade to latest). The documentation indicates that if you start mimedefang with the -E parameter, it uses embedded perl which is supposedly faster. I was wondering if someone could give me insight into why it's actually faster under the hood? And maybe any anecdotal or real statistics on how much faster. Also, should embedded perl be enabled by default? I see a --disable-embedded-perl option in the configure script. I did not configure my system to disable embedded perl, however when I try to experiment with the -E option, mimedefang complains that I did not enable embedded perl.. Thank you! ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl
On Mon, 18 Jun 2012 19:19:57 +0900 Nathan Findley wrote: > The man pages for the multiplexer indicate that when using embedded > perl the PID of the slave is not available ($$). Is there a way > around this? Wow, that man page is way out of date. :) I fixed that many versions of MIMEDefang ago. I'll have to remove that statement from the man page. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: File::Scan (was Re: [Mimedefang] Embedded Perl multiplexoronDebian - sarge Error)
I've been following this thread, and see where the module should not be used. I use a very simple mimedefang-filter with just filter_recipient in it. It does do the detect_and_load_perl_modules, but I don't really know when the bad module is used. I see a lot of d* files in my input queue with no q* files of delivered email. Should I worry and be doing something? And if so, what should be done? I use the RPM to install, so I'm not sure if that would be a problem with what you all are talking about or not. Thanks Steve Campbell [EMAIL PROTECTED] Charleston Newspapers - Original Message - From: "Kayne Kruse" <[EMAIL PROTECTED]> To: Sent: Thursday, July 20, 2006 3:36 PM Subject: Re: File::Scan (was Re: [Mimedefang] Embedded Perl multiplexoronDebian - sarge Error) ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: File::Scan (was Re: [Mimedefang] Embedded Perl multiplexor onDebian - sarge Error)
On Thu, 2006-07-20 at 10:03 -0400, David F. Skoll wrote: > I can't in good > conscience have built-in support for such a terrible module. > > Regards, > > David. Ironically, I think this has been the cause of my hanging mimedefang i hadnt posted about. Watching the logs I dont appear to have any slaves just hanging around busy doing nothing but making sendmail return tempfails. I havent had one hung slave that didnt kill properly today. Nothing to a milter error state. :D I think I missed the other thread about File::Scan vs PDFs, if it was on this list, I have been MIA thanks to Katrina. I've since said bye bye to the land of hurricanes and Louisiana politics. Life is completely different now. :D Thanks again guys, KK signature.asc Description: This is a digitally signed message part ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: File::Scan (was Re: [Mimedefang] Embedded Perl multiplexoronDebian - sarge Error)
::snip::
> Only reason I had it to begin with was as an extra scanner.
> Call me paranoid, but the cost of two scanners is worth the
> cpu time just to be extra safe.
We killed-off File::Scan several months ago, when it was corrupting PDF
files from some of our production batch jobs, depending upon the encoding
used.
Until the next release, when David officially removes support for
File::Scan, you can still add the following lines (suggested months ago by
David, I believe), to the top of mimedefang-filter, to disable File::Scan in
your installation...
$Features{"File::Scan"} = 0;
$Features{"Virus:FileScan"} = 0;
detect_and_load_perl_modules();
Ken
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: File::Scan (was Re: [Mimedefang] Embedded Perl multiplexor onDebian - sarge Error)
On Thu, 2006-07-20 at 10:03 -0400, David F. Skoll wrote: > In fact, I'd never looked closely at File::Scan before. > > I have removed support for File::Scan from MIMEDefang completely; the > next release won't support it. If you want to continue to use it, you'll > have to write the supporting code in your filter file. I can't in good > conscience have built-in support for such a terrible module. > > Regards, > > David. Well, join the club. I guess its only defense is, where else is it going to put that information. I guess they could have distributed it with a database type file, but have not. Im still okay on that front tho thanks to clam. Only reason I had it to begin with was as an extra scanner. Call me paranoid, but the cost of two scanners is worth the cpu time just to be extra safe. Its not like Im doing 10s of 1000s of emails a day. Averaging around 150 mails/hr through 1 mimedefang box hideously underpowered. :D KK signature.asc Description: This is a digitally signed message part ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
File::Scan (was Re: [Mimedefang] Embedded Perl multiplexor on Debian - sarge Error)
Kayne Kruse wrote: > Now Take david, hes a rather nice person. Now that I said that, why do > you support this monstrosity? :) That was some hideous code, makes me > want to puke shellcodes in the morning! :D Anywho, note taken and I > yanked that crap out. In fact, I'd never looked closely at File::Scan before. I have removed support for File::Scan from MIMEDefang completely; the next release won't support it. If you want to continue to use it, you'll have to write the supporting code in your filter file. I can't in good conscience have built-in support for such a terrible module. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded Perl multiplexor on Debian - sarge Error
Remove File::Scan from your system. It's buggy and evil. Regards, David. Now Take david, hes a rather nice person. Now that I said that, why do you support this monstrosity? :) That was some hideous code, makes me want to puke shellcodes in the morning! :D Anywho, note taken and I yanked that crap out. Kayne ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl multiplexor on Debian - sarge Error
Kayne Kruse wrote:
> I just noticed this error, and was curious as to what in the filter is
> the FD.
> Jul 19 17:03:02 mailfilter mimedefang-multiplexor[4711]: WARNING:
> Something in your Perl filter appears to have opened a file descriptor
> outside of any function. [...]
> lr-x-- 1 root root 64 Jul 19 17:19 7 -> /usr/share/perl5/File/Scan.pm
> Now You see I think its Mimedefangs own open File::Scan causing this
> warning. Whats the fix for the annoyance?
Remove File::Scan from your system. It's buggy and evil.
The open descriptor results from the dubious practice of using
SelfLoader to load functions only on demand. SelfLoader keeps a file
descriptor open. File::Scan uses this to load its file patterns as
Perl routines.
Did I mention that File::Scan was evil? Read the code sometime.
Here's a snippet:
if($type == 1) {
} elsif($type == 2) {
if($subtype == 1) {
if($total==10240) {
/\x48\x45\x4c\x4f\x44\x1d\x4d\x41\x49\xec\xc8\x1e\x52\xbc\x6e\x5d\xc3\x43\x50\x54/s
and $virus = "W32/[EMAIL PROTECTED]", last LINE;
}
if($total==1024) {
/\x0f\x4d\x61\x70\x56\x5f\xc5\x77\x4f\x66\x18\x10\x1e\x55\x6e\x56\x6d\x11\x90\x2f\x62\x08\x72\x73\xb3\x30\x0c\x99\x45\x6e\x76\x22\x6f\xdf\x52\xfc\x7b\x3c\x56\x61\xfb\xe6\x62\xac\x19\x67\x44\x1a\x76\xb1\x54\x79\x70\x4c\x0f\x53\xf5\xbf\x6c\x8e\x6d\x54\x69\x79/s
and $virus = "W32/[EMAIL PROTECTED]", last LINE;
}
} elsif($subtype == 2) {
if($total==4096) {
... and so on.
Regards,
David.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded perl problems with 2.45, again ... (reload broken)
> +(eval 'use Net::DNS; $Features{"Net::DNS"} = 1;')
> +or $Features{"Net::DNS"} = 0;
After looking at Net::DNS I suspect that a circular reference is the problem
here.
I dunno how to solve it.
Martin
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Re: MIMEDefang embedded perl stability issues
* Chris Gauch <[EMAIL PROTECTED]>
> I'm on Redhat Linux 9 for the OS, latest kernel, etc. (through RHN
> update). MIMEDefang version is 2.43, Spamassassin 2.63, and ClamAV
> 0.72. This problem was fairly consistent when using the embedded perl
> interpreter, the MD multiplexor would crash at almost the exact same
> time every morning (around 4-5am). I don't encounter the problem when
> I'm not using embedded perl.
That time of morning is when various cron jobs are run by default, such
as those under /etc/cron.{hourly,daily}. I have seen Mandrake and RedHat
systems croak in a variety of entertaining ways due to these cron jobs
when the system is under heavy load or using excessive amounts of RAM
for something.
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded Perl & External Perl Modules
--- "David F. Skoll" <[EMAIL PROTECTED]> wrote:
> On Wed, 5 May 2004, Chris Masters wrote:
>
> > Hmmm. I initialise all connection handles in a
> > getConnection function that checks the connection
> and
> > opens one if necessary (because there never used
> to be
> > a filter_initialize function and it's very
> resilient).
>
> > Any other reasons?
>
> We've noticed a problem with error reporting and the
> embedded interpreter,
> but haven't been able to track it down. Try
> wrapping your use
> statements in a BEGIN { eval { use XXX; } or die
> ("Error with use: $@"); }
>
Thanks David. Already doing that - the error is based
around the connection error:
Can't locate object method "new" via package
"Net::LDAP::Bind" at
/usr/lib/perl5/site_perl/5.8.0/Net/LDAP.pm line 227.
H. This is the same error if it cannot connect. I
can recreate and test further if required.
Cheers, Chris
code:
eval
{
md_syslog('info',"Connecting to LDAP Directory");
$ldap = Net::LDAP->new($LDAPServer,timeout =>
$LDAPTimeout);
$mesg = $ldap->bind($LDAP_user, password =>
$LDAP_passwd);
if ($mesg->code)
{
use Net::LDAP::Util qw(ldap_error_text);
$error = ldap_error_text($mesg->code);
md_syslog('err',"LDAP Connect Error: $error");
undef $ldap;
}
};
if($@ || !defined($ldap))
{
action_tempfail("LDAP Connect Error(Queuing
mail)");
md_syslog('err',"LDAP Error: $@");
undef $ldap;
}
__
Do you Yahoo!?
Win a $20,000 Career Makeover at Yahoo! HotJobs
http://hotjobs.sweepstakes.yahoo.com/careermakeover
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded Perl & External Perl Modules
On Wed, 5 May 2004, Chris Masters wrote:
> Hmmm. I initialise all connection handles in a
> getConnection function that checks the connection and
> opens one if necessary (because there never used to be
> a filter_initialize function and it's very resilient).
> Any other reasons?
We've noticed a problem with error reporting and the embedded interpreter,
but haven't been able to track it down. Try wrapping your use
statements in a BEGIN { eval { use XXX; } or die ("Error with use: $@"); }
Regards,
David.
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded Perl & External Perl Modules
--- Michael Sims <[EMAIL PROTECTED]> wrote: > Chris Masters wrote: > > I've just tested the embedded perl option and they > > don't seem to be functioning properly. The first > > module (Net::LDAP) that gets used doesn't function > > properly (cannot establish connections). > > > > If I change back to normal [non-embedded] mode > it's > > fine. > > > > Are there any issues with using perl modules when > > mimedefang runs in embedded mode? > > If you are creating a new instance of the Net::LDAP > object directly in your filter > file (outside of any subroutine), then try creating > the filter_initialize sub moving > that initialization inside it. > Hmmm. I initialise all connection handles in a getConnection function that checks the connection and opens one if necessary (because there never used to be a filter_initialize function and it's very resilient). Any other reasons? Thanks for your response Michael. __ Do you Yahoo!? Win a $20,000 Career Makeover at Yahoo! HotJobs http://hotjobs.sweepstakes.yahoo.com/careermakeover ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded Perl & External Perl Modules
Chris Masters wrote: > I've just tested the embedded perl option and they > don't seem to be functioning properly. The first > module (Net::LDAP) that gets used doesn't function > properly (cannot establish connections). > > If I change back to normal [non-embedded] mode it's > fine. > > Are there any issues with using perl modules when > mimedefang runs in embedded mode? There's a new section in the mimedefang-filter man page with the heading "INITIALIZATION AND CLEANUP" which says, in part: "... How-ever, if you are using an embedded Perl interpreter, then anything you call directly from outside a function definition is executed once only in the parent process. Anything in filter_initialize is executed once per slave. If you use any code that opens a descriptor (for example, a connection to a database server), you must run that code inside filter_initialize and not directly from the filter, because the multiplexor closes all open descriptors when it activates a new slave." If you are creating a new instance of the Net::LDAP object directly in your filter file (outside of any subroutine), then try creating the filter_initialize sub moving that initialization inside it. HTH ___ Michael Sims Project Analyst - Information Technology Crye-Leike Realtors Office: (901)758-5648 Pager: (901)769-3722 ___ ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded perl
On Sat, 24 Apr 2004, Les Mikesell wrote: > Would you have a lot of unnecessary overhead if, instead of > writing your own multiplexing and embedded perl, you ran > a stock apache/mod_perl or fastcgi and a really simple > front end that dispatches the jobs via http requests? I have no idea, and I have no inclination to find out. :-) > It would add some software dependencies, but all things > that are well tested, packaged in distributions, and > already understood by system administrators. apache + mod_perl is a very fragile thing, IMO. There are lots of things still broken in apache2's mod_perl. > You would also inherit a framework for web status checking and > administration. We already have that in our commercial product using the multiplexor control socket and other means, so it's already doable. Regards, David. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded perl
On Fri, 23 Apr 2004, David F. Skoll wrote: > On Fri, 23 Apr 2004, Henrik Schmiediche wrote: > > What is the advantage to Embedding Perl in the muliplexor? > > Faster startup time and much less memory use. The man mimedefang-multiplexor man page says: -E Specifies that the multiplexor should create an embedded Perl interpreter. This can improve performance dramatically. But see the section "EMBEDDING PERL" for more information. When I read this it wasn't clear whether "performance" applied only to speed, or to memory use, or to both. Perhaps the second sentence should be "This can dramatically decrease both startup time and memory use." Just a thought. -David ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded perl
On Sat, 2004-04-24 at 09:10, David F. Skoll wrote: > Yes; you need either a shared library libperl.so, or a static one (libperl.a) > on your system. Would you have a lot of unnecessary overhead if, instead of writing your own multiplexing and embedded perl, you ran a stock apache/mod_perl or fastcgi and a really simple front end that dispatches the jobs via http requests? It would add some software dependencies, but all things that are well tested, packaged in distributions, and already understood by system administrators. You would also inherit a framework for web status checking and administration. --- Les Mikesell [EMAIL PROTECTED] ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded perl
On Fri, 23 Apr 2004, Henrik Schmiediche wrote: > How can you test if a given implementation is safe? Look at "configure.in" and search for "AC_MSG_CHECKING([if it is safe to destroy and recreate a Perl interpreter])" > Does Perl need to be compiled in any special way to enable embedded Perl to > work? Yes; you need either a shared library libperl.so, or a static one (libperl.a) on your system. Regards, David. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
RE: [Mimedefang] Embedded perl
Thanks... How can you test if a given implementation is safe? Does Perl need to be compiled in any special way to enable embedded Perl to work? Sincerely, - Henrik -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David F. Skoll Sent: Friday, April 23, 2004 5:29 PM To: [EMAIL PROTECTED] Subject: Re: [Mimedefang] Embedded perl On Fri, 23 Apr 2004, Henrik Schmiediche wrote: > What is the advantage to Embedding Perl in the muliplexor? Faster startup time and much less memory use. > Why is it not enabled by default? Because it's broken on some platforms. It's fine on Linux, mostly fine on Solaris 9, and all bets are off on other platforms. The innards of Perl are very scary... Regards, David. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded perl
On Fri, 23 Apr 2004, Henrik Schmiediche wrote: > What is the advantage to Embedding Perl in the muliplexor? Faster startup time and much less memory use. > Why is it not enabled by default? Because it's broken on some platforms. It's fine on Linux, mostly fine on Solaris 9, and all bets are off on other platforms. The innards of Perl are very scary... Regards, David. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl problems
Verify it is using the newer embedded version by calling manually. what is the full memory listing for all the mimedefang process? post the result of "ps axuww | grep mimedefang" cut off parts so it doesn't word wrap. Josh Kelley said: > I've verified that the -E is listed in the ps auxwww listing, and I've > verified that embedded support was compiled in. Any other suggestions? -- Luke Computer Science System Administrator Security Administrator,College of Engineering Montana State University-Bozeman,Montana ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl problems
Lucas Albers wrote: You should show this sort of command arguments when you run ps axuww|grep mimedefang /usr/bin/mimedefang-multiplexor -p /var/spool/MIMEDefang/mimedefang-multiplexor.pid -E -m 4 -x 9 -U defang -i 30 -b 600 -W 1 -l -q 10 -s /var/spool/MIMEDefang/mimedefang-multiplexor.sock Notice the -E. Verify that it had embedded support when it was compiled. I've verified that the -E is listed in the ps auxwww listing, and I've verified that embedded support was compiled in. Any other suggestions? Josh Kelley ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl problems
Josh Kelley said: > I'm using the provided init script for Red Hat, so I can set the > appropriate option in /etc/sysconfig/mimedefang to make the init script > pass the -E option. > > I've confirmed that ps lists the children as "mimedefang-multiplexor" > rather than "perl mimedefang.pl", which should mean that embedded Perl > is enabled. You should show this sort of command arguments when you run ps axuww|grep mimedefang /usr/bin/mimedefang-multiplexor -p /var/spool/MIMEDefang/mimedefang-multiplexor.pid -E -m 4 -x 9 -U defang -i 30 -b 600 -W 1 -l -q 10 -s /var/spool/MIMEDefang/mimedefang-multiplexor.sock Notice the -E. Verify that it had embedded support when it was compiled. -- Luke Computer Science System Administrator Security Administrator,College of Engineering Montana State University-Bozeman,Montana ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl problems
Lucas Albers wrote: What version of perl are you running? 5.8.0, the version that ships with RHEL 3. Look at the shared memory, it will indicate that each slave is sharing 2/3 of it's memory with the master. Like such: defang 9 0 31992 31M 19972 S 0.0 6.3 0:00 mimedefang-mult defang 9 0 31040 30M 20600 S 0.0 6.1 0:00 mimedefang-mult defang 9 0 30220 29M 29740 S 0.0 5.9 0:00 mimedefang-mult defang 9 0 30212 29M 29732 S 0.0 5.9 0:09 mimedefang-mult defang 9 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang defang 4 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang defang 9 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang defang 9 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang See the number after 31m that is 20m or so? You can see te 3rd and 4th slave aren't using any extra memory, they are sharing all from the parent. That's what I was looking at. The 5th column for the mimedefang-multiplexors was only 3m or so, the same as it is when I have embedded Perl disabled. You need to specifically enable the embedded option with the -E switch on the multiplexor startup script. I'm using the provided init script for Red Hat, so I can set the appropriate option in /etc/sysconfig/mimedefang to make the init script pass the -E option. I've confirmed that ps lists the children as "mimedefang-multiplexor" rather than "perl mimedefang.pl", which should mean that embedded Perl is enabled. Josh Kelley ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl problems
Josh Kelley said: > I'm trying to get the multiplexor's embedded Perl mode to work, since > messages on the mailing list indicated that it would save a lot of > memory, and our mail server (running Red Hat Enterprise Linux 3) is a > bit limited on memory. > > First, a minor point: it doesn't seem to be helping the memory usage > Second, a major point: when I enable embedded mode, slaves apparently > tend to become busy and never report that they're done. > > Thanks. > What version of perl are you running? Look at the shared memory, it will indicate that each slave is sharing 2/3 of it's memory with the master. Like such: defang 9 0 31992 31M 19972 S 0.0 6.3 0:00 mimedefang-mult defang 9 0 31040 30M 20600 S 0.0 6.1 0:00 mimedefang-mult defang 9 0 30220 29M 29740 S 0.0 5.9 0:00 mimedefang-mult defang 9 0 30212 29M 29732 S 0.0 5.9 0:09 mimedefang-mult defang 9 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang defang 4 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang defang 9 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang defang 9 0 1632 1628 1408 S 0.0 0.3 0:00 mimedefang See the number after 31m that is 20m or so? You can see te 3rd and 4th slave aren't using any extra memory, they are sharing all from the parent. That is the amount of memory it is sharing, so memory usage is (Total Mem X n-slaves - (Shared Mem X n-slaves) 31M+30M - 20M = 40M You save about 10M per slave over non embedded. You need to specifically enable the embedded option with the -E switch on the multiplexor startup script. -- Luke Computer Science System Administrator Security Administrator,College of Engineering Montana State University-Bozeman,Montana ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
