[Mimedefang] OT: Quoted text detection
In my mimedefang-filter, I'm trying to segment quoted text from an original email I realize this question is more to do with actual email processing, but is there any trick within MD, MIME Tools or another library that can help me out? I realize there is no 100% accurate way to do it. I'm just trying to get to 80%+. There are just a variety of different ways email clients mark quoted sections within HTML and Text parts. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Is calling action_rebuild() idempotent?
If I call action_rebuild() a couple times in my filter_end code, could it have different effects depending on when it's called? ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] delete part best practice
Given a parent entity and child entity, what's the best way to delete the child entity? I see code that clears all parts for an entity and then selectively adds them back in. I would like to scan through and selectively remove instead. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Part’s parent content-type
Thank you Dianne! This is going to sound silly, but what I'm looking for is a function like $parent_part = get_parent ($top_entity, $part); So given the top level entity and a specific part, the function will return the MIME::Entity for the parent of $part. I'm struggling with getting the recursion on this. Any tips on a simple way to do this? ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Part’s parent content-type
When iterating through the parts of a MIME::Entity using parts_DFS, what would be the best way to get a reference to a part’s parent entity or parent entity type? I’m trying to do some logic on an HTML part depending on if the parent type is multipart/related or not. I was thinking of just setting a variable as I iterate through the parts, but it can get tricky if I don’t know the depth level I’m at. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Easy way to run a test of filter from CLI
If I have a .eml raw file of an email, is there a simple way to run it through a mimedefang.pl/mimedefang-filter process and get the output? I wanted to created a very basic test suite for parsing of some emails without needed to actually send the email through the MTA. Any tips on how to accomplish this is much appreciated. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] MIMEDefang Digest, Vol 165, Issue 5
Thanks Dianne! Our server has 16GB of ram and we have a 2GB RAMDISK setup for md. In the multiplexor man page, I saw this quote: "Using the -R option has the side-effect of permitting new connections from the loopback address to queue." Does this mean you can have loopback connections queue and non-loopback connections get a standard busy warning if the max processes are used up? Regarding your comment about the downside is that it would "Hold open more connections and use more milter threads." I wasn't quite sure what you meant by "using more milter threads"? Do you mean that it will just use MX_MAXIMUM mimedefang.pl processes for longer? ie, until the traffic spike is fully processed? > > > Assume a burst of emails comes in and all 20 md processes are busy > > working. > > > 1) What should the 21st and higher connecting clients > > experience? > > Sendmail will appear to respond more slowly than usual. > > > 2) Is there a configurable limit to the number of queued > > connections? > > Yep. The "-q" option to mimedefang-multiplexor. See the -q and -Q > command-line options in the mimedefang-multiplexor(8) man page. > > > 3) What's the downside of letting the connections queue > > other than holding open more TCP connections? > > Holding open more connections and using more milter threads. > > > 4) Any other tips on the settings you use in production? > > If you actually need to queue requests more than occasionally, you > need a more powerful server. The queueing feature is supposed to handle > transient bursts of traffic. It isn't meant to squeeze more steady-state > performance out of a server. > > > The reason I'm asking is because we recently had a burst of traffic > > that caused the IO on our server to go up to a point where everything > > became unresponsive for a couple minutes. We had our MX_MAXIMUM set > > to 110. > > How much RAM did you have? You really, really, really don't want a > MIMEDefang scanning server to start swapping. > > Regards, > > Dianne. > > > ** ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] ALLOW_NEW_CONNECTIONS_TO_QUEUE behavior
Hi, I wanted to get a better understanding of of the nuances of ALLOW_NEW_CONNECTIONS_TO_QUEUE in a high volume burst scenario. MX_MAXIMUM=20 MX_QUEUE_TIMEOUT=300 Assume a burst of emails comes in and all 20 md processes are busy working. 1) What should the 21st and higher connecting clients experience? 2) Is there a configurable limit to the number of queued connections? 3) What's the downside of letting the connections queue other than holding open more TCP connections? 4) Any other tips on the settings you use in production? The reason I'm asking is because we recently had a burst of traffic that caused the IO on our server to go up to a point where everything became unresponsive for a couple minutes. We had our MX_MAXIMUM set to 110. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] mimedefang-filter $Subject variable encoding
Should the $Subject variable be mime encoded? Is this the correct
way to update the $Subject of an email:
my $subject = mime_to_perl_string($Subject);
$subject = "Foo and maybe Unicode chars";
$Subject = encode_mimeword($subject);
action_change_header('Subject', $Subject);
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Understanding the -q flag
I saw in the documentation that the -q flag will allow mimedefang to accept more connections than the number of mimedefang slaves and it will queue connections if the slave limit is exceeded. I wanted to understand the implications and potential downsides of enabling this better.. So if your slave limit is 100 and they are all processing emails, but you need to process 500 emails that moment, does this mean 400 emails will be queued for processing as soon as the slaves start freeing up? Does this open up a potential denial of service attack? or is it relatively light-weight to keep a connection in queue? Any other gotchas we should be aware of that may not be explicitly mentioned in the docs? Thank you! ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Embedded Perl (continued)
Apologies for starting a new thread. I couldn't find any messages in my inbox to reply to. Thanks Paul, Bill and Diane for your replies. My situation is that the number of mimedefang.pl processes jumps to about 70 during peak loads (we are processing a couple hundres messages per minute on average). Our filter file is in need of some optimizations(since each mimedefang.pl is taking about 125mb of resident memory), but I'm wondering if using embedded perl will help in this situation. I see you mentioned using embedded perl prevents forking entire processes.. So does this mean each request is handled by a thread within the main process instead? So would my RAM requirements be reduced drastically? In my peak case, I roughly calculate my RAM usage just for md.pl to be about 8GB. If embedded perl makes this go down a lot, it's a major win for me. Thanks again for your help ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Embedded Perl (continued)
We have 16GB of ram, though there are other processes running on this machine such as DB that will be segmented later. I'm curious how much resident memory each of your mimedefang.pl processes uses? I haven't been tracking my mimedefang.pl memory usage over time, so I was a little surprised to see it at 125Mb. Before I go down a rabbit hole of minimizing it, i want to make sure it's actually significantly higher than your situation. Also, Am I right in thinking the forking issue is not such a big deal because the processes are pre-forked and stay running for some amount of time and eventually get cleared down to your minimum setting. I have my min processes set to 10, and max to 100.. And my monitoring system shows that I have about 20 running mimedefang.pl processes on average. On Tue, Sep 22, 2015 at 8:12 AM, Dianne Skoll <d...@roaringpenguin.com> wrote: > On Tue, 22 Sep 2015 07:57:18 -0700 > Amit Gupta <amit...@gmail.com> wrote: > >> My situation is that the number of mimedefang.pl processes jumps to >> about 70 during peak loads (we are processing a couple hundres >> messages per minute on average). > > How much RAM do you have? 70 parallel scanners is not outlandish on > busy machines. Our biggest scanning machine is configured to allow > up to 400 scanners. It's a pretty powerful machine with 48GB of RAM, > though, and our volume is 5-10x yours. > >> I see you mentioned using embedded perl prevents >> forking entire processes. > > No... it still forks each time, but it doesn't exec a new program. > >> So would my RAM requirements be reduced drastically? > > Probably not. As I said, embedded Perl helps a little bit, but not > dramatically. > > Regards, > > Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Embedded Perl
Hi, I'm using Centos7.1 and mimedefang 2.75 (still need to upgrade to latest). The documentation indicates that if you start mimedefang with the -E parameter, it uses embedded perl which is supposedly faster. I was wondering if someone could give me insight into why it's actually faster under the hood? And maybe any anecdotal or real statistics on how much faster. Also, should embedded perl be enabled by default? I see a --disable-embedded-perl option in the configure script. I did not configure my system to disable embedded perl, however when I try to experiment with the -E option, mimedefang complains that I did not enable embedded perl.. Thank you! ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Perl version
I've been using mimedefang on centos 6.x with the default perl version of 5.10 for a while now. Everything working well. I wanted to do an upgrade to perl to 5.20. Is there any obvious issue I will encounter? I plan to yum remove perl and then install perl from source.(or possibly from rpm if I find them). 5.10 is 7 years old. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Upgrading MIME::Tools to 5.505?
I noticed in mimedefang.pl there is the following statement: use MIME::Tools 5.410 (); The latest MIME::Tools seems to be 5.505.. MD 2.75 still references the old version. Should we be updating this? Or is this planned for the future? ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
