[Mimedefang] replacing body
Sorry if this is a dumb question from a Perl newbie. Still learning. Previously, Dianne provided some pseudo-code for use in filter_end to recursively traverse the MIME tree and rebuild it. I didn't forget that. But for someone new to MIMEDefang and Perl, and for my first version of the filter, I'm trying to get as much done as possible using the automated traversal provided by filter() and filter_multipart(). Reading MIME::Entity doc, is says that "bodyhandle [VALUE]" can be used to Get or Set. And currently, I've been using it to GET the bodyhandle inside filter(): $bh = $entity->bodyhandle $bh_string = $bh->as_string; Now supposed I'd like to substitute $new_string as the new body. How do I do that? Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] replace_entire_message() help
> > 1) When building $new_entity, does every field need to be included? > > Yes. > > 2) I can't figure out how to replace the Subject header - even with a > > string literal. For example, suppose I have the following in > > filter_end(): > > You can't replace top-level headers this way. You need to use > action_change_header(...) Thanks Dianne. But I guess I'm misunderstanding something fundamental because I don't understand how those two answers are not in conflict with each other. In other words, if the top-level headers can't be replaced by new values in the MIME::Entity->build() method, then why do I still need to include them when building the new entity? Perhaps you meant I need to include all header fields if they are part of a body part in a multi-part message? Again, pardon if my nomenclature/phraseology isn't exactly correct. Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] replace_entire_message() help
I need some help with the replace_entire_message() function. Apologies in advance if the problem is my novice perl knowledge. Assume $entity is the existing entity and $new_entity is the newly built entity. 1) When building $new_entity, does every field need to be included? Or will extra info be copied from the existing $entity? For example, suppose I have the following in filter_end(): $new_entity = MIME::Entity->build( Subject => $foo, Data=> $bar, ); Do I also need to specifically set Bcc, Cc, Date, From, Message-ID, etc. equal to the corresponding $entity values? Or will the extra fields be copied from $entity? 2) I can't figure out how to replace the Subject header - even with a string literal. For example, suppose I have the following in filter_end(): $new_subject = 'Replacement Subject'; $new_body= 'Replacement Body'; $new_entity = MIME::Entity->build( Type=> 'text/plain', Subject => $new_subject, Data=> $new_body, ); replace_entire_message($new_entity); What comes through the filter (received in the user's mailbox) is a text/plain message with the original subject and the replacement body. What am I doing wrong? Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to quoted-printable
> -Original Message- > } Unfortunately, it's an amateur radio BBS written in the 80s that > > Which app is it? JNOS http://www.langelaar.net/projects/jnos2/ > } only understands plain text. It was pretty advanced in its time > > Something this old may not understand quoted-printable. It doesn't really understand MIME. So it spits out quoted-printable as the text that it is. > I have a similar requirement to modify a message where I'm > using action_external_filter(). My filter() function looks like > this: Excellent. Examples of similar actions (even if they're not directly applicable) are a real help to a MIMEDefang and Perl newbie. > > There should probably be some more error checking such as making > sure that action_external_filter() succeeds. Perhaps if you look > at the definition of action_external_filter() in mimedefang.pl it > will give you some ideas. It's a fairly simple function (most of > it is error checking). > > VA7JMN Thanks. I'll do that. Michael N6MEF ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to quoted-printable
> -Original Message- > But you have a tool here that is flexible enough to do it. Yes, indeed. > I suppose you want to take the text/plain MIME part and remove any > other parts. Others would be just an HTML alternative or something > that is not text. Yup. And I'm doing that already, and adding a footer to tell the recipient what was removed. The case I'm trying to solve is when the text/plain or text/html part has Content-Transfer-Encoding of Base64. > If you happen to know that the original was always plain ascii with > normal line lengths (80 or less) then I'd tempted to say just go from > base64 to ascii. Yup. And "groff -T ascii" has been suggested, just in case there are emoji's or other things to squash. But the quoted-printable suggestion would be less destructive so I'm trying that out first. Either way, 7bit or quoted-printable, I apparently need to more carefully parse the MIME structure and ultimately rebuild the message. Hence, my questions. > It does make me wonder why it is base64 at all then, but I've been > seeing mail from Exchange that is always base64, so maybe that is what > you are dealing with. In most cases, it's the carrier or email service provider or sender's client app and it's often not something the sending user can control. So rejecting the message results in a complete loss of communications. Example: send a simple text message like "Hello" from Sprint to an email address and they wrap it in HTML and encode it with Base64. Sending from Verizon, a simple text message arrives as 7bit plain text. A short text, like "Hello", with one emoji arrives as quoted-printable. A short text message with multiple emoji's arrives as base64. A slightly longer text message, like "Me and the kids are safe at Mom's house. Call when you can.", with multiple emoji's arrives as an attachment. Different things also happen from smart phone email apps. So, "there be dragons". And ultimately, this may be beyond my ability to solve. But I'm not ready to give up yet. If I can convert at least some messages that otherwise would not be readable, it's a win. Michael a.k.a Don Quixote de La Mancha ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to quoted-printable
> -Original Message- > I won't give the actual Perl code, but here's a basic sketch, completely > untested. I really have no idea whether or not this will work, but I > think the basic approach is correct. :) Thanks Dianne. That definitely points me in the right direction. Much appreciated. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to quoted-printable
> -Original Message- > > >Unfortunately, it's an amateur radio BBS written in the 80s that only > understands plain text. > > It's still extremely useful and widely used, especially in emergency > comms > > So worthwhile investing some cash and getting it updated so that it can > cope with the modern world? [clipped] While I appreciate everyone's attempt to warn me that "there be dragons", the facts are that the client is what it is, it serves hundreds of people (in my location alone), and the problem needs solving. And, while I may be a newbie to MIME and MIMEDefang (and Perl, for that matter), I'm definitely NOT a newbie in most other respects. I've been doing network engineering since the days of the NSFnet and I'm pretty certain I understand this particular application need better than anyone else here. So, going forward, I'm hopeful I can get advice and help on MIME and MIMEDefang here. And warnings about dangers of this or that detail or approach are, of course, very helpful. But suggesting that the client be changed doesn't really answer the questions that I'm asking. Paul, Dianne: regarding readability of quoted-printable: again, I'm a newbie to MIMEDefang, not the world. So I understand and appreciate the warning but I still have to solve the problem. The incoming messages were originated as plain ASCII text. So, the quoted-printable results I'm seeing aren't hard to read at all. But if worse comes to worse, I may end up squashing it to plain ASCII text. We'll see. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to quoted-printable
> -Original Message- > If the old client that you are accommodating is running on (or even just > displaying on) a reasonably modern OS, there's a strong chance that > you'll actually do better for readability by using UTF-8 as the > character set and "8bit" encoding instead of QP. Many (but not all) > older mail clients don't really know MIME but do happily pass through > all 8 bits of every byte to their output, whether it's a terminal driver > or a GUI API. Thanks Bill. Unfortunately, it's an amateur radio BBS written in the 80s that only understands plain text. It was pretty advanced in its time in that it uses SMTP internally. But no MIME, no UTF-8, etc. It's still extremely useful and widely used, especially in emergency comms, delivering mail over VHF radio where other connectivity isn't available. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to quoted-printable
> On Tue, 10 Oct 2017 16:11:07 -0400 > Joseph Brennan wrote: > > > On Tue, Oct 10, 2017 at 1:48 PM, Michael Fox wrote: > > > I need to convert base64 to something readable. > > > To be honest I can't think of a reason to do this. Nevertheless, I need to for an old client. So, back to my question: if ($bh = $entity->bodyhandle) ; $qp_body = encode_qp($bh->as_string) } It seems to work (I use md_syslog() to show the first 20 chars). Is that right? Or is there a better way? Then, I see an action_change_header(), but no action_change_body(). So, presume I need to build a new entity by copying headers from the old entity, replacing the Content-Transfer-Encoding header, and replacing the body. Then use replace_entire_message(). Does that sound about right? Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] base64 to quoted-printable
I need to convert base64 to something readable. Per previous suggestions on this list, I'm going to convert it to quoted-printable. So far, I'm using: if ($bh = $entity->bodyhandle) ; $qp_body = encode_qp($bh->as_string) } It seems to work (I use md_syslog() to show the first 20 chars). Is that right? Or is there a better way? Then, I see an action_change_header(), but no action_change_body(). So, presume I need to build a new entity by copying headers from the old entity, replacing the Content-Transfer-Encoding header, and replacing the body. Then use replace_entire_message(). Does that sound about right? Also, I've got to believe someone else has needed to do this before. But I'm finding it tough to locate examples. I'm looking at docs for mime::tools, mime::body, etc. but I'm hoping to locate examples of the entire process. Any pointers? Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] head->get('Subject') adds "#012"
Thanks to Bill and Joseph. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] head->get('Subject') adds "#012"
I'm using the following to retrieve the message subject header:
$subj = $entity->head->get('Subject');
First is there a better way?
Second, what I get back always has "#012" added to the end. For example, if
the actual subject is "foo", then the above get() method returns "foo#012".
Why is this extra "#012" appearing? And, is there a way to get just the
subject?
Thanks,
Michael
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] multipart and filenames
I've got a couple of questions about the example filter handling of composite MIME types: /usr/share/doc/mimedefang/examples/suggested-minimum-filter-for-windows-clie nts According to RFC 2045, three composite message types exist: "message", "multipart", or "X-" Question 1: Are all of these composite types handled by the filter_multipart() function? If not, how are the others handled? These composite types include one or more discrete types, such as "text", "image", "audio", etc. My observation is that when a file is attached to a message, it is added as a discrete type, such as "text", "image", "audio", etc., which is handled by the filter() function. So, my observation is that the multipart MIME part never has an associated filename. But the discrete part within the multipart message may have an associated filename. Question 2: In the example file, filter_multipart calls the filter_bad_filenames() function to check for bad filename extensions. I see the utility of this test in the filter() function. But does a multipart message ever really have an associated filename? If so, can you help me understand, perhaps with examples, when filter_multipart() would need to check filenames? Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] return values
> -Original Message- > > I will fix the filter. > > Regards, > > Dianne. Excellent. Thanks. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] return values
Perhaps another couple of newbie questions. But I'm having some difficulty
understanding the return actions from filter() and filter_multipart() as
they appear in the example
/usr/share/doc/mimedefang/examples/suggested-minimum-filter-for-windows-clie
nts.gz.
Issue 1: filter and filter_multipart: return value or not?
/usr/bin/mimedefang.pl does not seem to expect a return value from either
filter() or filter_multipart().
if (defined(&filter_multipart)) {
push_status_tag("In filter_multipart routine");
filter_multipart($in, $fname, $extension, $type);
pop_status_tag();
}
if (defined(&filter)) {
push_status_tag("In filter routine");
filter($in, $fname, $extension, $type);
pop_status_tag();
}
. yet in the example "suggested-minimum-filter.", sometimes they return
nothing and sometimes they return a true/false value returned from another
function.
Returns nothing:
return if message_rejected(); # Avoid unnecessary work
Returns result of another function:
return action_accept();
Shouldn't filter() and filter_multipart() simply use "return;" (with no
return value)? If not, please help me understand.
Issue 2: bounce or discard, which is it?
In the "suggested-minimum-filter." filter function, there is this:
# Block message/partial parts
if (lc($type) eq "message/partial") {
md_graphdefang_log('message/partial');
action_bounce("MIME type message/partial not accepted here");
return action_discard();
}
I'm confused by the "return action_discard()". How can it be both bounced
AND discarded?
Thanks,
Michael
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] best practices for handling filename extensions
> -Original Message- > I am mainly not blocking by filename extensions, but by content. I am > blocking: Thanks Frank. Very helpful ideas. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] suspicious characters
> -Original Message- > suspicious := > If header or body has a \r without \n > If the body has an embedded \0 Jan-Pieter, Steffen, Thanks much. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] best practices for handling filename extensions
I'm looking to understand best practices with regard to rejecting filename extensions. The example provided in /usr/share/doc/mimedefang shows a very long list of extensions to be rejected. I know some hosted mail providers don't allow .exe. It annoys me but I just change the extension and it goes through. And I know that some providers don't allow .zip. So folks using those providers just change it to .piz and it goes through. I presume this is, indeed, a little safer, since the recipient has to take an extra step to change the extension. And, presumably, they would only do that if they knew what they were getting. But I wonder if that's just the appearance of additional security or if it's a true improvement. So, what do the folks here with much more experience than I do, and why? Thanks much, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] suspicious characters
I'm trying to understand what triggers the setting of $SuspiciousCharsInHeaders and $SuspiciousCharsInBody? All I can find are circular definitions that vaguely mention possible exploits. But no specifics are given. Before I use either of these, I'd like to understand better what constitutes "suspicious" in both cases. So, can someone provide a concrete/specific definition of "suspicious" characters in headers? In the body? Also, what do others do? Do you bounce every message that for which $SuspiciousCharsInHeaders is true? How about every message for which $SuspiciousCharsInBody is true? Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] At least I can be a negative example... (was Re: Reload doesnt work)
> I'll do better going forward, I promise. No worries, at least from this newbie. You're one of the few folks I've seen on these e-mail-related lists that takes the time to fully explain answers and provide sufficient detail for so that us newbies actually learn something. So you're still in the black in my book. M ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Reload doesnt work
To get your system to reload the > mimedefang-filter, use the md-mx-ctrl tool: > > md-mx-ctrl reread > > OR if you are not using the mimedefang-multiplexor (which I think > everyone should...) you can keep using the systemctl stop and start > functions while screaming curses at the abomination that is systemd and > its deeply ignorant design. Bill: Since "reload" *IS* something that can be defined in the ExecReload variable in the system unit file, maybe this more of a packaging problem than a systemd problem? Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> It may > simply be that in earlier versions the distribution init scripts' > "restart" command was used directly rather than generating a generic > systemd unit file that synthesizes restart with plain "start" and "stop" > commands. That makes sense. In my own case, I started with MD on Ubuntu 14.04 (MD v2.73). The documentation I found said to use "/etc/init.d/mimedefang reread" after changes to the filter file. But that had no effect (that I could see). So I started using "/etc/init.d/mimedefang restart". Once I got things working, I installed on Ubuntu 16.04 (v2.78). Ubuntu 16.04 uses systemd. So I simply converted to "systemctl restart mimedefang". And that's when I saw the problem caused by no "wait"ing. BTW, in Ubuntu 16.04: -- systemctl reread mimedefang results in "Unknown operation reread" -- systemctl reload mimedefang does run. But I haven't tested to see if it actually picks up the changes to the filter file. I wonder: Is the Debian package maintainer on this list? We seem to be speculating on questions that the packager could answer definitively. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] base64 to text
> >>> I'd like to be able to deliver HTML and Base64 messages to those > >>> clients. > > Honestly, I would de-MIME the whole message, re-format it and replace the > entire message with the new one. Yes, that makes sense. Does there exist a repository of functions written by folks for MD? It sure would help us noobs to not have to re-invent the wheel. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> I'm uneasy telling any author that they're wrong about their own code > but as far as I can tell, that is only conditionally true: Bill, you and Dianne are both right, depending on the version. See my previous message. I provided a diff between v2.78 and v2.79. In v2.79, 'wait' is always passed to stop_it, so the if statement you showed will always evaluate to true But in v2.78, which is where I had my problem, the main part of the script passes $2 to stop_it. So stop_it may or may not see "wait". Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] base64 to text
I have what is probably an unusual problem to solve. I'm willing to do the work to solve it. But as someone new to MIMEDefang in particular and MIME details in general, I want to first get a sanity check to see if what I'd like to do is reasonably possible, and then see if perhaps parts of the problem have already been solved by others. Problem: I have hundreds of old (but necessary) email clients that only understand plain text. They can accept a single-part MIME text/plain message, I think because they just ignore the header. (For the curious, these are amateur radio BBS systems that speak SMTP, but were developed before MIME existed. They still have relevance/usefulness for communications in the field, especially emergency communications over great distances). I'd like to be able to deliver HTML and Base64 messages to those clients. Already Solved: I'm using the built-in remove_redundant_html_parts() function and it works great, where that situation applies. Yet to Be Solved/Can this be Solved?: I'd like be able to deal with text-to-email. For example, Sprint takes a basic ascii-text message, wraps it in html, and then base64-encodes it prior to sending. The recipient receives a single-part mime message with Content-Type of text/html; charset="UTF-8", and Content-Transfer-Encoding of base64. I think what would need to be done is to decode the base64, which would result in plain html (not a multi-part html with redundant plain text), and then strip the extra html. I'd love to be able to do both steps. But even decoding base64 to the underlying html would make the message readable by a human. And, being new to MIMEDefang, I don't know if this type of pipeline processing of a single mime part is viable. It's certainly different than just throwing away redundant parts. So, is this something that's in the realm of reasonable for MIMEDefang to handle? If so, would I need to start from scratch? Or, do at least parts of the solution exists somewhere? Thanks in advance, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> > That way, they could define restart any way they like, such as "stop > > wait" followed by "start". > > Nope. A service definition in a unit file has an ExecStart definition > and maybe an ExecStop definition, but there is no support in systemd for > an ExecRestart attribute. It's a design flaw in systemd, which ALWAYS > implements "systemctl restart" as a stop then a start. Ah. I guess I was thinking of reload. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
> Dianne will likely give a definitive authoritative answer but having > adapted the MD startup script to multiple system control environments, I > don't see anything especially risky. The only potential issue I can > think of is if you're trying to shut down or reboot in a pathological > circumstance where the mimedefang.pl worker perl sub-processes refuse to > exit and the supervisory process (e.g. init) has no last resort > mechanism to arbitrarily slaughter hung children. There's nothing quite > so annoying as a "hung going down" system. Thanks Bill, That makes sense. One option would be for the MIMEDefang team to provide the necessary systemd control files (whatever they're called). That way, they could define restart any way they like, such as "stop wait" followed by "start". I know systemd is hated by many. But it is what Ubuntu 16.04 LTS uses. Without the proper files, apparently what happens is systemd auto-generates the file(s) it needs from the init script. And in the autogenerated file, a 'restart' is simply a 'stop' followed by a 'start'. I guess they can depend on at least those two basic commands being in every init script. And because 'stop' doesn't wait unless the added "wait" option is given, we have the problem. A friend of mine found the actual "fix" that was submitted. It's a bit different than what I did. Instead of commenting out the if statement in stop_it() like I did, they changed the line that calls stop_it from "stop_ip $2" to "stop_it wait", so stop_it is always called with the "wait" option. See below. Same result - the (formerly conditional) wait part always runs. 1 line changed instead of 2. Michael *** mimedefang-16.042017-09-23 17:53:51.046863700 -0700 --- mimedefang-17.042017-09-23 17:50:18.456864400 -0700 *** *** 412,418 ;; stop) ! stop_it $2 ;; restart|force-reload) --- 412,418 ;; stop) ! stop_it wait ;; restart|force-reload) ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
Thanks Bill, After some more research I found the bug and I think I've "fixed" it. See below. > > Option 1: Use inet socket > > > > /etc/default/mimedefang: > > SOCKET=inet:8899@localhost > > > > /etc/postfix/main.cf > > smtpd_milters = inet:localhost:8899 > > > > This works for Ubuntu 14.04/MIMEDefang 2.73. > > But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already > > running, and MIMEDefang is restarted, I get the following in > > /var/log/mail.err: > > > > mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port > > inet:8899@localhost: Address already in use > > mimedefang[27605]: MIMEDefang-2.78: Unable to create listening > > socket on conn inet:8899@localhost > > That's odd. Given your other options, I'd try to figure out what exactly > is going on here and fix it. I.e. what process actually has taken that > port. Yes, odd. I've continued to research and just found this, which is my problem. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807078 The problem occurs for me in MIMEDefang 2.78 (package for Ubuntu 16.04), which is the version reported in the above bug. Apparently, the problem is that when using "systemctl restart mimedefang" it doesn't wait for itself to completely stop before it tries to start again. And I can confirm that. If I use "systemctl stop ...", then wait for it to stop, then "systemctl start ...", then it works. The bug report says it's fixed in v2.79 and v2.79 is in the Ubuntu 17.04 package. According to https://ubuntu.pkgs.org/17.04/ubuntu-universe-i386/mimedefang_2.79-2_i386.deb.html , on 2016-12-23 the init stop action was changed to always wait. So here's what I did to "fix" it in Ubuntu 16.04/MIMEDefang 2.78: In /etc/init.d/mimedefang, in the stop_it() function: #if [ "$1" = "wait" ] ; then #fi In other words, just comment out the conditional wait so that the "waiting" part inside the IF statement always runs. When I tried to run systemctl restart mimedefang, I got: Warning: mimedefang.service changed on disk. Run 'systemctl daemon-reload' to reload units. So, I did that. And then: systemctl restart mimedefang seems to work just fine. I'd appreciate it if any of the developers can verify that what I did shouldn't cause some other unintended consequence. Thanks much, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] REVISED: postfix/mimedefang socket
Thanks Richard, BTW, I also installed from packages. So, with a unix socket you still have the same problem I do with the inet socket, which is: the order of start/restart matters and can break things. That's not good. This seems like a clear bug in MIMEDefang. Evidence: 1) Other milter(s) share a socket with Postfix and don't care which order they are started/restarted (example: OpenDKIM) 2) Dovecot shares a socket with Postfix and doesn't care which order they are started/restarted 3) Amavisd-new is a different mechanism, but it also doesn't care which order it is started/restarted 4) And even MIMEDefang didn't care about the order in v2.73 I'm new to the list. What's the process for reporting/resolving issues? Thanks, Michael > -Original Message- > From: Richard Laager [mailto:rlaa...@wiktel.com] > Sent: Friday, September 22, 2017 9:51 PM > To: Michael Fox > Cc: mimedefang@lists.roaringpenguin.com > Subject: Re: [Mimedefang] REVISED: postfix/mimedefang socket > > On 09/22/2017 12:47 PM, Michael Fox wrote: > > Option 3: Use unix socket in Postfix chroot jail > > This looks to be what I do. I'm running Postfix and MIMEDefang on > Ubuntu, both from packages. Postfix runs as the postfix user, and > there's a defang group. I run Postfix in a chroot. > > These appear to be the relevant parts of my install script: > > adduser --quiet postfix defang > > install -d -o defang -g defang -m 750 \ > /var/spool/postfix/var/spool/MIMEDefang > > chown -R defang:defang \ > /var/lib/MIMEDefang \ > /var/spool/MIMEDefang \ > /var/spool/postfix/var/spool/MIMEDefang > > sed -i 's|^\(# > \)\?\(SOCKET\)=.*|\2=/var/spool/postfix/var/spool/MIMEDefang/mimedefang.so > ck|' \ > /etc/default/mimedefang > > I believe we have some sort of trouble if one of the daemons is > restarted, but not the other, or if it's done in the wrong order or > something. I don't have a lot of specifics off the top of my head. In > practice, we hardly ever restart one or the other. It's usually either > stopping both (and starting MIMEDefang first, to give slaves a chance to > spin up), or rebooting the server. > > I hope this helps. If you have specific questions, I'll try to dig into > my config if I can. I'm currently out of the office, though. > > -- > Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] REVISED: postfix/mimedefang socket
Sorry if this has been asked/answered before. I've searched and searched and found no consistent, complete answers. What are the steps to configure MIMEDefang and Postix to share a socket? Here's what I tried: Option 1: Use inet socket /etc/default/mimedefang: SOCKET=inet:8899@localhost /etc/postfix/main.cf smtpd_milters = inet:localhost:8899 This works for Ubuntu 14.04/MIMEDefang 2.73. But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already running, and MIMEDefang is restarted, I get the following in /var/log/mail.err: mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port inet:8899@localhost: Address already in use mimedefang[27605]: MIMEDefang-2.78: Unable to create listening socket on conn inet:8899@localhost Option 2: Use unix socket in MIMEDefang directory -- /etc/default/mimedefang: MD_ALLOW_GROUP_ACCESS=yes add user "postfix" to group "defang" usermod -a -G defang postfix /etc/postfix/main.cf: smtpd_milters=unix:/var/spool/MIMEDefang/mimedefang.sock This fails, presumably because Postfix is chroot'ed and can't access anything outside it's chroot directory (/var/spool/postfix). In /var/log/mail.err: postfix/smtpd[31110]: warning: connect to Milter service unix:/var/spool/MIMEDefang/mimedefang.sock: No such file or directory Option 3: Use unix socket in Postfix chroot jail - I suppose I could configure /etc/default/mimedefang:SOCKET to point to a file in /var/spool/postfix/private, where the other sockets are located. But that directory is owned by postfix, group=root, permissions 0700. So I don't know how mimedefang could write to it. So, I'm stuck. Any help would be appreciated. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] postfix/mimedefang socket
Sorry if this has been asked/answered before. I've searched and searched and found no consistent, complete answers. In order for MIMEDefang and Postfix to share a unix socket, I think I need to do the following: /etc/default/mimedefang: MD_ALLOW_GROUP_ACCESS=yes add user "postfix" to group "defang" /etc/postfix/main.cf: smtpd_milters=unix:/var/spool/MIMEDefang/mimedefang.sock Is that right? Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] remove_redundant_html > delete_header?
> If you have Mimedefang change the message in any way, you should > remove the DKIM headers. OK. Thanks. > Since you are checking DKIM with a separate milter, you can configure > Spamassassin not to check it. Yeah, I was hoping for a way to not throw away the DKIM check as a contributing factor to the spam score. But it seems the only way to do that would be to run SpamAssassin from MIMEDefang. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] remove_redundant_html > delete_header?
Im new to the list so be gentle! Im using Postfix with the following milters/filters in this order: * pre-queue milters: opendkim, mimedefang (I plan to add opendmarc after opendkim) * post-queue content filter: amavisd-new (with spamassassin, clamd) I have an old client that doesnt understand HTML. So, Im using remove_redundant_html_parts(). And that works great. But by changing the message, the DKIM-Signature is no longer going to match. Since OpenDKIM runs before MIMEDefang, it is still able to validate the signature. So far, so good. But amavis/spamassassin runs after mimedefang. So Im wondering if I should take further action so that spamassassin doesnt see a mismatched signature. Specifically: If remove_redundant_html_parts() returns true (it removed HTML parts), then should I also delete the DKIM-Signature header? And, if I remove the DKIM-Signature header, should I also remove the Authentication-Results: header? In other words, whats the best method for keeping all of the players happy? Any thoughts/wisdom would be helpful. Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] newbie prerequisite install questions
> On 21 Nov 2016, at 13:04, Michael Fox wrote: [details clipped] > Yes. *ALL* of those are Distribution names, which is why they don't use > the :: module delimiter. You should be able to get them all using CPAN. Thanks Bill. That's exactly what I needed. Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] newbie prerequisite install questions
Thanks Mickey. That doesn't appear to be a "module", but rather an archive of multiple modules, right? (I'm new to CPAN, too). So, is the instruction to download, unpack and install this archive, using sudo? And, for my other question, should I install the other modules listed on the Download page using (sudo) CPAN? Michael > -Original Message- > From: MIMEDefang [mailto:mimedefang-boun...@lists.roaringpenguin.com] On > Behalf Of Mickey Hill > Sent: Monday, November 21, 2016 9:23 AM > To: MIMEDefang Mailing List > Subject: Re: [Mimedefang] newbie prerequisite install questions > > On Nov 21, 2016, at 9:16 AM, Michael Fox wrote: > > Then “MailTools-1” links to http://search.cpan.org/search?q=MailTools, > which has 28 pages of modules, none of which seem to be named “MailTools” > or “Mail::Tools”. I just don’t know what I’m supposed to do there. > Please explain. > > Change your CPAN search to Distributions only. > > http://search.cpan.org/search?mode=dist&query=MailTools > > That gives you http://search.cpan.org/~markov/MailTools-2.18/ > > -- > Mickey Hill > > > ___ > NOTE: If there is a disclaimer or other legal boilerplate in the above > message, it is NULL AND VOID. You may ignore it. > > Visit http://www.mimedefang.org and http://www.roaringpenguin.com > MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com > http://lists.roaringpenguin.com/mailman/listinfo/mimedefang ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] newbie prerequisite install questions
Im new to MIMEdefang and could use some more specific instructions on installation. Sorry for the many nit questions. But I want to start off right and Im just not finding clear installation instructions. I read http://www.mimedefang.org/static/mimedefang-lisa04.pdf, slide 30, and it says only Make sure you have indicated prerequisites installed. It doesnt say what those are. The download page lists Perl Modules from CPAN. But it doesnt say they are prerequisites either. Are those the prerequisites? The Perl modules listed on the download page dont have the usual :: format. So I could use some clearer instructions. For most, it looks like I should probably just replace the - with :: and then use my CPAN client to install that module: MIME-tools: Use CPAN client to install MIME::Tools IO-stringy: Use CPAN client to install IO::Stringy MIME-Base64:Use CPAN client to install MIME::Base64 Digest-SHA1:Use CPAN client to install Digest::SHA1 Unix-Syslog:Use CPAN client to install Unix::Syslog Net-DNSBL-Client: Use CPAN client to install Net::DNSBL::Client Is that correct? Then MailTools-1 links to http://search.cpan.org/search?q=MailTools, which has 28 pages of modules, none of which seem to be named MailTools or Mail::Tools. I just dont know what Im supposed to do there. Please explain. Thanks, Michael ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
