Re: [Mimedefang] base64 to quoted-printable

[Michael Fox]

> -Original Message-
> } Unfortunately, it's an amateur radio BBS written in the 80s that
> 
>  Which app is it?

JNOS
http://www.langelaar.net/projects/jnos2/


> } only understands plain text.  It was pretty advanced in its time
> 
>  Something this old may not understand quoted-printable.

It doesn't really understand MIME.  So it spits out quoted-printable as the 
text that it is.

 
>  I have a similar requirement to modify a message where I'm
> using action_external_filter().  My filter() function looks like
> this:

Excellent.  Examples of similar actions (even if they're not directly 
applicable) are a real help to a MIMEDefang and Perl newbie.


> 
> There should probably be some more error checking such as making
> sure that action_external_filter() succeeds.  Perhaps if you look
> at the definition of action_external_filter() in mimedefang.pl it
> will give you some ideas.  It's a fairly simple function (most of
> it is error checking).
> 
> VA7JMN

Thanks.  I'll do that.

Michael
N6MEF


___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[John Nemeth]

On Oct 11, 11:54pm, "Michael Fox" wrote:
} 
} > -Original Message-
} > If the old client that you are accommodating is running on (or even just
} > displaying on) a reasonably modern OS, there's a strong chance that
} > you'll actually do better for readability by using UTF-8 as the
} > character set and "8bit" encoding instead of QP. Many (but not all)
} > older mail clients don't really know MIME but do happily pass through
} > all 8 bits of every byte to their output, whether it's a terminal driver
} > or a GUI API.
} 
} Thanks Bill.
} 
} Unfortunately, it's an amateur radio BBS written in the 80s that

 Which app is it?

} only understands plain text.  It was pretty advanced in its time

 Something this old may not understand quoted-printable.

} in that it uses SMTP internally.  But no MIME, no UTF-8, etc.
} It's still extremely useful and widely used, especially in
} emergency comms, delivering mail over VHF radio where other
} connectivity isn't available.

 I have a similar requirement to modify a message where I'm
using action_external_filter().  My filter() function looks like
this:

sub filter () {
my($entity, $fname, $ext, $type) = @_;

return if message_rejected(); # Avoid unnecessary work

if ($fname eq "Cisco_fax.tif") {
#  convert incoming fax to pdf
md_syslog('info', 'converting tiff to pdf');
action_external_filter($entity, "/usr/pkg/bin/tiff2pdf -o 
./FILTEROUTPUT ./FILTERINPUT");
my $head = $entity->head;
$head->replace('content-type', 'application/pdf; name="Cisco_fax.pdf"');
}

return action_accept();
}

There should probably be some more error checking such as making
sure that action_external_filter() succeeds.  Perhaps if you look
at the definition of action_external_filter() in mimedefang.pl it
will give you some ideas.  It's a fairly simple function (most of
it is error checking).

VA7JMN

Note that I've only had licence for a couple of years and am still
getting my feet wet, but I'm definitely interested in data and
emergency comms.  I've recently started playing with RMS Express
(over telnet).  The next thing I want to do is get a TNC.

}-- End of excerpt from "Michael Fox"
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Michael Fox]

> -Original Message-
> But you have a tool here that is flexible enough to do it.

Yes, indeed.  

 
> I suppose you want to take the text/plain MIME part and remove any
> other parts. Others would be just an HTML alternative or something
> that is not text.

Yup.  And I'm doing that already, and adding a footer to tell the recipient 
what was removed.  The case I'm trying to solve is when the text/plain or 
text/html part has Content-Transfer-Encoding of Base64.


> If you happen to know that the original was always plain ascii with
> normal line lengths (80 or less) then I'd tempted to say just go from
> base64 to ascii.

Yup.  And "groff -T ascii" has been suggested, just in case there are emoji's 
or other things to squash.  But the quoted-printable suggestion would be less 
destructive so I'm trying that out first.  Either way, 7bit or 
quoted-printable, I apparently need to more carefully parse the MIME structure 
and ultimately rebuild the message.  Hence, my questions.

 
> It does make me wonder why it is base64 at all then, but I've been
> seeing mail from Exchange that is always base64, so maybe that is what
> you are dealing with.

In most cases, it's the carrier or email service provider or sender's client 
app and it's often not something the sending user can control.  So rejecting 
the message results in a complete loss of communications.

Example: send a simple text message like "Hello" from Sprint to an email 
address and they wrap it in HTML and encode it with Base64.

Sending from Verizon, a simple text message arrives as 7bit plain text.  A 
short text, like "Hello", with one emoji arrives as quoted-printable.  A short 
text message with multiple emoji's arrives as base64.  A slightly longer text 
message, like "Me and the kids are safe at Mom's house.  Call when you can.", 
with multiple emoji's arrives as an attachment.  

Different things also happen from smart phone email apps.  So, "there be 
dragons".  And ultimately, this may be beyond my ability to solve.  But I'm not 
ready to give up yet.  If I can convert at least some messages that otherwise 
would not be readable, it's a win.

Michael
a.k.a Don Quixote de La Mancha



___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Michael Fox]

> -Original Message-
> I won't give the actual Perl code, but here's a basic sketch, completely
> untested.  I really have no idea whether or not this will work, but I
> think the basic approach is correct. :)

Thanks Dianne.  That definitely points me in the right direction.  Much 
appreciated.

Michael




___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Joseph Brennan]

On Fri, Oct 13, 2017 at 9:53 AM, Michael Fox  wrote:
>> -Original Message-
>>
>> >Unfortunately, it's an amateur radio BBS written in the 80s that only
>> understands plain text.
>> > It's still extremely useful and widely used, especially in emergency
>> comms


Interesting case. Most likely none of us have needed to do this type
of conversion. But you have a tool here that is flexible enough to do
it.

I suppose you want to take the text/plain MIME part and remove any
other parts. Others would be just an HTML alternative or something
that is not text.

If you happen to know that the original was always plain ascii with
normal line lengths (80 or less) then I'd tempted to say just go from
base64 to ascii.

It does make me wonder why it is base64 at all then, but I've been
seeing mail from Exchange that is always base64, so maybe that is what
you are dealing with.

Joe Brennan

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Dianne Skoll]

On Fri, 13 Oct 2017 10:20:04 -0400
Dianne Skoll  wrote:

>   # non-recursive case: Single part.
>   # If $in_entity is base-64 encoded, make a new $new_entity
>   # that is qp-encoded and call $out_entity->attach($new_entity)
>   # and return 1 to indicate that a change was made.
>   # Otherwise, call $out_entity->attach($in_entity)
>   # and return 0.

Oh, sorry.  ->attach should be ->add_part.

Regards,

Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Dianne Skoll]

On Fri, 13 Oct 2017 06:53:33 -0700
"Michael Fox"  wrote:

> While I appreciate everyone's attempt to warn me that "there be
> dragons", the facts are that the client is what it is, it serves
> hundreds of people (in my location alone), and the problem needs
> solving.  

OK.  You will most likely have to do all your changes in
filter_end.  You'll need to traverse the MIME tree, looking for
text/* parts.  If any are Base-64 encoded, you'll need to load them in
and re-encode them using Quoted-Printable.  As you do this, you need
to build up a *new* replacement MIME::Entity.  Finally, call
replace_entire_message with your new entity as the argument.

I won't give the actual Perl code, but here's a basic sketch, completely
untested.  I really have no idea whether or not this will work, but I
think the basic approach is correct. :)

sub re_encode
{
my ($in_entity, $out_entity, $did_something) = @_;

# If it's multipart, recurse
if ($in_entity->is_multipart) {
my $new_entity = MIME::Entity->build(Type => $in_entity->type);
$out_entity->add_part($new_entity);
foreach my $p (@{$in_entity->parts}) {
if (re_encode($p, $new_entity, $did_something)) {
$did_something = 1;
}
}
return $did_something;
}

# non-recursive case: Single part.
# If $in_entity is base-64 encoded, make a new $new_entity
# that is qp-encoded and call $out_entity->attach($new_entity)
# and return 1 to indicate that a change was made.
# Otherwise, call $out_entity->attach($in_entity)
# and return 0.
}

sub filter_end
{
my ($entity) = @_;
my $replacement_entity = MIME::Entity->build(Type => 'multipart/mixed');
my $changes_made = re_encode($entity, $replacement_entity, 0);
if ($changes_made) {
# Recursively remove single-part "multiparts" starting
# from top-level.
while ($replacement_entity->is_multipart && 
scalar(@{$replacement_entity->parts}) == 1) {
$replacement_entity = $replacement_entity->parts(0);
}
replace_entire_message($replacement_entity);
}
}

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Frank Doepper]

Am 11.10.17 um 23:54 schrieb Michael Fox:

> Unfortunately, it's an amateur radio BBS written in the 80s that only
> understands plain text.  It was pretty advanced in its time in that it
> uses SMTP internally.  But no MIME, no UTF-8, etc.  It's still extremely
> useful and widely used, especially in emergency comms, delivering mail
> over VHF radio where other connectivity isn't available.

I'd suggest
http://search.cpan.org/~sburke/Text-Unidecode-1.30/lib/Text/Unidecode.pm
and doing it in the receiver's procmailrc if there is any.

Best regards
Frank
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Michael Fox]

> -Original Message-
> 
> >Unfortunately, it's an amateur radio BBS written in the 80s that only
> understands plain text.
> > It's still extremely useful and widely used, especially in emergency
> comms
> 
> So worthwhile investing some cash and getting it updated so that it can
> cope with the modern world?  [clipped]

While I appreciate everyone's attempt to warn me that "there be dragons", the 
facts are that the client is what it is, it serves hundreds of people (in my 
location alone), and the problem needs solving.  

And, while I may be a newbie to MIME and MIMEDefang (and Perl, for that 
matter), I'm definitely NOT a newbie in most other respects.  I've been doing 
network engineering since the days of the NSFnet and I'm pretty certain I 
understand this particular application need better than anyone else here.

So, going forward, I'm hopeful I can get advice and help on MIME and MIMEDefang 
here.  And warnings about dangers of this or that detail or approach are, of 
course, very helpful.  But suggesting that the client be changed doesn't really 
answer the questions that I'm asking.

Paul, Dianne: regarding readability of quoted-printable:  again, I'm a newbie 
to MIMEDefang, not the world.  So I understand and appreciate the warning but I 
still have to solve the problem.  The incoming messages were originated as 
plain ASCII text.  So, the quoted-printable results I'm seeing aren't hard to 
read at all.  But if worse comes to worse, I may end up squashing it to plain 
ASCII text.  We'll see.

Michael



___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Paul Murphy]

>Unfortunately, it's an amateur radio BBS written in the 80s that only 
>understands plain text.
> It's still extremely useful and widely used, especially in emergency comms

So worthwhile investing some cash and getting it updated so that it can cope 
with the modern world?  And presumably the receiving equipment is 8-bit capable 
and not also something from the 80s?

As Dianne has already pointed out to you, converting everything to 
Quoted-Printable doesn't necessarily make it more readable.  In the most 
extreme case, you'll encode spaces as well as extended characters, and 
especially when you're trying to pass addresses or telephone numbers then this 
could turn out to be problematic  - try this one out:

We're=20off=20to=20the=20seaside,=20where=20the=20temperature=3D20=C2=B0C=20:-)=20=20El=20Ni=C3=B1o=20has=20some=20benefits!=20=20Call=20me=20on=20(+44)=20123=20456=207890

Paste it into http://www.webatic.com/run/convert/qp.php if you're struggling.

Now tell me that in an emergency situation you can extract the phone number 
quickly and without errors, and that you got the temperature from that message. 
 If an address has Spanish, French, German, etc accented characters, they'll 
also be encoded rather than converted to the nearest unaccented character, so 
in the example above " El Niño" becomes "El=20Ni=C3=B1o" rather than "El Nino". 
 

Eventually this will come back to bite you.  Time to bring it at least into the 
90s, if not into the 21st century...

Paul.


___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Michael Fox]

> -Original Message-
> If the old client that you are accommodating is running on (or even just
> displaying on) a reasonably modern OS, there's a strong chance that
> you'll actually do better for readability by using UTF-8 as the
> character set and "8bit" encoding instead of QP. Many (but not all)
> older mail clients don't really know MIME but do happily pass through
> all 8 bits of every byte to their output, whether it's a terminal driver
> or a GUI API.

Thanks Bill.

Unfortunately, it's an amateur radio BBS written in the 80s that only 
understands plain text.  It was pretty advanced in its time in that it uses 
SMTP internally.  But no MIME, no UTF-8, etc.  It's still extremely useful and 
widely used, especially in emergency comms, delivering mail over VHF radio 
where other connectivity isn't available.

Michael



___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Bill Cole]

On 10 Oct 2017, at 18:24, Michael Fox wrote:

Then, I see an action_change_header(), but no action_change_body().  
So,
presume I need to build a new entity by copying headers from the old 
entity,
replacing the Content-Transfer-Encoding header, and replacing the 
body.

Then use replace_entire_message().

Does that sound about right?


Yes. For what you are determined to do, that would be the way. Note that 
if the message body has been cryptographically signed in any way (DKIM, 
OpenPGP, S/MIME, whatever) your replacement of the message will 
invalidate the signature.


Also, I've got to believe someone else has needed to do this before.  
But

I'm finding it tough to locate examples.


Faith is an amazing thing... :)

The irony is that this was more likely to be needed in the early 90's, 
when there were many clients that could handle QP and not Base64, but 
there wasn't any comprehensive toolkit for manipulating MIME messages. 
By the time the tools got really good, it was no longer much needed or 
generally safe to do the conversion to accommodate mail clients.


If the old client that you are accommodating is running on (or even just 
displaying on) a reasonably modern OS, there's a strong chance that 
you'll actually do better for readability by using UTF-8 as the 
character set and "8bit" encoding instead of QP. Many (but not all) 
older mail clients don't really know MIME but do happily pass through 
all 8 bits of every byte to their output, whether it's a terminal driver 
or a GUI API.

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Michael Fox]

> On Tue, 10 Oct 2017 16:11:07 -0400
> Joseph Brennan  wrote:
> 
> > On Tue, Oct 10, 2017 at 1:48 PM, Michael Fox  wrote:
> > > I need to convert base64 to something readable.
> 
> > To be honest I can't think of a reason to do this. 

Nevertheless, I need to for an old client.

So, back to my question:

if ($bh = $entity->bodyhandle) ;

$qp_body = encode_qp($bh->as_string)

}

It seems to work (I use md_syslog() to show the first 20 chars).  Is that
right?  Or is there a better way? 

Then, I see an action_change_header(), but no action_change_body().  So,
presume I need to build a new entity by copying headers from the old entity,
replacing the Content-Transfer-Encoding header, and replacing the body.
Then use replace_entire_message().

Does that sound about right?

Michael


___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Dianne Skoll]

On Tue, 10 Oct 2017 16:11:07 -0400
Joseph Brennan  wrote:

> On Tue, Oct 10, 2017 at 1:48 PM, Michael Fox  wrote:
> > I need to convert base64 to something readable.

> To be honest I can't think of a reason to do this. Any client from
> this century can read base64. Mimedefang and Spamassassin can read it
> too, so searching content just works.

And QP is not necessarily any more readable than B64, depending
on the source language.

Is =D7=A2=D7=91=D7=A8=D7=99=D7=AA=E2=80=8E any more readable than
16LXkdeo15nXquKAjg== ?

:)

Regards,

Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] base64 to quoted-printable

[Joseph Brennan]

On Tue, Oct 10, 2017 at 1:48 PM, Michael Fox  wrote:
> I need to convert base64 to something readable.
>
> Also, I've got to believe someone else has needed to do this before.


To be honest I can't think of a reason to do this. Any client from
this century can read base64. Mimedefang and Spamassassin can read it
too, so searching content just works.

(You may have explained a special case, and if so, I'm sorry I forgot.)


Joseph Brennan

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

[Mimedefang] base64 to quoted-printable

[Michael Fox]

I need to convert base64 to something readable.  Per previous suggestions on
this list, I'm going to convert it to quoted-printable.  So far, I'm using:

 

if ($bh = $entity->bodyhandle) ;

$qp_body = encode_qp($bh->as_string)

}

 

It seems to work (I use md_syslog() to show the first 20 chars).  Is that
right?  Or is there a better way? 

 

Then, I see an action_change_header(), but no action_change_body().  So,
presume I need to build a new entity by copying headers from the old entity,
replacing the Content-Transfer-Encoding header, and replacing the body.
Then use replace_entire_message().

 

Does that sound about right?

 

Also, I've got to believe someone else has needed to do this before.  But
I'm finding it tough to locate examples.  I'm looking at docs for
mime::tools, mime::body, etc. but I'm hoping to locate examples of the
entire process.  Any pointers?

 

Thanks,

Michael

 


___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang