Re: [Mimedefang] utf-8 issue?
I was signed up for that ... but Dianne Skoll sold her company and I ended up here. I have no idea id "the remind list" still works or not. I am not interested in what is going on here ... just "remind" > > What does all this talk about spam and anti virus stuff have ro do with > > the program remind? > > Nothing. But this is a mailing list about the email filtering tool > MIMEDefang. Remind is a different tool written by the same author. Remind > has its own mailing list at > https://dianne.skoll.ca/mailman/listinfo/remind-fans ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
> What does all this talk about spam and anti virus stuff have ro do with the > program remind? Nothing. But this is a mailing list about the email filtering tool MIMEDefang. Remind is a different tool written by the same author. Remind has its own mailing list at https://dianne.skoll.ca/mailman/listinfo/remind-fans Confidentiality Notice: This electronic message and any attachments may contain confidential or privileged information, and is intended only for the individual or entity identified above as the addressee. If you are not the addressee (or the employee or agent responsible to deliver it to the addressee), or if this message has been addressed to you in error, you are hereby notified that you may not copy, forward, disclose or use any part of this message or any attachments. Please notify the sender immediately by return e-mail or telephone and delete this message from your system. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
Hi > What does all this talk about spam and anti virus stuff have ro do with the > program remind? Sorry, if this seemed off-topic. MIMEDefang makes heavy use of MIME::Tools to unpack MIME Parts and extract files and filenames. So I supposed the problem described could have the same issue as the one I encountered and could be fixed within the affected mimedefang-filter with the code snippled I supplied. Mit freundlichen Grüssen -Benoît Panizzon- -- I m p r o W a r e A G-Leiter Commerce Kunden __ Zurlindenstrasse 29 Tel +41 61 826 93 00 CH-4133 PrattelnFax +41 61 826 93 01 Schweiz Web http://www.imp.ch __ ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
What does all this talk about spam and anti virus stuff have ro do with the program remind? ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
Hi
I suppose I came over a very similar issue while building an spamtrap
which extracts attachements with MIME::Parser and feeds them down the
throat of clamav.
It boils more or down how perl threat UTF-8 strings on file functions
and how posix systems don't care about the encoding of filenames
What I found out to work with try and error, still not sure why:
use utf8;
use open ':std', ':encoding(UTF-8)';
[...]
my $filename = $part->head->recommended_filename;
open(SPAMFILE,">$spamdir/$filename");
print SPAMFILE $attachment;
close(SPAMFILE);
if (utf8::is_utf8($filename)) {
print "Filename IS UTF8, encode Filename for AV\n";
utf8::encode($filename); # This encodes UTF-8 Perl Strings to binary
string.
}
my ($path,$found) = $av->scan_path("$spamdir/$filename"); # Now Umlauts and
similar characters in the filename do not cause the file not to be found.
Mit freundlichen Grüssen
-Benoît Panizzon-
--
I m p r o W a r e A G-Leiter Commerce Kunden
__
Zurlindenstrasse 29 Tel +41 61 826 93 00
CH-4133 PrattelnFax +41 61 826 93 01
Schweiz Web http://www.imp.ch
__
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On 29 Jan 2020, at 8:54, Alexey Lobanov wrote: Hi all. I ask you very much to continue the thread started in 2017. The problem persists, and it is really annoying. In brief: Mimedefang dies while message checking if *all *of following conditions happen: 1. Attachment extension is "bad" and the attachment should be quarantined. 2. Attachment file name is non-ASCII and so why is encoded in UTF-8. 3. UTF-8 header encoding in original message is "incorrect" in some way. Not sure what it means; many other Unicode filenames are processed and quarantined properly. I have supplied example strings to the maillist in 2018, and can add more examples. I'm unable to reproduce this with those scraps. If you have (or can construct) a full message which reproduces the problem, please provide it. I see that Dianne acknowledged the issue (https://lists.roaringpenguin.com/pipermail/mimedefang/2018-June/038300.html) but it was not fixed in 2.84 and I'm not sure exactly what the fix might be. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
Hi all.
I ask you very much to continue the thread started in 2017. The problem
persists, and it is really annoying.
In brief: Mimedefang dies while message checking if *all *of following
conditions happen:
1. Attachment extension is "bad" and the attachment should be quarantined.
2. Attachment file name is non-ASCII and so why is encoded in UTF-8.
3. UTF-8 header encoding in original message is "incorrect" in some way.
Not sure what it means; many other Unicode filenames are processed and
quarantined properly.
I have supplied example strings to the maillist in 2018, and can add
more examples. Fresh one:
Jan 29 13:13:29 mimedefang.pl[24376]: ACE0720490:
MDLOG,ACE0720490,bad_filename,%42D%426%41F.rar,application/x-rar,,,=?UTF-8?B?0K3QptCf?=
Jan 29 13:13:29 mimedefang-multiplexor[27228]: ACE0720490: Slave 1
stderr: open body: Invalid argument at /usr/share/perl5/MIME/Entity.pm
line 1892.
Jan 29 13:13:29 mimedefang-multiplexor[27228]: Slave 1 died prematurely
-- check your filter rules
Jan 29 13:13:29 postfix/cleanup[3830]: ACE0720490: milter-reject:
END-OF-MESSAGE from f558.i.mail.ru[94.100.184.166]: 4.7.1 Service
unavailable - try again later;
Any ideas or workarounds? Changing mime-tools version changes line
number in Entity.pm, and nothing more.
Older examples are below.
Alexey
On 25.06.2018 17:17, Alexey Lobanov wrote:
> Hi all.
>
> I continue the years old thread related to improper processing of
> Unicode strings either in Mimedefang or Perl libs. Last message
> https://lists.roaringpenguin.com/pipermail/mimedefang/2017-December/038247.html
>
> As far as I can see, the exception happens only when Mimedefang tries
> to remove an attachment with non-ASCII (e.g, Russian) name. If
> attachment is permitted and passed, exception does not appear. Looks
> like also, that other Russian names in same message do not generate an
> exception; not sure in exception logging scheme. Please note also that
> about 50% of attachments in our environment have Cyrillic names in
> UTF-8, but the issue happens few times a year only.
>
> The example log ("use diagnostics;" in mimedefang.pl) is below.
>
> The visible names of attachments are: "Ат. профессора.rar", "ВАК
> дмн.rar", "Диплом БДМУ 1982.rar" and "Категория 2016.rar".
>
> The MIME headers for first attachment are:
>
> Content-Type: application/octet-stream;
> name="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LAucmFy?="
> Content-Disposition: attachment;
> filename="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LA=?=
> =?UTF-8?B?LnJhcg==?="
> Content-Transfer-Encoding: base64
> X-Attachment-Id: f_jimx17ff8
>
> Debian Jessie, Mimedefang 2.83 built from source, no embedded Perl
> mode, Mime-Tools 5.009, Perl v5.20.2.
>
> Yes, I have the whole message and I can provide any other
> non-confidential details from it.
>
>
>
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%410%442.
> %43F%440%43E%444%435%441%441%43E%440%430.rar,application/octet-stream,,,Re:
> LEC
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: Strings with code points over 0xFF may not be mapped into
> in-memory file
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: handles (#1)
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: (W utf8) You tried to open a reference to a scalar for
> read or append
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: where the scalar contained code points over 0xFF.
> In-memory files
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: model on-disk files and can only contain bytes.
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr:
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%412%410%41A
> %434%43C%43D.rar,application/octet-stream,,,Re:
> LEC
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%414%438%43F%43B%43E%43C
> %411%414%41C%423
> 1982.rar,application/octet-stream,,,Re: LEC
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%41A%430%442%435%433%43E%440%438%44F
> 2016.rar,application/octet-stream,,,Re: LEC
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,mail_in,Re: LEC
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: Uncaught exception from user code:
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: open body: Invalid argument at
> /usr/share/perl5/MIME/Entity.pm line 1892.
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr:
> MIME::Entity::print_bodyhandle(MIME::Entity=HASH(0xa10a63c),
> GLOB(0x9b07fe8)) called at /usr/share/perl5/MIME/Entity.pm line 1873
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6
Re: [Mimedefang] utf-8 issue?
On Thu, 28 Jun 2018 10:57:06 +0300 Alexey Lobanov wrote: > It is default code with extended extension list. Trojan writers like > these file types novadays. Ah, OK, the re_match routine in MIMEDefang won't handle wide characters properly. I'll look into fixing it for a future release. I can't promise a delivery date as I'm currently recovering from major surgery and don't have the energy to do much work in a given day. :( Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
Hi.
On 28.06.2018 1:51, Dianne Skoll wrote:
Hi,
Can you post the relevant part of your filter code that triggers
this problem?
It is default code with extended extension list. Trojan writers like
these file types novadays.
==
sub filter_bad_filename {
my($entity) = @_;
my($bad_exts, $re);
# Bad extensions
$bad_exts =
'(arj|rar|r15|lzh|uue|7z|xz|xlsm|epf|docm|dotm|dot|tgz|ace|z|gz|ace|jar|ade|adp|app|asd|asf|asx|bas|bat|cab|chm|cmd|com|cpl|dll|exe|fxp|hlp|hta|hto|inf|ini|ins|isp|jse?|lib|lnk|mdb|mde|msc|msi|msp|mst|ocx|pcd|pif|prg|reg|scr|sct|sh|shb|shs|sys|vb|vbe|vbs|vxd|wsc|wsf|wsh|\{[^\}]+\})';
# Do not allow:
# - CLSIDs {foobarbaz}
# - bad extensions (possibly with trailing dots) at end
$re = '\.' . $bad_exts . '\.*$';
return 1 if (re_match($entity, $re));
# Look inside ZIP files
if (re_match($entity, '\.zip$') and
$Features{"Archive::Zip"}) {
my $bh = $entity->bodyhandle();
if (defined($bh)) {
my $path = $bh->path();
if (defined($path)) {
return re_match_in_zip_directory($path, $re);
}
}
}
return 0;
}
Also, MIME::tools 5.009 doesn't exist; I assume it was a typo and
you meant 5.509?
Correct. Just mistype.
--
Alexey
Regards,
Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
Hi, Can you post the relevant part of your filter code that triggers this problem? Also, MIME::tools 5.009 doesn't exist; I assume it was a typo and you meant 5.509? Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] utf-8 issue?
Hi all.
I continue the years old thread related to improper processing of
Unicode strings either in Mimedefang or Perl libs. Last message https://lists.roaringpenguin.com/pipermail/mimedefang/2017-December/038247.html
As far as I can see, the exception happens only when Mimedefang
tries to remove an attachment with non-ASCII (e.g, Russian) name.
If attachment is permitted and passed, exception does not appear.
Looks like also, that other Russian names in same message do not
generate an exception; not sure in exception logging scheme.
Please note also that about 50% of attachments in our environment
have Cyrillic names in UTF-8, but the issue happens few times a
year only.
The example log ("use diagnostics;" in mimedefang.pl) is below.
The visible names of attachments are: "Ат. профессора.rar", "ВАК
дмн.rar", "Диплом БДМУ 1982.rar" and "Категория 2016.rar".
The MIME headers for first attachment are:
Content-Type: application/octet-stream;
name="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LAucmFy?="
Content-Disposition: attachment; filename="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LA=?=
=?UTF-8?B?LnJhcg==?="
Content-Transfer-Encoding: base64
X-Attachment-Id: f_jimx17ff8
Debian Jessie, Mimedefang 2.83 built from source, no embedded
Perl mode, Mime-Tools 5.009, Perl v5.20.2.
Yes, I have the whole message and I can provide any other
non-confidential details from it.
Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
MDLOG,C6CE82122B,bad_filename,%410%442.
%43F%440%43E%444%435%441%441%43E%440%430.rar,application/octet-stream,,,Re:
LEC
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: Strings with code points over 0xFF may not be
mapped into in-memory file
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: handles (#1)
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: (W utf8) You tried to open a reference to a
scalar for read or append
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: where the scalar contained code points over
0xFF. In-memory files
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: model on-disk files and can only contain
bytes.
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr:
Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
MDLOG,C6CE82122B,bad_filename,%412%410%41A
%434%43C%43D.rar,application/octet-stream,,,Re:
LEC
Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
MDLOG,C6CE82122B,bad_filename,%414%438%43F%43B%43E%43C
%411%414%41C%423 1982.rar,application/octet-stream,,,Re:
LEC
Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
MDLOG,C6CE82122B,bad_filename,%41A%430%442%435%433%43E%440%438%44F
2016.rar,application/octet-stream,,,Re:
LEC
Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
MDLOG,C6CE82122B,mail_in,Re:
LEC
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: Uncaught exception from user code:
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: open body: Invalid argument at
/usr/share/perl5/MIME/Entity.pm line 1892.
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr:
MIME::Entity::print_bodyhandle(MIME::Entity=HASH(0xa10a63c),
GLOB(0x9b07fe8)) called at /usr/share/perl5/MIME/Entity.pm line
1873
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr:
MIME::Entity::print_body(MIME::Entity=HASH(0xa10a63c),
GLOB(0x9b07fe8)) called at /usr/share/perl5/MIME/Entity.pm line
1785
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr:
MIME::Entity::print(MIME::Entity=HASH(0xa10a63c), GLOB(0x9b07fe8))
called at /usr/share/perl5/MIME/Entity.pm line 1846
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: MIME::Entity::print_body
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: (MIME::Entity=HASH(0xa106420), GLOB(0x9b07fe8))
called at /usr/local/bin/mimedefang.pl line 6160
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr:
main::do_scan("/var/spool/MIMEDefang/mdefang-W13qP1p") called at
/usr/local/bin/mimedefang.pl line 5663
Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
Worker 0 stderr: main::handle_scan("C6CE82122B",
"/var/spool/MIMEDefang/mdefang-W13qP
Re: [Mimedefang] utf-8 issue?
On Thu, 14 Dec 2017 17:26:38 -0500 "Bill Cole" wrote: > > This will cause other problems down the line. I suggest you > > study the section "Byte and Character Semantics" in the perlunicode > > man page. > Thanks for the reference and for your patience with my confusion. I > understand much better now: anything expressed as a string has to > make sense as a series of encoded characters, not a series of bytes. Right. We completely overhauled our commercial CanIt software to handle Unicode and it took me about four weeks to completely grok how Perl handles Unicode and how the Encode module should be used. It was quite painful. :) Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On 14 Dec 2017, at 9:14 (-0500), Dianne Skoll wrote:
On Wed, 13 Dec 2017 20:37:36 -0500
"Bill Cole" wrote:
-return IO::File->new(\ $self->{MBS_Data}, $mode);
+return IO::Scalar->new(\ $self->{MBS_Data}, $mode);
This will cause other problems down the line. I suggest you
study the section "Byte and Character Semantics" in the perlunicode
man page.
Thanks for the reference and for your patience with my confusion. I
understand much better now: anything expressed as a string has to make
sense as a series of encoded characters, not a series of bytes.
And as `strings PerlIO/scalar/scalar.bundle` says: "Strings with code
points over 0xFF may not be mapped into in-memory file handles"
Modern Perl does let you open a "file" by passing a reference
to a scalar; it has built-in in-memory I/O.
Indeed. I was fooled by the fact that deep inside the 'open' call stack,
both $! and $^E get set to "No such file or directory" but that's
because XSLoader looks for a .bs while loading PerlIO::scalar...
I couldn't find exactly where the "Invalid Argument" error (and matching
numeric errno) is being generated, since it's in a debug-resistant
return that pops 4 stack layers in one step, but I'm perfectly happy to
believe that it's buried in PerlIO/scalar/scalar.bundle and is entirely
proper.
--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On Wed, 13 Dec 2017 20:37:36 -0500
"Bill Cole" wrote:
> -return IO::File->new(\ $self->{MBS_Data}, $mode);
> +return IO::Scalar->new(\ $self->{MBS_Data}, $mode);
This will cause other problems down the line. I suggest you
study the section "Byte and Character Semantics" in the perlunicode
man page.
Modern Perl does let you open a "file" by passing a reference
to a scalar; it has built-in in-memory I/O.
Regards,
Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On Wed, 13 Dec 2017 21:03:27 -0500 "Bill Cole" wrote: > It is entirely possible (as stated explicitly in RFC2045) for a MIME > entity to contain unencoded binary data: any arbitrary stream of > bytes. Stream of *bytes* yes. But Perl native characters > 0xFF are not bytes. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On 13 Dec 2017, at 16:39, Dianne Skoll wrote: It doesn't make sense because MIME messages are alway 8-bit messages; you need to encode everything as UTF-8 first before passing to MIME::Entity->build(). Re-reading that, I disagree. It is entirely possible (as stated explicitly in RFC2045) for a MIME entity to contain unencoded binary data: any arbitrary stream of bytes. Email normally cannot, but that's a transport issue. Since the BINARYMIME extension for SMTP exists, it is even possible to send mail (with some MTAs) as an arbitrary stream of bytes. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On 13 Dec 2017, at 16:39, Dianne Skoll wrote:
On Wed, 13 Dec 2017 16:28:58 -0500
"Bill Cole" wrote:
Unfortunately, I tested a bit more and found that bug is still extant
in 5.509, when tested with the one-liner in that bug report.
That "bug" is a WONTFIX.
Fair enough. I guess the answer to that specific bug report is in fact
"DON'T DO THAT!"
However, I was wondering how "Invalid Argument" was spitting out in that
particular place with 2 possibly different root causes, so I spent some
of my copious free time in the perl debugger and came up with this patch
(also added to the bug ticket):
--- Body.pm.old 2017-04-05 12:55:57.0 -0400
+++ Body.pm.new 2017-12-13 19:04:27.0 -0500
@@ -139,6 +139,7 @@
### System modules:
use Carp;
use IO::File;
+use IO::Scalar;
### The package version, both in 1.23 style *and* usable by MakeMaker:
$VERSION = "5.509";
@@ -522,7 +523,7 @@
die "bad mode: $mode";
}
-return IO::File->new(\ $self->{MBS_Data}, $mode);
+return IO::Scalar->new(\ $self->{MBS_Data}, $mode);
}
It is unclear to me how the original ever failed to fail, since
IO::File->new() really wants a filename or nothing and
MIME::Body->{MBS_Data} isn't a filename. With that patch and a slightly
modified one-liner (redirecting stderr because it emits a non-fatal
""Wide character in print"" warning) I get success:
bigsky:~ root# perl -MMIME::Entity \
>-e 'MIME::Entity->build( Data => "\x{1f4a9}\n" )->print' \
>2>/dev/null && echo "Success!"
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.509 (Entity 5.509)
💩
Success!
(I'm bracing for either a terse tutorial on how That's Not How This All
Works or a terser "Wrong!")
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On Wed, 13 Dec 2017 16:28:58 -0500
"Bill Cole" wrote:
> Unfortunately, I tested a bit more and found that bug is still extant
> in 5.509, when tested with the one-liner in that bug report.
That "bug" is a WONTFIX. You can NOT feed MIME::Entity->build()
data with raw characters > 0xFF. It doesn't make sense because
MIME messages are alway 8-bit messages; you need to encode everything
as UTF-8 first before passing to MIME::Entity->build(). I should
comment on the ticket. The correct way to build the entity
would be:
use Encode;
use MIME::Entity;
my $data = "\x{1f4a9}";
my $e = MIME::Entity->build(Data => Encode::encode('UTF-8', $data,
Encode::FB_CROAK));
$e->print();
So the question is... how on Earth are characters > 0xFF getting passed
to MIME::Entity->build() from within mimedefang.pl?
I will close the bug on CPAN.
Regards,
Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On 13 Dec 2017, at 2:31, Mark Coetser wrote:
On 13/12/2017 00:09, Bill Cole wrote:
[...]
I'd bet this is the same as this bug:
https://rt.cpan.org/Public/Bug/Display.html?id=105377
Which remains open but maybe not unfixed.
and the filter is pretty much the stock microsoft mimedefang-filter
and this does seem to be related from the upgrade from Debian Jessie
to Stretch, does anyone have any pointers?
It couldn't hurt to update MIME-Tools and it is certain that you're
not up-to-date with it, since line 1878 of the current Entity.pm is
blank. That error message would only be generated by line 1892 in the
current version. My bet is that the typo fix cited in the Changelog
for 5.507 also fixed this...
line 1878 in my /usr/share/perl5/MIME/Entity.pm is the following
my $IO = $self->open("r") || die "open body: $!";
Indicating that you are using the same version of MIME-Tools as the
person who opened Bug #105377, which was 5.506.
Unfortunately, I tested a bit more and found that bug is still extant in
5.509, when tested with the one-liner in that bug report.
Hopefully Dianne will note that. This could be a different issue
generating the same error in the same place OR it could be that she has
fixed something since which mostly avoids triggering it. It would still
make sense to update to MIME-Tools 5.509 and MIMEDefang 2.83 to rule out
the possibility that this is a fixed issue with the current versions.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On 12/12/2017 20:03, Dianne Skoll wrote:
On Tue, 12 Dec 2017 15:43:14 +0200
Mark Coetser wrote:
Error from multiplexor: ERR No response from slave
Reap: slave 1 (pid 15022) exited normally with status 22 (SLAVE DIED
UNEXPECTEDLY)
I've never seen this before. I'm also not convinced it's related
to the UTF-8 issue. Could you post the exact filter you are using?
I'm also running on Stretch, btw.
Regards,
Dianne.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
filter attached, I am doing some other stuff in there but as I said this
filter has been in place for many years and only recently have those
errors been occurring...
Thank you,
Mark Adrian Coetser
# -*- Perl -*-
#***
#
# mimedefang-filter
#
# Suggested minimum-protection filter for Microsoft Windows clients, plus
# SpamAssassin checks if SpamAssassin is installed.
#
# Copyright (C) 2002 Roaring Penguin Software Inc.
#
# This program may be distributed under the terms of the GNU General
# Public License, Version 2, or (at your option) any later version.
#
# $Id: mimedefang-filter,v 1.7 2009/10/12 09:49:13 root Exp $
#***
$SALocalTestsOnly = 0;
#***
# Set administrator's e-mail address here. The administrator receives
# quarantine messages and is listed as the contact for site-wide
# MIMEDefang policy. A good example would be 'defang-ad...@mydomain.com'
#***
$AdminAddress = 'postmas...@domain.co.za';
$AdminName = "PKF mail firewall";
#***
# Set the e-mail address from which MIMEDefang quarantine warnings and
# user notifications appear to come. A good example would be
# 'mimedef...@mydomain.com'. Make sure to have an alias for this
# address if you want replies to it to work.
#***
$DaemonAddress = 'mimedef...@domain.co.za';
$LocalHostName = 'mailhub01';
#***
# If you set $AddWarningsInline to 1, then MIMEDefang tries *very* hard
# to add warnings directly in the message body (text or html) rather
# than adding a separate "WARNING.TXT" MIME part. If the message
# has no text or html part, then a separate MIME part is still used.
#***
$AddWarningsInline = 0;
#***
# To enable syslogging of virus and spam activity, add the following
# to the filter:
# md_graphdefang_log_enable();
# You may optionally provide a syslogging facility by passing an
# argument such as: md_graphdefang_log_enable('local4'); If you do this, be
# sure to setup the new syslog facility (probably in /etc/syslog.conf).
# An optional second argument causes a line of output to be produced
# for each recipient (if it is 1), or only a single summary line
# for all recipients (if it is 0.) The default is 1.
# Comment this line out to disable logging.
#***
md_graphdefang_log_enable('mail', 1);
#***
# Uncomment this to block messages with more than 50 parts. This will
# *NOT* work unless you're using Roaring Penguin's patched version
# of MIME tools, version MIME-tools-5.411a-RP-Patched-02 or later.
#
# WARNING: DO NOT SET THIS VARIABLE unless you're using at least
# MIME-tools-5.411a-RP-Patched-02; otherwise, your filter will fail.
#***
# $MaxMIMEParts = 50;
#***
# Set various stupid things your mail client does below.
#***
# Set the next one if your mail client cannot handle multiple "inline"
# parts.
$Stupidity{"NoMultipleInlines"} = 0;
# Detect and load Perl modules
detect_and_load_perl_modules();
# This procedure returns true for entities with bad filenames.
sub filter_bad_filename {
my($entity) = @_;
my($bad_exts, $re);
# Bad extensions
$bad_exts =
'(ade|adp|app|asd|asf|asx|bas|bat|chm|cmd|com|cpl|crt|dll|exe|fxp|hlp|hta|hto|inf|ini|ins|isp|jse?|lib|lnk|mdb|mde|msc|msi|msp|mst|ocx|pcd|pif|prg|reg|scr|sct|sh|shb|shs|sys|url|vb|vbe|vbs|vcs|vxd|wmd|wms|wmz|wsc|wsf|wsh|\{[^\}]+\})';
# Do not all
Re: [Mimedefang] utf-8 issue?
On 12 Dec 2017, at 8:43 (-0500), Mark Coetser wrote: Hi I have 4 different mail hubs, all running latest debian ii perl 5.24.1-3+deb9u2i386 Larry Wall's Practical Extraction and Report Language ii mimedefang 2.79-2 i386 e-mail filter program for sendmail [...] stderr: open body: Invalid argument at /usr/share/perl5/MIME/Entity.pm line 1878. A google search shows this topic https://lists.roaringpenguin.com/pipermail/mimedefang/2013-February/036880.html I doubt that's related. But I am not doing anything with MIME::Entity; Not explicitly, but MD uses it (and the rest of MIME-Tools) extensively. I'd bet this is the same as this bug: https://rt.cpan.org/Public/Bug/Display.html?id=105377 Which remains open but maybe not unfixed. and the filter is pretty much the stock microsoft mimedefang-filter and this does seem to be related from the upgrade from Debian Jessie to Stretch, does anyone have any pointers? It couldn't hurt to update MIME-Tools and it is certain that you're not up-to-date with it, since line 1878 of the current Entity.pm is blank. That error message would only be generated by line 1892 in the current version. My bet is that the typo fix cited in the Changelog for 5.507 also fixed this... -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Currently Seeking Steady Work: https://linkedin.com/in/billcole ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] utf-8 issue?
On Tue, 12 Dec 2017 15:43:14 +0200 Mark Coetser wrote: > Error from multiplexor: ERR No response from slave > Reap: slave 1 (pid 15022) exited normally with status 22 (SLAVE DIED > UNEXPECTEDLY) I've never seen this before. I'm also not convinced it's related to the UTF-8 issue. Could you post the exact filter you are using? I'm also running on Stretch, btw. Regards, Dianne. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] utf-8 issue?
Hi I have 4 different mail hubs, all running latest debian ii perl 5.24.1-3+deb9u2i386 Larry Wall's Practical Extraction and Report Language ii mimedefang 2.79-2 i386 e-mail filter program for sendmail Recently I am seeing alot of the following errors Error from multiplexor: ERR No response from slave Reap: slave 1 (pid 15022) exited normally with status 22 (SLAVE DIED UNEXPECTEDLY) The above are logged in mail.err and are related to the following in mail.log which rejects the email with a "try again later" message until the email eveentually bounces after 5 days. stderr: open body: Invalid argument at /usr/share/perl5/MIME/Entity.pm line 1878. A google search shows this topic https://lists.roaringpenguin.com/pipermail/mimedefang/2013-February/036880.html But I am not doing anything with MIME::Entity; and the filter is pretty much the stock microsoft mimedefang-filter and this does seem to be related from the upgrade from Debian Jessie to Stretch, does anyone have any pointers? -- Thank you, Mark Adrian Coetser ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
