from:"Alexey Lobanov"

Re: [Mimedefang] utf-8 issue?

2020-01-29 Thread Alexey Lobanov

Hi all.

I ask you very much to continue the thread started in 2017. The problem
persists, and it is really annoying.

In brief: Mimedefang dies while message checking if *all *of following
conditions happen:

1. Attachment extension is "bad" and the attachment should be quarantined.

2. Attachment file name is non-ASCII and so why is encoded in UTF-8.

3. UTF-8 header encoding in original message is "incorrect" in some way.
Not sure what it means; many other Unicode filenames are processed and
quarantined properly.

I have supplied example strings to the maillist in 2018, and can add
more examples. Fresh one:

Jan 29 13:13:29 mimedefang.pl[24376]: ACE0720490:
MDLOG,ACE0720490,bad_filename,%42D%426%41F.rar,application/x-rar,,,=?UTF-8?B?0K3QptCf?=
Jan 29 13:13:29 mimedefang-multiplexor[27228]: ACE0720490: Slave 1
stderr: open body: Invalid argument at /usr/share/perl5/MIME/Entity.pm
line 1892.
Jan 29 13:13:29 mimedefang-multiplexor[27228]: Slave 1 died prematurely
-- check your filter rules
Jan 29 13:13:29 postfix/cleanup[3830]: ACE0720490: milter-reject:
END-OF-MESSAGE from f558.i.mail.ru[94.100.184.166]: 4.7.1 Service
unavailable - try again later;

Any ideas or workarounds? Changing mime-tools version changes line
number in Entity.pm, and nothing more.

Older examples are below.

Alexey

On 25.06.2018 17:17, Alexey Lobanov wrote:

> Hi all.
>
> I continue the years old thread related to improper processing of
> Unicode strings either in Mimedefang or Perl libs. Last message
> https://lists.roaringpenguin.com/pipermail/mimedefang/2017-December/038247.html
>
> As far as I can see, the exception happens only when Mimedefang tries
> to remove an attachment with non-ASCII (e.g, Russian) name. If
> attachment is permitted and passed, exception does not appear. Looks
> like also, that other Russian names in same message do not generate an
> exception; not sure in exception logging scheme. Please note also that
> about 50% of attachments in our environment have Cyrillic names in
> UTF-8, but the issue happens few times a year only.
>
> The example log ("use diagnostics;" in mimedefang.pl) is below.
>
> The visible names of attachments are: "Ат. профессора.rar", "ВАК
> дмн.rar", "Диплом БДМУ 1982.rar" and "Категория 2016.rar".
>
> The MIME headers for first attachment are:
>
> Content-Type: application/octet-stream; 
>   name="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LAucmFy?="
> Content-Disposition: attachment; 
> filename="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LA=?=
>   =?UTF-8?B?LnJhcg==?="
> Content-Transfer-Encoding: base64
> X-Attachment-Id: f_jimx17ff8
>
> Debian Jessie, Mimedefang 2.83 built from source, no embedded Perl
> mode, Mime-Tools 5.009, Perl v5.20.2.
>
> Yes, I have the whole message and I can provide any other
> non-confidential details from it.
>
> 
>
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%410%442.
> %43F%440%43E%444%435%441%441%43E%440%430.rar,application/octet-stream,,,Re:
> LEC
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: Strings with code points over 0xFF may not be mapped into
> in-memory file
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr:    handles (#1)
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: (W utf8) You tried to open a reference to a scalar for
> read or append
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: where the scalar contained code points over 0xFF. 
> In-memory files
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: model on-disk files and can only contain bytes.
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr:
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%412%410%41A
> %434%43C%43D.rar,application/octet-stream,,,Re: 
> LEC
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%414%438%43F%43B%43E%43C
> %411%414%41C%423
> 1982.rar,application/octet-stream,,,Re: LEC
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,bad_filename,%41A%430%442%435%433%43E%440%438%44F
> 2016.rar,application/octet-stream,,,Re: LEC
> Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
> MDLOG,C6CE82122B,mail_in,Re: LEC
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr: Uncaught exception from user code:
> Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B: Worker
> 0 stderr:    open body: Invalid argument at
> /usr/sha

Re: [Mimedefang] utf-8 issue?

2018-06-28 Thread Alexey Lobanov


Hi.

On 28.06.2018 1:51, Dianne Skoll wrote:


Hi,

Can you post the relevant part of your filter code that triggers
this problem?


It is default code with extended extension list. Trojan writers like 
these file types novadays.


==

sub filter_bad_filename  {
    my($entity) = @_;
    my($bad_exts, $re);

    # Bad extensions
    $bad_exts = 
'(arj|rar|r15|lzh|uue|7z|xz|xlsm|epf|docm|dotm|dot|tgz|ace|z|gz|ace|jar|ade|adp|app|asd|asf|asx|bas|bat|cab|chm|cmd|com|cpl|dll|exe|fxp|hlp|hta|hto|inf|ini|ins|isp|jse?|lib|lnk|mdb|mde|msc|msi|msp|mst|ocx|pcd|pif|prg|reg|scr|sct|sh|shb|shs|sys|vb|vbe|vbs|vxd|wsc|wsf|wsh|\{[^\}]+\})';


    # Do not allow:
    # - CLSIDs  {foobarbaz}
    # - bad extensions (possibly with trailing dots) at end
    $re = '\.' . $bad_exts . '\.*$';

    return 1 if (re_match($entity, $re));

    # Look inside ZIP files
    if (re_match($entity, '\.zip$') and
    $Features{"Archive::Zip"}) {
    my $bh = $entity->bodyhandle();
    if (defined($bh)) {
    my $path = $bh->path();
    if (defined($path)) {
    return re_match_in_zip_directory($path, $re);
    }
    }
    }
    return 0;
}





Also, MIME::tools 5.009 doesn't exist; I assume it was a typo and
you meant 5.509?


Correct. Just mistype.

--
Alexey



Regards,

Dianne.

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang


___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

[Mimedefang] utf-8 issue?

2018-06-26 Thread Alexey Lobanov


  
  
Hi all.
I continue the years old thread related to improper processing of
  Unicode strings either in Mimedefang or Perl libs. Last message https://lists.roaringpenguin.com/pipermail/mimedefang/2017-December/038247.html
As far as I can see, the exception happens only when Mimedefang
  tries to remove an attachment with non-ASCII (e.g, Russian) name.
  If attachment is permitted and passed, exception does not appear.
  Looks like also, that other Russian names in same message do not
  generate an exception; not sure in exception logging scheme.
  Please note also that about 50% of attachments in our environment
  have Cyrillic names in UTF-8, but the issue happens few times a
  year only.

The example log ("use diagnostics;" in mimedefang.pl) is below.
The visible names of attachments are: "Ат. профессора.rar", "ВАК
  дмн.rar", "Диплом БДМУ 1982.rar" and "Категория 2016.rar".
The MIME headers for first attachment are:
Content-Type: application/octet-stream; 
	name="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LAucmFy?="
Content-Disposition: attachment; filename="=?UTF-8?B?0JDRgi4g0L/RgNC+0YTQtdGB0YHQvtGA0LA=?=
	=?UTF-8?B?LnJhcg==?="
Content-Transfer-Encoding: base64
X-Attachment-Id: f_jimx17ff8
Debian Jessie, Mimedefang 2.83 built from source, no embedded
  Perl mode, Mime-Tools 5.009, Perl v5.20.2.

Yes, I have the whole message and I can provide any other
  non-confidential details from it.



Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
  MDLOG,C6CE82122B,bad_filename,%410%442.
%43F%440%43E%444%435%441%441%43E%440%430.rar,application/octet-stream,,,Re:
  LEC
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr: Strings with code points over 0xFF may not be
  mapped into in-memory file
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:    handles (#1)
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr: (W utf8) You tried to open a reference to a
  scalar for read or append
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr: where the scalar contained code points over
  0xFF.  In-memory files
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr: model on-disk files and can only contain
  bytes.
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:
  Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
  MDLOG,C6CE82122B,bad_filename,%412%410%41A
  %434%43C%43D.rar,application/octet-stream,,,Re:
  LEC
  Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
  MDLOG,C6CE82122B,bad_filename,%414%438%43F%43B%43E%43C
  %411%414%41C%423 1982.rar,application/octet-stream,,,Re:
  LEC
  Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
  MDLOG,C6CE82122B,bad_filename,%41A%430%442%435%433%43E%440%438%44F
  2016.rar,application/octet-stream,,,Re:
  LEC
  Jun 20 18:26:52 mail mimedefang.pl[25718]: C6CE82122B:
  MDLOG,C6CE82122B,mail_in,Re:
  LEC
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr: Uncaught exception from user code:
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:    open body: Invalid argument at
  /usr/share/perl5/MIME/Entity.pm line 1892.
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:   
  MIME::Entity::print_bodyhandle(MIME::Entity=HASH(0xa10a63c),
  GLOB(0x9b07fe8)) called at /usr/share/perl5/MIME/Entity.pm line
  1873
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:   
  MIME::Entity::print_body(MIME::Entity=HASH(0xa10a63c),
  GLOB(0x9b07fe8)) called at /usr/share/perl5/MIME/Entity.pm line
  1785
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:   
  MIME::Entity::print(MIME::Entity=HASH(0xa10a63c), GLOB(0x9b07fe8))
  called at /usr/share/perl5/MIME/Entity.pm line 1846
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:    MIME::Entity::print_body
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr: (MIME::Entity=HASH(0xa106420), GLOB(0x9b07fe8))
  called at /usr/local/bin/mimedefang.pl line 6160
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:   
  main::do_scan("/var/spool/MIMEDefang/mdefang-W13qP1p") called at
  /usr/local/bin/mimedefang.pl line 5663
  Jun 20 18:26:52 mail mimedefang-multiplexor[25717]: C6CE82122B:
  Worker 0 stderr:    main::handle_scan("C6CE82122B",
  "/var/spool/MIMEDefang/mdefang-W13qP

Re: [Mimedefang] utf-8 issue?

Re: [Mimedefang] utf-8 issue?

[Mimedefang] utf-8 issue?

3 matches

Site Navigation

Mail list logo

Footer information