Re: filter-dnsbl and Abusix

[Kirill A . Korinsky]

Greeting,

On Sat, 10 Feb 2024 20:10:36 +0100,
J Doe wrote:
> 
> ...however, if I try either the combined DNSBL from Abusix or the black
> DNSBL from Abusix, it will reject mail from Hotmail as well as e-mail
> that is hosted via 1&1's e-mail service.
> 
> The man page for filter-dnsbl mentions Abusix, so I am assuming it
> supports it, but why would this be happening ?  It seems like all
> responses via Abusix are detected as spam.
> 
> Does anyone else make use of Abusix and see this behaviour ?
> 

I use it.

And Abusix is expect that you're using it with their white list as well.

I do have a fork of filter-dnsbl which supports white listing, and which I'm
using for more than a month now.

Thus, in few weeks I plan to describe my email setup and share here a link, but
I need some time to stabelize software before I announce it.

-- 
wbr, Kirill

Re: need test from MacOS not Sonoma

[Kirill A. Korinsky]

Hey,

I do have a collection of macOS 10.5…13

-- 
wbr, Kirill

> On 9. Mar 2024, at 12:22, gil...@poolp.org wrote:
> 
> Hello,
> 
> Anyone on the list has a MacOS that's not Sonoma and can help track a bug ?
> 
> Gilles
> 

Disabling incoming SMTP connections: Client limit reached

[Kirill A . Korinsky]

Greetings,

I've noticed that my personal mail stop to working. After some digging in
the log I discovered the magic line:

  Apr 10 23:37:43 mx1 smtpd[84358]: warn: Disabling incoming SMTP connections: 
Client limit reached

which leads to this issue https://github.com/OpenSMTPD/OpenSMTPD/issues/698

After reading it and some context, like
https://www.mail-archive.com/misc@opensmtpd.org/msg03171.html I confuse.

What is current status of filters and this issue?

I'm running OpenSMTPD on OpenBSD 7.4, which I plan to upgrade in few days to
7.5, does this upgrade helps with this issue?

BTW it works without any question for month, I guess.

-- 
wbr, Kirill

Re: Disabling incoming SMTP connections: Client limit reached

[Kirill A . Korinsky]

Greetings,

On Fri, 12 Apr 2024 09:01:32 +0200,
gil...@poolp.org wrote:
> 
> This looks like clients hogging connections and not releasing them, or a leak
> within a filter.
> 
> - what do you see with the `fstat` command when the issue happens ?
> - do you see unusual trafic in your logs and/or `netstat` ?
> - any local script gone wrong in you `ps` output ?
> - and more importantly what's your configuration file like ?
> 

Unfortently I've restarted both mail server, and I can't answer to your
questions other than provide a config.

I've double checked it right now, and for 9 hours it hasn't got any unusual
issue.

Anyway, I've noticed an issue in hours, like 20.

> Give more details please

I'll try to give everything that I can. If you need more, feel free to ask.

1. smtpd.conf

I've removed srs keys, comments and short the list of used DNSBL:

pki mx.catap.net cert "/etc/ssl/mx.catap.net.crt"
pki mx.catap.net key  "/etc/ssl/private/mx.catap.net.key"

table aliasesfile:/etc/mail/aliases
table domainsfile:/etc/mail/domains
table credentialspasswd:/etc/mail/credentials

queue ttl 7d
bounce warn-interval 1h, 1d, 3d, 6d

admd mx.catap.net

smtp max-message-size 100M

listen on socket

action "local_mail" mbox alias 
match from local for local action "local_mail"

filter admdscrub proc-exec "filter-admdscrub -s"
filter "auth" proc-exec "filter-auth"

filter dnsbl proc-exec "filter-dnsbl -m \
   all.s5h.net \
   -w list.dnswl.org \
   zen.spamhaus.org \
   bl.local \
   -w wl.local "

listen on egress inet4 port smtp tls pki mx.catap.net \
   filter { admdscrub, "auth", dnsbl }

action deliver_lmtp lmtp "/var/dovecot/lmtp" rcpt-to virtual 
match from any for domain  action deliver_lmtp

filter dkimsign_rsa proc-exec "filter-dkimsign -a rsa-sha1 -D 
/etc/mail/domains \
   -s 20240125_rsa -k /etc/mail/dkim/20240125.rsa.key" user _dkimsign 
group _dkimsign

filter dkimsign_ed25519 proc-exec "filter-dkimsign -a ed25519-sha256 -D 
/etc/mail/domains \
   -s 20240125_ed25519 -k /etc/mail/dkim/20240125.ed25519.key" user 
_dkimsign group _dkimsign

filter dkimsign chain { dkimsign_rsa }

listen on egress port smtps \
   smtps pki mx.catap.net auth  mask-src filter dkimsign
listen on egress port submission \
   tls-require pki mx.catap.net auth  mask-src filter 
dkimsign

action "outbound" relay srs
match from any auth for any action "outbound"

2. filters

Almost all used filters is the fork with minimal changes, I plan to backport
it as soon as it stable enough. Anyway, the code available here:
 - https://github.com/catap/opensmtpd-filter-dkimsign
 - https://github.com/catap/opensmtpd-filter-dnsbl
 - https://github.com/catap/opensmtpd-filter-admdscrub

Plus I use a filter which implemets SPF and DKMI verify in one call which
is a good candidate to be be a source of leaking because before this error
message, it was warn of missed SPF domain from it. So, the source is here
https://github.com/catap/opensmtpd-filter-auth

3. Output of fstat | grep smtpd:

_smtpd   filter-dnsbl 72770 text /50153232  -rwxr-xr-x r
13864
_smtpd   filter-dnsbl 72770   wd /   2  drwxr-xr-x r  
512
_smtpd   filter-dnsbl 727700* unix stream 0x0
_smtpd   filter-dnsbl 727701* unix stream 0x0
_smtpd   filter-dnsbl 727702* unix stream 0x0
_smtpd   filter-dnsbl 727703 kqueue 0x0 0 state: W
_smtpd   filter-auth 57529 text /50161556  -rwxr-xr-x r91888
_smtpd   filter-auth 57529   wd /   2  drwxr-xr-x r  512
_smtpd   filter-auth 575290* unix stream 0x0
_smtpd   filter-auth 575291* unix stream 0x0
_smtpd   filter-auth 575292* unix stream 0x0
_smtpd   filter-auth 575293 kqueue 0x0 0 state: W
_smtpd   filter-admdscrub 67397 text /50153229  -rwxr-xr-x r
25688
_smtpd   filter-admdscrub 67397   wd /   2  drwxr-xr-x r
  512
_smtpd   filter-admdscrub 673970* unix stream 0x0
_smtpd   filter-admdscrub 673971* unix stream 0x0
_smtpd   filter-admdscrub 673972* unix stream 0x0
_smtpd   filter-admdscrub 673973 kqueue 0x0 0 state: W
_smtpd   table-passwd 44437 text /50153223  -rwxr-xr-x r
45000
_smtpd   table-passwd 44437   wd /   2  drwxr-xr-x r  
512
_smtpd   table-passwd 444370* unix stream 0x0
_smtpd   table-passwd 444371 / 3875217  crw-rw-rw-rw 
null
_smtpd   table-passwd 444372 / 3875217  crw-rw-rw-rw 
null
_smtpd   smtpd  16336 text /50051231  -r-xr-xr-x r   532008
_smtpd   smtpd  16336   wd /   2  drwxr-xr-x r  512
_smtpd   smtpd  163360* unix stream 0x0
_smtpd   smtpd  163

Inject Date and Message-Id

[Kirill A . Korinsky]

misc@,

Is it possible to inject into email date and message-id when email hasn't
got one?

Use case: my scaner sents email but qutie dummy which has format like:

  From: some@email
  To: my@email
  Subject: Scanned documents
  MIME-Version: 1.0
  Content-Type: multipart/mixed; boundary="__MIME_Section_Boundary__"

  This is a multi-part messsage in MIME format.

  --__MIME_Section_Boundary__
  Content-Type: application/pdf; name="=?utf-8?Q?scan.pdf?="
  Content-Description: =?utf-8?Q?scan.pdf?=
  Content-Disposition: attachment; filename="=?utf-8?Q?scan.pdf?="
  Content-Transfer-Encoding: base64

  JVBERi0xLjQKMSAwIG9iago8PCAvVHlwZSAvQ2F0YWxvZwovUGFnZXMgMiAwIFIKPj4KZW5kb2Jq
  CjMgMCBvYmoKPDwgL1R5cGUgL1BhZ2UKL1BhcmVudCAyIDAgUgovUmVzb3VyY2VzIDQgMCBSCi9N
  
  dHhyZWYKNDQ5ODU5OAolJUVPRg==

  --__MIME_Section_Boundary__--

as you may guess it's impossible to add current date to subject.

Some email cleint (at least Wunderlust which I use) creates a fake msgid
from some fields like From, To, Subject, Date, etc. when it missed to use it
as cache key.

So, literrally all emails from my scanner is cached as single email.

As simpler solution I see the behaviour for OpenSMTPd to inject MessageId
(when it missed) or date.

Or it's bad idea?

-- 
wbr, Kirill

Re: Inject Date and Message-Id

[Kirill A . Korinsky]

On Wed, 17 Apr 2024 09:48:14 +0200,
Kirill A. Korinsky wrote:
> 
> As simpler solution I see the behaviour for OpenSMTPd to inject MessageId
> (when it missed) or date.
> 
> Or it's bad idea?
> 

After reading the code I see that it should insert Date and MessageId if
client is connected to 587 port[1], in my case it uses 587 port and doesn't
inserted Message-Id nor Date header.

The listers looks like:

  listen on egress port smtps \
 smtps pki mx.catap.net auth  mask-src filter dkimsign
  listen on egress port submission \
 tls-require pki mx.catap.net auth  mask-src filter 
dkimsign

and that it uses 587 was just confirmed by tcpdump.

Footnotes:
[1]  
https://github.com/OpenSMTPD/OpenSMTPD/blob/v6.8.0p2/usr.sbin/smtpd/smtp_session.c#L2739-L2757

-- 
wbr, Kirill

Re: Inject Date and Message-Id

[Kirill A . Korinsky]

Hi,

On Wed, 17 Apr 2024 13:18:47 +0200,
Philipp wrote:
> 
> There was a bug in the code which mixed the byteorder of the port. This
> is fixed in 7.4. So an update might help.
> 

Confirmed, that upgrade to OpenBSD 7.5 fixes an issue.

-- 
wbr, Kirill

Re: Change "524 5.2.4 Mailing list expansion problem" to "550 Invalid recipient"?

[Kirill A . Korinsky]

On Tue, 14 May 2024 11:41:54 +0100,
Jesper Wallin  wrote:
> 
> ps, while writing this mail, satanist on IRC suggested the workaround
> of adding 'rcpt-to ', which might invalidate my
> suggestion/patch... Still, I find the mailing list expansion error more
> confusing than helpful.

I use rcpt-to virtual  in my mail server and it replies to spammer
with invlaid emails like this:

May 14 10:16:58 mx2 smtpd[5018]: 112db6ee6676ecb6 smtp failed-command 
command="RCPT TO:" result="524 5.2.4 Mailing list expansion 
problem: "

From my point of view your patch is the right move.

-- 
wbr, Kirill

New filters auth and sign

[Kirill A . Korinsky]

Greetings,

I'd like to announce a two new filters for OpenSMTD which better to use
together: auth and sign.

auth is a filter which verify DKMI, ARC and SPF, and iprev. It adds
Authentication-Results header or ARC-Authentication-Results.

sign is a filter which adds DKMI or ARC signature, or ARC seal.

For example, I run configuration:

  filter "auth" proc-exec "filter-auth"
  listen on egress port smtp ... filter { admdscrub, "auth", dnsbl }

  filter sign_ed25519 proc-exec "filter-sign -a ed25519-sha256 -D 
/etc/mail/domains \
 -s 20240125ed25519 -k /etc/mail/dkim/20240125.ed25519.key" user 
_dkimsign group _dkimsign
  filter sign_rsa proc-exec "filter-sign -a rsa-sha256 -D /etc/mail/domains \
 -s 20240125rsa -k /etc/mail/dkim/20240125.rsa.key" user _dkimsign 
group _dkimsign

  filter arc_auth proc-exec "filter-auth -A"
  filter arc_sign proc-exec "filter-sign -A -a rsa-sha256 -d mx.catap.net \
 -s 20240125rsa -k /etc/mail/dkim/20240125.rsa.key" user _dkimsign 
group _dkimsign
  filter arc_seal proc-exec "filter-sign -S -a rsa-sha256 -d mx.catap.net \
 -s 20240125rsa -k /etc/mail/dkim/20240125.rsa.key" user _dkimsign 
group _dkimsign

  filter sign chain { sign_ed25519 sign_rsa arc_auth arc_sign arc_seal }

  listen on egress port submission ... filter sign

Here all incomming messages is autorised by adding Authentication-Results,
and all outcomming messages:
 - signed by two DKMI signature with correct domain (list in /etc/mail/domains)
 - signed by one ARC signature with domain mx.catap.net
 - seal by one ARC seal with domain mx.catap.net

Yeah, it is possible to use different selectors for ARC signature and seal,
but I haven't tested it.

The code is based on Martijn van Duren's filter-dkimsign, filter-dkimverify
and filter-spf, and I also used some pices from spfwalk.c from OpenSMTPD.

Man pages for both filters are updated.

Thus, sign filter is drop-in replacment for filter-dkimsign.

Code available here:
 - https://github.com/catap/opensmtpd-filter-auth
 - https://github.com/catap/opensmtpd-filter-sign

I also attached ports for OpenBSD which I used to run it.

How stable it is? Well, enough to share and ask for feedback. It may
contains bugs, but it should be fine to use.

Produced signature was tested against gmail, yahoo, icloud.com and dkimpy
and it holds. Anyway, outlook.com fails on ARC signature with errors 35 or
47 (what does it mean?) and produced invalid signature as the next in ARC
chain (tested by dkimpy).

Thus, this email were sent via server which uses that filters, so, headers
from this email a good example.

-- 
wbr, Kirill


filters.tgz
Description: Binary data

Re: New filters auth and sign

[Kirill A . Korinsky]

On Sat, 01 Jun 2024 08:45:00 +0100,
"Corey Hickman"  wrote:
> 
> does it have policy server included? for instance, when DKIM fails, the 
> policy can be set up to deny the message.
> 

Right now it ignores DMARC as if it doesn't exist.

Doing a DMARC lookup for domain and inserting it's results into the header
is possible and not a big deal, but it has some issues.

The first is parsing the From header. It is durable, but different MUA may
follow different logic and parser for this can be quite complicated. And
complicated means bugs.

The second is more ideological. DMARC needs something that aggregates the
results and sends out reports. It shouldn't be a filter for smtpd. But a
filter can write it's decision to log, and something should harvest it to
process and create reports that need to be sent. Anyway, forensic reports,
which should be close to real-time and include a lot of things from the
original email, is a much more complicated story.

All this brings up the question of personal data / GDPR and DMARC. I know of
a very good analysis of DMARC and GDPR in the case of German law [1], which
can be summarized as a quote:

  The reports are fundamentally permitted and justified under data
  protection law. However, the principle of proportionality is to be
  complied with at all times.

Based on this analysis, I assume that only aggregated reports can be used
without legal headaches in the EU.

But implementing only a part of DMARC seems as much worse than not
implementing it at all, and implementing it in its entirety requires a lot
of pieces in place, much more than just a filter.

Thus, DMARC was discussed on the OpenBSD mailing lists a few months ago [2].

As a conclusion, I personally use the p=none policy, because I assume that
my mail should be delivered, and To is not the final destination, it's a
kind of starting direction of the mail's way to the recipient.

Footnotes:
[1]  
https://certified-senders.org/wp-content/uploads/2018/08/Report_DMARC_and_GDPR.pdf

[2]  https://marc.info/?l=openbsd-misc&m=171015367409290&w=2

-- 
wbr, Kirill

Re: filter-dnsbl: feature request and bug report

[Kirill A . Korinsky]

Greetings,

It was a while in this thread, but I don't forget.

I put all my ideas to filter-dnsbl as a fork which is available on GitHub:
https://github.com/catap/opensmtpd-filter-dnsbl

Here the quote from updated man page which summirizes changes:

 filter-dnsbl looks up the IP address of the sender in the blacklist (a
 domain name) and, by default drops the connection if it is found.  If the
 -m flag is specified, it will allow the message to continue, but such a
 message will be marked with X-Spam header with value Yes, and
 X-Spam-DNSBL header containing a list, and any existing headers starting
 with X-Spam will be stripped. If the -w flag is specified before
 blacklist, this list is treated as white list and X-Spam header isn't
 added, but X-Spam-DNSWL header is added instead X-Spam-DNSBL.
 Additionally, if the -d flag is specified before blacklist, it will use
 reverse DNS hostname instead of IP address for loopkup. For more verbose
 logging, the -v flag can be used.

 When DNS error happened it drops the connection, or adds X-Spam header
 with value Unknown and X-Spam-DNS with a list with cause an error if the
 -m flag is specified.

I've attached to this email port file for OpenBSD to to use it.

Additionally, you may grab build version for 7.5 from
https://mx0.catap.net/pub/ where I also keep filter-sign and filter-auth.

Any feedback and testing welcome.

Martijn, do you think this changes can be backported back?

--
wbr, Kirill


filter.tgz
Description: Binary data

Re: filter-dnsbl: feature request and bug report

[Kirill A . Korinsky]

On Sat, 01 Jun 2024 17:50:53 +0100,
Kirill A. Korinsky  wrote:
> 
> Here the quote from updated man page which summirizes changes:
> 

And quoted text had one missed feature. This filter supports -e argument for
each list to specified expected IP which means match. That allows, for
example, use hostkarma.junkemailfilter.com like this:

   -e 127.0.0.2 hostkarma.junkemailfilter.com \
   -w -e 127.0.0.1 hostkarma.junkemailfilter.com \

So, here response 127.0.0.2 means bad, and 127.0.0.1 good.

-- 
wbr, Kirill

Too many open files from fiter leads to smtpd death

[Kirill A . Korinsky]

misc@,

I just discovered that smtpd dies becuase of filter hit "too many open
files".

Last logs from smtpd:

Jun 11 13:06:03 mx1 smtpd[80363]: 1825a196e20867b3 mta disconnected 
reason=quit messages=1
Jun 11 13:07:06 mx1 smtpd[80363]: 1825a197ad6634d4 smtp connected 
address=198.2.134.32 host=mail134-32.atl141.mandrillapp.com
Jun 11 13:07:08 mx1 smtpd[80363]: 1825a197ad6634d4 smtp tls 
ciphers=TLSv1.3:TLS_AES_256_GCM_SHA384:256
Jun 11 13:07:08 mx1 smtpd[82182]: auth: 1825a197ad6634d4 Can't open 
tempfile: Too many open files
Jun 11 13:07:08 mx1 smtpd[80363]: dispatcher: tree_xpop(0xfae2ce35960, 
0x1825a197ad6634d4)
Jun 11 13:07:08 mx1 smtpd[61396]: smtpd: process dispatcher socket closed

inside filter the code which writes that logs looks like:

if ((msg->origf = tmpfile()) == NULL) {
auth_err(ctx, "Can't open tempfile");
return NULL;
}

..

void
auth_err(struct osmtpd_ctx *ctx, char *text)
{
struct message *msg = ctx->local_message;

fprintf(stderr, "%016"PRIx64" %s: %s\n",
ctx->reqid, text, strerror(errno));

if (msg != NULL)
msg->err = 1;
else
osmtpd_filter_disconnect(ctx, "Internal server error");
}

I expect that it clsoes connection, but not kills smtpd.

Am I wrong?

It is running on OpenBSD 7.5

The code of filter is hire: https://github.com/catap/opensmtpd-filter-auth

-- 
wbr, Kirill

Re: Too many open files from fiter leads to smtpd death

[Kirill A . Korinsky]

On Tue, 11 Jun 2024 19:23:27 +0100,
gil...@poolp.org wrote:
>
> I'm unsure what the filter really did and it is a bit harsh to read,
> but the rules are as follow:
>

Here I anounced it: 
https://www.mail-archive.com/misc@opensmtpd.org/msg06292.html

> I may be wrong about your bug but with just a quick glance I saw mem leaks
> and the error message you have seems to imply a leak of descriptor as well
> so I'd be tempted to assume that the filter is misbehaving and that it did
> not report the fd exhaustion properly to smtpd leading to termination.

Leak of descriptors is good point. On the same servers I had
https://www.mail-archive.com/misc@opensmtpd.org/msg06241.html a while ago
with the same filters which envolved since then a bit, but logic the same.

And for a few hours it had leaked a few descriptiors which I see via fstat,
so it probably is.

Also, I run two MXs and both of them fails the same way almost the same
moment, about 5 minutes between them.

I register session / message related things like this:

osmtpd_local_session(auth_session_new, auth_session_free);
osmtpd_local_message(auth_message_new, auth_message_free);

and I made an assumtion that both free method are called when I call:

osmtpd_filter_disconnect(ctx, "Internal server error");

am I right with that assumtion?

--
wbr, Kirill

Re: Too many open files from fiter leads to smtpd death

[Kirill A . Korinsky]

On Tue, 11 Jun 2024 23:10:07 +0100,
Kirill A. Korinsky  wrote:
> 
> I register session / message related things like this:
> 
>   osmtpd_local_session(auth_session_new, auth_session_free);
>   osmtpd_local_message(auth_message_new, auth_message_free);
> 
> and I made an assumtion that both free method are called when I call:
> 
> osmtpd_filter_disconnect(ctx, "Internal server error");
> 
> am I right with that assumtion?
> 

Seems that I was wrong. Or at least it hasn't happened each connection.

It defently may missbehave on error, and to avoid complexity of code I
migrated filter to osmtpd_err/osmtpd_errx in case of error which exists a
filter if syscall returns something unexcpected.

Also, I've added trace log each time when auth_{message,session}_{new,free}
is called, as first statment of the function.

And for the last night I had 14 leaked files. All of them had happened from
incomming connections from mail.nginx.org, and usuall log associeted with
connection looks like:

mx1$ grep 78ba6f2a7dceb938 /var/log/maillog 
   
Jun 13 05:55:11 mx1 smtpd[97991]: 78ba6f2a7dceb938 smtp connected 
address=206.251.255.65 host=mail.nginx.com
Jun 13 05:55:11 mx1 smtpd[84725]: auth: 78ba6f2a7dceb938 session_new
Jun 13 05:55:13 mx1 smtpd[97991]: 78ba6f2a7dceb938 smtp tls 
ciphers=TLSv1.3:TLS_AES_256_GCM_SHA384:256
Jun 13 05:55:14 mx1 smtpd[84725]: auth: 78ba6f2a7dceb938 message_new
mx1$ 

No dissconnect, no call auth_{message,session}_free, nothing.

-- 
wbr, Kirill

Re: Too many open files from fiter leads to smtpd death

[Kirill A . Korinsky]

On Tue, 11 Jun 2024 23:10:07 +0100,
Kirill A. Korinsky  wrote:
>
> Leak of descriptors is good point. On the same servers I had
> https://www.mail-archive.com/misc@opensmtpd.org/msg06241.html a while ago
> with the same filters which envolved since then a bit, but logic the same.
>
> And for a few hours it had leaked a few descriptiors which I see via fstat,
> so it probably is.
>
> Also, I run two MXs and both of them fails the same way almost the same
> moment, about 5 minutes between them.
>
> I register session / message related things like this:
>
>   osmtpd_local_session(auth_session_new, auth_session_free);
>   osmtpd_local_message(auth_message_new, auth_message_free);
>
> and I made an assumtion that both free method are called when I call:
>
> osmtpd_filter_disconnect(ctx, "Internal server error");
>
> am I right with that assumtion?
>

and I figured it out. Yes, both issues had the same root cause.

My filter runs a few DNS quereis to walk against SPF and get certificates
for DKIM and ARC signatures.

The last one is called by the code:

if ((query = res_query_async(sig->domain, C_IN, T_TXT, NULL)) == NULL)
osmtpd_err(1, "res_query_async");
if ((sig->query = event_asr_run(query, ar_rr_resolve, sig)) == NULL)
osmtpd_err(1, "event_asr_run");

and in may plan to execute more than one request for the same domain.

After a lot of hours of debuging I've added inside lookup function and
ar_rr_resolve log to print domain and address of sig object.

Well, it had proved that ar_rr_resolve not always call.

Huh.

tcpdump says that local DNS server replies, but the function never called.

The logic of filter is based on assumption that it will be called.

So, no call, it waits forever.

Really forewer.

Now wired things: seems some servers (at least Postfix at mail.nginx.org,
probably not only this) keeps connection... forever.

So, at some point all sockets are consumed by waited connections, and here
I had missbehaviour of filter on error (fixed) which lead to exit of smtpd.

Before I had implemented ARC signatures it uses DNS less and it may work
longer, that allows to achive: Disabling incoming SMTP connections: Client
limit reached.

Probably, after fixing a filter I also may achive that, but much faster.

Anyway, here the question: how should I run DNS queries? I've checked smtpd
code and seems that it is built on the same assumption that ar_rr_resolve
should be called.

Thus, I thought that I made something wrong and tried getrrsetbyname_async
without any success.

--
wbr, Kirill

Re: Network error on destination MXs on MX that starts with "_"

[Kirill A . Korinsky]

Hi Jesper,

On Tue, 11 Jun 2024 14:10:11 +0100,
Jesper Wallin  wrote:
> 
> This is indeed an invalid hostname.  A hostname must begin with a digit
> or letter. (https://www.rfc-editor.org/rfc/rfc1123#page-13)
>

You are wrong. For example, SRV records (RFC 2782) are widely used and have
the format _service._proto.name.

Next, I checked RFC 5321 section 5.1 and it uses the following wording:

   When a domain name associated with an MX RR is looked up and the
   associated data field obtained, the data field of that response MUST
   contain a domain name.  That domain name, when queried, MUST return
   at least one address record (e.g., A or  RR) that gives the IP
   address of the SMTP server to which the message should be directed.
   Any other response, specifically including a value that will return a
   CNAME record when queried, lies outside the scope of this Standard.
   The prohibition on labels in the data that resolve to CNAMEs is
   discussed in more detail in RFC 2181, Section 10.3 [38].

As you can see, it explicitly uses the domain name, not the hostname.

This means that the MX record is a domain name, and it can start with an
underscore.

-- 
wbr, Kirill

Re: Too many open files from fiter leads to smtpd death

[Kirill A . Korinsky]

On Fri, 14 Jun 2024 01:53:48 +0100,
Kirill A. Korinsky  wrote:
> 
> Thus, I thought that I made something wrong and tried getrrsetbyname_async
> without any success.
> 

Indeed I did: run asr_abort without decreasign counters which leads to
deadlock. Thus, asr debug log was useless due to missed debug output inside
asr_abort [1].

Anyway, I still not sure that it is correct behaviour that filter may block
processing of email indenfently.


Footnotes:
[1]  https://marc.info/?l=openbsd-misc&m=171853989911463&w=2

-- 
wbr, Kirill

Re: Too many open files from fiter leads to smtpd death

[Kirill A . Korinsky]

On Mon, 17 Jun 2024 23:56:34 +0100,
gil...@poolp.org wrote:
> 
> June 16, 2024 2:12 PM, "Kirill A. Korinsky"  wrote:
> 
> In your specific case, you had a descriptor leak and a deadlock... what would
> be the best option for OpenSMTPD to cope with these ? your filter isn't going
> to end up closing descriptors so you're already toast no matter what we do...
> as for the deadlock, we could try to detect it, but we're already going to be
> hitting the session timeout and the filter will still be in a broken state by
> the next time we enter it.
> 
> The only improvement I see here is that if we could detect deadlock early, we
> could kill smtpd right away.

I spent some time to think about it, and I feel that killing smtpd is the
right move.

As you points before it dies when filter misbehave, and not reply for a
while, let say for 15 minutes, it defently misbehaving.

From another hand, such timeout should be large enough and announced to
filter, because the filter may do DNS requests and it can be quite slow.

-- 
wbr, Kirill

filter-dkimverify status

[Kirill A. Korinsky]

Greetings,

This list seems to be the right place to ask about status of this filter 
http://imperialat.at/dev/filter-dkimverify/ 


As I see via snv log the last commit was about a year ago.

Is it working? Or?

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

Greetings,

This list seems as the right place to discuss about 
http://imperialat.at/dev/filter-dnsbl 

1. I'd like to report a small bug: OpenSMTPD injects header X-Spam: Yes when 
filter decided that it is junk, and this filter injects yes (in the lower 
case). Is it a bug?

2. Is it possible to add support of white list(s)? Let pass the message if it 
contains on that list and optionally adds X-Spam-DNSWL: Listed at ... header.

The nice example of white list is dnswl.mail.abusix.zone

Thanks.

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 19:22, Martijn van Duren  
> wrote:
> 
> On Thu, 2023-12-28 at 18:52 +0100, Kirill A. Korinsky wrote:
>> 
>> 
>> 1. I'd like to report a small bug: OpenSMTPD injects header X-Spam: Yes when 
>> filter decided that it is junk, and this filter injects yes (in the lower 
>> case). Is it a bug?
> 
> I'm not aware about any specific standard when it comes to this header.
> smtpd has `strcasecmp(line, "x-spam: yes")` in mail.maildir.c, so the
> capitalisation isn't important there. However, if you use smtpd with
> filter-dnsbl as a filter before forwarding it to another server which
> does check this header in a case-sensitive manner it would matter.
> So unless you can point to a piece of software which checks it in a
> specific capitalisation I don't see a direct reason to fix it.

Neither do I.

Example of software is sieve: as far as I know it doesn't support matching case 
insensitive strings.

>> 
>> 2. Is it possible to add support of white list(s)? Let pass the message if 
>> it contains on that list and optionally adds X-Spam-DNSWL: Listed at ... 
>> header.
> 
> I've thought about adding whitelist support, but it's tough...
> The RFC is explicitly vague on how to interpret responses, so there's
> no way to determine how a response it to be interpreted without
> extensive configuration either by the admin, or in the binary.
> The latter would require in-depth knowledge of the different lists
> on my end and actively maintain those, which I don't see myself doing.
> 
> Another reason why I don't see myself supporting whitelists is because
> I don't see their value. Mail is whitelist based in principle, so what
> blocking feature is it supposed to overwrite and how is filter-dnsbl
> supposed to do this? And that's not even starting on the prioritisation
> of which list takes precedence.
> 
> Do you have a specific use-case for whitelisting and if so, how would
> you suggest to implement it in a generic way without making the filter
> needlessly complicated?
> 
> In short: I'm not against whitelists per se, but without the specific
> use-case and a good plan of how to implement it without turning it
> into a coding/admin nightmare I don't see it happening.


I've read the code and I agree that implementing whitelist can be quite tricky.

Anyway, I do have one idea: let introduce flag -i (inverse). It should remove 
X-Spam: yes if matches with -m.

But I haven't see any easy way to implement it for non -m case.

During read the code of this filter I guess I've found third point which I'd 
like to raise: filter fails in the case when one of provided DNSBL returns 
error.

Shall it continue?

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 22:17, Martijn van Duren  
> wrote:
> 
> On Thu, 2023-12-28 at 20:05 +0100, Kirill A. Korinsky wrote:
>> 
>> 
>> Example of software is sieve: as far as I know it doesn't support matching 
>> case insensitive strings.
> 
> Does this currently pose a problem for anyone? If not I don't see a
> reason to make a new release for it.

Well, right now it should be written in sieve like this:

> if anyof(header :is "X-Spam" "yes", header :is "X-Spam" "Yes") {
>   fileinto :create "Junk";
> }

a bit ugly but I agree that it's not enough for a release.


>> 
>> I've read the code and I agree that implementing whitelist can be quite 
>> tricky.
>> 
>> Anyway, I do have one idea: let introduce flag -i (inverse). It should 
>> remove X-Spam: yes if matches with -m.
> 
> Why? What does this bring? Why do you need this?

Right now I'm making a kind of the experiment: living without any statistical 
analyser for my mail.

The hypotesa: usual mail traffic for family mail server is too small to make 
statistical analyser like spamassasian useful by impact of mails.

So, I'm testing another approach: only DNSBL which delivers everything that is 
matched into Junk folder.

Long story short I do have the following configuration:

 - NiX Spam and blocklist.de <http://blocklist.de/> at spamd
 - smptd configuration:

> filter dnsbl proc-exec "filter-dnsbl -m \
>all.spamrats.com \
>bip.virusfree.cz \
>bl.mailspike.org \
>bl.spamcop.net \
>bl.spameatingmonkey.net \
>cbl.abuseat.org \
>dnsbl-1.uceprotect.net \
>dnsbl.dronebl.org \
>mail-abuse.blacklist.jippg.org \
>psbl.surriel.com \
>rbl.0spam.org \
>truncate.gbudb.net \
>zen.spamhaus.org \
>XXX.combined.mail.abusix.zone \
>"
> 
> filter dnswl proc-exec "filter-dnsbl -m \
>XXX.white.mail.abusix.zone \
>"
> 
> filter "senderscore" proc-exec "filter-senderscore -junkBelow 70"
> 
> listen on egress inet4 port smtp tls pki mx1.catap.net \
>filter { "rdns", "fcrdns", dnsbl, dnswl, "senderscore" }


- and sieve script to move mail:

> if allof(
>   anyof(header :is "X-Spam" "yes", header :is "X-Spam" "Yes"),
>   not header :is "X-Spam-DNSBL" "Listed at white.mail.abusix.zone") {
> fileinto :create "Junk";
> }


this works like a charm, really.

The idea of both changes to use white lists to remove X-Spam: yes from both 
negative filters.

>> 
>> But I haven't see any easy way to implement it for non -m case.
>> 
>> During read the code of this filter I guess I've found third point which I'd 
>> like to raise: filter fails in the case when one of provided DNSBL returns 
>> error.
>> 
>> Shall it continue?
> 
> If a filter (or the intermediate DNS layer) returns an error we are in
> limbo. If we accept the mail, but it's listed we're probably delivering
> spam; if we reject the mail we're very likely to drop legit mail. Both
> are undesirable. Failing the message asking to try again later seems the
> safest option to me.

I see your point.

My point: user may wait messages and to be very nervous if it delayed for a 
while.

Important message means something like a ticket for a train in 5-15 minutes or 
something like that.

And here DNS seems like a single point of failure.

I think that it should be configurable by bypass DNS error by probability of 
delivering spam instead of delaying everything.

--
wbr, Kirill




signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 23:34, Martijn van Duren  
> wrote:
> 
> I've never used sieve, but this already is a custom rule and not a
> X-Spam specific header check from sieve itself. However, a quick
> online search shows me the i;ascii-casemap comparator. Maybe you
> can give that a try.
> 
> Also, smtpd.conf(5), from which I've taken the string, uses a lower
> case "yes".

Well... OpenSMTPD checks that value via strcasecmp to avoid case issue, see:
 
https://github.com/OpenSMTPD/OpenSMTPD/blob/7.4.0p1/usr.sbin/smtpd/mail.maildir.c#L192-L196

but setup it as Yes, see: 
https://github.com/OpenSMTPD/OpenSMTPD/blob/7.4.0p1/usr.sbin/smtpd/smtp_session.c#L2772-L2773


>> The idea of both changes to use white lists to remove X-Spam: yes from both 
>> negative filters.
> 
> The discussion about having a single whitelist overrule a whole sluice
> of lists apart. If you want whitelisting in filter-dnsbl it would
> require special handling of a whole range of different options:
> - Which response means what for what list

I assume any found response, like it works now with black listing.

> - Which option takes precedence in what situation

I assume that white list override the black one.

> - Do we need to remove existing X-Spam headers?

If we may do it in one run, we simple don't need to setup it.

> - Are there other headers that need removing/modifying?

I don't see any of this.

> This would make filter-dnsbl grow way beyond of what it was ever
> intended for into something where I'm afraid is not properly
> maintainable for both programmer and admin.
> 
> Maybe you can write your own filter-dnswl with filter-dnsbl and
> filter-admdscrub as inspiration.

Or I may make a series of patch over your code. It seems not that complicated.

 But I haven't see any easy way to implement it for non -m case.
 
 During read the code of this filter I guess I've found third point which 
 I'd like to raise: filter fails in the case when one of provided DNSBL 
 returns error.
 
 Shall it continue?
>>> 
>>> If a filter (or the intermediate DNS layer) returns an error we are in
>>> limbo. If we accept the mail, but it's listed we're probably delivering
>>> spam; if we reject the mail we're very likely to drop legit mail. Both
>>> are undesirable. Failing the message asking to try again later seems the
>>> safest option to me.
>> 
>> I see your point.
>> 
>> My point: user may wait messages and to be very nervous if it delayed for a 
>> while.
>> 
>> Important message means something like a ticket for a train in 5-15 minutes 
>> or something like that.
>> 
>> And here DNS seems like a single point of failure.
> 
> Sure, but if I'm in a hurry and need a ticket I'm not going to rely on
> mail anyway. Either I'm going to buy it at the door, or I hope they have
> an option to download the ticket from the browser (which most of the
> ticket purchases I make have an option for). Only as a last resort I'm
> going to rely mail and just hope that everything works as it should.

Well, this is an example from the last week :)

If I haven't open DB application for a while, more than a month it had missed 
updated of so-called Deutschlandticket, and I wait the email with approval code 
to re-download it to the application.

I know that is edge case, but DNS failure is also edge case.

>> I think that it should be configurable by bypass DNS error by probability of 
>> delivering spam instead of delaying everything.
> 
> Even as an option I'm not particularly fond of the idea... But if enough
> people think it's a worthwhile addition (you're the first one in 4 years
> to have raised this request) and the diff is small enough I might
> consider it.


From another hand, locally installed unbound should solve that issue on the 
first place.

--
wbr, Kirill




signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 23:46, Kirill A. Korinsky  wrote:
> 
>> On 28. Dec 2023, at 23:34, Martijn van Duren  
>> wrote:
>> This would make filter-dnsbl grow way beyond of what it was ever
>> intended for into something where I'm afraid is not properly
>> maintainable for both programmer and admin.
>> 
>> Maybe you can write your own filter-dnswl with filter-dnsbl and
>> filter-admdscrub as inspiration.
> 
> Or I may make a series of patch over your code. It seems not that complicated.

I mean something like that: https://github.com/catap/filter-dnsbl 
<https://github.com/catap/filter-dnsbl>

I've split it into two commits to make logic clear.


--
wbr, Kirill




signature.asc
Description: Message signed with OpenPGP

Run VM with 16G or more?

[Kirill A. Korinsky]

Greetings,

How can I run a VM with more than 16G of memory?

A naive approach fails with error:

> vmctl: start vm command failed: Cannot allocate memory


Yes, the host machine has that memory and much more.

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

Request review: updated filter-dnsbl

[Kirill A. Korinsky]

Folks,

I'd like to share with you an updated version of filter-dnsbl which is
available at https://github.com/catap/filter-dnsbl

The first changes it doesn't drop connection in case of DNS error when
-m is spcefied, instead it adds header X-Spam: unknown.

The second one is whitelists. It, again, works only with -m. It support
-w before each zone that turns it into whitelist zone. When anything is
found here, it supress X-Spam flag, if DNS error doesn't happened.

How it works. Let assume that we run this filter as:

filter-dnsbl -m \
all.spamrats.com \
combined.mail.abusix.zone \
-w white.mail.abusix.zone

When any DNS query returns error it adds: X-Spam: unknown

When any DNS query is matched it adds X-Spam-DNSBL or X-Spam-DNSWL with
value Listed at %s, the last one if -w specified before zone.

And it adds X-Spam: yes only if IP is matched against any of blacklists
and doesn't match against any whitelists.

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

Alisases, LMTP vs the Queue

[Kirill A . Korinsky]

Greetings,

Let assume that we have setup which defines a list of aliases:

  u...@domain.com:  u...@domain.com

and it delivering as:

  action deliver_lmtp lmtp "/var/dovecot/lmtp" rcpt-to virtual 
  match from any for any action deliver_lmtp

Unfortunately, this requires to add somewhere routing to vmail user.

A naive approach by adding to the end of aliases list:

  @: vmail

works, but it leads to queue all messages for non exist users.

Another approach is keeping at the end of aliases:

  u...@domain.com: vmail

what requires to duplicate list of known users from LMTP here, and makes error
for unknwon user quite wired, let me quote:

  524 5.2.4 Mailing list expansion problem: 

So, here the question: does it possible make this setup clean, without
duplicating information?

--
wbr, Kirill

Re: Alisases, LMTP vs the Queue

[Kirill A . Korinsky]

On Sat, 27 Jan 2024 11:26:11 +0100,
Kirill A. Korinsky wrote:
>
> So, here the question: does it possible make this setup clean, without
> duplicating information?
>

For example a solution which looks clean from my point of view.

Let assume that the setup is:

  action deliver_lmtp lmtp "/var/dovecot/lmtp" rcpt-to virtual  
userbase 
  match from any for any action deliver_lmtp

where credentials is password-like file with usual format:

  :::extra_fields

and we may use field  as the target user, if it isn't numeric. If no user
specified for a record, or no record after running aliases OpenSMTP should
return: user not exists, without an attempt of LMTP delivery.

This way it should be compatible with dovecot LMTP as well.

Thougths?

--
wbr, Kirill

Re: Status of filter-spf

[Kirill A . Korinsky]

On Mon, 29 Jan 2024 13:55:07 +0100,
Martijn van Duren wrote:
> 
> I've started filter-spf, but never gotten to finish it. No clue when
> or if I want to continue with it. At the moment my priorities are
> somewhere else.
> 

Got it.

Meanwhile I've discovered spfwalk.c which should speed up my work dramatically.

-- 
wbr, Kirill

Capture core for OpenSMTPD filter

[Kirill A . Korinsky]

Greetings,

I've finished the first version of opensmtpd-filter-auth and I've tried to use
it on real world traffic.

As expected it crashed instantly but without any core.

I've tried sysctl kern.nosuidcoredump with value 2 and 3, tried to create
/var/crash/filter-auth and /var/crash/smtpd, but it also doesn't help.

I had created /etc/login.conf.d/smtpd

  smtpd:\
  :coredumpsize=unlimited:\
  :tc=daemon:

without any sucess.

So, how can I enable capture core dump for OpenSMTPD filters on OpenBSD 7.4?

I feel that I've missed something, really.

--
wbr, Kirill