Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Avamander 
> Making arguments based upon extreme cases, assumptions, or
potential-for-collateral-damage is not scientific. "I know one that even
[...]" Anecdotal  evidence isn't scientific.

>From the perspective of your previous sentences that's kinda humorous. "To
avoid unnecessary costs incurred from disruption of service, excessive
traffic, annoyances using up *my* time, and countless other reasonable
rationale from *my* point of view." Because sure, a few (hypothetical RIPE
probe) connections are exactly that, zero exaggeration, right?

In the end such fail2ban-fueled (or similar) behaviour I initially
addressed, is exactly a non-scientific extreme-case assumption-based
approach. There are better and even more standard ways.

Crutch solutions out in the wild shouldn't be a showstopper for measuring
the ecosystem. (That is already quite neglected)

> What _objective_ risk/benefit analysis are you basing your opinions upon?

And you? What's the implication here about systems being as trigger-happy
as previously described?

Because sure, at some point rate limits make total sense, but certainly not
at the point where it would ban any potential RIPE probes.

>  Are you a systems administrator?

Let's not get into such measuring contests, even if it is the RIPE Atlas
mailing list.

On Tue, Sep 20, 2022 at 11:42 PM Paul Theodoropoulos via ripe-atlas <
ripe-atlas@ripe.net> wrote:

> On 9/20/2022 10:45 AM, Avamander wrote:
>
> Great to hear it works for you, but the potential unfortunate collateral
> from such a blanket action is not really RIPE Atlas' problem. There are
> more fine-grained methods against bruteforce attempts and open relay
> probes, than triggering on a few connections.
>
> What _objective_ risk/benefit analysis are you basing your opinions upon?
> Are you a systems administrator? My responsibility is to avoid unnecessary
> costs incurred from disruption of service, excessive traffic, annoyances
> using up *my* time, and countless other reasonable rationale from *my*
> point of view.
>
> You suggest that it is "not really RIPE Atlas' problem". That's very true.
> And it is not really my problem if I bounce yoinky, pointless probes of my
> server, and ruthlessly block them from contacting my server ever again. My
> server, my choice, my wallet, nobody's business but my own.
>
> Some webmasters ban IP's for simply visiting a domain, I know one that
> even dispatches an email to your ISP's abuse@ address upon visit. Should
> RIPE Atlas probes then not probe any HTTP servers? The answer is obviously
> no, they shouldn't care.
>
> Making arguments based upon extreme cases, assumptions, or
> potential-for-collateral-damage is not scientific. "I know one that even
> [...]" Anecdotal  evidence isn't scientific.
>
> Note, I run a probe myself. I don't block any RIPE Atlas traffic on my
> separate servers hosted on AWS, Oracle, and GCE.
>
> --
> Paul Theodoropoulos
> anastrophe.com 
> --
> ripe-atlas mailing list
> ripe-atlas@ripe.net
> https://lists.ripe.net/mailman/listinfo/ripe-atlas
>
-- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Paul Theodoropoulos via ripe-atlas 

(Coffee is to blame for the length of this)
It's worth bearing in mind that the internet is a large place, populated 
with large, larger, very large, enormous, gargantuanand extremely 
tiny, extremely budget-limited, personal, private servers that cannot 
afford the charges for non-productive traffic. The internet is not just 
Proofpoint or Sendgrid or "Meta" or etc.; it is also vast numbers of 
people in second- and third-world countries trying to engage with the 
world and learn, and who literally can't afford excess, unproductive bytes 
hitting their server that will cost them money.


I used to be the sole systems admin for a site that sent variously 
anywhere from a million to eleven million emails per day (yes, all opt-in, 
legitimate email for our customers, just for the record), so I've spent a 
fair bit of my life going zen analyzing logs for hours and hours. I'm now 
retired and run my own personal mail and webservers, on one t3.micro on 
AWS that costs me a few hundred bucks over the course of three years 
commitment, and a free server on each of GCE and Oracle, the former of 
which also costs me a few bucks a month in traffic charges.


I'm not poor, and can afford these charges. So I don't block any RIPE 
Atlas traffic (to my knowledge), as I don't have (or need) extreme 
restrictions on incoming traffic. I do occasionally sift through my logs 
to find new pointless traffic that's 'annoying' my systems, and may set 
something up to block them. I run a couple of stratum one timeservers from 
home, part of the ntp pool project, and regularly have to block nitwits 
(some of them ISP's themselves) who hammer my little primary RPI with 
multiple connections per minute. I have to block those malefactors simply 
because they interfere with legitimate traffic. There are countless ways 
that 'internet hooligans' and/or the clueless can cause harm with useless 
traffic.


There are potentially millions of individuals trying to contribute for the 
benefit of the internet who don't have the lack of concern for costs that 
I have. Thus why there are so many Raspberry Pi probes out there in 
non-first-world nations; and likely no small number of tiny personal 
mailservers running on a shared 56k line, the cost of which is _just_ 
bearable to maintain.


There is no one-size-fits-all on the internet. It is common - and very 
human - to view the internet and world through our own perspective.


There are other perspectives.

Okay, that's my coffee-fueled bloviation for the day, and with that, I 
acknowledge the significant digression involved...so I'll shut up now.


Cheers!

On 9/20/2022 13:04 PM, Simon Brandt via ripe-atlas wrote:
There's literally no danger for you (recipient), if the sender 
terminates the connection before the an e-mail has been successfully 
transmitted. No need to ban the ip address. You would only risk to block 
a legitimate ip address which might have trouble sending e-mails to you. 
If you want to do this because you don't want to see such connections in 
you logs that's fine. But i do not share your opinion, that this is a 
common configuration.


However, we have digressed from the topic.

BR,
Simon



On 20.09.22 21:47, Eric Kuhnke wrote:
No legitimate incoming SMTP traffic comes from IPs that abort 
connections to my mx prior to attempting to deliver mail, so however I 
choose to declutter my log files has absolutely zero real world impact 
in deliverablity of legitimate incoming mail. Nor does it hurt anybody 
at the other end whatever the connect-and-do-nothing software at the 
other side.


On Tue, Sep 20, 2022, 12:37 PM  wrote:

>> because common configurations of fail2ban [...] absolutely will
ban your IP [...] after multiple connection attempts and no
successful mail transfer

I would consider this a heavy misconfiguration. Please explain to
me what incomplete SMTP connections have in common with spammers,
virus/worm/trojan compromised hosts or open relay searching bots.
Those bad senders WANT to _successfully_ deliver mails to you. They
will never abort the connection on purpose. For example: bots which
search for open relays ALWAYS try to send mails with a foreign
sender and recipient domain. That's how you discover them. But as
suggested, the Atlas SMTP check should not send E-Mails at all, not
even send MAIL FROM: or RCPT TO: command.

You will not achieve mitigation of inbound spam/malware/phishing
traffic by blocking IP addresses of hosts from incomplete SMTP
sessions. Usually, incomplete SMTP sessions indicate a
misconfiguration. For example: forced TLS enabled, but expired
certificate or no matching cipher suites. But that is no reason to
ban the senders! I think you have to go a little bit deeper in your
logs and consider why mailtransfer was not successfull, before
blocking ip addresses.

I am no expert for fail2ban, but as far is i know, i searches for
failed login attem

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Paul Theodoropoulos via ripe-atlas 

On 9/20/2022 10:45 AM, Avamander wrote:
Great to hear it works for you, but the potential unfortunate collateral 
from such a blanket action is not really RIPE Atlas' problem. There are 
more fine-grained methods against bruteforce attempts and open relay 
probes, than triggering on a few connections.
What _objective_ risk/benefit analysis are you basing your opinions upon? 
Are you a systems administrator? My responsibility is to avoid unnecessary 
costs incurred from disruption of service, excessive traffic, annoyances 
using up *my* time, and countless other reasonable rationale from *my* 
point of view.


You suggest that it is "not really RIPE Atlas' problem". That's very true. 
And it is not really my problem if I bounce yoinky, pointless probes of my 
server, and ruthlessly block them from contacting my server ever again. My 
server, my choice, my wallet, nobody's business but my own.
Some webmasters ban IP's for simply visiting a domain, I know one that 
even dispatches an email to your ISP's abuse@ address upon visit. Should 
RIPE Atlas probes then not probe any HTTP servers? The answer is 
obviously no, they shouldn't care.
Making arguments based upon extreme cases, assumptions, or 
potential-for-collateral-damage is not scientific. "I know one that even 
[...]" Anecdotal  evidence isn't scientific.


Note, I run a probe myself. I don't block any RIPE Atlas traffic on my 
separate servers hosted on AWS, Oracle, and GCE.


--
Paul Theodoropoulos
anastrophe.com -- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Simon Brandt via ripe-atlas 

There's literally no danger for you (recipient), if the sender terminates the 
connection before the an e-mail has been successfully transmitted. No need to 
ban the ip address. You would only risk to block a legitimate ip address which 
might have trouble sending e-mails to you. If you want to do this because you 
don't want to see such connections in you logs that's fine. But i do not share 
your opinion, that this is a common configuration.

However, we have digressed from the topic.

BR,
Simon



On 20.09.22 21:47, Eric Kuhnke wrote:

No legitimate incoming SMTP traffic comes from IPs that abort connections to my 
mx prior to attempting to deliver mail, so however I choose to declutter my log 
files has absolutely zero real world impact in deliverablity of legitimate 
incoming mail. Nor does it hurt anybody at the other end whatever the 
connect-and-do-nothing software at the other side.

On Tue, Sep 20, 2022, 12:37 PM  wrote:

>> because common configurations of fail2ban [...] absolutely will ban your 
IP [...] after multiple connection attempts and no successful mail transfer

I would consider this a heavy misconfiguration. Please explain to me what 
incomplete SMTP connections have in common with spammers, virus/worm/trojan 
compromised hosts or open relay searching bots. Those bad senders WANT to 
_successfully_ deliver mails to you. They will never abort the connection on 
purpose. For example: bots which search for open relays ALWAYS try to send 
mails with a foreign sender and recipient domain. That's how you discover them. 
But as suggested, the Atlas SMTP check should not send E-Mails at all, not even 
send MAIL FROM: or RCPT TO: command.

You will not achieve mitigation of inbound spam/malware/phishing traffic by 
blocking IP addresses of hosts from incomplete SMTP sessions. Usually, 
incomplete SMTP sessions indicate a misconfiguration. For example: forced TLS 
enabled, but expired certificate or no matching cipher suites. But that is no 
reason to ban the senders! I think you have to go a little bit deeper in your 
logs and consider why mailtransfer was not successfull, before blocking ip 
addresses.

I am no expert for fail2ban, but as far is i know, i searches for failed 
login attempts. So that affects mostly authenticated SMTP connections (client 
E-Mail submission on tcp/465 or tcp/587), right? This should not concern us 
here.

I work with enterprise mailgateway solutions for years (mostly Proofpoint), 
but i have never encountered what you describe.

Reject or throttle because of too much connections at the same time? Yes.
Reject or throttle because of too much non-existing recipient adresses? Yes.
Reject or throttle because both sender and recipient domain is foreign? Yes.
Reject or throttle because of bad reputation (known spammer or TOR exit 
node ip address)? Yes.

But not because of incomplete SMTP connections. From my point of view, I 
can not confirm that this common behaviour.

BR,
Simon


On 20.09.22 19:22, Eric Kuhnke wrote:

I would discourage anyone from relying upon the data from "probing" the MX 
and SMTP daemons for a domain name no matter what port they run on, because common 
configurations of fail2ban used with postfix and others absolutely will ban your IP at 
the host operating system level (iptables or other) after multiple connection attempts 
and no successful mail transfer or authentication.

a probe of smtpd will look not much different from the many things out 
there on the internet already maliciously probing smtpd trying to find open 
relays.




On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas 
 wrote:

Hi Michel,

Currently, HTTP and SSL are separate measurements. But for SMTP it will 
probably not work this way... Encryption is not mandatory for SMTP and thus 
negotiated between client and server every time on port 25. Ports 465 and 587 
are used for Client Email Submission. You could run some measurements for these 
ports as well, but for the beginning, i would focus on server2server 
communication.

So here's what i think:

What we could measure:
- General reachability/availability of the e-mail service
- Response time of the remote server: time between connection establish 
and first SMTP response (220 service ready)
- Which enhanced status codes are offered by the server?
- Forward/Reverse DNS matching?
- SMTP banner matching the DNS name?
    - if not: what is it?
- Does the remote server offer encryption (250-STARTTLS)
- Which cipher settings are offered by the server (SSL/TLS Version, Key 
Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
    - alternatively: Which cipher setting has been negotiated between 
probe and server?
- Can we successfully establish a TLS connection?
- Certificate Check: is the server-certificate valid and does it ma

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Eric Kuhnke 
No legitimate incoming SMTP traffic comes from IPs that abort connections
to my mx prior to attempting to deliver mail, so however I choose to
declutter my log files has absolutely zero real world impact in
deliverablity of legitimate incoming mail. Nor does it hurt anybody at the
other end whatever the connect-and-do-nothing software at the other side.

On Tue, Sep 20, 2022, 12:37 PM  wrote:

> >> because common configurations of fail2ban [...] absolutely will ban
> your IP [...] after multiple connection attempts and no successful mail
> transfer
>
> I would consider this a heavy misconfiguration. Please explain to me what
> incomplete SMTP connections have in common with spammers, virus/worm/trojan
> compromised hosts or open relay searching bots. Those bad senders WANT to
> *successfully* deliver mails to you. They will never abort the connection
> on purpose. For example: bots which search for open relays ALWAYS try to
> send mails with a foreign sender and recipient domain. That's how you
> discover them. But as suggested, the Atlas SMTP check should not send
> E-Mails at all, not even send MAIL FROM: or RCPT TO: command.
>
> You will not achieve mitigation of inbound spam/malware/phishing traffic
> by blocking IP addresses of hosts from incomplete SMTP sessions. Usually,
> incomplete SMTP sessions indicate a misconfiguration. For example: forced
> TLS enabled, but expired certificate or no matching cipher suites. But that
> is no reason to ban the senders! I think you have to go a little bit deeper
> in your logs and consider why mailtransfer was not successfull, before
> blocking ip addresses.
>
> I am no expert for fail2ban, but as far is i know, i searches for failed
> login attempts. So that affects mostly authenticated SMTP connections
> (client E-Mail submission on tcp/465 or tcp/587), right? This should not
> concern us here.
>
> I work with enterprise mailgateway solutions for years (mostly
> Proofpoint), but i have never encountered what you describe.
>
> Reject or throttle because of too much connections at the same time? Yes.
> Reject or throttle because of too much non-existing recipient adresses?
> Yes.
> Reject or throttle because both sender and recipient domain is foreign?
> Yes.
> Reject or throttle because of bad reputation (known spammer or TOR exit
> node ip address)? Yes.
>
> But not because of incomplete SMTP connections. From my point of view, I
> can not confirm that this common behaviour.
>
> BR,
> Simon
>
>
> On 20.09.22 19:22, Eric Kuhnke wrote:
>
> I would discourage anyone from relying upon the data from "probing" the MX
> and SMTP daemons for a domain name no matter what port they run on, because
> common configurations of fail2ban used with postfix and others absolutely
> will ban your IP at the host operating system level (iptables or other)
> after multiple connection attempts and no successful mail transfer or
> authentication.
>
> a probe of smtpd will look not much different from the many things out
> there on the internet already maliciously probing smtpd trying to find open
> relays.
>
>
>
>
> On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas <
> ripe-atlas@ripe.net> wrote:
>
>> Hi Michel,
>>
>> Currently, HTTP and SSL are separate measurements. But for SMTP it will
>> probably not work this way... Encryption is not mandatory for SMTP and thus
>> negotiated between client and server every time on port 25. Ports 465 and
>> 587 are used for Client Email Submission. You could run some measurements
>> for these ports as well, but for the beginning, i would focus on
>> server2server communication.
>>
>> So here's what i think:
>>
>> What we could measure:
>> - General reachability/availability of the e-mail service
>> - Response time of the remote server: time between connection establish
>> and first SMTP response (220 service ready)
>> - Which enhanced status codes are offered by the server?
>> - Forward/Reverse DNS matching?
>> - SMTP banner matching the DNS name?
>> - if not: what is it?
>> - Does the remote server offer encryption (250-STARTTLS)
>> - Which cipher settings are offered by the server (SSL/TLS Version, Key
>> Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
>> - alternatively: Which cipher setting has been negotiated between
>> probe and server?
>> - Can we successfully establish a TLS connection?
>> - Certificate Check: is the server-certificate valid and does it match
>> the hostname?
>> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
>> 'report'?
>> - Forced Authentication: DANE available and check successfull?
>>
>>
>> What we should not do:
>> - send MAIL FROM: command
>> - send RCPT TO: command
>> - send DATA command
>> - measure e-mail delivery/roundtrip time, etc.
>> - Sending e-mails at all
>>
>> The Atlas probe should quit the connection after the data is collected.
>> An actual e-mail should not be send. The target mailserver would count this
>> session as "incomplete" or "aborted" whi

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Simon Brandt via ripe-atlas 

>> because common configurations of fail2ban [...] absolutely will ban your IP 
[...] after multiple connection attempts and no successful mail transfer

I would consider this a heavy misconfiguration. Please explain to me what 
incomplete SMTP connections have in common with spammers, virus/worm/trojan 
compromised hosts or open relay searching bots. Those bad senders WANT to 
_successfully_ deliver mails to you. They will never abort the connection on 
purpose. For example: bots which search for open relays ALWAYS try to send 
mails with a foreign sender and recipient domain. That's how you discover them. 
But as suggested, the Atlas SMTP check should not send E-Mails at all, not even 
send MAIL FROM: or RCPT TO: command.

You will not achieve mitigation of inbound spam/malware/phishing traffic by 
blocking IP addresses of hosts from incomplete SMTP sessions. Usually, 
incomplete SMTP sessions indicate a misconfiguration. For example: forced TLS 
enabled, but expired certificate or no matching cipher suites. But that is no 
reason to ban the senders! I think you have to go a little bit deeper in your 
logs and consider why mailtransfer was not successfull, before blocking ip 
addresses.

I am no expert for fail2ban, but as far is i know, i searches for failed login 
attempts. So that affects mostly authenticated SMTP connections (client E-Mail 
submission on tcp/465 or tcp/587), right? This should not concern us here.

I work with enterprise mailgateway solutions for years (mostly Proofpoint), but 
i have never encountered what you describe.

Reject or throttle because of too much connections at the same time? Yes.
Reject or throttle because of too much non-existing recipient adresses? Yes.
Reject or throttle because both sender and recipient domain is foreign? Yes.
Reject or throttle because of bad reputation (known spammer or TOR exit node ip 
address)? Yes.

But not because of incomplete SMTP connections. From my point of view, I can 
not confirm that this common behaviour.

BR,
Simon


On 20.09.22 19:22, Eric Kuhnke wrote:

I would discourage anyone from relying upon the data from "probing" the MX and 
SMTP daemons for a domain name no matter what port they run on, because common 
configurations of fail2ban used with postfix and others absolutely will ban your IP at 
the host operating system level (iptables or other) after multiple connection attempts 
and no successful mail transfer or authentication.

a probe of smtpd will look not much different from the many things out there on 
the internet already maliciously probing smtpd trying to find open relays.




On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas 
 wrote:

Hi Michel,

Currently, HTTP and SSL are separate measurements. But for SMTP it will 
probably not work this way... Encryption is not mandatory for SMTP and thus 
negotiated between client and server every time on port 25. Ports 465 and 587 
are used for Client Email Submission. You could run some measurements for these 
ports as well, but for the beginning, i would focus on server2server 
communication.

So here's what i think:

What we could measure:
- General reachability/availability of the e-mail service
- Response time of the remote server: time between connection establish and 
first SMTP response (220 service ready)
- Which enhanced status codes are offered by the server?
- Forward/Reverse DNS matching?
- SMTP banner matching the DNS name?
    - if not: what is it?
- Does the remote server offer encryption (250-STARTTLS)
- Which cipher settings are offered by the server (SSL/TLS Version, Key 
Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
    - alternatively: Which cipher setting has been negotiated between probe 
and server?
- Can we successfully establish a TLS connection?
- Certificate Check: is the server-certificate valid and does it match the 
hostname?
- Forced Encryption: MTA-STS available and 'enforced' or 'testing' or 
'report'?
- Forced Authentication: DANE available and check successfull?


What we should not do:
- send MAIL FROM: command
- send RCPT TO: command
- send DATA command
- measure e-mail delivery/roundtrip time, etc.
- Sending e-mails at all

The Atlas probe should quit the connection after the data is collected. An actual e-mail should 
not be send. The target mailserver would count this session as "incomplete" or 
"aborted" which is totally fine. If someone would want to monitor what happens after a 
mailserver has successfully accepted a testmail, he should better use a monitoring 
service/solution. We measure the INTERnet, not what comes after (Intra) :)

I don't expect any "spam" problems. Since the Atlas probes wouldn't send e-mails, there's 
nothing a spamfilter could analyze. The only thing that could theoretically happen, is that the probes ip 
addresses are flagged as bad by services like Spamhaus etc. and thus be listed

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Avamander 
>  but has had the greatly beneficial effect of cutting down on the
absolutely incredible volume of virus/worm/trojan compromised hosts out on
the internet that are probing for open relays or trying repeated
login/password combinations, which does nothing but clutter up log files.

Great to hear it works for you, but the potential unfortunate collateral
from such a blanket action is not really RIPE Atlas' problem. There are
more fine-grained methods against bruteforce attempts and open relay
probes, than triggering on a few connections.

Some webmasters ban IP's for simply visiting a domain, I know one that even
dispatches an email to your ISP's abuse@ address upon visit. Should RIPE
Atlas probes then not probe any HTTP servers? The answer is obviously
no, they shouldn't care.

On Tue, Sep 20, 2022 at 8:39 PM Eric Kuhnke  wrote:

> I have the various postfix smtpd that I control set to be what you would
> call "trigger happy" and it has had absolutely **zero** impact blocking
> legitimate incoming smtp traffic from other domains' legitimate MX, but has
> had the greatly beneficial effect of cutting down on the absolutely
> incredible volume of virus/worm/trojan compromised hosts out on the
> internet that are probing for open relays or trying repeated login/password
> combinations, which does nothing but clutter up log files.
>
> A single connection to check that my MX answers and exists on ports 25 or
> other is not enough to trigger it but multiple repeated same attempts
> coming from the same origin IP in a span of multiple hours will.
>
>
>
>
>
> On Tue, 20 Sept 2022 at 10:29, Avamander  wrote:
>
>> > a probe of smtpd will look not much different from the many things out
>> there on the internet already maliciously probing smtpd trying to find open
>> relays.
>>
>> It would look very different though, there's a large difference between
>> trying to probe for an open relay and just making a connection without any
>> specific commands being issued.
>>
>> > because common configurations of fail2ban used with postfix and others
>> absolutely will ban your IP at the host operating system level (iptables or
>> other) after multiple connection attempts and no successful mail transfer
>> or authentication.
>>
>> If a few connections are sufficient to trigger that, then such
>> configurations are simply too trigger-happy. Checking the existence and
>> availability of a domain's MX is a very common operation. Things like
>> rspamd have such functionality built-in and I'm sure there are many others.
>>
>> On Tue, Sep 20, 2022 at 8:23 PM Eric Kuhnke 
>> wrote:
>>
>>> I would discourage anyone from relying upon the data from "probing" the
>>> MX and SMTP daemons for a domain name no matter what port they run on,
>>> because common configurations of fail2ban used with postfix and others
>>> absolutely will ban your IP at the host operating system level (iptables or
>>> other) after multiple connection attempts and no successful mail transfer
>>> or authentication.
>>>
>>> a probe of smtpd will look not much different from the many things out
>>> there on the internet already maliciously probing smtpd trying to find open
>>> relays.
>>>
>>>
>>>
>>>
>>> On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas <
>>> ripe-atlas@ripe.net> wrote:
>>>
 Hi Michel,

 Currently, HTTP and SSL are separate measurements. But for SMTP it will
 probably not work this way... Encryption is not mandatory for SMTP and thus
 negotiated between client and server every time on port 25. Ports 465 and
 587 are used for Client Email Submission. You could run some measurements
 for these ports as well, but for the beginning, i would focus on
 server2server communication.

 So here's what i think:

 What we could measure:
 - General reachability/availability of the e-mail service
 - Response time of the remote server: time between connection establish
 and first SMTP response (220 service ready)
 - Which enhanced status codes are offered by the server?
 - Forward/Reverse DNS matching?
 - SMTP banner matching the DNS name?
 - if not: what is it?
 - Does the remote server offer encryption (250-STARTTLS)
 - Which cipher settings are offered by the server (SSL/TLS Version, Key
 Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
 - alternatively: Which cipher setting has been negotiated between
 probe and server?
 - Can we successfully establish a TLS connection?
 - Certificate Check: is the server-certificate valid and does it match
 the hostname?
 - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
 'report'?
 - Forced Authentication: DANE available and check successfull?


 What we should not do:
 - send MAIL FROM: command
 - send RCPT TO: command
 - send DATA command
 - measure e-mail delivery/roundtrip time, etc.
 - Sending e-mails at all

 The

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Eric Kuhnke 
I have the various postfix smtpd that I control set to be what you would
call "trigger happy" and it has had absolutely **zero** impact blocking
legitimate incoming smtp traffic from other domains' legitimate MX, but has
had the greatly beneficial effect of cutting down on the absolutely
incredible volume of virus/worm/trojan compromised hosts out on the
internet that are probing for open relays or trying repeated login/password
combinations, which does nothing but clutter up log files.

A single connection to check that my MX answers and exists on ports 25 or
other is not enough to trigger it but multiple repeated same attempts
coming from the same origin IP in a span of multiple hours will.





On Tue, 20 Sept 2022 at 10:29, Avamander  wrote:

> > a probe of smtpd will look not much different from the many things out
> there on the internet already maliciously probing smtpd trying to find open
> relays.
>
> It would look very different though, there's a large difference between
> trying to probe for an open relay and just making a connection without any
> specific commands being issued.
>
> > because common configurations of fail2ban used with postfix and others
> absolutely will ban your IP at the host operating system level (iptables or
> other) after multiple connection attempts and no successful mail transfer
> or authentication.
>
> If a few connections are sufficient to trigger that, then such
> configurations are simply too trigger-happy. Checking the existence and
> availability of a domain's MX is a very common operation. Things like
> rspamd have such functionality built-in and I'm sure there are many others.
>
> On Tue, Sep 20, 2022 at 8:23 PM Eric Kuhnke  wrote:
>
>> I would discourage anyone from relying upon the data from "probing" the
>> MX and SMTP daemons for a domain name no matter what port they run on,
>> because common configurations of fail2ban used with postfix and others
>> absolutely will ban your IP at the host operating system level (iptables or
>> other) after multiple connection attempts and no successful mail transfer
>> or authentication.
>>
>> a probe of smtpd will look not much different from the many things out
>> there on the internet already maliciously probing smtpd trying to find open
>> relays.
>>
>>
>>
>>
>> On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas <
>> ripe-atlas@ripe.net> wrote:
>>
>>> Hi Michel,
>>>
>>> Currently, HTTP and SSL are separate measurements. But for SMTP it will
>>> probably not work this way... Encryption is not mandatory for SMTP and thus
>>> negotiated between client and server every time on port 25. Ports 465 and
>>> 587 are used for Client Email Submission. You could run some measurements
>>> for these ports as well, but for the beginning, i would focus on
>>> server2server communication.
>>>
>>> So here's what i think:
>>>
>>> What we could measure:
>>> - General reachability/availability of the e-mail service
>>> - Response time of the remote server: time between connection establish
>>> and first SMTP response (220 service ready)
>>> - Which enhanced status codes are offered by the server?
>>> - Forward/Reverse DNS matching?
>>> - SMTP banner matching the DNS name?
>>> - if not: what is it?
>>> - Does the remote server offer encryption (250-STARTTLS)
>>> - Which cipher settings are offered by the server (SSL/TLS Version, Key
>>> Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
>>> - alternatively: Which cipher setting has been negotiated between
>>> probe and server?
>>> - Can we successfully establish a TLS connection?
>>> - Certificate Check: is the server-certificate valid and does it match
>>> the hostname?
>>> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
>>> 'report'?
>>> - Forced Authentication: DANE available and check successfull?
>>>
>>>
>>> What we should not do:
>>> - send MAIL FROM: command
>>> - send RCPT TO: command
>>> - send DATA command
>>> - measure e-mail delivery/roundtrip time, etc.
>>> - Sending e-mails at all
>>>
>>> The Atlas probe should quit the connection after the data is collected.
>>> An actual e-mail should not be send. The target mailserver would count this
>>> session as "incomplete" or "aborted" which is totally fine. If someone
>>> would want to monitor what happens after a mailserver has successfully
>>> accepted a testmail, he should better use a monitoring service/solution. We
>>> measure the INTERnet, not what comes after (Intra) :)
>>>
>>> I don't expect any "spam" problems. Since the Atlas probes wouldn't send
>>> e-mails, there's nothing a spamfilter could analyze. The only thing that
>>> could theoretically happen, is that the probes ip addresses are flagged as
>>> bad by services like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i
>>> really don't see a reason why that should happen. There wouldn't be any
>>> activity originating from the probes which could be classified as bad. IP
>>> addresses are normally only blacklisted,

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Avamander 
> a probe of smtpd will look not much different from the many things out
there on the internet already maliciously probing smtpd trying to find open
relays.

It would look very different though, there's a large difference between
trying to probe for an open relay and just making a connection without any
specific commands being issued.

> because common configurations of fail2ban used with postfix and others
absolutely will ban your IP at the host operating system level (iptables or
other) after multiple connection attempts and no successful mail transfer
or authentication.

If a few connections are sufficient to trigger that, then such
configurations are simply too trigger-happy. Checking the existence and
availability of a domain's MX is a very common operation. Things like
rspamd have such functionality built-in and I'm sure there are many others.

On Tue, Sep 20, 2022 at 8:23 PM Eric Kuhnke  wrote:

> I would discourage anyone from relying upon the data from "probing" the MX
> and SMTP daemons for a domain name no matter what port they run on, because
> common configurations of fail2ban used with postfix and others absolutely
> will ban your IP at the host operating system level (iptables or other)
> after multiple connection attempts and no successful mail transfer or
> authentication.
>
> a probe of smtpd will look not much different from the many things out
> there on the internet already maliciously probing smtpd trying to find open
> relays.
>
>
>
>
> On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas <
> ripe-atlas@ripe.net> wrote:
>
>> Hi Michel,
>>
>> Currently, HTTP and SSL are separate measurements. But for SMTP it will
>> probably not work this way... Encryption is not mandatory for SMTP and thus
>> negotiated between client and server every time on port 25. Ports 465 and
>> 587 are used for Client Email Submission. You could run some measurements
>> for these ports as well, but for the beginning, i would focus on
>> server2server communication.
>>
>> So here's what i think:
>>
>> What we could measure:
>> - General reachability/availability of the e-mail service
>> - Response time of the remote server: time between connection establish
>> and first SMTP response (220 service ready)
>> - Which enhanced status codes are offered by the server?
>> - Forward/Reverse DNS matching?
>> - SMTP banner matching the DNS name?
>> - if not: what is it?
>> - Does the remote server offer encryption (250-STARTTLS)
>> - Which cipher settings are offered by the server (SSL/TLS Version, Key
>> Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
>> - alternatively: Which cipher setting has been negotiated between
>> probe and server?
>> - Can we successfully establish a TLS connection?
>> - Certificate Check: is the server-certificate valid and does it match
>> the hostname?
>> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
>> 'report'?
>> - Forced Authentication: DANE available and check successfull?
>>
>>
>> What we should not do:
>> - send MAIL FROM: command
>> - send RCPT TO: command
>> - send DATA command
>> - measure e-mail delivery/roundtrip time, etc.
>> - Sending e-mails at all
>>
>> The Atlas probe should quit the connection after the data is collected.
>> An actual e-mail should not be send. The target mailserver would count this
>> session as "incomplete" or "aborted" which is totally fine. If someone
>> would want to monitor what happens after a mailserver has successfully
>> accepted a testmail, he should better use a monitoring service/solution. We
>> measure the INTERnet, not what comes after (Intra) :)
>>
>> I don't expect any "spam" problems. Since the Atlas probes wouldn't send
>> e-mails, there's nothing a spamfilter could analyze. The only thing that
>> could theoretically happen, is that the probes ip addresses are flagged as
>> bad by services like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i
>> really don't see a reason why that should happen. There wouldn't be any
>> activity originating from the probes which could be classified as bad. IP
>> addresses are normally only blacklisted, if they send unwanted mails like
>> spam. Also: there are a lot of "check you mailserver" or "check your
>> SSL/TLS" websites. The RIPE Atlas probes would behave the same way. No big
>> deal.
>>
>> Maybe we can think of an "extended" SMTP measurement where RIPE Atlas
>> sends actual e-mails, but that would require (in my opinion), that the
>> person who is creating the measurement somehow provides proof, to be the
>> owner of the target mailserver.
>>
>>
>> BR,
>> Simon
>>
>>
>> On 15.09.22 12:03, Michel Stam wrote:
>>
>> Hello Simon,
>>
>> Thank you for the suggestion.
>>
>> I have a couple of questions to get a better idea:
>>
>>- Can you maybe describe what a SMTP measurement would look like?
>>   - Simple EHLO/HELO
>>   - Sending an email to a designated address (which would then
>>   validate that the SMTP server is capable of re

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Eric Kuhnke 
I should add that having RIPE atlas probes in random IP blocks begin
probing peoples' smtpd without successfully authenticating or transferring
mail un a very high likelihood of getting those particular IPs or parent
netblocks added to lists of abusive smtp traffic, which is not something
that many probe hosts want to see happen.


On Tue, 20 Sept 2022 at 10:22, Eric Kuhnke  wrote:

> I would discourage anyone from relying upon the data from "probing" the MX
> and SMTP daemons for a domain name no matter what port they run on, because
> common configurations of fail2ban used with postfix and others absolutely
> will ban your IP at the host operating system level (iptables or other)
> after multiple connection attempts and no successful mail transfer or
> authentication.
>
> a probe of smtpd will look not much different from the many things out
> there on the internet already maliciously probing smtpd trying to find open
> relays.
>
>
>
>
> On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas <
> ripe-atlas@ripe.net> wrote:
>
>> Hi Michel,
>>
>> Currently, HTTP and SSL are separate measurements. But for SMTP it will
>> probably not work this way... Encryption is not mandatory for SMTP and thus
>> negotiated between client and server every time on port 25. Ports 465 and
>> 587 are used for Client Email Submission. You could run some measurements
>> for these ports as well, but for the beginning, i would focus on
>> server2server communication.
>>
>> So here's what i think:
>>
>> What we could measure:
>> - General reachability/availability of the e-mail service
>> - Response time of the remote server: time between connection establish
>> and first SMTP response (220 service ready)
>> - Which enhanced status codes are offered by the server?
>> - Forward/Reverse DNS matching?
>> - SMTP banner matching the DNS name?
>> - if not: what is it?
>> - Does the remote server offer encryption (250-STARTTLS)
>> - Which cipher settings are offered by the server (SSL/TLS Version, Key
>> Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
>> - alternatively: Which cipher setting has been negotiated between
>> probe and server?
>> - Can we successfully establish a TLS connection?
>> - Certificate Check: is the server-certificate valid and does it match
>> the hostname?
>> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
>> 'report'?
>> - Forced Authentication: DANE available and check successfull?
>>
>>
>> What we should not do:
>> - send MAIL FROM: command
>> - send RCPT TO: command
>> - send DATA command
>> - measure e-mail delivery/roundtrip time, etc.
>> - Sending e-mails at all
>>
>> The Atlas probe should quit the connection after the data is collected.
>> An actual e-mail should not be send. The target mailserver would count this
>> session as "incomplete" or "aborted" which is totally fine. If someone
>> would want to monitor what happens after a mailserver has successfully
>> accepted a testmail, he should better use a monitoring service/solution. We
>> measure the INTERnet, not what comes after (Intra) :)
>>
>> I don't expect any "spam" problems. Since the Atlas probes wouldn't send
>> e-mails, there's nothing a spamfilter could analyze. The only thing that
>> could theoretically happen, is that the probes ip addresses are flagged as
>> bad by services like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i
>> really don't see a reason why that should happen. There wouldn't be any
>> activity originating from the probes which could be classified as bad. IP
>> addresses are normally only blacklisted, if they send unwanted mails like
>> spam. Also: there are a lot of "check you mailserver" or "check your
>> SSL/TLS" websites. The RIPE Atlas probes would behave the same way. No big
>> deal.
>>
>> Maybe we can think of an "extended" SMTP measurement where RIPE Atlas
>> sends actual e-mails, but that would require (in my opinion), that the
>> person who is creating the measurement somehow provides proof, to be the
>> owner of the target mailserver.
>>
>>
>> BR,
>> Simon
>>
>>
>> On 15.09.22 12:03, Michel Stam wrote:
>>
>> Hello Simon,
>>
>> Thank you for the suggestion.
>>
>> I have a couple of questions to get a better idea:
>>
>>- Can you maybe describe what a SMTP measurement would look like?
>>   - Simple EHLO/HELO
>>   - Sending an email to a designated address (which would then
>>   validate that the SMTP server is capable of relaying etc.)
>>- How would DNSBL or other spam prevention techniques fit into this?
>>- What would the result be?
>>   - Delay until mail received
>>   - Response time by the actual mail server
>>   - Using the Received: headers to get a “traceroute” like result.
>>- What about the more uncommon ports such as 565 (SMTP+SSL/TLS) or
>>587 (mail submission port).
>>- How can we prevent this implementation from having RIPE Atlas be
>>identified as a spam bot network?
>>
>>
>> Best regards,
>>

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Eric Kuhnke 
I would discourage anyone from relying upon the data from "probing" the MX
and SMTP daemons for a domain name no matter what port they run on, because
common configurations of fail2ban used with postfix and others absolutely
will ban your IP at the host operating system level (iptables or other)
after multiple connection attempts and no successful mail transfer or
authentication.

a probe of smtpd will look not much different from the many things out
there on the internet already maliciously probing smtpd trying to find open
relays.




On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas <
ripe-atlas@ripe.net> wrote:

> Hi Michel,
>
> Currently, HTTP and SSL are separate measurements. But for SMTP it will
> probably not work this way... Encryption is not mandatory for SMTP and thus
> negotiated between client and server every time on port 25. Ports 465 and
> 587 are used for Client Email Submission. You could run some measurements
> for these ports as well, but for the beginning, i would focus on
> server2server communication.
>
> So here's what i think:
>
> What we could measure:
> - General reachability/availability of the e-mail service
> - Response time of the remote server: time between connection establish
> and first SMTP response (220 service ready)
> - Which enhanced status codes are offered by the server?
> - Forward/Reverse DNS matching?
> - SMTP banner matching the DNS name?
> - if not: what is it?
> - Does the remote server offer encryption (250-STARTTLS)
> - Which cipher settings are offered by the server (SSL/TLS Version, Key
> Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
> - alternatively: Which cipher setting has been negotiated between
> probe and server?
> - Can we successfully establish a TLS connection?
> - Certificate Check: is the server-certificate valid and does it match the
> hostname?
> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
> 'report'?
> - Forced Authentication: DANE available and check successfull?
>
>
> What we should not do:
> - send MAIL FROM: command
> - send RCPT TO: command
> - send DATA command
> - measure e-mail delivery/roundtrip time, etc.
> - Sending e-mails at all
>
> The Atlas probe should quit the connection after the data is collected. An
> actual e-mail should not be send. The target mailserver would count this
> session as "incomplete" or "aborted" which is totally fine. If someone
> would want to monitor what happens after a mailserver has successfully
> accepted a testmail, he should better use a monitoring service/solution. We
> measure the INTERnet, not what comes after (Intra) :)
>
> I don't expect any "spam" problems. Since the Atlas probes wouldn't send
> e-mails, there's nothing a spamfilter could analyze. The only thing that
> could theoretically happen, is that the probes ip addresses are flagged as
> bad by services like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i
> really don't see a reason why that should happen. There wouldn't be any
> activity originating from the probes which could be classified as bad. IP
> addresses are normally only blacklisted, if they send unwanted mails like
> spam. Also: there are a lot of "check you mailserver" or "check your
> SSL/TLS" websites. The RIPE Atlas probes would behave the same way. No big
> deal.
>
> Maybe we can think of an "extended" SMTP measurement where RIPE Atlas
> sends actual e-mails, but that would require (in my opinion), that the
> person who is creating the measurement somehow provides proof, to be the
> owner of the target mailserver.
>
>
> BR,
> Simon
>
>
> On 15.09.22 12:03, Michel Stam wrote:
>
> Hello Simon,
>
> Thank you for the suggestion.
>
> I have a couple of questions to get a better idea:
>
>- Can you maybe describe what a SMTP measurement would look like?
>   - Simple EHLO/HELO
>   - Sending an email to a designated address (which would then
>   validate that the SMTP server is capable of relaying etc.)
>- How would DNSBL or other spam prevention techniques fit into this?
>- What would the result be?
>   - Delay until mail received
>   - Response time by the actual mail server
>   - Using the Received: headers to get a “traceroute” like result.
>- What about the more uncommon ports such as 565 (SMTP+SSL/TLS) or 587
>(mail submission port).
>- How can we prevent this implementation from having RIPE Atlas be
>identified as a spam bot network?
>
>
> Best regards,
>
> Michel
>
> On 3 Sep 2022, at 14:48, Simon Brandt via ripe-atlas 
> wrote:
>
> Hello,
>
> i'd like to start a discussion about having a RIPE Atlas SMTP measurements.
> First of all: yes, i know there's a big obstacle for such a measurement
> type. A lot of probes are deployed using enduser internet-connections like
> dsl, cable, etc. with dynamic/eyeball IP addresses. Those IP spaces are
> usually blocked by SMTP servers as approach to reduce spam mails. For
> Example: by using blocklists like Spamha

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Avamander 
> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
'report'?

TLS-RPT would also be a good addition to MTA-STS and DANE checks.

> - Forced Authentication: DANE available and check successful?

Just mentioning it, but when DANE is measured then DNSSEC should be as well.

> What we could measure:

In theory things like TCP Fast Open would be nice to measure as well.

On Tue, Sep 20, 2022 at 7:23 PM Simon Brandt via ripe-atlas <
ripe-atlas@ripe.net> wrote:

> Hi Michel,
>
> Currently, HTTP and SSL are separate measurements. But for SMTP it will
> probably not work this way... Encryption is not mandatory for SMTP and thus
> negotiated between client and server every time on port 25. Ports 465 and
> 587 are used for Client Email Submission. You could run some measurements
> for these ports as well, but for the beginning, i would focus on
> server2server communication.
>
> So here's what i think:
>
> What we could measure:
> - General reachability/availability of the e-mail service
> - Response time of the remote server: time between connection establish
> and first SMTP response (220 service ready)
> - Which enhanced status codes are offered by the server?
> - Forward/Reverse DNS matching?
> - SMTP banner matching the DNS name?
> - if not: what is it?
> - Does the remote server offer encryption (250-STARTTLS)
> - Which cipher settings are offered by the server (SSL/TLS Version, Key
> Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
> - alternatively: Which cipher setting has been negotiated between
> probe and server?
> - Can we successfully establish a TLS connection?
> - Certificate Check: is the server-certificate valid and does it match the
> hostname?
> - Forced Encryption: MTA-STS available and 'enforced' or 'testing' or
> 'report'?
> - Forced Authentication: DANE available and check successfull?
>
>
> What we should not do:
> - send MAIL FROM: command
> - send RCPT TO: command
> - send DATA command
> - measure e-mail delivery/roundtrip time, etc.
> - Sending e-mails at all
>
> The Atlas probe should quit the connection after the data is collected. An
> actual e-mail should not be send. The target mailserver would count this
> session as "incomplete" or "aborted" which is totally fine. If someone
> would want to monitor what happens after a mailserver has successfully
> accepted a testmail, he should better use a monitoring service/solution. We
> measure the INTERnet, not what comes after (Intra) :)
>
> I don't expect any "spam" problems. Since the Atlas probes wouldn't send
> e-mails, there's nothing a spamfilter could analyze. The only thing that
> could theoretically happen, is that the probes ip addresses are flagged as
> bad by services like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i
> really don't see a reason why that should happen. There wouldn't be any
> activity originating from the probes which could be classified as bad. IP
> addresses are normally only blacklisted, if they send unwanted mails like
> spam. Also: there are a lot of "check you mailserver" or "check your
> SSL/TLS" websites. The RIPE Atlas probes would behave the same way. No big
> deal.
>
> Maybe we can think of an "extended" SMTP measurement where RIPE Atlas
> sends actual e-mails, but that would require (in my opinion), that the
> person who is creating the measurement somehow provides proof, to be the
> owner of the target mailserver.
>
>
> BR,
> Simon
>
>
> On 15.09.22 12:03, Michel Stam wrote:
>
> Hello Simon,
>
> Thank you for the suggestion.
>
> I have a couple of questions to get a better idea:
>
>- Can you maybe describe what a SMTP measurement would look like?
>   - Simple EHLO/HELO
>   - Sending an email to a designated address (which would then
>   validate that the SMTP server is capable of relaying etc.)
>- How would DNSBL or other spam prevention techniques fit into this?
>- What would the result be?
>   - Delay until mail received
>   - Response time by the actual mail server
>   - Using the Received: headers to get a “traceroute” like result.
>- What about the more uncommon ports such as 565 (SMTP+SSL/TLS) or 587
>(mail submission port).
>- How can we prevent this implementation from having RIPE Atlas be
>identified as a spam bot network?
>
>
> Best regards,
>
> Michel
>
> On 3 Sep 2022, at 14:48, Simon Brandt via ripe-atlas 
> wrote:
>
> Hello,
>
> i'd like to start a discussion about having a RIPE Atlas SMTP measurements.
> First of all: yes, i know there's a big obstacle for such a measurement
> type. A lot of probes are deployed using enduser internet-connections like
> dsl, cable, etc. with dynamic/eyeball IP addresses. Those IP spaces are
> usually blocked by SMTP servers as approach to reduce spam mails. For
> Example: by using blocklists like Spamhaus PBL. So a proper SMTP
> measurement wouldn't work.
>
> BUT we could have an easy way for RIPE Atlas probe hosters to signalize,
> that their probe is

Re: [atlas] RIPE Atlas SMTP Measurement

2022-09-20 Thread Simon Brandt via ripe-atlas 

Hi Michel,

Currently, HTTP and SSL are separate measurements. But for SMTP it will 
probably not work this way... Encryption is not mandatory for SMTP and thus 
negotiated between client and server every time on port 25. Ports 465 and 587 
are used for Client Email Submission. You could run some measurements for these 
ports as well, but for the beginning, i would focus on server2server 
communication.

So here's what i think:

What we could measure:
- General reachability/availability of the e-mail service
- Response time of the remote server: time between connection establish and 
first SMTP response (220 service ready)
- Which enhanced status codes are offered by the server?
- Forward/Reverse DNS matching?
- SMTP banner matching the DNS name?
    - if not: what is it?
- Does the remote server offer encryption (250-STARTTLS)
- Which cipher settings are offered by the server (SSL/TLS Version, Key 
Exchange Algorithms, Encryption Algorithms, Hashing Algorithms)
    - alternatively: Which cipher setting has been negotiated between probe and 
server?
- Can we successfully establish a TLS connection?
- Certificate Check: is the server-certificate valid and does it match the 
hostname?
- Forced Encryption: MTA-STS available and 'enforced' or 'testing' or 'report'?
- Forced Authentication: DANE available and check successfull?


What we should not do:
- send MAIL FROM: command
- send RCPT TO: command
- send DATA command
- measure e-mail delivery/roundtrip time, etc.
- Sending e-mails at all

The Atlas probe should quit the connection after the data is collected. An actual e-mail should not 
be send. The target mailserver would count this session as "incomplete" or 
"aborted" which is totally fine. If someone would want to monitor what happens after a 
mailserver has successfully accepted a testmail, he should better use a monitoring 
service/solution. We measure the INTERnet, not what comes after (Intra) :)

I don't expect any "spam" problems. Since the Atlas probes wouldn't send e-mails, there's nothing a 
spamfilter could analyze. The only thing that could theoretically happen, is that the probes ip addresses are 
flagged as bad by services like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i really don't see a 
reason why that should happen. There wouldn't be any activity originating from the probes which could be 
classified as bad. IP addresses are normally only blacklisted, if they send unwanted mails like spam. Also: 
there are a lot of "check you mailserver" or "check your SSL/TLS" websites. The RIPE 
Atlas probes would behave the same way. No big deal.

Maybe we can think of an "extended" SMTP measurement where RIPE Atlas sends 
actual e-mails, but that would require (in my opinion), that the person who is creating 
the measurement somehow provides proof, to be the owner of the target mailserver.


BR,
Simon


On 15.09.22 12:03, Michel Stam wrote:

Hello Simon,

Thank you for the suggestion.

I have a couple of questions to get a better idea:

  * Can you maybe describe what a SMTP measurement would look like?
  o Simple EHLO/HELO
  o Sending an email to a designated address (which would then validate 
that the SMTP server is capable of relaying etc.)
  * How would DNSBL or other spam prevention techniques fit into this?
  * What would the result be?
  o Delay until mail received
  o Response time by the actual mail server
  o Using the Received: headers to get a “traceroute” like result.
  * What about the more uncommon ports such as 565 (SMTP+SSL/TLS) or 587 (mail 
submission port).
  * How can we prevent this implementation from having RIPE Atlas be identified 
as a spam bot network?

Best regards,

Michel


On 3 Sep 2022, at 14:48, Simon Brandt via ripe-atlas  
wrote:

Hello,

i'd like to start a discussion about having a RIPE Atlas SMTP measurements.
First of all: yes, i know there's a big obstacle for such a measurement type. A 
lot of probes are deployed using enduser internet-connections like dsl, cable, 
etc. with dynamic/eyeball IP addresses. Those IP spaces are usually blocked by 
SMTP servers as approach to reduce spam mails. For Example: by using blocklists 
like Spamhaus PBL. So a proper SMTP measurement wouldn't work.

BUT we could have an easy way for RIPE Atlas probe hosters to signalize, that 
their probe is eligible for SMTP measurements:

Step 1: enable "Simple DNS Entry"
Step 2: create a matching reverse DNS record for the probes IP address

Everybody who is able so configure a reverse DNS record for his probes IP address, is 
most likely using a non-dynamic/non-home ip address space e.g. a datacenter or office 
network. If an ISP provides the option for his customers to configure a reverse DNS 
record, it's most likely a "business-customer" subnet which can be used to run 
mailservers. After Step 1+2 are done, the RIPE Atlas platform would easily be able to 
verify if forward-confirmed reverse DNS is successful, and if so, automatically enable 
that pro