It's a good topic, and one I'm fairly passionate about. Obviously at
small scale it's super easy to tell when anything is off from normal,
but as you grow it's more difficult to rely on eyes and ears. But that
was kind of my dream: I want to be as present as though I'm one admin,
logged into one machine, merely watching it function and asking "Why?"
when something unusual happens (CPU spike, queue higher than it's been
this year to date, a flood of connections from X IP, etc). I want to
scale that, I want to scale me.
So that's really what I do. I just scale me. If you were sitting in an
SSH session tailing a log and just watching for anything that sets off a
mental alarm, what would the things be that would trigger that mental
alarm? I take the answer to that and have automated checks which then do
one of two things:
1. Alert me for human review.
2. Perform the reaction that I would have performed if I were sitting
there watching at the time.
It can be kind of a mess but right now I'm at over 14,000 clients
(exponentially more if counting customers of my customers) and growing
rapidly. Thus far I've been able to grow myself by way of coding checks
and balances that operate like I think. That's pretty vague so I'll give
an example.
In rspamd I have this map configured:
COMPD_RCPT {
type = "rcpt";
header = "subject";
filter = "email";
map = "${LOCAL_CONFDIR}/local.d/compd_rcpt.map";
symbol = "COMPD_RCPT";
prefilter = true;
action = "reject";
regexp = true;
}
Then I have this running on cron:
https://paste.mxrouteapps.com/?6603394e7d823164#4r5qkNXATJTko55DWmwxjrrbTLCvJ9t5ry61cf5zfHE5
Every morning I get up and I check /root/ALERT_RCPT.log and then open a
ticket with the customer. This is where the next automation will be as
the scale continues to grow, automatically targeting the user and
opening a ticket with them.
Now what that map does, it lists the recipient emails used by specific
spammers who send "test" emails to verify SMTP credentials before they
start a campaign. Most of them use the same recipient email every time,
so all I have to do is look for it and know "That user's password is
compromised."
For even more fun, I have a basic HTML page hidden behind authentication
which lists two columns. On one side, the top 15 senders of this hour.
On the other side, the top 15 senders of the last hour. Forcing yourself
to be familiar with the top users of your platform by observing how much
of your infrastructure they are utilizing creates a mental place where
you can immediately recognize when something is off. Toss it on a
monitor, have the entire abuse team just stare at it every time they
glance away from their work. While you might think that would outgrow
it's usefulness with scale, I've worked at large enough scale that I
simply don't think it to be so. The top resource users on your platform
will change over time, but the vast majority will always be too low
utilization to be noteworthy.
Even still, if it were to be outgrown, a good database system could keep
track of senders enough to say "This person who only sent 1 email a day
for the last year just sent 600, might be worth checking the logs to see
if they're alright."
And that's really where it all comes back to: What do I want to know?
What would concern me to see? What would I do if I saw it? Then, quite
simply, turn that logic into code and make it work for you.
Hope that wasn't too vague to be useful!
Jarland
On 2022-04-22 15:28, Cyril - ImprovMX via mailop wrote:
Hi Jarland,
that was very interesting, thank you for sharing these details.
I'm curious to know how you caught this in the first place. It would
be interesting to know some technics on how to catch bad behaviors
before they get out of hand and many of us here might be interested in
the how-tos and might also learn a lot from this (me first).
thank you in advance :)
Best,
Cyril
Le ven. 22 avr. 2022 à 00:57, Jarland Donnell via mailop
<mailop@mailop.org> a écrit :
Hey friends,
This week at MXroute we saw an increase in compromised email
accounts.
Apologies if you saw virus spam coming from our network. Typically,
these events are caught instantly. In cases that use new patterns
and
techniques, under 1 hour. This time, it went on intermittently for
about
half a day on 4/20 (I wish it was for THAT reason), and it happened
a
few times in the days prior. What we found was that every one of
these
outbound emails contained this virus:
https://www.virustotal.com/gui/file/707d507f138a450fb4c7b5c906f280259f23f5aac808b8dfcd23b66d0d679441/detection
It's not difficult to assume that the users received the same virus
beforehand, whether by email or otherwise. The virus appears to use
each
infected computer as part of a botnet, and each computer is involved
in
authenticating over SMTP and sending out copies of the virus. The
only
thing I never saw was our infected users connecting to our servers
to
send the spam, it was always other residential IPs in various
locations.
The emails themselves are difficult to nail down into patterns.
Subjects
ranged from "Firstname Lastname" (name of recipient perhaps?) to
spoofed
PayPal receipts. The emails mostly contained HTML and often some
links,
couldn't find a case of either being the same twice. The To/From
headers
did use a consistent spoofing trend and although the actual content
of
them rarely if ever repeated, the style is easy to grab on to:
Example:
From: "serv...@paypal.com" <engineering@{our customer's
domain}.com>
To: "2...@mail01.netgate.com.uy" <2...@mail01.netgate.com.uy>
Frankly, the "To" header alone may be a consistent way to just shut
these down until the patterns change. However, ensuring that the
virus
signature is present on inbound and outbound scanners is more likely
to
be effective for a longer period of time, I suspect.
Finally, the last part of the trend which is noteworthy, all of the
email accounts sent out a "test" email first. Unfortunately, it was
rarely to the same recipient. Rarely, but not never. Two sample test
email logs:
2022-04-20 12:53:58 1nh9qH-0008OE-RW <= winfield@{our customer's
domain}.com H=host-79-10-109-221.business.telecomitalia.it [1]
(localhost)
[79.10.109.221] P=esmtpsa X=TLS1.2:ECDHE-ECDSA-AES128-GCM-SHA256:128
CV=no A=login:winfield@{our customer's domain}.com S=735 T="ZYaeD, X
"
from <winfield@{our customer's domain}.com> for
toewebvastsatw...@gmx.com
2022-04-20 12:53:40 1nh9q0-00HECV-Al <= register@{our customer's
domain}.org H=mail.dfclark.co.uk [2] (localhost) [149.255.169.82]
P=esmtpsa
X=TLS1.3:TLS_AES_256_GCM_SHA384:256 CV=no A=login:register@{our
customer's domain}.org S=691 T="wFd2, MkRWa2 A" from <register@{our
customer's domain}.org> for toewebvastsatw...@gmx.com
And for kicks, a few sample email subjects: https://clbin.com/dGT1y
(sorted by count of repeat subjects from one compromised account)
I sincerely hope that this helps someone else in the never-ending
effort
to keep IPs clean.
Sincerely,
Jarland Donnell
MXroute Administrator
_______________________________________________
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
Links:
------
[1] http://host-79-10-109-221.business.telecomitalia.it
[2] http://mail.dfclark.co.uk
_______________________________________________
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
_______________________________________________
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
