Re: [mailop] Increase in virus activity this week @ MXroute (perhaps others?)
That's super interesting (sorry for my late response here).
One thing that comes to mind reading the way you process the emails, is the
amount of false-positive you might get by implementing scripts that way.
I've been trying to do the same on our end, trying to automate the search
of abuses and notifying me of any suspicious activity, and there are
unfortunately many false positives that pop up, requiring more work to be
implemented.
I feel like it's a whack-a-mole game, but I definitely share the idea that
the human brain has the capacity of filtering out specific data that a
script can hardly produce by default.
When we have an issue arising, I tend to open the logs and just look at
them, and often, a pattern emerges.
Thank you for sharing your structure and your scripts, I really appreciate
it!
Good luck on fighting abusers.
Best,
Cyril
Le dim. 24 avr. 2022 à 02:36, Byung-Hee HWANG via mailop
a écrit :
> (... sorry for top-posting ...)
>
> Dear Jarland,
>
> In the whole story, i feel that you are NICE guy!
> NICE(= faithful + technical + reasonable)
>
> Thanks ^^^
>
> Sincerely, Linux fan Byung-Hee
>
> Jarland Donnell via mailop writes:
>
> > It's a good topic, and one I'm fairly passionate about. Obviously at
> > small scale it's super easy to tell when anything is off from normal,
> > but as you grow it's more difficult to rely on eyes and ears. But that
> > was kind of my dream: I want to be as present as though I'm one admin,
> > logged into one machine, merely watching it function and asking "Why?"
> > when something unusual happens (CPU spike, queue higher than it's been
> > this year to date, a flood of connections from X IP, etc). I want to
> > scale that, I want to scale me.
> >
> > So that's really what I do. I just scale me. If you were sitting in an
> > SSH session tailing a log and just watching for anything that sets off
> > a mental alarm, what would the things be that would trigger that
> > mental alarm? I take the answer to that and have automated checks
> > which then do one of two things:
> >
> > 1. Alert me for human review.
> > 2. Perform the reaction that I would have performed if I were sitting
> > there watching at the time.
> >
> > It can be kind of a mess but right now I'm at over 14,000 clients
> > (exponentially more if counting customers of my customers) and growing
> > rapidly. Thus far I've been able to grow myself by way of coding
> > checks and balances that operate like I think. That's pretty vague so
> > I'll give an example.
> >
> > In rspamd I have this map configured:
> >
> > COMPD_RCPT {
> > type = "rcpt";
> > header = "subject";
> > filter = "email";
> > map = "${LOCAL_CONFDIR}/local.d/compd_rcpt.map";
> > symbol = "COMPD_RCPT";
> > prefilter = true;
> > action = "reject";
> > regexp = true;
> > }
> >
> > Then I have this running on cron:
> >
> >
> https://paste.mxrouteapps.com/?6603394e7d823164#4r5qkNXATJTko55DWmwxjrrbTLCvJ9t5ry61cf5zfHE5
> >
> > Every morning I get up and I check /root/ALERT_RCPT.log and then open
> > a ticket with the customer. This is where the next automation will be
> > as the scale continues to grow, automatically targeting the user and
> > opening a ticket with them.
> >
> > Now what that map does, it lists the recipient emails used by specific
> > spammers who send "test" emails to verify SMTP credentials before they
> > start a campaign. Most of them use the same recipient email every
> > time, so all I have to do is look for it and know "That user's
> > password is compromised."
> >
> > For even more fun, I have a basic HTML page hidden behind
> > authentication which lists two columns. On one side, the top 15
> > senders of this hour. On the other side, the top 15 senders of the
> > last hour. Forcing yourself to be familiar with the top users of your
> > platform by observing how much of your infrastructure they are
> > utilizing creates a mental place where you can immediately recognize
> > when something is off. Toss it on a monitor, have the entire abuse
> > team just stare at it every time they glance away from their
> > work. While you might think that would outgrow it's usefulness with
> > scale, I've worked at large enough scale that I simply don't think it
> > to be so. The top resource users on your platform will change over
> > time, but the vast majority will always be too low utilization to be
> > noteworthy.
> >
> > Even still, if it were to be outgrown, a good database system could
> > keep track of senders enough to say "This person who only sent 1 email
> > a day for the last year just sent 600, might be worth checking the
> > logs to see if they're alright."
> >
> > And that's really where it all comes back to: What do I want to know?
> > What would concern me to see? What would I do if I saw it? Then, quite
> > simply, turn that logic into code and make it work for you.
> >
> > Hope that wasn't too vague to be useful!
> >
> > Jarland
>
> --
> ^고맙습니다 _布德天下_ 감사합니다_^))//
> ___
Re: [mailop] Increase in virus activity this week @ MXroute (perhaps others?)
(... sorry for top-posting ...)
Dear Jarland,
In the whole story, i feel that you are NICE guy!
NICE(= faithful + technical + reasonable)
Thanks ^^^
Sincerely, Linux fan Byung-Hee
Jarland Donnell via mailop writes:
> It's a good topic, and one I'm fairly passionate about. Obviously at
> small scale it's super easy to tell when anything is off from normal,
> but as you grow it's more difficult to rely on eyes and ears. But that
> was kind of my dream: I want to be as present as though I'm one admin,
> logged into one machine, merely watching it function and asking "Why?"
> when something unusual happens (CPU spike, queue higher than it's been
> this year to date, a flood of connections from X IP, etc). I want to
> scale that, I want to scale me.
>
> So that's really what I do. I just scale me. If you were sitting in an
> SSH session tailing a log and just watching for anything that sets off
> a mental alarm, what would the things be that would trigger that
> mental alarm? I take the answer to that and have automated checks
> which then do one of two things:
>
> 1. Alert me for human review.
> 2. Perform the reaction that I would have performed if I were sitting
> there watching at the time.
>
> It can be kind of a mess but right now I'm at over 14,000 clients
> (exponentially more if counting customers of my customers) and growing
> rapidly. Thus far I've been able to grow myself by way of coding
> checks and balances that operate like I think. That's pretty vague so
> I'll give an example.
>
> In rspamd I have this map configured:
>
> COMPD_RCPT {
> type = "rcpt";
> header = "subject";
> filter = "email";
> map = "${LOCAL_CONFDIR}/local.d/compd_rcpt.map";
> symbol = "COMPD_RCPT";
> prefilter = true;
> action = "reject";
> regexp = true;
> }
>
> Then I have this running on cron:
>
> https://paste.mxrouteapps.com/?6603394e7d823164#4r5qkNXATJTko55DWmwxjrrbTLCvJ9t5ry61cf5zfHE5
>
> Every morning I get up and I check /root/ALERT_RCPT.log and then open
> a ticket with the customer. This is where the next automation will be
> as the scale continues to grow, automatically targeting the user and
> opening a ticket with them.
>
> Now what that map does, it lists the recipient emails used by specific
> spammers who send "test" emails to verify SMTP credentials before they
> start a campaign. Most of them use the same recipient email every
> time, so all I have to do is look for it and know "That user's
> password is compromised."
>
> For even more fun, I have a basic HTML page hidden behind
> authentication which lists two columns. On one side, the top 15
> senders of this hour. On the other side, the top 15 senders of the
> last hour. Forcing yourself to be familiar with the top users of your
> platform by observing how much of your infrastructure they are
> utilizing creates a mental place where you can immediately recognize
> when something is off. Toss it on a monitor, have the entire abuse
> team just stare at it every time they glance away from their
> work. While you might think that would outgrow it's usefulness with
> scale, I've worked at large enough scale that I simply don't think it
> to be so. The top resource users on your platform will change over
> time, but the vast majority will always be too low utilization to be
> noteworthy.
>
> Even still, if it were to be outgrown, a good database system could
> keep track of senders enough to say "This person who only sent 1 email
> a day for the last year just sent 600, might be worth checking the
> logs to see if they're alright."
>
> And that's really where it all comes back to: What do I want to know?
> What would concern me to see? What would I do if I saw it? Then, quite
> simply, turn that logic into code and make it work for you.
>
> Hope that wasn't too vague to be useful!
>
> Jarland
--
^고맙습니다 _布德天下_ 감사합니다_^))//
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
Re: [mailop] Increase in virus activity this week @ MXroute (perhaps others?)
It's a good topic, and one I'm fairly passionate about. Obviously at
small scale it's super easy to tell when anything is off from normal,
but as you grow it's more difficult to rely on eyes and ears. But that
was kind of my dream: I want to be as present as though I'm one admin,
logged into one machine, merely watching it function and asking "Why?"
when something unusual happens (CPU spike, queue higher than it's been
this year to date, a flood of connections from X IP, etc). I want to
scale that, I want to scale me.
So that's really what I do. I just scale me. If you were sitting in an
SSH session tailing a log and just watching for anything that sets off a
mental alarm, what would the things be that would trigger that mental
alarm? I take the answer to that and have automated checks which then do
one of two things:
1. Alert me for human review.
2. Perform the reaction that I would have performed if I were sitting
there watching at the time.
It can be kind of a mess but right now I'm at over 14,000 clients
(exponentially more if counting customers of my customers) and growing
rapidly. Thus far I've been able to grow myself by way of coding checks
and balances that operate like I think. That's pretty vague so I'll give
an example.
In rspamd I have this map configured:
COMPD_RCPT {
type = "rcpt";
header = "subject";
filter = "email";
map = "${LOCAL_CONFDIR}/local.d/compd_rcpt.map";
symbol = "COMPD_RCPT";
prefilter = true;
action = "reject";
regexp = true;
}
Then I have this running on cron:
https://paste.mxrouteapps.com/?6603394e7d823164#4r5qkNXATJTko55DWmwxjrrbTLCvJ9t5ry61cf5zfHE5
Every morning I get up and I check /root/ALERT_RCPT.log and then open a
ticket with the customer. This is where the next automation will be as
the scale continues to grow, automatically targeting the user and
opening a ticket with them.
Now what that map does, it lists the recipient emails used by specific
spammers who send "test" emails to verify SMTP credentials before they
start a campaign. Most of them use the same recipient email every time,
so all I have to do is look for it and know "That user's password is
compromised."
For even more fun, I have a basic HTML page hidden behind authentication
which lists two columns. On one side, the top 15 senders of this hour.
On the other side, the top 15 senders of the last hour. Forcing yourself
to be familiar with the top users of your platform by observing how much
of your infrastructure they are utilizing creates a mental place where
you can immediately recognize when something is off. Toss it on a
monitor, have the entire abuse team just stare at it every time they
glance away from their work. While you might think that would outgrow
it's usefulness with scale, I've worked at large enough scale that I
simply don't think it to be so. The top resource users on your platform
will change over time, but the vast majority will always be too low
utilization to be noteworthy.
Even still, if it were to be outgrown, a good database system could keep
track of senders enough to say "This person who only sent 1 email a day
for the last year just sent 600, might be worth checking the logs to see
if they're alright."
And that's really where it all comes back to: What do I want to know?
What would concern me to see? What would I do if I saw it? Then, quite
simply, turn that logic into code and make it work for you.
Hope that wasn't too vague to be useful!
Jarland
On 2022-04-22 15:28, Cyril - ImprovMX via mailop wrote:
Hi Jarland,
that was very interesting, thank you for sharing these details.
I'm curious to know how you caught this in the first place. It would
be interesting to know some technics on how to catch bad behaviors
before they get out of hand and many of us here might be interested in
the how-tos and might also learn a lot from this (me first).
thank you in advance :)
Best,
Cyril
Le ven. 22 avr. 2022 à 00:57, Jarland Donnell via mailop
a écrit :
Hey friends,
This week at MXroute we saw an increase in compromised email
accounts.
Apologies if you saw virus spam coming from our network. Typically,
these events are caught instantly. In cases that use new patterns
and
techniques, under 1 hour. This time, it went on intermittently for
about
half a day on 4/20 (I wish it was for THAT reason), and it happened
a
few times in the days prior. What we found was that every one of
these
outbound emails contained this virus:
https://www.virustotal.com/gui/file/707d507f138a450fb4c7b5c906f280259f23f5aac808b8dfcd23b66d0d679441/detection
It's not difficult to assume that the users received the same virus
beforehand, whether by email or otherwise. The virus appears to use
each
infected computer as part of a botnet, and each computer is involved
in
authenticating over SMTP and sending out copies of the virus. The
only
thing I never saw was our infected users connecting to our servers
to
send the spam, it was alway
Re: [mailop] Increase in virus activity this week @ MXroute (perhaps others?)
Hi Jarland,
that was very interesting, thank you for sharing these details.
I'm curious to know how you caught this in the first place. It would be
interesting to know some technics on how to catch bad behaviors before they
get out of hand and many of us here might be interested in the how-tos and
might also learn a lot from this (me first).
thank you in advance :)
Best,
Cyril
Le ven. 22 avr. 2022 à 00:57, Jarland Donnell via mailop
a écrit :
> Hey friends,
>
> This week at MXroute we saw an increase in compromised email accounts.
> Apologies if you saw virus spam coming from our network. Typically,
> these events are caught instantly. In cases that use new patterns and
> techniques, under 1 hour. This time, it went on intermittently for about
> half a day on 4/20 (I wish it was for THAT reason), and it happened a
> few times in the days prior. What we found was that every one of these
> outbound emails contained this virus:
>
> https://www.virustotal.com/gui/file/707d507f138a450fb4c7b5c906f280259f23f5aac808b8dfcd23b66d0d679441/detection
>
> It's not difficult to assume that the users received the same virus
> beforehand, whether by email or otherwise. The virus appears to use each
> infected computer as part of a botnet, and each computer is involved in
> authenticating over SMTP and sending out copies of the virus. The only
> thing I never saw was our infected users connecting to our servers to
> send the spam, it was always other residential IPs in various locations.
> The emails themselves are difficult to nail down into patterns. Subjects
> ranged from "Firstname Lastname" (name of recipient perhaps?) to spoofed
> PayPal receipts. The emails mostly contained HTML and often some links,
> couldn't find a case of either being the same twice. The To/From headers
> did use a consistent spoofing trend and although the actual content of
> them rarely if ever repeated, the style is easy to grab on to:
>
> Example:
> From: "serv...@paypal.com"
> To: "2...@mail01.netgate.com.uy" <2...@mail01.netgate.com.uy>
>
> Frankly, the "To" header alone may be a consistent way to just shut
> these down until the patterns change. However, ensuring that the virus
> signature is present on inbound and outbound scanners is more likely to
> be effective for a longer period of time, I suspect.
>
> Finally, the last part of the trend which is noteworthy, all of the
> email accounts sent out a "test" email first. Unfortunately, it was
> rarely to the same recipient. Rarely, but not never. Two sample test
> email logs:
>
> 2022-04-20 12:53:58 1nh9qH-0008OE-RW <= winfield@{our customer's
> domain}.com H=host-79-10-109-221.business.telecomitalia.it (localhost)
> [79.10.109.221] P=esmtpsa X=TLS1.2:ECDHE-ECDSA-AES128-GCM-SHA256:128
> CV=no A=login:winfield@{our customer's domain}.com S=735 T="ZYaeD, X "
> from for
> toewebvastsatw...@gmx.com
>
> 2022-04-20 12:53:40 1nh9q0-00HECV-Al <= register@{our customer's
> domain}.org H=mail.dfclark.co.uk (localhost) [149.255.169.82] P=esmtpsa
> X=TLS1.3:TLS_AES_256_GCM_SHA384:256 CV=no A=login:register@{our
> customer's domain}.org S=691 T="wFd2, MkRWa2 A" from customer's domain}.org> for toewebvastsatw...@gmx.com
>
> And for kicks, a few sample email subjects: https://clbin.com/dGT1y
> (sorted by count of repeat subjects from one compromised account)
>
> I sincerely hope that this helps someone else in the never-ending effort
> to keep IPs clean.
>
> Sincerely,
> Jarland Donnell
> MXroute Administrator
> ___
> mailop mailing list
> mailop@mailop.org
> https://list.mailop.org/listinfo/mailop
>
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
[mailop] Increase in virus activity this week @ MXroute (perhaps others?)
Hey friends,
This week at MXroute we saw an increase in compromised email accounts.
Apologies if you saw virus spam coming from our network. Typically,
these events are caught instantly. In cases that use new patterns and
techniques, under 1 hour. This time, it went on intermittently for about
half a day on 4/20 (I wish it was for THAT reason), and it happened a
few times in the days prior. What we found was that every one of these
outbound emails contained this virus:
https://www.virustotal.com/gui/file/707d507f138a450fb4c7b5c906f280259f23f5aac808b8dfcd23b66d0d679441/detection
It's not difficult to assume that the users received the same virus
beforehand, whether by email or otherwise. The virus appears to use each
infected computer as part of a botnet, and each computer is involved in
authenticating over SMTP and sending out copies of the virus. The only
thing I never saw was our infected users connecting to our servers to
send the spam, it was always other residential IPs in various locations.
The emails themselves are difficult to nail down into patterns. Subjects
ranged from "Firstname Lastname" (name of recipient perhaps?) to spoofed
PayPal receipts. The emails mostly contained HTML and often some links,
couldn't find a case of either being the same twice. The To/From headers
did use a consistent spoofing trend and although the actual content of
them rarely if ever repeated, the style is easy to grab on to:
Example:
From: "serv...@paypal.com"
To: "2...@mail01.netgate.com.uy" <2...@mail01.netgate.com.uy>
Frankly, the "To" header alone may be a consistent way to just shut
these down until the patterns change. However, ensuring that the virus
signature is present on inbound and outbound scanners is more likely to
be effective for a longer period of time, I suspect.
Finally, the last part of the trend which is noteworthy, all of the
email accounts sent out a "test" email first. Unfortunately, it was
rarely to the same recipient. Rarely, but not never. Two sample test
email logs:
2022-04-20 12:53:58 1nh9qH-0008OE-RW <= winfield@{our customer's
domain}.com H=host-79-10-109-221.business.telecomitalia.it (localhost)
[79.10.109.221] P=esmtpsa X=TLS1.2:ECDHE-ECDSA-AES128-GCM-SHA256:128
CV=no A=login:winfield@{our customer's domain}.com S=735 T="ZYaeD, X "
from for
toewebvastsatw...@gmx.com
2022-04-20 12:53:40 1nh9q0-00HECV-Al <= register@{our customer's
domain}.org H=mail.dfclark.co.uk (localhost) [149.255.169.82] P=esmtpsa
X=TLS1.3:TLS_AES_256_GCM_SHA384:256 CV=no A=login:register@{our
customer's domain}.org S=691 T="wFd2, MkRWa2 A" from customer's domain}.org> for toewebvastsatw...@gmx.com
And for kicks, a few sample email subjects: https://clbin.com/dGT1y
(sorted by count of repeat subjects from one compromised account)
I sincerely hope that this helps someone else in the never-ending effort
to keep IPs clean.
Sincerely,
Jarland Donnell
MXroute Administrator
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
