Re: China’s Slow Transnational Network

[Ben Cannon]

oops. missed a spot.

-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC 
b...@6by7.net 




> On Mar 2, 2020, at 2:36 PM, David Burns  wrote:
> 
> Did you compare CERNET with commodity networks?  (My anecdotal observations 
> from a couple years ago suggest that Internet2 to CERNET is very good when 
> other paths are poor to unusable.)
> 
> --David Burns

Re: China’s Slow Transnational Network

[Mark Tinka]

On 15/Mar/20 22:51, Frank Habicht wrote:

>
> thanks for the "quotes", Mark. I agree.
>
> https://www.caida.org/publications/presentations/2018/investigating_causes_congestion_african_afrinic/investigating_causes_congestion_african_afrinic.pdf
>
> page 23:
> Results Overview
> • No evidence of widespread congestion
>- 2.2% of discovered link showed evidence of congestion at the end of
>  our measurements campaign
>
> page 34:
> Conclusions
> • Measured IXPs were congestion-free, which promotes peering in the
>   region
>
> https://conferences.sigcomm.org/imc/2017/papers/imc17-final182.pdf
>
> my conclusion: s/congestion/congestion or the lack thereof/g
>
> Frank Habicht
>
> PS: yes, i could name peers that once had inadequate links into an IXP.
> but for how long did that happen? (yes..., any minute is too long...)

Indeed.

There was a time when backhaul links between ISP routers at the exchange
point and their nearest PoP were based on E1's, wireless, e.t.c. But
that could be said of, pretty much, every exchange point that kicked off
inside of the last 2.5 decades.

Nowadays, such links, if they exist, are the very deep exception, not
the rule.

Mark.

Re: COVID-19 vs. our Networks

[Stephen Fulton]

In $dayjob I constantly see the lack of understanding of the difference 
between what the Internet is and what a path engineered private circuit 
is (eg. pseudowire, wave, whatever).  The latest fight is over SD-WAN 
and those who think it will replace MPLS entirely and they won't need 
those expensive routers anymore.  But I digress.


Mark's comment and others like it are the correct approach Mike.  If 
your private WAN is most critical, then invest in and manage user 
complaints about poor Internet service.  ISP's, IXP's and CDN's are not 
going to twist themselves into knots to solve your problems, even if 
someone calls it an emergency.  Sorry.


Stephen


On 2020-03-15 02:01, Mark Tinka wrote:



On 14/Mar/20 19:14, Mike Bolitho wrote:


/
/

I work for a hospital, we ran into some issues last week due to 
congestion that was totally outside of our control that was off of our 
WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission 
critical at the time but it was still disruptive. As more and more 
people are driven home during this time, more and more people will be 
using bandwidth intensive streaming and online gaming products. If 
more and more TSP coded entities are running into issues, ISPs, IXPs, 
and CDNs will be forced to act.


Hmmh, if that level of priority is required, I'd probably build my own 
network, and not rely on public infrastructure like the Internet.


Mark.

Re: COVID-19 vs. our Networks

[Paul Nash]

> … as soon as they enter the Province
> from outside Canada they are "requested" to self-isolate for 14-days.
> This is for citizens.  Don't know what the policy is for non-Canadians.

Maybe not so much in practice.  I landed at Pearson late last night, returning 
from South Africa via Amsterdam.  Other than the standard passport checks, no 
sign of any screening.  Yay Doug Ford and his merry men.

I’m in self-quarantine for the next 14 days, working from home, in case of any 
symptoms.  I obviously hope the there are none, and that I can go back to visit 
clients, but I feel that that would be a really stupid thing to do right now.

In the meantime, schools are shut down, and I have two children back home from 
university.

paul


> 
>> (Fortunately, I'm in a position to hide in my apartment and only
> emerge
>> for grocery shopping at 2AM until things wind down... Hope everybody
> else
>> has a good contingency plan)
> 
> Yeah, sounds like a plan.
> 
> -- 
> The fact that there's a Highway to Hell but only a Stairway to Heaven
> says a lot about anticipated traffic volume.
> 
> 
> 

Re: China’s Slow Transnational Network

[Frank Habicht]

On 15/03/2020 13:07, Mark Tinka wrote:
> On 15/Mar/20 09:55, Pengxiong Zhu wrote:
> 
>> I know Caida has one paper on the congestion on Africa's IXPs substrate.
> 
> I can't think of a single IXP in Africa that is "congested".

thanks for the "quotes", Mark. I agree.

https://www.caida.org/publications/presentations/2018/investigating_causes_congestion_african_afrinic/investigating_causes_congestion_african_afrinic.pdf

page 23:
Results Overview
• No evidence of widespread congestion
   - 2.2% of discovered link showed evidence of congestion at the end of
 our measurements campaign

page 34:
Conclusions
• Measured IXPs were congestion-free, which promotes peering in the
  region

https://conferences.sigcomm.org/imc/2017/papers/imc17-final182.pdf

my conclusion: s/congestion/congestion or the lack thereof/g

Frank Habicht

PS: yes, i could name peers that once had inadequate links into an IXP.
but for how long did that happen? (yes..., any minute is too long...)

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 15/Mar/20 20:11, Owen DeLong wrote:
> I can top that. I was at a Data Center Real Estate conference some years back 
> when virtualization was all the rage.
>
> Admittedly, a lot of the people present (including this guy) were real-estate 
> types, not technical, 
>
> “Eventually, we’ll even be able to virtualize the machines and the network 
> and we won’t need all these datacenters
> and the amount of power required will be much less.”
>
> Kid you not… He literally thought we could virtualize away all of the 
> physical infrastructure.

Hehehe - that's not entirely unbelievable. Although it gave me a good
knee-tap :-).

#WhenTheInternetMakesYouThinkYouCanFly

Mark.

Re: COVID-19 vs. our Networks

[Owen DeLong]

> On Mar 15, 2020, at 08:13 , Mark Tinka  wrote:
> 
> 
> 
> On 15/Mar/20 16:59, Keith Medcalf wrote:
>> If it is "critical" you need a dedicated circuit.  If it is "meh, who gives 
>> a shit", then you can go though the Internet.
>> 
>> The root of the issue is that some idiot did a bad Risk Assessment.  Hope it 
>> got fired or killed so it won't do this again in the future.
>> 
>> Hope you also learned something as well.  Freedom of the Press belongs to He 
>> Who Owns the Press.  If you are using someone else's presses (particularly 
>> without directly paying and contracting with that party for the use of their 
>> presses), you will live or die according to the whim of the owner of the 
>> Press, and there is SFA you can do about that.  That is how the world has 
>> worked for billions of years.  You would think people would understand that 
>> by now.
> 
> The Internet has become its own enemy.
> 
> The time I realized it gives people more mental than practical hope in
> the possibility of anything is when a pre-sales engineer once asked me
> if we could deliver a circuit to a customer without using a CPE, because
> that would increase their acquisition costs. RFC's 1149, 2549 and 6214
> came to memory. This was 2012.
> 
> The Internet has become so ubiquitous and inspired significant (almost
> unreasonable) possibilities that it is just about preposterous to
> convince those that need to sign invoices that "Ummh, you get what you
> pay for is as relevant in 2020 as it was in 1980".
> 
> Then again, you can buy an SDN or an SD-WAN or an IoT, to back up your
> Big Data over the 5G connection you gathered it, and all will be well.
> 
> Mark.
> 

I can top that. I was at a Data Center Real Estate conference some years back 
when virtualization was all the rage.

Admittedly, a lot of the people present (including this guy) were real-estate 
types, not technical, 

“Eventually, we’ll even be able to virtualize the machines and the network and 
we won’t need all these datacenters
and the amount of power required will be much less.”

Kid you not… He literally thought we could virtualize away all of the physical 
infrastructure.

Owen

Re: backtracking forged packets?

[Octolus Development]

Regardless if you leave half-open SYN_RECV, you will still get abuse reports 
and blacklist it. These providers who blacklist and report you, are reporting 
you for bruteforce.. and they only check for SYN_RECV.

>From my experience there is nothing you can do about that, as you should not 
>monitor TCP Services for SYN_RECV only.. but rather the full TCP-handshake to 
>be completed.. if not, there are large chances it's spoofed.

Only solution we found was using SYNPROXY / CONNTRACK to validate.


On 15.03.2020 00:51:11, Damian Menscher via NANOG  wrote:
I don't recommend filtering the SYN-ACK packets. That's what Octolus did, and 
the result was leaving half-open SYN_RECV connections on all the nodes used for 
reflection. That has two downsides:

- the reflectors will retry the SYN-ACK (several times), which increases your 
PPS load (amplifying the attack)
- the providers may notice the large number of SYN_RECV connections from your 
network and put you on a blacklist

I don't want to leave you with the impression that it's hopeless... these 
attacks aren't impossible to stop --- it just requires convincing the transit 
providers to care.


Damian

On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me [http://ddostest.me] via 
NANOG mailto:nanog@nanog.org]> wrote:

Hi Bill,
thanks for sharing the data. Indeed, I can't offer you a way to backtrack the 
spoofed packets.
Anyway, I'm not sure what could you do legally as there is a very high chance 
that these people are not in the USA and the CFAA won't apply to them.

Here is what I would do if I was in your situation.
Since these packets are spoof and malformed, I would block all SYN/ACK based on 
the length.
Depending on your hardware, it's very easy to inspect only the SYN/ACK by 
length if you have modern hardware. On linux/unix, it's also very 
straightforward. I'm not sure for windows though.

Here is the details of the analysis:
Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, 
SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be others, 
but let's use the basic one.
In your case, there are none. There is only MSS and the SYN length is 44 bytes. 
These are spoof packets maybe generated by either TCP-AMP like reported earlier.

I would try to block all SYN/ACK coming toward your network with a length of 44 
bytes or lower. But, this is weird because it should be 54 bytes. Maybe there 
is some offloading of some sort in your gear.

Now depending on your hardware, it could work or it could kill your router as 
it will punt the cpu. I guess you have some modern gear.

What I do when I am not sure about the length, I start to accept and log at 60 
bytes, then 58, 56, 54... 44 until I catch the gremlins.
Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower than 
X bytes. It won't kill or block anything legitimate if you do it properly. :)
What will happen is that you will not reply to these spoof SYN/ACK with a RST 
and still allowing RST for legit SYN and SYN/ACK. Akamai and your service 
providers will be happy and should not penalize you.

I'm pretty sure that it will help you as it did for me in the past.

Let me know if it's not clear and/or which part is foggy and I'll try to give 
more details and better explanation.
Regards,

Jean St-Laurent

On 2020-03-14 11:46, William Herrin wrote:

On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me [http://ddostest.me] via 
NANOG  [mailto:nanog@nanog.org] wrote:
can you post some forged packets please? You can send them offlist if you 
prefer.
Hi Jean, Here are a couple examples (PDT this morning): 08:22:43.413250 IP (tos 
0x0, ttl 55, id 10108, offset 0, flags [none], proto ICMP (1), length 56) 
45.89.93.26 > 199.33.225.218 [http://199.33.225.218]: ICMP host 45.89.93.26 
unreachable - admin prohibited filter, length 36 IP (tos 0x0, ttl 69, id 10108, 
offset 0, flags [DF], proto TCP (6), length 40) 199.33.225.218.9851 > 
45.89.93.26.443: [|tcp] 0x: 4500 0038 277c  3701 28da 2d59 5d1a 0x0010: 
c721 e1da 030d 4b61   4500 0028 0x0020: 277c 4000 4506 dae4 c721 e1da 
2d59 5d1a 0x0030: 267b 01bb a057 e903 08:25:47.787326 IP (tos 0x0, ttl 54, id 
0, offset 0, flags [DF], proto TCP (6), length 44) 104.87.78.95.80 > 
199.33.225.143.8667: Flags [S.], cksum 0xc97a (correct), seq 1216155085, ack 
11765035, win 29200, options [mss 1156], length 0 0x: 4500 002c  4000 
3606 e564 6857 4e5f 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b 0x0020: 
6012 7210 c97a  0204 0484 I have observed no consistency in the remote IP 
addresses. I receive no more than a few of each and they don't line up with 
particular networks. Remote ports are heavily 80, 443, 22, 25, etc. but a 
smattering of less common ports too. I'm not seeing any RSTs at all nor any 
port-unreachables. Lots of syn/acks and a few time exceeded and host 
unreachables. I don't know what to make of that. On Sat, Mar 14, 2020 at 1:46 
AM Andrew Smith  
[mailto:andrew.william.sm...@g

Re: backtracking forged packets?

[Amir Herzberg]

Bill said:

To be clear: the majority of the addresses at my end are not
> associated with live hosts. There's nothing there to respond.
>

I forgot to ask/mention , but that's actually a common scenario. Of course
in that case your router is _supposed_ to respond with ICMP host
unreachable which would have similar impact to RST (but I bet it doesn't -
which spec says to do it, many routers are not configured to do this). So
attackers may idd have identified your address block as a good set of IP
addresses to spoof when they attack different servers. It may help , if you
configure your router to send ICMP unreachable (or RST, but that may be
harder/less efficient, I think). Initially, this will cause the victim
servers to stop re-sending you the syn/ack, so you'll feel some relief, and
also you'll reduce the attack on these servers, which is A Good Deed.
Eventually, hopefully, this may cause attackers to remove your IP prefix
from their list of IP addresses which work well as spoofed source, but this
will probably take quite a while. Sorry...

>
> My surprise about the lack of RSTs is the lack of RSTs from the remote
> servers back to the addresses which have been spoofed. If the attacker
> was hitting random ports on those hosts, I'd expect to see some RSTs.
>

yes, but I bet attacker is not hitting random ports, attacker is hitting
real servers in TCP listen.

(sorry don't have time to netflow... have tons of work to do)

-- 
Amir Herzberg

Comcast professor of Security Innovations, University of Connecticut

Homepage: https://sites.google.com/site/amirherzberg/home

Foundations of Cyber-Security (part I: applied crypto, part II:
network-security):
https://www.researchgate.net/project/Foundations-of-Cyber-Security



On Sun, Mar 15, 2020 at 12:50 PM William Herrin  wrote:

> On Sun, Mar 15, 2020 at 9:07 AM Amir Herzberg 
> wrote:
> > Not sending RST could even result in you receiving ICMP unreachable -
> esp. indicating filtering as you received - since server admins may have
> installed a filter against your prefix (to deal with such abuse). So, I
> wonder, it is possible that your network/FW/provider already filter the RST
> responses so they don't reach the (victim) servers?
>
> Hi Amir,
>
> To be clear: the majority of the addresses at my end are not
> associated with live hosts. There's nothing there to respond.
>
> My surprise about the lack of RSTs is the lack of RSTs from the remote
> servers back to the addresses which have been spoofed. If the attacker
> was hitting random ports on those hosts, I'd expect to see some RSTs.
>
> If you happen to have decent netflow, try looking for packets sourced
> from 199.33.224.0/24. You'll find a legitimate route in your tables
> ending at AS11875 but today, at least, there are no legitimate packets
> sourced from that address block.
>
> Regards,
> Bill
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>

Re: backtracking forged packets?

[William Herrin]

On Sun, Mar 15, 2020 at 9:07 AM Amir Herzberg  wrote:
> Not sending RST could even result in you receiving ICMP unreachable - esp. 
> indicating filtering as you received - since server admins may have installed 
> a filter against your prefix (to deal with such abuse). So, I wonder, it is 
> possible that your network/FW/provider already filter the RST responses so 
> they don't reach the (victim) servers?

Hi Amir,

To be clear: the majority of the addresses at my end are not
associated with live hosts. There's nothing there to respond.

My surprise about the lack of RSTs is the lack of RSTs from the remote
servers back to the addresses which have been spoofed. If the attacker
was hitting random ports on those hosts, I'd expect to see some RSTs.

If you happen to have decent netflow, try looking for packets sourced
from 199.33.224.0/24. You'll find a legitimate route in your tables
ending at AS11875 but today, at least, there are no legitimate packets
sourced from that address block.

Regards,
Bill


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: backtracking forged packets?

[Jean | ddostest.me via NANOG]

I believe that Oculus blocked the RST and not the SYN/ACK.

It sounds the same but, it's not.

I see 2 options here:

1. Continue to be DDoS and abuse. The result is maybe they will move on, 
but I doubt.


2. Try to block the malformed SYN/ACK and it will probably solve your 
issue. You have nothing to lose to try as you can easily fallback.


You can let us know what are the results and if I was wrong, I will 
publicly state that I was wrong.


But, make sure you do it properly. _Do not block the RST_, you need to 
block the malformed SYN/ACK before they hit your open reflectors.


Jean


On 2020-03-15 12:05, Amir Herzberg wrote:
Bill: I agree with Damian that you should try to ensure responding 
with RST to SYN/ACK; in fact, attackers are sometimes (often?) 
_looking_ for networks that do not send RST in response to unsolicited 
SYN/ACK, to spoof their addresses in syn-flooding and other attacks 
(eg., syn-ack) against victim servers.


Not sending RST could even result in you receiving ICMP unreachable - 
esp. indicating filtering as you received - since server admins may 
have installed a filter against your prefix (to deal with such abuse). 
So, I wonder, it is possible that your network/FW/provider already 
filter the RST responses so they don't reach the (victim) servers?


BTW, I'm covering these issues in my DoS lecture as part of the 
Net-Sec course (see URL below). The foils are available (although not 
yet latest version, will upload `soon' :) ), the text of the net-sec 
(2nd) part - not so much, it may take me quite a while to make this 
(2nd) part useable.

--
Amir Herzberg

Comcast professor of Security Innovations, University of Connecticut

Homepage: https://sites.google.com/site/amirherzberg/home

Foundations of Cyber-Security (part I: applied crypto, part II: 
network-security):

https://www.researchgate.net/project/Foundations-of-Cyber-Security



On Sat, Mar 14, 2020 at 7:51 PM Damian Menscher via NANOG 
mailto:nanog@nanog.org>> wrote:


I don't recommend filtering the SYN-ACK packets.  That's what
Octolus did, and the result was leaving half-open SYN_RECV
connections on all the nodes used for reflection.  That has two
downsides:

  - the reflectors will retry the SYN-ACK (several times), which
increases your PPS load (amplifying the attack)
  - the providers may notice the large number of SYN_RECV
connections from your network and put you on a blacklist

I don't want to leave you with the impression that it's
hopeless... these attacks aren't impossible to stop --- it just
requires convincing the transit providers to care.

Damian

On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me
 via NANOG mailto:nanog@nanog.org>> wrote:

Hi Bill,

thanks for sharing the data. Indeed, I can't offer you a way
to backtrack the spoofed packets.

Anyway, I'm not sure what could you do legally as there is a
very high chance that these people are not in the USA and the
CFAA won't apply to them.

Here is what I would do if I was in your situation.

Since these packets are spoof and malformed, I would block all
SYN/ACK based on the length.

Depending on your hardware, it's very easy to inspect _only
the SYN/ACK by length_ if you have modern hardware. On
linux/unix, it's also very straightforward. I'm not sure for
windows though.

Here is the details of the analysis:

Today, all the SYN and SYN/ACK includes a minimum of options
like MSS, WS, SACK, NOP (Only a spacer, sometimes 2) and
extended TS. There might be others, but let's use the basic one.

In your case, there are none. There is only MSS and the SYN
length is 44 bytes. These are spoof packets maybe generated by
either TCP-AMP like reported earlier.

I would try to block all SYN/ACK coming toward your network
with a length of 44 bytes or lower. But, this is weird because
it should be 54 bytes. Maybe there is some offloading of some
sort in your gear.

Now depending on your hardware, it could work or it could kill
your router as it will punt the cpu. I guess you have some
modern gear.

What I do when I am not sure about the length, I start to
accept and log at 60 bytes, then 58, 56, 54... 44 until I
catch the gremlins.

Once you found the sweet spot, you drop all SYN/ACK toward
your /23 lower than X bytes. It won't kill or block anything
legitimate if you do it properly. :)

What will happen is that you will not reply to these spoof
SYN/ACK with a RST and still allowing RST for legit SYN and
SYN/ACK. Akamai and your service providers will be happy and
should not penalize you.

I'm pretty sure that it will help you as it did for me in the
past.

Let me know if it

Re: WIKI documentation Software?

[Andrew Latham]

Mediawiki is ideal in my use. If security is a concern just front it with
oauth2 via Caddy[2] and maybe have a look at how Wikipedia self documents
infrastructure[3]

1. https://www.mediawiki.org/wiki/MediaWiki
2. https://caddyserver.com/
3. https://wikitech.wikimedia.org/wiki/Main_Page


On Sun, Mar 15, 2020 at 9:27 AM Grimes, Greg 
wrote:
>
>
> I know I'm a bit late to the conversation.  We have been using PMWiki for
well over 10 yrs now.  At the time we started using it there weren't a lot
of Wikis out there.  MediaWiki obviously was the most popular, but it did
not provide the level of secure access that we wanted.  We didn't want
everyone to be able to edit certain pages.  It was also very easy to
integrate into CAS.  I wrote a cookbook for it years ago.  We use groups to
allow only certain people to edit certain pages.  We also restrict viewing
of some pages.  Our Security Group keeps some of their stuff restricted.  I
work for the network team and we prevent everyone but our team from editing
our pages.  They can view them, just can't mess with them.  Hope this helps
someone.
>
> https://www.pmwiki.org/
>
>
> --
>
> Greg T. Grimes
> Senior Network Analyst
> Information Technology Services
> Mississippi State University
> greg.gri...@msstate.edu
> 662-325-9311
>
> 
> From: NANOG  on behalf of Yang Yu <
yang.yu.l...@gmail.com>
> Sent: Sunday, March 15, 2020 06:59
> To: Brielle
> Cc: NANOG list
> Subject: Re: WIKI documentation Software?
>
> On Sat, Mar 14, 2020 at 7:07 AM Brielle  wrote:
> >
> > I personally like Dokuwiki a lot.
> >
> > From a usability standpoint, once you spend a few learning the
interface, it’s very simplistic and not overwhelming in features.  You can
always add extensions for stuff you need that isn’t there out of box.
> >
> > From a technical standpoint, it doesn’t need a database.  The entire
structure is text files, so it can be run on even a super small VM, and
doing backups is as easy as tarballing the data directory.
> >
> > It’s got support for LDAP for authentication too, which might be useful.
>
> +1 for dokuwiki
>
> easy to maintain, has enough features while not become distracting
>
> only complaint is that it doesn't support markdown, but the syntax is
> easy enough (much easier than MediaWiki imo)



--
- Andrew "lathama" Latham -

Re: backtracking forged packets?

[Amir Herzberg]

Bill: I agree with Damian that you should try to ensure responding with RST
to SYN/ACK; in fact, attackers are sometimes (often?) _looking_ for
networks that do not send RST in response to unsolicited SYN/ACK, to spoof
their addresses in syn-flooding and other attacks (eg., syn-ack) against
victim servers.

Not sending RST could even result in you receiving ICMP unreachable - esp.
indicating filtering as you received - since server admins may have
installed a filter against your prefix (to deal with such abuse). So, I
wonder, it is possible that your network/FW/provider already filter the RST
responses so they don't reach the (victim) servers?

BTW, I'm covering these issues in my DoS lecture as part of the Net-Sec
course (see URL below). The foils are available (although not yet latest
version, will upload `soon' :) ), the text of the net-sec (2nd) part - not
so much, it may take me quite a while to make this (2nd) part useable.
-- 
Amir Herzberg

Comcast professor of Security Innovations, University of Connecticut

Homepage: https://sites.google.com/site/amirherzberg/home

Foundations of Cyber-Security (part I: applied crypto, part II:
network-security):
https://www.researchgate.net/project/Foundations-of-Cyber-Security



On Sat, Mar 14, 2020 at 7:51 PM Damian Menscher via NANOG 
wrote:

> I don't recommend filtering the SYN-ACK packets.  That's what Octolus did,
> and the result was leaving half-open SYN_RECV connections on all the nodes
> used for reflection.  That has two downsides:
>
>   - the reflectors will retry the SYN-ACK (several times), which increases
> your PPS load (amplifying the attack)
>   - the providers may notice the large number of SYN_RECV connections from
> your network and put you on a blacklist
>
> I don't want to leave you with the impression that it's hopeless... these
> attacks aren't impossible to stop --- it just requires convincing the
> transit providers to care.
>
> Damian
>
> On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG <
> nanog@nanog.org> wrote:
>
>> Hi Bill,
>>
>> thanks for sharing the data. Indeed, I can't offer you a way to backtrack
>> the spoofed packets.
>>
>> Anyway, I'm not sure what could you do legally as there is a very high
>> chance that these people are not in the USA and the CFAA won't apply to
>> them.
>>
>> Here is what I would do if I was in your situation.
>>
>> Since these packets are spoof and malformed, I would block all SYN/ACK
>> based on the length.
>>
>> Depending on your hardware, it's very easy to inspect *only the SYN/ACK
>> by length* if you have modern hardware. On linux/unix, it's also very
>> straightforward. I'm not sure for windows though.
>>
>> Here is the details of the analysis:
>>
>> Today, all the SYN and SYN/ACK includes a minimum of options like MSS,
>> WS, SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be
>> others, but let's use the basic one.
>>
>> In your case, there are none. There is only MSS and the SYN length is 44
>> bytes. These are spoof packets maybe generated by either TCP-AMP like
>> reported earlier.
>>
>> I would try to block all SYN/ACK coming toward your network with a length
>> of 44 bytes or lower. But, this is weird because it should be 54 bytes.
>> Maybe there is some offloading of some sort in your gear.
>>
>> Now depending on your hardware, it could work or it could kill your
>> router as it will punt the cpu. I guess you have some modern gear.
>>
>> What I do when I am not sure about the length, I start to accept and log
>> at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins.
>>
>> Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower
>> than X bytes. It won't kill or block anything legitimate if you do it
>> properly. :)
>>
>> What will happen is that you will not reply to these spoof SYN/ACK with a
>> RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your
>> service providers will be happy and should not penalize you.
>>
>> I'm pretty sure that it will help you as it did for me in the past.
>>
>> Let me know if it's not clear and/or which part is foggy and I'll try to
>> give more details and better explanation.
>>
>> Regards,
>>
>> Jean St-Laurent
>> On 2020-03-14 11:46, William Herrin wrote:
>>
>> On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via 
>> NANOG  wrote:
>>
>> can you post some forged packets please? You can send them offlist if
>> you prefer.
>>
>> Hi Jean,
>>
>> Here are a couple examples (PDT this morning):
>>
>> 08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none],
>> proto ICMP (1), length 56)
>> 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable -
>> admin prohibited filter, length 36
>> IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP
>> (6), length 40)
>> 199.33.225.218.9851 > 45.89.93.26.443: [|tcp]
>> 0x:  4500 0038 277c  3701 28da 2d59 5d1a
>> 0x0010:  c721 e1da 030d 4b61   4500 0028
>> 0x0020

Re: WIKI documentation Software?

[Grimes, Greg]

I know I'm a bit late to the conversation.  We have been using PMWiki for well 
over 10 yrs now.  At the time we started using it there weren't a lot of Wikis 
out there.  MediaWiki obviously was the most popular, but it did not provide 
the level of secure access that we wanted.  We didn't want everyone to be able 
to edit certain pages.  It was also very easy to integrate into CAS.  I wrote a 
cookbook for it years ago.  We use groups to allow only certain people to edit 
certain pages.  We also restrict viewing of some pages.  Our Security Group 
keeps some of their stuff restricted.  I work for the network team and we 
prevent everyone but our team from editing our pages.  They can view them, just 
can't mess with them.  Hope this helps someone.

https://www.pmwiki.org/


--

Greg T. Grimes
Senior Network Analyst
Information Technology Services
Mississippi State University
greg.gri...@msstate.edu
662-325-9311


From: NANOG  on behalf of Yang Yu 

Sent: Sunday, March 15, 2020 06:59
To: Brielle
Cc: NANOG list
Subject: Re: WIKI documentation Software?

On Sat, Mar 14, 2020 at 7:07 AM Brielle  wrote:
>
> I personally like Dokuwiki a lot.
>
> From a usability standpoint, once you spend a few learning the interface, 
> it’s very simplistic and not overwhelming in features.  You can always add 
> extensions for stuff you need that isn’t there out of box.
>
> From a technical standpoint, it doesn’t need a database.  The entire 
> structure is text files, so it can be run on even a super small VM, and doing 
> backups is as easy as tarballing the data directory.
>
> It’s got support for LDAP for authentication too, which might be useful.

+1 for dokuwiki

easy to maintain, has enough features while not become distracting

only complaint is that it doesn't support markdown, but the syntax is
easy enough (much easier than MediaWiki imo)

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 15/Mar/20 16:59, Keith Medcalf wrote:
> If it is "critical" you need a dedicated circuit.  If it is "meh, who gives a 
> shit", then you can go though the Internet.
>
> The root of the issue is that some idiot did a bad Risk Assessment.  Hope it 
> got fired or killed so it won't do this again in the future.
>
> Hope you also learned something as well.  Freedom of the Press belongs to He 
> Who Owns the Press.  If you are using someone else's presses (particularly 
> without directly paying and contracting with that party for the use of their 
> presses), you will live or die according to the whim of the owner of the 
> Press, and there is SFA you can do about that.  That is how the world has 
> worked for billions of years.  You would think people would understand that 
> by now.

The Internet has become its own enemy.

The time I realized it gives people more mental than practical hope in
the possibility of anything is when a pre-sales engineer once asked me
if we could deliver a circuit to a customer without using a CPE, because
that would increase their acquisition costs. RFC's 1149, 2549 and 6214
came to memory. This was 2012.

The Internet has become so ubiquitous and inspired significant (almost
unreasonable) possibilities that it is just about preposterous to
convince those that need to sign invoices that "Ummh, you get what you
pay for is as relevant in 2020 as it was in 1980".

Then again, you can buy an SDN or an SD-WAN or an IoT, to back up your
Big Data over the 5G connection you gathered it, and all will be well.

Mark.

RE: COVID-19 vs. our Networks

[Keith Medcalf]

If it is "critical" you need a dedicated circuit.  If it is "meh, who gives a 
shit", then you can go though the Internet.

The root of the issue is that some idiot did a bad Risk Assessment.  Hope it 
got fired or killed so it won't do this again in the future.

Hope you also learned something as well.  Freedom of the Press belongs to He 
Who Owns the Press.  If you are using someone else's presses (particularly 
without directly paying and contracting with that party for the use of their 
presses), you will live or die according to the whim of the owner of the Press, 
and there is SFA you can do about that.  That is how the world has worked for 
billions of years.  You would think people would understand that by now.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

>-Original Message-
>From: NANOG  On Behalf Of Mike Bolitho
>Sent: Saturday, 14 March, 2020 12:02
>To: Clayton Zekelman 
>Cc: nanog 
>Subject: Re: COVID-19 vs. our Networks
>
>First of all, we use a mixture of layer 2/3 private lines and DIA
>circuits. You don't know our infrastructure, stop being condescending. It
>goes against the spirit of this mailing list.
>
>Second, yes, the Internet is protected. Both public and private lines. I
>know this because we have TSP coded circuits and I spent four years at a
>Tier I ISP servicing TSP coded circuits
>
>Third, the trouble we had was a third party service having congestion
>issues. They are hosted by the same CDN as Call of Duty. The problem was
>both outside of our control and our third party service's control. The
>chokepoint was between ISPs/IXPs and the CDN. I've seen this time and
>time again while working at the aforementioned ISP. Saturated links on
>ISP/IXP/CDN networks. This is where the TSP code comes in. In this day
>and age of cloud services, it is financially unfeasible for every company
>to have a private line to every single cloud provider. That's
>preposterous to even suggest.
>
>- Mike Bolitho
>
>
>On Sat, Mar 14, 2020 at 10:40 AM Clayton Zekelman  > wrote:
>
>
>
>
>   The Internet is not a telecommunications service, according to your
>FCC.  If you want predictability, buy WAN circuits, not Internet
>circuits.   If your provider is co-mingling Internet and WAN traffic
>(i.e. circuits with defined endpoints vs. public Internet or VPN), then
>you need to talk to them about their prioritization.
>
>   If you have mission critical applications, put them on mission
>critical infrastructure, not the public Internet.
>
>   Oh, that's right - Internet circuits are cheaper than WAN circuits
>
>
>Clayton Zekelman
>Managed Network Systems Inc. (MNSi)
>3363 Tecumseh Rd. E
>Windsor, Ontario
>N8W 1H4
>
>tel. 519-985-8410
>fax. 519-985-8409

Re: sflow -> aggregated aspath visualization?

[Mark Tinka]

On 15/Mar/20 13:32, Yang Yu wrote:

>
>
> I haven't used Kentik in production, but heard good things about it

We've been on Kentik for about a year. Good things...

Definitely beats the Arbor/Netscout number we had before.

Mark.

Re: WIKI documentation Software?

[Yang Yu]

On Sat, Mar 14, 2020 at 7:07 AM Brielle  wrote:
>
> I personally like Dokuwiki a lot.
>
> From a usability standpoint, once you spend a few learning the interface, 
> it’s very simplistic and not overwhelming in features.  You can always add 
> extensions for stuff you need that isn’t there out of box.
>
> From a technical standpoint, it doesn’t need a database.  The entire 
> structure is text files, so it can be run on even a super small VM, and doing 
> backups is as easy as tarballing the data directory.
>
> It’s got support for LDAP for authentication too, which might be useful.

+1 for dokuwiki

easy to maintain, has enough features while not become distracting

only complaint is that it doesn't support markdown, but the syntax is
easy enough (much easier than MediaWiki imo)

Re: sflow -> aggregated aspath visualization?

[Yang Yu]

On Sat, Mar 14, 2020 at 12:33 PM Adam Thompson 
wrote:

> I’m looking for product recommendations:
>
>
>
> We’ve noticed that about 20% of our traffic here lately has decamped from
> the free (or, at least, flat-rate) connection to CANARIE (our R&E network)
> and its various connected content-delivery networks, and onto our
> commercial provider.
>
> While this is presumptively a legitimate shift, we’d like to better
> understand these changes when they occur, in a way that our executive can
> understand at a glance.
>
> We do have sFlow (et al.) going to an Arbor PeakFlow box for analysis, but
> it’s lacklustre at best at understanding changes like this.
>
> I want:
>
>- Top #n ASNs by traffic volume, per router/interface, stacked chart
>- Some way to visualize large jumps in that dataset, e.g. if
>Cloudflare ditched their CANARIE connection and now that traffic all goes
>commercial, I don’t know what sort of graphic would be useful, maybe a
>stacked polar chart so you could see when an AS jumped from one sector to
>another?  Even stacked bar charts could be useful.
>
>
I haven't used Kentik in production, but heard good things about it

https://techfieldday.com/video/the-kentik-experience-an-overview-demo-with-akshay-dhawale/
https://techfieldday.com/video/kentik-interconnection-and-metrics-from-kentik-for-service-provider-networks/



Just a reminder network devices might not export 100% samples/flows
correctly (sampling rate/export rate limitation, dropped packets on
ingress/egress, recirculated packet, policy routing actions, multiple
routing tables/vrf). The accuracy/availability of metadata in flow itself
(sFlow Extended Flow Data, sFlow input/output/source interface, IPFIX
information elements that are not directly extracted from packet lookup
header) might have limitations

Re: China’s Slow Transnational Network

[Mark Tinka]

On 15/Mar/20 09:55, Pengxiong Zhu wrote:

> I know Caida has one paper on the congestion on Africa's IXPs substrate.

I can't think of a single IXP in Africa that is "congested".

Do you have more data?


> However, we did find Kenya, Nigeria and South Africa have better
> transnational performance than China,...

Not having great big firewalls tends to help :-).


> while the performance of Ghana and Egypt was worse than China, at
> least that's what we saw from the web4africa VPSes we brought.

While I can't speak to the national backbones of Ghana and Egypt, it
would be good to obtain multiple perspectives, just to be sure.


> Sorry I am a bit confused here. What do you mean by "these networks"?
> When you say "peering outside of China", who is peering who exactly?

I know about Chinese operators who will deliberately congest peering
ports to influence 3rd party network behaviour.

Mark.

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

>
> Most countries in Africa do not implement great big firewalls. Our
> problems are quite different :-\...
>

I know Caida has one paper on the congestion on Africa's IXPs substrate.
However, we did find Kenya, Nigeria and South Africa have better
transnational performance than China, while the performance of Ghana and
Egypt was worse than China, at least that's what we saw from the web4africa
VPSes we brought.

We've seen somewhat similar behaviour from these networks when peering
> outside of China
>

Sorry I am a bit confused here. What do you mean by "these networks"? When
you say "peering outside of China", who is peering who exactly?

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Tue, Mar 3, 2020 at 12:17 AM Mark Tinka  wrote:

>
>
> On 3/Mar/20 00:57, Tom Paseka via NANOG wrote:
>
> >
> > Prices are set artificially high, so their interconnection partners
> > wont purchase enough capacity. additionally, the three don't
> > purchase enough to cover demand for their own network. Results in
> > congestion.
>
> We've seen somewhat similar behaviour from these networks when peering
> outside of China, perhaps, to influence the flow of money and traffic.
>
> We have zero patience for such things.
>
> Mark.
>