Re: Command and Control Centres | COVID-19

[Mel Beckman]

We have data centers in two states, with a six-hour drive between them. Our 
normal practice has been to drive from the main DC to the satellite every few 
weeks to catch up on maintenance and non-urgent infrastructure changes such as 
server installs. For urgent things we have some part-time “remote hands” techs 
that are limited to cable changes, reboots, and diagnostic assistance. 
Everything else is done unattended using remote management.

But we do have a large customer install coming up in a few weeks that will be 
more of a problem. It can’t really be rescheduled, because the circuits 
involved are part of a larger critical network infrastructure. But there will 
be some racking and stacking that requires several people working in the same 
area. This project also requires a generator upgrade, which is outdoor 
construction work. I’m hoping the CV situation improves so that we don’t have 
to use some kind of lightweight hazmat gear. 

 -mel 

> On Apr 6, 2020, at 9:27 AM, Ben Cannon  wrote:
> 
> We are leaving it at everyone’s discretion where we can. We have a big open 
> pair of GNOCs with nobody closer than 6 feet by far, and with one or two 
> people in each while everyone else is WFH, it’s essentially the run of the 
> place.  
> 
> Anything that can be done at home is generally encouraged - even before this. 
>  
> 
> So it’s really only the very sensitive ops that are even noticing.
> 
> -Ben
> 
>> On Apr 6, 2020, at 7:51 AM, Scott E. MacKenzie  wrote:
>> 
>> All,
>> 
>> This question has arisen and I was wondering if I could request some
>> feedback from the community.  We operate a 24x7x365 Command and
>> Control Centre that provides mission critical services (Security
>> Operations, Network Operations, and Enterprise Management) as does
>> many on this list.
>> 
>> How many on the list have sent all personnel home using work from home
>> practices and home many have opted to run skeleton crews while
>> implementing tight social distancing restrictions?  How many are
>> operating status quo?
>> 
>> We are trying to find a balanced position and I was wondering what is
>> the communities position on this topic?
>> 
>> 
>> Scott

[NANOG-announce] Get $10k for your education!

[NANOG Marketing]

*Empowering the Next Generation  *

We believe in fostering the education and professional development of our
next generation of networking engineers, operators, and architects. NANOG’s
annual scholarships provide students with the funding and resources they
need to grow and evolve in their education and careers.

Applications for our $10,000 awards are OPEN  Learn more
 about eligibility and
selection process, plus view past recipients, before beginning your
application.

Apply Now 
___
NANOG-announce mailing list
NANOG-announce@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog-announce

Re: Command and Control Centres | COVID-19

[Ben Cannon]

We are leaving it at everyone’s discretion where we can. We have a big open 
pair of GNOCs with nobody closer than 6 feet by far, and with one or two people 
in each while everyone else is WFH, it’s essentially the run of the place.  

Anything that can be done at home is generally encouraged - even before this.  

So it’s really only the very sensitive ops that are even noticing.

-Ben

> On Apr 6, 2020, at 7:51 AM, Scott E. MacKenzie  wrote:
> 
> All,
> 
> This question has arisen and I was wondering if I could request some
> feedback from the community.  We operate a 24x7x365 Command and
> Control Centre that provides mission critical services (Security
> Operations, Network Operations, and Enterprise Management) as does
> many on this list.
> 
> How many on the list have sent all personnel home using work from home
> practices and home many have opted to run skeleton crews while
> implementing tight social distancing restrictions?  How many are
> operating status quo?
> 
> We are trying to find a balanced position and I was wondering what is
> the communities position on this topic?
> 
> 
> Scott

Re: Command and Control Centres | COVID-19

[Bill Blackford]

I think that several businesses already have a BCP in place that includes
work from home and a pre-built VPN infrastructure. I can't speak for
business units I'm unfamiliar with, but for Engineering/Ops, this is status
quo.

On Mon, Apr 6, 2020 at 7:52 AM Scott E. MacKenzie 
wrote:

> All,
>
> This question has arisen and I was wondering if I could request some
> feedback from the community.  We operate a 24x7x365 Command and
> Control Centre that provides mission critical services (Security
> Operations, Network Operations, and Enterprise Management) as does
> many on this list.
>
> How many on the list have sent all personnel home using work from home
> practices and home many have opted to run skeleton crews while
> implementing tight social distancing restrictions?  How many are
> operating status quo?
>
> We are trying to find a balanced position and I was wondering what is
> the communities position on this topic?
>
>
> Scott
>


-- 
Bill Blackford

Logged into reality and abusing my sudo privileges.

Re: Command and Control Centres | COVID-19

[Jeff Shultz]

On Mon, Apr 6, 2020 at 7:53 AM Scott E. MacKenzie  wrote:
>
> All,
>
> This question has arisen and I was wondering if I could request some
> feedback from the community.  We operate a 24x7x365 Command and
> Control Centre that provides mission critical services (Security
> Operations, Network Operations, and Enterprise Management) as does
> many on this list.
>
> How many on the list have sent all personnel home using work from home
> practices and home many have opted to run skeleton crews while
> implementing tight social distancing restrictions?  How many are
> operating status quo?
>
> We are trying to find a balanced position and I was wondering what is
> the communities position on this topic?
>
>
> Scott

We're tiny - only a 3 person shop. But so far it's been the two
workerbees rotating between home (we have VOIP phones and VPN access)
and the office, and the boss is in or out on his own call. And we all
have boxes of gloves.

One of the workerbees does the majority of fiber hanging, and he'll
come in in the morning to do the day's worth of that if needed when
he's on his "home" week.

It's been working so far.

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: FCC and FTC Demand Cut-Off Robercallers of Coronavirus Scams

[Tom Beecher]

I was watching one of Jim Browning's s Youtube series the other day, the
one where he got into a scam call center's network so completely that he
had access to their entire operation, including CCTV cameras, and
eventually got BBC Panorama involved, which got the place shut down.

He mentioned that with all the work he does in this area, he has thousands
of call recordings of scams taking place, and he's working with a
university to try and see if they can make an AI call bot, essentially
Lenny 2.0, that would be near impossible for scammers to detect.

Interested to see how that shakes out. :)

On Sat, Apr 4, 2020 at 1:21 PM Dan Hollis  wrote:

> Sadly I've discovered that >95% of scammers have caught on to lenny by
> now, and hang up within the first few seconds of hearing him.
>
> I guess they've been thoroughly lenny'd already so he's no longer
> effective.
>
> -Dan
>
> On Fri, 3 Apr 2020, JASON BOTHE via NANOG wrote:
>
> > I just need my phone to have a warm transfer button with or without
> supervision to Lenny https://toao.net/595-lenny
> >
> >> On Apr 3, 2020, at 18:40, Grimes, Greg  wrote:
> >>
> >> 
> >> I was thinking the EXACT same thing!!
> >>
> >> --
> >> Greg Grimes
> >> Senior Network Analyst
> >> Information Technology Services
> >> Mississippi State University
> >> 662-325-9311(w)
> >>
> >> From: NANOG  on behalf of Clayton Zekelman
> 
> >> Sent: Friday, April 3, 2020 6:21:43 PM
> >> To: Sean Donelan ; nanog@nanog.org 
> >> Subject: Re: FCC and FTC Demand Cut-Off Robercallers of Coronavirus
> Scams
> >>
> >>
> >> Finally, but why did it take a pandemic to get them to do this?
> >>
> >> At 07:14 PM 03/04/2020, Sean Donelan wrote:
> >>
> >>> A sternly worded, finger-wagging press release.
> >>>
> >>>
> >>>
> https://secure-web.cisco.com/1LWeqrXLGvJdEBzB1uFe8kj9AI4aQYKo58pr0a7HHabjzZjlUFhbYW9sw_3phNW8RRZcfh4T01zhFWJzwlT5koYKFBC0X9DhlUbUeWJCpcaJDWoGiw4jEvVGWiHMyWhb-DgXaHwKqs4DEaqsgXzJvXllUvcmj0hqGdV7dPWOJjhFPMUEnjT8Grl3W7MQ7A5v1nC1W9_K01pTSV8PsbPRjlTzYrA20dcqjx74JJSmlZDRnMsoPxJoZcH2jQ00PAsRaeGGdnA4EE5KwbCYlUu4M0UhYHkKmOkfjSJjZyBfAALCJQveH8qYnTfkSoI5OzVKm1aapZq23qJsbv8OReeHk8w/https%3A%2F%2Fwww.fcc.gov%2Fdocument%2Ffcc-ftc-demand-gateway-providers-cut-covid-19-robocall-scammers
> >>>
> >>> [...]
> >>> The FCC¢s Enforcement Bureau and FTC¢s Bureau of
> >>> Consumer Protection wrote to three gateway
> >>> providers that are facilitating these scam
> >>> COVID-19-related robocalls: SIPJoin of Suffolk,
> >>> Virginia; Connexum of Orange, California; and
> >>> VoIP Terminator/BLMarketing of Lake Mary,
> >>> Florida. The companies have been identified by
> >>> the Traceback Group, a consortium of phone
> >>> companies that help officials track down suspect
> >>> calls, managed by the trade association
> >>> USTelecom. The Commissions also wrote to
> >>> USTelecom to ask its members to begin blocking
> >>> calls from these providers if the flood of
> >>> robocalls is  not cut off within 48 hours.
> >>
> >> --
> >>
> >> Clayton Zekelman
> >> Managed Network Systems Inc. (MNSi)
> >> 3363 Tecumseh Rd. E
> >> Windsor, Ontario
> >> N8W 1H4
> >>
> >> tel. 519-985-8410
> >> fax. 519-985-8409
> >>
> >>
> >
>

Re: IS-IS on FRR - Is Anyone Running It?

[John St.Martin]

Mark, Thanks for sharing your experiences with FRR. I don't work with
IS-IS, but have found the development to be very active and fixing
reproducible bugs quickly.

It look like FRR put a patch in for the bug you referenced and have a test
build from 3/21 available which allows for up to 16k MTU @
https://ci1.netdef.org/browse/FRR-FRRPULLREQ-11364/artifact

I hope this helps and please continue to share your progress with the
community.

On Sat, Apr 4, 2020, 11:18 AM Mark Tinka  wrote:

>
>
> On 3/Apr/20 21:28, Eduardo Schoedler wrote:
> > Mark,
> >
> > Did you tried this:
> >
> https://lists.freebsd.org/pipermail/freebsd-current/2006-December/068011.html
> >
> > There are some knobs for Freebsd:
> > http://nginx.org/en/docs/freebsd_tuning.html
>
> So the problem really isn't FreeBSD itself. IS-IS looks at MTU in order
> to work, and if nothing is manually set, it infers the MTU from the
> physical interface over which it is enabled.
>
> While the current IS-IS implementation in FRR is buggy to the extent
> that it does not assume MTU can be larger than 8,192 bytes, that does
> not prevent an operator from telling it what MTU it should use for IIH
> messages, provided the physical interface can support it.
>
> Suffice it to say, I already have a number of Sysctl and kernel knobs in
> FreeBSD to tune the system for the Anycast services we run on them. I'd
> be disinclined to mess about with that as I don't think it has any
> bearing on an over-the-top service such as IS-IS.
>
> Quagga runs well (OSFP though, I'll admit), and I'll keep looking for
> answers on IS-IS in FRR until it stops making sense.
>
> Mark.
>

Command and Control Centres | COVID-19

[Scott E. MacKenzie]

All,

This question has arisen and I was wondering if I could request some
feedback from the community.  We operate a 24x7x365 Command and
Control Centre that provides mission critical services (Security
Operations, Network Operations, and Enterprise Management) as does
many on this list.

How many on the list have sent all personnel home using work from home
practices and home many have opted to run skeleton crews while
implementing tight social distancing restrictions?  How many are
operating status quo?

We are trying to find a balanced position and I was wondering what is
the communities position on this topic?


Scott

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 14:52, Radu-Adrian Feurdean wrote:

>
> Beware of bad dog^H^H^H NCS model. On NCS5000 (don't know about 5500 - those 
> arrived months after me leaving $job[-1]) you will get the nasty surprise of 
> discovering that they have some limits to 9150(IP)/9164(eth), even if you can 
> set the mtu higher (to an unusable 9192 or 92zz bytes).

A case of classic IOS XR not sync'ing with a Broadcom-based chipset,
perhaps?

Y'all know my feelings on that :-).

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Radu-Adrian Feurdean]

On Mon, Apr 6, 2020, at 10:58, Mark Tinka wrote:
> Within our core, we run 9,178 bytes (which translates to 9,192 bytes on
> Junos and IOS XR), to support the transport of Jumbo frames for

Beware of bad dog^H^H^H NCS model. On NCS5000 (don't know about 5500 - those 
arrived months after me leaving $job[-1]) you will get the nasty surprise of 
discovering that they have some limits to 9150(IP)/9164(eth), even if you can 
set the mtu higher (to an unusable 9192 or 92zz bytes).

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 12:57, Dale Shaw wrote:

>
> Since vsphere67u3 (ESXi 6.7 U3), you can set an MTU of up to 9,190 bytes:

So we moved from 6.0 to 6.7 last year, when we refreshed the servers.

Not sure why they hid the fact that they can breach the 9,000 mark:

   
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc/GUID-D05444F8-8C34-43CD-93D0-5F3C2369DE01.html
   
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.networking.doc/GUID-40856C1E-7631-4228-A111-13A783316595.html



> The host UI doesn't let you go >9000 bytes so (for now) you need to
> use the CLI.

Yeah... RR's are critical infrastructure. I'd prefer not to mess
with them beyond what is necessary :-).


>
> Are your FRR VM vNICs attached to standard or distributed vswitch port
> groups?

So we only worry about this on our RR's.

Other VM's that may be running are not a huge concern.

Despite our network being able to carry 9,192 bytes, we only guarantee
9,000 for services and customers, end-to-end. Same applies to internal
services we operate.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Dale Shaw]

Hi Mark,

On Mon, 6 Apr 2020 at 18:51, Mark Tinka  wrote:
>
> However, VMware ESXi does not
> support anything larger than 9,000 bytes, and that is where we run our
RR's.

Since vsphere67u3 (ESXi 6.7 U3), you can set an MTU of up to 9,190 bytes:

[root@esxi01o:~] esxcli network vswitch standard list
vSwitch2
   Name: vSwitch2
   Class: cswitch
   Num Ports: 3456
   Used Ports: 1
   Configured Ports: 128
   MTU: 9000
   CDP Status: listen
   Beacon Enabled: false
   Beacon Interval: 1
   Beacon Threshold: 3
   Beacon Required By:
   Uplinks:
   Portgroups: pg-vlan204-LAB-STORAGE, pg-trunk, pg-vlan201-LAB-VMGMT

[root@esxi01o:~] esxcli network vswitch standard set -m=9190 -v vSwitch2
[root@esxi01o:~] esxcli network vswitch standard list
vSwitch2
   Name: vSwitch2
   Class: cswitch
   Num Ports: 3456
   Used Ports: 1
   Configured Ports: 128
   MTU: 9190
   CDP Status: listen
   Beacon Enabled: false
   Beacon Interval: 1
   Beacon Threshold: 3
   Beacon Required By:
   Uplinks:
   Portgroups: pg-vlan204-LAB-STORAGE, pg-trunk, pg-vlan201-LAB-VMGMT

The host UI doesn't let you go >9000 bytes so (for now) you need to use the
CLI.

Are your FRR VM vNICs attached to standard or distributed vswitch port
groups?

You'll also need >9,000 byte MTU support in the vNIC driver, obviously.

Cheers,
Dale

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 12:47, Saku Ytti wrote:

>
> Good good, if you are particularly concerned, match 8870 etype,
> len>1500 and drop on upstream router :). So should someone do
> something funky on your FRR, at least that's not the moment you test
> what your rest of the network think of large LSPs.

:-).

First, let me get the adjacencies up. FRR seems to have its own ideas
about what HMAC-MD5 is vs. the upstream IOS XE number.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 13:40, Mark Tinka  wrote:

> On these servers, I'm pushing only 2 routes into the IS-IS domain.

Good good, if you are particularly concerned, match 8870 etype,
len>1500 and drop on upstream router :). So should someone do
something funky on your FRR, at least that's not the moment you test
what your rest of the network think of large LSPs.

-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 12:31, Saku Ytti wrote:

> So FRR should have an addition of LSP-MTU which should default
> to 1492B to avoid interoperability issues when it must generate large
> LSP PDU.

A couple of weeks ago, my Google-fu led to me some kind of "lsp-mtu"
command for FRR. I tried it everywhere but it wasn't supported.


>
> So better not make config where FRR needs to inject larger LSP PDU,
> might be more excitement than what people would like. Someone can test
> what happens when you redistribute more prefixes than can fit in 1492B
> LSP PDU and if those LSPs propagate to Ciscos and Juniper, blackhole
> or crash the network.

On these servers, I'm pushing only 2 routes into the IS-IS domain.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 13:12, Mark Tinka  wrote:

> > https://github.com/FRRouting/frr/blob/58980443821edf95719984e01f31720bd1dc7f79/isisd/isis_constants.h#L172-L180
> >
> > But as long as you don't pad, your PDU shouldn't exceed 1500B.
>
> Good man, you gave me an idea.

There are other interesting implications about this high MTU commit -
https://github.com/FRRouting/frr/commit/316a98ecd151dfba86a4a87e11c98cb2a0e94518

If in FRR you generate large LSP (many interfaces or prefixes), FRR
may actually generate as large LSP as MTU allows. This means it
doesn't interoperate between Cisco and Juniper, which will fragment
generated LSPs at 1492B, no matter what CLSN MTU you have. On Cisco
you can increase LSP MTU to 4352, on Juniper you cannot increase above
1492B. So FRR should have an addition of LSP-MTU which should default
to 1492B to avoid interoperability issues when it must generate large
LSP PDU.

So better not make config where FRR needs to inject larger LSP PDU,
might be more excitement than what people would like. Someone can test
what happens when you redistribute more prefixes than can fit in 1492B
LSP PDU and if those LSPs propagate to Ciscos and Juniper, blackhole
or crash the network.

-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 12:00, Saku Ytti wrote:

> Aah found it, it does do Cisco hack.
>
> https://github.com/FRRouting/frr/blob/58980443821edf95719984e01f31720bd1dc7f79/isisd/isis_constants.h#L172-L180
>
> But as long as you don't pad, your PDU shouldn't exceed 1500B.

Good man, you gave me an idea.

Ran through the documentation and found out how to deal with Hello
Padding in FRR:

interface em0
 ip router isis 1
 ipv6 router isis 1
 no isis hello padding

Looks like IS-IS now initializes:

2020/04/06 10:09:21 ISIS: Opened BPF device /dev/bpf0
2020/04/06 10:09:21 ISIS: BPF buffer len = 4096
2020/04/06 10:09:21 ISIS: failed to set BPF buffer len (4096 to 9000)
2020/04/06 10:09:21 ISIS: circuit already connected
2020/04/06 10:09:21 ISIS: circuit already connected
2020/04/06 10:09:21 ISIS: circuit already connected

Let me continue with the configuration and report back.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 11:58, Saku Ytti wrote:

> Why is PDU 9014, if you don't have padding? I wonder what FRR even
> does at >1500B, I don't see '8870' in source code, so I don't think it
> supports the EthernetII hack.

I wondered about that when I saw it, and just assumed that FRR was
account for the 14 bytes Ethernet header.


>
> I wonder did someone verify the fix, to see what it even does? To me
> it looks like what it'll do, it'll use what ever ethernetII matches
> byte-size of frame, which is decidedly, not correct and would not
> interoperate after 'fix'.

Hard to say. I can only try once it ships.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

Aah found it, it does do Cisco hack.

https://github.com/FRRouting/frr/blob/58980443821edf95719984e01f31720bd1dc7f79/isisd/isis_constants.h#L172-L180

But as long as you don't pad, your PDU shouldn't exceed 1500B.


On Mon, 6 Apr 2020 at 12:58, Saku Ytti  wrote:
>
> From your original post:
>
> > 2020/03/21 03:12:36 ISIS: isis_send_pdu_bcast: sock_buff size 8192 is less 
> > than output pdu size 9014 on circuit em0
> > 2020/03/21 03:12:36 ISIS: [EC 67108865] ISIS-Adj (1): Send L2 IIH on em0 
> > failed
>
> Why is PDU 9014, if you don't have padding? I wonder what FRR even
> does at >1500B, I don't see '8870' in source code, so I don't think it
> supports the EthernetII hack.
>
> I wonder did someone verify the fix, to see what it even does? To me
> it looks like what it'll do, it'll use what ever ethernetII matches
> byte-size of frame, which is decidedly, not correct and would not
> interoperate after 'fix'.
>
> --
>   ++ytti



-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 11:48, Saku Ytti wrote:

>
> Ok, and because this particular FRR VM does more than just ISIS, you
> want the extra mtu between 9k and 8192?

Yes sir.

FRR is just another service running on the box, mainly to pull traffic
toward it.


>
> Change MTU after starting ISIS? :>

Hehe, I'm old now - I try to look for excitement outside of network
operations :-).

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

>From your original post:

> 2020/03/21 03:12:36 ISIS: isis_send_pdu_bcast: sock_buff size 8192 is less 
> than output pdu size 9014 on circuit em0
> 2020/03/21 03:12:36 ISIS: [EC 67108865] ISIS-Adj (1): Send L2 IIH on em0 
> failed

Why is PDU 9014, if you don't have padding? I wonder what FRR even
does at >1500B, I don't see '8870' in source code, so I don't think it
supports the EthernetII hack.

I wonder did someone verify the fix, to see what it even does? To me
it looks like what it'll do, it'll use what ever ethernetII matches
byte-size of frame, which is decidedly, not correct and would not
interoperate after 'fix'.

-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 12:43, Mark Tinka  wrote:

> Hello Padding is disabled by default in our IS-IS core, so the other
> side isn't doing it already.
>
> The problem you have is IS-IS on FreeBSD won't initialize because it
> sees the physical interface running at 9,000 bytes, and yet its code can
> only go up to 8,192 bytes.

Ok, and because this particular FRR VM does more than just ISIS, you
want the extra mtu between 9k and 8192?

Change MTU after starting ISIS? :>

-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 11:28, Saku Ytti wrote:

> You want to run your physical at high MTU and you also like ISIS to come up?

The services running on the server benefit from the high MTU. That's the
whole point of the server... to run the services, not to run a routing
protocol.

So I don't want to lower the physical MTU because that will impact the
performance of the services on the server that depend on it.

For OSPF on Quagga, we just run "ip ospf mtu-ignore" and "ipv6 ospf6
mtu-ignore" to get around this, and this works fine.


>
> FRR doesn't seem to support Cisco proprietary EthernetII based ISIS,
> so it'll only do 802.3 standard. Have you tried to just use what ever
> interface MTU, and NOT pad on far-end cisco? If I read the code right,
> this should be then standard max_size 802.3 and it should come up.

Hello Padding is disabled by default in our IS-IS core, so the other
side isn't doing it already.

The problem you have is IS-IS on FreeBSD won't initialize because it
sees the physical interface running at 9,000 bytes, and yet its code can
only go up to 8,192 bytes.

Fair point, that bug has been fixed and just needs to be committed to
upcoming code. But again, if I can manually tell IS-IS to run at a lower
MTU than what it is trying to infer from the interface, we shall be golden.

How do I do that, is the question?


>
> If you force Cisco to pad, it'll use the EthernetI hack, which is not
> supported by FRR, and it won't work. I noticed in lab just now, if I
> set 'pad sometimes' on IOS-XR, it won't pad at all for >1500B MTUs, so
> seems like 'sometimes' stops Cisco from using the EthernetII hack.

    "no hello padding" is default for our core.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 11:25, Saku Ytti wrote:

> Quite and then you specified but for ISIS you run 8000. I'm only
> talking about your ISIS here, not rest. And that 8k doesn't do
> anything useful,

Except for our CSR1000v on ESXi. We only ever needed it when we went
that route. We didn't need it prior to that.


>  so you can run 1500B to your FRR and not lose
> anything.

Which is my point - there is nowhere in FRR to set this, that is
documented or has worked from experiment/general knowledge.


> As I explained this is not true, by standard ISIS cannot do this for
>> 1500B MTU. As ISIS is 802.3 and 802.3 cannot send larger frames.
> Cisco has proprietary hack to do EthernetII for hellos, but even that
> doesn't guarantee anything, because ISIS will still come up, even if
> it receives subMTU hello as 802.3.

Again, this is only required on our CSR1000v ESXi deployments, because
our experience was that IS-IS would not establish because neighbors were
running at a different MTU from the RR's, and IS-IS on the RR's was
trying to setup adjacencies at a physically higher MTU than ESXi could
support.

So lowering it fixed it, and we just picked 8,000. We could have picked
2,000, or 1,500, or whatever. The actual number doesn't matter, as long
as it's the same on the boxes that need it.

We spent about 6hrs troubleshooting this back in 2014, so we didn't put
it in just for fun.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 12:16, Mark Tinka  wrote:

> Which is what this whole thread is about. How do I set that, manually,
> without changing the physical interface MTU?

You want to run your physical at high MTU and you also like ISIS to come up?

FRR doesn't seem to support Cisco proprietary EthernetII based ISIS,
so it'll only do 802.3 standard. Have you tried to just use what ever
interface MTU, and NOT pad on far-end cisco? If I read the code right,
this should be then standard max_size 802.3 and it should come up.

If you force Cisco to pad, it'll use the EthernetI hack, which is not
supported by FRR, and it won't work. I noticed in lab just now, if I
set 'pad sometimes' on IOS-XR, it won't pad at all for >1500B MTUs, so
seems like 'sometimes' stops Cisco from using the EthernetII hack.

-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 12:13, Mark Tinka  wrote:

> Ummh, not really. As mentioned, we run 9,178 bytes on IOS and IOS XE,
> and 9,192 bytes on Junos and IOS XR, in our network.

Quite and then you specified but for ISIS you run 8000. I'm only
talking about your ISIS here, not rest. And that 8k doesn't do
anything useful, so you can run 1500B to your FRR and not lose
anything.

> The 8,000 bytes we run is done only on the IOS, IOS XE and CSR1000v
> boxes. It doesn't mean that we are trying to transport IS-IS PDU's at
> 8,000 bytes. It just means we can, and are capping it to just about what
> VMware ESXi can support.

As I explained this is not true, by standard ISIS cannot do this for
>1500B MTU. As ISIS is 802.3 and 802.3 cannot send larger frames.
Cisco has proprietary hack to do EthernetII for hellos, but even that
doesn't guarantee anything, because ISIS will still come up, even if
it receives subMTU hello as 802.3.


-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 11:05, Saku Ytti wrote:

> And the only thing this 'ISIS MTU' (think you mean CLNS MTU)...

Yes, I mean "clns mtu".


>  does, is
> for some cases make ISIS hellos larger. If you don't pad ISIS hellos,
> or if you have standard compliant ISIS, it doesn't do anything past
> 1500B.

Agreed.

But in the network devices, IS-IS will infer the interface MTU to setup
adjacencies. It doesn't matter what it is, it just needs to either be
the same or low enough for the interface to support it.



> More accurately ISIS doesn't have a way to fragment LSP into two LSP,
> so the whole ISIS network needs the same LSP-MTU, which has nothing to
> do with 'clns mtu', LSP-MTU does not increase with increasing CLNS
> MTU. There is no utility of having a harmonised CLNS MTU,...

Do you run CSR1000v on ESXi?


>  so if you
> want to speak to FRR, you can just do 1500B and it's fine.

Which is what this whole thread is about. How do I set that, manually,
without changing the physical interface MTU?

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 10:18, Saku Ytti wrote:

> And to your original question, no, nothing in Mark's ISIS network is
> above 1500, for the same reason.

Ummh, not really. As mentioned, we run 9,178 bytes on IOS and IOS XE,
and 9,192 bytes on Junos and IOS XR, in our network.

The 8,000 bytes we run is done only on the IOS, IOS XE and CSR1000v
boxes. It doesn't mean that we are trying to transport IS-IS PDU's at
8,000 bytes. It just means we can, and are capping it to just about what
VMware ESXi can support.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 11:50, Mark Tinka  wrote:

> and switches can forward up to 9,178 bytes, we set IS-IS MTU to 8,000
> bytes on all of them, as a lowest common denominator so that our RR's
> can participate in the IS-IS domain.

And the only thing this 'ISIS MTU' (think you mean CLNS MTU) does, is
for some cases make ISIS hellos larger. If you don't pad ISIS hellos,
or if you have standard compliant ISIS, it doesn't do anything past
1500B.

> It's got nothing to do with trying to transport CLNS frames or fragment
> them, since as you rightly point out, fragmentation is not supported at
> Layer 2.

More accurately ISIS doesn't have a way to fragment LSP into two LSP,
so the whole ISIS network needs the same LSP-MTU, which has nothing to
do with 'clns mtu', LSP-MTU does not increase with increasing CLNS
MTU. There is no utility of having a harmonised CLNS MTU, so if you
want to speak to FRR, you can just do 1500B and it's fine.

-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 09:58, Radu-Adrian Feurdean wrote:

> I won't speak for Mark, but NO, when you're carrying somebody's else's 
> traffic you do your best to have the MTU on each and every backbone link 
> "high enough" : preferably in the 9200(bytes) range, so you can easily 
> transport 9000(bytes) client packets, and by no means so small that you need 
> to fragment 1500(IP)/1514(Eth) byte packets. If things are really-really bad, 
> 1600 bytes towards the edge.

Yes, this.

Within our core, we run 9,178 bytes (which translates to 9,192 bytes on
Junos and IOS XR), to support the transport of Jumbo frames for
customers that need it, typically l2vpn type services.

If VMware ESXi did not limit itself to 9,000 bytes, we wouldn't have
needed to set the IS-IS MTU at all, as it would have been inferred from
the physical interface of each device.


>
> Back to the original question, I would expect FRR to be able to manually 
> specify a MTU/frame-size, like any other decent NOS (even if it's not a full 
> NOS).

One would have thought.

Even if no one is running it, I'd have assumed the developers would have
said something. My Google-fu is generally okay, but not on this one :-).

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 6/Apr/20 07:51, Saku Ytti wrote:

>
> I'm not sure what 'globally our IS-IS domain runs 8000 bytes' means.
> Your LSP MTU is like 1492B, there isn't a mechanism to fragment and
> reassemble LSP in-transit. ISIS network doesn't support different MTU
> sizes and I've not heard anyone being brave enough to increase LSP MTU
> above 1492B.
> The only thing that is larger in your network is hellos, and I'm not
> even sure how that works, considering 802.3 cannot signal larger
> frames than 1500B.

I mean that every router and switch interface in our network runs at
9,178 bytes. Ordinarily, we wouldn't need to do anything to IS-IS as it
would just take that and run with it. However, VMware ESXi does not
support anything larger than 9,000 bytes, and that is where we run our RR's.

So to make it all harmonious, while the actual data plane in our routers
and switches can forward up to 9,178 bytes, we set IS-IS MTU to 8,000
bytes on all of them, as a lowest common denominator so that our RR's
can participate in the IS-IS domain.

It's got nothing to do with trying to transport CLNS frames or fragment
them, since as you rightly point out, fragmentation is not supported at
Layer 2.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Mark Tinka]

On 5/Apr/20 21:24, Randy Bush wrote:
> ok, if IS-IS is kinda working on FRR, at least enough to get loopbacks
> and external interfaces around a pop, i gotta ask.

Not for me. I can't get it to even start, i.e., no link adjacencies due
to an inability to agree on MTU.

So if anyone has IS-IS on FRR running at all, please reach out.

Mark.

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 11:01, Radu-Adrian Feurdean
 wrote:

> Ethernet cannot signal MTU. But if you have equipment at both sides of a P2P 
> link, you don't need any signalling. You check the MTU suits your needs and 
> put it in statically. Same for NNIs : the MTU is signalled in a document 
> called "contract" or "agreement". And no, the guy that is being woken up at 
> 3am for an issue, if he/she doesn't know that we run Jumbo, then he/she 
> should start updating the CV.

You might want to have read the message and have a basic understanding
of ISIS before posting. ISIS is not ethernetII, ISIS is 802.3, which
in fact does signal packet size, and cannot signal larger than 1500B.
And to your original question, no, nothing in Mark's ISIS network is
above 1500, for the same reason.


-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Radu-Adrian Feurdean]

On Mon, Apr 6, 2020, at 07:51, Saku Ytti wrote:

> I'm not sure what 'globally our IS-IS domain runs 8000 bytes' means.
> Your LSP MTU is like 1492B, there isn't a mechanism to fragment and
> reassemble LSP in-transit. ISIS network doesn't support different MTU
> sizes and I've not heard anyone being brave enough to increase LSP MTU
> above 1492B.

I won't speak for Mark, but NO, when you're carrying somebody's else's traffic 
you do your best to have the MTU on each and every backbone link "high enough" 
: preferably in the 9200(bytes) range, so you can easily transport 9000(bytes) 
client packets, and by no means so small that you need to fragment 
1500(IP)/1514(Eth) byte packets. If things are really-really bad, 1600 bytes 
towards the edge.

> The only thing that is larger in your network is hellos, and I'm not
> even sure how that works, considering 802.3 cannot signal larger
> frames than 1500B.

Ethernet cannot signal MTU. But if you have equipment at both sides of a P2P 
link, you don't need any signalling. You check the MTU suits your needs and put 
it in statically. Same for NNIs : the MTU is signalled in a document called 
"contract" or "agreement". And no, the guy that is being woken up at 3am for an 
issue, if he/she doesn't know that we run Jumbo, then he/she should start 
updating the CV.

Back to the original question, I would expect FRR to be able to manually 
specify a MTU/frame-size, like any other decent NOS (even if it's not a full 
NOS).

Re: IS-IS on FRR - Is Anyone Running It?

[Saku Ytti]

On Mon, 6 Apr 2020 at 10:36, Anoop Ghanwani  wrote:

>> The only thing that is larger in your network is hellos, and I'm not
>> even sure how that works, considering 802.3 cannot signal larger
>> frames than 1500B.
>>
> Probably this method:
> https://en.wikipedia.org/wiki/EtherType#Jumbo_frames

Mikael Abrahamson and I quickly looked at this, some platforms (IOS,
IOS-XR, EOS) use ethernetII + 8870, but only for hellos and revert
back to standard 802.3 after. Some platforms (Junos) accept 8840
hello, but do not send it.
So after 1500B MTU detection does not work by-standard and everyone
has been forced to implement CiscoISIS to just get ISIS up, and Cisco
is forced to accept subMTU hello equally to interoperate with non8840
hosts.

It's really time to kill 802.3 and CLNS and do pure ISIS over
ethernetII+IPv6, no one really understands how all this works, it's
complex and fragile.


-- 
  ++ytti

Re: IS-IS on FRR - Is Anyone Running It?

[Anoop Ghanwani]

On Sun, Apr 5, 2020 at 10:52 PM Saku Ytti  wrote:

> The only thing that is larger in your network is hellos, and I'm not
> even sure how that works, considering 802.3 cannot signal larger
> frames than 1500B.
>
> Probably this method:
https://en.wikipedia.org/wiki/EtherType#Jumbo_frames