Re: 10G CPE w/VXLAN - vendors?

[Ryan Hamel]

I fully agree here too. That's why I proposed a "smarter" CPE to replace the 
standard appliances deployed on site, where the only thing changing is the 
configuration on the device itself, not product being handed off.

Ryan Hamel

From: NANOG  on behalf of Mark Tinka 

Sent: Wednesday, June 14, 2023 10:31 PM
To: nanog@nanog.org 
Subject: Re: 10G CPE w/VXLAN - vendors?

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.



On 6/14/23 21:16, Joe Freeman wrote:


I think you’re probably overthinking this a bit.



Why do you need to extend your vxlan/evpn to the customer premise? There are a 
number of 1G/10G even 100G CPE demarc devices out there that push/pop tags, 
even q-in-q, or 802.1ad. Assuming you have some type of aggregation node you 
bring these back to, tie those tags to the appropriate EVPN instance at the 
aggregation point. Don’t extend anything but a management tag and an S-tag 
essentially to the device at the customer premise.



You can even put that management tagged vlan in it’s own L3 segment, or a 
larger L3 network and impose security. This way you’re not exposing your whole 
service infrastructure to a bad actor that might unplug your cpe device and 
plug into your network directly.

The reason customers ask that their site be part of the customer's Metro-E 
backbone is so that they can enjoy link redundancy without paying for it.

Operators will generally have east and west links coming out of a Metro-E site. 
Customers who single-home into this device only have their last mile as the 
risk. But if the operator drops a Metro-E node into the customer's site, and 
cables it per standard, the customer has the benefit of last mile redundancy, 
because the internal fibre/copper patch to the operator's Metro-E switch does 
not really count as a (risky) last mile.

Sales people like to do this to engender themselves with the customer.

Customers like to do this to get a free meal.

Don't do it, because customer's always assume that that Metro-E node that is in 
their building "belongs to them".

Mark.

Re: 10G CPE w/VXLAN - vendors?

[Ryan Hamel]

> Putting the smart devices on the edge allows for a much-simplified core 
> topology.

>> Putting smart devices in the edge does simplify the network, yes. What 
>> doesn't is making the customer's site part of your edge.

If the customer's site goes offline, that is their problem. A CPE device is 
still a CPE device, no matter how smart it is. Setup IS-IS, BGP to route 
servers, LDP + MPLS if you don't go the VXLAN route, and that's it. I know 
Ciena's can do that on their more expensive 39xx models.

>> We've been running MPLS all the way into the access since 2009 (Cisco 
>> ME3600X/3800X). It is simpler than running an 802.1Q or Q-in-Q Metro-E 
>> backbone, and scales very well. Just leave your customers out of it.

There are a few tier 1's that have delivered Ethernet transport circuits on 
those exact boxes in the field as I speak. It works very well.

I also agree with your stance on Broadcom, it's hard to come up with 
alternatives that are not ADVA/Ciena/Cisco/RAD.

Ryan Hamel

From: NANOG  on behalf of Mark Tinka 

Sent: Wednesday, June 14, 2023 10:30 PM
To: nanog@nanog.org 
Subject: Re: 10G CPE w/VXLAN - vendors?

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.


On 6/14/23 22:04, Ryan Hamel wrote:

> Putting the smart devices on the edge allows for a much-simplified
> core topology.

Putting smart devices in the edge does simplify the network, yes. What
doesn't is making the customer's site part of your edge.

We've been running MPLS all the way into the access since 2009 (Cisco
ME3600X/3800X). It is simpler than running an 802.1Q or Q-in-Q Metro-E
backbone, and scales very well. Just leave your customers out of it.


>
> Either way, I was doing research on FPGA-based hardware a couple of
> weeks agoand came across this which may tick all the boxes.
> https://ethernitynet.com/products/enet-network-appliances/uep-60/ I do
> not know the vendor personally and have not worked on their hardware,
> so your mileage may vary.

There aren't a great deal of options in this space, unfortunately. What
is making it worse is most traditional vendors are relegating devices
designed for this to Broadcom chips, which is a problem because the
closer you get to the customer, the more you need to "touch" their
packets, and Broadcom chips, while fast and cheap, aren't terribly good
at working with packets in the way the customers these devices need to
address would like.

Cisco's ASR920 is still, by far, the best option here. Unfortunately, it
has a very small FIB, does not do 10Gbps at any scale, and certainly
does not 100Gbps. But, because most customers tend to run only p2p
EoMPLS services on it (that doesn't require any large FIB), the box is
still actively sold by Cisco even though in Internet years, it is older
than my grandfather's tobacco pipe.

Juniper are pushing their ACX7024, which we are looking at as a viable
option for replacing the ASR920. However, it's Broadcom... and while
Nokia's Broadcom option for the Metro-E network is using the same chip
as the Juniper one, they seem lazier to be more creative with how they
can touch customer packets vs. Juniper.

Cisco's recommended upgrade path is the NCS540, also a Broadcom box; the
heaviness that is IOS XR in a large scale deployment area like the
Metro-E backbone notwithstanding. The rumour is that Cisco want to
optimize Silicon One for their entire routing & switching range, small
and large. I'll believe it when I see it. Until then, I wouldn't touch
the NCS540.

Vendors are trying to do the least in the Metro-E space, knowing full
well how high the margins are. It's a bit disingenuous, considering they
will be shipping more Metro-E routers to customers than core or edge
routers. But, it is what it is.

Mark.

Re: 10G CPE w/VXLAN - vendors?

[Mark Tinka]

On 6/14/23 22:06, Adam Thompson wrote:

The redundant links to the customer site that traverse independent 
underlay carriers, and in some cases, equal-cost paths that we want to 
load-balance across, are the hard part.  I’m not going to trust STP 
for that, and we aim for <3sec failover where we do have redundant 
paths.  ERPS can handle the failover, but not the load-balancing.  Any 
L2-over-L3 encapsulation protocol can handle the failover + ECMP 
features, but I need to do it at ~10G (~20G if ECMP) wire speed.




We use MPLS for this. We can have as many as 6 paths coming out of a 
single Metro-E node. MPLS will handle it just fine.


Any Layer 2 option won't work the way you want it... they are simply not 
built for that level of redundancy or load balancing.


We have not tried to do this with VXLAN, and don't intend to.

Mark.

Re: 10G CPE w/VXLAN - vendors?

[Mark Tinka]

On 6/15/23 07:22, Marco Paesani wrote:


Huawei NE8000-M1C


I envy folk who aren't mobile operators that are brave enough to run 
Huawei for their IP/MPLS network deliberately, i.e., without influence 
from "management" because they got a good deal :-).


Not for us.

Mark.

Re: 10G CPE w/VXLAN - vendors?

[Mark Tinka]

On 6/14/23 21:16, Joe Freeman wrote:


I think you’re probably overthinking this a bit.

Why do you need to extend your vxlan/evpn to the customer premise? 
There are a number of 1G/10G even 100G CPE demarc devices out there 
that push/pop tags, even q-in-q, or 802.1ad. Assuming you have some 
type of aggregation node you bring these back to, tie those tags to 
the appropriate EVPN instance at the aggregation point. Don’t extend 
anything but a management tag and an S-tag essentially to the device 
at the customer premise.


You can even put that management tagged vlan in it’s own L3 segment, 
or a larger L3 network and impose security. This way you’re not 
exposing your whole service infrastructure to a bad actor that might 
unplug your cpe device and plug into your network directly.




The reason customers ask that their site be part of the customer's 
Metro-E backbone is so that they can enjoy link redundancy without 
paying for it.


Operators will generally have east and west links coming out of a 
Metro-E site. Customers who single-home into this device only have their 
last mile as the risk. But if the operator drops a Metro-E node into the 
customer's site, and cables it per standard, the customer has the 
benefit of last mile redundancy, because the internal fibre/copper patch 
to the operator's Metro-E switch does not really count as a (risky) last 
mile.


Sales people like to do this to engender themselves with the customer.

Customers like to do this to get a free meal.

Don't do it, because customer's always assume that that Metro-E node 
that is in their building "belongs to them".


Mark.

Re: 10G CPE w/VXLAN - vendors?

[Mark Tinka]

On 6/14/23 22:04, Ryan Hamel wrote:

Putting the smart devices on the edge allows for a much-simplified 
core topology.


Putting smart devices in the edge does simplify the network, yes. What 
doesn't is making the customer's site part of your edge.


We've been running MPLS all the way into the access since 2009 (Cisco 
ME3600X/3800X). It is simpler than running an 802.1Q or Q-in-Q Metro-E 
backbone, and scales very well. Just leave your customers out of it.





Either way, I was doing research on FPGA-based hardware a couple of 
weeks agoand came across this which may tick all the boxes. 
https://ethernitynet.com/products/enet-network-appliances/uep-60/ I do 
not know the vendor personally and have not worked on their hardware, 
so your mileage may vary.


There aren't a great deal of options in this space, unfortunately. What 
is making it worse is most traditional vendors are relegating devices 
designed for this to Broadcom chips, which is a problem because the 
closer you get to the customer, the more you need to "touch" their 
packets, and Broadcom chips, while fast and cheap, aren't terribly good 
at working with packets in the way the customers these devices need to 
address would like.


Cisco's ASR920 is still, by far, the best option here. Unfortunately, it 
has a very small FIB, does not do 10Gbps at any scale, and certainly 
does not 100Gbps. But, because most customers tend to run only p2p 
EoMPLS services on it (that doesn't require any large FIB), the box is 
still actively sold by Cisco even though in Internet years, it is older 
than my grandfather's tobacco pipe.


Juniper are pushing their ACX7024, which we are looking at as a viable 
option for replacing the ASR920. However, it's Broadcom... and while 
Nokia's Broadcom option for the Metro-E network is using the same chip 
as the Juniper one, they seem lazier to be more creative with how they 
can touch customer packets vs. Juniper.


Cisco's recommended upgrade path is the NCS540, also a Broadcom box; the 
heaviness that is IOS XR in a large scale deployment area like the 
Metro-E backbone notwithstanding. The rumour is that Cisco want to 
optimize Silicon One for their entire routing & switching range, small 
and large. I'll believe it when I see it. Until then, I wouldn't touch 
the NCS540.


Vendors are trying to do the least in the Metro-E space, knowing full 
well how high the margins are. It's a bit disingenuous, considering they 
will be shipping more Metro-E routers to customers than core or edge 
routers. But, it is what it is.


Mark.

Re: 10G CPE w/VXLAN - vendors?

[Marco Paesani]

Huawei NE8000-M1C

On Thu, Jun 15, 2023, 07:20 Mark Tinka  wrote:

>
>
> On 6/14/23 20:50, Adam Thompson wrote:
>
> Hello, all.
>
> I’m having difficulty finding vendors, never mind products, that fit my
> need.
>
>
>
> We have a small but growing number of L2 (bridged) customers that have
> diverse fiber paths available, and, naturally, want to make use of them.
>
> We have a solution for this: we extend the edge of our EVPN VXLAN fabric
> right to the customer premise.  The customer-prem device needs 4x10G SFP+
> cages (2 redundant paths, plus LAG to customer), and the switches we
> currently use, Arista 7020Rs, are quite expensive if I’m deploying one one
> per customer.  (Nice switches, but overkill here – I don’t need 40/100G,
> and I don’t need 24 SFP+ ports.  And they still take forever to ship.)
>
>
>
> We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode,
> which limits our choices somewhat.  I might be willing to entertain
> spinning up a separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and
> static VTEPs if it saves me a lot of pain.
>
>
>
> However, I’m having trouble finding small & cheap*er* 1U (or even
> desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the
> first place.
>
> Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik
> CRS309-1G-8S+IN / CRS317-1G-16S+RM, which are nice, but our policy requires
> vendor support contracts, so… no-go.)
>
>
>
> Vendors?  Model#s, if you happen to know any?
>
>
> You will have trouble finding such a device at the price you need because
> it is atypical to have your customer's CPE as part of your Metro-E backbone.
>
> Our sales people have asked for this more times than I can remember. We
> have continued to refuse for a reason.
>
> They've angled their query to extend our u-PE devices into the customer
> site, to which they can attach their CPE. We have refused that too, because
> most customer's do not allow 3rd party fibre x-connects into their site
> (for example, some country's embassy, a stock exchange building, a bank,
> e.t.c.), never mind the fact that most customer sites are not fitted with
> 24/7/365 availability and security. And we continue to refuse.
>
> My advice - don't do it. But it sounds like you want to, so...
>
> Mark.
>

Re: 10G CPE w/VXLAN - vendors?

[Mark Tinka]

On 6/14/23 20:50, Adam Thompson wrote:


Hello, all.

I’m having difficulty finding vendors, never mind products, that fit 
my need.


We have a small but growing number of L2 (bridged) customers that have 
diverse fiber paths available, and, naturally, want to make use of them.


We have a solution for this: we extend the edge of our EVPN VXLAN 
fabric right to the customer premise. The customer-prem device needs 
4x10G SFP+ cages (2 redundant paths, plus LAG to customer), and the 
switches we currently use, Arista 7020Rs, are quite expensive if I’m 
deploying one one per customer.  (Nice switches, but overkill here – I 
don’t need 40/100G, and I don’t need 24 SFP+ ports.  And they still 
take forever to ship.)


We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” 
mode, which limits our choices somewhat.  I might be willing to 
entertain spinning up a separate VXLAN mesh using RFC7438 §6.1 
(“vlan-based”) and static VTEPs if it saves me a lot of pain.


However, I’m having trouble finding small & cheap*/er/* 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, 
in the first place.


Who even makes CPE gear with SFP+ ports? (Other than Mikrotik 
CRS309-1G-8S+IN / CRS317-1G-16S+RM, which are nice, but our policy 
requires vendor support contracts, so… no-go.)


Vendors?  Model#s, if you happen to know any?



You will have trouble finding such a device at the price you need 
because it is atypical to have your customer's CPE as part of your 
Metro-E backbone.


Our sales people have asked for this more times than I can remember. We 
have continued to refuse for a reason.


They've angled their query to extend our u-PE devices into the customer 
site, to which they can attach their CPE. We have refused that too, 
because most customer's do not allow 3rd party fibre x-connects into 
their site (for example, some country's embassy, a stock exchange 
building, a bank, e.t.c.), never mind the fact that most customer sites 
are not fitted with 24/7/365 availability and security. And we continue 
to refuse.


My advice - don't do it. But it sounds like you want to, so...

Mark.

Re: 10G CPE w/VXLAN - vendors?

[Yan Filyurin]

There may be a few more places to go searching.  I am not saying you will
find anything, but worth looking into, assuming Mikrotik won't help. :)

Check out what various SD-WAN vendors have to offer.  Now, SD-WAN has about
46 definitions, as many as vendors (surviving vendors that is), but
underneath all of them, it is some sort of box with a CPU, a semi-smart NIC
with a bunch of ports and routing stack that happens to support L2
transport and can overlay it on top of any WAN transport, including regular
IP underlay that can run on these fiber paths. The one of note is Versa.
Besides BGP and overlaying, you may even get a useful multi-layer control
plane out of it, which under the hood of all marketing definitions is all
the things you are familiar with.   And data plane that can actually do
10G.

Check out some of the Broadcom Qumran half-ru switches.  Something like
that:

https://www.etb-tech.com/dell-networking-s4112f-on-switch-12-x-10gb-sfp-3-x-qsfp28-ports-sw00237.html

There are a few other vendors besides Dell and Dell OS does have your basic
P2P VXLAN and EVPN as VXLAN control plane. There are a few others including
open source options. But you are using these small half-ru Broadcom Qumran
and Trident reference designs.

And finally as you go on that search, you can always build your own.  All
you need is $100-200 mini-pc, Linux on it, some form of optimized forwarder
and open source routing stack.

There are people out there who supposedly did that with Raspberry Pis and
used Linksys routers.  Not that you should do it, but shows that there are
options and don't count on 10G!

Yan



On Wed, Jun 14, 2023 at 4:46 PM Arie Vayner  wrote:

> Not sure how much of "CPE" it needs to be, but for example the whole Cisco
> Catalyst 9K product line (including the smaller C9300 switches) support the
> whole EVPN/VXLAN stack).
> A similar set of products exist on the Arista side (e.g. 7xx switches) as
> well as Juniper EX4400 products...
>
> On Wed, Jun 14, 2023, 11:53 Adam Thompson  wrote:
>
>> Hello, all.
>>
>> I’m having difficulty finding vendors, never mind products, that fit my
>> need.
>>
>>
>>
>> We have a small but growing number of L2 (bridged) customers that have
>> diverse fiber paths available, and, naturally, want to make use of them.
>>
>> We have a solution for this: we extend the edge of our EVPN VXLAN fabric
>> right to the customer premise.  The customer-prem device needs 4x10G SFP+
>> cages (2 redundant paths, plus LAG to customer), and the switches we
>> currently use, Arista 7020Rs, are quite expensive if I’m deploying one one
>> per customer.  (Nice switches, but overkill here – I don’t need 40/100G,
>> and I don’t need 24 SFP+ ports.  And they still take forever to ship.)
>>
>>
>>
>> We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode,
>> which limits our choices somewhat.  I might be willing to entertain
>> spinning up a separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and
>> static VTEPs if it saves me a lot of pain.
>>
>>
>>
>> However, I’m having trouble finding small & cheap*er* 1U (or even
>> desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the
>> first place.
>>
>> Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik
>> CRS309-1G-8S+IN / CRS317-1G-16S+RM, which are nice, but our policy requires
>> vendor support contracts, so… no-go.)
>>
>>
>>
>> Vendors?  Model#s, if you happen to know any?
>>
>>
>>
>> Reply here or privately, whatever floats your boat – any pointers
>> appreciated!
>>
>>
>>
>> *Adam Thompson*
>>
>> Consultant, Infrastructure Services
>>
>> [image: [MERLIN logo]]
>>
>> 100 - 135 Innovation Drive
>>
>> Winnipeg, MB R3T 6A8
>>
>> (204) 977-6824 or 1-800-430-6404 (MB only)
>>
>> https://www.merlin.mb.ca
>>
>> Chat with me on Teams
>> 
>>
>>
>>
>

Re: 10G CPE w/VXLAN - vendors?

[Ryan Hamel]

The problem with these switch suggestions is the lack of RFC2544 testing, and 
jitter + latency monitoring required for meeting SLA. That is why I mentioned 
the FPGA solution.

Ryan Hamel


From: NANOG  on behalf of Brandon 
Price 
Sent: Wednesday, June 14, 2023 2:27:02 PM
To: Adam Thompson ; nanog 
Subject: RE: 10G CPE w/VXLAN - vendors?

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

The Juniper EX4100-F-12T is pretty nice. Fanless, 1RU, 4x SFP+, 2x 10G Copper 
which can also be used to power up the switch, and 12x 1G Copper ports. 
EVPN/VXLAN requires an additional license. They don’t break the bank, our use 
case is for a CPE as well.

Brandon

From: NANOG  On Behalf Of 
Adam Thompson
Sent: Wednesday, June 14, 2023 11:51 AM
To: nanog 
Subject: 10G CPE w/VXLAN - vendors?

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you are expecting this email and/or know the 
content is safe.

Hello, all.
I’m having difficulty finding vendors, never mind products, that fit my need.

We have a small but growing number of L2 (bridged) customers that have diverse 
fiber paths available, and, naturally, want to make use of them.
We have a solution for this: we extend the edge of our EVPN VXLAN fabric right 
to the customer premise.  The customer-prem device needs 4x10G SFP+ cages (2 
redundant paths, plus LAG to customer), and the switches we currently use, 
Arista 7020Rs, are quite expensive if I’m deploying one one per customer.  
(Nice switches, but overkill here – I don’t need 40/100G, and I don’t need 24 
SFP+ ports.  And they still take forever to ship.)

We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode, which 
limits our choices somewhat.  I might be willing to entertain spinning up a 
separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and static VTEPs if it 
saves me a lot of pain.

However, I’m having trouble finding small & cheaper 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the 
first place.
Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik CRS309-1G-8S+IN 
/ CRS317-1G-16S+RM, which are nice, but our policy requires vendor support 
contracts, so… no-go.)

Vendors?  Model#s, if you happen to know any?

Reply here or privately, whatever floats your boat – any pointers appreciated!

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99EC2.B891B0A0]Chat with me on 
Teams

RE: 10G CPE w/VXLAN - vendors?

[Brandon Price]

The Juniper EX4100-F-12T is pretty nice. Fanless, 1RU, 4x SFP+, 2x 10G Copper 
which can also be used to power up the switch, and 12x 1G Copper ports. 
EVPN/VXLAN requires an additional license. They don’t break the bank, our use 
case is for a CPE as well.

Brandon

From: NANOG  On Behalf Of 
Adam Thompson
Sent: Wednesday, June 14, 2023 11:51 AM
To: nanog 
Subject: 10G CPE w/VXLAN - vendors?

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you are expecting this email and/or know the 
content is safe.

Hello, all.
I’m having difficulty finding vendors, never mind products, that fit my need.

We have a small but growing number of L2 (bridged) customers that have diverse 
fiber paths available, and, naturally, want to make use of them.
We have a solution for this: we extend the edge of our EVPN VXLAN fabric right 
to the customer premise.  The customer-prem device needs 4x10G SFP+ cages (2 
redundant paths, plus LAG to customer), and the switches we currently use, 
Arista 7020Rs, are quite expensive if I’m deploying one one per customer.  
(Nice switches, but overkill here – I don’t need 40/100G, and I don’t need 24 
SFP+ ports.  And they still take forever to ship.)

We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode, which 
limits our choices somewhat.  I might be willing to entertain spinning up a 
separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and static VTEPs if it 
saves me a lot of pain.

However, I’m having trouble finding small & cheaper 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the 
first place.
Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik CRS309-1G-8S+IN 
/ CRS317-1G-16S+RM, which are nice, but our policy requires vendor support 
contracts, so… no-go.)

Vendors?  Model#s, if you happen to know any?

Reply here or privately, whatever floats your boat – any pointers appreciated!

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99EC2.B891B0A0]Chat with me on 
Teams

Re: 10G CPE w/VXLAN - vendors?

[Arie Vayner]

Not sure how much of "CPE" it needs to be, but for example the whole Cisco
Catalyst 9K product line (including the smaller C9300 switches) support the
whole EVPN/VXLAN stack).
A similar set of products exist on the Arista side (e.g. 7xx switches) as
well as Juniper EX4400 products...

On Wed, Jun 14, 2023, 11:53 Adam Thompson  wrote:

> Hello, all.
>
> I’m having difficulty finding vendors, never mind products, that fit my
> need.
>
>
>
> We have a small but growing number of L2 (bridged) customers that have
> diverse fiber paths available, and, naturally, want to make use of them.
>
> We have a solution for this: we extend the edge of our EVPN VXLAN fabric
> right to the customer premise.  The customer-prem device needs 4x10G SFP+
> cages (2 redundant paths, plus LAG to customer), and the switches we
> currently use, Arista 7020Rs, are quite expensive if I’m deploying one one
> per customer.  (Nice switches, but overkill here – I don’t need 40/100G,
> and I don’t need 24 SFP+ ports.  And they still take forever to ship.)
>
>
>
> We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode,
> which limits our choices somewhat.  I might be willing to entertain
> spinning up a separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and
> static VTEPs if it saves me a lot of pain.
>
>
>
> However, I’m having trouble finding small & cheap*er* 1U (or even
> desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the
> first place.
>
> Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik
> CRS309-1G-8S+IN / CRS317-1G-16S+RM, which are nice, but our policy requires
> vendor support contracts, so… no-go.)
>
>
>
> Vendors?  Model#s, if you happen to know any?
>
>
>
> Reply here or privately, whatever floats your boat – any pointers
> appreciated!
>
>
>
> *Adam Thompson*
>
> Consultant, Infrastructure Services
>
> [image: [MERLIN logo]]
>
> 100 - 135 Innovation Drive
>
> Winnipeg, MB R3T 6A8
>
> (204) 977-6824 or 1-800-430-6404 (MB only)
>
> https://www.merlin.mb.ca
>
> Chat with me on Teams
> 
>
>
>

Re: Your input sought on PeeringDB's Network Type field

[Aaron Wendel]

I just left a couple sections blank.


On 6/14/2023 3:31 PM, Justin Streiner wrote:

Leo:

The survey might also want to include response options along the lines 
of: "Don't know / N/A".


Thank you
jms


On Wed, Jun 14, 2023 at 12:18 PM Leo Vegoda  wrote:

Hi,

PeeringDB's Product Committee wants your input on whether the Network
Type field is useful. Should it go? Should it change?

We have published a very short blog post describing the options and
linking to the survey.

https://docs.peeringdb.com/blog/network_type_your_input_sought/

Your input will influence our decision.

Thanks,

Leo Vegoda for PeeringDB's Product Committee



--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: Your input sought on PeeringDB's Network Type field

[Justin Streiner]

Leo:

The survey might also want to include response options along the lines of:
"Don't know / N/A".

Thank you
jms


On Wed, Jun 14, 2023 at 12:18 PM Leo Vegoda  wrote:

> Hi,
>
> PeeringDB's Product Committee wants your input on whether the Network
> Type field is useful. Should it go? Should it change?
>
> We have published a very short blog post describing the options and
> linking to the survey.
>
> https://docs.peeringdb.com/blog/network_type_your_input_sought/
>
> Your input will influence our decision.
>
> Thanks,
>
> Leo Vegoda for PeeringDB's Product Committee
>

Re: 10G CPE w/VXLAN - vendors?

[Tarko Tikan]

hey,

equal-cost paths that we want to load-balance across, are the hard 
part.  I’m not going to trust STP for that, and we aim for <3sec 
failover where we do have redundant paths.  ERPS can handle the 
failover, but not the load-balancing.


You have EVPN already, perhaps just use active-active multihoming lag 
over those two paths? It'll give you loadbalancing in both directions.


--
tarko

RE: 10G CPE w/VXLAN - vendors?

[Adam Thompson]

The redundant links to the customer site that traverse independent underlay 
carriers, and in some cases, equal-cost paths that we want to load-balance 
across, are the hard part.  I’m not going to trust STP for that, and we aim for 
<3sec failover where we do have redundant paths.  ERPS can handle the failover, 
but not the load-balancing.  Any L2-over-L3 encapsulation protocol can handle 
the failover + ECMP features, but I need to do it at ~10G (~20G if ECMP) wire 
speed.

We provide IaaS services to our customers, which is why we’re stretching VLANs 
to them in the first place.  Viewed from the IaaS perspective, this is a bunch 
of DC-DC connections… but relative to the overall network, the customer-prem 
devices fall into the traditional “CPE” category.  (Most customers either just 
plug in bare fiber, or they connect to an intermediate carrier’s CPE.)

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99ECB.C1CEDE50]Chat with me on 
Teams

From: Joe Freeman 
Sent: Wednesday, June 14, 2023 2:16 PM
To: Adam Thompson ; nanog 
Subject: Re: 10G CPE w/VXLAN - vendors?

I think you’re probably overthinking this a bit.

Why do you need to extend your vxlan/evpn to the customer premise? There are a 
number of 1G/10G even 100G CPE demarc devices out there that push/pop tags, 
even q-in-q, or 802.1ad. Assuming you have some type of aggregation node you 
bring these back to, tie those tags to the appropriate EVPN instance at the 
aggregation point. Don’t extend anything but a management tag and an S-tag 
essentially to the device at the customer premise.

You can even put that management tagged vlan in it’s own L3 segment, or a 
larger L3 network and impose security. This way you’re not exposing your whole 
service infrastructure to a bad actor that might unplug your cpe device and 
plug into your network directly.



From: NANOG 
mailto:nanog-bounces+joe=netbyjoe@nanog.org>>
 on behalf of Adam Thompson 
mailto:athomp...@merlin.mb.ca>>
Date: Wednesday, June 14, 2023 at 2:52 PM
To: nanog mailto:nanog@nanog.org>>
Subject: 10G CPE w/VXLAN - vendors?
Hello, all.
I’m having difficulty finding vendors, never mind products, that fit my need.

We have a small but growing number of L2 (bridged) customers that have diverse 
fiber paths available, and, naturally, want to make use of them.
We have a solution for this: we extend the edge of our EVPN VXLAN fabric right 
to the customer premise.  The customer-prem device needs 4x10G SFP+ cages (2 
redundant paths, plus LAG to customer), and the switches we currently use, 
Arista 7020Rs, are quite expensive if I’m deploying one one per customer.  
(Nice switches, but overkill here – I don’t need 40/100G, and I don’t need 24 
SFP+ ports.  And they still take forever to ship.)

We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode, which 
limits our choices somewhat.  I might be willing to entertain spinning up a 
separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and static VTEPs if it 
saves me a lot of pain.

However, I’m having trouble finding small & cheaper 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the 
first place.
Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik CRS309-1G-8S+IN 
/ CRS317-1G-16S+RM, which are nice, but our policy requires vendor support 
contracts, so… no-go.)

Vendors?  Model#s, if you happen to know any?

Reply here or privately, whatever floats your boat – any pointers appreciated!

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99EC2.B891B0A0]Chat with me on 
Teams

Re: 10G CPE w/VXLAN - vendors?

[Ryan Hamel]

Putting the smart devices on the edge allows for a much-simplified core 
topology.

Either way, I was doing research on FPGA-based hardware a couple of weeks ago 
and came across this which may tick all the boxes. 
https://ethernitynet.com/products/enet-network-appliances/uep-60/ I do not know 
the vendor personally and have not worked on their hardware, so your mileage 
may vary.

Ryan


From: NANOG  on behalf of Joe Freeman 

Sent: Wednesday, June 14, 2023 12:19:26 PM
To: Adam Thompson ; nanog 
Subject: Re: 10G CPE w/VXLAN - vendors?

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

I think you’re probably overthinking this a bit.

Why do you need to extend your vxlan/evpn to the customer premise? There are a 
number of 1G/10G even 100G CPE demarc devices out there that push/pop tags, 
even q-in-q, or 802.1ad. Assuming you have some type of aggregation node you 
bring these back to, tie those tags to the appropriate EVPN instance at the 
aggregation point. Don’t extend anything but a management tag and an S-tag 
essentially to the device at the customer premise.

You can even put that management tagged vlan in it’s own L3 segment, or a 
larger L3 network and impose security. This way you’re not exposing your whole 
service infrastructure to a bad actor that might unplug your cpe device and 
plug into your network directly.



From: NANOG  on behalf of Adam 
Thompson 
Date: Wednesday, June 14, 2023 at 2:52 PM
To: nanog 
Subject: 10G CPE w/VXLAN - vendors?
Hello, all.
I’m having difficulty finding vendors, never mind products, that fit my need.

We have a small but growing number of L2 (bridged) customers that have diverse 
fiber paths available, and, naturally, want to make use of them.
We have a solution for this: we extend the edge of our EVPN VXLAN fabric right 
to the customer premise.  The customer-prem device needs 4x10G SFP+ cages (2 
redundant paths, plus LAG to customer), and the switches we currently use, 
Arista 7020Rs, are quite expensive if I’m deploying one one per customer.  
(Nice switches, but overkill here – I don’t need 40/100G, and I don’t need 24 
SFP+ ports.  And they still take forever to ship.)

We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode, which 
limits our choices somewhat.  I might be willing to entertain spinning up a 
separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and static VTEPs if it 
saves me a lot of pain.

However, I’m having trouble finding small & cheaper 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the 
first place.
Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik CRS309-1G-8S+IN 
/ CRS317-1G-16S+RM, which are nice, but our policy requires vendor support 
contracts, so… no-go.)

Vendors?  Model#s, if you happen to know any?

Reply here or privately, whatever floats your boat – any pointers appreciated!

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99EC2.B891B0A0]Chat with me on 
Teams

Re: 10G CPE w/VXLAN - vendors?

[Joe Freeman]

I think you’re probably overthinking this a bit.

Why do you need to extend your vxlan/evpn to the customer premise? There are a 
number of 1G/10G even 100G CPE demarc devices out there that push/pop tags, 
even q-in-q, or 802.1ad. Assuming you have some type of aggregation node you 
bring these back to, tie those tags to the appropriate EVPN instance at the 
aggregation point. Don’t extend anything but a management tag and an S-tag 
essentially to the device at the customer premise.

You can even put that management tagged vlan in it’s own L3 segment, or a 
larger L3 network and impose security. This way you’re not exposing your whole 
service infrastructure to a bad actor that might unplug your cpe device and 
plug into your network directly.



From: NANOG  on behalf of Adam 
Thompson 
Date: Wednesday, June 14, 2023 at 2:52 PM
To: nanog 
Subject: 10G CPE w/VXLAN - vendors?
Hello, all.
I’m having difficulty finding vendors, never mind products, that fit my need.

We have a small but growing number of L2 (bridged) customers that have diverse 
fiber paths available, and, naturally, want to make use of them.
We have a solution for this: we extend the edge of our EVPN VXLAN fabric right 
to the customer premise.  The customer-prem device needs 4x10G SFP+ cages (2 
redundant paths, plus LAG to customer), and the switches we currently use, 
Arista 7020Rs, are quite expensive if I’m deploying one one per customer.  
(Nice switches, but overkill here – I don’t need 40/100G, and I don’t need 24 
SFP+ ports.  And they still take forever to ship.)

We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode, which 
limits our choices somewhat.  I might be willing to entertain spinning up a 
separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and static VTEPs if it 
saves me a lot of pain.

However, I’m having trouble finding small & cheaper 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the 
first place.
Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik CRS309-1G-8S+IN 
/ CRS317-1G-16S+RM, which are nice, but our policy requires vendor support 
contracts, so… no-go.)

Vendors?  Model#s, if you happen to know any?

Reply here or privately, whatever floats your boat – any pointers appreciated!

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99EC2.B891B0A0]Chat with me on 
Teams

RE: Comcast Business Account Website Broken

[Greg Dickinson]

Have you tried incognito mode?  Sometimes some Extensions will mess with the 
request-header and break some web sites (I'm looking at you, Cisco Licensing).  
FYI I'm able to access the site but don't have a login to test further.

Greg Dickinson, CCNA
Network Engineer
 


-Original Message-
From: NANOG  On Behalf 
Of Matt Hoppes
Sent: Wednesday, June 14, 2023 1:21 PM
To: North American Network Operators' Group 
Subject: Comcast Business Account Website Broken

This Message originates from outside Bryant Bank.   Please use caution when 
opening this correspondence, attachments or hyperlinks (URLs).  If you have 
questions, please contact IT Support.  Thank you.

For the last two weeks we have been unable to pay any bills on the business 
Comcast website.

Clicking on any billing link results in:

400 Bad Request
Request Header Or Cookie Too Large

This is going to the URL of:
https://secure-web.cisco.com/1OR1dkhCBJgJzALRxId_YRkGbls0qVudj2He-TbqJEuGPQCp5H7HGrMHbNAnn7j6HO75WfwxBwNZA7uTz61S1fajaiH69duoalzSw_v-x-2PihXk6NrbaBtP6OwJkuA_Mh8Jh9AtAFo9hyJKvkPIJUQoH8rSkZxm6IsFUlX9_sMFTJFNlqwYGivVV88UBLTDZiZMZrgJ_Bhaw1e2MYl6pYMSAUW7_nkh7MA5BJpQfYcyRDYFTlW9yHYbmBAbGR2t3IjGlIwtI_yIPsUpec0d94CurROXZFokaCDBCKXrdOvRsnznTZMOjkUB0lxoZ47u7/https%3A%2F%2Fbusiness.comcast.com%2Foauth%2Foauth2%2Fauthorize%3Fclient_id%3Dcomcast-business-myaccount-prod%26response_type%3Dcode%26redirect_uri%3Dhttps%253A%252F%252Fbusiness.comcast.com%252Faccount%252Fbilling


Even trying to log out from the customer portal results in:

Sorry

Something went wrong. Please check back later.

This is going to this URL:
https://secure-web.cisco.com/1AgJ4TJJqRM8pVst8n8EDUOS8sls4KEZ5v4bwFmKGYsj2r38_gl4fWo2g1tC_CEgIWK8Er7Zukc27DpTBW5fdJIpdIcA0cvi5y3eNo0mxDJfm4wtVelqtgxVQOiVFBWBBdral4d8Fwqu9xs7MqIU6HJe22S0CoHq6yct6ywyDHoaxA7KhM9sEMp6YZ4bIqjcCQwcjoPH5x3RuiMJYHXuQug_NN-NqAVsQ4ZkfqXWbHBVTGVf1hw1pMap2UPagI2Jl_a0VZRLAS_G9_f0hTFuhGgV4rE62_u0zV_rbEt7OnFH4pt5x0QhSMllg5c0qltwO/https%3A%2F%2Fbusiness.comcast.com%2Faccount%2Flogout

Hoping someone on here can get this to the right people to fix.  I'm sure 
Comcast would love to get payments from their commercial customer base.


NOTICE: This electronic mail message and any files transmitted with it are 
intended exclusively for the individual or entity to which it is addressed. The 
message, together with any attachment, may contain confidential and/or 
privileged information. Any unauthorized review, use, print, save, copy, 
disclosure or distribution is strictly prohibited. If you have received this 
message in error, please immediately advise the sender by reply email and 
delete copies.  Thank you.

10G CPE w/VXLAN - vendors?

[Adam Thompson]

Hello, all.
I’m having difficulty finding vendors, never mind products, that fit my need.

We have a small but growing number of L2 (bridged) customers that have diverse 
fiber paths available, and, naturally, want to make use of them.
We have a solution for this: we extend the edge of our EVPN VXLAN fabric right 
to the customer premise.  The customer-prem device needs 4x10G SFP+ cages (2 
redundant paths, plus LAG to customer), and the switches we currently use, 
Arista 7020Rs, are quite expensive if I’m deploying one one per customer.  
(Nice switches, but overkill here – I don’t need 40/100G, and I don’t need 24 
SFP+ ports.  And they still take forever to ship.)

We use RFC7438 §6.3 “vlan-aware-bundle” mode, not §6.1 “vlan-based” mode, which 
limits our choices somewhat.  I might be willing to entertain spinning up a 
separate VXLAN mesh using RFC7438 §6.1 (“vlan-based”) and static VTEPs if it 
saves me a lot of pain.

However, I’m having trouble finding small & cheaper 1U (or even 
desktop/wallmount) devices that have 4 SFP+ cages, and can do VXLAN, in the 
first place.
Who even makes CPE gear with SFP+ ports?  (Other than Mikrotik CRS309-1G-8S+IN 
/ CRS317-1G-16S+RM, which are nice, but our policy requires vendor support 
contracts, so… no-go.)

Vendors?  Model#s, if you happen to know any?

Reply here or privately, whatever floats your boat – any pointers appreciated!

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN logo]]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
[cid:image002.png@01D99EC2.B891B0A0]Chat with me on 
Teams

Re: Comcast Business Account Website Broken

[Aaron de Bruyn via NANOG]

Someone else here gave me a pointer when I was running into this on the USPS 
site.

Clear your cookies for that site. (In Chrome/Edge, go to the site, open up the 
dev tools, go to the "Application" tab, find cookies, delete them all).

Something probably went a little nuts with the site and ended up creating too 
much data in one or more cookies.

-A

On Wed Jun 14, 2023, 06:20 PM GMT, Matt Hoppes 
 wrote:
> For the last two weeks we have been unable to pay any bills on the
> business Comcast website.
>
> Clicking on any billing link results in:
>
> 400 Bad Request
> Request Header Or Cookie Too Large
>
> This is going to the URL of:
> https://business.comcast.com/oauth/oauth2/authorize?client_id=comcast-business-myaccount-prod_type=code_uri=https%3A%2F%2Fbusiness.comcast.com%2Faccount%2Fbilling
>
>
> Even trying to log out from the customer portal results in:
>
> Sorry
>
> Something went wrong. Please check back later.
>
> This is going to this URL:
> https://business.comcast.com/account/logout
>
> Hoping someone on here can get this to the right people to fix. I'm
> sure Comcast would love to get payments from their commercial customer base.

Comcast Business Account Website Broken

[Matt Hoppes]

For the last two weeks we have been unable to pay any bills on the 
business Comcast website.


Clicking on any billing link results in:

400 Bad Request
Request Header Or Cookie Too Large

This is going to the URL of:
https://business.comcast.com/oauth/oauth2/authorize?client_id=comcast-business-myaccount-prod_type=code_uri=https%3A%2F%2Fbusiness.comcast.com%2Faccount%2Fbilling


Even trying to log out from the customer portal results in:

Sorry

Something went wrong. Please check back later.

This is going to this URL:
https://business.comcast.com/account/logout

Hoping someone on here can get this to the right people to fix.  I'm 
sure Comcast would love to get payments from their commercial customer base.

Re: Your input sought on PeeringDB's Network Type field

[Rubens Kuhl]

Clarify: sure.
Remove: don't remove. Please. Pretty please.


Rubens

On Wed, Jun 14, 2023 at 12:18 PM Leo Vegoda  wrote:
>
> Hi,
>
> PeeringDB's Product Committee wants your input on whether the Network
> Type field is useful. Should it go? Should it change?
>
> We have published a very short blog post describing the options and
> linking to the survey.
>
> https://docs.peeringdb.com/blog/network_type_your_input_sought/
>
> Your input will influence our decision.
>
> Thanks,
>
> Leo Vegoda for PeeringDB's Product Committee

Re: afrinic rpki issue

[Alex Band]

Hi Carlos,

Happy to hear everything is working fine with the latest version of Routinator.

At lot of work has been put into making fetching and validating RPKI data more 
robust since the (over two year old) version of Routinator that you were 
running. 

I want to make an important point for the entire NANOG community:

As developers and operators, we’re still learning a lot about RPKI as it grows 
and evolves in the real world. Maintainers of relying party software [1] are 
actively adapting and improving their software every day to accommodate this. 

This is security software. Please keep it updated.

Cheers,

Alex

[1] https://rpki.readthedocs.io/en/latest/ops/tools.html#relying-party-software

> On 14 Jun 2023, at 18:43, Carlos Friaças  wrote:
> 
> 
> Greetings,
> 
> My issue seems to be solved.
> 
> It seems the Afrinic glitch is incompatible with the version of routinator i 
> was using. So i updated to the last version (0.12.1), and now i can get 
> Afrinic's ROAs again :-)
> 
> Thanks Alex and Cedrick!
> 
> Best Regards,
> Carlos

Re: afrinic rpki issue

[Carlos Friaças via NANOG]

Greetings,

My issue seems to be solved.

It seems the Afrinic glitch is incompatible with the version of routinator 
i was using. So i updated to the last version (0.12.1), and now i can get 
Afrinic's ROAs again :-)


Thanks Alex and Cedrick!

Best Regards,
Carlos

Your input sought on PeeringDB's Network Type field

[Leo Vegoda]

Hi,

PeeringDB's Product Committee wants your input on whether the Network
Type field is useful. Should it go? Should it change?

We have published a very short blog post describing the options and
linking to the survey.

https://docs.peeringdb.com/blog/network_type_your_input_sought/

Your input will influence our decision.

Thanks,

Leo Vegoda for PeeringDB's Product Committee

Re: afrinic rpki issue

[Carlos Friaças via NANOG]

On Wed, 14 Jun 2023, Alex Band wrote:


Hi Carlos,


Hi Alex, All,


Because of the issues that AfriNIC is facing, they are forcing all 
traffic from HTTPS to rsync, so you should check if rsync can properly 
set up outbound connections from your machine. What?s the output you get 
when you rsync rsync://rpki.afrinic.net/repository/ ?



drwxr-xr-x  4,096 2023/06/14 12:04:28 .
-rw-r--r--496 2020/04/08 19:58:03 AfriNIC-simple.tal
-rw-r--r--  1,216 2020/03/30 13:00:32 AfriNIC.cer
drwxr-xr-x  4,096 2023/06/09 13:50:13 
04E8B0D80F4D11E0B657D8931367AE7D

drwxr-xr-x 32,768 2023/06/14 12:04:28 afrinic
drwxr-xr-x  4,096 2023/06/14 01:05:30 apnic
drwxr-xr-x  8,192 2023/06/14 11:42:38 arin
drwxr-xr-x120 2023/06/14 01:15:32 lacnic
drwxr-xr-x 16,384 2023/06/14 12:04:01 member_repository
drwxr-xr-x  4,096 2023/06/14 01:20:30 ripe


Seems to be working...




I do an interactive Routinator validation run with debug logging enabled, like 
so:

$ routinator -vv vrps -f summary

Then I see the following in the logs:

[WARN] RRDP https://rrdp.afrinic.net/notification.xml: Getting notification 
file failed with status 204 No Content
[INFO] RRDP https://rrdp.afrinic.net/notification.xml: Update failed and 
current copy is expired since 2023-05-30 10:43:44 UTC.
[INFO] RRDP https://rrdp.afrinic.net/notification.xml: Falling back to rsync.
[INFO] rsyncing from rsync://rpki.afrinic.net/repository/.




Found valid trust anchor https://rpki.afrinic.net/repository/AfriNIC.cer. 
Processing.

RRDP https://rrdp.afrinic.net/notification.xml: Updating server
RRDP https://rrdp.afrinic.net/notification.xml: malformed XML
rsync://rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/62gPOPXWxxu0sQa4vQZYUBLaMbY.mft: 
failed to validate
CA for 
rsync://rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/ 
rejected, resources marked as unsafe:

   0.0.0.0/0
   ::/0
   AS0-AS4294967295






Then, rsyncing the contents works just fine; objects are fetched and validated. Some 
objects fail validation with "certificate is not yet valid.?, "certificate has 
been revoked.? and ?Object not found.? but that appears unrelated to the connectivity 
issues they?re facing.

I end up with the following totals:

Summary at 2023-06-14 13:43:24.366013 UTC
afrinic:  ROAs:5756 verified;
   VRPs:7121 verified,6820 final;
   router certs:   0 verified;
router keys:   0 verified,   0 final.
  ASPAs:   0 verified,   0 final.


Where do you see this?
Command output?


Summary at 2023-06-14 14:11:34.413948850 UTC
ripe: 39230 verified ROAs, 212122 verified VRPs, 6 unsafe VRPs, 212117 
final VRPs.
apnic: 24878 verified ROAs, 111967 verified VRPs, 0 unsafe VRPs, 111699 
final VRPs.
arin: 64077 verified ROAs, 79176 verified VRPs, 0 unsafe VRPs, 78064 final 
VRPs.
lacnic: 17966 verified ROAs, 32624 verified VRPs, 5 unsafe VRPs, 31033 
final VRPs.

afrinic: 0 verified ROAs, 0 verified VRPs, 0 unsafe VRPs, 0 final VRPs.
total: 146151 verified ROAs, 435889 verified VRPs, 11 unsafe VRPs, 432913 
final VRPs.





If you want some logs to compare, you can have a look here:
https://routinator.do.nlnetlabs.nl/log


Thanks.



It all still works without any extra configuration in Routinator.


Well, for me it's still not really working yet :-)

Thanks anyway.


Cheers,
Carlos



Cheers,

Alex




On 14 Jun 2023, at 15:15, Carlos Friaças via NANOG  wrote:


Hi All,

Did this issue resurface some days ago...?
I had nearly 6000 ROAs on June 1st.
That went to ZERO on June 2nd.

I'm using routinator. Should i have changed something in my config to 
accomodate for some change?

Best Regards,
Carlos



On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:


Hi Job,
Thank you for this good analysis and for sharing your findings.
The issue has since been fixed and the team will publish a post-mortem 
accordingly once we are done with making sure the issue will not
reappear.
Your recommendation is well noted and I cc my colleague so that they can take 
that into consideration in our improvement roadmap.
Best regards,
==
Cedrick Adrien MBEYET
Ebene Cybercity, Mauritius
+230 5851 7674
+++ Never give up, Keep moving forward +++
On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG  wrote:
 Hi all,

 It appears PacketVis correctly identified an issue.

 AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
 'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
 'K1eJenypZMPIt_e92qek2jSpj4A.mft'.

 The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
 Authorities. This Manifest represents the demarcation point between
 "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
 members". In other words; this is an important top-level Manifest in the
 critical path towards the ROAs of the Afrinic members.

Re: afrinic rpki issue

[Alex Band]

Hi Carlos,

Because of the issues that AfriNIC is facing, they are forcing all traffic from 
HTTPS to rsync, so you should check if rsync can properly set up outbound 
connections from your machine. What’s the output you get when you rsync 
rsync://rpki.afrinic.net/repository/ ?

I do an interactive Routinator validation run with debug logging enabled, like 
so:

$ routinator -vv vrps -f summary

Then I see the following in the logs:

[WARN] RRDP https://rrdp.afrinic.net/notification.xml: Getting notification 
file failed with status 204 No Content
[INFO] RRDP https://rrdp.afrinic.net/notification.xml: Update failed and 
current copy is expired since 2023-05-30 10:43:44 UTC.
[INFO] RRDP https://rrdp.afrinic.net/notification.xml: Falling back to rsync.
[INFO] rsyncing from rsync://rpki.afrinic.net/repository/.

Then, rsyncing the contents works just fine; objects are fetched and validated. 
Some objects fail validation with "certificate is not yet valid.”, "certificate 
has been revoked.” and “Object not found.” but that appears unrelated to the 
connectivity issues they’re facing. 

I end up with the following totals:

Summary at 2023-06-14 13:43:24.366013 UTC
afrinic:  ROAs:5756 verified;
VRPs:7121 verified,6820 final;
router certs:   0 verified;
 router keys:   0 verified,   0 final.
   ASPAs:   0 verified,   0 final.

If you want some logs to compare, you can have a look here:
https://routinator.do.nlnetlabs.nl/log

It all still works without any extra configuration in Routinator. 

Cheers,

Alex



> On 14 Jun 2023, at 15:15, Carlos Friaças via NANOG  wrote:
> 
> 
> Hi All,
> 
> Did this issue resurface some days ago...?
> I had nearly 6000 ROAs on June 1st.
> That went to ZERO on June 2nd.
> 
> I'm using routinator. Should i have changed something in my config to 
> accomodate for some change?
> 
> Best Regards,
> Carlos
> 
> 
> 
> On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:
> 
>> Hi Job,
>> Thank you for this good analysis and for sharing your findings.
>> The issue has since been fixed and the team will publish a post-mortem 
>> accordingly once we are done with making sure the issue will not
>> reappear.
>> Your recommendation is well noted and I cc my colleague so that they can 
>> take that into consideration in our improvement roadmap.
>> Best regards,
>> ==
>> Cedrick Adrien MBEYET
>> Ebene Cybercity, Mauritius
>> +230 5851 7674
>> +++ Never give up, Keep moving forward +++
>> On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG  
>> wrote:
>>  Hi all,
>> 
>>  It appears PacketVis correctly identified an issue.
>> 
>>  AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
>>  'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
>>  'K1eJenypZMPIt_e92qek2jSpj4A.mft'.
>> 
>>  The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
>>  Authorities. This Manifest represents the demarcation point between
>>  "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
>>  members". In other words; this is an important top-level Manifest in the
>>  critical path towards the ROAs of the Afrinic members.
>> 
>>  There was a ~ 7 hour gap in the validity window of this Manifest and its
>>  companion CRL (from 20221120T000311Z until 20221120T071514Z). The
>>  serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.
>> 
>>  rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
>>  CRL Serial Number:1E19
>>  CRL valid since:  Nov 18 00:03:11 2022 GMT
>>  CRL valid until:  Nov 20 00:03:11 2022 GMT
>> 
>>  CRL Serial Number:1E1A
>>  CRL valid since:  Nov 20 07:15:12 2022 GMT
>>  CRL valid until:  Nov 22 07:15:12 2022 GMT
>> 
>>  rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
>>  Manifest Number:  12B2
>>  Manifest valid since: Nov 18 00:03:13 2022 GMT
>>  Manifest valid until: Nov 20 00:03:13 2022 GMT
>> 
>>  Manifest Number:  12B3
>>  Manifest valid since: Nov 20 07:15:14 2022 GMT
>>  Manifest valid until: Nov 22 07:15:14 2022 GMT
>> 
>>  (The above can be reconstructed using archives from 
>> http://www.rpkiviews.org)
>> 
>>  The rcynic validator hosted at Afrinic also noticed a gap in objects:
>>  https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html
>> 
>>  A possible recommendation might be to increase the validity window of
>>  these two objects from a sliding 48-hour window to a 1 or 2 week window.
>>  This way any stalling in the issuance process wouldn't case operational
>>  issues on the weekend.
>> 
>>  Kind regards,
>> 
>>  Job
>> 
>>  [1]: SKI EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
>>  

Re: afrinic rpki issue

[Cedrick Adrien Mbeyet]

Hi Carlos,
We currently have a degradation on our RPKI services. We had to disable the
RRDP service request so it can fall back to RSYNC in the meantime that the
team works on ways to optimize the availability of the service. However,
this was prior to 1st of June. We will still investigate just to be on the
safe side though so far everything looks good on our side.
For reference of the mentioned degradation, you can check the below link
https://status.afrinic.net/notices/dkpzrtgqzftlclyg-rrdp-service-degradation
Best regards,

==
Cedrick Adrien MBEYET
Ebene Cybercity, Mauritius
+230 5851 7674

+++ Never give up, Keep moving forward +++


On Wed, Jun 14, 2023 at 5:15 PM Carlos Friaças  wrote:

>
> Hi All,
>
> Did this issue resurface some days ago...?
> I had nearly 6000 ROAs on June 1st.
> That went to ZERO on June 2nd.
>
> I'm using routinator. Should i have changed something in my config to
> accomodate for some change?
>
> Best Regards,
> Carlos
>
>
>
> On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:
>
> > Hi Job,
> >
> > Thank you for this good analysis and for sharing your findings.
> > The issue has since been fixed and the team will publish a post-mortem
> accordingly once we are done with making sure the issue will not
> > reappear.
> > Your recommendation is well noted and I cc my colleague so that they can
> take that into consideration in our improvement roadmap.
> > Best regards,
> >
> > ==
> > Cedrick Adrien MBEYET
> > Ebene Cybercity, Mauritius
> > +230 5851 7674
> >
> > +++ Never give up, Keep moving forward +++
> >
> >
> > On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG 
> wrote:
> >   Hi all,
> >
> >   It appears PacketVis correctly identified an issue.
> >
> >   AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
> >   'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
> >   'K1eJenypZMPIt_e92qek2jSpj4A.mft'.
> >
> >   The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
> >   Authorities. This Manifest represents the demarcation point between
> >   "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf
> of its
> >   members". In other words; this is an important top-level Manifest
> in the
> >   critical path towards the ROAs of the Afrinic members.
> >
> >   There was a ~ 7 hour gap in the validity window of this Manifest
> and its
> >   companion CRL (from 20221120T000311Z until 20221120T071514Z). The
> >   serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.
> >
> >
> rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
> >   CRL Serial Number:1E19
> >   CRL valid since:  Nov 18 00:03:11 2022 GMT
> >   CRL valid until:  Nov 20 00:03:11 2022 GMT
> >
> >   CRL Serial Number:1E1A
> >   CRL valid since:  Nov 20 07:15:12 2022 GMT
> >   CRL valid until:  Nov 22 07:15:12 2022 GMT
> >
> >
> rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
> >   Manifest Number:  12B2
> >   Manifest valid since: Nov 18 00:03:13 2022 GMT
> >   Manifest valid until: Nov 20 00:03:13 2022 GMT
> >
> >   Manifest Number:  12B3
> >   Manifest valid since: Nov 20 07:15:14 2022 GMT
> >   Manifest valid until: Nov 22 07:15:14 2022 GMT
> >
> >   (The above can be reconstructed using archives from
> http://www.rpkiviews.org)
> >
> >   The rcynic validator hosted at Afrinic also noticed a gap in
> objects:
> >
> https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html
> >
> >   A possible recommendation might be to increase the validity window
> of
> >   these two objects from a sliding 48-hour window to a 1 or 2 week
> window.
> >   This way any stalling in the issuance process wouldn't case
> operational
> >   issues on the weekend.
> >
> >   Kind regards,
> >
> >   Job
> >
> >   [1]: SKI
> EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
> >   [2]: SKI
> 2B:57:89:7A:7C:A9:64:C3:C8:B7:F7:BD:DA:A7:A4:DA:34:A9:8F:80
> >
> >
> >
> >   On Sat, Nov 19, 2022 at 08:36:23PM -0800, Randy Bush wrote:
> >   > From: PacketVis 
> >   > Date: Sun, 20 Nov 2022 04:30:44 +
> >   >
> >   > Possible TA malfunction or incomplete VRP file: 73.95% of the
> ROAs disappeared from afrinic
> >   >
> >   > See more details about the event:
> >   >
> https://packetvis.com/#/bgp/event/905ec8b7d37e89a2d7b547bca99fd57e-372b0bf3-9056-407e-9e8d-e986567155fc/4f309cb51ba9314fafa64da53d007e342fac
> >   a613
> >
> >
> >

Re: afrinic rpki issue

[Carlos Friaças via NANOG]

Hi All,

Did this issue resurface some days ago...?
I had nearly 6000 ROAs on June 1st.
That went to ZERO on June 2nd.

I'm using routinator. Should i have changed something in my config to 
accomodate for some change?


Best Regards,
Carlos



On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:


Hi Job,

Thank you for this good analysis and for sharing your findings.
The issue has since been fixed and the team will publish a post-mortem 
accordingly once we are done with making sure the issue will not
reappear.
Your recommendation is well noted and I cc my colleague so that they can take 
that into consideration in our improvement roadmap.
Best regards,

==
Cedrick Adrien MBEYET
Ebene Cybercity, Mauritius
+230 5851 7674

+++ Never give up, Keep moving forward +++


On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG  wrote:
  Hi all,

  It appears PacketVis correctly identified an issue.

  AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
  'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
  'K1eJenypZMPIt_e92qek2jSpj4A.mft'.

  The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
  Authorities. This Manifest represents the demarcation point between
  "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
  members". In other words; this is an important top-level Manifest in the
  critical path towards the ROAs of the Afrinic members.

  There was a ~ 7 hour gap in the validity window of this Manifest and its
  companion CRL (from 20221120T000311Z until 20221120T071514Z). The
  serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.

  rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
      CRL Serial Number:        1E19
      CRL valid since:          Nov 18 00:03:11 2022 GMT
      CRL valid until:          Nov 20 00:03:11 2022 GMT

      CRL Serial Number:        1E1A
      CRL valid since:          Nov 20 07:15:12 2022 GMT
      CRL valid until:          Nov 22 07:15:12 2022 GMT

  rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
      Manifest Number:          12B2
      Manifest valid since:     Nov 18 00:03:13 2022 GMT
      Manifest valid until:     Nov 20 00:03:13 2022 GMT

      Manifest Number:          12B3
      Manifest valid since:     Nov 20 07:15:14 2022 GMT
      Manifest valid until:     Nov 22 07:15:14 2022 GMT

  (The above can be reconstructed using archives from 
http://www.rpkiviews.org)

  The rcynic validator hosted at Afrinic also noticed a gap in objects:
  https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html

  A possible recommendation might be to increase the validity window of
  these two objects from a sliding 48-hour window to a 1 or 2 week window.
  This way any stalling in the issuance process wouldn't case operational
  issues on the weekend.

  Kind regards,

  Job

  [1]: SKI EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
  [2]: SKI 2B:57:89:7A:7C:A9:64:C3:C8:B7:F7:BD:DA:A7:A4:DA:34:A9:8F:80



  On Sat, Nov 19, 2022 at 08:36:23PM -0800, Randy Bush wrote:
  > From: PacketVis 
  > Date: Sun, 20 Nov 2022 04:30:44 +
  >
  > Possible TA malfunction or incomplete VRP file: 73.95% of the ROAs 
disappeared from afrinic
  >
  > See more details about the event:
  
>https://packetvis.com/#/bgp/event/905ec8b7d37e89a2d7b547bca99fd57e-372b0bf3-9056-407e-9e8d-e986567155fc/4f309cb51ba9314fafa64da53d007e342fac
  a613

Re: Software to document fiber networks - in house only

[Denis Fondras]

Le Tue, Jun 13, 2023 at 03:12:29PM -0300, Jean Franco a écrit :
> Hi all,
> 
> I know this must have been on the table before, but I'm looking for a
> in-house solution, something I can host on our own datacenter to document
> fiber networks, maps and so forth.
> 

I use a mix of Qgis, PostgreSQL, PostGIS and the GraceTHD model (which seems to 
be
France specific though). This requires a bit of work at the beginning but works
really great once installed, is self-hosted and does not require too much
attention afterwards. Qfield can be used on mobile for usage by field
technicians.