Re: CloudFlare issues?

[Fredrik Korsbäck]

On 2019-06-24 20:16, Mark Tinka wrote:
> 
> 
> On 24/Jun/19 16:11, Job Snijders wrote:
> 
>>
>> - deploy RPKI based BGP Origin validation (with invalid == reject)
>> - apply maximum prefix limits on all EBGP sessions
>> - ask your router vendor to comply with RFC 8212 ('default deny')
>> - turn off your 'BGP optimizers'
> 
> I cannot over-emphasize the above, especially the BGP optimizers.
> 
> Mark.
> 

+1

https://honestnetworker.net/2019/06/24/leaking-your-optimized-routes-to-stub-networks-that-then-leak-it-to-a-tier1-transit-that-doesnt-filter/



-- 
hugge

Re: Cheap switch with a couple 100G

[Fredrik Korsbäck]

On 2018-11-25 21:16, Mike Hammett wrote:
> No, not new. No need to buy new switches when there are so many used 
> available (except for now needing 100G). Switches
> have an extremely long life. I have a client that has 15 year old Foundry 
> switches that just work, though we're looking
> to replace them to get some 10G ports.
> 
> The pricing was part of my point. For years everyone says just skip 40G and 
> go to 100G. The price difference isn't that
> much  but it is.
> 
> "Everyone just skipped 40G and went for 100G." Then why is there such an 
> availability of 40G switches? Obviously they
> weren't skipped, but purchased and then later replaced.
> 
> Looks like $280 for an LR4 40G and $800 for an LR4 100G. Still a premium for 
> 100G over 40G.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com

Whole bunch of Apples and Oranges mixed here.

* 40G in the ISP/Telco world is more or less non-existant. There was never a 
good uptake on any products aimed towards
ISP/Telco that did 40G cost-effectively. This is partly because 40G was/is 
never a thing in the DWDM-world either. There
was some STM256 built and some OTU3 built aswell but not nearly in the same 
density as 10G was and 100G is built today,
it had a weird timing with being fairly close to 100G and abit to distant to 
cost-effective 10G so it never had any good
usecase. So if you buy services from a Telco, its very likely that 40G handoff 
is the least preffered option for them.
Or most likely not a option at all (like from every company ive been involved 
with)

* You compare market-economics with what new stuff cost vs used stuff on 
ebay/liquidators cost. Thats not a sane way of
doing math, the reason why old 10/40 nexus and arista etc is plentiful on ebay 
is because people switched it out to
something newer, faster. There isn't currently anything faster on the market 
then 100G-switches so naturally there isn't
much of that available used. Now that at least some product-families (spine 
switches) getting 400G with the new
TH3-switches we can assume that there is a decent amount of older 100G spines 
getting decommisioned, and hence getting
available in the second hand market (maybe even late next year). However there 
has been _a lot_ of datacenters being
lifted from 10G/40G to 25/50/100G architecture the last year, so it makes sense 
that the predecessor is on ebay.

If you are to buy *new* stuff id say that going for 100G of 40G architecture 
makes sense in almost every aspect,
regardless of what products and usecase you have. If you rely on second-hand, 
then sure, 40G might be a decent choice.
However i would cry if id have to buy 40G optics today since i know they be 
binned in a year anyway. (4x10G PIR is still
usable optics in 100G land though).


-- 
hugge

Re: AS3266: BitCanal hijack factory, courtesy of many connectivity providers

[Fredrik Korsbäck]

On 2018-07-09 17:24, Fredrik Korsbäck wrote:
> On 2018-07-06 21:18, Tom Paseka via NANOG wrote:
>> Hi,
>>
>> I've been casually observing the connectivity to Bitcanal / AS3266 /
>> AS197426 since the thread started.
>>
>> After GTT shared that bitcanal had been disconnected, bitcanal was only
>> visible behind Cogent. But the Cogent path now also seems to have been
>> disconnected. After Cogent they popped up behind BICS (but just for a few
>> days), that circuit seems to have been disconnected too.
>>
>> On the IX front: I noticed that Bitcanal's IP addresses on LINX (since
>> yesterday) and FranceIX (since today) are no longer responding.
>>
>> It is good to see that discussing BGP hijacking abuse complaints actually
>> results in clean up activities. I hope the remaining IX's they're still
>> connected to can act too.
>>
>> Thanks!
>> -Tom
>>
> 
> And it also seems that they are now no longer reachable over the AMS-IX 
> fabric (and is no longer listed as a member).
> 
> I also noticed that hey are not reachable over the Megaport/ECIX fabric in 
> Frankfurt either (no arp or ping-reply) but
> is listed as member on the megaport website, so not sure whats going on there.
> 
> The only routes i can see now for 3266/197426 is two /24 v4 and one /29 v6 
> that jumps on over to portugal through 1299
> (telia) -> 174 (cogent) -> 29003 (refertelecom / iptelecom).
> 
> 

And now it also seems that NANOG-contributor Doug over at Dyn has done a 
complete wrap-up of the thing and he has
hilighted all the important aspects of this incident in a very educative manner.

https://dyn.com/blog/shutting-down-the-bgp-hijack-factory/

Thanks for this Doug!

I will bring this post up with my NOC and L2-teams since i think these type of 
incidents will become as common as
regular spam in the future...

-- 
hugge

Re: AS3266: BitCanal hijack factory, courtesy of many connectivity providers

[Fredrik Korsbäck]

On 2018-07-06 21:18, Tom Paseka via NANOG wrote:
> Hi,
> 
> I've been casually observing the connectivity to Bitcanal / AS3266 /
> AS197426 since the thread started.
> 
> After GTT shared that bitcanal had been disconnected, bitcanal was only
> visible behind Cogent. But the Cogent path now also seems to have been
> disconnected. After Cogent they popped up behind BICS (but just for a few
> days), that circuit seems to have been disconnected too.
> 
> On the IX front: I noticed that Bitcanal's IP addresses on LINX (since
> yesterday) and FranceIX (since today) are no longer responding.
> 
> It is good to see that discussing BGP hijacking abuse complaints actually
> results in clean up activities. I hope the remaining IX's they're still
> connected to can act too.
> 
> Thanks!
> -Tom
> 

And it also seems that they are now no longer reachable over the AMS-IX fabric 
(and is no longer listed as a member).

I also noticed that hey are not reachable over the Megaport/ECIX fabric in 
Frankfurt either (no arp or ping-reply) but
is listed as member on the megaport website, so not sure whats going on there.

The only routes i can see now for 3266/197426 is two /24 v4 and one /29 v6 that 
jumps on over to portugal through 1299
(telia) -> 174 (cogent) -> 29003 (refertelecom / iptelecom).


-- 
hugge

Re: The story about MyEtherWallet.com hijack or how to become a millionare in 2 hours.

[Fredrik Korsbäck]

Well there is quite abit of data around that particular server.

So it definitely happened.

https://twitter.com/GossiTheDog/status/988873775285460992

This tweet is a good start.

The server answer to me right now and google safe browsing has flagged it as 
well for being insecure (no the regular
cert-fail warning but deceptivness warning)

The SSL-cert is a self-signed one impersonating MyEtherWallet.com.

Id take it that 15169 accepted the prefix for some reason over a bilateral 
peering-sesssion (to the best of my knowledge
the equinix routeservers does indeed do filter, but please correct me on this 
one) with 10297 and hence poisoned the
8.8.8.8 resolver for some time with the wrong ip-addr.

> On Tue, Apr 24, 2018 at 08:35:17PM +0200,
>  Fredrik Korsbäck  wrote
>   a message of 28 lines which said:
> 
>> Surprised this hasnt "made the news" over at this list yet.
> 
> It was discussed several hours before on the Outages mailing list.
> 
> Also, there are not a lot of hard facts. The BGP hijacking is clear
> and easy to find in the usual places.
> 
> The supposed rogue DNS server is much more elusive. Nobody apparently
> thought of querying it with dig during the hijack. There are reports
> of people being directed to a rogue www.myetherwallet.com but, again,
> no detail, no IP address, not the certificate of the rogue server,
> nothing.
> 
>> seems to be some kind of transparent proxy out of russia with a
>> bogus SSL-cert (but still pretty good) (https://46.161.42.42/)
> 
> DNSDB does not confirm this:
> 
> %  isc-dnsdb-query rdata ip 46.161.42.42
> pigroot.sciencesupply.eu. IN A 46.161.42.42
> value.rollliquid.com. IN A 46.161.42.42
> campsprings.collaspepaw.com. IN A 46.161.42.42
> bronchopneumonic.collaspepaw.com. IN A 46.161.42.42
> server42.woodorganism.com. IN A 46.161.42.42
> ;;; Returned 5 RRs in 0.03 seconds.
> ;;; DNSDB
> 
> Currently, this machine does not accept connections.
> 
> 
> 
> 


-- 
hugge

Re: The story about MyEtherWallet.com hijack or how to become a millionare in 2 hours.

[Fredrik Korsbäck]

"that depends".

we for sure know that 150K or so got immediately snatched of the bat, but how 
much more wallets is at stake? no one knows.

What is known however is that they are trying to deploy smokescreens with tons 
of transfers moving ETH around wallets
and all seems to be ending up sooner or later in this account.

https://etherscan.io/address/0xb3e47070264f3595c5032ee94b620a583a39

Which is good for 17MUSD.

That doesn't really matter though - i wanna speak what we do about this in the 
DFZ.

Can someone from HE comment on how your ingress route-filtering policy looks 
like towards your customers? I typically
base my peering-relationships on people/operators that i have some kind of 
level of trust in.



> Is MyEtherWallet really doing 500k/hr in business though?
> 
>> On Apr 24, 2018, at 2:35 PM, Fredrik Korsbäck  wrote:
>>
>> Aloha.
>>
>> Surprised this hasnt "made the news" over at this list yet.
>>
>> https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f
>>
>> https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/2teeVLJ44RM/Yqk5GHSpCQAJ
>>
>> https://twitter.com/barton_paul/status/988788348272734217
>>
>> TLDR; So it seems that AS10297 (some small hostingprovider in the US) 
>> suddenly started to announce de-aggregated AWS
>> IP-space, containing quite alot of Route53 infrastructure, put up resolvers 
>> on their own on the hijacked IP-space and
>> pointed *ATLEAST* www.myetherwallet.com to a ip-address that seems to be 
>> some kind of transparent proxy out of russia
>> with a bogus SSL-cert (but still pretty good) (https://46.161.42.42/)
>>
>> I did digging in my own logs and played it through BGP-play - seems like it 
>> was in fact only Hurricane Electric (6939)
>> that actually propagated this prefix to the Internet. Which makes sense 
>> since we have seen them being part of the
>> problem in almost all recent hijacks.
>>
>> Can we do some collaborative digging in other tools you have handy (i guess 
>> thousand eyes probes etc could be of help
>> here) to track how big the propagation was?
>>
>> Being abit involved in the Ethereum world it could be noted that the login 
>> to MyEtherWallet.com is abit special since
>> you actually login with you wallet-seed and not user/pass to the site... 
>> giving the possibility to make really swift
>> transfers without having actual access to the real site (for good and 
>> bad).
>>
>> -- 
>> hugge @ 2603
>>
> 


-- 
hugge

The story about MyEtherWallet.com hijack or how to become a millionare in 2 hours.

[Fredrik Korsbäck]

Aloha.

Surprised this hasnt "made the news" over at this list yet.

https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f

https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/2teeVLJ44RM/Yqk5GHSpCQAJ

https://twitter.com/barton_paul/status/988788348272734217

TLDR; So it seems that AS10297 (some small hostingprovider in the US) suddenly 
started to announce de-aggregated AWS
IP-space, containing quite alot of Route53 infrastructure, put up resolvers on 
their own on the hijacked IP-space and
pointed *ATLEAST* www.myetherwallet.com to a ip-address that seems to be some 
kind of transparent proxy out of russia
with a bogus SSL-cert (but still pretty good) (https://46.161.42.42/)

I did digging in my own logs and played it through BGP-play - seems like it was 
in fact only Hurricane Electric (6939)
that actually propagated this prefix to the Internet. Which makes sense since 
we have seen them being part of the
problem in almost all recent hijacks.

Can we do some collaborative digging in other tools you have handy (i guess 
thousand eyes probes etc could be of help
here) to track how big the propagation was?

Being abit involved in the Ethereum world it could be noted that the login to 
MyEtherWallet.com is abit special since
you actually login with you wallet-seed and not user/pass to the site... giving 
the possibility to make really swift
transfers without having actual access to the real site (for good and bad).

-- 
hugge @ 2603

Re: Spiffy Netflow tools?

[Fredrik Korsbäck]

On 2018-03-13 00:24, mike.l...@gmail.com wrote:
> Howdy!
> 
> Checking out various Netflow tools and wanted to see what others are using? 
> 
> Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to 
> see any others.
> 
> Also curious about on-prem solutions as well.
> 
> Thanks!
> Mike
> 

Kentik is probably top of the foodchain right now.

But they are certainly not alone in the biz. Ontop of my head...

* Flowmon
* Talaia
* Arbor Peakflow
* Deepfield
* Pmacct + supporting toolkit
* NFsen/Nfdump/AS-stats
* Put kibana/ES infront of any collector
* Solarwinds something something
* Different vendor toolkits



-- 
hugge

Re: Contact info for AS1880 - STUPI.SE (Svensk Teleutveckling & Produktinnovation)

[Fredrik Korsbäck]

On 2018-03-05 04:20, Brian Kantor wrote:
> Does anyone have contact info for the peering folks at
> AS1880, Svensk Teleutveckling & Produktinnovation in Sweden?
> 
> They appear to be advertising a subnet of our network
> space without permission.  Their WHOIS entry at RIPE does
> not list any contact email addresses.
> 
> Any information would be appreciated.  Off-list is fine.
> 
> Thank you.
>   - Brian
> 

Thats Peter Lothbergs lab-asn.

I turned on the bat-signal and put out the breadcrumbs - expect an answer 
shortly.

I have limited commit in the network but is also one of the upstreams - feel 
free to send me the prefix offlist and ill
make sure that atleast we dont transit the network.

-- 
hugge @ 2603

Re: Blockchain and Networking

[Fredrik Korsbäck]

On 2018-01-13 03:26, Christopher Morrow wrote:

On Fri, Jan 12, 2018 at 5:20 PM,  wrote:


On Thu, 11 Jan 2018 15:28:19 -0500, William Herrin said:

On Thu, Jan 11, 2018 at 2:46 PM, Dale W. Carder  wrote:


Traceroute or any other path diagnostics comes to mind.



That's not obvious to me. Assuming the time-exceeded message was modified
to include the necessary data, how would blockchain authenticate the
responding router?


And do you really want to do *all* that on every single 'TTL Exceeded'
ICMP?  Sounds like
a *really* easy way to DDoS a router



pish-posh! the asics will do it.



Apparently we are not keeping up with the cool kids here.

https://www.openct.io/

* Open in the name
* .IO TLD
* A scrolling website

It all checks out as a legitimate web3.0 biz

###

* Blockchain as a Transport (BaaT): BaaT leverages blockchain to create an architecture that can connect geographically 
dispersed Layer 2 islands over any available infrastructure, including the public Internet. As the resulting solution 
securely supports all kinds of traffic: Unicast, Multicast and Broadcast - BaaT will become the transport service of 
choice for all businesses.


* Blockchain-Defined Wide Area Networks (BD-WAN): BD-WAN integrates blockchain with Software-Defined WAN (SD-WAN) for a 
secure, scalable virtualization of WAN transport technologies. Among the unique features (not available with most SD-WAN 
architectures):


The ability to dynamically establish and tear down logical and physical circuits so customers pay only for what they 
consume.


* Inter-Domain MPLS Traffic Engineering via blockchain (near zero time delay).

Trusted per-usage billings that are verified and hard-coded over the blockchain.

Full visibility and control over all transport facilities either via Fiber to the Premises (FTTP) or via partnership 
with key telco operators and metro Ethernet providers worldwide.


Bringing public cloud and content services seamlessly to the customers' 
doorstep as part of the standard offering.


###


I wanna buy all of these stuff right now!


--
hugge

Re: Attacks from poneytelecom.eu

[Fredrik Korsbäck]

Depends on what "legitimate" means.

We have a decent amount of traffic to the network (like 2Gbps sustained in any afternoon). Its typically a mix of 
bittorrent, tor-relay traffic, ftp-transfers and of course the expected scanners, malware-hosts, ddos-bots and such.


For me Poney/Illiad/Online.net/Scaleway has always been a bulletproof hoster (or bulletproof transit even), the response 
to abuse has always been NIL. I know tons of my customers just blocks out their whole ip-ranges in their SIP-servers and 
email-machines to lessen the white-noise.


However - judging from the Online.net website it atleast seems that they are trying to up their game and look like 
something that would be attractive to a legitimate business to consider. On the other hand, looking at 
http://as12876.net/  it looks more like something that would rather fit as a place where i put the shady stuff, so not 
sure where on the map they fall these days.





AS12876 is online.net... home of the €2.99 physical server, perfect for all of 
your favorite illegitimate activity. I’m curious how much traffic originates 
from that ASN that is actually legitimate... probably close to none.

Sent from my iPhone


On Jan 3, 2018, at 1:35 AM, Troy Mursch  wrote:

Dovid,

Back in September, I documented my poor experience with AS12876 here:
https://badpackets.net/ongoing-large-scale-sip-attack-
campaign-coming-from-online-sas-as12876/
Since then, their handling of abuse notifications (or lack thereof) has
largely remained the same. The volume of malicious traffic from their
network hasn't decreased either.

As you noted, others have reported similar issues with AS12876, including
my associate Dr. Neal Krawetz: https://twitter.com/h
ackerfactor/status/932593355648667649. I've also compiled a list of
complaints regarding AS12876 in this thread: https://twitter.com/ba
d_packets/status/937220987371732992


Thanks,
__

*Troy Mursch*

@bad_packets 


On Tue, Jan 2, 2018 at 6:51 PM, Dovid Bender  wrote:

Hi All,

Lately we have seen a lot of attacks from IPs where the PTR record ends in
poneytelecom.eu to PBX systems. A quick search on twitter (
https://twitter.com/hashtag/poneytelecom) shows multiple people
complaining
that they reported the IP's yet nothing happens. Has anyone had the
pleasure of dealing with them and have you gotten anywhere? I wonder if the
only option is public shaming.

I would rather not ban their AS as it may hurt legit traffic but I am out
of ideas at this point

TIA.

Dovid




--
hugge

Re: 40G and 100G optics options

[Fredrik Korsbäck]

> 19 dec. 2017 kl. 19:24 skrev Sabri Berisha :
> 
> - On Dec 18, 2017, at 9:49 AM, Fredrik Korsbäck hu...@nordu.net wrote:
> 
>> This is the "failure" of us (the business) choosing QSFP as the de-factor
>> formfactor for 100G, there is not power in
>> that cage to make 10km+ optics in an easy way. If we would have pushed for 
>> CFP4
>> as the "last" formfactor in 100G land we
>> would be much better off.
> 
> How about OSFP? The OSFP MSA has a large number of backers, including 
> Juniper, Arista, Finisar and Google. 

Yes, on OSFP we have the possbility of making this right again for 400G. It 
will not have the same backward compability as QSFP-DD and not the same 
faceplate density (but close enough i would say). But in return we would most 
likely see longrange optics MUCH earlier in the lifecycle of 400G


> 
> It's the vendors that chose to go for QSFP due to the density options in a 
> single RU chassis.
> 
> Thanks,
> 
> Sabri

Re: 40G and 100G optics options

[Fredrik Korsbäck]

This is the "failure" of us (the business) choosing QSFP as the de-factor formfactor for 100G, there is not power in 
that cage to make 10km+ optics in an easy way. If we would have pushed for CFP4 as the "last" formfactor in 100G land we 
would be much better off.


The options you have to choose from realistically is QSFP28-ER4L

It exist in a fec and non-fec option that does 30 and 40km respectively although a few vendors spec them at 25km cause 
its really hard to validate them in their upper ranges. But this is quite and expensive optic.


On the horizon you have QSFP28-4WDM20 and 4WDM40 which uses FEC. Not sure where 
the MSA stands on that.

If you need more then 30-40km reliable stuff you need to either to PAM4 (The "Inphi ColorZ" optic) which is 80km at 
best. These should not be confused for the regular 10G ZR-optics with a fixed lambda you can buy for $300 from china. 
These are dual-laser optics that runs two 50G PAM4 carriers with very low launchpower. They need pre and post-amps 
aswell as tuned DCM and a MUX to fully work. Also they are fixed wave so you need 44 different optics to populate a full 
MUX. These optics are great if you need tons of 100Gs on very short metro-spans.


It always almost ends up with coherent being the better option in the end anyway if you wanna go beyond 20-25km. There 
is like 30 different competing products on the market now that acts as a stupid little transponder with a QSFP28 in one 
end (where you plug your DAC in) and a CFP2-ACO (or DCO) in the other end that let you run a couple of hundred 
kilometers (or thousands) with a proper coherent DWDM signal.


Now that CFP2-DCO is GA finally i hope to see more linecards and switches that 
has CFP2 slots in them.



Hi

What options are available for 40G QSFP+ and 100G QSFP28 for 10+ km links?

I see a lot of switches offered with QSFP+ and QSFP28. But I do not seem
to find the necessary optics to build the links I want.

For example, take a look at the options available at Fiberstore:

https://www.fs.com/c/generic-40g-qsfp-891

Generic Compatible 40GBASE-LR4 and OTU3 QSFP+ 1310nm 10km LC Transceiver
for SMF
$340

Generic Compatible 40GBASE-ER4 and OTU3 QSFP+ 1310nm 40km LC Transceiver
for SMF
$1500

https://www.fs.com/c/generic-qsfp28-100g-transceivers-2858

Generic Compatible QSFP28 100GBASE-LR4 1310nm 10km Transceiver
$1999

Generic Compatible QSFP28 100GBASE-ER4 1310nm 40km Transceiver
$7000

That is it. Four modules and the 40km are prohibitive expensive. The
situation at other vendors appears to be similar.

I need stronger modules that can do more than 10 km without being
extremely expensive. Or DWDM modules in the 1550 nm band so I can use
external amplifiers. Am I looking in the wrong place? Is this expected
to be available in the near future?

Regards,

Baldur




--
hugge



signature.asc
Description: OpenPGP digital signature

Re: Arista Layer3

[Fredrik Korsbäck]

On 2017-11-30 19:36, Romeo Czumbil wrote:

So I've been using Arista as layer2 for quite some time, and I'm pretty happy 
with them.
Kicking the idea around to turn on some Layer3 features but I've been hearing 
some negative feedback.
The people that I did hear negative feedback don't use Arista themselves. (they 
just heard)

So do we have any Arista L3 people out here that can share some negatives or 
positives?

Use case: Just some MPLS IPv4/IPv6 routing, l2vpn OSPF/BGP
Maybe 20k routes (no full internet routes)
7050 Series
7280 Series

-Romeo



I have a whole bunch of 7280SR in production, acting as peering-aggregators to easy be able to scale out PNIs to the 
CDN/Clouds (where you sometimes needs to add 10G of capacity per PoP per month)


They work just fine. Simple PE-functions, a few hundred BGP-peers in each, full tables, as-path filtering (150k lines of 
config), route-maps and sub-route maps. It is certainly not as flexible and easy to work with as for example a 
MX-router. But on the other hand you get 1Tbit worth of ports for the same price as a 16x10G MX-card.


L3VPN, RSVP-TE which could be major things you need is coming to EOS "soon", 
february i think.

The boxes that is coming out here in Q1 (some even out) with Jericho+ and Jericho2 chipsets should be even better with 
even more tables that should suffice for quite some time, the 1mil limit on 7280SR can be borderline especially when you 
mix in L3VPN whenever thats coming.


Huawei (ce6870) and Cisco (ncs5500) is also selling the same boxes and rumours on the streets are that Juniper will also 
release a jericho-based PE-box.


Also Juniper has picked up alot of slack recently with the release of MX204, which seems for whats its worth be a really 
good contender in the "small but modern router" market which has been grossly overlooked by many vendors for quite some 
time.


Not sure where cisco really is with the 9901, which atleast looked really good 
on the CLUS presentations.

--
hugge



signature.asc
Description: OpenPGP digital signature

Re: Commodity routers/switches

[Fredrik Korsbäck]

On 2017-11-19 02:55, mike.l...@gmail.com wrote:

Howdy!

Looking to replace some edge routers for my small ISP. With all the various SDN 
platforms available along with various choices of bare-metal hardware 
platforms, im thinking i may go this route instead of going with 
Cisco/Juniper/Etc.

I only need a handful of 10G uplinks. The SuperMicro SSE-G3648B and the Penguin 
Arctica Network switches appear to fit my needs.

I am eyeing Cumulus Linux to run on these, but that isn’t set in stone.

They’ll likely be getting 2 full tables along with some peers.

Has anyone run SuperMicro or Penguin hardware with Cumulus in this type of 
scenario?

What were your experiences? How is BGP convergence time on x86 hardware these 
days?

Any insight would be appreciated.

Thank You,
Mike



Replacing a edge-router with a switch is nothing new, however make sure you 
actually replace it with the correct one.

The Supermicro looks like any generic Helix4-switch and is a ToR-switch for the datacenter. Its not very fitting for 
edge-routing. It does not have buffers at all and would make your sub-speed connections perform like shit, and also it 
has a tiny LPM table so you wont be able to fit anywhere near a full table in there


It seems that you want a cost-effective 1G solution given that you linked 
SSE-G3648B?

Merchant-switch silicon and edge-routing isn't very competitive on 1G/10G, both because traditional legacy-routers is 
somewhat cheap for 1G applications and also that 1G is virtually non-existant in datacenter enviroments these days so 
its hard to leverage the economy-of-scale from there on these swithces.


Look at Nokias portfolio for 1G/10G routers, they still care in that segment and is in Europe a very popular choice for 
broadband buildouts, as is Huaweis smaller NetEngines but that might not fly that well in the US. Juniper MX150 might 
also work depending on how much 1G you need, but you likely need more.


If you bump it up a notch to 10G/100G or 100G only the market for routing-merchant-silicon looks much better. I guess 
the most famous platform is the Arista 7280R that was the first Broadcom-based box that accepted 1M routes, had big 
buffers and didnt cost the equivalent of a bunch of new cars for a 1Tbit of capacity like J/C/N/H would charge you for a 
equivalent linecard to their edge-portfolios.


Cisco quickly released NCS550 productline as an answer, Huawei released CE6870-line (but didnt do the LEM/LPM hack that 
C/A did for full tables to protect NetEngine BU), Juniper pushes QFX10K which is somewhat equal to a Broadcom 
Jericho-based box. The only Whitebox-vendor i know off that actually has a Jericho (qumran) based box is Agema with the 
AGC7648S, not sure which stand-alone NOS that actually supports this box fully.


Now Jericho+ is also out and Jericho2 is around the corner so i guess we will see alot bigger and even more competetive 
switch-routers based on these chips. But it doesent really help much if you are operating in 1G/10G space.





--
hugge

Re: 4 or smaller digit ASNs

[Fredrik Korsbäck]

On 2017-10-12 07:01, James Breeden wrote:

Hello NANOG...

I have a client interested in picking up a new AS number but they really want 
it to be 3 or 4 digits in length.

Is there a process to request this from ARIN, or doss anyone know of unused 
ASns fitting this that anyone is looking to sell for some quick cash?

Thanks!
James




Sent via the Samsung Galaxy S7 active, an AT&T 4G LTE smartphone



AS251 comes to mind as one of few 3-digit ones that has existed, but doesent 
anymore (atleast not in DFZ).

But do you really want a used and abused old asn? The old logic with the lower 
ASN you have the bigger E-penis you got doesent really apply anymore since the 
biggest players on the Internet doesent have 3 or 4 digit ASNs anymore.

--
hugge

Re: 100G - Whitebox

[Fredrik Korsbäck]

The only viable merchant silicon chip that would be useful for a IXP is from 
the StrataDNX-family which house the jericho/qumran/petra/arad chips from 
broadcom. No packetbuffer in the exhangepoint will shred performance 
significantly, especially when one of your bursty 100G customers starts sending 
data into 1/10G customers. 

To the best of my knowledge the only one that offers DNX in whitebox-fashion is 
Agema and Edgecore. But why whitebox? Except on a very few occasions whitebox 
is just "i like paying hardware and software on different invoices = whitebox" 
the TCO is just the same but. As an exchangepoint i also see that it can hard 
to reap the benefits of all the hipstershit going on in these NOS-startups, you 
want spanning-tree, port-security, something to loadbalance over links and 
perhaps a overlaying-technology if the IXP becomes to big and distributed, like 
vxlan. This is to easy almost. 

Whenever i see unbuffered mix-speed IXPs i ask if i can pay 25% of the portcost 
since that is actually how much oumpfff i would get through the port.

// hugge @ 2603

> 20 aug. 2017 kl. 18:40 skrev Bill Woodcock :
> 
> Why don't we just swap out your 40g switch for a 100g switch?  You've had the 
> 40g one for a while, and we anticipate upgrades every 18-24 months. 
> 
>-Bill
> 
> 
>> On Aug 20, 2017, at 08:46, Mike Hammett  wrote:
>> 
>> I first sent to an IX-specific mailing list, but as I have yet to see the 
>> message hit the list, I figured I would post it here as well. 
>> 
>> We've had multiple requests for 100G interfaces (instead of Nx10G) and no 
>> one seems to care about the 40G interfaces we have available. 
>> 
>> Looking at cost effective options brings us to whitebox switches. Obviously 
>> there's a wide range of hardware vendors and there are a few OSes available 
>> as well. Cumulus seems to be the market leader, while IPinFusion seems to be 
>> the most feature-rich. 
>> 
>> We're not doing any automation on the switches at this time, so it would 
>> still need decent manual configuration. It wouldn't need a Cisco-centric CLI 
>> as we're quite comfortable managing standard Linux-type config files. We're 
>> not going all-in on some overlay either given that we wouldn't be replacing 
>> our entire infrastructure, only supplementing it where we need 100G. I know 
>> that LINX has gone IPinfusion. What OS would be appropriate for our usage? 
>> I'm not finding many good comparisons of the OSes out there. I'm assuming 
>> any of them would work, but there may be gotchas that a "cheapest that meets 
>> requirements" doesn't quite unveil. 
>> 
>> Any particular hardware platforms to go towards or avoid? Broadcom Tomahawk 
>> seems to be quite popular with varying control planes. LINX went Edgecore, 
>> which was on my list given my experience with other Accton brands. 
>> Fiberstore has a switch where they actually publish the pricing vs. a bunch 
>> of mystery. 
>> 
>> Thoughts? 
>> 
>> 
>> 
>> 
>> - 
>> Mike Hammett 
>> Intelligent Computing Solutions 
>> http://www.ics-il.com 
>> 
>> Midwest-IX 
>> http://www.midwest-ix.com 
> 

Re: Virtual or Remote Peering

[Fredrik Korsbäck]

How well does this service work? I understand it usually involves 
point-to-multipoint Switched Ethernet with VLANs and resold IX ports. Sounds 
like a service for ISP that would like to peer, but have relatively small 
volumes for peering purposes or lopsided volumes.


Roderick Beck

Director of Global Sales

United Cable Company

DRG Undersea Consulting

Affiliate Member

www.unitedcablecompany.com

85 Király utca, 1077 Budapest

rod.b...@unitedcablecompany.com

36-30-859-5144


[1467221477350_image005.png]



Its like buying regular ip-transit, but worse.




--
hugge

Re: Anyone using Arista 7280R as edge router?

[Fredrik Korsbäck]

On 14/04/17 15:51, David Hubbard wrote:
> Hey all, have some Brocade MLXe’s that can no longer handle a full v4 and v6 
> route table while also having VRF support (dumb CAM profile limitations in 
> the software).  Mine don’t do anything fancy; just BGP to a few upstream 
> peers and OSPF/OSPFv3 to the inside, management VRF, some ACL’s.  I’m looking 
> at the ASR9001 with add-on ports since I need (10) 10gig.  However, I’ve also 
> been running some Arista 7280SE’s for the past 18 months with no issues, and 
> they want me to consider their 7280R since it would give me more ports, in 
> addition to some higher speed ports, which would be nice if I ever want to 
> upgrade some of our peering to 40 or 100gig.
> 
> Arista’s specs say the 7500R / 7280R can handle 1M ipv4+ipv6 routes in 
> hardware (FIB):
> 
> https://www.arista.com/assets/data/pdf/Whitepapers/FlexRoute-WP.pdf
> 
> In theory, it would last at least a few years if the v4 table doesn’t get too 
> crazy between now and then.
> 
> Curious if anyone has deployed a 7500R or 7280R in this role and what the 
> feedback has been?
> 
> The 9001’s 4M ‘credits’ for the combo of v4 +(2)v6 routes obviously goes much 
> further, but I think either one would make it to their expected end of life, 
> or if not on the Arista side, I’d probably have spent half as much.
> 
> Thanks,
> 
> David
> 

I have a bunch of 7280R in a edge-peering role in as2603 network. Works really 
well and now with the latest additions of subroutemaps and a very optimistic 
road-map for features i think these style boxes will definitely see more 
market. Comparing the price of a 7820R 1Tb box with like a Juniper MX with 1Tb 
worth of ports its not even on the same play-field.

I wouldn't count on the 1M routes to last a lifetime but on the other hand its 
very easy to re-skill the box to something else. If we look at broadcoms 
road-map there is also new 7280s destined to come out quite soon with the 
Jericho+ chipset, broadcom promises 20-25% table-increase so we will see what 
arista, cisco and the others boils it down to when the chip has gotten a box 
wrapped around it.



-- 
hugge

Re: importance of fiber cleaning

[Fredrik Korsbäck]

On 21/09/16 11:53, Mel Beckman wrote:
> This is a very comprehensive article, and worth handing out to techs. I have 
> one comment on Balder’s OTDR suggestion, and one on the article’s microscope 
> instructions.
> 
> Although it certainly can’t hurt to run an OTDR test (except for extended 
> downtime), I fear hauling out the extra gear will prompt many techs to put 
> off fiber cleaning. In my experience, just doing the cleaning solves 99.9% of 
> the problem. Anything that an OTDR would pick up would likely severely impact 
> performance, while dirty connector will just increase the error rate.
> 
> Also, the article didn’t mention eye safety when using a fiber microscope. 
> The example showed a USB digital video microscope, but many maintainer kits 
> in the field have much cheaper direct-view optical microscopes. Viewing an 
> energized fiber with a direct-view microscope can cause major eye damage. I 
> recommend all fiber kits throw out their optical scopes and substitute a USB 
> or WiFI scope (some of these can be used with a cell phone or tablet). 
> 
>  -mel

Cool that this article got posted here (im not the author, but im the dude in 
the pictures).

One reason we didn't bring up OTDR in the article is that OTDR-test only works 
on certain occasions. If you have problems with a circuit and decide to take it 
down to clean it up and inspect patches and ODFs you need to also disconnect 
the other side if you feel doing a OTDR-test. Launching light from a 
OTDR-instrument where you have a regular transceiver on the other side gives a 
potential risk
of destroying it. Also if running the OTDR towards a TX will also skew up the 
results since there is already light in the fibre.

We have however (both me and jörgen) done a few articles where we go through 
OTDR-stuff.

https://www.sunet.se/blogg/sites-and-fibers-being-delivered/
https://www.sunet.se/blogg/first-test-on-real-fiber/
https://www.sunet.se/blogg/full-speed-ahead/

https://www.sunet.se/blogg/otdr-grundlaggande-om/ (use google translate on that 
one, i can get it translated if its of interest)

-- 
hugge

Re: Arista unqualified SFP

[Fredrik Korsbäck]

On 18/08/16 14:45, Mark Tinka wrote:
> 
> 
> On 18/Aug/16 14:42, Nick Hilliard wrote:
> 
>>
>> It is always better to clarify this sort of thing with the account
>> management team before purchasing, and preferably have it in email or
>> writing.  After that, the best approach is to ask support and/or account
>> management nicely rather than "bitching and moaning" as someone else
>> suggested - diplomacy is usually a better long term basis for having a
>> good relationship with your vendor.  Often it's useful to point out
>> discussions like this which indicate that it's been enabled for other
>> people.
> 
> +1.
> 
> We politely said to Arista, "We like your box, but we're afraid that if
> we can't use our existing optics, we'd all miss out on a good
> opportunity working together".
> 
> That did the job.
> 
> Mark.
> 

I think someone from Arista said...

"We are never going to lose an affair due to not supporting 3-d party optics, 
but we will try to convince the customer to buy our stuff, since that's what we 
do, we sell stuff".

In these cheap arista switches, filling them with optics (if the optics is from 
ANET themselves) is usually the same cost as buying like five switches, so of 
course they want a share of that and they will try to convince you that 3rd 
party optics will make the switch go up in flames etc etc.

So its kinda easy... just present three choices.

1. Arista Switch + Arista Optics (at the same price as your favourite 3rd party 
vendor)
2. Arista Switch + 3rd party optics
3. No Arista switch.

I know which one you are gonna get.

-- 
hugge

Re: Arista unqualified SFP

[Fredrik Korsbäck]

On 18/08/16 13:29, Dovid Bender wrote:
> And I was about to jump on to the Arista train.
> 
> Regards,
> 
> Dovid
> 
> -Original Message-
> From: Stanislaw 
> Sender: "NANOG" Date: Thu, 18 Aug 2016 13:24:05 
> To: nanog list
> Subject: Re: Arista unqualified SFP
> 
> Hi all,
> If somebody is following my epic adventure of getting uqualified SFP to 
> work on Aristas, here is the unhappy end of it.
> 
> I've written to Arista support and got the following dialogue:
> Support guy:
> Hi,
> Thank you for contacting Arista Support. My name is  and I'll be 
> assisting you on this case.
> Could you please provide the "show version" output from this switch?
> 
> Me:
> Hi,
> Here it is:
> 
> 
> Support guy:
> Hi,
> Thank you for the information.
> Unfortunately, we are unable to activate your 3rd party components. To 
> ensure ongoing quality, Arista devices are designed to support only 
> properly qualified transceivers.
> Please let me know if you have any other questions.
> 
> Me:
> I do not understand,
> But there is a command which allows using non-Arista transceivers. Why 
> have you implemented it but don't provide an access key to your 
> customers when they ask for it?
> If it is required to sign some papers which declare that I am aware of 
> all the risks and losing my warranty - I agree with that, lets do it. 
> Any way what are the conditions to receive that access key?
> 
> Support guy:
> I'm afraid that there is nothing I'm able to do regarding this 
> situation. If you have any other questions regarding enabling 3rd party 
> options in Arista switches, I suggest to contact your local account team 
> (or sales) for further discussion on this matter.
> 

So. Since when does one handle a business-decisions with the TAC? handing out 
the key means that ANET will not never ever be able to sell you any optics, 
because that's how it works when you ride on the 3rd party optics train. Also 
the TAC need to be flagged to ignore non-official transcievers when sending in 
your issues so they know they don't have to bitch about that.

Id suggest you call your SE/TAM instead of TAC for this.

Or buy something where you can brand the EEPROM with something more appropriate 
that a ANET-switch like

-- 
hugge

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Fredrik Korsbäck]

On 17/06/16 01:09, Baldur Norddahl wrote:
> Hi,
> 
> I have studied Netnod extensively because we want to become members, but we
> can not simply because it is too expensive. I just signed a deal with he.net
> for a flatrate 10G transit for about the same as the 10G Comix port cost.
> The difference being that the he.net port provides much more value and
> besides also provides indirect one-step-away peering with everyone on Comix.
> 
> So from my perspective it is clear that Netnod has a pricing problem here.
> Especially for the lower speeds (10G). There is also a value problem
> because the only Comix peer that moves a lot of traffic to us is Akamai,
> and they promised that we could peer directly skipping the middleman.
> 
> We are based in Copenhagen. The Netnod IX in Stockholm would provide a lot
> more value, but to get there we have to add the cost for transport and
> after doing that, the comparison to the 10G he.net transit is just silly.
> 
> Here is an opinion: If the IX pricing is comparable to transit, the service
> needs to be too. Netnod will need to connect the five (I think there are
> five) Netnod IX'es into one big domain. I am meeting with NL-IX next week
> and as I understand it, that is exactly what they did - we will probably
> buy NL-IX and skip Netnod for this single reason.
> 
> I feel that smaller providers are being let down by the IX community at
> this point. The value of "smaller" is going to get larger if the IX
> community does not move with the transit providers. We want to take part
> but there is a limit of how much over price you can sign onto and keep your
> job.
> 
> Regards,
> 
> Baldur

For me (and middle-sized non-profit ISP) this is something that will eventually 
solve itself, because of people tend to vote with the wallets.

As of now (looking at job's excellent spreadsheet) we can see that regular 
transit is cheaper or atleast on-par with 6-8 of the more expensive IXPs in 
that spreadsheet. We can use hurricane-electrics longlasting google-ad that 
promises $0.26/Mbps on a 10G as a baseline, there is cheaper and much more 
expensive bits to be had naturally but for the sake of argument. While transit 
is almost always a
better thing then connecting a IXP if cheap bits is of your prime concern, 
there is still quality to gain by being able to take care of some of the 
traffic yourself over an IXP. But this also has a tippingpoint in terms of 
price.

The other battle IXPs needs to take into consideration is the price of PNI´s 
nowadays. It is MUCH easier to get really good value from private interconnects 
seeing as most operators has more traffic to less destinations then what we had 
a few years back, and $BIGCLOUD is usually very forthcoming into helping out 
with this since the bits is the product they sell, not the transport of the 
bits.

A 10G port at DECIX Frankfurt is 22800EUR/year. For 22800/year you can go out 
in the market today, buy a jericho-switch which does internet-scale tables and 
BGP and has buffers and all the stuff you need for a peering-edge and will get 
1T worth of faceplate connectors, support for that same switch for about three 
years and you still has some change left for icecream. Thats alot of possible 
PNI´s.
Then we need to factor in costs of crossconnects - if you are in a 
$BIGNAMEDATACENTER  will probably have to pay anything between 19 - 300 
EURO/month for said crossconnect (or the other company has to, or you split 
it). If you are in $NOTSOBIGNAMEDATACENTER you can perhaps pull the patch 
yourself and only pay the CAPEX of 11EUR of buying the patch so this varies 
heavily.

What i do when i think about these things is to produce a matrix where the cost 
of the IXP in that region is measured towards a full TCO of a PNI, factoring in 
what a 10G/100G port costs in terms of buying hardware and then all the OPEX 
surrounding it in terms of 
hardwaresupport/xconns/powerconsumtion/patching/documentation of patching etc 
etc.

For me - PNIs make economical sense if i can shave of 313Mbit/s of my IXP 
(cheap DC / high IXP cost) in the lowside, and about 3.67Gbit/s on the highside 
(expensive DC / cheap IX). And this is with quite high port CAPEX (juniper mx), 
when and if peering-edge is being produced in something like a Jericho-box i 
expect these requirements will drop to the floor.

Last year i added 0 new IXPs, upgraded 0 IXPs, but i added over 30 new PNI's.

If IXPs wants more of those bits, adjusting prices much more aggresively is 
what can bring this back to their market, instead of the 
datacenter-crossconnect market.

-- 
hugge

Re: NOC AS1836 green.ch AG

[Fredrik Korsbäck]

On 03/05/16 21:23, Marco Paesani wrote:
> Hi Arnold,
> nobody answer at 'peer...@green.ch' for this reason I write here on NANOG.
> Ciao,
> 
> 
> Marco Paesani
> 
> 
> Skype: mpaesani
> Mobile: +39 348 6019349
> Success depends on the right choice !
> Email: ma...@paesani.it
> 

Marco.

As I've told you when you hunted me down the last time through backside 
channels.. :)

When you don't get an answer on peering@ addresses its not because they don't 
read the emails. Instead of declining a peering-request formally people tend to 
just use the silent treatment and have that represent a "No". If you don't hear 
back - you probably don't fulfill the requirements to be eligible to peer with 
a network and for a big network with selective or restrictive policy, answering
"no" to emails all day long isn't productive for anyone.

This is one of MANY unwritten rules in the peering-world which can be hard to 
grasp at first. To understand the nomenclature and the "rules", going on a 
field trip to conferences such as NANOG, RIPE, EPF, GPF and the such is a great 
way to understand the game, so one can act accordingly to how the playfield 
looks like.

-- 
hugge @ AS2603

Re: New Switches with Broadcom StrataDNX

[Fredrik Korsbäck]

On 18/04/16 20:01, Colton Conor wrote:
> As a follow up to this post, it look like the Arista 7500R series has this
> new chip inside of it.
> 
> On Wed, Jan 20, 2016 at 9:34 AM, Jeff Tantsura 
> wrote:
> 
>> That's right, logic is in programming chips, not their property. You just
>> need to know what to program ;-)
>>
>> Regards,
>> Jeff

Not only the big one, The new Arista 7280R is also the new BRCM DNX aka Jericho.

Aswell as Cisco NCS550X series.

-- 
hugge

Re: PCH Peering Paper

[Fredrik Korsbäck]

On 11/02/16 00:34, Patrick W. Gilmore wrote:
> I quoted a PCH peering paper at the Peering Track. (Not violating rules, 
> talking about myself.)
> 
> The paper is:
>   
> https://www.pch.net/resources/Papers/peering-survey/PCH-Peering-Survey-2011.pdf
> 
> I said “99.97%” of all peering sessions have nothing behind them more than a 
> “handshake” or an email. It seems I was in error. Mea Culpa.
> 
> The number in the paper, on page one is, 99.52%.
> 
> Hopefully everyone will read the paper, and perhaps help create better data.
> 

Well, how about crowdsourcing some data?

3145 eBGP settlement-free peering-sessions (v4 and v6 combined) in US and EU. 
350k routes recieved over SFI peering.

1 Written contract in EU for SFI
1 Written contract in US for SFI

R&E Sector

-- 
Apparently not a peering coordinator.
Fredrik "hugge" Korsbäck
AS2603