Re: Frustration with increasing information demands from Network Vendors

[Charles Polisher]

On 10/11/24 07:19, Nick Hilliard wrote:

Rich Kulawiec wrote on 11/10/2024 15:07:

Every answer to every question at every
site should be different and every one of them should be wrong.


This approach does lead to interesting conversations with tech support 
people when you explain to them that your birthday is some time in the 
1700s or 1800s.


But a token is a token, right?

Nick
(fwiw, my cat's name is "ofo0tL1!Rgz8WPQ+")


Comcast rep thought her computer was misbehaving
because my "high school's name" looked like line noise.

Awkward conversation with support when AT&T support
wanted to confirm my email address (f...@attsucks.com).

Re: constant FEC errors juniper mpc10e 400g

[Charles Polisher]

On 4/18/24 11:45, Aaron Gould wrote:


Thanks.  What "all the ethernet control frame juju" might you be 
referring to?  I don't recall Ethernet, in and of itself, just sending 
stuff back and forth.  Does anyone know if this FEC stuff I see 
concurring is actually contained in Ethernet Frames?  If so, please 
send a link to show the ethernet frame structure as it pertains to 
this 400g fec stuff.  If so, I'd really like to know the header 
format, etc.


-Aaron



IEEE Std 802.3™‐2022 Standard for Ethernet
(§65.2.3.2 FEC frame format p.2943)
https://ieeexplore.ieee.org/browse/standards/get-program/page/series?id=68

Also helpful, generally:
ITU-T 2000 Recommendation G975 Forward Error Correction for Submarine 
Systems

https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-G.975-200010-I!!PDF-E&type=items

Re: SRI's Dan Lynch dies

[Charles Spurgeon]

Well said. Dan Lynch's requirement that vendor gear must interop on
the show floor was a big deal and required extra work to accomplish
compared to other "dog and pony" shows.

His ground rules for the Interop conferences had the effect of
ensuring that competent people showed up to set up and run the gear
being shown. Interoperability also ensured that vendors got the
opportunity to find bugs and fix things (whether they wanted to or
not) which helped improve the quality of vendor offerings.

Another benefit was that the people who showed up to make things work
on the show floor were often available during the show as resources to
answer questions or help with issues.

As a result of Dan's efforts the Interop conferences were a very
valuable resource both for attendees and vendors for a number of
years.

-Charles

--
Charles Spurgeon
c.spurg...@austin.utexas.edu
Networking at the University of Texas at Austin (1988-present)
Networking at Stanford University (1981-1988)   

* Sajit Bhaskaran  [2024-04-01 11:12:39 -0700]:

> RIP Dan Lynch. It is worth adding that he was also the founder of
> the Interop shows in the mid 80s which achieved a great deal in
> terms of advancing TCP/IP adoption, and inter-operability testing
> was a big deal back then when the future of TCP/IP was also not at
> all certain, as it was in competition then with the ISO/OSI protocol
> suite. Dan's efforts and passion as an entrepreneur created an
> exponentially growing community of users and vendors all over the
> world that made the TCP/IP protocol suite the de facto standard.
> Thanks very much for sharing. Today we take the Internet for
> granted. It could have been very different.
> 
> On 3/31/2024 12:19 PM, Jay R. Ashworth wrote:
> >>From Lauren Weinstein @ PRIVACY Digest:
> >
> >"""
> >Dan Lynch, one of the key people involved in building the Internet and
> >ARPANET before it, has died.
> >
> >Dan was director of computing facilities at SRI International, where
> >ARPANET node #2 was located and he worked on development of TCP/IP, and
> >where the first packets were received from our site at UCLA node #1 to
> >SRI, and later at USC-ISI led the team that made the transition from the
> >original ARPANET NCP protocols to TCP/IP for the Internet. And much more.
> >
> >Peace. -L
> >"""
> >
> >He was well written up across the web, but here's a 2021 piece for those
> >who aren't as familiar with his background:
> >
> >https://www.internethalloffame.org/2021/04/19/dan-lynchs-love-brilliant-complexity-fuels-early-internet-development-growth/
> >
> >And his IHoF induction speech:
> >
> >http://opentranscripts.org/transcript/dan-lynch-ihof-2019-speech/
> >
> >I would note his age here, as obits usually do, but it seems unusually 
> >difficult
> >to learn.
> >
> >Happy landings, Mr Lynch.
> >
> >Cheers,
> >-- jra
> >>This message is from an external sender. Learn more about why this <<
> >>matters at https://links.utexas.edu/rtyclf.<<

-- 
Charles Spurgeon
c.spurg...@austin.utexas.edu

ITS Networking
University of Texas at Austin
512.475.9265 desk
512.750.3675 Cell

Re: Without further comment:

[Charles Polisher]

On 4/1/24 07:14, chris wrote:

ROFL. networking is a stream of zeros and one's. You are either 0 or 1 :))


Completely ignoring the real hardware layer where
it's all about eye diagrams, transitioning constantly.
Between voltage levels. Or I guess lumens. Or phase
shifts. Pick your poison^H^H^H^H^H^Htransport medium.

Your welcome!

Re: edgecast - lots of traffic at ~3:00 a.m.

[Charles Monson]

I'm seeing an uptick from Apple's AS6185, along with the usual CDNs,
all around that time. Looks like there is a new iOS update (17.3).

On Tue, Jan 23, 2024 at 9:19 AM Aaron Gould  wrote:
>
> Anyone else see a lot of traffic inbound from the Internet last night
> (early this morning) at ~3:00 a.m. central time?  I see an IP Address,
> (93.184.215.240 - EdgeCast), which I think is EdgIO (fka limelight).
> Any idea what this is related to? (something tells me it's a game update)
>
> --
> -Aaron
>

Re: Backward Compatibility Re: 202401100645.AYC Re: IPv4 address block

[Charles Polisher]

Owen DeLong wrote:

> Some, but not a lot. In the case of the DTMF transition, the
> network and handsets were all under the central control of a
> single provider at a time when they could have forced the change
> if they really wanted to. After all, nobody was going to cancel
> their phone service altogether (or such a small fraction of
> subscribers as to count as a rounding error anyway) over the
> issue and AT&T could simply have shipped replacement phones
> with instructions for returning the older phone and done a
> retrofit operation if they really wanted to drive the transition.

True, yet there's a missing piece to that description: ROI.
In the regulated environment with a mandated X% Return On Invest-
ment (X ≈ 15 IIRC) a bigger expense pie was a better pie because
a bigger expense pie meant a bigger return. This was an inexorable
force that influenced every substantive decision. An expanding
rate base was the One True Path to advancing against the demon
competitors: AT&T and other RBOCs.

In the Bell System setting, before and after Divestiture, a
perpetual and costly migration from IPv4 to IPv6 with all the
attendant cost burdens would have been well tolerated, even
welcomed, in the "C Suite" anyways.

--
Charles Polisher

Re: Issues with prefix / help needed

[Charles Monson]

On Mon, Mar 27, 2023 at 9:05 AM Kevin McCormick  wrote:
>
> IRR Explorer is showing RPKI-Invalid. Maybe RPKI is causing the issue or 
> there is an issue with IRR Explorer?
>
> https://irrexplorer.nlnog.net/prefix/86.104.228.0/24
>
> I do see RIPE and Cloudflare are showing RPKI as valid.
>
> https://rpki-validator.ripe.net/ui/86.104.228.0%2F24/45021?include=related_alloc
>
> https://rpki.cloudflare.com/?view=validator&validateRoute=45021_86.104.228.0%2F24
>
> Curious why IRR Explorer is showing invalid.
>
> Thank you,
>
> Kevin McCormick
>

That seems to just be indicating there are route-objects in RADB that
don't match RPKI, and not related to anything in BGP.

BCP38 For BGP Customers

[Charles Rumford via NANOG]

Hello -

I'm are currently working on getting BCP38 filtering in place for our BGP 
customers. My current plan is to use the Juniper uRPF feature to filter out 
spoofed traffic based on the routing table. The mentality would be: "If you 
don't send us the prefix, then we don't accept the traffic". This has raised 
some issues amongst our network engineers regarding multi-homed customers.


One of the issues raised was if a multi-homed BGP customer revoked a prefix from 
one of their peerings, but continued sending us traffic on the link then we 
would drop the traffic.


I would like to hear what others are doing for BCP38 deployments for BGP 
customers. Are you taking the stance of "if you don't send us the prefix, then 
we don't accept the traffic"? Are you putting in some kind of fall back filter 
in based on something like IRR data?


Thanks!

--
Charles Rumford (he/his/him)
Network Engineer | Deft
1-312-268-9342 | charl...@deft.com
deft.com

Re: Geolocation data management practices?

[Charles Polisher]

On 4/21/22 06:14, Rubens Kuhl wrote:

Besides geofeed, there are also geoidx records in IRRs but whether
geolocation services actually use geofeed or geoidx remains to be
seen. You can see some geoidx: at this IRR entry in TC:
https://bgp.net.br/whois/?q=-s%20TC%20-i%20mnt-by%20MAINT-AS271761

Regarding LACNIC, what LACNIC, NIC.mx and NIC.br do is to select which
RIR or NIR services requests depending on the organisation's country.


Also:

RFC 3693: Geopriv Requirements 
<https://datatracker.ietf.org/doc/html/rfc3693>


RFC 5870: A Uniform Resource Identifier for Geographic Locations ('geo' 
URI) <https://datatracker.ietf.org/doc/html/rfc5870>


RFC 6288: URN Namespace for the Defence Geospatial Information Working 
Group (DGIWG) <https://datatracker.ietf.org/doc/html/rfc6288>


RFC 6397: Multi-Threaded Routing Toolkit (MRT) BGP Routing Information 
Export Format with Geo-Location Extensions 
<https://datatracker.ietf.org/doc/html/rfc6397>


RFC 6772: Geolocation Policy: A Document Format for Expressing Privacy 
Preferences for Location Information 
<https://datatracker.ietf.org/doc/html/rfc6772>


RFC 7942: The GeoJSON Format <https://datatracker.ietf.org/doc/html/rfc7942>

RFC 8142: GeoJSON Text Sequences 
<https://datatracker.ietf.org/doc/html/rfc8142>


RFC 8805: A Format for Self-Published IP Geolocation Feeds 
<https://datatracker.ietf.org/doc/html/rfc8805>


RFC 9092: Finding and Using Geofeed Data 
<https://datatracker.ietf.org/doc/html/rfc9092>


--
Charles Polisher
(/Pedantic, I?/)

Re: BGP Route Monitoring

[Charles Monson]

This sounds like something BMP might be useful for. I haven't used it, but
I would look at OpenBMP (https://github.com/SNAS/openbmp) as a starting
point. I'm not familiar with what commercial offerings are out there, but
I'm sure there are some.

On Thu, Jan 6, 2022 at 7:45 AM Sandoiu Mihai  wrote:

> Hi
>
>
>
> I am looking for a route monitoring product that does the following:
>
> -checks if a specific bgp route from a specific neighbor is present the
> BGP table (in some vrf, not necessarily internet routed vrf) of an ASR9K
> running IOS XR
>
> -sends a syslog message or an alarm if the route goes missing
>
>
>
> The use case is the following: we are receiving same routes over 2 or more
> bgp peerings, due to best route we cannot really see at the moment if one
> of the routes ceased to be received over a certain peering.
>
>
>
> Alternative approach: a product that measures the number of bgp received
> prefixes from a certain peer.
>
>
>
> Do you know of such product that is readily available and does not require
> ssh sessions to the routers and parsing the outputs?
>
> I am trying to find a solution that does not require much scripting or
> customization.
>
>
>
> Many thanks.
>
>
>
> Regards
>
> Mihai
>
>
>

Re: (Slightly OT?) K8S Platform As A Service Recommendations

[Charles N Wyble]

On 4/7/21 11:38 PM, Raymond Burkholder wrote:
On 4/7/21 9:16 AM, Charles N Wyble wrote:> Does anyone have a 
recommendation for a self-hosted, on premise,

> platform as a service layer for k8s (specifically k3s)?
FWIW:

Maybe you don't need kubernetes:
https://endler.dev/2019/maybe-you-dont-need-kubernetes/



I have considered not running k8s. I didn't run it for a long time. I 
kept an eye on developments and waited for it to mature.


However the amount of applications and services I am now needing to 
support and the HA requirements and need for standardization etc I 
don't know of a better option.





Manually install a single node Kubernetes cluster on Debian
http://meta.libera.cc/2021/03/manually-install-single-node-kubernetes.html 



Or run Salt or something and spin up LXC containers.




Sure and how do I manage IP addresses? Ports? HA? Containers 
(LXC/docker) is the easy part (on a relative basis anyway!) . It's the 
meta stuff around it that gets messy.  The orchestration piece of the 
containers is the difficult part.



As I mentioned, we already have a mature stack outside the app runtime 
layer (for certs/LDAP/database etc). We just want applications/services 
on k8s. Minimize the complexity/blast radius! :)

(Slightly OT?) K8S Platform As A Service Recommendations

[Charles N Wyble]

Hello all,


I know this is primarily a networking list, but I know lots of server 
admins hang out here.


Does anyone have a recommendation for a self-hosted, on premise, 
platform as a service layer for k8s (specifically k3s)?


I have written up some context here:

https://github.com/TSYSGroup/docs-techops/blob/master/Applications/AppRuntimeLayerTodo.md

tl:dr : I have about 70 to 200 apps / (micro) services that will need to 
run across a handful of k3s servers . I already have HA 
database/networking/certificate/application load 
balanacer/authentication stacks in production use, I am currently 
running the actual websites/applications on a single Ubuntu LAMP server 
and want to build out an HA runtime layer for all the 
properties/applications and need a way to orchestrate k3s/metallb


Rancher rio has come up a few times in my 
research:https://bram.dingelstad.works/blog/finding-the-right-paas-for-k8s/ 



In addition to the web apps , I will also will be running a number of 
r&d applications and CUDA enabled containers (across a mix of physical 
x86/jetson/tegra machines with k3s workers).


Suggestions/comments/questions/flames welcome :)

On or off list as you prefer.

Re: wow, lots of akamai

[Charles Polisher]

On 4/5/21 10:23 PM, Robert Brockway wrote:

On Thu, 1 Apr 2021, Jean St-Laurent via NANOG wrote:

What happened is that it would create a kind of internal DDoS and 
they would all timed out and give a weird error message. Something 
very useful like Error Code 0x8098808 Please call our support line at 
this phone number.


If only there was a way to address the Thundering Herd problem before 
the cloud. :)


This simple change to add 3 lines of code to add a random artificial 
boot penalty of few seconds, completely solve the problem.


Bingo.  Now, the trick is to catch this before it causes an self-DDoS.

This is a problem that has been recognised for decades and this is 
unfortunately a good example of how operational experience is still 
not being distributed properly.  Too many managers think that 
operational work is obvious and just a result of common sense.  It isn't.


Same problem as disk drives powering up simultaneously
in datacenters. SCSI drives have (had?) a random delay
mechanism to distribute the initial power surge over a few
seconds.

Re: CIDR cleanup

[Charles Cloughly]

Not Perl, though this may be useful depending on your environment:
https://github.com/rus-cert/compress-cidr

The examples are for IPv6, though I use it to consolidate lists of IPv4 in a 
variety of jobs/scripts without issue. YMMV.



From: NANOG on behalf of John Von Essen
Sent: Thursday, October 1, 2020 6:32 AM
To: NANOG
Subject: CIDR cleanup




Sorry if this is slightly off-topic, but I am writing some code for a custom 
GeoDNS routemap. My starting data set is a raw list of /24 subnets, no prefix 
aggregation has been done. In other words, its the entire BGP routing table in 
/24 prefixes - tagged by Geo region. Each region is its own txt file with a 
dump of /24’s. As a result, these lists are HUGE. I want to aggregate the 
prefixes as much as possible to create a smaller routemap.



So right now it looks like:



...

105.170.72.0/24 brs

105.170.73.0/24 brs

105.170.74.0/24 brs

105.170.75.0/24 brs

105.170.76.0/24 brs

105.170.77.0/24 brs

105.170.78.0/24 brs

105.170.79.0/24 brs

105.170.80.0/24 brs

105.170.81.0/24 brs

105.170.82.0/24 brs

105.170.83.0/24 brs

105.170.84.0/24 brs

…



and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be aggregated to 
105.170.72.0/21 and so on. I normally use Perl, does anyone now if there is a 
perl module that will automatically do this prefix aggregation? I tried to 
write my code to do this, and its not trivial, just lookinh for a shortcurt. I 
did a breif glance at some CIDR related Perl cpan modules, and nothing has 
jumped out.



Thanks

John

Re: backtracking forged packets?

[Charles Polisher via NANOG]

On 2020-03-13 23:23, William Herrin wrote:
> Can anyone suggest tools, techniques and helpful contacts for
> backtracking spoofed packets? At the moment someone is forging TCP
> syns from my address block. I'm getting the syn/ack and icmp
> unreachable backscatter. Enough that my service provider briefly
> classified it a DDOS. I'd love to find the culprit.

FWIW, Bellovin et al proposed an ICMP traceback mechanism in 2001
( https://tools.ietf.org/html/draft-ietf-itrace-04 ), but it seems
not to have progressed. Abstract:

 It is often useful to learn the path that packets take through the
 Internet, especially when dealing with certain denial-of-service
 attacks. We propose a new ICMP message, emitted randomly by routers
 along the path and sent randomly to the destination (to provide
 useful information to the attacked party) or to the origin (to
 provide information to decipher reflector attacks).

-- 
Chuck Polisher

Re: [j-nsp] MX10003 rack size

[Anderson, Charles R]

1000mm deep.  APC AR3100 racks are 600mm x 1070mm.  APC also makes 1200mm deep 
ones, and 750mm wide ones, and both together.

On Wed, Aug 07, 2019 at 04:12:26PM +, Richard McGovern wrote:
> Pete "1000 deep rack"??  Is that fathoms __
> 
> Richard McGovern
> Sr Sales Engineer, Juniper Networks 
> 978-618-3342
>  
> I’d rather be lucky than good, as I know I am not good
> I don’t make the news, I just report it
>  
> 
> On 8/7/19, 6:20 AM, "Pete Webb"  wrote:
> 
> No mate,
> I made the same mistake.
> Minimum you can get away with is 1000 deep racks, and even then you have 
> to leave the front air filter off.
> 
> Pete
> 
> -Original Message-
> From: juniper-nsp  On Behalf Of 
> Sander Steffann
> Sent: 30 July 2019 13:32
> To: nanog ; Juniper List 
> Subject: [j-nsp] MX10003 rack size
> 
> Hi,
> 
> Has anyone ever managed to fit a Juniper MX10003 in a 90cm deep rack? 
> Without applying power tools to either the rack or the router ;)
> 
> Cheers,
> Sander

Re: few big monolithic PEs vs many small PEs

[Anderson, Charles R]

On Fri, Jun 21, 2019 at 09:01:38AM -0500, Aaron Gould wrote:
> I was reading this and thought, planet earth is a single point of failure.
> 
> ...but, I guess we build and design and connect as much redundancy (logic, 
> hw, sw, power) as the customer requires and pays for and that we can 
> truly accomplish.

Fate sharing is also an important concept in system design.

Re: BGP prefix filter list

[Anderson, Charles R]

What about these ones?

https://teamarin.net/2019/05/13/taking-a-hard-line-on-fraud/

On Wed, May 15, 2019 at 01:43:30PM +0200, Baldur Norddahl wrote:
> Hello
> 
> This morning we apparently had a problem with our routers not handling 
> the full table. So I am looking into culling the least useful prefixes 
> from our tables. I can hardly be the first one to take on that kind of 
> project, and I am wondering if there is a ready made prefix list or similar?
> 
> Or maybe we have a list of worst offenders? I am looking for ASN that 
> announces a lot of unnecessary /24 prefixes and which happens to be far 
> away from us? I would filter those to something like /20 and then just 
> have a default route to catch all.
> 
> Thanks,
> 
> Baldur

Re: [EXT] RE: Widespread Firefox issues

[Charles Bronson]

From: NANOG  on behalf of Keith Medcalf 

Sent: Saturday, May 4, 2019 3:14:53 AM
To: NANOG list
Cc: Constantine A. Murenin
Subject: [EXT] RE: Widespread Firefox issues


HTTPS: has nothing to do with the website being "secure".  https: means that 
transport layer security (encryption) is in effect.  https: is a PRIVACY 
measure, not a SECURITY measure.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.


>-Original Message-
>From: NANOG [ mailto:nanog-boun...@nanog.org] On Behalf Of Constantine
>A. Murenin
>Sent: Friday, 3 May, 2019 21:02
>To: Brielle Bruns
>Cc: NANOG list
>Subject: Re: Widespread Firefox issues
>
>On Fri, 3 May 2019 at 20:57, Brielle Bruns  wrote:
>
>
>   Just an FYI since this is bound to impact users:
>
>   https://bugzilla.mozilla.org/show_bug.cgi?id=1548973
>
>   Basically, Mozilla forgot to renew an intermediate cert, and
>people's
>   Firefox browsers have mass-disabled addons.
>
>   Whoops.
>
>
>
>This is why it's important that every single website on the internet
>is available ONLY over HTTPS.  Don't forget to install an HSTS
>policy, too, so, if anyone ever visits Kazakhstan or a security-
>conscious corporate office, they'll be prevented from accessing the
>cute pictures of cats on your fully static website.  Of course, don't
>forget to abandon HTTP, too, and simply issue 301 Moved Permanently
>redirects from all HTTP targets to HTTPS, to cover all the bases.
>
>Backwards compatibility?  Don't you worry — no browser lets anyone
>remove HSTS, once installed, so, you're golden.  And HTTPS links
>won't fallback to HTTP, either, so, you're good there, too — your
>cute cats are safe and secure, and once folks link to your new site
>under https://, your future self will be safe and secure from ever
>having the option to go insecure again.  I mean, why would anyone go
>"insecure"?  Especially now with LetsEncrypt?
>
>
>Oh, wait…
>
>
>Wait a moment, and who's the biggest player behind the HTTPS-only
>movement?  Oh, and Mozilla's one of the biggest backers of
>LetsEncrypt, too?  I see…  Well, nothing to see here, move along!
>#TooBigToFail.
>
>
>C.

I may be wrong and if so, I am happy to be corrected, but I don't think that 
statement is entirely true. The certificate not only encrypts the connection, 
it also verifies that you are connecting to the server you intend to. That 
second component is a security measure.


Charles Bronson

Re: a quick survey about LLDP and similar

[Anderson, Charles R]

On Thu, Feb 28, 2019 at 10:00:55AM +0100, Pierfrancesco Caci wrote:
> 
> Hello,
> having a bit of a debate in my team about turning on LLDP and/or CDP.
> I would appreciate if you could spend a minute answering this
> survey so I have some numbers to back up my reasoning, or to accept
> defeat.
> 
> https://www.surveymonkey.com/r/TH3WCWP
> 
> Feel free to cross-post to other relevant lists. 
> 
> Thank you
> 
> Pf

We require LLDP/LLDP-MED to configure our VOIP phones.

For trunk links, it is extremely helpful to verify correct topology.

For datacenters, it is EXTREMELY helpful to verify hypervisor connectivity.

Re: fs.com dwdm equipment

[Anderson, Charles R]

I concur.  I have also used CWDM and DWDM optics and they are fine.  I have had 
one QSFP+ optic go bad.

On Mon, Feb 18, 2019 at 07:47:10PM +, Brian R wrote:
> Samir,
> 
> I have purchased over a thousand SFPs from Fiber Store.  I can recall less 
> than 5 having problems when we received them (not all even DOA) and I know 
> less than 10 dead even after deployment.  Some we have ad running for 4+ 
> years.  We have done very little with their SFP+ equipment, really only 
> testing and a few lower priority links.
> The only downside to the SFPs that we found was the variance of power.  Say 
> an Adtran, Cisco, Juniper, HP SFP is rated from -3dB to -8 db (all units I 
> have used them with), the equivalent FS direct SFP we have seen as hot as 3dB 
> and as weak as -15dB.  These extremes are fairly rare but we have still seen 
> them.
> Distribution (approximate):
> 80% SM single fiber SFPs (mostly 10km - 40km, some 60km & 80km)
> 7% 1Gb Copper
> 5% MM SFPs
> 5% SM dual fiber SFPs
> 3% others (SFP+, GPON, testing, etc)
> 
> I have not used them for any DWDM applications and only used them a few times 
> on an older CWDM link that used standard SFPs into a MUX.  This was not over 
> great distances (less than 40 miles).  With WDM SFP power consistency was 
> important so we did not play much with it, granted most of the SFPs I am 
> purchasing are in the $10-$25 range so take the extremes with a grain of salt.
> 
> Their sales has always been very responsive and helpful.  The 
> support/engineering, the few times I worked with them, were helpful but the 
> language barrier was harder here.
> 
> Brian
> 
> 
> From: NANOG  on behalf of Samir Rana 
> 
> Sent: Sunday, February 17, 2019 12:42 PM
> To: nanog@nanog.org
> Subject: fs.com dwdm equipment
> 
> Hello All,
> 
> Does anybody have experience with fs.com their production environment? Are you they working without any issue? How's 
> their warranty support if the issue arises?
> 
> Thanks in advance for all the answers and help.

Re: A few GPON questions...

[Anderson, Charles R]

On Tue, Dec 11, 2018 at 07:07:49PM +0100, Baldur Norddahl wrote:
> >
> >
> > And WDM gear if necessary...heck even passive CWDM if you have a riser
> > space issue.
> >
> 
> WDM is much more expensive than GPON.
> 
> I am still waiting for one of the 10G PON variants to become available. We
> want to deliver 10G to customers as >1G is becoming common on CPE Wi-Fi
> routers. But doing it with WDM is too expensive and p2p uses more fiber
> than we have.

Passive CWDM is cheap and supports 10gig.

Re: A few GPON questions...

[Anderson, Charles R]

On Tue, Dec 11, 2018 at 05:36:47PM +, Aled Morris via NANOG wrote:
> On Tue, 11 Dec 2018 at 17:30, Jason Lixfeld  wrote:
> > There’s only so much space in conduits, risers and ducts.  At some point, 
> > scale would press this up against physical infrastructure realities 
> > depending on how far the active gear at the head end is from the subscriber.
> 
> A point made earlier was that typically in a campus environment, most
> every riser cupboard has access to power so you can easily build a
> regular Ethernet LAN with a switch on every floor/corridor/hub.
> Basically, everywhere that you'd put a GPON splitter.

And WDM gear if necessary...heck even passive CWDM if you have a riser space 
issue.

Re: Cogent charging 50/mo for BGP (not IPs, the service)

[Anderson, Charles R]

I was told they only charge it if you have bigger than a /29 from them.

On Wed, Oct 17, 2018 at 04:12:01PM +, David Hubbard wrote:
> They charge it even if you’re using your own address space.  It’s a fee 
> simply for establishing BGP with them on a given circuit.  I believe if you 
> used static routes and their space, you would not have to pay it.
> 
> From: NANOG  on behalf of Josh Luthman 
> 
> Date: Wednesday, October 17, 2018 at 12:10 PM
> To: Brielle Bruns 
> Cc: NANOG list 
> Subject: Re: Cogent charging 50/mo for BGP (not IPs, the service)
> 
> I view Cogent IP space as a way to lock customers to their service, ie make 
> them sticky.
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> On Wed, Oct 17, 2018, 12:03 PM Brielle Bruns 
> mailto:br...@2mbit.com>> wrote:
> On 10/17/2018 9:47 AM, Josh Luthman wrote:
> > Has anyone else dealt with this mess?  Even my Cogent rep admits it's
> > unique to their business.
> 
> That sounds like the BS the first company I worked for tried to pull.
> 
> One would think they'd welcome customers bringing their own IP space
> since it saves them money by not using up precious Cogent IPv4 address
> space.
> 
> Hell, I even have BGP for v4 and v6 over my CenturyLink biz fiber, and
> its available as part of the enhanced package they offer with no extra fees.

Re: Youtube Outage

[Charles Mills]

The reports I've seen showing it as a worldwide outage.

On Tue, Oct 16, 2018 at 10:14 PM Nathan Brookfield <
nathan.brookfi...@simtronic.com.au> wrote:

> Australia too….
>
>
>
> *From:* NANOG  *On Behalf Of *Oliver O'Boyle
> *Sent:* Wednesday, October 17, 2018 1:08 PM
> *To:* marshall.euba...@gmail.com
> *Cc:* North American Network Operators' Group 
> *Subject:* Re: Youtube Outage
>
>
>
> Same in Montreal.
>
>
>
> On Tue, Oct 16, 2018 at 9:52 PM Marshall Eubanks <
> marshall.euba...@gmail.com> wrote:
>
> Reports (and humor) are flooding twitter.
> On Tue, Oct 16, 2018 at 9:44 PM Ross Tajvar  wrote:
> >
> > You beat my email by seconds. Yes, it is widespread.
> >
> > On Tue, Oct 16, 2018 at 9:39 PM, Kenneth McRae via NANOG <
> nanog@nanog.org> wrote:
> >>
> >> Is this widespread?
> >
> >
>
>
>
>
> --
>
> :o@>
>
>
>

Re: ARIN RPKI TAL deployment issues

[Anderson, Charles R]

On Wed, Sep 26, 2018 at 02:18:43PM -0700, Mark Milhollan wrote:
> On Tue, 25 Sep 2018, Job Snijders wrote:
> 
> >We really need to bring it back down to "apt install rpki-cache-validator"
> 
> You say this as if no packager has a way to display and perhaps require 
> approval of the license nor any way to fetch something remote as part of 
> the installation process, e.g., the Microsoft "freely" supplied TTF 
> files ...
>   [...]
> 
> I bet apt, dnf, pacman, pkg_add, yum, etc., do as well -- actually I 
> know some of those do.  Perhaps fetching as part of installing is less 
> desireable than already present at the outset, but it might appease ARIN 
> and be workable (or superior) for many.

rpm/yum/dnf do NOT have a way to allow the installation of packages to
interact with the user.  They specifically block this functionality
since it goes against their design of allowing non-interactive
installs.

RE: [EXT] Fwd: Re: problems sending to prodigy.net hosted email

[Charles Bronson]

If this isn't pertinent to the list, feel free to answer privately. How did you 
implement the server that got rid of ARP storms?


Charles Bronson



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell
Sent: Monday, March 19, 2018 9:31 PM
To: nanog@nanog.org
Subject: [EXT] Fwd: Re: problems sending to prodigy.net hosted email

Two DNS servers hosted on one box (or VM object), even with two addresses, is 
easily compromised by DDoS amplification attacks.  That's the norm for a number 
of "web control panel" systems like Plesk and CPanel.

It depends on the scale of your operations.  Last time I was in that situation, 
I had roughly 25,000 domains spread across 30 servers.  Life became MUCH 
simpler when I put up dedicated, and high-power, physical systems running 
non-recursive BIND for DNS1 and DNS2, as well as another pair of boxes running 
recursive servers as DNS3 and DNS4.

Getting QMail and Exim to "smart host" to my monster MX servers proved to be 
pretty easy, and I even was able to get the web servers to tell me when a 
mailbox was full so I could reject the SMTP exchange at the edge, instead of 
generating backscatter.

And, with a pool of roughly 4,000 IP addresses, I got rid of ARP storms in our 
network by putting up a little server called "ackbar", that was configured to 
respond to all otherwise unused IP address in our pool. 
(Edge routers were Cisco 7000 class, with DS3 uplinks.)

Lessons learned well.

 Forwarded Message 
Subject: Re: problems sending to prodigy.net hosted email
Date: Mon, 19 Mar 2018 17:55:33 +0100
From: Chris 
To: C. Jon Larsen 
CC: nanog@nanog.org

On Mon, 19 Mar 2018 11:56:16 -0400 (EDT) C. Jon Larsen wrote:

> > Why not? Never had a problem with multiple services on linux, in 
> > contrast to windows where every service requires its own box (or at 
> > least vm).
> 
> Go for it ! Failure is an awesome teacher :)

Don't really see a problem, especially since you normally always have two DNS 
servers...

--
Papst Franziskus ruft zum Kampf gegen Fake News auf. Wir finden, der Mann, der 
sich als Stellvertreter Christi ausgibt, von dem er behauptet, dessen Mutter 
sei zeitlebens Jungfrau gewesen, er hätte über Wasser gehen und selbiges in 
Wein verwandeln können, hat vollkommen recht.

RE: Charter engineer

[Manser, Charles J]

Mr. Carman,

Did someone already reach out to you off-list?

Charles Manser | Principal Engineer I, Network Security | [c] 813-422-4281
14810 Grasslands Dr, Englewood, CO 80112

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Samual Carman
Sent: Sunday, May 14, 2017 1:28 PM
To: nanog@nanog.org
Subject: Charter engineer 

Can a charter engineer please contact me off list I am getting slammed from a 
charter ip address on a local cable node and normal support channels have been 
unhelpful at bet and unwilling to escalate the issue if anyone else has any 
suggestion please feel free to contact Contact may be delayed as I will flying 
back from Dubai today

In addition would a charter voice /internet engineer please contact me off list 
 or someone who specialize in fax machines on the charter network Thanks Sam 
Mettai Inc Yakima, Branch Sent from my home please excuse grammar and spelling 
issues Sent from my iPhone

Get Outlook for iOS<https://aka.ms/o0ukef>
E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or legally privileged 
information. If you are not the intended recipient of this message or if this 
message has been addressed to you in error, please immediately alert the sender 
by reply e-mail and then delete this message and any attachments. If you are 
not the intended recipient, you are notified that any use, dissemination, 
distribution, copying, or storage of this message or any attachment is strictly 
prohibited.

RE: ticketmaster.com 403 Forbidden

[Manser, Charles J]

All,

Thank you for the suggestions. All (3) of the e-mail addresses associated with 
their ARIN records bounced back.

Remote Server returned '< #5.7.133 smtp;550 5.7.133 
RESOLVER.RST.SenderNotAuthenticatedForGroup; authentication required; Delivery 
restriction check failed because the sender was not authenticated when sending 
to this group>' 

It can be difficult for consumers to work these issues individually, so we 
reached out to the NANOG community for an assist. The problem seemed widespread 
and not isolated to single customers and referring them to a web form did not 
seem like an option.

Good news: I am making some progress with the Live Nation/Ticketmaster team.

"Thank you for bringing this to our attention. We are conducting an 
investigation on suspicious activity that has been observed on the range of 
IP's are associated to your connectivity and will make every effort to do this 
as fast as possible."

Thank you all again for the help and I will keep the archive updated if we 
reach a repeatable resolution.

Regards,
 
Charles Manser | Principal Engineer I, Network Security 
charles.man...@charter.com

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of joel jaeggli
Sent: Monday, February 06, 2017 7:38 PM
To: Suresh Ramasubramanian ; mike.l...@gmail.com; Ethan E. 
Dee 
Cc: Niels Bakker ; nanog@nanog.org
Subject: Re: ticketmaster.com 403 Forbidden

On 2/6/17 8:49 AM, Suresh Ramasubramanian wrote:
> My guess is you have or had sometime in the long distant past a scalper 
> operating on your network, using automated ticket purchase bots.
>
> If you still have that scalper around, you might want to turf him.  If he’s 
> ancient history, saying so might induce them to remove the block.
Note that scalper bots benefit from pools of residential ip addresses to
work with in subverting the anti-bot countermeasures of ticket sale
platforms. so there are the legitimate possibility that subverted hosts
are being used for that sort of thing.
> --srs
>
> On 06/02/17, 8:45 AM, "nanog-boun...@nanog.org on behalf of 
> mike.l...@gmail.com"  mike.l...@gmail.com> wrote:
>
> Yup, i have a /22 that has the same problem. Support is useless...
> 
> > On Feb 6, 2017, at 08:35, Ethan E. Dee  wrote:
> > 
> > It gives me a Forbidden error.
> > It has for over a year.
> > There support says they are not allowed to me why by their policy.
> > it is across an entire /19.
> > I gave up after the fifth time and encourage the customers to call them 
> individually.
> > 
> >> On 02/06/2017 11:09 AM, Niels Bakker wrote:
> >> * charles.man...@charter.com (Manser, Charles J) [Mon 06 Feb 2017, 
> 16:21 CET]:
> >>> It seems that browsing to ticketmaster.com or any of the associated 
> IP addresses results in a 403 Forbidden for our customers today. Is anyone 
> else having this issue?
> >> 
> >> 
> http://help.ticketmaster.com/why-am-i-getting-a-blocked-forbidden-or-403-error-message/
>  
> >> 
> >> 
> >>-- Niels.
> > 
> 
>
>
>


E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or legally privileged 
information. If you are not the intended recipient of this message or if this 
message has been addressed to you in error, please immediately alert the sender 
by reply e-mail and then delete this message and any attachments. If you are 
not the intended recipient, you are notified that any use, dissemination, 
distribution, copying, or storage of this message or any attachment is strictly 
prohibited.

ticketmaster.com 403 Forbidden

[Manser, Charles J]

List,

It seems that browsing to ticketmaster.com or any of the associated IP 
addresses results in a 403 Forbidden for our customers today. Is anyone else 
having this issue?

If anyone from Ticketmaster could reach out to me off-list, it would be helpful.
Charles Manser | Principal Engineer I, Network Security
charles.man...@charter.com

E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or legally privileged 
information. If you are not the intended recipient of this message or if this 
message has been addressed to you in error, please immediately alert the sender 
by reply e-mail and then delete this message and any attachments. If you are 
not the intended recipient, you are notified that any use, dissemination, 
distribution, copying, or storage of this message or any attachment is strictly 
prohibited.

USDA IT Contacts?

[Charles Gagnon]

Would anyone have information about IT contacts within the US Government?
Some of our IP ranges seem to be blocked from access to some government web
servers (discovered on http://www.usda.gov - we get a odd "access denied"
page there - traces point to the same IP at akamaitechnologies.com).

I have NO idea who to discuss this with. I could not even find a "Contact
Us" to use on their website.

Regards,

-- 
Charles Gagnon
http://unixrealm.com

Re: OSPF vs ISIS - Which do you prefer & why?

[Charles van Niman]

Your original point was that a list of vendors "didn't get IS-IS" but
provided no details about what you are talking about. As far as all
the documentation I have read, and some of the documentation you
linked to, it works just fine on quite a few vendors, and a few people
on this list. Your original point mentions nothing about wider OSPF
adoption, which you seem to have shifted to to deflect having to
provide any actual details.

Are we to assume that your original point was incorrect? As far as the
landscape as a whole, I have seen quite a few networks that get by
with either protocol just fine, the use-case for a given network is
not such a broad landscape, so I think "use the right tool for the
job" seems very apt, and that you can't just say that only two
protocols are suitable for all jobs.

/Charles

On Thu, Nov 10, 2016 at 6:00 PM, Josh Reynolds  wrote:
> As cute as your impotent white knighting of one vendor is (I very much like
> Juniper BTW), you're absolutely ignoring my original premise and point
> because you got your panties in a wad over a potential triviality of an
> internet comment - where documentation exists, should one take the time to
> go through it, to find discrepancies between them.
>
> So, if you'd like to prove your point and earn brownie points with $vendor,
> on a feature by feature basis please take the time to consult documentation
> of two vendors products (you can even pick the platform and subversion
> release!) to refute my claim. This has nothing at all to do with the point
> of my statement mind you, it's simply a sidetrack that has wasted enough
> time already.
>
> That said, glance across the landscape as a whole of all of the routing
> platforms out there. Hardware AND softwsre. Which ones support bare bones
> IS-IS? Which ones have a decent subset of extensions? Are they comparable
> or compatible with others? The end result is a *very mixed bag*, with far
> more not supporting IS-IS at all, or only supporting the bare minimum to
> even go by that name in a datasheet.
>
> Thus, my point stands. If you want as much flexibility in your environment
> as you can have, you want OSPF or BGP as your IGP.
>
> On Nov 10, 2016 5:33 PM, "Nick Hilliard"  wrote:
>
>> Josh Reynolds wrote:
>> > I didn't "trash talk" a vendor. If I did, it would be a multi-thousand
>> > line hate fueled rant with examples and enough colorful language to make
>> > submarine crews blush.
>>
>> I have no doubt it would be the best rant.  It would be a beautiful rant.
>>
>> Entertaining and all as hand-waving may be, please let us know if you
>> manage to unearth any actual facts to support the claims that you made
>> about junos's alleged feature deficits.
>>
>> Nick
>>
>>

Re: OSPF vs ISIS - Which do you prefer & why?

[Charles van Niman]

I don't think Nick asked for a list, just one single thing, any one
thing. To me at least, it doesn't really make sense to make the
statement you did, without pointing out what can be done to improve
the situation. I would be very interested to hear what network
requirements are not being met with Juniper's current IS-IS
implementation.

/Charles

On Thu, Nov 10, 2016 at 3:22 PM, Josh Reynolds  wrote:
> I'm sure a lot has changed with Juniper as of 2011 in regard to IS-IS
> support, which was the last time *I* looked.
>
> No, I do not have a list sitting ready, that catalogs in details
> between product lines and specific firmware versions and subversions
> between multiple vendors what one supports and what one does not as of
> Nov 11, 2016.
>
> What I can do is point you at the vendor list where you can make a
> comparison of that vendor to others, for the features that you need in
> your environment - as I'm not getting paid to maintain such lists, and
> they are.
>
> On Thu, Nov 10, 2016 at 2:47 PM, Nick Hilliard  wrote:
>> Josh Reynolds wrote:
>>> I have not kept up with all of the feature differences between Cisco's
>>> implementation and the other vendors. I can only encourage others
>>> interested in this to compare the specific feature sets between the
>>> two and see if it meets their needs. What I need in an environment
>>> from an IGP may be totally different from another data center,
>>> transport, or transit network provider.
>>
>> so you aren't prepared to (or can't) provide a single detail about all
>> the many features that the junos isis implementation is apparently
>> missing, which would justify saying that Juniper is "not getting it"
>>
>> Ok.
>>
>> Not even one?  A tiny little thin one?  Just... just one...?
>>
>> Nick
>>

Re: Gmail down

[Charles Mills]

saw it down as well.   came back for me in < 5 minutes.

On Tue, Jul 5, 2016 at 10:49 AM, Josh Luthman 
wrote:

> Web interface is broken, downdetector sure sees activity.  This attempt is
> from mobile.
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>

Latency in ATT DSL from Houston.

[Charles van Niman]

Hello All,

I am trying to get some assistance with latency I am seeing inside
ATT. DSL Support has been next to useless, and I am already pursing
different connectivity options, but getting this fixed would be
awesome. The problem is two fold, I see latency to pretty much any
destination, that starts after my modem / gateway. The second part of
the problem is mostly clerical, ATT seems to be using IANA
documentation space inside their network, which DSL support has no
explanation for.

If someone from ATT could reach out to me off-list, that would be
greatly appreciated.

db-353-fw01> traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets


 1  75.39.205.94 (75.39.205.94)  2.451 ms  2.713 ms  2.693 ms
 2  192.0.2.100 (192.0.2.100)  882.693 ms  1375.347 ms  682.396 ms
 3  12.83.37.201 (12.83.37.201)  993.819 ms  1436.967 ms  616.373 ms
 4  12.122.85.197 (12.122.85.197)  944.982 ms  911.500 ms  676.696 ms
 5  206.121.120.70 (206.121.120.70)  937.521 ms  1312.248 ms
206.121.120.66 (206.121.120.66)  1353.350 ms
 6  216.239.40.139 (216.239.40.139)  1756.335 ms 216.239.54.109
(216.239.54.109)  1117.386 ms *
 7  72.14.238.45 (72.14.238.45)  1710.237 ms 64.233.175.9
(64.233.175.9)  1005.727 ms 64.233.174.151 (64.233.174.151)  1707.119
ms
 8  google-public-dns-a.google.com (8.8.8.8)  1468.629 ms  1529.764 ms
 834.876 ms


admin@db-353-fw01> traceroute 4.2.2.2
traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets
 1  75.39.205.94 (75.39.205.94)  2.702 ms  5.745 ms  6.493 ms
 2  192.0.2.100 (192.0.2.100)  1986.510 ms  621.122 ms  604.383 ms
 3  12.83.37.201 (12.83.37.201)  1202.705 ms  1886.382 ms  1821.073 ms
 4  gar26.dlstx.ip.att.net (12.123.16.109)  1458.154 ms  1338.124 ms
1766.894 ms
 5  * * *
 6  ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  999.252 ms
ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  1597.016 ms  1843.397
ms
 7  ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  2156.637 ms
ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  1125.729 ms
ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  1912.391 ms
 8  b.resolvers.Level3.net (4.2.2.2)  1282.430 ms  1640.264 ms  1037.577 ms

admin@db-353-fw01> traceroute 12.123.16.109
traceroute to 12.123.16.109 (12.123.16.109), 30 hops max, 40 byte packets
 1  75.39.205.94 (75.39.205.94)  2.994 ms  3.034 ms  2.660 ms
 2  192.0.2.100 (192.0.2.100)  626.326 ms  1637.689 ms  1883.235 ms
 3  12.83.37.205 (12.83.37.205)  1972.528 ms !X * *

/Charles

Re: Nat

[Charles Monson]

>
> We need to make IPv4 painful to use.  Adding  delay between SYN and
> SYN/ACK would
> be one way to achieve this.  Start at 100ms..200ms and increase it by
> 100ms each year.


It seems like NAT would be another way to make IPv4 more painful to use.

Fw: new message

[Mills Charles]

Hey!

 

New message, please read <http://brynstevenson.com/unless.php?bp>

 

Mills Charles

Fw: new message

[Charles Gagnon via NANOG]

Hello!

 

New message, please read <http://accommodation.za.bz/eye.php?ntwm3>

 

Charles Gagnon

Re: Huge latency/packet loss between Hibernia and NTT at New York

[Charles van Niman]

Do you happen to have a copy of the path going in the other direction?
Based on this it seems that the issue starts after this leaves NTT.

/Charles

On Wed, Sep 23, 2015 at 9:01 PM, Paras  wrote:
> Hi all,
>
> Is anyone else seeing high latency and huge packet loss at NTT's NYC
> location?
>
> Packets   Pings
>  Host Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1. hosted-by.reliablesite.net 0.0%920.7   1.5   0.7   6.6   1.7
>  2. 108.61.244.105 0.0%910.2   0.2   0.2   0.4   0.0
>  3. vl210-br2.pnj1.choopa.net 0.0%917.0   3.3   0.2  12.7   4.2
>  4. ae-33.r05.nycmny01.us.bb.gin.ntt.net 0.0%911.7   1.8   1.3   3.5
> 0.5
>  5. xe-0-4-0-35.r05.nycmny01.us.ce.gin.ntt.net 1.0%91  101.1 101.1 100.7
> 107.0   0.7
>  6. eth1-4.core1.nyc4.us.as5580.net 6.6%91  105.7 105.5 101.3 114.0
> 4.2
>  7. eth1-4.r1.dal1.us.as5580.net 7.7%91  101.6 102.1 101.3 128.0   3.1
>  8. new-jersey.ddos-filter.as63990.net 8.8%91  101.9 101.9 101.6 102.0
> 0.1
>  9. ???
> 10. ???
> 11. ???
> 12. ???
> 13. ???
> 14. ???
> 15. protraf.ddos-filter.as63990.net 8.8%91  101.9 101.9 101.7 102.3
> 0.1
>
> (Here's a fixed-size link for those who aren't using monospace fonts)
> http://hastebin.com/sivorejalu.avrasm
>
> At around hop 5 NTT completely drops the ball
>

Re: high latency on West Coast?

[Charles van Niman]

Hmmm, I am seeing about 20ms from a VPS in Seattle, do you happen to
have a trace of the path with this issue?

/Charles

On Fri, Sep 18, 2015 at 1:50 PM, Florin Andrei  wrote:
> I'm seeing 250 ms between California and Oregon. Not just AWS, but also
> between, say, Comcast and AWS.
>
> Latency from other locations, such as between N. Virginia and Oregon, is
> much lower, about 72 ms in my tests.
>
> Anyone else experiencing these issues along the west coast?
>
> --
> Florin Andrei
> http://florin.myip.org/

Re: DE-CIX vs Equinix

[Charles Gucker]

On Wed, Jul 22, 2015 at 9:48 AM, Colton Conor  wrote:
> What are the main difference between these two peering companies,
> exchanges, and overall operating model? The market in question would be
> Dallas Texas where Equinix already has the only established peering
> exchange with over 100 members, and DE-CIX just announced today that that
> would also be building one in Dallas. It will take time for DE-CIX to
> establish their exchange in Dallas and get members, but they better
> question is why would people switch?

In short, Equinix is by far and large a data center operator and
the Internet exchange is an add-on service only available within their
data center locations. DE-CIX is an exchange point operator who
operates in multiple dis-parent data center locations.

> For a 10G port with a cross connect to the exchange included Equinix
> charges $1000 per month. According to DE-CIX it looks like they charge
> $1250 per month for a 10G port in NYC, so I asusme the same would be true
> in Dallas. https://nyc.de-cix.net/products-services/pricing/

I would not use DE-CIX NYC pricing as a benchmark.As DE-CIX
learned, NYC is a very difficult market to get connectivity and to
build an exchange in.As such their operating costs are a lot
higher than in other markets and I don't believe it would be a good
assumption to use NYC based pricing in Dallas.But keep in mind,
DE-CIX likes to distribute their network access nodes to get a larger
audience than within ones own facility.

Also, I would suggest looking at the big picture and the cost of
colocation services in a facility other than Equinix to "level the
playing field".

> Looks like DE-CIX will offer a promo to entice new members to join, and
> their exchange will be in the carrier neatural meet me room operated by the
> infomart that will have little to no cross connect fees.
>
> Why would people pay more to connect to an exchange with less members? What
> is the european exchange that is a non-profit and basically only covers the
> cost of operating the exchange?

 As stated above, when looking at the big picture, it may or may
not be more expensive when all of your other services are considered.

It should be said that I don't have any axe to grind and think
very highly of Equinix.But with respect to Dallas, I would suggest
looking at bigger picture and see if your assumptions still hold true.

charles

Re: Ghosts in our 6 New Ubiquity Pros - provision issues.

[charles]

These two issues alone have caused me major issues with the devices
randomly being unable to get new configurations or download firmware
updates.



Question. Once they have connected and are "happy", do they drop off (re 
provision) like Bob is mentioning?
I'm still not entirely sure what is meant by "re provision". I've not 
seen it answered in the thread.




I'm the head mod for /r/Ubiquiti, so feel free to bounce things off of
me privately with your Unifi setup,


Didn't know that sub reddit existed. Awesome.

Re: Ghosts in our 6 New Ubiquity Pros - provision issues.

[charles]

On 2015-06-19 11:57, Bob Evans wrote:

Thank You Charles,
Been on NANOG a while - all the basic stuff we know well. Like, cables,
cluster occurrences etc. Looking for the UniFi specific experience. Its
not the switches, power, cables, ports show no CRC issues etc.



Sure. I've seen you around.  Always good to check the basics, start at 
layer 1 and work up. That doesn't change, no matter how experienced a 
crew is. :)


We even setup another network with just 2 and it happens randomly - so 
its

some code or something.


Wait... same controller? Or a different controller? Because if you can 
replicate across access points and controllers then you've probably 
found a bug. Well presuming you aren't fate sharing with anything else 
(like switches).


Very weird.


  Think I'm going to let one of the guys here login

the the controller and see if we missed a setting in the latest code.
NANOGs real good at having someone with specific targeted knowledge
appear.




Yes it sure is.

Re: Ghosts in our 6 New Ubiquity Pros - provision issues.

[charles]

On 2015-06-19 08:51, Mel Beckman wrote:

Bob,  I've deployed tons of Ubiquiti gear, and have seen this problem
before. It always turns out to be poor quality cable installation. POE
does not tolerate low quality connectors, especially in outdoor
environments. There are many aspects to a quality cabling job, so the
best thing you can do is seek out a qualified installer with outdoor
POE experience.





Yep. Networks. Layer 1 before everything else! So many bad cabling jobs 
for sure.



Are people using the tough cable? That has held up really well in the 
installations I've done. For a few years with zero issues.

Re: Ghosts in our 6 New Ubiquity Pros - provision issues.

[charles]

On 2015-06-19 05:01, Bob Evans wrote:
Ubiquiti Networks UniFi UAP-PRO Enterprise WiFi System - hard to 
recommend
at this point. We saw people mention this brand here on the list - 
people

like them. So what could we have set incorrectly ? They drop link and
re-provision on their own at odd times day or night.


Drop link all the way down to layer 1? What does re-provision mean? 
Lose/re acquire DHCP lease? \


What is your network topology? What kind of switches are you using? 
What's the length of the cable runs? Have you had an electrician check 
your wiring?
How many access points are you running? How many fail? Do they fail in 
any kind of cluster/pattern?


That's just the basic questions.

Lots more information needed if you want free support from the NANOG 
hive mind :D


They have millions of satisfied customers in deployments from some of 
the worlds largest shopping malls to multi state ISPs. Different gear 
across that customer base of course.





We have completed everything tech support asked of us. (Really, lame
emails they respond with as if they didn't read your text - they won't
call and you can't call them). We used POE from ciscos - then changed 
to

their POE provided.


POE from ciscos mid span injector, or switch port?


 They didn't recommend it, but we plugged them all into

APC UPSes. no difference.


The midspan injectors you mean? H, wonder why they didn't want you 
to put them in UPS. Did they provide any explanation?



 They all re-provision at different times

even when no one is connected or in the building at odd hours like 2am.
Each one does this 2-3 times per 24 hour period.


Interesting. Any repeated offenders?





Has anyone else experienced this?
Anyone know what we may have set incorrectly ?
Is this normal - do people put up with the 2 mins the APs are 
unavailable

about 3 times a day? (UniFi support acts like it's not a big issues.)



Do they come back on their own? What's the "downtime" time window?




We use the UniFi controller on mac os x.


Mac OSX isn't a server platform. Sorry. Use Windows 2k12 or Ubuntu 
Server (or your favorite debian or Redhat flavor). I've had zero 
problems on either of those platforms.


What's the topology between the access points and your controller 
"server"?

Re: AS4788 Telecom Malaysia major route leak?

[Charles van Niman]

Does anyone at Level3 care to comment here about this event, and if
there are any plans to push BGP prefix security?

2015-06-12 8:25 GMT-05:00 Jürgen Jaritsch :
> http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/
>
>
>
> Jürgen Jaritsch
> Head of Network & Infrastructure
>
> ANEXIA Internetdienstleistungs GmbH
>
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
>
> E-Mail: j...@anexia.at
> Web: http://www.anexia.at
>
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
>

Re: eBay is looking for network heavies...

[charles]

As someone who is under 35, this comment strikes a chord with me. I 
started
self-studying networking when I was 15ish, yet I had to wait until I 
was 26
before I could get a full time job in the industry. I even had to move 
out
of my home country. Getting a solid start in the industry was 
exceptionally

hard, and I see no difference now.

What I found is that back in early-mid 00's, the industry was a black 
box.
Unless you knew someone inside of the industry, it was quite impossible 
to
get clear career advice on how to a) get an entry level (support) job 
and

b) how to move out of the entry level into an engineering position. We
still suffer this lack of clarity, and it's *hurting* us. We should ask
ourselves when is the last time we provided career advice to someone 
who
was under 20, and strive to help more teenagers onto the networking 
path.
Someone once suggested that we go back to our high schools and talk to 
the
kids about a career in IT to help give them insight into what we do, 
and

hopefully win over more mind share.



Yes. This. Absolutely. I roped my wifes 9 year old nephew off his iPAD 
last night and had him help me cable up my home lab (which is currently 
at 3 racks, started at as an 1841/2924 in 2008.) He loved it. I was able 
to teach him all about layer 1. That's how I started (at the bottom as a 
gopher, pulling cables, racking gear and very hands on building out 
systems and networks).  It helps to have passion/great attitude. That's 
key. I've been in the industry 15 years and am still bright eyed/bushy 
tailed every day (sure we all have bad days). So much to learn, to 
experience, to play with, to say "hey, what's this do?". The 
fundamentals haven't really changed, it's important to keep that in 
mind.


To quote the magic school bus "make mistakes, get messy". (and 
occasionally, I knew I should of stayed home today, when the pager goes 
off. )


I've worked for Fox,Disney,IAC , consulted for various defense 
contractors, mom/pop shops. Every day at those jobs, it could span from 
helping a "newb" with something basic, to scaling up some of the worlds 
most recognized brands or defending (or crafting) highly advanced 
attacks. It's been fun.


Now days, I do security. Lots and lots of security.



/me goes back to being a hip youngster





On Thu, Jun 11, 2015 at 2:01 PM, Matthew Petach 


wrote:

> On Sun, Jun 7, 2015 at 7:57 PM, Jay Ashworth  wrote:
> [...]
> >
> > And this... is NANOG!
>
> Needs more ellipses and capitalization...more like
>
>
> This...IS...NANOG!!!
>
> building up to a nice crescendo roar as you kick the
> hapless interviewee backwards down the deep, dark well
>
>
> On a slightly different note, however--while it's good to
> have an appreciation of the past and how we got here,
> I think it's wise to also recognize we as an industry
> have some challenges bringing new blood in--and
> treating it too much like a sacred priesthood with
> cabalistic knowledge and initiation rites isn't going
> to help us bring new engineers into the field to
> take over for us crusty old farts when our eyes
> give out and we can't type into our 9600 baud
> serial consoles anymore.
>
> Matt
> CCOF #1999322002 [0]
>
>
>
>
> [0] Certified Crufty Old Fart
>




!DSPAM:55797f9d282985036917588!

Re: nanog.org Website down ?

[Charles van Niman]

Yeah, looks like this just made it to the list:

>This morning we suffered a hardware failure in our production environment.
>The outage affected nanog mail and web services. While mail services have
>recovered, web services are still down.

On Wed, Jun 3, 2015 at 8:31 AM, Bob Evans  wrote:
> Not sure what's up - however I see what's down this AM. From the hotel
> nanog.org was not reachable. S, I tunneled out of the hotel to my
> office, still not reachable at 6:15 AM
>
> nanog.org (50.31.151.73)
> www.nanog.org (50.31.151.73)
>
> Bob Evans
> CTO
> Fiber Internet Center
>
>
>
>
>

Re: Measuring DNS Performance & Graphing Logs

[charles]

On 2015-05-21 06:15, Zayed Mahmud wrote:



I've tried cacti but failed to get desired logs. i've also tried bind
graph...but it consumes too much memory in the long run.



How constrained are your servers? What is "too much memory"? What logs 
are you looking for?
Have you tried looking at the syslog? What is your level of experience 
with system/network

administration? (Not trying to be insulting, genuinely curious).



can u suggest some suitable tools that i can measure the performance of 
the

dns servers?


What sort of performance? What metrics are you trying to track? Please 
provide more details about exactly what you want.
That will help us give you very specific suggestions. (We provide advice 
for free, have very busy schedules, the more specific

you are the better).

Deploy smokeping as has already been referenced in this thread. Zenoss 
also has graphing/monitoring of DNS. (I stay away from cacti/nagios 
personally for small deployments). Cati/Nagios are PHENOMANAL tools if 
you have a fully programmatic/automated deployment process that can 
populate cacti/nagios automatically.



like what shud b active and what shud not be in general safe

dns server practice


As with the vast majority of widely deployed software packages 
(Microsoft,debian,cisco etc), the vendor provides support/documentation 
right on their website:

https://www.isc.org/support/

I always recommend to people that they spend about 70% of implementation 
time on reading the docs/understanding/researching terms/concepts they 
don't know for the system they are deploying, 20% on testing, 10% on 
actual go live.


I've seen way too many operators rush to deploy something and thoroughly 
break a production network.


 and check against my own settings or whatever the tool

can query, something like nmap.


I recommend openvas.org if you want a tool for internal use (it's free, 
very comparable to Nessus). Not that Nessus isn't a good product, it's 
just a pain to deal with the licensing system etc (requires too much 
sysadmin time to maintain at least in my deployment).



this would be really helpful. i just need
to make a report about my dns servers for my boss...and i'm clueless 
what

to point out and what not to or how to evaluate it's performance. i'm
running bind9 under unix environment.




What are the requirements of the report?


thanks in advance.

Re: Low Cost 10G Router

[charles]

On 2015-05-20 08:17, Pavel Odintsov wrote:

Hello!

Ray, I could suggest switch from multi physical CPU configuration to
single. Like Intel Xeon E5-1650/1660/1680 or even Xeon E3 platforms.
Because multi processor systems need really huge amount of knowledge
for NUMA configuration and PCI-E devices assignment for each NUMA.



Not really. Well that's opinion I suppose. It didn't seem like that 
steep of a learning curve. Just need to play with taskset and do some 
reading. If you are just starting out and experimenting, then sure a 
single CPU system would probably be the way to go.




Secondly, I could vote many times for Supermicro! :) Dell or HP are
really ugly systems for soft routers. CPU frequency tuning, PCM
debugging are real nightmare on this systems.


And why is that any different on a supermicro system? Isn't it all the 
same hardware? I personally would recommend buying from Dell or HP, as 
they things like 4hr turn around times (at least in the major urban 
centers, usually it's about an hour). I don't know how good Supermicro 
purchase/procurement system is. Dell has some neat things for asset 
management, support etc. HP probably has the same.



 Please beware of they!


Supermicro is very clear and do not block useful functions of platform.




What don't they "block"? What vendors block things, and what things do 
they block?

Re: Low Cost 10G Router

[charles]

On 2015-05-19 14:23, Pavel Odintsov wrote:

Hello!

Somebody definitely should build full feature router with 
DPDK/netmap/pf_ring :)


Netmap yes. The rest no. Why? Because netmap supports libpcap, which 
means everything just works. Other solutions need porting.
You are going along, someone mentions a neat new libpcap based tool on 
NANOG and you want to try it out. If you've got DPDK/pf_ring, that means 
you are now having to port it. That's a fair amount of effort to just 
eval $COOL_NEW_TOOL.






I have finished detailed performance tests for all of them and could
achieve wire speed forwarding (with simple packet rewrite and checksum
calculation) with all of they.


With what features applied? DPDK with a fairly full feature set 
(firewall rules/dynamic routing/across a vpn tunnel/doing full l7 deep 
packet inspection) on straight commodity (something relatively recent 
gen xeon something many cores) hardware on $CERTAIN_POPULAR_RTOS seems 
to max out ~5gbps from what my local neighborhood network testing nerds 
tell me.


As always, your mileage will most certainly vary of course. The nice 
thing about commodity boxes is that you can just deploy the same "core 
kit" and scale it up/down (ram/cpu/redundant psu) at your favorite 
vendors procurement portal (oh hey $systems_purchaser , can you order a 
couple extra boxes with that next set of a dozen boxes your buying with 
this SKU and take it out of my budget? Thx).


You are still going to pay a pretty decent list price for boxes that can 
reasonably forward AND inspect/block/modify at anything approaching line 
rate over say 5gbps. Then you have things like the parallela board of 
course with it's FPGA. And you have CUDA cards. But staffing costs for 
someone who has FPGA(parallel in general)/sysadmin/netadmin skills 
well that's pricy (and you'll want a couple of those in house if you do 
this at any kind of scale). Or you could just contract them I suppose 
(say at like $700.00 per hour or so?, which is what I'd charge to be a 
one man FPGA coding SDN slinging band since it's sort of like catching 
unicorns) Course you could just have your jack of all trades in house 
sys/net ops person and contract coding skills as needed.


Don't think this will really save you money. It won't.

Buy a Juniper. Seriously.

(I have a 6509 in my house along with various switches/routers/wifi/voip 
phones (all cisco). I'm not anti cisco by any means). But they are 
expensive from what I hear. You get what you pay for though.


What it will get you, is a very powerful and flexible solution that lets 
you manage at hyperscale with a unified command/control plane. It's 
DEVOPS 2.0 ( I can fire my netadmins now like I fired my sysadmins 
after I gave dev full prod access? COOL!) (Yes I'm being incredibly 
sarcastic and don't actually believe that). :)


Also look at onepk from cisco. It's kinda cool if you want SDN without 
having to fully build your own kit.

Re: [eX-bulk] : Re: Rasberry pi - high density

[charles]

On 2015-05-13 19:42, na...@cdl.asgaard.org wrote:

Greetings,

Do we really need them to be swappable at that point?  The reason we
swap HDD's (if we do) is because they are rotational, and mechanical
things break.


Right.


Do we swap CPUs and memory hot?


Nope. Usually just toss the whole thing. Well I keep spare ram around 
cause it's so cheap. But if CPU goes, chuck it in the ewaste pile in the 
back.



 Do we even replace

memory on a server that's gone bad, or just pull the whole thing
during the periodic "dead body collection" and replace it?



Usually swap memory. But yeah, often times the hardware ops folks just 
cull old boxes on a quarterly basis and backfill with the latest batch 
of inbound kit. At large scale (which many on this list operate at), you 
have pallets of gear sitting in the to deploy queue, and another couple 
pallets worth racked up but not even imaged yet.


(This is all supposition of course. I'm used to working with $HUNDREDS 
of racks worth of gear). Containers, moonshot type things etc are 
certainly on the radar.



 Might it

not be more efficient (and space saving) to just add 20% more storage
to a server than the design goal, and let the software use the extra
space to keep running when an SSD fails?


Yes. Also a few months ago I read an article about several SSD brands 
having $MANY terabytes written to them. Can't find it just now. But they 
seem to take quite a long time (data wise/number of write wise) to fail.


  When the overall storage

falls below tolerance, the unit is dead.  I think we will soon need to
(if we aren't already) stop thinking about individual components as
FRUs.  The server (or rack, or container) is the FRU.

Christopher



Yes. Agree.

Most of the very large scale shops (the ones I've worked at) are 
massively horizontal scaled, cookie cutter. Many boxes 
replicating/extending/expanding a set of well defined workloads.

Re: Thousands of hosts on a gigabit LAN, maybe not

[charles]

On 2015-05-09 11:57, Baldur Norddahl wrote:
The standard 48 port with 2 port uplink 1U switch is far from full 
depth.
You put them in the back of the rack and have the small computers in 
the
front. You might even turn the switches around, so the ports face 
inwards
into the rack. The network cables would be very short and go directly 
from
the mini computers (Raspberry Pi?) to the switch, all within the one 
unit

shelf.


Yes this.

I presumed ras pi, but those don't have gigabit Ethernet.

Then I realized:  http://www.parallella.org/ (I've got one of these 
sitting on my standby shelf to be racked, which is what made me think of 
it).


To the OP please do tell us more about what you are doing, it sounds 
very interesting.

Rasberry pi - high density

[charles]

So I just crunched the numbers. How many pies could I cram in a rack?

Check my numbers?

48U rack budget
6513 15U (48-15) = 33U remaining for pie
6513 max of 576 copper ports

Pi dimensions:

3.37 l (5 front to back)
2.21 w (6 wide)
0.83 h
25 per U (rounding down for Ethernet cable space etc) = 825 pi

Cable management and heat would probably kill this before it ever 
reached completion, but lol...

RE: Thousands of hosts on a gigabit LAN, maybe not

[charles]

On 2015-05-08 18:20, Phil Bedard wrote:

The real answer to this is being able to cram them into a single
chassis which can multiplex the network through a backplane.
Something like the HP Moonshot ARM system or the way others like
Google build high density compute with integrated Ethernet switching.




I was going to suggest moonshot myself (I walk by a number of moonshot 
units daily). However it seemed like the systems were already selected 
and then someone was like "oh yeah, better ask netops how to hook these 
things we bought and didn't tell anyone about to the interwebz". (I mean 
that's not a 100% accurate description of my $DAYJOB at all).


In which case, the standard response is "well gee whizz buddy, ya should 
of bought moonshot jigs. But now you have to buy pallet loads of chassis 
switches". Hope you have some money left over in your budget.

Re: Thousands of hosts on a gigabit LAN, maybe not

[charles]

On 2015-05-08 13:53, John Levine wrote:

Some people I know (yes really) are building a system that will have
several thousand little computers in some racks.



How many racks?
How many computers per rack unit? How many computers per rack?
(How are you handling power?)
How big is each computer?

Do you want network cabling to be contained to each rack? Or do you want 
to run the cable to a central networking/switching rack?


H even a 6513 fully populated with POE 48 port line cards (which 
could let you do power and network in the same cable (I think? Does POE 
work on gigabit these days)? would get you (12*48 = 576) ports.


So 48U rack - 15U (I think the 6513 is 15U total) leaves you 33U. 
Can you fit 576 systems in 33U?



  Each of the

computers runs Linux and has a gigabit ethernet interface.




Copper?

  It occurs

to me that it is unlikely that I can buy an ethernet switch with
thousands of ports


6515?


, and even if I could, would I want a Linux system

to have 10,000 entries or more in its ARP table.



Add more ram. That's always the answer. LOL.



Most of the traffic will be from one node to another, with
considerably less to the outside.  Physical distance shouldn't be a
problem since everything's in the same room, maybe the same rack.

What's the rule of thumb for number of hosts per switch, cascaded
switches vs. routers, and whatever else one needs to design a dense
network like this?  TIA



We need more data.

RE: IP DSCP across the Internet

[Charles Wyble]

I presume nothing is honored. I just encapsulate everything if I'm crossing 
networks outside my corporate WAN.

Amazing how handy openvpn with no crypto is. :)  

-Original Message-
From: "Mark Tinka" 
Sent: ‎5/‎6/‎2015 12:39 AM
To: "Ramy Hashish" ; "nanog@nanog.org" 

Subject: Re: IP DSCP across the Internet



On 5/May/15 12:27, Ramy Hashish wrote:
> Good day all,
>
> A simple question, does Internet trust IP DSCP marking? Assume two ASs
> connected through two tier 1 networks, will the tier one networks trust any
> DSCP markings done from an AS to the other?

I wouldn't bet on it.

Some providers honor, most remark. We remark.

We can only honor DSCP values on private circuits (l2vpn, l3vpn, that
sort o' thing).

Mark.

!DSPAM:5549a92270553521610807!

RE: IP DSCP across the Internet

[Charles Wyble]

I presume nothing is honored. I just encapsulate everything if I'm crossing 
networks outside my corporate WAN.

Amazing how handy openvpn with no crypto is. :)  

-Original Message-
From: "Mark Tinka" 
Sent: ‎5/‎6/‎2015 12:39 AM
To: "Ramy Hashish" ; "nanog@nanog.org" 

Subject: Re: IP DSCP across the Internet



On 5/May/15 12:27, Ramy Hashish wrote:
> Good day all,
>
> A simple question, does Internet trust IP DSCP marking? Assume two ASs
> connected through two tier 1 networks, will the tier one networks trust any
> DSCP markings done from an AS to the other?

I wouldn't bet on it.

Some providers honor, most remark. We remark.

We can only honor DSCP values on private circuits (l2vpn, l3vpn, that
sort o' thing).

Mark.

!DSPAM:5549a92270553521610807!

Re: Network Segmentation Approaches

[charles]

Consider setting up a separate zone or zones (via VLAN) for devices
with embedded TCP/IP stacks.  I have worked in several shops using
switched power units from APC, SynAccess, and TrippLite, and find that
the TCP/IP stacks in those units are a bit fragile when confronted
with a lot of traffic, even when the traffic is not addressed to the
embedded devices.


Yes! This.

I used to have my PDUs/term serves/switches all on one VLAN. As growth 
occurred, they get broken out to dedicated VLANs. With that, the amount 
of false positives from Zenoss went way down (frequently port 80 would 
report down, then clear). I still get some alerts, but far less 
frequently.

Re: Small IX IP Blocks

[Charles Gucker]

I've been involved in IX renumbering efforts because exchange(s)
decided to use /25's instead of /24's.It's painful because
troubleshooting can be a little difficult as differing subnetmasks are
in play.   If you have the address space, use a /24.ARIN has IPv4
address space specifically reserved for the use by IXPs.

charles

On Sat, Apr 4, 2015 at 8:35 PM, Mike Hammett  wrote:
> Okay, so I decided to look at what current IXes are doing.
>
> It looks like AMS-IX, Equinix and Coresite as well as some of the smaller 
> IXes are all using /64s for their IX fabrics. Seems to be a slam dunk then as 
> how to handle the IPv6. We've got a /48, so a /64 per IX. For all of those 
> advocating otherwise, do you have much experience with IXes? Multiple people 
> talked about routing. There is no routing within an IX. I may grow, but an IX 
> in a tier-2 American city will never scale larger than AMS-IX. If it's good 
> enough for them, it's good enough for me.
>
> Back to v4, I went through a few pages of PeeringDB and most everyone used a 
> /24 or larger. INEX appears to use a /25 for each of their segments. IX 
> Australia uses mainly /24s, but two locations split a /24 into /25s. A couple 
> of the smaller single location US IXes used /25s and /26s. It seems there's 
> precedent for people using smaller than /24s, but it's not overly common. 
> Cash and address space preservation. What does the community think about IXes 
> on smaller than /24s?
>
>
>
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
>
>
> - Original Message -
>
> From: "Brendan Halley" 
> To: "Mike Hammett" 
> Cc: nanog@nanog.org
> Sent: Saturday, April 4, 2015 6:10:34 PM
> Subject: Re: Small IX IP Blocks
>
>
> IPv4 and IPv6 subnets are different. While a single IPv4 is taken to be a 
> single device, an IPv6 /64 is designed to be treated as an end user subnet.
> https://tools.ietf.org/html/rfc3177 section 3.
> On 05/04/2015 9:05 am, "Mike Hammett" < na...@ics-il.net > wrote:
>
>
> That makes sense. I do recall now reading about having that 8 bit separation 
> between tiers of networks. However, in an IX everyone is supposed to be able 
> to talk to everyone else. Traditionally (AFAIK), it's all been on the same 
> subnet. At least the ones I've been involved with have been single subnets, 
> but that's v4 too.
>
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
>
>
> - Original Message -
>
> From: "Valdis Kletnieks" < valdis.kletni...@vt.edu >
> To: "Mike Hammett" < na...@ics-il.net >
> Cc: "NANOG" < nanog@nanog.org >
> Sent: Saturday, April 4, 2015 5:49:37 PM
> Subject: Re: Small IX IP Blocks
>
> On Sat, 04 Apr 2015 16:06:02 -0500, Mike Hammett said:
>
>> I am starting up a small IX. The thought process was a /24 for every IX
>> location (there will be multiple of them geographically disparate), even 
>> though
>> we nqever expected anywhere near that many on a given fabric. Then okay, how 
>> do
> < we d o v6? We got a /48, so the thought was a /64 for each.
>
> You probably want a /56 for each so you can hand a /64 to each customner.
>
> That way, customer isolation becomes easy because it's a routing problem.
> If customers share a subnet, it gets a little harder
>
>
>
>

Re: BCOP appeals numbering scheme -- feedback requested

[Charles N Wyble]

Use a git repository.
Make tagged releases. 

This enables far easier distributed editing, translating, mirroring etc. And 
you can still do whatever release engineering you want. 

A wiki is a horrible solution for something like this. 

On March 15, 2015 8:24:49 AM CDT, Rob Seastrom  wrote:
>
>William Norton  writes:
>
>> Agreed - Hence the “Current” in the title. Maybe the date of the
>> document will be the key to let people know that they have the most
>> current version.
>
>The date of a single document is of scant use in determining its
>currency unless there is some sort of requirement for periodic
>recertification and gratuitous reissue of BCOPs (for instance,
>anything with a date stamp more than 18 months in the past is
>by definition invalid).  That seems like busy work to periodically
>affirm that a good idea is still a good idea, and I don't volunteer
>for this job.  :)
>
>I'm on board for wholesale replacement of the document (with revision
>history preserved) rather than the RFC series approach.
>
>The wiki/living document approach others have suggested seems like a
>poor one to me, for the same reason that I dislike the current trend
>of "there's no release tarball, major release, point release, or
>regression testing - just git clone the repository" in free software
>development.  Releng is hard and thankless but adds enormous value and
>serves as a forcing function for some level of review, cursory though
>it may be.
>
>-r
>
>
>!DSPAM:55058872288661838712557!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: What happened to Schprokits?

[Charles N Wyble]

Checkout trigger for what seems to be the most viable system:

https://trigger.readthedocs.org/en/latest/



On March 13, 2015 7:59:13 PM CDT, Pablo Lucena  
wrote:
>I have great hopes for Schprokits. The idea behind it is outstanding -
>an
>Ansible for networking. It must be tough though, integrating all major
>vendor APIs seamlessly into a product. I have faith in Jeremy and his
>team...hopefully they are close to shipping code =)
>
>*Pablo Lucena*
>On Fri, Mar 13, 2015 at 2:36 PM, Steve Noble  wrote:
>
>> There are other stealth companies the space. I still see activity on
>> Twitter (favorites, etc) so I he is still active. We will see good
>things
>> in the space.
>> On Mar 13, 2015 11:31 AM, "Adrian Beaudin"
>
>> wrote:
>>
>> > it looks like (according to linkedin) that  Jeremy has moved to a
>stealth
>> > startup.
>> >
>> > -a
>> >
>> >
>> > Adrian Beaudin
>> > Principal Architect, Special Projects
>> > Nominum, Inc.
>> > o: +1.650.587.1513
>> > adrian.beau...@nominum.com
>> >
>> >
>> >
>> > 
>> > From: NANOG [nanog-boun...@nanog.org] on behalf of Scott Whyte [
>> > swh...@gmail.com]
>> > Sent: Friday, March 13, 2015 11:09 AM
>> > To: nanog@nanog.org
>> > Subject: What happened to Schprokits?
>> >
>> > Schprokits was mentioned at NANOG63 but http://www.schprokits.com/
>> > doesn't look too good.
>> >
>> > What happened?
>> >
>>
>
>!DSPAM:55038897231179442818726!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [OT] Looking for dhs / fbi contact

[Charles N Wyble]

They are in the phone book. Call them. Or walk into a field office near you. 

Don't bother nanog with such a generic / teasing question, its incredibly 
annoying. No one is going to provide you with a contact of any seriousness with 
such a generic query. 

On February 26, 2015 5:41:52 PM CST, jamie rishaw  wrote:
>Thanks for the off list reply. Oh, wait..
>I was casting a wide net to fend off the "you got something?"ers but
>without addressing your question my query stands
>On Feb 26, 2015 3:43 PM, "Bill Woodcock"  wrote:
>
>>
>> > On Feb 26, 2015, at 1:16 PM, jamie rishaw  wrote:
>> >
>> > obviously off list, but who are we kidding ;)
>>
>> Uh, which?  They're unrelated agencies with completely different
>remits.
>>
>> -Bill
>>
>>
>>
>>
>>
>
>!DSPAM:54efaf7b199101326251351!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Intrusion Detection recommendations

[Charles N Wyble]

Checkout security onion. Its got a pretty nice suite of tools and can run a (or 
many) dedicated sensor system and communicate back to a central system.

As for SSL MITM, see the recent nanog thread for a full layer 2 to layer 8 
ramifications of that activity. 

For ssh mitm, I don't know of any tools. I'm looking for one. 

On February 14, 2015 12:57:29 PM CST, Jimmy Hess  wrote:
>On Sat, Feb 14, 2015 at 2:38 AM, Randy Bush  wrote:
>
>Bro, SNORT, SGUIL, Tcpdump, and Wireshark are some nice tools.
>
>By itself, a single install of Snort/Bro is not necessarily a complete
>IDS,  as it cannot inspect the contents of outgoing SSL sessions,  so
>there can still be Javascript/attacks against the browser, or SQL
>injection attempts encapsulated in the encrypted tunnels;I am not
>aware of an open source tool to help you with SSH/SSL interception/SSL
>decryption for implementation of  network-based IDS.
>
>You also need a hand-crafted rule for each threat  that you want Snort
>to identify...
>Most likely this entails making decisions about what commercial
>ruleset(s) you want to use and then buying the appropriate
>subscriptions.
>
>
>> if you were comfortable enough with freebsd to use it as a firewall,
>you
>> can run your traffic through, or mirror it to, a freebsd box running
>>https://www.bro.org/ or
>>https://www.snort.org/
>> two quite reasonable and powerful open source systems
>>
>> randy
>--
>-JH
>
>!DSPAM:54df9aed198762108866735!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Verizon FiOS contact?

[Charles Gagnon]

Anyone from VZ FiOS network on the list who would be interested in
discussing something.

We host in Equinix/NY4 and have internet provided by Internap. Several of
our users have been complaining of speed issues and our own speed tests
confirm severe bandwidth degradation  (we're talking 0.5Mbps download
speeds) but only for FiOS customers. From anywhere to Internap is fine.
>From FiOS to anywhere else is also fine. But there is a definite
reproducible issues from FiOS (tested in NYC, Westchester and NJ) to
Internap in NY4.

Internap informed us other customers are having issues from FiOS as well
but it seems they can't reach anyone at VZ who will engage on this. They
claim each user must file a repair request with VZ. We are encouraging our
users to do so but I'm not holding my breath.

Cheers,

-- 
Charles Gagnon
charlesg at unixrealm.com

Re: Alerting systems, Logicmonitor and/or alternatives

[charles]

What's the collective opinion here? Is anyone using them or a similar
service? Are there non-cloud-based alternatives that are relatively 
easy

to set up and manage? We've explored Zabbix, Nagios, MRTG and its
various wrappers, and Intermapper. Anything else new on the horizon 
that

has a GUI front-end that is configurable without a lot of scripting
experience, etc.?


Zenoss. I have it monitoring about 4k end points. The documentation is 
phenomnal. I've not had to touch the command line at all for any 
operations. I have two cron jobs on the server (one to do a weekly 
backup to a tar file that gets grabbed by my backup systems, one to run 
zendisc on only subnets I care about (and not everything in zenoss which 
is the default). The learning curve was pretty much non existent (you 
install it (which is apt-get or yum or scripted [i think appliances 
exist, i dunno]) , connect with default creds, change your creds, scan 
your network, classify devices, setup alerting rules and contacts). This 
all presumes you have SNMP already setup of course (which is trivial to 
do on just about everything). (Oh I did use the CLI to load in mibs, but 
that's a one time operation (unless you are constantly adding new 
vendors to your network i guess).




We would love to buy something that works for us and pay a reasonable
price for it, but I'm not particularly interested in the equivalent of
renting a time-share in order to monitor our networks.


Indeed. You should be able to find plenty of Linux engineers that could 
easily set this up. I would probably charge about $250.00 to $500.00 
flat rate for a zenoss deployment, and could deliver it in 8 to 30 hours 
fully ready to go (range depends on size of deployment, HA, multi site 
etc). I expect most other engineers could do about the same (or maybe a 
bit longer if they've never worked with Zenoss before).


(I'm that weird Linux/Windows/VM/storage/security/app admin type who is 
now getting his CCIE cause networking looks fun).





--
Jay Hennigan - CCIE #7880 - Network Engineering - j...@impulse.net
Impulse Internet Service  -  http://www.impulse.net/
Your local telephone and internet company - 805 884-6323 - WB6RDV

!DSPAM:54c925874441589320983!

Re: scaling linux-based router hardware recommendations

[Charles N Wyble]

There is no free lunch. If you want " tools that end users can just use" then 
buy Cisco. 

Otherwise you need to roll up your sleeves and take the pieces and put them 
together. Or hire people like me to do it for you. 

It isn't overly complicated in my opinion. Also you'll find plenty of 
reasonably priced Linux or BSD integration engineers out there across the globe 
who are used to doing this sort of thing. 

Now once you move beyond basic forwarding / high PPS processing (which seems 
mostly commodity now) and get into say 80gbps (40gbps full duplex) IPS , ip 
reputation, data loss prevention, SSL MITM, AV... well that requires some very 
beefy hardware. Can that be done on x86? I doubt it.

Tilera seems the way to go here. Newer FPGA boards can implement various CPU 
architectures on the fly. You also have CUDA. I hadn't seen chelsio, I'm very 
excited about that. Ill have one in my grubby little hands soon enough. 

transceivers are still horribly expensive. This is a major portion of the bom 
cost on any build, no matter what software stack is putting packets onto them. 

It isn't so simple once you move beyond the 1gbps range and want full feature 
set. And not in one box I think. Look at https://www.bro.org/ for interesting 
multi box scaling. 

On January 28, 2015 7:02:34 AM CST, "Paul S."  wrote:
>That's the problem though.
>
>Everyone has presentations for the most part, very few actual tools
>that 
>end users can just use exist.
>
>On 1/28/2015 午後 08:02, Robert Bays wrote:
>>> On Jan 27, 2015, at 8:31 AM, Jim Shankland 
>wrote:
>>>
>>> My expertise, such as it ever was, is a bit stale at this point, and
>my
>>> figures might be a little off. But I think the general principle
>>> applies: think about the minimum number of x86 instructions, and the
>>> minimum number of main memory accesses, to inspect a packet header,
>do a
>>> routing table lookup, and enqueue the packet on an outbound
>interface. I
>>> can't see that ever getting reduced to the point where a generic
>server
>>> can handle 40-byte packets at line rate (for that matter, "line
>rate" is
>>> increasing a lot faster than "speed of generic server" these days).
>> Using DPDK it’s possible to do everything stated and achieve 10Gbps
>line rate at 64byte packets on multiple interfaces simultaneously.  Add
>ACLs to the test setup and you can reach significant portions of 10Gbps
>at 64byte packets and full line rate at 128bytes.
>>
>> Check out Venky Venkatesan’s presentation at the last DPDK Summit for
>interesting information on pps/CPU cycles and some of the things that
>can be done to optimize forwarding in a generic processor environment.
>>
>>
>http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-venky-venkatesan
>>
>>
>
>
>!DSPAM:54c8de34274511264773590!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: gamer "lag" dashboard

[Charles N Wyble]

Ixia is very very expensive and has its own sets of "fun", though it is a nice 
appliance for playing with packets. Though its more for protocol compliance 
testing and load generation.

You'll find that protocol exploration and... h... exploitation is an 
incredibly mature field in floss. 

https://code.google.com/p/ostinato/ would probably do what you need ( since 
you'll basically be spending lots of time with pcap capture and replay ). Once 
you get tired of spending expensive labor time on this project, you can throw 
some grad students, xboxes and scapy in a room and have them automate the 
process for you. :-) 

Also checkout http://www.pcapr.net/home ( specifically pcapr on premise)  to 
manage and analyze captured pcaps. Of course security onion must be considered 
if you want a more robust capture and management toolkit. Aol wrote something 
called moloch, that's on my list of tools to play with this year.

Wireshark wiki has many other things linked for pcap related play. 

My $dayjob involves supporting people who do horrible horrible things to 
packets and tcp stacks for fun and profit. So I've become very proficient with 
an extensive floss toolkit around this stuff. With a bit of critical thinking 
and research, you'll be able to devise a strategy that works.

Also +1 for Zenoss. That is a fantastic NMS. Written in python, so hooking up 
scapy to do periodic game latency checks would be slick and a natural fit. 

On January 19, 2015 5:18:38 PM CST, Josh Luthman  
wrote:
>IXIA would be the first product to look at as far as emulating traffic.
>
>
>Josh Luthman
>Office: 937-552-2340
>Direct: 937-552-2343
>1100 Wayne St
>Suite 1337
>Troy, OH 45373
>
>On Mon, Jan 19, 2015 at 6:16 PM, George Herbert
>
>wrote:
>
>> Emulating game traffic...  Good luck with that.  You'll probably have
>to
>> figure it out and build your own models per service, though a lot is
>> encapsulated in https.
>>
>> In terms of showing it to the public, look at Zabbix and Zenoss; both
>do
>> dashboards and managing multiple realtime monitoring / performance
>info
>> feeds well.
>>
>> George William Herbert
>> Sent from my iPhone
>>
>> > On Jan 19, 2015, at 2:10 PM, Michael O Holstein <
>> michael.holst...@csuohio.edu> wrote:
>> >
>> > ?Can someone point me in the right direction for something that
>allows
>> creation of a "dashboard" with current and statistical latency to the
>> various game servers (PC, Xbox, PS4, etc) ? .. I'm in the education
>space
>> and we get lots of questions/complains about this and would like a
>way to
>> make the stats public.
>> >
>> >
>> > I could roll something with RRD and Smokeping but with all the
>> packet-shaping crapola (including that which we use here) I need
>something
>> that emulates the actual game traffic as would be classified by all
>the
>> network crap that endeavors to mess with it.
>> >
>> >
>> > (not intended to be an argument about QoS and prioritization,
>responses
>> addressing either --or the politics thereof-- really aren't helpful).
>> >
>> >
>> > TIA,
>> >
>> >
>> > Michael Holstein
>> >
>> > Network & Data Security
>> >
>> > Cleveland State University
>>
>
>!DSPAM:54bd9147175514905077569!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: gamer "lag" dashboard

[Charles N Wyble]

As a zenoss plugin, I agree. 

On January 19, 2015 7:22:36 PM CST, Roland Dobbins  wrote:
>
>On 20 Jan 2015, at 5:10, Michael O Holstein wrote:
>
>> I need something that emulates the actual game traffic as would be 
>> classified by all the network crap that endeavors to mess with it.
>
>That sounds like a great open-source project - let us know when you're 
>done!
>
>;>
>
>---
>Roland Dobbins 
>
>!DSPAM:54bdae36220661660451680!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: gamer "lag" dashboard

[Charles N Wyble]

SSL is no problem. We just had a whole thread about breaking it. :-) 


On January 19, 2015 5:16:43 PM CST, George Herbert  
wrote:
>Emulating game traffic...  Good luck with that.  You'll probably have
>to figure it out and build your own models per service, though a lot is
>encapsulated in https.
>
>In terms of showing it to the public, look at Zabbix and Zenoss; both
>do dashboards and managing multiple realtime monitoring / performance
>info feeds well.
>
>George William Herbert
>Sent from my iPhone
>
>> On Jan 19, 2015, at 2:10 PM, Michael O Holstein
> wrote:
>> 
>> ?Can someone point me in the right direction for something that
>allows creation of a "dashboard" with current and statistical latency
>to the various game servers (PC, Xbox, PS4, etc) ? .. I'm in the
>education space and we get lots of questions/complains about this and
>would like a way to make the stats public.
>> 
>> 
>> I could roll something with RRD and Smokeping but with all the
>packet-shaping crapola (including that which we use here) I need
>something that emulates the actual game traffic as would be classified
>by all the network crap that endeavors to mess with it.
>> 
>> 
>> (not intended to be an argument about QoS and prioritization,
>responses addressing either --or the politics thereof-- really aren't
>helpful).
>> 
>> 
>> TIA,
>> 
>> 
>> Michael Holstein
>> 
>> Network & Data Security
>> 
>> Cleveland State University
>
>!DSPAM:54bd909e175152519182214!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: DDOS solution recommendation

[Charles N Wyble]

Also how are folks testing ddos protection? What lab gear,tools,methods are you 
using to determine effectiveness of the mitigation. 

On January 8, 2015 11:01:47 AM CST, "Manuel Marín"  wrote:
>Nanog group
>
>I was wondering what are are using for DDOS protection in your
>networks. We
>are currently evaluating different options (Arbor, Radware, NSFocus,
>RioRey) and I would like to know if someone is using the cloud based
>solutions/scrubbing centers like Imperva, Prolexic, etc and what are
>the
>advantages/disadvantages of using a cloud base vs an on-premise
>solution.
>It would be great if you can share your experience on this matter.
>
>Thank you
>
>!DSPAM:54aeb96d198072115716976!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Comcast thinks it ok to install public wifi in your house

[Charles Mills]

In the US at least you have to authenticate with your Comcast credentials
and not like a traditional open wifi where you can just make up an email
and accept the terms of service.  I also understand that it is a different
IP than the subscriber.  Based on this the subscriber should be protected
from anyone doing anything illegal and causing the SWAT team to pay a
visit.  I haven't upgraded my gear though.

Now..they are doing this on your electric bill and taking up space (albeit
a small amount of it) in your home.

Chuck



On Wed, Dec 10, 2014 at 9:35 PM, Jeroen van Aart  wrote:

> Why am I not surprised?
>
> Whose fault would it be if your comcast installed public wifi would be
> abused to download illegal material or launch a botnet, to name some random
> fun one could have on your behalf. :-/
>
> (apologies if this was posted already, couldn't find an email about it on
> the list)
>
> http://www.theregister.co.uk/2014/12/10/disgruntled_
> customers_lob_sueball_at_comcast_over_public_wifi/
>
> "A mother and daughter are suing Comcast claiming the cable giant's router
> in their home was offering public Wi-Fi without their permission.
>
> Comcast-supplied routers broadcast an encrypted, private wireless network
> for people at home, plus a non-encrypted network called XfinityWiFi that
> can be used by nearby subscribers. So if you're passing by a fellow user's
> home, you can lock onto their public Wi-Fi, log in using your Comcast
> username and password, and use that home's bandwidth.
>
> However, Toyer Grear, 39, and daughter Joycelyn Harris – who live together
> in Alameda County, California – say they never gave Comcast permission to
> run a public network from their home cable connection.
>
> In a lawsuit [PDF] filed in the northern district of the golden state, the
> pair accuse the ISP of breaking the Computer Fraud and Abuse Act and two
> other laws.
>
> Grear – a paralegal – and her daughter claim the Xfinity hotspot is an
> unauthorized intrusion into their private home, places a "vast" burden on
> electricity bills, opens them up to attacks by hackers, and "degrades"
> their bandwidth.
>
> "Comcast does not, however, obtain the customer's authorization prior to
> engaging in this use of the customer's equipment and internet service for
> public, non-household use," the suit claims.
>
> "Indeed, without obtaining its customers' authorization for this
> additional use of their equipment and resources, over which the customer
> has no control, Comcast has externalized the costs of its national Wi-Fi
> network onto its customers."
>
> The plaintiffs are seeking monetary damages for themselves and on behalf
> of all Comcast customers nation-wide in their class-action case – the
> service was rolled out to 20 million customers this year."
>
> --
> Earthquake Magnitude: 4.8
> Date: 2014-12-10  22:10:36.800 UTC
> Date Local: 2014-12-10 13:10:36 PST
> Location: 120km W of Panguna, Papua New Guinea
> Latitude: -6.265; Longitude: 154.4004
> Depth: 35 km | e-quake.org
>

Re: Incident notification

[Charles N Wyble]

Pushover and email to sms from both an inband and off site monitoring vm. 

On November 21, 2014 9:52:00 AM CST, Thijs Stuurman  
wrote:
>Nanog list members,
>
>I was looking at some statistic and noticed we are sending out a
>massive amount of SMS messages from our monitoring systems.
>This left me wondering if there isn't a better (and cheaper)
>alternative to this, something just as reliant but IP based. We all
>have smartphones these days anyway.
>
>Therefore my question, what are you using to notify admins of
>incidents?
>
>Kind regards / Met vriendelijke groet,
>
>Thijs Stuurman
>
>
>
>[IS Logo]
>
>
>
>
>IS Group
>
>Wielingenstraat 8
>
>T
>
>+31 (0)299 476 185
>
>i...@is.nl
>
>1441 ZR Purmerend
>
>F
>
>+31 (0)299 476 288
>
>www.is.nl
>
>
>
>IS Group is ISO 9001:2008, ISO/IEC 27001:2005, ISO 20.000-1:2005, ISAE
>3402 certified. De datacenters zijn PCI DSS en ISO 14001 compliant.
>
>
>
>!DSPAM:546f5ff6238696356864932!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Craigslist hacked?

[Charles Mills]

Not seeing that here  The local site and the general http;//
www.craigslist.org both look to be going to the correct site.

On Sun, Nov 23, 2014 at 10:41 PM, Brian Henson  wrote:

> Is anyone else seeing their local craigslist redirected to another site
> other than craigslist? I see it loading http://digitalgangster.com/5um.
>

Re: NTT high packet loss from US and BR to AU?

[Charles van Niman]

Howdy all,

  I've been lurking for a long time, first time writing in. Please
excuse my inexperience. Javier, can you provide full traces, and
source/destination addresses?

/Charles

On Wed, Oct 22, 2014 at 11:18 PM, Javier J 
wrote:

> Anyone else notice this?
>
> Or is this an AWS issue in APAC that hasn't been reported yet?
>
> AU-NY(aws)
> 18. xe-1.level3.lsanca03.us.bb.gin.n 72.0%
>
> BR(aws)-AU(aws)
> 11. ae-9.r20.snjsca04.us.bb.gin.ntt.net 71.4%
>
>
> NJ/NYC to AU(aws)
> 9. ae-9.r20.asbnva02.us.bb.gin.ntt.net 45.9% 772 10.1 16.4 9.2 94.4 13.3
> 10. ae-2.r21.lsanca03.us.bb.gin.ntt.net 40.5% 772 69.6 72.7 69.3 149.2 9.0
>

Scaling from home broom closet to multisite "home" data center/WAN network on a budget

[charles]

Hi everybody,

It's been a long time since I've kicked up a new thread here on ye ol 
nanog.


Recently I've been putting some serious thought into home "budget" data 
centers. What started out as a little router/switch/virt server lab by 
me/myself/I in 2008, has turned into a multisite (7 points of presence 
spread across two states), multi rack (223U, 10 racks), multi carrier 
(time warner, suddenlink, att dsl, att uverse, blended bandwidth) affair 
today.


I've realized that my friends and I may need to approach things a bit 
more. professionally/seriously/circumspectly (is that a word?).


We run OSPF (and soon i/eBGP including out to amprnet) between all the 
sites, have a multi vendor (cisco/dell/pfsense/mikrotik/linux) l3/l2 
network. All the gear has been sourced from flea markets/ebay etc.


Anyone else on here have a similar situation? How do you deal with it? 
(More details/questions after the jump)


I mean my little lab is not quite as  *ahem* involved as mr morris 
home data center

http://smorris.uber-geek.net/lab.htm

but it's also just a whee bit bigger then the typical /r/homelab posts.

https://commons.thefnf.org/index.php/FNF_Lab << needs a bit more work to 
reflect current reality, but you can get the general idea)


So how to home/small (multisite) business datacenter/network on 
budgetz? Not so much the software side of things (networking/operating 
system etc), but more the actual layer 1 stuff (and of course software 
to manage all that).


Things like:

Power

* Any good resources for wiring optimization/layout/dos/dont's? (like 
say how to hook up surge protector -> ups -> pdu -> gear ?)


* Network UPS tools is what we are considering "betting on" for managing 
our multi vendor (APC/Dell/Cyclades) PDU/UPS setup across all sites. 
(The joys of sourcing from flea markets). Does anything better exist? 
Any issues with running this across a WAN? Any tips for tying it into 
LDAP? (We have a large amount of lab gear that we wish to make available 
to the public for use by reservation, we don't want them being able to 
turn off our production gear) :)



Temperature / Environmental monitoring

* I've found the temper USB sensors to be perfect for per rack 
temperature statistics. I believe they will do humidity as well. Any 
other good environmental sensors for cheap? I know about PacketFlux 
which is cool for an all in one solution. (I'd love to use netbotz or 
something, but those are pricy).


Cooling

* BTU calculation resources?
* Cooling sizing?
* How hot can gear really run?
* Is per rack cooling/airflow via floor fans necessary?

For now we all just have room AC cooling the gear. So far all seems 
well, curious if folks have done stuff like pipe AC output directly into 
a cabinet etc?


Noise dampening

Any good room partitions or other solutions for noise cancelling? We've 
got combination of closed cabs and open racks, so a rack based solution 
won't quite cut it.



Also what about insurance?

Probably a thousand other questions/comments/ideas/suggestions, but I 
think this should kick off a great discussion!


It's funny how it kind of all just combines/grows and next thing you 
know, you've got yourself a whole little internet as it were. :)

Re: RADB

[Charles Gucker]

Take a look:

https://www.arin.net/resources/routing/

charles


On Wed, Oct 8, 2014 at 10:35 PM, Brandon Wade  wrote:
>
>
>>> For a newbie, how does one go about learning the basic's of IRRd.
>
> That pretty much sums it up. I feel like I'm stuck reading RFC's that are too 
> overly complex for something that seems like it shouldn't be complex. Anyone 
> know of a
> quick 101 intro to routing registries with a simple example of an AS
> that has two upstream providers and a handful of peers and downstream
> AS's?
>
> Brandon
>
>
> On Wednesday, October 8, 2014 8:15 PM, Faisal Imtiaz 
>  wrote:
>
>
>
> I can relate to this, having gone thru a similar process/experience fairly 
> recently in using IRRd..
>
> So the real question Brandon is asking..
>
> For a newbie, how does one go about learning the basic's of IRRd.
>
> Speaking for myself, if there is a good answer, I would welcome it.
>
> Here is what I had to do...
>
>   1. Used Radb lookup on different AS-Set and ASN to get a feel on how others 
> were using this resource.
>
>2.  Went thru the ARIN IRRd Tutorial / info pages on how to create records 
> etc.
>
>3.  Did Google searches on finding some of the older NANOG presentations 
> about IRRd.
>
> Regards
>
> Faisal Imtiaz
> Snappy Internet & Telecom
>
>
>
> - Original Message -
>> From: "Brandon Wade" 
>> To: nanog@nanog.org
>> Sent: Wednesday, October 8, 2014 4:44:07 PM
>> Subject: RADB
>>
>> Hi,
>>
>> I really don't know where else to post this. I recently subscribed to RADB
>> and added route objects and route6 objects for our prefixes we announce. Of
>> course an aut-num object was created and I created a list of ASN's that are
>> downstream customers in an as-set list. But, since this is my first time
>> ever subscribing to a routing registry, I really don't know for sure that
>> I'm doing everything correctly. So, I submitted an e-mail to RADB and
>> requested that they review what I've done and to see if I'm doing it
>> correctly. Well, the reply I received was far less than I could have ever
>> imagined:
>>
>> "RADb is a self-serve service. If you have specific questions, we can
>> address those. However, the type of audit requested is not a part of the
>> standard offering associated with this service."
>>
>> So my question is, how am I supposed to verify that what I've done is what is
>> supposed to be done and that I am doing this correctly?
>>
>> My next question is, why would RADB offer zero support for confirming this?
>> And lastly, why should my organization pay $500 per year to a service that
>> is unwilling to assist in making sure their subscriber is using their
>> service properly?
>>
>> Best regards,
>> Brandon Wade
>>

Re: RADB

[Charles Gucker]

You can also verify the object configurations from another IRRd, such
as Level(3)

whois -h filtergen.level3.net "RADB::YOUR-AS-SET
-searchpath=RIPE;ARIN;RADB -recurseok -warnonly"

You can limit the searchpath to just include RADB if you wish, but
it's good to know what else is out there.

charles


On Wed, Oct 8, 2014 at 4:44 PM, Brandon Wade  wrote:
> Hi,
>
> I really don't know where else to post this. I recently subscribed to RADB 
> and added route objects and route6 objects for our prefixes we announce. Of 
> course an aut-num object was created and I created a list of ASN's that are 
> downstream customers in an as-set list. But, since this is my first time ever 
> subscribing to a routing registry, I really don't know for sure that I'm 
> doing everything correctly. So, I submitted an e-mail to RADB and requested 
> that they review what I've done and to see if I'm doing it correctly. Well, 
> the reply I received was far less than I could have ever imagined:
>
> "RADb is a self-serve service. If you have specific questions, we can
> address those. However, the type of audit requested is not a part of the 
> standard offering associated with this service."
>
> So my question is, how am I supposed to verify that what I've done is what is 
> supposed to be done and that I am doing this correctly?
>
> My next question is, why would RADB offer zero support for confirming this? 
> And lastly, why should my organization pay $500 per year to a service that is 
> unwilling to assist in making sure their subscriber is using their service 
> properly?
>
> Best regards,
> Brandon Wade

Re: Facebook down?

[Charles Mills]

W. PA. too.  Looks pretty widespread.


On Wed, Sep 3, 2014 at 3:46 PM, aUser  wrote:

> Appears to be in Oregon, Southern Oregon.  Mobile too.
>
> Sent from my iPhone 5S.
>
> > On Sep 3, 2014, at 12:45 PM, Marshall Eubanks <
> marshall.euba...@gmail.com> wrote:
> >
> > This message has no content.
>

Re: fire ants

[charles]

On 2014-08-12 15:06, me wrote:

Ran across this paper the other day and didn't know how big a problem
it was. Looks like Eduardo's post confirms it.

http://www.rainbowtech.net/products/docs/c51ce4107047eb1b2dc/Ants%20in%20OSP%20Equipment.pdf.pdf



Now that is fascinating. I like how they reproduced the issue via an ant 
farm. That's pretty slick.

Re: [HFC] pooling modems in layer2

[charles]

On 2014-08-12 09:23, Toney Mareo wrote:

Hello

I think it's kind of an isp secret but I would be curious how do
people distribute modems to pools before they would even reach the
actual IP network so on layer2:

http://dl.packetstormsecurity.net/papers/evaluation/docsis/Service_Distribution.jpg


For this I would like to get some clarification because I do not work
in the telco industry. As I can figure out of the docsis, cablelabs
documents. The CMTS device is connected to the coax segments through
fiber. Therefore one could say that the "modem facing" side is a fiber
optic interface but it's not 1000 Base-FX, not a regular Ethernet over
fiber. It sends signals through a broad range of frequencies.


Sounds about right to me.




So what I would like to accomplish to provide a different pool of dhcp
servers, which provides different config file, tod server, router, dns
etc. infos to the modems but to do all this in Layer2.



Why? Do you have a bunch of cable modems and a CMTS? If so, does the 
documentation not cover this? Or are you trying to hack your cable 
modem/cable provider?



I don't have hands on experience with CMTS-es but I would think that
they are able to pool clients by MACs and able to send eg 500 clients
to DHCP server1 and the other 1500 to DHCP server2 before they would
even get an IP, so I talking of pure layer2 here!

Let's say if the CMTS device does not support this, what are the other
options for routing layer2 traffic coming out of the CMTS?


Um. Probably via RADIUS and via VLAN assignment?

 If I would

know more about the device I would say that put a linuxbox after it
(on the ISP facing nic) and mark the packets going out with
arptables/ebtables then send them out of different nics to different
dhcp servers.


Most likely they just use VLANs. This rack of CMTS gear is on port 22 of 
the agg switch, vlan 2 and ip helper is set for vlan 2 to the desired 
dhcp server (which is most likely an HA floating IP if not a full blown 
VIP etc).

Re: Dealing with abuse complaints to non-existent contacts

[charles]

On 2014-08-10 10:19, Gabriel Marais wrote:

Hi Nanog

I'm curious.

I have been receiving some major ssh brute-force attacks coming from 
random
hosts in the 116.8.0.0 - 116.11.255.255 network. I have sent a 
complaint to
the e-mail addresses obtained from a whois query on one of the IP 
Addresses.


Did they have a dedicated abuse e-mail? Did you receive an automated 
confirmation (which generally means the communication went into some 
sort of ticket queue as opposed to 
$random_employee_malbox_who_has_moved_on .


How did you format the e-mail? What information did you provide?

(Folks here, what do you look for in an abuse complaint to take it 
seriously)? I imagine many here have template/ticket systems for abuse 
communications? What info do you ask for in those communications?




My e-mail bounced back from both recipients. Once being rejected by 
filter

and the other because the e-mail address doesn't exist. I would have
thought that contact details are rather important to be up to date, or 
not?


Yes. For operators who actually care about running their networks and 
being good citizens. At least that's my opinion.




Besides just blocking the IP range on my firewall, I was wondering what
others would do in this case?




Well of course fail2ban is always good.

My personal preference is only expose HTTPS/SMTPS/IMAPS to the world. 
Zero management traffic on the front channel. SSH is only possible once 
you have connected to the VPN (which is running on 443 on another IP and 
is accessible without any firewall restrictions).

Re: EFF gets into the CPE router software business..

[Charles N Wyble]

Well yes. :)

Plenty of relatively inexpensive x86 based kit out there. Maybe with TPM? Never 
looked.  Atom can push a good amount of packets. 

I am in the process of building an HCL for the various bits of the 
FreedomStack. (CPE/distribution/core etc). My family is  a very heavy internet 
user. Both directions. An atom pfsense router and netgear 3800 has done the 
trick. Now to package them up with a slick / simplified / turnkey configuration 
and not have people balk at the price.

I hadn't taken much security/TPM wise into account. Would be a good way to help 
folks deal with the  increased expense. NSA proof, Snowden endorsed! :)



On July 25, 2014 6:42:13 PM CDT, valdis.kletni...@vt.edu wrote:
>On Fri, 25 Jul 2014 13:11:29 -0500, char...@thefnf.org said:
>> On 2014-07-25 12:22, valdis.kletni...@vt.edu wrote:
>> > The second big challenge is that to the best of my knowledge, there
>exist
>> > no router-class hardware that includes a TPM chip,
>>
>> OpenWRT x86? Run it on a decently specced laptop a couple gens old
>(like
>> a Dell Latitude 6500 or so). That's got TPM, plenty of ram.
>> Of course you can run on a server board (Dell Poweredge or
>something). I
>> prefer pfsense myself for full blown kit.
>
>Yeah, but it's hard to justify a PowerEdge for a Joe Sixpack consumer
>CPE
>(admittedly, I managed to leave that phrase out of 'router-class', mea
>culpa).
>
>
>
>
>
>!DSPAM:53d2eb62262122034419612!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [OPINION] Best place in the US for NetAdmins

[charles]

On 2014-07-22 18:20, Nolan Rollo wrote:

I've been trying to decide for a while what makes a good home for a
Network Admin... access to physical, reliable upstream routes? good
selection of local taverns? What, in your opinion, makes a good
location for a Network Admin and where in the US would you find that?


H. That's a great question.

Well does the network admin mostly travel to job sites? Or work 
remotely? If either/both are true,
I'd suggest the DFW area. It's a major hub in both internet and travel 
respects. (I fly American Airlines exclusively, I live in Austin. Most 
flights are AUS-DFW-$FINALHOP).




Also, I'd like to introduce myself [[ o/ ]] I've been watching the
list for a while now and have found it helpful with picking up some
"best practices", getting use-case scenarios you might not see in text
books.


Is that code for "all you crazies doing crazy things for crazier 
bosses?" :)


Welcome to the list sir!


 I attended Michigan Tech for Computer Networking and System

Administration and have been bouncing around for a couple of years
trying to find my calling.


Yeah. That happens.


 I've been working a lot with VoIP and

that's been my interest ever since middle school. I've been mainly
playing with stub networks for most of my life but have recently
started working with larger routed networks, leading me to subscribe
to the NANOG list.


Excellent!


My latest endeavor was acquiring and ASN and a /24

from ARIN and multihoming a very small MSP.


Oooo. How did that go for you? What upstreams did you connect with? How 
painful was it? How much convincing did it take to get management to go 
along? What are the post implementation improvements? etc etc.


I've been fortunate enough

to have really sharp mentors to help answer any questions I've had
along the way. I know there must be quite a few people like myself
that are lurking on the list and I just wanted to thank you guys for
answering other questions and providing input on
  topics that have come through the list.




Yes. Many lurkers, many off list replies to most threads. Did you get 
any awesome off list replies? Summarize them back to the list?

Re: Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

[charles]

I highly recommend pfsense for a firewall (been using pfsense and
m0n0wall for years), but do have some concerns about using it at scale
for (several) thousands of users.


So far it's gone fairly well for the existing subscriber base. The 
current service footprint is ~1k homes. I think it's running on a Dell 
Poweredge ~29xxish , don't know for sure.


 Most of this relates to NAT/State

tracking, some of it hardware related, some of it software.


Right.

 If

possible, I would suggest you obtain a routable IP address per user
and avoid the pitfalls of NAT (I know at some point this may become
expensive).


Exactly.

If you start with IPv6 from day 1 you are in a lot better

place to encourage customers to upgrade to IPv6 capable gear.



Yes. We are doing v6 to every end user CPE. Absolutely. It will be 
there, be turned on and we hope to send all netflix/facebook/google etc 
traffic over v6. The v4 will be CGN. (We think we can only get a /24 
reasonably).


@Comcast v6 team (and really anyone who has a large dualstack network 
(*waves* at Owen),


So you guys have v6 turned up. You passed 1tb of traffic. Didn't comcast 
also write some floss code for CGN? So presumably you'll have to start 
doing CGN soon.


Thoughts on long tail v4 only internet being seriously degraded by large 
scale CGN? (Maybe that's a new thread?) If the major properties are v6, 
shouldn't that be enough to keep the support costs down? (My friends in 
the MMORPG "cloud gaming" space tell me that my approach could wreak 
havoc with many game engines).


Thoughts on what happens when you've got v6 at your door and v4 at your 
CO? Who is running a network like this today (I imagine most small ISPs 
will be in that boat soon)?


(And also, what's up with people complaining about ARIN fees?). The air 
fiber radios FNF is installing in KC cost 5k capex. So enough already 
about a ONE TIME 1k fee and get your v6 space! (I agree with the posters 
who said if you can't afford the arin fee, GET OUT OF BUSINESS).



 I would

also suggest using stateless firewall rules and routing on your WAN
devices.


That does seem to be the common wisdom. I'm actually not 100% sure what 
we've got in line. It's OpenWRT based all around, so I'm sure IPTABLES 
(and maybe even some ebtables).



 This should simplify the functions performed by these boxes

to reduce the need to troubleshoot, apply updates, etc (resulting in
better availability).


Yeah. Of course.



 I haven't used pfsense in an ISP WAN router

capacity, and personally feel a router from Cisco, MikroTik, or
Ubiquiti's EdgeOS devices, etc may be more appropriate in this role.



I've got pretty much every Cisco router/switch in our lab, and an 
EdgeRouter.


What mikrotik should I evaluate?

Our lab : https://commons.thefnf.org/index.php/FNF_Lab



If you've automatically discounted big name gear due to upfront costs,
you might consider buying from a used equipment reseller (I can
recommend a few, if needed).


No. It's mostly for the customization/scripting etc. "SDN" and all that 
jazz.  ;)





If you do need to use NAT, I feel like 500+ users sharing a single NAT
IP will result in poor quality of service and more admin overhead.


Quite possibly. However if it's just for long tail v4 only sites, I 
wonder how much it matters?



 My

gut feeling is that <50 may be more appropriate, depending on the
quality of service you want to provide. This provides some headroom if
one user makes many connections (p2p, virus infection, DoS attack) and
also lessens the number of subs you need to look at in cases of abuse
that are reported as an IP/port. Individual pfsense servers in a
cluster may provide scalable CGN services. I'm not sure how you want
to handle logging of all that data, but pfsense should allow you to
define rules that allow stateless auditing (ip 1.2.3.4, ports
1000-2000 always NAT to sub A). The XML config file or possibly the
shell is probably the easiest way to define such rulesets at scale.



Right right. I'm very familiar with the XML config and CLI. We've gotten 
to know pfSense well in our AutoTunnel (RADIUS) work. We patched (and 
released back to upstream) hostapd and other bits to actually correctly 
implement the RFC :D


So we've got a solution that is multi gateway. So based on the login 
creds you use, you get dropped into an appropriate vlan / BMX tunnel and 
get routed out the appropriate gateway.





I didn't see it mentioned, where (and to whom) are you multihoming?


Kansas City Kansas. Joesdatacenter.com is the current tower PoP. We can 
get transit from him, of course peer with KCIX , and we'll probably get 
transit from another local ISP in town (CTC). Of course level3/att/vz et 
al are all in town/on net and just a very short fiber hop away from Joes 
if we want to go that route.



 Do

you have a good working relationship with these folks (cell phone,
email contacts that reach someone promptly)?


Yes. Very much so.

Will you be considered a

facili

Re: EFF gets into the CPE router software business..

[charles]

On 2014-07-25 12:22, valdis.kletni...@vt.edu wrote:

On Thu, 24 Jul 2014 22:06:38 -0700, George Herbert said:

Any idea how well CeroWRT stands up to nation-state level intrusion 
efforts?


If they are as determined as FBI v Scarfo (the FBI pulled a black bag 
job
to install a keystroke logger in a mobster's PC to capture his PGP 
passphrase),
it's pretty much "game over".  Isn't much the average router-class 
hardware

can do to protect itself at that point.


Of course. Physical access is root access. We know this.




The second big challenge is that to the best of my knowledge, there 
exist

no router-class hardware that includes a TPM chip,


OpenWRT x86? Run it on a decently specced laptop a couple gens old (like 
a Dell Latitude 6500 or so). That's got TPM, plenty of ram.
Of course you can run on a server board (Dell Poweredge or something). I 
prefer pfsense myself for full blown kit.



 which means that you're
not going to be able to implement a trusted boot environment.  This 
means that
we're stuck with trusting at least part of the boot process (though we 
can
probably trust the first stage boot loader on a 3800, as that appears 
to be
in an actual ROM, and we'll have to trust the bootstrap code on the 
flash,
but if we use a signed kernel, everything after that can have some 
trust

attached.)


Right.




There's a number of attack surfaces left on CeroWRT, starting with the 
usual
"find a 0-day and point it" - good targets there are the Linux network 
stack,
the IPtables code, dropbear (which is nice, but almost certainly not 
audited
as heavily as OpenSSH), and Luci.  And yes, reflecting an attack off a 
browser
behind the router is *very* much in scope - *most* of the pwned router 
attacks

we see come from javascript or other executables pointed at the usually
well-known router address from a PC behind the router.



Agree 100%

All the way to pulling a MITM on downloads from Dave Taht's 
repositories.  The
combination of DNSSEC, trusted crypto signatures on the dowload 
package, and
OpeWireless's plans to use Tor to do the software download should make 
it a

*lot* harder to attach via that route.



Oooo. I'll have to clone that methodology for the FNF downloads.

Re: Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

[charles]

On 2014-07-24 11:39, Josh Baird wrote:

FCC licensing?  No licenses as long as you operate in unlicensed
bands (ie, 900mhz/2.4ghz/5). 


Yes. This is correct. Also no licensing needed for 24ghz. We are rolling 
out a dual uplink 24ghz AirFiber back bone in the next couple of weeks.


The FNF has obtained a 3.65ghz license and that's come in very handy in 
some of the very noisy parts of our footprint.




 On Thu, Jul 24, 2014 at 5:10 AM, hayden  wrote:


Sorry, no feedback from me.. I have couple of questions though, how
much licensing do you need to go through, to actually start a WISP?


Well. I'd recommend being incorporated. Which isn't licensing per se. 
I'd also recommend being bonded/insured. Just good general business 
practices.




Also, Kansas.. Are you concerned that you’ll have to compete with
Google Fiber at some point?




Not really. We are serving areas that Google Fiber has decided to not 
service.

Re: EFF gets into the CPE router software business..

[charles]

On 2014-07-25 00:06, George Herbert wrote:
Any idea how well CeroWRT stands up to nation-state level intrusion 
efforts?



Interesting question.

It uses OpenWRT as a base. IPTables for the firewall. So that's a pretty 
big code base right there (though certainly a bit less than a comparable 
x86 Linux box). Most people use it with LUCI (web UI). So that adds more 
code.


Is this attack from the WAN side? Or from a comped browser on the LAN 
side?



Interesting discussion for a Friday! :)

Re: EFF gets into the CPE router software business..

[charles]

On 2014-07-24 12:04, Valdis Kletnieks wrote:

So the EFF is pushing development of an open CPU router

https://www.eff.org/deeplinks/2014/07/building-open-wireless-router
https://openwireless.org/

It's currently targeting WNDR3800's and based on the CeroWRT software
(which works pretty well in my own experience).

What will possibly be interesting in this forum is that it's explicitly
targeting having open guest wireless access (unlike the stuff being 
pushed
by some ISPs, where you can roam but only to other customers of the 
same

ISP).

!DSPAM:53d13dc965333732154236!


The Free Network Foundation (which I co founded and am CTO of) has been 
helping several groups in the USA do this for ~1 year now. EFF is simply 
rebranding/respinning community networking, but they are pretty new to 
the USA Free Networks party overall. They just have a bigger 
budget/brand recognition (though FreedomTower has become a pretty 
resilient brand based on the e-mails we get on a daily basis). Also I'm 
not sure of the level of support/hand holding/documentation etc EFF will 
provide for folks wanting to build a network off this setup (I'm 
guessing not much).  Also most incumbent carriers prevent sharing (where 
FNF supported/assisted/collaborative/affiliated US based efforts back 
haul (over high capacity wifi or VPN over incumbent circuits) to 
wholesale colocation facilities POP and do things like monitor abuse@ 
contacts etc. (Ya know, actually responsibly run an ISP).


I'd rather of seen them partner with FNF, (or actually much more 
preferable would be upstream wrt projects like QMP) and not spin YET 
ANOTHER FIRMWARE.


I'm glad they picked CeroWRT though.

Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

[charles]

Hey everybody,

So all this talk about monopolies, small ISPs vs the big bad netflix  , 
muni fiber etc etc has been interesting. Lots and lots of talk, lots of 
interesting links etc.


I'm an action/results oriented individual, and have been working on 
actually building out a grassroots ISP, instead of just talking about 
it. :)


Over the past year or so, I've been involved with an effort to launch a 
community ISP in the Kansas City MO area. It's got several towers up now 
and a decent amount of users. It's been funded by the community that it 
serves. Feel free to ask any questions you have about the details. It's 
an open network in all aspects (design, business model etc). It is 
intentionally designed/operated in such a way that all aspects can be 
disclosed.


We are now ready to take the next step and obtain an ASN and v6 space 
(also looks like we can get a /24 of v4 space as well).


What are the things that we should do before we get those resources? 
What should we do immediately after? What books/rfc/bcp should we be 
most familiar with?


As is typical of many small outfits, we have an incredibly high degree 
of software skill, and a limited budget which goes entirely to hardware.


This is a greenfield network. We've got Ubiquiti gear for the backbone. 
Running a mix of QMP routers with BMX6 as the IGP linked over AirOS l2 
bridge "pseudowires". We'll be homed to two AS upstreams. Using pfSense 
as the WAN edge routers.


From all my reading of the list, it seems like key things to do in this 
scenario:


1) Have full flow telemetry at all points to help with (D)DOS 
mitigation.

2) Do CGN in pools (so perhaps ~500 to 1k users behind each IP)?
3) Provision a /56 of v6 space to each end user. I was thinking of 
having the CPE with CeroWRT and be multi SSID with a /64 per. I'm 
interested in folks thoughts on this?

4) Upsell a public v4 address if someone requires it
5) Of course implement bcp38

I'm mostly interested in technical feedback. Business model etc type 
feedback is welcome as well, but not the primary purpose of this 
message. :)


Thanks!

Charles Wyble
CTO Free Network Foundation

Re: Inevitable death, was Re: Verizon Public Policy on Netflix

[Charles Gucker]

On Tue, Jul 15, 2014 at 12:21 AM, Brett Glass  wrote:
> Perhaps it's best to think of it this way: I'm outsourcing some backbone
> routing functions to my upstreams, which (generously) aren't charging me
> anything extra to do it. In my opinion, that's a good business move.

Last comment on the thread.   And the truth will set you free!
Please have your upstream provider peer with Netflix and all will be
right in the world.As a single-homed customer of said ISP, you are
subject to their rules. No need for your involvement in this old
routing protocol and numbers business, let them do it as it's their
business, not yours.  I will not respond further and we can let
this thread finally die.

- charles

Re: Inevitable death, was Re: Verizon Public Policy on Netflix

[Charles Gucker]

> But regardless of the financial arrangements, such a connection doesn't
> require an ASN or BGP. In fact, it doesn't even require a registered IP
> address at either end! A simple Ethernet connection (or a leased line of any
> kind, in fact; it could just as well be a virtual circuit) and a static
> route would work just fine.

Anybody else feel a vendor t-shirt in the works?

"Who needs BGP to peer, a static route would work just fine!"

Time to get back into the Hot Tub Time Machine and back on point.
*hangs head in shame*

Re: Verizon Public Policy on Netflix

[charles]

On 2014-07-10 21:40, Randy Bush wrote:

Trying to play both sides of the issue like that in the same
paragraph is just...dizzying.


if we filtered or otherwise prevented conjecturbation, jumping to
conclusions based on misuse of tools, hyperbole, misinformation, fud,
and downright lying, how would we know the list exploder was working?


Randy,

The ipv6 vs NAT discussions of course!

Re: Verizon Public Policy on Netflix

[Charles Gucker]

> If Netflix continues on its current course, ALL ISPs -- not just rural ones,
> will eventually be forced to rebel. And it will not be pretty.

I call hogwash.ALL ISPs are in the business of providing access to
the Internet.If you feel the need to rebel, then I suggest you
look at creative ways to increase revenue from your customers, not
threaten to cut off a portion of the Internet that "cost too much".

A point that seems to be missed in this whole discussion.It was
your choice to provide services in a rural area, not Netflix, Akamai
or the like.If your business model is flawed, then don't expect
somebody else to step in and fix it for you. Bandwidth is
expensive to procure in a rural area, if you wish to change that,
maybe it's time to find some investors and build your own network into
an urban area where bandwidth, and interconnections in general, are
more reasonably priced.

Also, based on your logic within this whole thread, if I was a
customer of yours, I'd expect you to pay me to use your services as
you would be looking to get paid for my use of third party services.
 Also, I believe that what happened between Comcast and Netflix is
temporary, much like what happened between Comcast and Level(3).

charles

Re: Peering Latency

[Charles N Wyble]

Is it Friday already? Or is this not a troll email? Its hard to tell. 

If its not a troll: Put up some smokeping boxes. Graph it for a few nights. 
Gather details. Send us those. That is far more interesting/(damning?)

If its a troll: *grabs popcorn and gets comfortable* . we've not had a good 
"zomg the pipes, they are teh fullz, woe is Netflix" (and the obligatory 
cgn/v6/software vs hardware router sub thread divergences). 

Very nicely struck balance sir! 


On July 2, 2014 11:19:07 PM CDT, Sam Norris  wrote:
>Hey all - new to the list but not to the community...
>
>Wondering if this is typical when there is too small of a pipe between
>peering
>arrangements:
>
>From Level3 to Time Warner
>
> ADDRESSSTATUS
>   24.69.133.206 4ms 4ms 4ms 
>   34.69.153.222 9ms 4ms 4ms 
>   4 4.69.158.78 8ms 4ms 4ms  (L3)
>   566.109.9.121 28ms 53ms 29ms   (TWC)   <--
>   6107.14.19.87 30ms 28ms 28ms 
>   766.109.6.213 27ms 28ms 28ms 
>   8  72.129.1.1 32ms 32ms 32ms 
>   9  72.129.1.7 27ms 26ms 25ms 
>  10   67.52.158.145 28ms 29ms 31ms 
>
>From TWC to Level3
>
> # ADDRESS RT1   RT2   RT3   STATUS
>
>2 24.43.183.345ms   5ms   6ms 
> 3 72.129.1.14 8ms   8ms   8ms
>
> 4 72.129.1.2  6ms   8ms   8ms
>
> 5 107.14.19.307ms   8ms   8ms
>
> 6 66.109.6.4  8ms   8ms   8ms
>
> 7 107.14.19.865ms   5ms   5ms
>
>8 66.109.9.12234ms  33ms  31ms  (TWC)   
><--
>
> 9 4.69.158.65 31ms  30ms  29ms  (L3)
>10 4.69.153.22133ms  33ms  34ms  
>11 4.69.133.20532ms  32ms  31ms
>
>
>I am showing, typically at night, a 20-40ms jump when hopping from
>Level3 to
>Time Warner and back in Tustin, CA.  This does not happen when using
>Cogent or
>other blended providers bandwidth.   I believe they are probably
>stuffing too
>many bits thru the peering there and wondering whats the best way to
>prove to
>them both (we pay for both) that they need to fix it.
>
>During non-peak traffic times these look normal (sub 10s).
>
>Sam
>
>
>!DSPAM:53b5890e239912186872586!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Next steps in extortion case - ideas?

[Charles N Wyble]

Sue him for slander? 

Contact the US DOJ and request extortion charges be filed? I mean if someone 
was committing a crime against me, I'd certainly be in contact with law 
enforcement to have charges filed and a warrant out for arrest. 

You shouldn't have called him. He has certainly changed his phone number. He 
also now most likely has your personal phone number. 

Contact law enforcement. That's what you should of done instead of calling him. 
I'd also consult your attorney. Ironically enough , the person you contacted 
could potentially try and turn the tables on you. Did you record the telephone 
conversation? 

On June 28, 2014 9:32:15 AM CDT, Markus  wrote:
>Hi list,
>
>nothing operational here, but there are many smart minds on this list 
>and people working for telcos, ISPs and law enforcement agencies, so 
>maybe you are willing to give me some advice in the following case:
>
>There's an individual out there on the web that has been blackmailing 
>hundreds of people and companies in a specific area of business for 
>years. His scheme is: 1. Contact the alleged "debtor" via e-mail and 
>inform him about an existing debt claim by a third party. 2. Offer the 
>debtor a deadline to pay the debt and warn the debtor if he shouldn't 
>pay he'll be prosecuted and his case will be "made public". 3. Once the
>
>deadline has elapsed, he'll publish completely false information made 
>out of thin air on the web, in particular Facebook, Twitter, a blog, a 
>website, including pictures of the debtor and serious accusations like 
>"This debtor is a child molestor" or "This debtor is part of the mafia"
>
>and other crazy stuff that you can usually only see in movies. All of 
>course with real names, company information (if applicable) and 
>basically everything he can find out about the debtor. 4. Then, the 
>individual hopes that the debtor will be intimidated because the debtor
>
>is afraid of the false information about him, which will show up on 
>Google etc., and will finally pay to get this false information removed
>
>from the web.
>
>In all cases the published "background information" about the debtors
>is 
>false, made out of thin air, and over the top. Just the names and 
>pictures are correct. Intentional slander in order to get the debtor to
>
>pay. If any of the published information was true, then every 2nd
>debtor 
>would be a child molestor and every other debtor part of the mafia.
>
>That individual is hiding his real identity really well, obviously, and
>
>he knows what he's doing. Domain hosted in Russia, taking good care his
>
>IP address won't show up in the mail headers, using false names and 
>identities, phone numbers registered through some DID provider who 
>doesn't collect personal information about the DID owner etc.
>
>I am one of the accused and had lots of false information about myself 
>and my company published by him. This is why I started to have an 
>interest to track his real identity down. I took 2 days out of my life 
>and researched high and low and finally found his personal phone number
>
>along with a name, a picture of him and several possible addresses (in 
>the US).
>
>I cannot be sure that the name, picture and addresses are correct, but
>I 
>called him on his personal phone number and after having spoken with
>him 
>before under his false identity, I can confirm that it's the same
>person 
>(the voice is the same). He was quite surprised to say the least.
>
>In case it matters, according to a LRN lookup the number belongs to 
>Omnipoint Communications, which is part of T-Mobile USA, I think.
>
>My idea is to somehow confirm his identity and confirm my research by 
>matching the voice of the false identity (available from a message he 
>left on my voicemail and also from his voicemail intro) to the real 
>person. I'm thinking about hiring a private investigator in the US (I'm
>
>in Germany) to drive up to the addresses I can provide the PI with and 
>find the person that matches the voice / maybe even the picture. The PI
>
>then must document the outcome in a way that it can be used in court. 
>I'm wanting to go the PI route because it will be the fastest way to 
>possibly gather evidence, I assume, as opposed to commissioning a
>lawyer 
>who will then in turn contact law enforcement etc.
>
>Unfortunately I do not have the authority to access the personal data
>of 
>the person that pays the monthly bill for the phone number that I
>called 
>him on, otherwise that would be the fastest way I suppose. I spent
>money 
>for some pay-sites that do some reverse phone lookup and stuff like 
>that, and although the information was helpful, I cannot be sure that 
>it's accurate.
>
>My goal is to confirm his real identity/name and address in order to 
>start a lawsuit and have a lawyer, or maybe even law enforcement, 
>investigate this case and ultimately, put an end to his slander 
>activities, not just for my case but for all hundreds before me and 
>those which are to com

Re: ipmi access

[charles]

On 2014-06-02 07:19, Andrew Latham wrote:
I use OpenVPN to access an Admin/sandboxed network with insecure 
portals,

wiki, and ipmi.



Same here.  My entire in band management plane (DRAC 
(disk/cpu/temperature etc telemetry to my OpenManage/Zenoss server), 
OpenSSH and 80/443 for backend stuffs) is all behind OpenVPN. Zero 
outside exposure.


Out of band, is a cyclades (acs48) directly on the internet with all my 
consoles hooked up and it controls daisy chained Cyclades PDUs. I have 
fairly strong passwords on it, everything is SSH.


How important is it to setup ACLs on it? Like say some VPS that's 
outside my infra and lock the Cyclades down to that? Is that really a 
much higher level of security?

Re: Off Topic Friday

[charles]

On 2014-05-30 16:09, Alain Hebert wrote:

Well happy friday.

We're planning to build a MPLS lab this summer.


What's this? Operational related content on a Friday? *angrily hurls 
popcorn across the room*. LOL.


MPLS lab sounds cool. For students? Already experienced engineers? 
Simulating a production environment? Tell us more details please! :)




If anyone has suggestion for cheap hardware to simulate a
implementation of all the RFC's from the Core to the CPE.


H. Good question. Any particular vendors you want to emulate? C/J I 
presume (maybe interop between them)? They do have official VM versions 
freely available of representative sampling of their product lines now 
days. Start there? Run it on a digtial ocean cloud? (They are currently 
$MY_FAV_CLOUD_PROVIDER) Or if you've got some spare server class kit 
kicking around your nearest colo then just load up Ubuntu 12.04 and 
Virtualbox (or heck, even w2k12/w8 which is what I'm using for my 
network vm lab). I've got...


HP (comware something something (v7? I dunno) . Oh and they have some 
cloud router deal to, I think I grabbed that.
Arista (got this at SCALE this year). Not sure if you can grab it 
"officially offically"

Cisco (asa, asr and that cloud router thing)
Juniper (mx emulator something someting)
Huewai (eNSP)
GNS3

I'm not at home right now, I'll put the links to the above item 
downloads on my lab wiki and update the thread (I'm embarrassed I didn't 
do that already).



Of course there is also OpenDaylight and other Linuxish stuff. But 
that's beyond MPLS. Linux/BSD MPLS seems to be not so great last I 
looked.





it would be greatly appreciated.


Same here.



From our research,

It is pretty much all over the place, like the vendor did it on
purpose to milk all the money they could from the spec :).


Haha.



As usual, off-list would be best.


If it's cool with 10k of my closest friends, I'd say this would be a 
welcome on list topic?

Re: IPAM DDI Software, Subscriber Management, CMDB and Per Customer VLANs

[charles]

On 2014-05-13 16:37, Kyle Leissner wrote:

I would like recommendations on the following software/hardware
elements required to run an access network. Assume you are building a
greenfield network using a combination of access technologies such as
DSL, GPON, AE, and WiFi.



What a timely thread! With all the talk the past several days about 
incumbents and lack of alternatives, I'm glad to see someone starting a 
new network!


If it's not ultra proprietary, what (major) geographical region are you 
looking to start in, how many homes/businesses do you intend to pass? Or 
is this all theoretical?


I've recently helped a coalation of non profits start an access network 
in Kansans City Missiouri/Kansas. It passes about 1,000 homes. Uses wifi 
exclusively. Meraki / Ubiquiti gear in the access layer, Ubiquiti gear 
in the backbone. We've been ironing out things like grounding/access to 
facilities, user access policies, dealing with bandwidth hogs etc etc. 
Now we are getting to the support suite and asking some of the same 
questions you are.


One thing I don't see you mention below is a network monitoring system. 
What are you using for that?




IPAM / DDI Solution: Needs full support for IPv6,


Of course. That's important.


Customer VLANs,


QinQ? You looking at offering metro-e services?

 RFC

1918,


ewww. v6 sir! Greenfield network and everything.


VRF, Overlapping Address Space,


ewww again. Those are horrible hacks, v6 all the things.


integration with DNS, DNSSEC,


So what does that mean? Create forward/reverse zone entries? Do you want 
to be able to delegate zone editing to customers? You'll need strong 
ACLs and what not. What does integration with DNSSEC mean to you?




Integration with DHCP,


v4? v6? SLACC? RADVD?



and integration with ARIN.


You mean the ARIN API? So you can setup auto SWIP?


 Looks like there are

both open source and commercial solutions available according to old
NANOG posts.


Indeed. I've been looking at http://nocproject.org/ which should cater 
to most of the above requirements.



 Which cater to service providers? Who are the leaders in

this space? Does anyone have experience with dealing with multiple
vendors?


Multiple vendors in what regard? You mean integrating offerings from 
multiple vendors?


Honestly I'd spend money on a couple good integration engineers. What 
you are looking for almost certainly will need a good amount of 
perl/python/bash glue to work. You could also throw money at proprietary 
solutions, which might get you what you want.




Subscriber Management/BRAS/BNG: Redback was the big player back in the
day, but I believe they are no longer. Juniper has their Subscriber
Management feature pack on their MX routers, and Cisco has their
Broadband Network Gateway on their ASR routers. Besides these two
vendors I am not sure what other solutions are out there. I believe
both of these solutions communicate upstream to external radius
servers and DHCP servers. Is anyone using Subscriber Management, or is
there another way of doing it?


What is subscriber management? You mean like provisioning and such?

Ah here is a description:
"Broadband Subscriber Management is a method of dynamically provisioning 
and managing subscriber access in a multiplay or triple play network 
environment. This method uses AAA configuration in conjunction with 
dynamic profiles to provide dynamic, per-subscriber authentication, 
addressing, access, and configuration for a host of broadband services 
including Internet access, gaming, IPTV, Video on Demand (VoD), and 
subscriber wholesaling."


We (Free Network Foundation) are doing this with RADIUS. FreeRadius on 
the backend, hostapd on the access layer (fairly heavily modified, we'll 
be submitting patches upstream soon), pfsense (with pfblocker, but used 
in a reverse manner). This gives us full AAA capabilities. It's somewhat 
"hacked" together, but our testing has seen good results so far. We hope 
to deploy in limited production test this weekend.





CMDB: A centralized database to keep track of all assets within the
network would be nice. I would assume this would need to tie in with
the IPAM solution and billing systems.



Yes. Agreed. I've not necessarily come up with a good system for this. 
I'm using a combination of Zenoss / Observium (will retire Observium 
once I have figured out the Zenoss API).




If you had your choice starting from the ground up how would you
deploy an access network today?


Well since I'm in the process of doing that:

v6 only (though to be honest, we are v4 right now, but heavily testing 
v6. Still lots of broken stuff, like gaming)

All FLOSS.
   Pfsense for internet edge (OSPF/BGP) routing, full l7 
firewalling/IDS/IPS, proxy/caching
   Zenoss (up/down, trending)  Observium (used as a CMDB, will be 
retiring for Zenoss soon)
   Slack/Rundeck (configuration management, command dispatching). Since 
everything is *NIX with a shell, I can just treat the routers/access 

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

[charles]

On 2014-05-14 02:04, Jean-Francois Mezei wrote:

On 14-05-13 22:50, Daniel Staal wrote:

They have the money.  They have the ability to get more money.  *They 
see

no reason to spend money making customers happy.*  They can make more
profit without it.


There is the issue of control over the market. But also the pressure
from shareholders for continued growth.



Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections).
More bandwidth across backbone pipes.


What is growth for access network?
More subscribers.

Except that the incumbent carriers have shown they have no interest in 
providing decent bandwidth to anywhere but the most profitable rate 
centers. I'd say about 2/3 of the USA is served with quite terrible 
access.





The problem with the internet is that while it had promises of wild
growth in the 90s and 00s, once penetration reaches a certain level,
growth stabilizes.


Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.



When you combine this with threath to large incumbents's media and 
media

distribution endeavours by the likes of Netflix (and cat videos on
Youtube), large incumbents start thinking about how they will be able 
to

continue to grow revenus/profits when customers will shift spending to
vspecialty channels/cableTV to Netflix and customer growth will not
compensate.


Except they aren't. Even in the most profitable rate centers, they've 
declined to really invest in the networks. They aren't a real business. 
You have to remember that. They have regulatory capture, natural/defacto 
monopoly etc etc. They don't operate in the real world of 
risk/reward/profit/loss/uncertainty like any other real business has to.




So they seek new sources of revenues, and/or attempt to thwart
competition any way they can.


No to the first. Yes to the second. If they were seeking new sources of 
revenue, they'd be massively expanding into un/der served markets and 
aggressively growing over the top services (which are fat margin). They 
did a bit of an advertising campaign of "smart home" offerings, but that 
seems to have never grown beyond a pilot.




The current trend is to "if you can't fight them, jon them" where
cablecos start to include the Netflix app into their proprietary 
set-top

boxes. The idea is that you at least make the customer continue to use
your box and your remote control which makes it easier for them to
switch between netflix and legacy TV.



True. I don't know why one of the cablecos hasn't licensed roku, added 
cable card and made that available as a "hip/cool" set top box offering 
and charge another 10.00 a month on top of the standard dvr rental.




Would be interesting to see if those cable companies that are agreeing
to add the Netflix app onto their proprietary STBs also  play peering
capacity games to degrade the service or not.


So how is the content delivered? Is it over the internet? Or is it over 
the cable plant, from cable headends?