Re: Level(3) ex-twtelecom midwest packet loss (4323)

[James Sink]

Seemed reasonably accurate to me.


[image: Phone.com] James SinkSenior Network Engineer
js...@phone.com(800) 997-9179 x506Try Phone.com free!
CONFIDENTIALITY NOTICE: This
e-mail and any attachments are for the exclusive and confidential use of
the intended recipient. If you received this in error, please do not read,
distribute, or take action in reliance upon this message. Instead, please
notify us immediately by return e-mail and promptly delete this message and
its attachments from your computer system.

On Fri, Aug 28, 2015 at 8:45 AM, Mel Beckman  wrote:

> Blake,
>
> There's no call to be blatantly offensive like that.
>
> -mel
>
> > On Aug 28, 2015, at 8:35 AM, Blake Dunlap  wrote:
> >
> > I'll just leave this here
> >
> >
> https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/
> >
> >> On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady 
> wrote:
> >> Mike, I would take it to mean someone screwed something up and they
> don't
> >> want to admit to it.  :-)  That's just a guess.
> >>
> >> --
> >>
> >> Jason Canady
> >> Unlimited Net, LLC
> >> Responsive, Reliable, Secure
> >>
> >> www.unlimitednet.us
> >> ja...@unlimitednet.us
> >> twitter: @unlimitednet
> >>
> >>
> >>> On 8/28/15 12:00 AM, Mike Hammett wrote:
> >>>
> >>> 08/28/2015 3:08 AM GMT
> >>> Event Conclusion Summary
> >>>
> >>> Start: August 27, 2015 13:20 GMT
> >>> Stop: August 28, 2015 00:00 GMT
> >>>
> >>> Root Cause: A protocol issue impacted IP services in multiple markets.
> >>> Fix Action: Adjustments were made to clear the errors.
> >>>
> >>> Summary:
> >>> The IP NOC began investigating the root cause with Tier III Technical
> >>> Support. It was reported that the issue was causing packet loss for
> >>> customers. Operations Engineering teams were engaged, and Field
> Services
> >>> were dispatched to a site in Chicago, IL to assist with investigations.
> >>> Troubleshooting identified a protocol issue, and Operations Engineering
> >>> worked with Tier III Technical Support to perform adjustments on the
> links.
> >>> It was confirmed that the errors cleared. The traffic load was also
> lowered
> >>> on cards in Chicago to alleviate any further issues. Should any
> additional
> >>> impact be experienced, please contact the Level 3 Technical Service
> Center.
> >>>
> >>> What the hell is a "protocol issue"?
> >>>
> >>> I'm not an idiot, you can tell me specifically what happened...
> >>>
> >>>
> >>>
> >>>
> >>> -
> >>> Mike Hammett
> >>> Intelligent Computing Solutions
> >>> http://www.ics-il.com
> >>>
> >>> - Original Message -
> >>>
> >>> From: "Ryan Gelobter" 
> >>> To: "Mel Beckman" 
> >>> Cc: "" 
> >>> Sent: Thursday, August 27, 2015 3:14:59 PM
> >>> Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)
> >>>
> >>> If you have access to the Level3 portal you should see ticket #9639047
> >>> under Network Events now.
> >>>
> >>> Event Summary:IP Network Event ~ Multiple Markets
> >>>
> >>> 08/27/2015 8:05 PM GMT
> >>>
> >>> Level 3 Tier III and Operations Engineering teams have identified
> Internet
> >>> Protocols dropping, affecting customer services. Restoration efforts
> are
> >>> in progress, but an estimated time of restoral is not available at this
> >>> time.
> >>>
> >>> 08/27/2015 6:36 PM GMT
> >>>
> >>> IP and Transport Tier III, Operations Engineering and Field Services
> >>> continue collaboratively working the issue.
> >>>
> >>>
> >>> 08/27/2015 4:59 PM GMT
> >>>
> >>> Operations Engineering is engaged and Field Services is on site in
> >>> Chicago,
> >>> IL investigating the issue.
> >>>
> >>>
> >>> 08/27/2015 4:38 PM GMT
> >>>
> >>> The engineers are currently migrating traffic in efforts of restoring
> >>> services while troubleshooting continues. Field Services is being
> >>> dispatched to a Chicago, IL site to assist.
> >>>
> >>> 08/27/2015 4:21 PM GMT
> >>>
> >>> IP services are affected across multiple markets and the root cause is
> >>> currently under investigation. The IP NOC and IP and Transport Tier III
> >>> are
> >>> actively troubleshooting and working to isolate the cause. The
> engineers
> >>> have detected peering issues which are resulting in packet loss for
> >>> customers. Please be advised that updates will be provided at minimum
> of
> >>> hourly unless otherwise noted.
> >>
>

Re: free Tools to monitor website performance

[James Sink]

Monitis has a free option called monitor.us and you can set up several
different kinds of probes in many locations.


[image: Phone.com] James SinkSenior Network Engineer
js...@phone.com(800) 997-9179 x506Try Phone.com free!
CONFIDENTIALITY NOTICE: This
e-mail and any attachments are for the exclusive and confidential use of
the intended recipient. If you received this in error, please do not read,
distribute, or take action in reliance upon this message. Instead, please
notify us immediately by return e-mail and promptly delete this message and
its attachments from your computer system.

On Sat, Aug 8, 2015 at 7:56 PM, Chaim Rieger  wrote:

> I think that Apica might have a free option as well. (www.apicasystem.com)
>
>
>
>
> On 08/08/2015 11:38 AM, Daren Darrow wrote:
>
>> Pingdom is the most affordable one I've seen recently.  You can try it out
>> with one URL for free
>> https://www.pingdom.com/free
>>
>> It also has some other nice free tools.
>> http://tools.pingdom.com/fpt/
>>
>>
>>
>> --
>> Daren Darrow, dar...@gmail.com
>>
>> On Thu, Aug 6, 2015 at 12:02 AM, Bryan Tong  wrote:
>>
>> Hello,
>>>
>>> We have been using Zabbix with great success on 900 hosts. I would
>>> recommend it, however I must agree the learning curve can be pretty
>>> steep.
>>> I think of Zabbix more like a piece of networking equipment where it wont
>>> do anything until everything is configured correctly. It is far from plug
>>> and play, but very powerful and flexible.
>>>
>>> Thanks
>>>
>>> On Thu, Aug 6, 2015 at 12:59 AM, David Hofstee 
>>> wrote:
>>>
>>> We use Zabbix for local monitoring. Quite powerful (Nagios crapped out a
 lot on larger setups, although 300 is not large). There is a learning

>>> curve
>>>
 for Zabbix.

 We have a few VPS'es outside our network for DNS reasons. They are
 configured as (pushing) monitoring nodes too. Bye,


 David Hofstee

 Deliverability Management
 MailPlus B.V. Netherlands (ESP)

 -Oorspronkelijk bericht-
 Van: NANOG [mailto:nanog-boun...@nanog.org] Namens sathish kumar Ippani
 Verzonden: Thursday, August 6, 2015 4:24 AM
 Aan: nanog@nanog.org
 Onderwerp: free Tools to monitor website performance

 Hi All,

 Thanks to all for reviewing my topic, may it is slightly off topic.

 We have almost 300 URL's (local and web) and we want to monitor few of
 them which are very critical URL's for web access and local access.

 I would like to know is there any free tool or software with I can use
 to
 monitor url performance in terms of response time. Which gives more
 information like how much time it taken to connect the server and time
 to
 load the page and total response time.

 Thanks in advance.



 --
 With Regards,

 Sathish Ippani


>>>
>>> --
>>> eSited LLC
>>> (701) 390-9638
>>>
>>>
>

RE: network quality measurement probes+reporting

[James Sink]

The licenses can get pricey, but AppNeta is worth a look.

http://www.appneta.com/products/pathview/

-James

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Saku Ytti
Sent: Tuesday, August 26, 2014 10:13 AM
To: nanog@nanog.org
Subject: network quality measurement probes+reporting

Anyone can recommend or even just name drop network quality measurement kit?

I'm only familiar with IP SLA, RPM and Creanord (and various inhouse tools:)

What I'd like to see
  - 1us or better one-way jitter (no need for clock sync, just accurate clock)
  - tens or 100us one-way latency (as good as cheaply can get sync, ntp is ok)
  - 1us or better RTT
  - any-to-any measurement, not just hub<->spoke (or sufficiently cheap hub)
  - 100pps to 100 measurement points on 8 CoS (ish, less may be acceptable)
  - randomized payload pattern + verification (to catch bit mangling)
  - randomized sport/dport (to put traffic in each ECMP/LAG combo, long term)
  - programmatic accesss and useful documents to measurement data (e.g. some 
OSS TSDB)
  - high quality, fast, useful graphical reporting and alarming
  - support for TWAMP and OWAMP responders
  - indicative price <100kEUR CAPEX and <2000EUR YRC for 100 nodes solution

-- 
  ++ytti

RE: CPE dns hijacking malware

[James Sink]

"Personally I have fond memories of going into my neighbor's router, flashing 
it with dd-wrt which allowed manual channel setting, and moving it off of the 
same wifi channel mine was on That was probably not a great idea, but you 
do what you have to sometimes."

Props on that, but wouldn't it have been easier to simply change your channel 
setting?
-James

-Original Message-
From: Tom Morris [mailto:bluen...@gmail.com] 
Sent: Tuesday, November 12, 2013 9:59 AM
Cc: NANOG list
Subject: Re: CPE dns hijacking malware

EXTREMELY common. Almost all Comcast Cable CPE has this same login, cusadmin / 
highspeed At least on AT&T U-Verse gear, there's a sticker on the modem with 
the password which is a hash of the serial number or something equally unique.

Almost all home routers also tend to have the default credentials.

I'm actually surprised it was this long before XSS exploits and similar garbage 
started hitting them.

Personally I have fond memories of going into my neighbor's router, flashing it 
with dd-wrt which allowed manual channel setting, and moving it off of the same 
wifi channel mine was on That was probably not a great idea, but you do 
what you have to sometimes.


On Tue, Nov 12, 2013 at 10:57 AM, Matthew Galgoci wrote:

> > Date: Tue, 12 Nov 2013 06:35:51 +
> > From: "Dobbins, Roland" 
> > To: NANOG list 
> > Subject: Re: CPE  dns hijacking malware
> >
> >
> > On Nov 12, 2013, at 1:17 PM, Jeff Kell  wrote:
> >
> > > (2) DHCP hijacking daemon installed on the client, supplying the
> hijacker's DNS servers on a DHCP renewal.  Have seen both, the latter 
> being more
> > > common, and the latter will expand across the entire home subnet 
> > > in
> time (based on your lease interval)
> >
> > I'd (perhaps wrongly) assumed that this probably wasn't the case, as 
> > the
> OP referred to the CPE devices themselves as being malconfigured; it 
> would be helpful to know if the OP can supply more information, and 
> whether or not he'd a chance to examine the affected CPE/end-customer setups.
> >
>
> I have encountered a family members provider supplied CPE that had the 
> web server exposed on the public interface with default credentials 
> still in place. It's probably more common than one would expect.
>
> --
> Matthew Galgoci
> Network Operations
> Red Hat, Inc
> 919.754.3700 x44155
> --
> "It's not whether you get knocked down, it's whether you get up." - 
> Vince Lombardi
>
>


--
--
Tom Morris, KG4CYX
Mad Scientist and Operations Manager, WDNA-FM 88.9 Miami - Serious Jazz!
786-228-7087
151.820 Megacycles

RE: How big is the Internet?

[James Sink]

Pretty big, but they gotta keep it trimmed down to fit on a floppy disk. 

Details within -> http://www.cidr-report.org

-James

-Original Message-
From: Sean Donelan [mailto:s...@donelan.com] 
Sent: Wednesday, August 14, 2013 7:32 AM
To: nanog@nanog.org
Subject: How big is the Internet?


Researchers have complained for years about the lack of good statistics about 
the internet for a couple fo decades, since the end of NSFNET statistics.

What are the current estimates about the size of the Internet, all IP networks 
including managed IP and private IP, and all telecommunications including 
analog voice, video, sensor data, etc?

CAIDA, ITU, Telegeography and some vendors like Cisco have released forecasts 
and estimates.  There are occasional pieces of information stated by companies 
in their investor documents (SEC 10-K, etc).

RE: Strange entries from AS1 in global table

[James Sink]

That's correct, I have seen L3 use that for MPLS as recently as a few months 
ago. 
-James

-Original Message-
From: Brad Fleming [mailto:bdfle...@gmail.com] 
Sent: Thursday, August 08, 2013 7:49 AM
To: Humberto Galiza
Cc: NANOG Mailing List
Subject: Re: Strange entries from AS1 in global table

I think Level(3) uses it for at least some L3 MPLS VPN stuff. We peer with that 
AS for dedicated SIP service transport for example.


On Aug 8, 2013, at 5:25 AM, Humberto Galiza  wrote:

> Looking at our routers I can see this:
> 3549 3356 26114 1 i
> 12956 1239 23520 23383 1 ?
> 
> but neither 26114 or 23383 are Brazilian ISP´s. Anyway, I guess it´s 
> probably leaked routes or even use of AS 1 as private one (I don´t 
> think level3 guys are using this AS anymore...).
> 
> Cheers,
> Humberto Galiza
> 
> 
> 2013/8/4 Anurag Bhatia :
>> Hello everyone
>> 
>> 
>> I was looking at global IPv4 table and saw some strange entries from AS1.
>> As per ARIN whois AS1 seems to be with Level3 but I noticed few 
>> prefixes of Brazil based ISP - Netvip
>> 
>> 
>> http://bgp.he.net/AS1#_prefixes
>> 
>> 
>> 
>> Looking at any prefix in detail, it seems like there are multiple 
>> ASNs announcing same prefix.
>> 
>> E.g 177.185.96.0/24 - is being announced by AS1 as well as AS52931 
>> (Netvip's allocated ASN). Same is true with 177.185.98.0/24, 
>> 177.185.98.0/24and so on.
>> http://bgp.he.net/net/177.185.96.0/24
>> 
>> So seems like AS1 acting like a mirror for all announcements of 
>> AS52931. To see who exactly gave "transit" to AS1 by Netvip in 
>> Brazil, I checked Oregon and noticed these routes:
>> 
>> 3356 3549 16735 52931 1 i
>> 
>> 
>> 
>> Seems like AS52931 itself is acting as transit for AS1 (and AS16735 
>> which seems like a backbone ISP in Brazil) is not filtering these 
>> routes further passing to Level3 (AS3549+AS3356).
>> 
>> 
>> 
>> I am curious to know what could be possible reason for an ASN like 
>> AS1 acting in exact mirror of AS52931? Could it be a case of internal 
>> use of
>> AS1 (assuming it to be private ASN)? May be it's a case of leaked 
>> internal routes?
>> 
>> 
>> 
>> Appreciate your time & answer.
>> 
>> 
>> 
>> Thanks.
>> 
>> --
>> 
>> Anurag Bhatia
>> anuragbhatia.com
>> 
>> Linkedin  | 
>> Twitter
>> Skype: anuragbhatia.com
> 

RE: tools and techniques to pinpoint and respond to loss on a path

[James Sink]

Have you looked into Cisco's OER?
-James

-Original Message-
From: Andy Litzinger [mailto:andy.litzin...@theplatform.com] 
Sent: Monday, July 15, 2013 2:19 PM
To: nanog@nanog.org
Subject: tools and techniques to pinpoint and respond to loss on a path

Hi,

Does anyone have any recommendations on how to pinpoint and react to packet 
loss across the internet?  preferably in an automated fashion.  For detection 
I'm currently looking at trying smoketrace to run from inside my network, but 
I'd love to be able to run traceroutes from my edge routers triggered during 
periods of loss.  I have Juniper MX80s on one end- which I'm hopeful I'll be 
able to cobble together some combo of RPM and event scripting to kick off a 
traceroute.  We have Cisco4900Ms on the other end and maybe the same thing is 
possible but I'm not so sure.

I'd love to hear other suggestions and experience for detection and also for 
options on what I might be able to do when loss is detected on a path.

In my specific situation I control equipment on both ends of the path that I 
care about with details below.

we are a hosted service company and we currently have two data centers, DC A 
and DC B.  DC A uses juniper MX routers, advertises our own IP space and takes 
full BGP feeds from two providers, ISPs A1 and A2.  At DC B we have a smaller 
installation and instead take redundant drops (and IP space) from a single 
provider, ISP B1, who then peers upstream with two providers, B2 and B3

We have a fairly consistent bi-directional stream of traffic between DC A and 
DC B.  Both of ISP A1 and A2 have good peering with ISP B2 so under normal 
network conditions traffic flows across ISP B1 to B2 and then to either ISP A1 
or A2

oversimplified ascii pic showing only the normal best paths:

  -- ISP A1--ISP B2--
DC A--| |---  
ISP B1 - DC B
 -- ISP A2--ISP B2--


with increasing frequency we've been experiencing packet loss along the path 
from DC A to DC B.  Usually the periods of loss are brief,  30 seconds to a 
minute, but they are total blackouts.

  I'd like to be able to collect enough relevant data to pinpoint the trouble 
spot as much as possible so I can take it to the ISPs and request a solution.  
The blackouts are so quick that it's impossible to log in and get a trace- 
hence the desire to automate it.

I can provide more details off list if helpful- I'm trying not to vilify 
anyone- especially without copious amounts of data points.

As a side question, what should my expectation be regarding packet loss when 
sending packets from point A to point B across multiple providers across the 
internet?  Is 30 seconds to a minute of blackout between two destinations every 
couple of weeks par for the course?  My directly connected ISPs offer me an 
SLA, but what should I reasonably expect from them when one of their upstream 
peers (or a peer of their peers) has issues?  If this turns out to be BGP 
reconvergence or similar do I have any options?

many thanks,
-andy