Anyone from Charter or Spectrum?

[Joseph Jenkins]

Seeing some issues with packet loss from my users on Charter and Spectrum
in California when trying to access my network on Level3/Lumen. Trying to
figure out if there are any issues on the Charter or Spectrum side and how
to help out my users.

Anyone with experience working the Oracle Cloud and IPSec connections?

[Joseph Jenkins]

We are in the process of setting up connectivity to the Oracle Cloud over
ipsec connections. I have it setup with 3 /16 subnets to route back to our
on premise network. The tunnel comes up when interesting traffic comes from
one of the subnets, however when traffic is generated from another subnet
the first subnet stops getting return traffic from OCI. Anyone with any
experience that can shed some light on this?

Re: Centurylink having a bad morning? [EXTERNAL]

[Joseph Jenkins]

Well at least it looks like the issue is starting to resolve  and stuff is
coming back up.

On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes <
mattli...@rivervalleyinternet.net> wrote:

> Is this what happens when your entire network is database driven?
>

Re: Centurylink having a bad morning?

[Joseph Jenkins]

AS3356 is the Level3 internet.

On Sun, Aug 30, 2020 at 8:09 AM Ian Bowers  wrote:

> AS3356 is the one I've seen all the chatter about this morning.
>
> On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor 
> wrote:
>
>> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
>> > Woke up this morning to a bunch of reports of issues with connectivity
>> > had to shut down some Level3/CTL connections to get it to return to
>> normal.
>>
>>
>>
>> Just to confirm we're seeing this on AS3356 and not AS209, correct?
>>
>>
>> We have links to both and shut down AS3356 which seems to have cleared
>> "most" of the problems.
>>
>>
>> --
>> inoc.net!rblayzor
>> XMPP: rblayzor.AT.inoc.net
>> PGP:  https://pgp.inoc.net/rblayzor/
>>
>

Re: Centurylink having a bad morning?

[Joseph Jenkins]

That might be because of this:

The IP NOC with the assistance of the Operations Engineering team confirmed
a routing issue to be preventing BGP sessions from establishing correctly.
A configuration adjustment was deployed at a high level, and sessions began
to re-establish with stability. As the change propagates through the
affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your
services locally at your equipment, or manually reset your BGP session. If
after that action has been performed a service issue prevails, please
contact the CenturyLink Repair Center for troubleshooting assistance

On Sun, Aug 30, 2020 at 7:42 AM Drew Weaver  wrote:

> Something just now changed in this situation and now it seems to have
> gotten worse.
>
>
>
> *From:* Jason Kuehl 
> *Sent:* Sunday, August 30, 2020 10:00 AM
> *To:* Drew Weaver 
> *Cc:* R. Leigh Hennig ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> People are rebooting ghosting now.
>
>
>
> https://twitter.com/ir_kujoe/status/1300066569645707265
>
>
>
> Seeing other reports of this too.
>
>
>
> On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver 
> wrote:
>
> That site seems to be just for their cloud products, is there one of these
> for their actual network?
>
>
>
> *From:* R. Leigh Hennig 
> *Sent:* Sunday, August 30, 2020 8:54 AM
> *To:* Drew Weaver ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> Global impact with issues reported by Fastly, Cloudflare, OpenDNS.
>
>
>
> https://status.ctl.io/
>
>
> STARTED
>
> Sun Aug 30 2020 08:13 (EDT)
> Sun Aug 30 2020 12:13 (UTC)
> AFFECTED SERVICES
>
> External Cloud Network (CA3)
> DATE
> LATEST UPDATE
>
> Sun Aug 30 2020 08:13 (EDT)
> Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue
> affecting some services in the CA3 data center. Ensuring the reliability of
> our services is our top priority. We will continue to provide status
> updates as this incident progresses. If you need further support, please
> contact us at h...@ctl.io.
>
>
>
>
>
>
>
> Sent from ProtonMail Mobile
>
>
>
>
>
> On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG 
> wrote:
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>
>
>
>
>
>
>
> --
>
> Sincerely,
>
> Jason W Kuehl
> Cell 920-419-8983
> jason.w.ku...@gmail.com
>

Re: Centurylink having a bad morning?

[Joseph Jenkins]

Latest updates from my tickets:
08/30/2020 14:28:20 GMT - The IP NOC confirmed a routing issue and
commenced with troubleshooting efforts. Routing configuration adjustments
have been made and service affecting alarms are beginning to clear.

08/30/2020 11:38:15 GMT - The IP NOC is engaged in cooperative escalated
investigations to isolate and troubleshoot the fault at this time.

08/30/2020 11:03:09 GMT - On August 30, 2020 at 10:00 GMT, CenturyLink
identified a Market Wide service impact. As this network fault is impacting
multiple clients, the event has increased visibility with CenturyLink
leadership. As such, client trouble tickets associated to this fault have
been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please
be advised that updates for this event will be relayed at a minimum of
hourly unless otherwise noted. The information conveyed hereafter is
associated to live troubleshooting effort and as the discovery process
evolves through to service resolution, ticket closure, or post incident
review, details may evolve.



On Sun, Aug 30, 2020 at 7:30 AM Antonios Chariton 
wrote:

> Reporting from Europe, any IP with them in the path is unreachable from
> various providers. I guess they wanted to try IPv6-only.. :P IPv6 is fine,
> working fine, IPv4 not at all..
>
> Antonis
>
> > On 30 Aug 2020, at 14:58, Tomas Lynch  wrote:
> >
> > Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose.
> It is also affecting some data centers in Europe too. but haven't seen
> flaps there, just suboptimal routing.
> >
> > On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver 
> wrote:
> > Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
> >
> >
> >
> > From: Tomas Lynch 
> > Sent: Sunday, August 30, 2020 8:45 AM
> > To: Mel Beckman 
> > Cc: Drew Weaver ; nanog@nanog.org
> > Subject: Re: Centurylink having a bad morning?
> >
> >
> >
> > BGP sessions randomly flapping or having routing issues in different
> cities since ~5AM EST
> >
> >
> >
> > On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman  wrote:
> >
> > The CL portal loads for me, and I can log in, but it is slower than
> usual. Not seeing traffic issues on our CL circuits.
> >
> > -mel via cell
> >
> >
> >
> >
> > On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
> wrote:
> >
> > 
> >
> > Hello,
> >
> >
> >
> > Woke up this morning to a bunch of reports of issues with connectivity
> had to shut down some Level3/CTL connections to get it to return to normal.
> >
> >
> >
> > As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
> >
> >
> >
> > Just wondering what others are seeing.
> >
> >
> >
>
>

Re: Centurylink having a bad morning?

[Joseph Jenkins]

Now if you call into CL you get a message stating their technicians are
working on an ip outage.

On Sun, Aug 30, 2020 at 6:56 AM Chase Christian  wrote:

> Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00
> Pacific:
>
> Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y
> (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
> Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state Established event HoldTime new state Idle
> Aug 30 03:07:37 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state OpenConfirm event RecvKeepAlive new state Established
> Aug 30 03:15:38 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state Established event HoldTime new state Idle
> Aug 30 03:17:15 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state OpenConfirm event RecvKeepAlive new state Established
> Aug 30 03:19:55 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor
> 4.35.X.Y+52091 (proto) 6/7 (Cease/connection collision resolution) 0 bytes
> Aug 30 03:20:11 rtr02 Rib: %BGP-3-NOTIFICATION: received from neighbor
> 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
> Aug 30 03:20:11 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state Established event RecvNotify new state Idle
>
> And incoming traffic from AS3356 and AS209 both dropped to very low
> volumes.
>
> On Sun, Aug 30, 2020 at 5:58 AM Jason Kuehl 
> wrote:
>
>> Well, When I tried calling I got a fast busy, so that's nice.
>>
>> On Sun, Aug 30, 2020 at 8:33 AM David Hubbard <
>> dhubb...@dino.hostasaurus.com> wrote:
>>
>>> Same.  Also, as reported on outages list, what’s even worse is that they
>>> appear to be continuing to propagate advertisements from circuits whose
>>> sessions have been turned down.  I validated ours still were via a couple
>>> looking glass portals.  Down Detector shows nearly every major service
>>> provider impacted.
>>>
>>>
>>>
>>> They’re not reachable so who knows if they’re even working on it.  I
>>> feel like they’ve been cutting heavily on the network ops side in recent
>>> years…
>>>
>>>
>>>
>>> *From: *NANOG 
>>> on behalf of Drew Weaver via NANOG 
>>> *Reply-To: *Drew Weaver 
>>> *Date: *Sunday, August 30, 2020 at 8:23 AM
>>> *To: *"nanog@nanog.org" 
>>> *Subject: *Centurylink having a bad morning?
>>>
>>>
>>>
>>> Hello,
>>>
>>>
>>>
>>> Woke up this morning to a bunch of reports of issues with connectivity
>>> had to shut down some Level3/CTL connections to get it to return to normal.
>>>
>>>
>>>
>>> As of right now their support portal won’t load:
>>> https://www.centurylink.com/business/login/
>>>
>>>
>>>
>>> Just wondering what others are seeing.
>>>
>>>
>>>
>>
>>
>> --
>> Sincerely,
>>
>> Jason W Kuehl
>> Cell 920-419-8983
>> jason.w.ku...@gmail.com
>>
>

Re: Centurylink having a bad morning?

[Joseph Jenkins]

Finally got through on their support line and spoke to level1. The only
thing the tech could say was it was an issue with BGP route reflectors and
it started about 3am(pacific). They were still trying to isolate the issue.
I've tried failing over my circuits and no go, the traffic just dies as L3
won't stop advertising my routes.

On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
wrote:

> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>

Question on BlackBox or Commworks

[Joseph Jenkins]

Do you know or have experience with either company? Do they have their own
techs are they just bidding out for local techs in the area? I have work
that needs to be done all across the US and just trying to look for some
options.

Re: anyone on from hotmail.com, msn.com, live.com smtp?

[Joseph Jenkins]

No not at all, just replaced with a random address.

So nothing from MS as of yet, best I have gotten is these two addresses to
go and file complaints. It's been 2 days with no response, MS isn't so
attentive it seems.

https://sender.office.com

https://support.microsoft.com/en-us/supportrequestform/8ad563e3-288e-2a61-8122-3ba03d6b8d75

So at this point we are just queuing up the messages waiting for MS to
allow us to send again. What a pain.

On Sat, May 9, 2020 at 12:22 PM Robert Story  wrote:

> On Fri 2020-05-08 19:35:08-0700 Joseph wrote:
> > We are getting this messages when sending emails from our domain.
> > We've submitted tickets, but haven't received a response yet. Anyone
> > have any insights?
> >
> > 550 5.7.1 Unfortunately, messages from [1.1.1.1] weren't sent.
>
>
> > It's our own address and everything on the Microsoft site says we are
> > clean with no issues.
>
> 1.1.1.1 is your address? You are cloudfare's anycast dns server?
>
>
> $ whois 1.1.1.1
> [...]
> inetnum:1.1.1.0 - 1.1.1.255
> netname:APNIC-LABS
> descr:  APNIC and Cloudflare DNS Resolver project
>
> --
> Robert Story 
> USC Information Sciences Institute 
>

anyone on from hotmail.com, msn.com, live.com smtp?

[Joseph Jenkins]

We are getting this messages when sending emails from our domain. We've
submitted tickets, but haven't received a response yet. Anyone have any
insights?

550 5.7.1 Unfortunately, messages from [1.1.1.1] weren't sent. Please
contact your Internet service provider since part of their network is on
our block list (S3140). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors. [
AM5EUR03FT014.eop-EUR03.prod.protection.outlook.com]

It's our own address and everything on the Microsoft site says we are clean
with no issues.

Anyone from Home Town Communications in Florida that can contact me off list?

[Joseph Jenkins]

Need some help tracking down a device.

Thank you,

Joe

Re: QFX5k question

[Joseph Jenkins]

I have 4 QFX51xx switches in a virtual chassis and have no problems pushing
that much traffic through them for several hundred servers with 10GB
uplinks.


On March 23, 2019 at 12:42:52 PM, Mehmet Akcin (meh...@akcin.net) wrote:

Hey there,

I am trying to get my hands on some QFX5000s and I have a rather quick
question.

In the past, I often used MX + EX where MX did routing and I connected all
uplinks/peering and EX, and EX did switching, i connected my servers to ex.

in QFX, I am trying to see if I need EX or not? more importantly (besides
from what juniper papers say) are there any known issues people run into
for a small scale deployment. (100mbps-1gbps range 1 rack, 20 servers)

my plan is to have QFX to it all, but i am worried, if this is too much for
QFX, if you have relative experience on this , feel free to let me know

thanks in advance

mehmet

Re: Centurylink/Level3 Plans

[Joseph Jenkins]

So the reps/SEs/Techs that I have spoken with are saying they are going to
keep them separate. In my case I am tw customer and was worried about their
network when L3 took them over. I was even more worried when Centurylink
then took over L3 that I was going to lose one of the networks. However I
gotten assurances from everyone there that they are going to maintain all 3
networks as separate for the foreseeable future.


On July 19, 2018 at 11:15:39 AM, James Breeden (ja...@arenalgroup.co) wrote:

Does anyone know what the plans or even watercooler chat is concerning
Centurylink (AS209) and Level3 (AS3356) integration, direct peering, or
continuing separation?

We are looking at a IP Transit deal involving one or both networks and
while I'd love to have transit routes from both, I don't want to design to
be shot in the foot later on either if they are talking about soon-to-be
integrated or something.

TIA...

James W. Breeden
Managing Partner

[logo_transparent_background]
Arenal Group: Arenal Consulting Group | Acilis Telecom | Pines Media
PO Box 1063 | Smithville, TX 78957
Email: ja...@arenalgroup.co | office
512.360. | cell 512.304.0745 | www.arenalgroup.co<
http://www.arenalgroup.co>

Re: Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Joseph Jenkins]

This are also no new vlans being used at all. They are all already existing
on the switches involved and nothing is being added. In fact what makes
this even weirder is that I already have that exact same port configuration
running on port 1/0/67 of the Juniper and it doesn't cause me any issues
nor did it cause any issues when the config was applied. This existing port
1/0/67 has gone down/up as the firewall has been rebooted and doesn't cause
any issues or hiccups on the network. For reference the attached firewall
is an ASA which doesn't do spanning tree anyways.

set interfaces ge-1/0/67 description "Firewall Port-2"
set interfaces ge-1/0/67 unit 0 family ethernet-switching interface-mode
trunk
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 9-10
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 29
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 31-32
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 43
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 50-51
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 56
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 58
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 66
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 68
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 90
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 143
set interfaces ge-1/0/67 unit 0 family ethernet-switching vlan members 170

On Thu, Apr 5, 2018 at 2:34 PM, Joseph Jenkins 
wrote:

> Steve let me clarify the config I am applying has nothing to do with an
> LACP trunk or any of my existing LACP trunks. It is a completely different
> configuration on a completely different interface, the only similarity is
> that I am trying to configure a trunk interface on the Juniper side for
> multiple vlans. There is no LACP configuration involved.
>
> On Thu, Apr 5, 2018 at 2:26 PM, Naslund, Steve 
> wrote:
>
>> It really does not resolve anything it just allows a bad configuration to
>> work.  The guard is there so that if one side is configured as a channel
>> and the other side is not, the channel gets shut down.  Allowing it to
>> remain up can cause a BPDU loop.  Your spanning tree is trying to tell you
>> something, you should listen or you could get really hard to isolate issues.
>>
>> Steven Naslund
>> Chicago IL
>>
>> >-Original Message-
>> >From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joseph Jenkins
>> >Sent: Thursday, April 05, 2018 4:16 PM
>> >To: Robert Webb
>> >Cc: nanog@nanog.org
>> >Subject: Re: Juniper Config Commit causes Cisco Etherchannels to go into
>> err-disable state
>> >
>> >No there isn't, but from what I am getting responses both onlist and off
>> list is to just run this on the Cisco switches:
>> >
>> >no spanning-tree etherchannel guard misconfig
>> >
>> >and that should resolve the issue.
>> >
>> >Thanks Everyone.
>>
>>
>

Re: Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Joseph Jenkins]

Steve let me clarify the config I am applying has nothing to do with an
LACP trunk or any of my existing LACP trunks. It is a completely different
configuration on a completely different interface, the only similarity is
that I am trying to configure a trunk interface on the Juniper side for
multiple vlans. There is no LACP configuration involved.

On Thu, Apr 5, 2018 at 2:26 PM, Naslund, Steve  wrote:

> It really does not resolve anything it just allows a bad configuration to
> work.  The guard is there so that if one side is configured as a channel
> and the other side is not, the channel gets shut down.  Allowing it to
> remain up can cause a BPDU loop.  Your spanning tree is trying to tell you
> something, you should listen or you could get really hard to isolate issues.
>
> Steven Naslund
> Chicago IL
>
> >-Original Message-
> >From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joseph Jenkins
> >Sent: Thursday, April 05, 2018 4:16 PM
> >To: Robert Webb
> >Cc: nanog@nanog.org
> >Subject: Re: Juniper Config Commit causes Cisco Etherchannels to go into
> err-disable state
> >
> >No there isn't, but from what I am getting responses both onlist and off
> list is to just run this on the Cisco switches:
> >
> >no spanning-tree etherchannel guard misconfig
> >
> >and that should resolve the issue.
> >
> >Thanks Everyone.
>
>

Re: Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Joseph Jenkins]

No there isn't, but from what I am getting responses both onlist and off
list is to just run this on the Cisco switches:

no spanning-tree etherchannel guard misconfig

and that should resolve the issue.

Thanks Everyone.

On Thu, Apr 5, 2018 at 2:10 PM, Robert Webb  wrote:

> I don't see any issue with the snippet of the config you provided for the
> "Firewall Port". Is there a chance that the port ge-0/0/67 is referenced
> somewhere else in the Juniper config that when applying your trunk setup is
> causing issues?
>
> Just throw that out off the top of my head and not really thinking it
> through.
>
> Robert
>
> -----Original Message-
> From: NANOG  On Behalf Of Joseph Jenkins
> Sent: Thursday, April 5, 2018 4:58 PM
> To: nanog@nanog.org
> Subject: Juniper Config Commit causes Cisco Etherchannels to go into
> err-disable state
>
> I have cases open with both Cisco and Juniper on this, but wanted to see
> if anyone else had seen an issue like this because support has no idea.
>
> I have a Juniper QFX 5100 Core running in Virtual Chassis mode with 4
> switches. I have 4 separate stacks of Cisco 3750 switches with 2x1GB
> uplinks bound into 4 different LACP trunks. I have had it happen twice now
> where I apply a trunk port config(not an LACP trunk) to a port that isn't a
> part of any of the LACP trunks and it causes all 4 of the Etherchannels on
> the Cisco stacked switches to go into an err-disable state with these
> messages:
>
> Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
> on Gi1/0/48, putting Gi1/0/48 in err-disable state
>
> Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
> on Po17, putting Gi1/0/48 in err-disable state
>
> Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
> on Po17, putting Po17 in err-disable state
>
> Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface
> GigabitEthernet1/0/48, changed state to down
>
> Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
> on Gi2/0/48, putting Gi2/0/48 in err-disable state (CA-TOR-1-7-2)
>
> Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface
> GigabitEthernet2/0/48, changed state to down
>
> Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface
> Port-channel17, changed state to down
>
> Here is the config I am applying to the port that has caused this issue to
> happen twice now:
>
> set interfaces ge-0/0/67 description "Firewall Port"
> set interfaces ge-0/0/67 unit 0 family ethernet-switching interface-mode
> trunk set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan
> members 9-10 set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan
> members 29 set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan
> members 31-32 set interfaces ge-0/0/67 unit 0 family ethernet-switching
> vlan members 43 set interfaces ge-0/0/67 unit 0 family ethernet-switching
> vlan members 50-51 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 56 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 58 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 66 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 68 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 90 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 143 set interfaces ge-0/0/67 unit 0 family
> ethernet-switching vlan members 170
>
> The issue happens within a couple of minutes of committing the config on
> the Juniper side, there are no cables plugged into port 0/0/67 so
> technically there shouldn't be any BPDU's sent out since there isn't a port
> change.
>
> Juniper Support wants me to turn on trace option and then run though a
> bunch of scenarios, the issue is that testing this takes down my network.
>
> Just wanted to put it out there to see if anyone else had run into a
> situation similar to this.
>
> TIA
>
> Joe
>

Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Joseph Jenkins]

I have cases open with both Cisco and Juniper on this, but wanted to see if
anyone else had seen an issue like this because support has no idea.

I have a Juniper QFX 5100 Core running in Virtual Chassis mode with 4
switches. I have 4 separate stacks of Cisco 3750 switches with 2x1GB
uplinks bound into 4 different LACP trunks. I have had it happen twice now
where I apply a trunk port config(not an LACP trunk) to a port that isn't a
part of any of the LACP trunks and it causes all 4 of the Etherchannels on
the Cisco stacked switches to go into an err-disable state with these
messages:

Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
on Gi1/0/48, putting Gi1/0/48 in err-disable state

Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
on Po17, putting Gi1/0/48 in err-disable state

Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
on Po17, putting Po17 in err-disable state

Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet1/0/48, changed state to down

Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected
on Gi2/0/48, putting Gi2/0/48 in err-disable state (CA-TOR-1-7-2)

Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet2/0/48, changed state to down

Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface
Port-channel17, changed state to down

Here is the config I am applying to the port that has caused this issue to
happen twice now:

set interfaces ge-0/0/67 description "Firewall Port"
set interfaces ge-0/0/67 unit 0 family ethernet-switching interface-mode
trunk
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 9-10
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 29
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 31-32
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 43
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 50-51
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 56
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 58
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 66
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 68
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 90
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 143
set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 170

The issue happens within a couple of minutes of committing the config on
the Juniper side, there are no cables plugged into port 0/0/67 so
technically there shouldn't be any BPDU's sent out since there isn't a port
change.

Juniper Support wants me to turn on trace option and then run though a
bunch of scenarios, the issue is that testing this takes down my network.

Just wanted to put it out there to see if anyone else had run into a
situation similar to this.

TIA

Joe

gmail for business email contact - server blacklisted

[Joseph Jenkins]

Anyone on here from gmail that can help with an issue. One of your email
servers has been blacklisted by spam cop and is still in rotation. It's
causing individual senders to be blocked on our side. If possible can you
remove that server from rotation?

IP is: 209.85.161.176

Re: Google Captcha on web searches

[Joseph Jenkins]

I have about a 600 users. We aren’t dual stick only ipv4 at this point. Someone 
contacted me off list and gave me some insight as to what to key on.


Joe 

> On Nov 10, 2015, at 9:48 AM, Josh Luthman  wrote:
> 
> It's done per /32 I believe.  Do you have a lot of NATed users?
> 
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> On Nov 10, 2015 12:29 PM, "Joseph Jenkins"  <mailto:j...@breathe-underwater.com>> wrote:
> We started getting a Google Captcha for our web searches this morning. Does 
> anyone have contact info for Google so that I can contact them and figure out 
> where the traffic is coming from on my side or what service it is going to so 
> that I can track down the users?
> 
> Thanks,
> 
> Joe

Google Captcha on web searches

[Joseph Jenkins]

We started getting a Google Captcha for our web searches this morning. Does 
anyone have contact info for Google so that I can contact them and figure out 
where the traffic is coming from on my side or what service it is going to so 
that I can track down the users?

Thanks,

Joe Jenkins
909.636.2097

Re: DDoS Mitigation

[Joseph Jenkins]

Depends on the service, you might have better luck with versign or prolexic and 
they can get the services up and running quickly.
Joe Jenkins
909.636.2097

> On Nov 4, 2015, at 9:33 AM, Mario Eirea  wrote:
> 
> Hello everyone,
> 
> Looking to find out how the pricing model works for DDoS mitigation and what 
> to expect as far as ballpark pricing from my ISP. Some background, we are 
> getting hit with a chargen attack that comes and goes and is saturating our 
> 500mb connection. Tried hitting up the ISP for UDP block on 19 but they want 
> us to go through our rep, in the process making this go on longer that is 
> necessary. Any feedback would be appreciated.
> 
> Thanks,
> 
> -ME

Re: Level3 routing issue US west coast

[Joseph Jenkins]

Level3 had an issue with one of their core routers in Los Angeles last 
night(7pm Pacific) and early this morning(1am Pacific). Last update to my 
trouble ticket had the issue still being reviewed by engineering, but that a 
core router was dropping packets.

> On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch  wrote:
> 
> Hi,
> 
> does anyone else experience issues with the Level3 network at the US west 
> coast? We see lots of broken paths like this:
> 
>  Packets   
> Pings
> Host   Loss%   Snt   Last   Avg  
> Best  Wrst StDev
> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com0.0%   2310.6   0.5   
> 0.2  18.1   1.2
> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com0.0%   2310.5   9.9   
> 0.3 361.1  40.1
> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com0.0%   2306.5   7.7   
> 6.3  49.7   5.3
> 4. win-b4-link.telia.net0.0%   2306.6   6.8   
> 6.4  20.2   1.5
> 5. level-ic-1573273-wien-b4.c.telia.net 0.0%   2306.6   9.3   
> 6.3  69.1   9.6
> 6. ae-2-70.edge1.SanJose3.Level3.net   38.4%   230  164.8 165.0 
> 164.5 194.9   2.6
> 7. ae-2-70.edge1.SanJose3.Level3.net   45.9%   230  164.7 164.8 
> 164.5 174.1   0.9
> 8. 4.53.208.10234.3%   230  634.9 310.7 
> 168.5 680.1 199.7
> 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1%   230  412.0 455.2 
> 304.9 954.6 203.4
> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6%   230  323.4 441.4 
> 323.1 822.0 182.1
> 11. 211.115.201.92  38.4%   230  289.8 412.9 
> 289.6 846.7 185.4
> 12. 61.250.89.2 35.8%   230  290.6 439.4 
> 290.2 804.3 205.1
> 13. ???
> 
> Trace from NYC is also broken:
> 
>  Packets  
>  Pings
> Host   Loss%   Snt   Last   
> Avg  Best  Wrst StDev
> 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com0.0%300.4   
> 4.3   0.4  57.8  13.3
> 2. nyk-b5-link.telia.net0.0%300.3   
> 0.4   0.3   0.9   0.1
> 3. ???
> 4. ae-3-80.edge1.SanJose3.Level3.net   17.2%30   71.7  
> 73.2  71.7  98.7   5.5
> 5. ae-3-80.edge1.SanJose3.Level3.net0.0%30   71.8  
> 71.8  71.7  72.0   0.1
> 6. 4.53.208.10231.0%30  569.6 
> 250.7  70.5 579.3 231.2
> 7. 63.218.250.73   31.0%30  672.6 
> 355.5 178.0 672.6 232.1
> 
> 
> At 10:24 UTC+2 it was even more broken:
> 
>  Packets   
> Pings
> Host   Loss%   Snt   Last   Avg  
> Best  Wrst StDev
> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com0.0%   3260.4   0.5   
> 0.3  39.7   2.2
> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com0.0%   3260.5   6.7   
> 0.3 198.1  26.3
> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com0.0%   3266.6   7.6   
> 6.4  43.6   4.5
> 4. win-b4-link.telia.net0.0%   3266.7   7.4   
> 6.3  43.1   3.2
> 5. level-ic-1573273-wien-b4.c.telia.net 0.0%   3266.9   9.2   
> 6.3  73.2  10.1
> 6. ae-1-60.edge5.LosAngeles1.Level3.net62.6%   326  164.7 165.5 
> 164.5 176.7   1.5
>ae-2-70.edge1.SanJose3.Level3.net
> 7. ae-1-60.edge5.LosAngeles1.Level3.net63.1%   326  164.8 165.8 
> 164.6 204.2   3.9
>ae-2-70.edge1.SanJose3.Level3.net
> 8. 205.129.5.7074.2%   326  799.9 487.2 
> 169.0 799.9 305.7
>4.53.208.102
> 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2%   326  1359. 701.0 
> 308.7 3716. 510.6
> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1%   326  960.4 643.0 
> 323.4 960.4 307.6
> 11. 211.115.201.92  68.9%   326  925.3 674.2 
> 289.8 932.3 296.6
> 12. 61.250.89.2 72.9%   326  928.5 637.2 
> 291.9 928.5 304.3
> 13. ???
> 
> 
> best regards
> 
> Jürgen Jaritsch
> Head of Network & Infrastructure
> 
> ANEXIA Internetdienstleistungs GmbH
> 
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
> 
> E-Mail: j...@anexia.at
> Web: http://www.anexia.at
> 
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
> 

Re: BGPMON Alert Questions

[Joseph Jenkins]

Tried the recipients mailbox is full, but it looks like all of the bgpmon
alerts have cleared.


On Wed, Apr 2, 2014 at 1:40 PM, Aris Lambrianidis wrote:

> Contacted ip@indosat.com about this, I urge others to do the same.
>
> --Aris
>
>
> On Wed, Apr 2, 2014 at 9:33 PM, Andrew (Andy) Ashley
> wrote:
>
> > Hi All,
> >
> > I am a network admin for Aware Corporation AS18356 (Thailand), as
> > mentioned in the alert.
> > We operate a BGPMon PeerMon node on our network, which peers with the
> > BGPMon service as a collector.
> >
> > It is likely that AS4761 (INDOSAT) has somehow managed to hijack these
> > prefixes and CAT (Communications Authority of Thailand AS4651) is not
> > filtering them,
> > hence they are announced to us and are triggering these BGPMon alerts.
> >
> > I have had several mails to our NOC about this already and have responded
> > directly to those.
> > I suggest contacting Indosat directly to get this resolved.
> > AS18356 is a stub AS, so we are not actually advertising these learned
> > hijacked prefixes to anyone but BGPMon for data collection purposes.
> >
> > Thanks.
> >
> > Regards,
> >
> > Andrew Ashley
> >
> > Office: +27 21 673 6841
> > E-mail: andre...@aware.co.th
> > Web: www.aware.co.th
> >
> >
> >
> > On 2014/04/02, 21:05, "Vlade Ristevski"  wrote:
> >
> > >I just got the same alert for one of my prefixes one minute ago.
> > >
> > >On 4/2/2014 2:59 PM, Frank Bulk wrote:
> > >> I received a similar notification about one of our prefixes also a few
> > >> minutes ago.  I couldn't find a looking glass for AS4761 or AS4651.
> > >>But I
> > >> also couldn't hit the websites for either AS, either.
> > >>
> > >> Frank
> > >>
> > >> -Original Message-
> > >> From: Joseph Jenkins [mailto:j...@breathe-underwater.com]
> > >> Sent: Wednesday, April 02, 2014 1:52 PM
> > >> To: nanog@nanog.org
> > >> Subject: BGPMON Alert Questions
> > >>
> > >> So I setup BGPMON for my prefixes and got an alert about someone in
> > >> Thailand announcing my prefix.  Everything looks fine to me and I've
> > >> checked a bunch of different Looking Glasses and everything announcing
> > >> correctly.
> > >>
> > >> I am assuming I should be contacting the provider about their
> > >> misconfiguration and announcing my prefixes and get them to fix it.
>  Any
> > >> other recommendations?
> > >>
> > >> Is there a way I can verify what they are announcing just to make sure
> > >>they
> > >> are still doing it?
> > >>
> > >> Here is the alert for reference:
> > >>
> > >> Your prefix:  8.37.93.0/24:
> > >>
> > >> Update time:  2014-04-02 18:26 (UTC)
> > >>
> > >> Detected by #peers:   2
> > >>
> > >> Detected prefix:  8.37.93.0/24
> > >>
> > >> Announced by: AS4761 (INDOSAT-INP-AP INDOSAT Internet Network
> > >> Provider,ID)
> > >>
> > >> Upstream AS:  AS4651 (THAI-GATEWAY The Communications
> Authority
> > >>of
> > >> Thailand(CAT),TH)
> > >>
> > >> ASpath:   18356 9931 4651 4761
> > >>
> > >>
> > >>
> > >
> > >--
> > >Vlad
> > >
> > >
> >
>

BGPMON Alert Questions

[Joseph Jenkins]

So I setup BGPMON for my prefixes and got an alert about someone in
Thailand announcing my prefix.  Everything looks fine to me and I've
checked a bunch of different Looking Glasses and everything announcing
correctly.

I am assuming I should be contacting the provider about their
misconfiguration and announcing my prefixes and get them to fix it.  Any
other recommendations?

Is there a way I can verify what they are announcing just to make sure they
are still doing it?

Here is the alert for reference:

Your prefix:  8.37.93.0/24:

Update time:  2014-04-02 18:26 (UTC)

Detected by #peers:   2

Detected prefix:  8.37.93.0/24

Announced by: AS4761 (INDOSAT-INP-AP INDOSAT Internet Network
Provider,ID)

Upstream AS:  AS4651 (THAI-GATEWAY The Communications Authority of
Thailand(CAT),TH)

ASpath:   18356 9931 4651 4761

Question on Route-Set for Arin DB

[Joseph Jenkins]

So the Routing Database is something that I am just learning about and trying 
to find out if I need to create a Route-set or not.  I just created my MNTNER 
ID and I also created the Route Objects for my two /24s that were given to my 
by my carriers.  Do I need a route-set or aut-num object created?

Still trying to get my head wrapped around the need for this.  I read through 
this tutorial:

http://www.nanog.org/meetings/nanog51/presentations/Sunday/NANOG51.Talk34.NANOG51%20IRR%20Tutorial.pdf

and didn't get a really clear idea as to if I needed these.

TIA,

Joe

Re: Looking for some guidance on creating route objects for ARIN

[Joseph Jenkins]

Nevermind, someone already jumped in and helped me.  Thanks though for the 
email.
Joe Jenkins
909.636.2097

On Feb 10, 2014, at 11:43 AM, Jay Hennigan  wrote:

> On 2/10/14 9:08 AM, Joseph Jenkins wrote:
>> I am trying to get the routing objects database.  However I am getting back 
>> failures for the messages that I send in.  I am wondering is it possible to 
>> get route objects created for the two /24s that I was given from my carriers 
>> allocations?  If so what is the process to update the route objects 
>> database?  I have my MNTNERID for my company, but when I use that to try and 
>> update the objects and attach them to my AS I am getting a failure.
> 
> What are you sending (passwords redacted), to whom, and what failure
> message are you receiving?
> 
> -- 
> --
> Jay Hennigan - CCIE #7880 - Network Engineering - j...@impulse.net
> Impulse Internet Service  -  http://www.impulse.net/
> Your local telephone and internet company - 805 884-6323 - WB6RDV
> 

Looking for some guidance on creating route objects for ARIN

[Joseph Jenkins]

I am trying to get the routing objects database.  However I am getting back 
failures for the messages that I send in.  I am wondering is it possible to get 
route objects created for the two /24s that I was given from my carriers 
allocations?  If so what is the process to update the route objects database?  
I have my MNTNERID for my company, but when I use that to try and update the 
objects and attach them to my AS I am getting a failure.

TIA

Joe Jenkins

Re: BGP multihoming with two address spaces

[Joseph Jenkins]

I am announcing two separate /24s.  8.37.93.0 and 207.114.212.0.


Joe

On Jan 29, 2014, at 4:21 AM, Sasa Ristic  wrote:

> How are you announcing your address space now?
> 
> On Wed, Jan 29, 2014 at 12:32 PM, Joseph Jenkins
>  wrote:
>> I am seeking some feedback/help with my BGP configuration.  I am peering 
>> with two providers level3 and tw.  Unfortunately all of my address spaces 
>> are preferring the route over tw rather than level3.  I have tried 
>> Prepending my AS and the carriers AS to the path on the tw side and I see 
>> those update being accepted by internet routers, but everyone is still 
>> preferring to install the tw routes rather than level3.  I was trying to 
>> advertise each provider's address space out their connections and then use 
>> the other for backup.  Now however everything is coming in through tw and no 
>> one seems to like level3.
>> 
>> 
>> Thanks in advance for any guidance or assistance.
>> 
>> Joe
>> 
> 
> 
> 
> -- 
> Ristic Sasa
> --
> mob: +381652221123
> fax: +381618208488
> --
> Molim Vas da ne štampate ovaj e-mail ukoliko Vam zaista nije potreban
> na papiru. Hvala!
> 

BGP multihoming with two address spaces

[Joseph Jenkins]

I am seeking some feedback/help with my BGP configuration.  I am peering with 
two providers level3 and tw.  Unfortunately all of my address spaces are 
preferring the route over tw rather than level3.  I have tried Prepending my AS 
and the carriers AS to the path on the tw side and I see those update being 
accepted by internet routers, but everyone is still preferring to install the 
tw routes rather than level3.  I was trying to advertise each provider's 
address space out their connections and then use the other for backup.  Now 
however everything is coming in through tw and no one seems to like level3.


Thanks in advance for any guidance or assistance.

Joe