Re: Hawaiian ILEC infrastructure and fire

[Aaron1]

“Big, undersea, mpls network”.  Doesn’t get much cooler than that ;)

Aaron

> On Aug 16, 2023, at 9:51 PM, scott via NANOG  wrote:
> 
> 
> 
>> On 8/17/23 2:03 AM, John Levine wrote:
>> According to Eric Kuhnke :
>>> -=-=-=-=-=-
>>> 
>>> It's my understanding that the Hawaiian ILEC is now owned by Cincinnati
>>> Bell, which is also a unique historical artifact, as it was its own
>>> independent corporation/operating entity in the region of Cincinnati during
>>> the era of the pre-1984 Bell system.
>> Not that unique, SNET was also a Bell affiliate in most of Connecticut.
>> Hawaiian Tel has a very painful history. It was independent until
>> 1967, then bought by GTE, then merged into Verizon along with the rest
>> of GTE in 2000, then sold to a hedge fund in 2004 which knew nothing
>> about telephony and ran it into bankruptcy, then an independent public
>> company from 2010 to 2017, when it was bought by Cincinnati Bell,
>> which in turn was bought in 2021 by Australian conglomerate Macquarie.
> 
> Yep, that's it.  And the hedge fund (The Carlyle Group) thing was a complete 
> disaster.  I was here for all that.  Fugly is all I can say.
> 
> 
> 
>> Running phone systems on islands is very expensive. There's only
>> 160,000 people on Maui, about the same as Salinas CA, but separated
>> from the rest of the world by a lot of water.
> 
> We have a lot of undersea fiber and it is all connected into one big MPLS 
> network for the internet stuff.  There is still SS7 stuff out there, too.  I 
> am unfamiliar with that part.
> 
> scott

Re: Hawaiian ILEC infrastructure and fire

[Jason Kuehl]

I would be willing to travel down to help restore infra; I did this back
around Sandy as well. Is there anyone we can contact?

On Wed, Aug 16, 2023 at 10:51 PM scott via NANOG  wrote:

>
>
> On 8/17/23 2:03 AM, John Levine wrote:
> > According to Eric Kuhnke :
> >> -=-=-=-=-=-
> >>
> >> It's my understanding that the Hawaiian ILEC is now owned by Cincinnati
> >> Bell, which is also a unique historical artifact, as it was its own
> >> independent corporation/operating entity in the region of Cincinnati
> during
> >> the era of the pre-1984 Bell system.
> >
> > Not that unique, SNET was also a Bell affiliate in most of Connecticut.
> >
> > Hawaiian Tel has a very painful history. It was independent until
> > 1967, then bought by GTE, then merged into Verizon along with the rest
> > of GTE in 2000, then sold to a hedge fund in 2004 which knew nothing
> > about telephony and ran it into bankruptcy, then an independent public
> > company from 2010 to 2017, when it was bought by Cincinnati Bell,
> > which in turn was bought in 2021 by Australian conglomerate Macquarie.
>
> Yep, that's it.  And the hedge fund (The Carlyle Group) thing was a
> complete disaster.  I was here for all that.  Fugly is all I can say.
>
>
>
> > Running phone systems on islands is very expensive. There's only
> > 160,000 people on Maui, about the same as Salinas CA, but separated
> > from the rest of the world by a lot of water.
>
> We have a lot of undersea fiber and it is all connected into one big
> MPLS network for the internet stuff.  There is still SS7 stuff out
> there, too.  I am unfamiliar with that part.
>
> scott
>


-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: Hawaiian ILEC infrastructure and fire

[scott via NANOG]

On 8/17/23 2:03 AM, John Levine wrote:

According to Eric Kuhnke :

-=-=-=-=-=-

It's my understanding that the Hawaiian ILEC is now owned by Cincinnati
Bell, which is also a unique historical artifact, as it was its own
independent corporation/operating entity in the region of Cincinnati during
the era of the pre-1984 Bell system.


Not that unique, SNET was also a Bell affiliate in most of Connecticut.

Hawaiian Tel has a very painful history. It was independent until
1967, then bought by GTE, then merged into Verizon along with the rest
of GTE in 2000, then sold to a hedge fund in 2004 which knew nothing
about telephony and ran it into bankruptcy, then an independent public
company from 2010 to 2017, when it was bought by Cincinnati Bell,
which in turn was bought in 2021 by Australian conglomerate Macquarie.


Yep, that's it.  And the hedge fund (The Carlyle Group) thing was a 
complete disaster.  I was here for all that.  Fugly is all I can say.





Running phone systems on islands is very expensive. There's only
160,000 people on Maui, about the same as Salinas CA, but separated
from the rest of the world by a lot of water.


We have a lot of undersea fiber and it is all connected into one big 
MPLS network for the internet stuff.  There is still SS7 stuff out 
there, too.  I am unfamiliar with that part.


scott

Re: Hawaiian ILEC infrastructure and fire

[scott via NANOG]

On 8/17/23 1:08 AM, Eric Kuhnke wrote:
It's my understanding that the Hawaiian ILEC is now owned by Cincinnati 
Bell, which is also a unique historical artifact, as it was its own 
independent corporation/operating entity in the region of Cincinnati 
during the era of the pre-1984 Bell system.


Yes, HT was bought by Cin Bell.  CB was then bought by an out of country 
company and are changing their name to altafiber.


scott






Somewhat like how GTE was independent in other places in the country.

https://en.wikipedia.org/wiki/Cincinnati_Bell 



Some of the Hawaii ILEC structures I have seen photos of in other 
non-fire-affected places and other islands have a resemblance to designs 
that were built by BCTel, the ILEC in British Columbia, at the time when 
GTE was a shareholder in BCTel.




On Wed, Aug 16, 2023 at 10:50 AM Jay Hennigan > wrote:


On 8/16/23 09:32, Jay R. Ashworth wrote:

 > Well, it sounds like the historical Bell System attitude has
transitioned
 > forwards to ... newer transport.  Good.

Legacy GTE in this case, but agreed.

 > Best of luck to you all, out there.

Indeed.

-- 
Jay Hennigan - j...@west.net 

Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: Hawaiian ILEC infrastructure and fire

[scott via NANOG]

On 8/16/23 4:32 PM, Jay R. Ashworth wrote:

- Original Message -

From: "scott via NANOG" 



On 8/11/23 4:06 AM, Mark Tinka wrote:

It's like a war zone.


Yes, it definitely looks like that. We have connectivity to some of the
edges and have put up hotspots, so folks can go to the hotspot areas and
get internet access.


Well, it sounds like the historical Bell System attitude has transitioned
forwards to ... newer transport.  Good.



Yeah, the mindset of keeping it all running whatever we need to do is 
still strong here.  We have been having looong conf calls with many 
folks on it.




Best of luck to you all, out there.


Thanks.

scott




Cheers,
-- jra

Re: Hawaiian ILEC infrastructure and fire

[John Levine]

According to Eric Kuhnke :
>-=-=-=-=-=-
>
>It's my understanding that the Hawaiian ILEC is now owned by Cincinnati
>Bell, which is also a unique historical artifact, as it was its own
>independent corporation/operating entity in the region of Cincinnati during
>the era of the pre-1984 Bell system.

Not that unique, SNET was also a Bell affiliate in most of Connecticut.

Hawaiian Tel has a very painful history. It was independent until
1967, then bought by GTE, then merged into Verizon along with the rest
of GTE in 2000, then sold to a hedge fund in 2004 which knew nothing
about telephony and ran it into bankruptcy, then an independent public
company from 2010 to 2017, when it was bought by Cincinnati Bell,
which in turn was bought in 2021 by Australian conglomerate Macquarie.

Running phone systems on islands is very expensive. There's only
160,000 people on Maui, about the same as Salinas CA, but separated
from the rest of the world by a lot of water.

-- 
Regards,
John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

Re: Hawaiian ILEC infrastructure and fire

[scott via NANOG]

On 8/16/23 3:58 AM, TJ Trout wrote:
I found it interesting that *all*? cellular service on west maui died? 
Does every carrier single-home via waves served out of the Lahaina CO? 
Or maybe they aren't allowed to have generators in Maui? Seems like they 
would have diverse paths to major sites

--

Many do mobile backhaul over various providers, including Hawaiian 
Telcom.  We do that over MPLS.  The Lahaina CO is an HT property and we 
maintain the stuff that kept it relatively safe; air handlers, filters, 
generators, battery banks, etc.  All fiber was gone.  The fire was 
intense due to the wind speed.  There was a hurricane near the islands. 
Likely, even the cell towers were melted.  I cannot speak to what the 
cell folks have in place.  Last, it's an island and diverse paths are 
short in number.


scott









On Tue, Aug 15, 2023 at 6:55 PM scott > wrote:




 > On Tue, Aug 15, 2023, 5:21 PM scott via NANOG mailto:nanog@nanog.org>
 >
 >     On 8/11/23 4:06 AM, Mark Tinka wrote:
 >      > It's like a war zone.
 >
 >     Yes, it definitely looks like that. We have connectivity to
some of the
 >     edges and have put up hotspots, so folks can go to the
hotspot areas
 >     and
 >     get internet access.


On 8/16/23 12:39 AM, TJ Trout wrote:

  > Scott: Just an FYI that anecdotal reports from social media
coming in or
  > stating that residents have been unable to connect to the Wi-Fi
hotspots
  > that the local government have been promoting in the Lahaina area.
--


I don't have anything to do with that as I work in the core and we got
the node up for west Maui, so I am done. (:  But I wonder if those are
different wifis.  I'd imagine the focus now is plant poles, hang fiber
and get the Access part of the network fully up before getting those
up,
if they're the same ones.

scott

Re: Hawaiian ILEC infrastructure and fire

[Eric Kuhnke]

It's my understanding that the Hawaiian ILEC is now owned by Cincinnati
Bell, which is also a unique historical artifact, as it was its own
independent corporation/operating entity in the region of Cincinnati during
the era of the pre-1984 Bell system.

Somewhat like how GTE was independent in other places in the country.

https://en.wikipedia.org/wiki/Cincinnati_Bell

Some of the Hawaii ILEC structures I have seen photos of in other
non-fire-affected places and other islands have a resemblance to designs
that were built by BCTel, the ILEC in British Columbia, at the time when
GTE was a shareholder in BCTel.



On Wed, Aug 16, 2023 at 10:50 AM Jay Hennigan  wrote:

> On 8/16/23 09:32, Jay R. Ashworth wrote:
>
> > Well, it sounds like the historical Bell System attitude has transitioned
> > forwards to ... newer transport.  Good.
>
> Legacy GTE in this case, but agreed.
>
> > Best of luck to you all, out there.
>
> Indeed.
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>
>

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/16/23 16:16, michael brooks - ESC wrote:

Perhaps (probably) naively, it seems to me that DPA would have been a 
useful BGP attribute. Can anyone shed light on why this RFC never 
moved beyond draft status? I cannot find much information on this 
other than IETF's data tracker 
(https://datatracker.ietf.org/doc/draft-ietf-idr-bgp-dpa/) and RFC6938 
(which implies DPA was in use, but then was deprecated).


I've never heard of this draft until now, but reading it, I can see why 
it would likely not be adopted today (not sure what the consensus would 
have been back in the '90's).


DPA looks like MED on drugs.

Not sure operators want remote downstream ISP's arbitrarily choosing 
which of their peering interconnects (and backbone links) carry traffic 
from source to them. BGP is a poor communicator of bandwidth and 
shilling cost, in general. Those kinds of decisions tend to be locally 
made, and permitting outside influence could be a rather hard sell.


It reminds me of how router vendors implemented GMPLS in the hopes that 
optical operators would allow their customers to build and control 
circuits in the optical domain in some fantastic fashion.


Or how router vendors built Sync-E and PTP into their routers hoping 
that they could sell timing as a service to mobile network operators as 
part of a RAN backhaul service.


Some things just tend to be sacred.

Mark.

Re: Hawaiian ILEC infrastructure and fire

[Jay Hennigan]

On 8/16/23 09:32, Jay R. Ashworth wrote:


Well, it sounds like the historical Bell System attitude has transitioned
forwards to ... newer transport.  Good.


Legacy GTE in this case, but agreed.


Best of luck to you all, out there.


Indeed.

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: Hawaiian ILEC infrastructure and fire

[Jay R. Ashworth]

- Original Message -
> From: "scott via NANOG" 

> On 8/11/23 4:06 AM, Mark Tinka wrote:
>> It's like a war zone.
> 
> Yes, it definitely looks like that. We have connectivity to some of the
> edges and have put up hotspots, so folks can go to the hotspot areas and
> get internet access.

Well, it sounds like the historical Bell System attitude has transitioned 
forwards to ... newer transport.  Good.

Best of luck to you all, out there.

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: NTP Sync Issue Across Tata (Europe)

[Tom Beecher]

>
> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.


100% correct. From the PDF :

4.31 JT summarised its findings in relation to the ‘Panic Timer’ on the
> Cisco IOS XR NTP Client, namely that: JT’s efforts in understanding the
> root cause, and mitigation steps to take to avoid future incidents have
> focused on the Cisco NTP Client behaviour, and notably Cisco’s decision to
> not implement the ‘Panic Timer’ on their IOS XR operating system. Arguably,
> whilst the NTP server injected an invalid time into the network, it is the
> NTP Clients filtering and selection algorithms which are responsible for
> detecting and disregarding falsetickers, and it was the Cisco NTP Clients
> failure to appropriately handle this which triggered the network incident.
> 43 […] Further detailed soak testing, log analysis and debug analysis
> corroborated that the Cisco IOS XR NTP Client did not implement the ‘Panic
> Timer’ that would normally cause an NTP Client to ignore an NTP Server
> exceeding 1000 seconds variance.


On Wed, Aug 16, 2023 at 10:50 AM Mel Beckman  wrote:

> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.
>
>
>
>  -mel beckman
>
> On Aug 16, 2023, at 3:51 AM, Matthew Richardson 
> wrote:
>
> Mel Beckman wrote:-
>
> Do you have a citation for your Jersey event? I doubt GPS caused the
> problem, but I'd like to see the documentation.
>
>
> The event took place on the evening of Sunday 12 July 2020, and seems NOT
> to have been due to an issue caused directly by GPS, but rather to
> misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
> subsequently issued the following comprehensive document:-
>
>
> https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf
>
> By way of summary, JT operated two GPS derived NTP servers, with all of
> their routers were pointing to both.  On the evening in question, one of
> the two reset its clock back to 27 November 2000.
>
> Their interior routing protocol used amongst their mesh of routers was
> IS-IS which was using authentication.  The authentication [section 4.19]
> was described having a "password validity start date" of 01 July 2012.
> Thus, any routers which had picked up the time from the faulty source no
> longer had valid IS-IS authentication and were thus isolated.
>
> Whilst only 15% of their routers were affected, this was enough to cause an
> almost total failure in their network, affecting telephony (fixed & mobile)
> and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
> the emergency services in these parts, where any failures attract the
> attention of our regulator.
>
> The details of why the clock "failed" start at section 4.23, and seem to
> relate a GPS week number rollover.
>
> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.
>
> One curious aspect is that some routers followed the "bad" time, which is
> alluded to in section 4.31.
>
> Something not discussed in that report is that JT's email failed during the
> incident despite its being hosted on Office365.  The reason was that the
> two authoritative DNS servers for jtglobal.com were hosted in Jersey
> inside
> their network.  As that network was wholly disconnected, there was no DNS
> and hence no email.  Despite my having raised this since with their senior
> management, their DNS remains hosted in this way:-
>
> matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @
> ns1.jtibs.net
>
> ;; Got answer:
>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
>
> ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
>
>
> ;; QUESTION SECTION:
>
> ;jtglobal.com.INNS
>
>
> ;; ANSWER SECTION:
>
> jtglobal.com.60INNSns2.jtibs.net.
>
> jtglobal.com.60INNSns1.jtibs.net.
>
>
> ;; ADDITIONAL SECTION:
>
> ns1.jtibs.net.60INA212.9.0.135
>
> ns2.jtibs.net.60INA212.9.0.136
>
> ns1.jtibs.net.60IN2a02:c28::d1
>
> ns2.jtibs.net.60IN2a02:c28::d2
>
>
> Rediculously (and again despite my agitation to their management) our
> government domain gov.je has similar DNS fragility:-
>
> matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
>
> ;; Got answer:
>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
>
> ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
>
>
> ;; QUESTION SECTION:
>
> ;gov.je.INNS
>
>
> ;; ANSWER SECTION:
>
> gov.je.3600INNSns2.gov.je.
>
> gov.je.3600INNSns1.gov.je.
>
>
> ;; ADDITIONAL SECTION:
>
> ns2.gov.je.3600INA212.9

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

Interesting software bug, but not really germane to this discussion, other than 
as a cautionary tale about time distribution architectures.


 -mel beckman

On Aug 16, 2023, at 3:51 AM, Matthew Richardson  
wrote:

Mel Beckman wrote:-

Do you have a citation for your Jersey event? I doubt GPS caused the problem, 
but I'd like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both.  On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication.  The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365.  The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network.  As that network was wholly disconnected, there was no DNS
and hence no email.  Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com 
@ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;jtglobal.com.INNS

;; ANSWER SECTION:
jtglobal.com.60INNSns2.jtibs.net.
jtglobal.com.60INNSns1.jtibs.net.

;; ADDITIONAL SECTION:
ns1.jtibs.net.60INA212.9.0.135
ns2.jtibs.net.60INA212.9.0.136
ns1.jtibs.net.60IN2a02:c28::d1
ns2.jtibs.net.60IN2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;gov.je.INNS

;; ANSWER SECTION:
gov.je.3600INNSns2.gov.je.
gov.je.3600INNSns1.gov.je.

;; ADDITIONAL SECTION:
ns2.gov.je.3600INA212.9.21.137
ns1.gov.je.3600INA212.9.21.9

--
Best wishes,
Matthew

--
From: Mel Beckman 
To: Matthew Richardson 
Cc: Nanog 
Date: Tue, 8 Aug 2023 15:12:29 +
Subject: Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation 
for your Jersey event? I doubt GPS caused the problem, but I'd like to see the 
documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP 
with known, well documented vulnerabilities and many security incidents, versus 
the risk of some theoretical GPS-based vulnerability, for which mitigations 
such as geographic diversity are readily available. Sure, you could use 
Internet NTP as a last resort should GPS fail globally (perhaps due to a 
theoretical - but conceivable - meteor storm). But that would be a fall-back. I 
would not mix the systems.

-mel

On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
wrote:

?Mel Beckman wrote:-

It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression via high powered jamming. But that you can do with a
geographically diverse GPS NTP network.

Whilst looking forw

Re: NTP Sync Issue Across Tata (Europe)

[Tom Beecher]

Thanks for that link.

This is jumping out at me though :

Their interior routing protocol used amongst their mesh of routers was
> IS-IS which was using authentication.  The authentication [section 4.19]
> was described having a "password validity start date" of 01 July 2012.
> Thus, any routers which had picked up the time from the faulty source no
> longer had valid IS-IS authentication and were thus isolated.
>

It's been a while, but last time I remember diving into the IS-IS weeds ,
the time of the transmitting system wasn't part of a Hello.  Is this a
Cisco specific option they toss in a TLV?

On Wed, Aug 16, 2023 at 9:04 AM Matthew Richardson via NANOG <
nanog@nanog.org> wrote:

> Mel Beckman wrote:-
>
> >Do you have a citation for your Jersey event? I doubt GPS caused the
> problem, but I’d like to see the documentation.
>
> The event took place on the evening of Sunday 12 July 2020, and seems NOT
> to have been due to an issue caused directly by GPS, but rather to
> misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
> subsequently issued the following comprehensive document:-
>
>
> https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf
>
> By way of summary, JT operated two GPS derived NTP servers, with all of
> their routers were pointing to both.  On the evening in question, one of
> the two reset its clock back to 27 November 2000.
>
> Their interior routing protocol used amongst their mesh of routers was
> IS-IS which was using authentication.  The authentication [section 4.19]
> was described having a "password validity start date" of 01 July 2012.
> Thus, any routers which had picked up the time from the faulty source no
> longer had valid IS-IS authentication and were thus isolated.
>
> Whilst only 15% of their routers were affected, this was enough to cause an
> almost total failure in their network, affecting telephony (fixed & mobile)
> and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
> the emergency services in these parts, where any failures attract the
> attention of our regulator.
>
> The details of why the clock "failed" start at section 4.23, and seem to
> relate a GPS week number rollover.
>
> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.
>
> One curious aspect is that some routers followed the "bad" time, which is
> alluded to in section 4.31.
>
> Something not discussed in that report is that JT's email failed during the
> incident despite its being hosted on Office365.  The reason was that the
> two authoritative DNS servers for jtglobal.com were hosted in Jersey
> inside
> their network.  As that network was wholly disconnected, there was no DNS
> and hence no email.  Despite my having raised this since with their senior
> management, their DNS remains hosted in this way:-
>
> >matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @
> ns1.jtibs.net
> >;; Got answer:
> >;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
> >;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
> >
> >;; QUESTION SECTION:
> >;jtglobal.com. IN  NS
> >
> >;; ANSWER SECTION:
> >jtglobal.com.  60  IN  NS  ns2.jtibs.net.
> >jtglobal.com.  60  IN  NS  ns1.jtibs.net.
> >
> >;; ADDITIONAL SECTION:
> >ns1.jtibs.net. 60  IN  A   212.9.0.135
> >ns2.jtibs.net. 60  IN  A   212.9.0.136
> >ns1.jtibs.net. 60  IN  2a02:c28::d1
> >ns2.jtibs.net. 60  IN  2a02:c28::d2
>
> Rediculously (and again despite my agitation to their management) our
> government domain gov.je has similar DNS fragility:-
>
> >matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @
> ns1.gov.je
> >;; Got answer:
> >;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
> >;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
> >
> >;; QUESTION SECTION:
> >;gov.je.   IN  NS
> >
> >;; ANSWER SECTION:
> >gov.je.3600IN  NS  ns2.gov.je.
> >gov.je.3600IN  NS  ns1.gov.je.
> >
> >;; ADDITIONAL SECTION:
> >ns2.gov.je.3600IN  A   212.9.21.137
> >ns1.gov.je.3600IN  A   212.9.21.9
>
> --
> Best wishes,
> Matthew
>
>  --
> >From: Mel Beckman 
> >To: Matthew Richardson 
> >Cc: Nanog 
> >Date: Tue, 8 Aug 2023 15:12:29 +
> >Subject: Re: NTP Sync Issue Across Tata (Europe)
>
> >Until the Internet NTP network can be made secure, no. Do you have a
> citation for your Jersey event? I doubt GPS caused the problem, but I’d
> like to see the documentation.
> >
> >Using GPS for time sync is simple risk management: the risk of Internet
> NTP with known, well documented vulnerabilities and many security
> incidents, versus the risk o

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.



 -mel beckman

On Aug 16, 2023, at 3:51 AM, Matthew Richardson  
wrote:

Mel Beckman wrote:-

Do you have a citation for your Jersey event? I doubt GPS caused the problem, 
but I'd like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both.  On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication.  The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365.  The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network.  As that network was wholly disconnected, there was no DNS
and hence no email.  Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com 
@ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;jtglobal.com.INNS

;; ANSWER SECTION:
jtglobal.com.60INNSns2.jtibs.net.
jtglobal.com.60INNSns1.jtibs.net.

;; ADDITIONAL SECTION:
ns1.jtibs.net.60INA212.9.0.135
ns2.jtibs.net.60INA212.9.0.136
ns1.jtibs.net.60IN2a02:c28::d1
ns2.jtibs.net.60IN2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;gov.je.INNS

;; ANSWER SECTION:
gov.je.3600INNSns2.gov.je.
gov.je.3600INNSns1.gov.je.

;; ADDITIONAL SECTION:
ns2.gov.je.3600INA212.9.21.137
ns1.gov.je.3600INA212.9.21.9

--
Best wishes,
Matthew

--
From: Mel Beckman 
To: Matthew Richardson 
Cc: Nanog 
Date: Tue, 8 Aug 2023 15:12:29 +
Subject: Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation 
for your Jersey event? I doubt GPS caused the problem, but I'd like to see the 
documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP 
with known, well documented vulnerabilities and many security incidents, versus 
the risk of some theoretical GPS-based vulnerability, for which mitigations 
such as geographic diversity are readily available. Sure, you could use 
Internet NTP as a last resort should GPS fail globally (perhaps due to a 
theoretical - but conceivable - meteor storm). But that would be a fall-back. I 
would not mix the systems.

-mel

On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
wrote:

?Mel Beckman wrote:-

It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression via high powered jamming. But that you can do with a
geographically diverse GPS NTP network.

Whilst looking forward to being corrected, GPS (even across multiple
locations) seems to be a SINGLE source of time.  You seem (have I
misunderstood?) to be a p

Early Registration Ends Sunday! + Sponsorships Available + Final Call for Presentations

[Nanog News]

*Early Registration Ends Sunday! **Don't Miss Your Chance to Receive a
Discounted Rate *

NANOG 89 will take place off the coast of San Diego at the Loews Coronado
Bay Resort from 16 - 18 October.

*REGISTER NOW
*

*NANOG 89 Sponsorships Available! *
*Invest in the Strength of the Community We've Built *

*Sponsorship Benefits:*

+ Showcase your newest technologies + solutions
+ Increase your brand’s visibility + reach
+ Amplify your organization’s message
+ Connect with industry influencers + decision-makers
+ Empower people + inspire change

*MORE INFO
*

*Final Call for Presentations*
*Submission Deadline is Monday!*

We want to hear from you! Present on the stage or virtually for our next
meeting.

Anyone interested in sharing their experiences, ideas, or the latest
technologies with the greater NANOG community must submit a presentation
proposal by Monday, 28 Aug 2023.

*MORE INFO
*

*Video of the Week  *
*Network Telemetry on Modern Routers w/ Pavel Odintsov*

Watch a detailed overview of traffic telemetry protocols in modern routing
platforms.The speaker, Pavel Odintsov will cover well-known protocols such
as Netflow, IPFIX, sFlow, and port mirror.

He will additionally provide deep dive into current protocols such as
inline monitoring services and IPFIX 315.

*WATCH NOW * 

Destination Preference Attribute for BGP

[michael brooks - ESC]

Perhaps (probably) naively, it seems to me that DPA would have been a
useful BGP attribute. Can anyone shed light on why this RFC never moved
beyond draft status? I cannot find much information on this other than
IETF's data tracker (
https://datatracker.ietf.org/doc/draft-ietf-idr-bgp-dpa/) and RFC6938
(which implies DPA was in use, but then was deprecated).




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
720.972.4110
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: NTP Sync Issue Across Tata (Europe)

[sronan]

Throw  PTP in the mix for the greater accuracy required for some wireless (5G) configurations, and the situation becomes even more complicated.On Aug 14, 2023, at 6:55 AM, Forrest Christian (List Account)  wrote:We're going to have to somewhat disagree here...I may not have been 100% clear about what I see as the most common risks for GPS.  The reason I suggest that NTP risks and GPS risks are similar is not primarily due to intentional time injection hacks (although that is a risk).   Instead, it's related to GPS failure modes, and the increased commonality of GPS jamming causing those failures.   I will 100% concede that NTP carries far more spoofing or intentional DoS risks.   But GPS is far more likely to suffer a failure in the absence of a bad actor than NTP.The reason for this is that the GPS signal is incredibly weak, and it's incredibly easy to break GPS reception.   Good antenna placement and antennas that try to reject terrestrial signals help but don't always prevent the failures from happening.   Because GPS is used more and more to track objects and people, people who don't want to be tracked are starting to buy and use jammers.  In addition, it's becoming increasingly common for gamers to spoof their GPS location (and, as a result, time) via GPS injection.  So the kid down the street trying to cheat at pokemon go or the truck driver not wanting to get in trouble for speeding may unintentionally cause your time server to quit working correctly.  Not to mention the random piece of electronic gear which malfunctions and spews noise across the GPS band.So, yes, I will 100% agree with you that NTP carries more intentional hacking risk.   But I'm going to argue that GPS carries a significantly higher risk of a jamming-related failure.   Without good statistics, it's hard to tell which is more prevalent.   I see a lot more GPS failures from my viewpoint, but I also have to talk to customers who are having precision timing issues due to GPS failures.   My intuitive feeling is that in the absence of bad actors, NTP is significantly more reliable than GPS.   In the presence of remote bad actors, I'll grant that NTP is 100% the loser here.  When everything is working, GPS will provide better time.  Adding a holdover oscillator to GPS does help in marginal situations, but doesn't resolve all of the GPS issues.In those situations where time is not critical, either NTP or GPS is a good solution, and it largely comes down to which you prefer.   I deal with way too many antennas so I'd rather just harden a NTP server.   You might deal with way too many hackers getting in your systems so you might prefer relying on a GPS antenna.   Either way, most of the time you're going to get decent time service.   We could go into a lot of details about how each system can fail, but for non-time critical applications I'm not sure either would come out a clear winner.  I know you believe GPS does, and I believe that it isn't 100% clear which one is better for those "just want time that works most of the time" applications.   We could argue all day about this and we won't get anywhere beyond us disagreeing about this.Once you get to more time-critical apps where actual budget is going to be expended on ensuring reliable NTP services are available 24x7, then neither a default configuration NTP server nor a single GPS receiver will provide reliable time.   Selecting servers and hardening firewalls to limit the likelihood of time injection can work wonders on NTP robustness.   GPS works too if you provide enough GPS timing sources that multiple locations would have to be jammed at once.   Providing a mix of these is even better.  If I was to go GPS-only I'd probably try to ensure a minimum of 3 different GPS receivers at 3 different locations, with internal NTP servers pulling from each of the GPS-connected NTP servers.   5 would even be better.   An even more robust option would be to go with 5 GPS receivers and 2 or 3 NTP-connected stratum 1 time sources.   In this last case, you could spoof ALL of the NTP servers and the GPS would still be in control.  You could also have signal failures at 3 of the GPS sites and the NTP connections would provide redundant time sources.   Only with GPS failures at multiple sites AND NTP failures or spoofing happening at the same time would one have an issue where the NTP servers could possibly fail to receive correct time.On Mon, Aug 14, 2023 at 2:00 AM Mel Beckman  wrote:




Forrest,


I think you’re gilding the lilly. My original recommendation was to use GPS as primary, for its superior accuracy and resistance
 to attack, and have anti-GPS back up.  If you want automatic fail over, do that in an intermediate server on your site that makes a conscious test and decision to fail over to Internet NTP.


You’re mistaken to say that the vulnerability of GPS is remotely comparable  to the vulnerability of Internet-based NTP.
 To interfere with a GPS-derived clock, an attac

Re: Dodgy AS327933 ...?

[Tom Beecher]

>
> That's a terrible excuse for the shitty concepts behind MikroTik's CLI.
>

Fear of being sued into oblivion by a massive corporation, even if they're
in the wrong, has influenced many choices in technology.

To be clear, I am not stating that Mikrotik made the CLI choices they did
BECAUSE of that concern. I have no knowledge there. I do know that Much
Larger Vendors have absolutely made tradeoffs because they didn't want to
deal with legal actions from Another Large Vendor or Patent Troll, so it's
not exactly uncommon.

On Tue, Aug 15, 2023 at 5:43 PM Chris Cappuccio  wrote:

> Tom Beecher [beec...@beecher.cc] wrote:
> > >
> > > It should be a huge embarrasment to the designers. They survive on low
> > > price and unique features. It would be quite amazing to have a CLI
> without
> > > the nonsense.
> > >
> >
> > That ship sailed years ago. Even though the legal precedent was set after
> > Cisco vs Arista that CLI elements that are of common , standard usage
> > aren't copyrightable, nobody wants to take the risk. So they come up with
> > other ways that tend to be not good.
> >
>
> That's a terrible excuse for the shitty concepts behind MikroTik's CLI.
>
> I'm not saying my stuff is much better, but yeah, actually I am.
>

Re: NTP Sync Issue Across Tata (Europe)

[Matthew Richardson via NANOG]

Mel Beckman wrote:-

>Do you have a citation for your Jersey event? I doubt GPS caused the problem, 
>but I’d like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both.  On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication.  The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365.  The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network.  As that network was wholly disconnected, there was no DNS
and hence no email.  Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

>matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com 
>@ns1.jtibs.net
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
>;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
>
>;; QUESTION SECTION:
>;jtglobal.com. IN  NS
>
>;; ANSWER SECTION:
>jtglobal.com.  60  IN  NS  ns2.jtibs.net.
>jtglobal.com.  60  IN  NS  ns1.jtibs.net.
>
>;; ADDITIONAL SECTION:
>ns1.jtibs.net. 60  IN  A   212.9.0.135
>ns2.jtibs.net. 60  IN  A   212.9.0.136
>ns1.jtibs.net. 60  IN  2a02:c28::d1
>ns2.jtibs.net. 60  IN  2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

>matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
>;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
>
>;; QUESTION SECTION:
>;gov.je.   IN  NS
>
>;; ANSWER SECTION:
>gov.je.3600IN  NS  ns2.gov.je.
>gov.je.3600IN  NS  ns1.gov.je.
>
>;; ADDITIONAL SECTION:
>ns2.gov.je.3600IN  A   212.9.21.137
>ns1.gov.je.3600IN  A   212.9.21.9

--
Best wishes,
Matthew

 --
>From: Mel Beckman 
>To: Matthew Richardson 
>Cc: Nanog 
>Date: Tue, 8 Aug 2023 15:12:29 +
>Subject: Re: NTP Sync Issue Across Tata (Europe)

>Until the Internet NTP network can be made secure, no. Do you have a citation 
>for your Jersey event? I doubt GPS caused the problem, but I’d like to see the 
>documentation.
>
>Using GPS for time sync is simple risk management: the risk of Internet NTP 
>with known, well documented vulnerabilities and many security incidents, 
>versus the risk of some theoretical GPS-based vulnerability, for which 
>mitigations such as geographic diversity are readily available. Sure, you 
>could use Internet NTP as a last resort should GPS fail globally (perhaps due 
>to a theoretical — but conceivable — meteor storm). But that would be a 
>fall-back. I would not mix the systems.
>
> -mel
>
>> On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
>> wrote:
>> 
>> ?Mel Beckman wrote:-
>> 
>>> It's a problem that has received a lot of attention in both NTP and
>>> aviation navigation circles. What is hard to defend against is total signal
>>> suppression via high powered jamming. But that you can do with a
>>> geographically diverse GPS NTP network.
>> 
>> Whilst looking forward to being corrected, GPS (even across multiple
>> locations) seems to be a SINGLE source of time.  You seem (have I
>> misunderstood?) to be a proponent