Re: NIST NTP servers

[B F]

All,  
Thanks very much for all the replies. Extremely helpful.
"...ask someone what time it is and they'll tell you how to build a watch."
Luckily I got both.
Ed



 Original message 
From: Lamar Owen  
Date: 5/14/2016  10:27 AM  (GMT-05:00) 
To: NANOG  
Subject: Re: NIST NTP servers 

On 05/13/2016 04:38 PM, Mel Beckman wrote:
> But another key consideration beyond accuracy is the reliability of a 
> server's GPS constellation view. If you can lose GPS sync for an hour or more 
> (not uncommon in terrain-locked locations), the NTP time will go free-running 
> and could drift quite a bit. You need an OCXO to minimize that drift to 
> acceptable levels.
While this is drifting a bit off-topic for NANOG (and drifting into the 
topic range for time-n...@febo.com), I'll just add one more thing to 
this.  The Hold time (when the oscillator is free-running) is a very 
important consideration, especially, as you say, when terrain is an 
issue. For us it is even more important, as the 10MHz output from the 
timing rack is used as a site-wide frequency standard.  Of course, you 
never discipline a cesium PRS, but the rubidium secondary is disciplined 
by circuitry in the SSU2000.

Back in the days of common backbone delivery over SONET discussion of 
cesium standards would have been on-topic, as some SONET gear (Nortel 
Optera for instance) needs a master clock; especially if you were 
delivering channelized circuits or interfacing with customers and telcos 
with DS3 or even DS1 circuits or DS0 fractions within them.  Ethernet is 
far more forgiving.

Re: NIST NTP servers

[Lamar Owen]

On 05/13/2016 04:38 PM, Mel Beckman wrote:

But another key consideration beyond accuracy is the reliability of a server's 
GPS constellation view. If you can lose GPS sync for an hour or more (not 
uncommon in terrain-locked locations), the NTP time will go free-running and 
could drift quite a bit. You need an OCXO to minimize that drift to acceptable 
levels.
While this is drifting a bit off-topic for NANOG (and drifting into the 
topic range for time-n...@febo.com), I'll just add one more thing to 
this.  The Hold time (when the oscillator is free-running) is a very 
important consideration, especially, as you say, when terrain is an 
issue. For us it is even more important, as the 10MHz output from the 
timing rack is used as a site-wide frequency standard.  Of course, you 
never discipline a cesium PRS, but the rubidium secondary is disciplined 
by circuitry in the SSU2000.


Back in the days of common backbone delivery over SONET discussion of 
cesium standards would have been on-topic, as some SONET gear (Nortel 
Optera for instance) needs a master clock; especially if you were 
delivering channelized circuits or interfacing with customers and telcos 
with DS3 or even DS1 circuits or DS0 fractions within them.  Ethernet is 
far more forgiving.

Re: NIST NTP servers

[Mel Beckman]

"Either method needs the specs" should read "Either method meets the specs."

 -mel beckman

> On May 13, 2016, at 1:39 PM, Mel Beckman  wrote:
> 
> Lamar,
> 
> Because you need microsecond-level time accuracy (which is beyond NTP's 
> capabilities) you'll requires an adjunct protocol, such as PPS, to get that.  
> For continued NTP delivery despite periodic GPS signal loss, then you need an 
> OCXO internal clock. 
> 
> But anyone satisfied with NTP's millisecond time accuracy at worst needs a $1 
> temperature-compensated internal clock. Either method needs the specs for a 
> Stratum 1 time source on a local network. 
> 
> As you point out, the hobbyist SBCs can't deliver even basic clock accuracy.  
> 
> But another key consideration beyond accuracy is the reliability of a 
> server's GPS constellation view. If you can lose GPS sync for an hour or more 
> (not uncommon in terrain-locked locations), the NTP time will go free-running 
> and could drift quite a bit. You need an OCXO to minimize that drift to 
> acceptable levels. 
> 
> But I see that the price premium for an OCXO clock is only $100 to $200 on 
> low-cost (I.e., ~$500) commercial NTP servers. So buyers need only make a 
> minor cost adjustment to have very good, and inexpensive, COTS NTP 
> performance and reliability. 
> 
> -mel beckman
> 
>>> On May 13, 2016, at 9:30 AM, Lamar Owen  wrote:
>>> 
>>> On 05/13/2016 10:38 AM, Mel Beckman wrote:
>>> You make it sound like TXCOs are rare, but they're actually quite common in 
>>> most single board computers. True, you're probably not gonna find them in 
>>> the $35 cellular-based SBCs, but since these temperature compensated 
>>> oscillators cost less than a dollar each in quantity, they're quite common 
>>> in most industrial species for well under $100.
>> 
>> Correct, they're not rare in the industrial line (for that matter you can 
>> get TCXO-equipped RTL-SDR dongles, but that's not NTP-related).  Something 
>> like a Transko TFC or TX-P or similar is enough for reasonable timing for 
>> basic purposes, and they're not expensive.  They're also not a stock item on 
>> the consumer-level SBC's either.  I looked at one of our half-dozen ODroid 
>> C2's, and the main processor clock, X3, is under the heatsink, so I can't 
>> see what part is being used.  X1 and X2 are outside, and it doesn't appear 
>> that they are TCXO modules, although I didn't use a magnifier to check the 
>> part number and might have made an error.
>> 
>> The Nicegear DS3231 RTC has a TCXO, and might be the best low-cost choice at 
>> $12 (need to have an RPi, ODroid, or similar on which to mount it).  It's 
>> not that TCXO's are rare or expensive, it's that they're not often 
>> considered to be important to accuracy in many circles.
>> 
>>> An Ovenized XCO is absolutely not required for IT-grade NTP servers.
>> 
>> No, but it is for my purposes here.  But, as I said in my post:
>> 
>> 
>>> You really have to have at least a temperature compensated quartz crystal 
>>> oscillator (TCXO) to even begin to think about an NTP server, for anything 
>>> but the most rudimentary of timing.  Ovenized quartz oscillators (OCXO) and 
>>> rubidium standards are the next step up, ...
>> 
>> I was just saying that OCXO and Rb are just the next step up if you would 
>> like more stability, that's all.  For 'within a second' on a GPS-disciplined 
>> clock (even without the 1PPS signal) you wouldn't necessarily need TXCO.  
>> But that's what I meant by 'anything but the most rudimentary of timing.'  
>> Rudimentary is down to the millisecond in my environment.  PTP takes you to 
>> the next level, and beyond that you don't use network timing but put a 
>> dedicated time distribution network running IRIG-B or similar.  But that is 
>> beyond the scope of a typical IT NTP server's needs.
>> 

Re: NIST NTP servers

[Mel Beckman]

Lamar,

Because you need microsecond-level time accuracy (which is beyond NTP's 
capabilities) you'll requires an adjunct protocol, such as PPS, to get that.  
For continued NTP delivery despite periodic GPS signal loss, then you need an 
OCXO internal clock. 

But anyone satisfied with NTP's millisecond time accuracy at worst needs a $1 
temperature-compensated internal clock. Either method needs the specs for a 
Stratum 1 time source on a local network. 

As you point out, the hobbyist SBCs can't deliver even basic clock accuracy.  

But another key consideration beyond accuracy is the reliability of a server's 
GPS constellation view. If you can lose GPS sync for an hour or more (not 
uncommon in terrain-locked locations), the NTP time will go free-running and 
could drift quite a bit. You need an OCXO to minimize that drift to acceptable 
levels. 

But I see that the price premium for an OCXO clock is only $100 to $200 on 
low-cost (I.e., ~$500) commercial NTP servers. So buyers need only make a minor 
cost adjustment to have very good, and inexpensive, COTS NTP performance and 
reliability. 

 -mel beckman

> On May 13, 2016, at 9:30 AM, Lamar Owen  wrote:
> 
>> On 05/13/2016 10:38 AM, Mel Beckman wrote:
>> You make it sound like TXCOs are rare, but they're actually quite common in 
>> most single board computers. True, you're probably not gonna find them in 
>> the $35 cellular-based SBCs, but since these temperature compensated 
>> oscillators cost less than a dollar each in quantity, they're quite common 
>> in most industrial species for well under $100.
> 
> Correct, they're not rare in the industrial line (for that matter you can get 
> TCXO-equipped RTL-SDR dongles, but that's not NTP-related).  Something like a 
> Transko TFC or TX-P or similar is enough for reasonable timing for basic 
> purposes, and they're not expensive.  They're also not a stock item on the 
> consumer-level SBC's either.  I looked at one of our half-dozen ODroid C2's, 
> and the main processor clock, X3, is under the heatsink, so I can't see what 
> part is being used.  X1 and X2 are outside, and it doesn't appear that they 
> are TCXO modules, although I didn't use a magnifier to check the part number 
> and might have made an error.
> 
> The Nicegear DS3231 RTC has a TCXO, and might be the best low-cost choice at 
> $12 (need to have an RPi, ODroid, or similar on which to mount it).  It's not 
> that TCXO's are rare or expensive, it's that they're not often considered to 
> be important to accuracy in many circles.
> 
>> An Ovenized XCO is absolutely not required for IT-grade NTP servers.
> 
> No, but it is for my purposes here.  But, as I said in my post:
> 
> 
>> You really have to have at least a temperature compensated quartz crystal 
>> oscillator (TCXO) to even begin to think about an NTP server, for anything 
>> but the most rudimentary of timing.  Ovenized quartz oscillators (OCXO) and 
>> rubidium standards are the next step up, ...
> 
> I was just saying that OCXO and Rb are just the next step up if you would 
> like more stability, that's all.  For 'within a second' on a GPS-disciplined 
> clock (even without the 1PPS signal) you wouldn't necessarily need TXCO.  But 
> that's what I meant by 'anything but the most rudimentary of timing.'  
> Rudimentary is down to the millisecond in my environment.  PTP takes you to 
> the next level, and beyond that you don't use network timing but put a 
> dedicated time distribution network running IRIG-B or similar.  But that is 
> beyond the scope of a typical IT NTP server's needs.
> 

Re: NIST NTP servers

[Lamar Owen]

On 05/13/2016 10:38 AM, Mel Beckman wrote:

You make it sound like TXCOs are rare, but they're actually quite common in 
most single board computers. True, you're probably not gonna find them in the 
$35 cellular-based SBCs, but since these temperature compensated oscillators 
cost less than a dollar each in quantity, they're quite common in most 
industrial species for well under $100.


Correct, they're not rare in the industrial line (for that matter you 
can get TCXO-equipped RTL-SDR dongles, but that's not NTP-related).  
Something like a Transko TFC or TX-P or similar is enough for reasonable 
timing for basic purposes, and they're not expensive.  They're also not 
a stock item on the consumer-level SBC's either.  I looked at one of our 
half-dozen ODroid C2's, and the main processor clock, X3, is under the 
heatsink, so I can't see what part is being used.  X1 and X2 are 
outside, and it doesn't appear that they are TCXO modules, although I 
didn't use a magnifier to check the part number and might have made an 
error.


The Nicegear DS3231 RTC has a TCXO, and might be the best low-cost 
choice at $12 (need to have an RPi, ODroid, or similar on which to mount 
it).  It's not that TCXO's are rare or expensive, it's that they're not 
often considered to be important to accuracy in many circles.



An Ovenized XCO is absolutely not required for IT-grade NTP servers.


No, but it is for my purposes here.  But, as I said in my post:



You really have to have at least a temperature compensated quartz crystal 
oscillator (TCXO) to even begin to think about an NTP server, for anything but 
the most rudimentary of timing.  Ovenized quartz oscillators (OCXO) and 
rubidium standards are the next step up, ...


I was just saying that OCXO and Rb are just the next step up if you 
would like more stability, that's all.  For 'within a second' on a 
GPS-disciplined clock (even without the 1PPS signal) you wouldn't 
necessarily need TXCO.  But that's what I meant by 'anything but the 
most rudimentary of timing.'  Rudimentary is down to the millisecond in 
my environment.  PTP takes you to the next level, and beyond that you 
don't use network timing but put a dedicated time distribution network 
running IRIG-B or similar.  But that is beyond the scope of a typical IT 
NTP server's needs.

Re: NIST NTP servers

[Sharon Goldberg]

Since we are on the subject, I would strongly recommend that you don't run
NTP on Linux 2.2.13, since its especially vulnerable to our IPv4
fragmentation attack.  "SunOS" also seems vulnerable, but I am not 100%
sure what systems that say they are "SunOS" actually are.  These OS will
fragment packets to 64 bytes, and are vulnerable to frag attacks using
"tiny" fragments.

See Section VI of our paper:
https://eprint.iacr.org/2015/1020.pdf

You can also test your OS here (scroll to the bottom).
http://www.cs.bu.edu/~goldbe/NTPattack.html


On Fri, May 13, 2016 at 10:46 AM, Chuck Anderson  wrote:

> On Fri, May 13, 2016 at 10:12:49AM -0400, Lamar Owen wrote:
> > On 05/11/2016 09:46 PM, Josh Reynolds wrote:
> > >maybe try [setting up an NTP server] with an odroid?
> > >
> > ...
> >
> > I have several ODroid C2's, and the first thing to note about them
> > is that there is no RTC at all.  Also, the oscillator is just a
> > garden-variety non-temperature-compensated quartz crystal, and not
> > necessarily a very precise one, either (precise quartz oscillators
> > can cost more than the whole ODroid board costs).  The XU4 and other
> > ODroid devices make nice single-board ARM computers, but have pretty
> > ratty oscillator precision.
> >
> > You really have to have at least a temperature compensated quartz
> > crystal oscillator (TCXO) to even begin to think about an NTP
> > server, for anything but the most rudimentary of timing.  Ovenized
> > quartz oscillators (OCXO) and rubidium standards are the next step
> > up, and most reasonably good GPS-disciplined clocks have at least an
> > ovenized quartz oscillator module (the Agilent Z3816 and kin are of
> > this type).
>
> Does anyone know of any COTS NTP servers that are based on non-ancient
> Linux kernel versions?  In 2012 we bought new GPS/CDMA NTP servers
> with OCXO that are based on Linux 2.4, but they are fiddly as you can
> imagine with such an ancient software stack.
>
> What would people recommend for NTP server hardware/software?
>
>


-- 
Sharon Goldberg
Computer Science, Boston University
http://www.cs.bu.edu/~goldbe

Re: NIST NTP servers

[Chuck Anderson]

On Fri, May 13, 2016 at 10:12:49AM -0400, Lamar Owen wrote:
> On 05/11/2016 09:46 PM, Josh Reynolds wrote:
> >maybe try [setting up an NTP server] with an odroid?
> >
> ...
> 
> I have several ODroid C2's, and the first thing to note about them
> is that there is no RTC at all.  Also, the oscillator is just a
> garden-variety non-temperature-compensated quartz crystal, and not
> necessarily a very precise one, either (precise quartz oscillators
> can cost more than the whole ODroid board costs).  The XU4 and other
> ODroid devices make nice single-board ARM computers, but have pretty
> ratty oscillator precision.
> 
> You really have to have at least a temperature compensated quartz
> crystal oscillator (TCXO) to even begin to think about an NTP
> server, for anything but the most rudimentary of timing.  Ovenized
> quartz oscillators (OCXO) and rubidium standards are the next step
> up, and most reasonably good GPS-disciplined clocks have at least an
> ovenized quartz oscillator module (the Agilent Z3816 and kin are of
> this type).

Does anyone know of any COTS NTP servers that are based on non-ancient
Linux kernel versions?  In 2012 we bought new GPS/CDMA NTP servers
with OCXO that are based on Linux 2.4, but they are fiddly as you can
imagine with such an ancient software stack.

What would people recommend for NTP server hardware/software?

Re: NIST NTP servers

[Laszlo Hanyecz]

On 2016-05-13 14:12, Lamar Owen wrote:

On 05/11/2016 09:46 PM, Josh Reynolds wrote:

maybe try [setting up an NTP server] with an odroid?


...


You really have to have at least a temperature compensated quartz 
crystal oscillator (TCXO) to even begin to think about an NTP server, 
for anything but the most rudimentary of timing.




There are WWVB clocks that try to sync nightly.  Many of them don't even 
have a second indicator, but they give reliable time to the minute.  NTP 
is a lot better than this as it continuously disciplines the clock 
instead of just lining it up once a day, but we're talking about doing 
this over the internet where we measure latency in milliseconds.  If 
you're working down at the picosecond level you will probably not be 
using NTP to distribute your clock signal.  Running an NTP client 
against pool servers is a lot better than not running it at all, but 
running it against a fancy local server with a GPSDO hooked up to it is 
only marginally better than the pool servers.


It all depends on what you want to do but a cheap ARM or Intel Atom 
computer works well for an NTP server (remember millisecond level 
accuracy).  If you can afford to build a secure bunker with armed guards 
and redundant everything for your time server that's good, but a few RPi 
style computers with GPS hats are almost as good, and you can buy a lot 
of them for very little money..


-Laszlo

Re: NIST NTP servers

[Mel Beckman]

Lamar,

You make it sound like TXCOs are rare, but they're actually quite common in 
most single board computers. True, you're probably not gonna find them in the 
$35 cellular-based SBCs, but since these temperature compensated oscillators 
cost less than a dollar each in quantity, they're quite common in most 
industrial species for well under $100.

An Ovenized XCO is absolutely not required for IT-grade NTP servers. If you 
need sub-microsecond  low-jitter leading-edge clocks, for BITS timing of SONET 
or radio networks for example, then an OXCO is helpful. But NTP itself is not 
that accurate. NTP can usually maintain time to only within tens of 
milliseconds over the public Internet, and can only achieve better than one 
millisecond accuracy in local area networks under ideal conditions. 

 -mel 

> On May 13, 2016, at 7:13 AM, Lamar Owen  wrote:
> 
>> On 05/11/2016 09:46 PM, Josh Reynolds wrote:
>> maybe try [setting up an NTP server] with an odroid?
>> 
> ...
> 
> I have several ODroid C2's, and the first thing to note about them is that 
> there is no RTC at all.  Also, the oscillator is just a garden-variety 
> non-temperature-compensated quartz crystal, and not necessarily a very 
> precise one, either (precise quartz oscillators can cost more than the whole 
> ODroid board costs).  The XU4 and other ODroid devices make nice single-board 
> ARM computers, but have pretty ratty oscillator precision.
> 
> You really have to have at least a temperature compensated quartz crystal 
> oscillator (TCXO) to even begin to think about an NTP server, for anything 
> but the most rudimentary of timing.  Ovenized quartz oscillators (OCXO) and 
> rubidium standards are the next step up, and most reasonably good 
> GPS-disciplined clocks have at least an ovenized quartz oscillator module 
> (the Agilent Z3816 and kin are of this type).
> 

Re: NIST NTP servers

[Lamar Owen]

On 05/11/2016 09:46 PM, Josh Reynolds wrote:

maybe try [setting up an NTP server] with an odroid?


...

I have several ODroid C2's, and the first thing to note about them is 
that there is no RTC at all.  Also, the oscillator is just a 
garden-variety non-temperature-compensated quartz crystal, and not 
necessarily a very precise one, either (precise quartz oscillators can 
cost more than the whole ODroid board costs).  The XU4 and other ODroid 
devices make nice single-board ARM computers, but have pretty ratty 
oscillator precision.


You really have to have at least a temperature compensated quartz 
crystal oscillator (TCXO) to even begin to think about an NTP server, 
for anything but the most rudimentary of timing.  Ovenized quartz 
oscillators (OCXO) and rubidium standards are the next step up, and most 
reasonably good GPS-disciplined clocks have at least an ovenized quartz 
oscillator module (the Agilent Z3816 and kin are of this type).

Re: NIST NTP servers

[Tony Finch]

Jean-Francois Mezei  wrote:
>
> Today, if someone were to jam the GPS signal in an areas in USA, you'd
> likely hear about large number of car accidents in the news before
> noticing your systems canMt get time from the GPS-NTP and went to a
> backup ip address (nist etc).

The USA and the UK governments regularly perform GPS jamming tests, but
they do so in remote areas. See
http://www.navcen.uscg.gov/?pageName=gpsServiceInterruptions
http://stakeholders.ofcom.org.uk/spectrum/gps-jamming-exercises/
(Dunno if other governments have similar exercises.)

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/  -  I xn--zr8h punycode
Lundy, Fastnet, Irish Sea, South Shannon: Northerly or northeasterly 4 or 5,
occasionally 6 except in south Shannon. Slight or moderate. Mainly fair. Good,
occasionally poor at first in south Lundy.

Re: NIST NTP servers

[George Herbert]

> On May 11, 2016, at 6:31 AM, Leo Bicknell  wrote:
> ...
> You're replacing one single point of failure with another.
> 
> Personally, my network gets NTP from 14 stratum 1 sources right now.
> You, and the hacker, do not know which ones.  You have to guess at least
> 8 to get me to move to your "hacked" time.  Good luck.

...except for people who think that N internet only servers is enough 
redundancy.

Pretty much anything with unfiltered outbound could put out enough forged UDP 
to effectively jam ALL the Stratum 1 servers for a given endpoint.


George William Herbert
Sent from my iPhone

RE: NIST NTP servers

[John Souvestre]

 >>> I know it's supposed to have better range and signal quality, but I
thought accuracy was about the same.  The variables that affect accuracy
are mostly external to the signal itself (propagation delay affected by
atmospheric conditions).

You are correct, but the what I read from NIST is that the Enhanced (PM)
format " will allow faster and more accurate synchronization, as well as
further address reception at particularly low SNIR."

So perhaps I should have said better "resolution" rather than "accuracy".  :)

John

    John Souvestre - New Orleans LA


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Chris Adams
Sent: 2016 May 12, Thu 21:21
To: nanog@nanog.org
Subject: Re: NIST NTP servers

Once upon a time, John Souvestre  said:
> The Enhanced WWVB signal has better range and more accuracy, but I don't
know if any receivers are available yet.

I know it's supposed to have better range and signal quality, but I
thought accuracy was about the same.  The variables that affect accuracy
are mostly external to the signal itself (propagation delay affected by
atmospheric conditions).

At one point, they were going to put a second transmitter closer to the
east coast, and it was going to be at the Army's Redstone Arsenal, next
to Huntsville, AL, where I live; I probably could have put a receiver in
a steel box and still had good signal!  NASA vetoed it though.
-- 
Chris Adams 

Re: NIST NTP servers

[Chris Adams]

Once upon a time, John Souvestre  said:
> The Enhanced WWVB signal has better range and more accuracy, but I don't know 
> if any receivers are available yet.

I know it's supposed to have better range and signal quality, but I
thought accuracy was about the same.  The variables that affect accuracy
are mostly external to the signal itself (propagation delay affected by
atmospheric conditions).

At one point, they were going to put a second transmitter closer to the
east coast, and it was going to be at the Army's Redstone Arsenal, next
to Huntsville, AL, where I live; I probably could have put a receiver in
a steel box and still had good signal!  NASA vetoed it though.
-- 
Chris Adams 

RE: NIST NTP servers

[John Souvestre]

 > ... a dedicated WWVB receiver.

The Enhanced WWVB signal has better range and more accuracy, but I don't know 
if any receivers are available yet.

John

John Souvestre - New Orleans LA

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jon Meek
Sent: 2016 May 11, Wed 10:40
To: nanog@nanog.org list
Subject: Re: NIST NTP servers

A note on using a Raspberry Pi as a NTP server. In my limited home lab
testing the RPi server had enough instability that Internet time sources
were always preferred by my workstation after ntpd had been running for a
while. Presumably this was due to the RPi's clock frequency drifting. At
some point I will look at it again.

If you do want to build your own Stratum 1 server you might want to glance
at:

https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf

and the references there.

I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but
the test device was clearly not up to the job. At some point I hope to
revisit this and do some more testing like I did for that poster. I'll add
in a CDMA server and a dedicated WWVB receiver.

Jon

Re: NIST NTP servers

[Lyndon Nerenberg]

[...]  but I would also have doubts over running anything business 
critical on a RP2.


We use them as reverse terminal servers, for dhcp/tftp bootstrapping other 
machines, and soon, NTP.  They are absolutely rock solid.  There's 
something to be said for "no moving parts inside."


--lyndon

Re: NIST NTP servers

[Laurent Dumont]

I did and it works! But as other mentioned, using a passive antenna 
means that you are very limited in where you can actually use the NTP 
server. The device failed to acquire a GPS lock with it was 2-3 feet 
away from a window. But when it did acquire a signal, it happily worked 
as a Stratum 1 device servicing what felt like all the CPE devices 
around Montreal.


It's definitely not something that can be massively deployed to data 
centers where the physical layout can change a lot. It might be worth 
looking into an active antenna but I would also have doubts over running 
anything business critical on a RP2.


https://coldnorthadmin.com/raspberry-pi-2-ntp-server-stratum-1/

Cheers,

Laurent

On 5/11/2016 6:47 AM, Dovid Bender wrote:

What about something like this?
http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
Has anyone used a Pi to create their own server?


On Wed, May 11, 2016 at 3:24 AM, Mel Beckman  wrote:


Regarding Roland’s reference to time and position spoofing via a hacked
GPS signal, the hacker has to get physical line of sight to the victim’s
antenna in order to succeed with this attack. That’s likely within a few
blocks, if not within a few feet. And a rooftop antenna might require a
drone attack. And how does the drone get guidance without a reliable GPS
signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my
experience designing data centers, that’s still pretty rare. Many NTP
systems use an active GPS antenna that can be hundreds of feet away. But
you can always put the $300 NTP server in an outdoor enclosure and power it
with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time
source.  Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated
yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity.
You just have to pay for it.

  -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke > wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein > wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman > wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratu

Re: NIST NTP servers

[Mel Beckman]

The WWV signal is still accurate within a few milliseconds. Light is fast. 
Really fast.

 -mel

> On May 12, 2016, at 10:19 AM, Jean-Francois Mezei 
>  wrote:
> 
> On 2016-05-11 10:30, Mel Beckman wrote:
> 
>> Read deeper into the thread and you'll find where I sourced inexpensive 
>> RF-based NTP servers using CDMA, GSM, and even WWV. 
> 
> For shortwave, you would need to calculate propagation delay between
> transmitter and receiver. (does signal reach via line of sight, bounce
> against ionosphere ?).
> 
> Since CDMA is dead outside the USA and drying in USA, I wouldn't rely on
> that.  If GSM towers rely on a GPS receiver on the tower and those
> towers are near enough to your location (< 30km), then chances are that
> blocked GPS signals at your location would also jam the signals at the
> GSM antenna.
> 
> And if you are setup to be totally autonomous in case of power failures,
> you need to know whether the GSM antenna you are relying on is also on
> permanent power backup or only has autonomy of a few hours.

Re: NIST NTP servers

[Jean-Francois Mezei]

On 2016-05-11 10:30, Mel Beckman wrote:

> Read deeper into the thread and you'll find where I sourced inexpensive 
> RF-based NTP servers using CDMA, GSM, and even WWV. 

For shortwave, you would need to calculate propagation delay between
transmitter and receiver. (does signal reach via line of sight, bounce
against ionosphere ?).

Since CDMA is dead outside the USA and drying in USA, I wouldn't rely on
that.  If GSM towers rely on a GPS receiver on the tower and those
towers are near enough to your location (< 30km), then chances are that
blocked GPS signals at your location would also jam the signals at the
GSM antenna.

And if you are setup to be totally autonomous in case of power failures,
you need to know whether the GSM antenna you are relying on is also on
permanent power backup or only has autonomy of a few hours.

Re: NIST NTP servers

[Jean-Francois Mezei]

On 2016-05-10 10:59, Stephane Bortzmeyer wrote:

> Yes, but they may switch it off for civilian use (by going encrypted,
> for instance) at any time, if it is better for *their* operations.


In the days of selected availability (GPS precision reduced on purpose),
the time signal was still very accurate from the point of view of using
it as a time source for computers.

When Clinton lifted SA, the military reserved the right to re-instate
it, and stated that it reserves the right to kill the civilian signal
outside the USA.

The EU considered launching their own constellation to counter the US
possibiliy of a shutdown.

Russia launched Glonass and eliminated the need for EU to launch their
own.  With Glonass now fairly common in GPS receivers, the USA can't
unilaterally shut it down anymore.

A satellite that is visible from Syria couldn't shutdown its signal
without affecting a WHOLE lot of other areas.  It is more likely that
the USA would just jam the frequencies from a plane over Syria or some
other means to geographically block those frequencies.

Today, if someone were to jam the GPS signal in an areas in USA, you'd
likely hear about large number of car accidents in the news before
noticing your systems canMt get time from the GPS-NTP and went to a
backup ip address (nist etc).

Re: NIST NTP servers

[Jared Mauch]

> On May 11, 2016, at 1:42 PM, Majdi S. Abbas  wrote:
> 
> On Wed, May 11, 2016 at 03:24:43PM +, Jay R. Ashworth wrote:
>> We're all aware this project is underway, right?
>> 
>>  https://www.ntpsec.org/
> 
>   Despite the name, I'm not aware of any significant protocol
> changes.  It's just a recent fork of the reference implementation
> minus the refclocks, which isn't particularly helpful if you /don't/
> trust network time sources.

I’ll also say that if you’re running NTP with -g beware.

"This option allows the time to be set to any value without restriction”

Game over if someone decided to go after you, you will never sync.  Make sure
systemd won’t just restart your daemon, if you get “invalid” time the process
dies and then you’re off.  Game over, press redo or back. (yay ti99/4a 
references)

- Jared

Re: NIST NTP servers

[Mike]

On 5/11/2016 11:24 AM, Jay R. Ashworth wrote:
> - Original Message -
>> From: "Jared Mauch" 
> 
>>> Yes, and properly monitor your ntpd instances.
>>
>> And upgrade them.
>>
>> Some software distributors don’t ship modern software.  if you
>> are using a distribution packaged ntpd it’s likely old and
>> difficult to determine its lineage due to how it’s packaged.
>>
>> If you’re using Redhat based systems consider using chrony
>> instead, even the new beta fedora 24 uses 4.2.6 derived code
>> vs 4.2.8
> 
> We're all aware this project is underway, right?
> 
>   https://www.ntpsec.org/
> 


Speaking of nascent time projects

http://nwtime.org/projects/ntimed/


So far, I like the overall architecture of the client/slave/master task
differentiation.

I've played around with the client in a test environment and ~it seems
to work OK~, but it looks like the project is still a bit in the future.

Re: NIST NTP servers

[Harlan Stenn]

Harlan Stenn writes:
> Sharon Goldberg writes:
> > Well, if you really want to learn about the NTP servers a target is using
> > you can always just sent them a regular NTP timing query (mode 3) and just
> > read off the IP address in the reference ID field of the response (mode 4).
> 
> Unless the server is an IPv6 server.  This trick only works for IPv4.
> 
> And we have a fix for all of this that will be out soon.

Also, the attacker will need to know the correct origin timestamp for
the brief window where that attack will work, and even if this happens
either the client or the server will see syslog entries alerting to the
abuse (if folks are running new enough versions of ntpd).

H

Re: NIST NTP servers

[Harlan Stenn]

Sharon Goldberg writes:
> Well, if you really want to learn about the NTP servers a target is using
> you can always just sent them a regular NTP timing query (mode 3) and just
> read off the IP address in the reference ID field of the response (mode 4).

Unless the server is an IPv6 server.  This trick only works for IPv4.

And we have a fix for all of this that will be out soon.
-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: NIST NTP servers

[Valdis . Kletnieks]

On Wed, 11 May 2016 17:23:31 -0700, Eric Kuhnke said:

> average of $150/mo x 500 = $75,000

Id worry more about the fact that somebody is willing to spend $75K/mo to
attack me than the fact that it might be possible to wiggle my time base a bit.
At that point, you *really* have to worry about other, more productive attacks,
such as social engineering (including spear phishing and bribery), or obtaining
a 0-day against something in your network.

Remember that the FBI has basically admitted that the most effective means of
uncloaking a Tor user has been browser 0-days, even though decloaking by pwning
a majority of the servers is possible

(Seriously - if *you* had $75K/mo to attack somebody, what would *you* spend
it on?)



pgpaZojwN0Y_I.pgp
Description: PGP signature

Re: NIST NTP servers

[Eygene Ryabinkin]

Wed, May 11, 2016 at 05:20:28PM -0700, Scott Weeks wrote:
> --- m...@beckman.org wrote:
>> From: Mel Beckman 
>> 
>> Accurate time to the millisecond is pretty much 
>> essential for any network troubleshooting. Say 
>> you want to diagnose a SIP problem. You collect 
>> transaction logs from both phones, the VoIP 
>> gateway, and the PBX. Now you try to merge them 
>> to derive the sequence of events. You NEED 
>> millisecond accuracy.
>> 
> 
> If all logs are sent to a unix server that does 
> syslogd the log entries would go into the file
> in order no matter what timestamp is on them.

Modulo latencies and jitter from different machines to the log server.
Millisecond precision can be harmed by this, easily.  Especially by
jitter and order of millisecond here isn't something non-existent
in a long-distant networks.
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.

Re: NIST NTP servers

[Josh Reynolds]

maybe try with an odroid?
On May 11, 2016 8:45 PM, "Jon Meek"  wrote:

> A note on using a Raspberry Pi as a NTP server. In my limited home lab
> testing the RPi server had enough instability that Internet time sources
> were always preferred by my workstation after ntpd had been running for a
> while. Presumably this was due to the RPi's clock frequency drifting. At
> some point I will look at it again.
>
> If you do want to build your own Stratum 1 server you might want to glance
> at:
>
> https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf
>
> and the references there.
>
> I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but
> the test device was clearly not up to the job. At some point I hope to
> revisit this and do some more testing like I did for that poster. I'll add
> in a CDMA server and a dedicated WWVB receiver.
>
> Jon
>

Re: NIST NTP servers

[Jon Meek]

A note on using a Raspberry Pi as a NTP server. In my limited home lab
testing the RPi server had enough instability that Internet time sources
were always preferred by my workstation after ntpd had been running for a
while. Presumably this was due to the RPi's clock frequency drifting. At
some point I will look at it again.

If you do want to build your own Stratum 1 server you might want to glance
at:

https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf

and the references there.

I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but
the test device was clearly not up to the job. At some point I hope to
revisit this and do some more testing like I did for that poster. I'll add
in a CDMA server and a dedicated WWVB receiver.

Jon

Re: NIST NTP servers

[Sharon Goldberg]

With the caveat that if some of the servers are inside your own private
network then learning who the servers are might be less useful.

But this could be an issue for targets who use servers that are exclusively
on the public internet.

On Wed, May 11, 2016 at 3:15 PM, Sharon Goldberg  wrote:

> Well, if you really want to learn about the NTP servers a target is using
> you can always just sent them a regular NTP timing query (mode 3) and just
> read off the IP address in the reference ID field of the response (mode 4).
>
>
> Reference ID reveals the target that the client is sync'd to.
>
> If you do this over and over as the client changes the servers it sync's
> to, you learn all the servers.
>
> Or if you are really keen you can use our "kiss-of-death" attack to learn
> all the servers a client is willing to take time from. See sections V.B-V.C
> of our paper.
>
> https://eprint.iacr.org/2015/1020.pdf
>
> Sharon
>
>
>
> On Wed, May 11, 2016 at 3:07 PM, Florian Weimer  wrote:
>
>> * Chris Adams:
>>
>> > First, out of the box, if you use the public pool servers (default
>> > config), you'll typically get 4 random (more or less) servers from the
>> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
>> > high chance of guessing the servers your system is using.
>>
>> A determined attacker will just run servers in the official pool.
>>
>>
>
>
> --
> Sharon Goldberg
> Computer Science, Boston University
> http://www.cs.bu.edu/~goldbe
>



-- 
Sharon Goldberg
Computer Science, Boston University
http://www.cs.bu.edu/~goldbe

Re: NIST NTP servers

[Sharon Goldberg]

Well, if you really want to learn about the NTP servers a target is using
you can always just sent them a regular NTP timing query (mode 3) and just
read off the IP address in the reference ID field of the response (mode 4).


Reference ID reveals the target that the client is sync'd to.

If you do this over and over as the client changes the servers it sync's
to, you learn all the servers.

Or if you are really keen you can use our "kiss-of-death" attack to learn
all the servers a client is willing to take time from. See sections V.B-V.C
of our paper.

https://eprint.iacr.org/2015/1020.pdf

Sharon



On Wed, May 11, 2016 at 3:07 PM, Florian Weimer  wrote:

> * Chris Adams:
>
> > First, out of the box, if you use the public pool servers (default
> > config), you'll typically get 4 random (more or less) servers from the
> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > high chance of guessing the servers your system is using.
>
> A determined attacker will just run servers in the official pool.
>
>


-- 
Sharon Goldberg
Computer Science, Boston University
http://www.cs.bu.edu/~goldbe

Re: NIST NTP servers

[Lyndon Nerenberg]

> On May 11, 2016, at 5:42 PM, Scott Weeks  wrote:
> 
> Wouldn't the buffers empty in a FIFO manner?

They will empty in whatever order the implementation decides to write them.

But what's more important is the order in which the incoming packets are 
presented to the syslogd process. If you're listening on TCP connections, the 
receive order is very much determined by the strategy the syslogd 
implementation uses to read from FDs with available data.  I.e. elevator scan, 
lowest/highest first, circular queue, ...  In a threaded implementation, your 
reader workers, buffer writers, etc., are all at the mercy of the threading 
implementation; it's difficult to control thread dispatch ordering at that 
level of granularity.

--lyndon

Re: NIST NTP servers

[Gary E. Miller]

Yo Scott!

On Wed, 11 May 2016 17:42:34 -0700
"Scott Weeks"  wrote:

> > If all logs are sent to a unix server that does 
> > syslogd the log entries would go into the file
> > in order no matter what timestamp is on them.  
> 
> syslogd can have quite large buffers.
> ---
> 
> 
> Wouldn't the buffers empty in a FIFO manner?

Nope, each local syslog waits until its local buffer is full, or timed
out, then sends to the main syslog server.

The default flush_timeout() for syslog-ng is 10 seconds.  That is a long
time when debugging SIP.

http://www.syslog.org/syslog-ng/v2/

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpBtkPYjkTeV.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Scott Weeks]

--- g...@rellim.com wrote:
From: "Gary E. Miller" 

Yo Scott!

On Wed, 11 May 2016 17:20:28 -0700
"Scott Weeks"  wrote:

> If all logs are sent to a unix server that does 
> syslogd the log entries would go into the file
> in order no matter what timestamp is on them.

syslogd can have quite large buffers.
---


Wouldn't the buffers empty in a FIFO manner?

scott

Re: NIST NTP servers

[Gary E. Miller]

Yo Scott!

On Wed, 11 May 2016 17:20:28 -0700
"Scott Weeks"  wrote:

> If all logs are sent to a unix server that does 
> syslogd the log entries would go into the file
> in order no matter what timestamp is on them.

syslogd can have quite large buffers.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpByl5us85nK.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Eric Kuhnke]

Compared to the scale of the budget of small research projects run by
national intelligence agency sized organizations, you wouldn't have to be
very well funded to run a sizeable proportion of all tor exit nodes with
some degree of plausible deniability...

500 credit cards

500 unique bililng names/addresses and sets of contact info

spread 500 1U servers around the world in as many geographically unique
locations as you can find, with every dedicated hosting/colo company...

average of $150/mo x 500 = $75,000



On Wed, May 11, 2016 at 5:08 PM,  wrote:

> On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said:
> > * Chris Adams:
> >
> > > First, out of the box, if you use the public pool servers (default
> > > config), you'll typically get 4 random (more or less) servers from the
> > > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > > high chance of guessing the servers your system is using.
> >
> > A determined attacker will just run servers in the official pool.
>
> Such attacks have allegedly been attempted against Tor by certain
> very well funded adversaries.
>
> Thus my statement that if you're seeing that scale attack on your time
> sources, the fact that your time source is being attacked is the *least*
> of your problems...
>

Re: NIST NTP servers

[Scott Weeks]

--- m...@beckman.org wrote:
From: Mel Beckman 

Accurate time to the millisecond is pretty much 
essential for any network troubleshooting. Say 
you want to diagnose a SIP problem. You collect 
transaction logs from both phones, the VoIP 
gateway, and the PBX. Now you try to merge them 
to derive the sequence of events. You NEED 
millisecond accuracy.


If all logs are sent to a unix server that does 
syslogd the log entries would go into the file
in order no matter what timestamp is on them.

scott

Re: NIST NTP servers

[Valdis . Kletnieks]

On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said:
> * Chris Adams:
>
> > First, out of the box, if you use the public pool servers (default
> > config), you'll typically get 4 random (more or less) servers from the
> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > high chance of guessing the servers your system is using.
>
> A determined attacker will just run servers in the official pool.

Such attacks have allegedly been attempted against Tor by certain
very well funded adversaries.

Thus my statement that if you're seeing that scale attack on your time
sources, the fact that your time source is being attacked is the *least*
of your problems...


pgplgKIyYlw3x.pgp
Description: PGP signature

Re: NIST NTP servers

[Eric Kuhnke]

Cellular carriers also use GPS timing for many reasons that are not readily
apparent at the layer 3 router/IP/BGP network level. One big need is RF
related, back-to-back sector antenna frequency re-use with GPS synced
timing on the remote radio heads, such as an ABAB configuration on a tower
or rooftop site.

The same with some much less costly near consumer grade WISP radio
platforms and PTP radio systems nowadays.

In such a configuration the GPS timing signal from the local GPS receiver
(small cone shaped or puck antennas at the site) is actually the primary,
and whatever NTP-based GPS signal the network node might have access to is
secondary.


On Wed, May 11, 2016 at 12:10 PM, Mel Beckman  wrote:

> No, many cell carriers run their own completely independent timing
> networks. I support some head-ends where they have rubidium clocks and a
> T1-delivered time source. They do reference GPS, and many cell sites have
> GPS as a backup clock (you can see their conical antennas on the very top
> of the tower). But most cellular providers are very protective of their
> time sources. They’re also typically clocking SONET networks too, which
> requires BITS.
>
>  -mel
>
>
> JAshworth said:
> > CDMA and GSM are false diversity: both network types nodes *get their
> time*
> > from GPS, so far as I know.
>
>
> > On May 11, 2016, at 10:54 AM, valdis.kletni...@vt.edu wrote:
> >
> > On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said:
> >
> >> CDMA and GSM are false diversity: both network types nodes *get their
> time*
> >> from GPS, so far as I know.
> >
> > I'll make the fairly reasonable assumption that most readers of this
> list have
> > networks that span multiple buildings.
> >
> > If somebody is managing to figure out that you have a GPS in Building
> 37, and a
> > GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks
> at
> > other locations and getting close enough to all of them at the same time
> to
> > conduct a successful spoofing attack, all just to move your time source a
> > few seconds off
> >
> > ...  then the fact that GPS is spoofable is probably *NOT* your biggest
> > security problem.
> >
>
>

Re: NIST NTP servers

[Mel Beckman]

No, many cell carriers run their own completely independent timing networks. I 
support some head-ends where they have rubidium clocks and a T1-delivered time 
source. They do reference GPS, and many cell sites have GPS as a backup clock 
(you can see their conical antennas on the very top of the tower). But most 
cellular providers are very protective of their time sources. They’re also 
typically clocking SONET networks too, which requires BITS.

 -mel


JAshworth said:
> CDMA and GSM are false diversity: both network types nodes *get their time* 
> from GPS, so far as I know.


> On May 11, 2016, at 10:54 AM, valdis.kletni...@vt.edu wrote:
> 
> On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said:
> 
>> CDMA and GSM are false diversity: both network types nodes *get their time*
>> from GPS, so far as I know.
> 
> I'll make the fairly reasonable assumption that most readers of this list have
> networks that span multiple buildings.
> 
> If somebody is managing to figure out that you have a GPS in Building 37, and 
> a
> GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at
> other locations and getting close enough to all of them at the same time to
> conduct a successful spoofing attack, all just to move your time source a
> few seconds off
> 
> ...  then the fact that GPS is spoofable is probably *NOT* your biggest
> security problem.
> 

Re: NIST NTP servers

[Florian Weimer]

* Chris Adams:

> First, out of the box, if you use the public pool servers (default
> config), you'll typically get 4 random (more or less) servers from the
> pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> high chance of guessing the servers your system is using.

A determined attacker will just run servers in the official pool.

Re: NIST NTP servers

[Lamar Owen]

On 05/11/2016 07:46 AM, Baldur Norddahl wrote:
But would you not need to actually spend three times $300 to get a 
good redundant solution?


While we are there, why not go all the way and get a rubidium standard 
with GPS sync? Anyone know of a (relatively) cheap solution with NTP 
output?


Ebay has several Symmetricom, Microsemi, Datum, Spectracom, and even 
Agilent solutions for prices from a few hundred US$ to a couple of 
thousand US$.  Even something like the Agilent Z3801, Z3805, or Z3816 
can be found for a few hundred US$.   New, these things are in the 
$10,000+ territory.  About the same range as mid-range ethernet gear.


I like our SSU2000, personally.

Re: NIST NTP servers

[Lamar Owen]

On 05/11/2016 12:05 AM, Joe Klein wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?


...

I remember it like it was only four years ago oh, wait

We have multiple sync sources ourselves, with a Symmetricom (formerly 
Datum) SSU2000 setup with a cesium PRS, a rubidium secondary, and an 
ovenized quartz for tertiary oscillators.  SSU2000 architecture is 
separate control and data planes, with time-sync on a different 
interface from the LAN-facing NTP NIC.  And the control plane is 
firewalled off from the main LAN.  An Agilent (now Symmetricom) Z3816 is 
secondary.


PC and SBC (RasPi, etc) oscillators are just not accurate enough for 
Stratum 1 standards; at best stratum 3 or 4, even when directly 
GPS-disciplined (stratum is NOT just a synonym for 'level' as a 
particular stratum really has stability, precision, and accuracy 
requirements).  WWV plus GPS; GPS as you may or may not be aware, is 
spoofable and is not as accurate as one might want.  Neither is WWV.


Good reference for time-nuts is, well, the 'time-n...@febo.com' mailing 
list.


(We're a radio astronomy observatory; accurate time and frequency 
standards are  a must here, especially as position accuracy of radio 
telescopes approach tens of arcseconds).

Re: NIST NTP servers

[Valdis . Kletnieks]

On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said:

> CDMA and GSM are false diversity: both network types nodes *get their time*
> from GPS, so far as I know.

I'll make the fairly reasonable assumption that most readers of this list have
networks that span multiple buildings.

If somebody is managing to figure out that you have a GPS in Building 37, and a
GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at
other locations and getting close enough to all of them at the same time to
conduct a successful spoofing attack, all just to move your time source a
few seconds off

...  then the fact that GPS is spoofable is probably *NOT* your biggest
security problem.



pgpYTudayH1Sq.pgp
Description: PGP signature

Re: NIST NTP servers

[Majdi S. Abbas]

On Wed, May 11, 2016 at 03:24:43PM +, Jay R. Ashworth wrote:
> We're all aware this project is underway, right?
> 
>   https://www.ntpsec.org/

Despite the name, I'm not aware of any significant protocol
changes.  It's just a recent fork of the reference implementation
minus the refclocks, which isn't particularly helpful if you /don't/
trust network time sources.

Long term, be looking at NTS:

https://datatracker.ietf.org/doc/draft-ietf-ntp-network-time-security/

In the meanwhile, I'd recommend something along the following
lines:

- Several nearby upstream servers configured per time server, per site
(As diversely as possible.)

- Diverse reference clocks (I run everything from WWV to GPS
  here.) providing authenticated time to your servers.

- That all your time servers in all sites be configured in an
authenticated full mesh of symmetric peers, allowing the other
sites to provide time to a site that has lost its upstream
servers or for whatever reason does not trust them at the moment.

And of course, ensure any hosts whose clocks you care about are
talking to at least a few of these, and preferably several.  I know the
common case configuration is either default/ntp-pool, or "we have two
time servers in this site and everything just chimes from them," but
neither is that great of a configuration.

--msa

Re: NIST NTP servers

[Jay R. Ashworth]

- Original Message -
> From: "Mel Beckman" 

> Read deeper into the thread and you'll find where I sourced inexpensive 
> RF-based
> NTP servers using CDMA, GSM, and even WWV. All radically different 
> technologies
> that are unlikely to have common failure modes. But yes, buying different
> brands can't hurt either.

CDMA and GSM are false diversity: both network types nodes *get their time* 
from GPS, so far as I know.

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: NIST NTP servers

[Jay R. Ashworth]

- Original Message -
> From: "Jared Mauch" 

>> Yes, and properly monitor your ntpd instances.
> 
> And upgrade them.
> 
> Some software distributors don’t ship modern software.  if you
> are using a distribution packaged ntpd it’s likely old and
> difficult to determine its lineage due to how it’s packaged.
> 
> If you’re using Redhat based systems consider using chrony
> instead, even the new beta fedora 24 uses 4.2.6 derived code
> vs 4.2.8

We're all aware this project is underway, right?

  https://www.ntpsec.org/

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: NIST NTP servers

[Scott Whyte]

On 5/10/16 21:05, Joe Klein wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
I believe this will become a stronger need over time, and apply to more 
than NTP: 
http://arstechnica.com/security/2016/02/using-ipv6-with-linux-youve-likely-been-visited-by-shodan-and-other-scanners/

6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  wrote:


I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

  -mel beckman


On May 10, 2016, at 5:18 PM, Chris Adams  wrote:

Once upon a time, Mel Beckman  said:

Boss: So how did a hacker get in and crash our accounting server, break

our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe Random Hacker isn't going to have a
high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the
common software won't step the clock more than IIRC 15 minutes.  Yes,
that can cause problems, but not the catastrophes of years in the future
or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
not so sure, and I haven't seen proof to the contrary.
--
Chris Adams 

RE: NIST NTP servers

[Chuck Church]

-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Leo Bicknell
>Sent: Wednesday, May 11, 2016 9:31 AM
>To: nanog@nanog.org
>Subject: Re: NIST NTP servers

>Personally, my network gets NTP from 14 stratum 1 sources right now.
>You, and the hacker, do not know which ones.  You have to guess at least
>8 to get me to move to your "hacked" time.  Good luck.

>Redundancy is the solution, not a new single point of failure.  GPS can be 
>part of the redundancy, not a sole solution.

This seems like the most reasonable advise.  If this truly becomes a concern, I 
would think IPS vendors could implement signatures to look for bad time.  Lots 
of ways to do this 
- look for a difference between the IPS realtime and NTP status versus the 
incoming packets.
- look for duplicate NTP responses, or responses that weren't requested 
- duplicate responses, but with differing TTLs, which might hint at one being 
spoofed.

Chuck

Re: NIST NTP servers

[Brandon Vincent]

GPS + a cesium or rubidium frequency standard is all you need.

Too expensive? Then time isn't important to your organization.

Re: NIST NTP servers

[Leo Bicknell]

In a message written on Wed, May 11, 2016 at 09:00:54AM -0500, Josh Reynolds 
wrote:
> I hope your receivers aren't all from a single source.

I have 4 each ACTS, GPS, and CDMA in my list, agumented with a pair
of PTP.  Amazingly right now all but 3 are within 2 microsconds of
each other, and those three outliers are 10 and 35 microseconds
off.  That's pretty impressive!

I didn't have to buy any of them, because various trustable entities
run those infrastructures.  Some of the trustable entites are the
same ones that send the time up to the GPS satellites. :)

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpKDSyi44ype.pgp
Description: PGP signature

Re: NIST NTP servers

[Mel Beckman]

Josh,

Read deeper into the thread and you'll find where I sourced inexpensive 
RF-based NTP servers using CDMA, GSM, and even WWV. All radically different 
technologies that are unlikely to have common failure modes. But yes, buying 
different brands can't hurt either. 

 -mel beckman

> On May 11, 2016, at 7:15 AM, Josh Reynolds  wrote:
> 
> I hope your receivers aren't all from a single source.
> 
> I was in Iraq when this (
> http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-on-gps/
> ) happened, which meant I had no GPS guided indirect fire assets for 2
> weeks.
> 
>> On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell  wrote:
>> In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman 
>> wrote:
>>> All because of misplaced trust in a tiny UDP packet that can worm its way 
>>> into your network from anywhere on the Internet.
>>> 
>>> I say you’re crazy if you don’t run a GPS-based NTP server, especially 
>>> given that they cost as little as $300 for very solid gear. Heck, get two 
>>> or three!
>> 
>> You're replacing one single point of failure with another.
>> 
>> Personally, my network gets NTP from 14 stratum 1 sources right now.
>> You, and the hacker, do not know which ones.  You have to guess at least
>> 8 to get me to move to your "hacked" time.  Good luck.
>> 
>> Redundancy is the solution, not a new single point of failure.  GPS
>> can be part of the redundancy, not a sole solution.
>> 
>> --
>> Leo Bicknell - bickn...@ufp.org
>> PGP keys at http://www.ufp.org/~bicknell/

Re: NIST NTP servers

[Mel Beckman]

Andreas,

Most data centers will require a remotely positioned NTP server, which is 
actually easier and cheaper than a remotely located active GPS antenna. I have 
placed the $300 commercial NTP servers in an environmental box on the roof, 
powering t by PoE, without problems. 

You don't need a redundant network either, nor redundant power. Just plunk down 
two of these gizmos, or as I suggested elsewhere, deploy one or more CDMA, GSM, 
or WWV-based clocks, for as much local infrastructure and signal source 
diversity as you like (I sourced some of these units earlier in the thread, all 
well less than $1K each. You pay more for diversity, but you get more too.

In response to the several DIYers on this thread: Anyone who thinks they're 
building a raspberry pi-based GPS NTP server for just $150 is kidding 
themselves. They forgot to include their labor, which is worth far more than 
the $300 for a commercial unit. The same goes for people who complain about 
even the minimal $300 price, forgetting that a commercial product has to pay 
for marketing, support, and make a profit. 

External NTP has two kinds of vulnerabilities: the ones we know, and the ones 
we don't. The ones we know are serious enough the pat the ones we don't should 
be considered with respect. Maybe diversity in Internet sources is a cure, 
maybe it isn't. But diverse RF sources is demonstrably more secure than the 
Internet.   

My point stands: secure external RF NTP sources are so plentiful that relying 
on Internet NTP is just plain crazy. 

 -mel beckman

> On May 11, 2016, at 7:12 AM, Andreas Ott  wrote:
> 
> Hi,
> 
>> Boss: That sounds expensive. How much are we talking?
>> IT guy: $300
> 
> Beware!
> 
> Over the past year we made engineering samples to deploy to datacenters.
> The goal was to use GPS and PPS to discipline ntpd appliances and serve 
> as stratum 1 to other NTP distribution servers without the $5k price tag
> of commercial NTP 1RU gear. We also deliberately not pursued the path of
> running antenna coax from the roof to a receiver, as that is not an
> option in all the datacenters where we need to deploy.
> 
> These appliances were built for lesss than $150 as 
> 
> (a) Raspberry-Pi with GPS receiver board
> 
> (b) Garmin 18(x) LVC with DB-9 to an "older whitebox server"
> 
> In my experience, most locations inside datacenters where you have good
> power and network connectivity have not "good enough" GPS signal reception
> due to servers emitting lots of RF noise in the range 1-2 GHz on the
> L-band. A brand new suite in the datacenter had OK GPS quality, but
> once we added 20+ racks with 40 servers each, GPS reception was pretty
> much gone. This hardware was an active antenna with less than 6 feet of
> cabling routed to the top of the network cabling rack. Most smartphones
> can run an app to show you the GPS signal on the phone, just walk around
> your datacenter and compare the signal.
> 
> The only workable solution was to move the GPS clock to a location
> where it had good GPS signal but neither redundant network nor conditioned
> power (aka. "on my desk near a south facing window"). It also works pretty 
> well "in my garage".
> 
> In places where GPS reception is good, you can achieve 10E-06 seconds
> accuracy over time even with cheap hardware. If you chose to run the DB-9
> NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor
> you can still get better than 10E-03 seconds accuracy.
> 
> 
> -andreas
> -- 
> Andreas Ott   (Time-Nut)   K6OTT   +1.408.431.8727   andr...@naund.org

Re: NIST NTP servers

[Josh Reynolds]

I hope your receivers aren't all from a single source.

I was in Iraq when this (
http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-on-gps/
) happened, which meant I had no GPS guided indirect fire assets for 2
weeks.

On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell  wrote:
> In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman 
> wrote:
>> All because of misplaced trust in a tiny UDP packet that can worm its way 
>> into your network from anywhere on the Internet.
>>
>> I say you’re crazy if you don’t run a GPS-based NTP server, especially given 
>> that they cost as little as $300 for very solid gear. Heck, get two or three!
>
> You're replacing one single point of failure with another.
>
> Personally, my network gets NTP from 14 stratum 1 sources right now.
> You, and the hacker, do not know which ones.  You have to guess at least
> 8 to get me to move to your "hacked" time.  Good luck.
>
> Redundancy is the solution, not a new single point of failure.  GPS
> can be part of the redundancy, not a sole solution.
>
> --
> Leo Bicknell - bickn...@ufp.org
> PGP keys at http://www.ufp.org/~bicknell/

Re: NIST NTP servers

[Andreas Ott]

Hi,

> Boss: That sounds expensive. How much are we talking?
> IT guy: $300

Beware!

Over the past year we made engineering samples to deploy to datacenters.
The goal was to use GPS and PPS to discipline ntpd appliances and serve 
as stratum 1 to other NTP distribution servers without the $5k price tag
of commercial NTP 1RU gear. We also deliberately not pursued the path of
running antenna coax from the roof to a receiver, as that is not an
option in all the datacenters where we need to deploy.

These appliances were built for lesss than $150 as 

(a) Raspberry-Pi with GPS receiver board

(b) Garmin 18(x) LVC with DB-9 to an "older whitebox server"

In my experience, most locations inside datacenters where you have good
power and network connectivity have not "good enough" GPS signal reception
due to servers emitting lots of RF noise in the range 1-2 GHz on the
L-band. A brand new suite in the datacenter had OK GPS quality, but
once we added 20+ racks with 40 servers each, GPS reception was pretty
much gone. This hardware was an active antenna with less than 6 feet of
cabling routed to the top of the network cabling rack. Most smartphones
can run an app to show you the GPS signal on the phone, just walk around
your datacenter and compare the signal.

The only workable solution was to move the GPS clock to a location
where it had good GPS signal but neither redundant network nor conditioned
power (aka. "on my desk near a south facing window"). It also works pretty 
well "in my garage".

In places where GPS reception is good, you can achieve 10E-06 seconds
accuracy over time even with cheap hardware. If you chose to run the DB-9
NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor
you can still get better than 10E-03 seconds accuracy.


-andreas
-- 
Andreas Ott   (Time-Nut)   K6OTT   +1.408.431.8727   andr...@naund.org

RE: NIST NTP servers

[Allan Liska]

On 5/10/2016 at 10:30 AM, "Chuck Church"  wrote:

>
>It doesn't really.  Granted there are a lot of CVEs coming out for 
>NTP the
>last year or so.  But I just don't think there are that many 
>attacks on it.
>It's just not worth the effort.  Changing time on devices is more 
>an
>annoyance than anything, and doesn't necessarily get you into a 
>device.
>Sure you can hide your tracks a little by altering time in logs 
>and altering
>it back, but that's more of an in-depth nation-state kind of 
>attack, not
>going to be a script kiddie kind of thing.  Just follow the best 
>practices
>for verifying packet sources and NTP security itself, and you 
>should be ok.
>
>Chuck

I would argue that the fact the NTP can, and has been, be used in DDoS 
amplification attacks is a serious concern for using protocol going forward.



allan

Re: NIST NTP servers

[Ryan Harden]

_Everything_ has vulnerabilities and using _any_ external source opens your 
network and infrastructure to disruptions. NTP has been used for DDoS 
amplification attacks recently, but so has DNS and other well known/heavily 
used protocols.

With the right protections, syncing with an external NTP source is perfectly 
acceptable and safe.
Further, it’s generally a good idea to ‘peer’ (not just sync) your NTP servers 
with a few external sources. This removes the dependence on a single source and 
helps ensure that your time source agrees with the rest of the world. Peering 
requires interaction with the owners of the remote site, which establishes a 
basic level of trust that they’ll provide an accurate and stable service.

I’ve attached a diagram (sanitized) of what our NTP service will look like 
after an upcoming refresh.
All external sources are trusted and will be peered. All time devices peer with 
four other sources to ensure there is always a live source to sync/peer with.
A DNS record with round-robin is used for local clients to connect to the local 
Stratum 2 devices. The Stratum 1 GPS will not be directly accessible by users.

/Ryan

[cid:5676FF89-CBC8-42F7-84CE-69F431C23E48@int.ancker.net]

Ryan Harden
Research and Advanced Networking Architect
University of Chicago - ASN160
P: 773.834.5441

On May 10, 2016, at 5:48 AM, Steven Miano 
mailto:mian...@gmail.com>> wrote:

NTP has vulnerabilities, so using an external source opens your networks
and infrastructure to disruptions.

Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict
incoming traffic and not rely on volunteers or external entities (which may
undergo maintenance or budget issues).

My preference is more so something akin to the GLN180PEX (I am not
affiliated or paid to endorse this product). It allows you to use commodity
hardware (like a decommissioned 1U or several preferably) and creation of
ones own reliable internal time source(s). Introducing black boxes into a
production (revenue generation or expected services by paying customers)
environment is undesirable.

From there setting up NTPd, Chronyd, and PTPd is up to you.

Relying on satellites may seem like just another external reliance, but the
next life is proposing a design life of 12 years.

On Mon, May 9, 2016 at 11:12 PM, Majdi S. Abbas 
mailto:m...@latt.net>> wrote:

On Tue, May 10, 2016 at 03:08:16AM +, Mel Beckman wrote:
NTP has vulnerabilities that make it generally unsuitable for
provider networks. I strongly recommend getting a GPS-based
time server. These are as cheap as $300. Here is one I use quite a bit:

   So how does this stop from distributing time to their
customers via NTP?

   GPS doesn't save the protocol, in particular where the S1
clocks involved are embedded devices with rather coarse clocks and
timestamping.

   --msa




--
Miano, Steven M.
http://stevenmiano.com

Re: NIST NTP servers

[Leo Bicknell]

In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman 
wrote:
> All because of misplaced trust in a tiny UDP packet that can worm its way 
> into your network from anywhere on the Internet.
> 
> I say you’re crazy if you don’t run a GPS-based NTP server, especially given 
> that they cost as little as $300 for very solid gear. Heck, get two or three!

You're replacing one single point of failure with another.

Personally, my network gets NTP from 14 stratum 1 sources right now.
You, and the hacker, do not know which ones.  You have to guess at least
8 to get me to move to your "hacked" time.  Good luck.

Redundancy is the solution, not a new single point of failure.  GPS
can be part of the redundancy, not a sole solution.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpZ8nfasXwtV.pgp
Description: PGP signature

Re: NIST NTP servers

[Eygene Ryabinkin]

Tue, May 10, 2016 at 04:59:02PM +0200, Stephane Bortzmeyer wrote:
> On Tue, May 10, 2016 at 10:52:28AM -0400,
>  valdis.kletni...@vt.edu  wrote 
>  a message of 37 lines which said:
> 
> > Note that they *do* have motivation to keep it working, simply
> > because so much of their *own* gear (from gear for individual
> > soldiers all the way to strategic bombers and aircraft carriers)
> > wants a working GPS signal...
> 
> Yes, but they may switch it off for civilian use (by going encrypted,
> for instance) at any time, if it is better for *their* operations.

Civilian signals are already degraded in terms of accuracy (without
assisted GPS) and military ones are encrypted (but see [1]).  The
primary reason for encryption is, by the way, to address spoofing
issues for the mil people themselves, but it is also good not to arm
the potential enemy with the precise coordinates or to be able to
do spoofing for them.

And since many civilian, but nonetheless, vital orgs are using
civilian parts, it could be hard to shut it off without affecting some
parts of the infrastructure.  Not that nobody wants that, but it will
give an unneeded resonance and could lead to the terrible mess.

[1] http://www.gps.gov/technical/codeless/
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.

Re: NIST NTP servers

[Baldur Norddahl]

But would you not need to actually spend three times $300 to get a good 
redundant solution?


While we are there, why not go all the way and get a rubidium standard 
with GPS sync? Anyone know of a (relatively) cheap solution with NTP output?


https://en.wikipedia.org/wiki/Rubidium_standard

Regards,

Baldur

On 2016-05-11 09:24, Mel Beckman wrote:

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

  -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
mailto:jskl...@gmail.com>> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
mailto:m...@beckman.org>> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
mailto:c...@cmadams.net>> wrote:

Once upon a time, Mel Beckman mailto:m...@beckman.org>> said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our n

Re: NIST NTP servers

[Steven Miano]

Building a S1 system with RaspberryPis would not fly in most of the
corporate/enterprise environments I've worked in (random 'appliances',
non-uniformity, and lack of support are all glaring issues).

Get a PCIe card with a BNC connector and dual power supplies for life in a
data center.

For home/hobby use a Garmin 18x LVC and any spare compute is a great
project: http://www.catb.org/gpsd/gpsd-time-service-howto.html

On Wed, May 11, 2016 at 6:47 AM, Dovid Bender  wrote:

> What about something like this?
> http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
> Has anyone used a Pi to create their own server?
>
>
> On Wed, May 11, 2016 at 3:24 AM, Mel Beckman  wrote:
>
> > Regarding Roland’s reference to time and position spoofing via a hacked
> > GPS signal, the hacker has to get physical line of sight to the victim’s
> > antenna in order to succeed with this attack. That’s likely within a few
> > blocks, if not within a few feet. And a rooftop antenna might require a
> > drone attack. And how does the drone get guidance without a reliable GPS
> > signal? :)
> >
> > Eric, I agree that sometimes a site can’t get a GPS signal, but in my
> > experience designing data centers, that’s still pretty rare. Many NTP
> > systems use an active GPS antenna that can be hundreds of feet away. But
> > you can always put the $300 NTP server in an outdoor enclosure and power
> it
> > with PoE.
> >
> > There’s always CDMA, GSM, and even WWV for a less-accurate plan B time
> > source.  Here’s a somewhat pricey ($700) CDMA gizmo I haven’t
> investigated
> > yet:
> >
> > http://www.beaglesoft.com/celsynhowworks.htm
> >
> > And their $400 WWV-based Stratum 1 time server:
> >
> > http://www.beaglesoft.com/radsynreceiver.htm
> >
> > So if you want non-Internet clock diversity, you can have clock
> diversity.
> > You just have to pay for it.
> >
> >  -mel
> >
> > On May 10, 2016, at 9:18 PM, Eric Kuhnke  > eric.kuh...@gmail.com>> wrote:
> >
> > For quite some time, in debian the default configuration for the
> ntpd.conf
> > that ships with the package for the ntpd is to poll from four different,
> > semi-randomly assigned DNS pool based sources. I believe the same is true
> > for redhat/centos.
> >
> > In the event that one out of four sources is wildly wrong the ntpd will
> > ignore it.
> >
> > If people have routers/networking equipment inside their network that
> only
> > supports retrieving ntp from one IP address (or hostname) and have
> manually
> > configured it to request time from a single external source, not their
> own
> > internal ntpd that is <10ms away, bad things could definitely happen.
> >
> > It is worthwhile to have both polling from external sources via IP as
> well
> > as GPS sync. Many locations in a network have no hope of getting a GPS
> > signal or putting an antenna with a clear view to the sky, but may be on
> a
> > network segment that is <4ms away from many other nodes where you can
> > colocate a 1U box and GPS antenna.
> >
> > On Tue, May 10, 2016 at 9:05 PM, Joe Klein  > jskl...@gmail.com>> wrote:
> >
> > Is this group aware of the incident with tock.usno.navy.mil &
> > tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost
> 12
> > years for the period of one hour, then return?
> >
> > The reasons were not fully explained, but the impact was global. Routers,
> > switches, power grids, phone systems, certificates, encryption, Kerberos,
> > logging and any tightly coupled transaction systems were impacted.
> >
> > So I began doing 'security research' on the topic (don't confuse me with
> > joe hacker), and discovered both interesting and terrifying issues,
> which I
> > will not disclose on an open forum.
> >
> > Needless to say, my suggestions are:
> > 1. Configure a trusted time source and good time stratum architecture for
> > your organization.
> > 2. When identifying your source of time, the majority of the technologies
> > can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
> > authentication.
> > 3. For distribution of time information inside your organization, ensure
> > your critical systems (Encryption, PKI, transactions, etc) are using your
> > redundant sources and authentication.
> > 4. Operating systems, programming languages, libraries, and applications
> > are sensitive to time changes and can fail in unexpected ways. Test them
> > before it's too late.
> > 5. Disallow internal system to seek NTP from other sources beyond your
> edge
> > routers.
> > 6. All core time systems should be monitored by your security team or
> SOC.
> >
> > One question, is this a topic anyone would find interested at a future
> > NANOG? Something like "Hacking and Defending time?".
> >
> >
> > Joe Klein
> > "Inveniam viam aut faciam"
> >
> > PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
> >
> > On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  > m...@beckman.org>> wrote:
> >
> > I don't pretend to know all the ways a hacker can find out what nap
> > ser

Re: NIST NTP servers

[Dovid Bender]

What about something like this?
http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
Has anyone used a Pi to create their own server?


On Wed, May 11, 2016 at 3:24 AM, Mel Beckman  wrote:

> Regarding Roland’s reference to time and position spoofing via a hacked
> GPS signal, the hacker has to get physical line of sight to the victim’s
> antenna in order to succeed with this attack. That’s likely within a few
> blocks, if not within a few feet. And a rooftop antenna might require a
> drone attack. And how does the drone get guidance without a reliable GPS
> signal? :)
>
> Eric, I agree that sometimes a site can’t get a GPS signal, but in my
> experience designing data centers, that’s still pretty rare. Many NTP
> systems use an active GPS antenna that can be hundreds of feet away. But
> you can always put the $300 NTP server in an outdoor enclosure and power it
> with PoE.
>
> There’s always CDMA, GSM, and even WWV for a less-accurate plan B time
> source.  Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated
> yet:
>
> http://www.beaglesoft.com/celsynhowworks.htm
>
> And their $400 WWV-based Stratum 1 time server:
>
> http://www.beaglesoft.com/radsynreceiver.htm
>
> So if you want non-Internet clock diversity, you can have clock diversity.
> You just have to pay for it.
>
>  -mel
>
> On May 10, 2016, at 9:18 PM, Eric Kuhnke  eric.kuh...@gmail.com>> wrote:
>
> For quite some time, in debian the default configuration for the ntpd.conf
> that ships with the package for the ntpd is to poll from four different,
> semi-randomly assigned DNS pool based sources. I believe the same is true
> for redhat/centos.
>
> In the event that one out of four sources is wildly wrong the ntpd will
> ignore it.
>
> If people have routers/networking equipment inside their network that only
> supports retrieving ntp from one IP address (or hostname) and have manually
> configured it to request time from a single external source, not their own
> internal ntpd that is <10ms away, bad things could definitely happen.
>
> It is worthwhile to have both polling from external sources via IP as well
> as GPS sync. Many locations in a network have no hope of getting a GPS
> signal or putting an antenna with a clear view to the sky, but may be on a
> network segment that is <4ms away from many other nodes where you can
> colocate a 1U box and GPS antenna.
>
> On Tue, May 10, 2016 at 9:05 PM, Joe Klein  jskl...@gmail.com>> wrote:
>
> Is this group aware of the incident with tock.usno.navy.mil &
> tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
> years for the period of one hour, then return?
>
> The reasons were not fully explained, but the impact was global. Routers,
> switches, power grids, phone systems, certificates, encryption, Kerberos,
> logging and any tightly coupled transaction systems were impacted.
>
> So I began doing 'security research' on the topic (don't confuse me with
> joe hacker), and discovered both interesting and terrifying issues, which I
> will not disclose on an open forum.
>
> Needless to say, my suggestions are:
> 1. Configure a trusted time source and good time stratum architecture for
> your organization.
> 2. When identifying your source of time, the majority of the technologies
> can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
> authentication.
> 3. For distribution of time information inside your organization, ensure
> your critical systems (Encryption, PKI, transactions, etc) are using your
> redundant sources and authentication.
> 4. Operating systems, programming languages, libraries, and applications
> are sensitive to time changes and can fail in unexpected ways. Test them
> before it's too late.
> 5. Disallow internal system to seek NTP from other sources beyond your edge
> routers.
> 6. All core time systems should be monitored by your security team or SOC.
>
> One question, is this a topic anyone would find interested at a future
> NANOG? Something like "Hacking and Defending time?".
>
>
> Joe Klein
> "Inveniam viam aut faciam"
>
> PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
>
> On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  m...@beckman.org>> wrote:
>
> I don't pretend to know all the ways a hacker can find out what nap
> servers a company uses, but I can envision a virus that could do that
> once
> behind a firewall. Every ntp response lists the current reference ntp
> server in the next higher stratum. There are many ways that process could
> harvest all ntp servers over time, and then pass the public IP back to a
> mother ship controller. It could be going on right now.
>
> My point is, when the fix is so cheap, why put up with this risk at all?
>
> -mel beckman
>
> On May 10, 2016, at 5:18 PM, Chris Adams  c...@cmadams.net>> wrote:
>
> Once upon a time, Mel Beckman mailto:m...@beckman.org>>
> said:
> Boss: So how did a hacker get in and crash our accounting server,
> break
> our VPNs, and kill our network perform

Re: NIST NTP servers

[Mel Beckman]

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

 -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
mailto:jskl...@gmail.com>> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
mailto:m...@beckman.org>> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
mailto:c...@cmadams.net>> wrote:

Once upon a time, Mel Beckman mailto:m...@beckman.org>> said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go
if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe R

Re: NIST NTP servers

[Mel Beckman]

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

 -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
mailto:jskl...@gmail.com>> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
mailto:m...@beckman.org>> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
mailto:c...@cmadams.net>> wrote:

Once upon a time, Mel Beckman mailto:m...@beckman.org>> said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go
if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe R

Re: NIST NTP servers

[Mel Beckman]

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

 -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
mailto:jskl...@gmail.com>> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
mailto:m...@beckman.org>> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
mailto:c...@cmadams.net>> wrote:

Once upon a time, Mel Beckman mailto:m...@beckman.org>> said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go
if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe R

Re: NIST NTP servers

[Eric Kuhnke]

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein  wrote:

> Is this group aware of the incident with tock.usno.navy.mil &
> tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
> years for the period of one hour, then return?
>
> The reasons were not fully explained, but the impact was global. Routers,
> switches, power grids, phone systems, certificates, encryption, Kerberos,
> logging and any tightly coupled transaction systems were impacted.
>
> So I began doing 'security research' on the topic (don't confuse me with
> joe hacker), and discovered both interesting and terrifying issues, which I
> will not disclose on an open forum.
>
> Needless to say, my suggestions are:
> 1. Configure a trusted time source and good time stratum architecture for
> your organization.
> 2. When identifying your source of time, the majority of the technologies
> can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
> authentication.
> 3. For distribution of time information inside your organization, ensure
> your critical systems (Encryption, PKI, transactions, etc) are using your
> redundant sources and authentication.
> 4. Operating systems, programming languages, libraries, and applications
> are sensitive to time changes and can fail in unexpected ways. Test them
> before it's too late.
> 5. Disallow internal system to seek NTP from other sources beyond your edge
> routers.
> 6. All core time systems should be monitored by your security team or SOC.
>
> One question, is this a topic anyone would find interested at a future
> NANOG? Something like "Hacking and Defending time?".
>
>
> Joe Klein
> "Inveniam viam aut faciam"
>
> PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
>
> On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  wrote:
>
> > I don't pretend to know all the ways a hacker can find out what nap
> > servers a company uses, but I can envision a virus that could do that
> once
> > behind a firewall. Every ntp response lists the current reference ntp
> > server in the next higher stratum. There are many ways that process could
> > harvest all ntp servers over time, and then pass the public IP back to a
> > mother ship controller. It could be going on right now.
> >
> > My point is, when the fix is so cheap, why put up with this risk at all?
> >
> >  -mel beckman
> >
> > > On May 10, 2016, at 5:18 PM, Chris Adams  wrote:
> > >
> > > Once upon a time, Mel Beckman  said:
> > >> Boss: So how did a hacker get in and crash our accounting server,
> break
> > our VPNs, and kill our network performance?
> > >>
> > >> IT guy: He changed our clocks.
> > >
> > > So, this has been repeated several times (with how bad things will go
> if
> > > your clocks get changed by years).  It isn't that easy.
> > >
> > > First, out of the box, if you use the public pool servers (default
> > > config), you'll typically get 4 random (more or less) servers from the
> > > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > > high chance of guessing the servers your system is using.
> > >
> > > Second, he'd have to guess at least three to "win".
> > >
> > > Third, at best, he'd only be able to change your clocks a little; the
> > > common software won't step the clock more than IIRC 15 minutes.  Yes,
> > > that can cause problems, but not the catastrophes of years in the
> future
> > > or Jan 1, 1970 mentioned in this thread.
> > >
> > > Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
> > > not so sure, and I haven't seen proof to the contrary.
> > > --
> > > Chris Adams 
> >
>

Re: NIST NTP servers

[Joe Klein]

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  wrote:

> I don't pretend to know all the ways a hacker can find out what nap
> servers a company uses, but I can envision a virus that could do that once
> behind a firewall. Every ntp response lists the current reference ntp
> server in the next higher stratum. There are many ways that process could
> harvest all ntp servers over time, and then pass the public IP back to a
> mother ship controller. It could be going on right now.
>
> My point is, when the fix is so cheap, why put up with this risk at all?
>
>  -mel beckman
>
> > On May 10, 2016, at 5:18 PM, Chris Adams  wrote:
> >
> > Once upon a time, Mel Beckman  said:
> >> Boss: So how did a hacker get in and crash our accounting server, break
> our VPNs, and kill our network performance?
> >>
> >> IT guy: He changed our clocks.
> >
> > So, this has been repeated several times (with how bad things will go if
> > your clocks get changed by years).  It isn't that easy.
> >
> > First, out of the box, if you use the public pool servers (default
> > config), you'll typically get 4 random (more or less) servers from the
> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > high chance of guessing the servers your system is using.
> >
> > Second, he'd have to guess at least three to "win".
> >
> > Third, at best, he'd only be able to change your clocks a little; the
> > common software won't step the clock more than IIRC 15 minutes.  Yes,
> > that can cause problems, but not the catastrophes of years in the future
> > or Jan 1, 1970 mentioned in this thread.
> >
> > Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
> > not so sure, and I haven't seen proof to the contrary.
> > --
> > Chris Adams 
>

Re: NIST NTP servers

[Roland Dobbins]

On 11 May 2016, at 8:59, Mel Beckman wrote:

My point is, when the fix is so cheap, why put up with this risk at 
all?


Time and Position Spoofing With Open Source Projects.

[.pdf link]



Just keep in mind, *nothing* is perfect.

---
Roland Dobbins 

Re: NIST NTP servers

[Mel Beckman]

I don't pretend to know all the ways a hacker can find out what nap servers a 
company uses, but I can envision a virus that could do that once behind a 
firewall. Every ntp response lists the current reference ntp server in the next 
higher stratum. There are many ways that process could harvest all ntp servers 
over time, and then pass the public IP back to a mother ship controller. It 
could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all? 

 -mel beckman

> On May 10, 2016, at 5:18 PM, Chris Adams  wrote:
> 
> Once upon a time, Mel Beckman  said:
>> Boss: So how did a hacker get in and crash our accounting server, break our 
>> VPNs, and kill our network performance?
>> 
>> IT guy: He changed our clocks.
> 
> So, this has been repeated several times (with how bad things will go if
> your clocks get changed by years).  It isn't that easy.
> 
> First, out of the box, if you use the public pool servers (default
> config), you'll typically get 4 random (more or less) servers from the
> pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> high chance of guessing the servers your system is using.
> 
> Second, he'd have to guess at least three to "win".
> 
> Third, at best, he'd only be able to change your clocks a little; the
> common software won't step the clock more than IIRC 15 minutes.  Yes,
> that can cause problems, but not the catastrophes of years in the future
> or Jan 1, 1970 mentioned in this thread.
> 
> Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
> not so sure, and I haven't seen proof to the contrary.
> -- 
> Chris Adams 

Re: NIST NTP servers

[Chris Adams]

Once upon a time, Mel Beckman  said:
> Boss: So how did a hacker get in and crash our accounting server, break our 
> VPNs, and kill our network performance?
> 
> IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe Random Hacker isn't going to have a
high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the
common software won't step the clock more than IIRC 15 minutes.  Yes,
that can cause problems, but not the catastrophes of years in the future
or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
not so sure, and I haven't seen proof to the contrary.
-- 
Chris Adams 

Re: NIST NTP servers

[Mel Beckman]

Boss: So how did a hacker get in and crash our accounting server, break our 
VPNs, and kill our network performance?

IT guy: He changed our clocks.

Boss: How did he do that?

IT guy: We have an opening in our firewall that permits time clock packets to 
come from anywhere in the world, under certain conditions.

Boss: Why didn’t you block that?

IT guy: Well, we filtered to only accept clock settings from a trusted source, 
but the hacker lied and pretended to be that protected source.

Boss: I thought encryption was supposed to prevent that.

IT guy: Time clock packets aren’t encrypted. There is no standard for that.

Boss: Not even a password?

IT guy: Yes, there is a sophisticated authentication mechanism, but it doesn’t 
work.

Boss: So how could we have prevented this?

IT guy: We could have purchased our own time server synchronized to the U.S. 
Department of Standards atomic clock via Global Positioning System satellites 
using a special antenna. Then we wouldn’t need time from the Internet.

Boss: That sounds expensive. How much are we talking?

IT guy: $300

Boss: You’re fired.


On May 10, 2016, at 1:51 PM, Jared Mauch 
mailto:ja...@puck.nether.net>> wrote:


On May 10, 2016, at 4:40 PM, Gary E. Miller 
mailto:g...@rellim.com>> wrote:

Yo Jared!


Yo, Gary!

On Tue, 10 May 2016 16:29:26 -0400
Jared Mauch mailto:ja...@puck.nether.net>> wrote:

If you’re using Redhat based systems consider using chrony
instead, even the new beta fedora 24 uses 4.2.6 derived code
vs 4.2.8

Or, new but under heavy development: NTPsec : https://www.ntpsec.org/

It is a fork of classic NTPD, but was not vulnerable to most of the
recent NTPD CVEs.


Yeah, there are some issues here in how the NTP community has implemented
solutions without discussing with each other through the community splits.

The NTPWG at IETF has been in a bit of stasis for years now because the
various aspects of how it works, and those who present sometimes don’t
output in the most organized fashion requiring a lot of effort on the
receiver.

There’s also a very narrow universe of people who actually care about the
implementations and details, with people like Majdi, Harlan and Miroslav
understanding the needs more than I’ve seen anyone from the ntpsec/cisco
funded side grasp the nuances of.

As a general statement, we are well served by having diverse and robust
implementations, but as we’ve seen in the (mostly) router space that NANOG
community cares about.. there are far more BGP implementations than NTP.

This isn’t good if the community wants to move to a model of certificate based
routing and the dependent infrastructure is weak.

I would suggest moving parts of this discussion to either the NTP Pool or the
NTPWG mailing lists.

- jared

Re: NIST NTP servers

[Jared Mauch]

> On May 10, 2016, at 4:40 PM, Gary E. Miller  wrote:
> 
> Yo Jared!
> 

Yo, Gary!

> On Tue, 10 May 2016 16:29:26 -0400
> Jared Mauch  wrote:
> 
>> If you’re using Redhat based systems consider using chrony 
>> instead, even the new beta fedora 24 uses 4.2.6 derived code
>> vs 4.2.8
> 
> Or, new but under heavy development: NTPsec : https://www.ntpsec.org/
> 
> It is a fork of classic NTPD, but was not vulnerable to most of the 
> recent NTPD CVEs.


Yeah, there are some issues here in how the NTP community has implemented
solutions without discussing with each other through the community splits.

The NTPWG at IETF has been in a bit of stasis for years now because the
various aspects of how it works, and those who present sometimes don’t
output in the most organized fashion requiring a lot of effort on the
receiver.

There’s also a very narrow universe of people who actually care about the
implementations and details, with people like Majdi, Harlan and Miroslav
understanding the needs more than I’ve seen anyone from the ntpsec/cisco
funded side grasp the nuances of.

As a general statement, we are well served by having diverse and robust
implementations, but as we’ve seen in the (mostly) router space that NANOG
community cares about.. there are far more BGP implementations than NTP.

This isn’t good if the community wants to move to a model of certificate based
routing and the dependent infrastructure is weak.

I would suggest moving parts of this discussion to either the NTP Pool or the
NTPWG mailing lists.

- jared

Re: NIST NTP servers

[Gary E. Miller]

Yo Jared!

On Tue, 10 May 2016 16:29:26 -0400
Jared Mauch  wrote:

> If you’re using Redhat based systems consider using chrony 
> instead, even the new beta fedora 24 uses 4.2.6 derived code
> vs 4.2.8

Or, new but under heavy development: NTPsec : https://www.ntpsec.org/

It is a fork of classic NTPD, but was not vulnerable to most of the 
recent NTPD CVEs.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgp5kLhcr45Li.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Jared Mauch]

> On May 10, 2016, at 4:21 PM, Harlan Stenn  wrote:
> 
>> Configure all of your devices to get NTP from the servers you run
>> using authentication.
> 
> Yes, and properly monitor your ntpd instances.

And upgrade them.

Some software distributors don’t ship modern software.  if you
are using a distribution packaged ntpd it’s likely old and
difficult to determine its lineage due to how it’s packaged.

If you’re using Redhat based systems consider using chrony 
instead, even the new beta fedora 24 uses 4.2.6 derived code
vs 4.2.8

- Jared

Re: NIST NTP servers

[Gary E. Miller]

Yo Chuck!

On Tue, 10 May 2016 16:18:41 -0400
"Chuck Church"  wrote:

> Ok, annoyance might have been a little light on the severity wording.

Yup.

> Still, modifying all your incoming NTP packets from all your sources
> to actually get your NTP servers to agree on a bad time is tricky.
> That is assuming you've got multiple links, multiple sources from
> multiple organizations (more than 4), they're all authenticated,
> etc.

NTP Authentication (autokey) has been broken, and no one used it anyway.  

If I have a copy of your ntp.conf I can spoof all your chimers.  Not
hard at all.  This is UDP after all.

> Even if a criminal was to do all that damage you listed, it
> still probably doesn't result in obtaining sensitive data or money
> that would be the main motivators for such extreme hacking.

Correct, it would just get me fired due to the extended downtime.

Or maybe my company just decided to pay the ransom to get un-DoS'ed.
I still get fired.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpGSd6Se1CbY.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Mel Beckman]

Accurate time to the millisecond is pretty much essential for any network 
troubleshooting. Say you want to diagnose a SIP problem. You collect 
transaction logs from both phones, the VoIP gateway, and the PBX. Now you try 
to merge them to derive the sequence of events. You NEED millisecond accuracy.

But more importantly, Gary is right about the risks. I’ve had several customers 
receive major NTP DoS attacks using forged source addresses. In today’s 
Internet, there is very little source address verification (despite several 
mechanisms being proposed). Everyone relies on the originating network 
preventing spoofing, but thousands of ISPs — particularly overseas — do not do 
spoof checks. 

And the issues of NTP pollution are even more dangerous. As Gary notes, 
changing dates is a risk. A big enough change (say 30 days) would be 
catastrophic to most accounting systems. A big leap — a year or more — could 
expire software license and disable all kinds of encryption. We haven’t even 
discussed multi-stage attacks, where NTP is used to disrupt systems at multiple 
points, and then the attacker storms in and takes over unnoticed during the 
confusion.

All because of misplaced trust in a tiny UDP packet that can worm its way into 
your network from anywhere on the Internet.

I say you’re crazy if you don’t run a GPS-based NTP server, especially given 
that they cost as little as $300 for very solid gear. Heck, get two or three!

 -mel

> On May 10, 2016, at 12:58 PM, Gary E. Miller  wrote:
> 
> Yo Chuck!
> 
> On Tue, 10 May 2016 10:29:35 -0400
> "Chuck Church"  wrote:
> 
>> Changing time on
>> devices is more an annoyance than anything, and doesn't necessarily
>> get you into a device.
> 
> So, you are not worried about getting DoS'ed?
> 
> How about you set the time on your server ahead by 5 years.  Got any
> idea what would happen?
> 
> Most of your passwords would expire.
> 
> All your SSL certs would expire.
> 
> All your TOTPs, like Google Authenticator would fail.
> 
> All your IPSEC tunnels would drop, and refuse to restart.
> 
> Many of your cron jobs would got nuts, possibly deleting all your logs.
> 
> Much of your DNSSEC would expire.
> 
> Many of your backups would be deleted since they 'expired'.
> 
> Until recently, setting your iPhone to 1 Jan 1970 would brick it.
> 
> I'm sure there are many more examples, but likely you can no longer log
> in, via SSH or HTTPS, and your iPhone is dead.  I think any of those
> would qualify as more than an annoyance.
> 
> RGDS
> GARY
> ---
> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>   g...@rellim.com  Tel:+1 541 382 8588

Re: NIST NTP servers

[Harlan Stenn]

Leo Bicknell writes:
> ...
> 
> The correct answer here is to run multiple NTP servers in your
> network.  And by servers I mean real servers, with good quality
> oscellators on the motherboard.  Then configure them to talk to
> _many_ sources.  You need 4 sources of time minimum to redundantly
> detect false tickers.  If you're serious about it then find ~10
> Stratum 1 sources (ideally authenticated and from trusted entities),

Byzantine General's problem.

With 3 sources you can detect *1* false ticker.

But if one of those becomes unreachable you only have 2 time sources.
Dilemma.

With 4 sources you run the risk of 2 going one way, and 2 going another
way.  This happened to several folks recently, when some sites put NTP
servers on the 'net that offered leap-smeared time.  That's really a
different problem where one of the effects is that it causes "time
islands".

> one of which could be GPS as several have suggested.  You'll then
> have high quality false ticker rejection.

For extra points, use GPS receivers from different manufacturers, using
whatever "variety" you can get for all of the components involved.

Are you mounting each GPS receiver inside a coffee can to prevent
drive-by jamming?

Are the cables properly grounded?  Using gas discharge tubes?
Periodically tested/inspected?

How much fun do you want to have thinking about all of these cases?

> Configure all of your devices to get NTP from the servers you run
> using authentication.

Yes, and properly monitor your ntpd instances.

-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

RE: NIST NTP servers

[Chuck Church]

-Original Message-
From: Gary E. Miller [mailto:g...@rellim.com] 
Sent: Tuesday, May 10, 2016 3:58 PM
To: Chuck Church 
Cc: 'Majdi S. Abbas' ; nanog@nanog.org
Subject: Re: NIST NTP servers

Yo Chuck!

On Tue, 10 May 2016 10:29:35 -0400
"Chuck Church"  wrote:

> Changing time on
> devices is more an annoyance than anything, and doesn't necessarily 
> get you into a device.

So, you are not worried about getting DoS'ed?

How about you set the time on your server ahead by 5 years.  Got any idea
what would happen?

Most of your passwords would expire.

All your SSL certs would expire.

All your TOTPs, like Google Authenticator would fail.

All your IPSEC tunnels would drop, and refuse to restart.

Many of your cron jobs would got nuts, possibly deleting all your logs.

Much of your DNSSEC would expire.

Many of your backups would be deleted since they 'expired'.

Until recently, setting your iPhone to 1 Jan 1970 would brick it.

I'm sure there are many more examples, but likely you can no longer log in,
via SSH or HTTPS, and your iPhone is dead.  I think any of those would
qualify as more than an annoyance.

RGDS
GARY



Ok, annoyance might have been a little light on the severity wording.
Still, modifying all your incoming NTP packets from all your sources to
actually get your NTP servers to agree on a bad time is tricky.  That is
assuming you've got multiple links, multiple sources from multiple
organizations (more than 4), they're all authenticated, etc.  Even if a
criminal was to do all that damage you listed, it still probably doesn't
result in obtaining sensitive data or money that would be the main
motivators for such extreme hacking.   If I had an iPhone, perhaps I'd worry
about that as well.  But fortunately, not an issue ;)

Chuck

Re: NIST NTP servers

[Jared Mauch]

> On May 10, 2016, at 3:58 PM, Gary E. Miller  wrote:
> 
> I'm sure there are many more examples, but likely you can no longer log
> in, via SSH or HTTPS, and your iPhone is dead.  I think any of those
> would qualify as more than an annoyance.

An unnamed vendor has code where if the clock on their router is not
set SSH won’t work as the crypto package signature says the
package isn’t valid.

Many of the not-before and not-after certificate systems have some fairly
serious issues.

https://www.cs.bu.edu/~goldbe/pub-index.html#NTP

is one place to start when it comes to on-path and off-path
NTP attacks that can skew clocks.

- jared

Re: NIST NTP servers

[Gary E. Miller]

Yo Chuck!

On Tue, 10 May 2016 10:29:35 -0400
"Chuck Church"  wrote:

> Changing time on
> devices is more an annoyance than anything, and doesn't necessarily
> get you into a device.

So, you are not worried about getting DoS'ed?

How about you set the time on your server ahead by 5 years.  Got any
idea what would happen?

Most of your passwords would expire.

All your SSL certs would expire.

All your TOTPs, like Google Authenticator would fail.

All your IPSEC tunnels would drop, and refuse to restart.

Many of your cron jobs would got nuts, possibly deleting all your logs.

Much of your DNSSEC would expire.

Many of your backups would be deleted since they 'expired'.

Until recently, setting your iPhone to 1 Jan 1970 would brick it.

I'm sure there are many more examples, but likely you can no longer log
in, via SSH or HTTPS, and your iPhone is dead.  I think any of those
would qualify as more than an annoyance.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpNcDMQ1pC0W.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Laszlo Hanyecz]

On 2016-05-10 15:36, Mike wrote:

On 5/10/2016 11:22 AM, Leo Bicknell wrote:

In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:

In search of stable, disparate stratum 1 NTP sources.

http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm


We tried using “time.nist.gov” which returns varying round-robin addresses
(as the link says), but Cisco IOS resolved the FQDN and embedded the
numeric address in the “ntp server” config statement.

Depending on your hardware platform your Cisco Router is likely not
a great NTP server.  IOS is not designed for hyper-accuracy.


After letting the new server config go through a few days of update cycles,
the drift, offset and reachability stats are not anywhere as good as what
the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.

The correct answer here is to run multiple NTP servers in your
network.  ...
[snip]


I think the correct answer here starts with a question --- what level of
time accuracy is required for the local NTP server(s)? Which then begs
the question, what level of accuracy is needed for the clients?

A shop with a client need for nanosecond accuracy begs for an entirely
different solution set than a shop where a millisecond of accuracy is
needed on the clients, and still a different solution set that a shop
where "a few milliseconds either way" is quite OK.




You can get pretty nutty with this, and it's fun to do, but regular NTP 
over the internet is good enough for millisecond accuracy.  A default 
install with pool servers is pretty good.  A custom config with a local 
NTP server (with less, possibly more symmetric network latency) is a 
little better, but for common sysadmin needs like cron jobs and log 
correlation, you likely won't notice a difference.  I would worry more 
about having that config distributed correctly and monitoring all your 
servers to make sure their NTP is healthy, rather than worrying about 
the source of your NTP sync.  The pool servers are fine, and NTP is good 
at deciding when they're acting up.  The computer running NTP doesn't 
have to be anything special, but beware of VMs - depending on the 
virtualization type and the guest OS, you may not even be able to get 
NTP to engage because of the clock instability.  Chrony might work 
better for VMs.  For a linux NTP server, I prefer modern Intel CPUs with 
invariant tsc - linux will use it as a clocksource (cat 
/sys/devices/system/clocksource/clocksource0/current_clocksource
) .  A Raspberry Pi or something in between also works equally well if 
you're going to be syncing this over a jittery shared network anyway.  I 
would suggest having more than one server, in different locations if you 
can, and if you're able to supplement with pool servers, add those too.  
The most likely failure mode you're going to have is that it doesn't 
work at all because it's not running, it can't correct the local clock 
because of excess instability, or you lost all network connections.  
Worrying about whether you have 4 or 8 servers is kind of moot in any of 
those (much more likely) faults.


-Laszlo

Re: NIST NTP servers

[Mike]

On 5/10/2016 11:22 AM, Leo Bicknell wrote:
> In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:
>> In search of stable, disparate stratum 1 NTP sources.
> 
> http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm
> 
>> We tried using “time.nist.gov” which returns varying round-robin addresses
>> (as the link says), but Cisco IOS resolved the FQDN and embedded the
>> numeric address in the “ntp server” config statement.
> 
> Depending on your hardware platform your Cisco Router is likely not
> a great NTP server.  IOS is not designed for hyper-accuracy.
> 
>> After letting the new server config go through a few days of update cycles,
>> the drift, offset and reachability stats are not anywhere as good as what
>> the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.
> 
> The correct answer here is to run multiple NTP servers in your
> network.  ...
>[snip]


I think the correct answer here starts with a question --- what level of
time accuracy is required for the local NTP server(s)? Which then begs
the question, what level of accuracy is needed for the clients?

A shop with a client need for nanosecond accuracy begs for an entirely
different solution set than a shop where a millisecond of accuracy is
needed on the clients, and still a different solution set that a shop
where "a few milliseconds either way" is quite OK.

Re: NIST NTP servers

[Valdis . Kletnieks]

On Tue, 10 May 2016 08:07:15 -0700, Brandon Vincent said:
> On May 10, 2016 7:59 AM, "Stephane Bortzmeyer"  wrote:
> > Yes, but they may switch it off for civilian use (by going encrypted,
> > for instance) at any time, if it is better for *their* operations.
>
> I think you are referring to selective availability which degraded the
> positional accuracy for civilians.

I seem to recall that the positional accuracy is degraded to "on the order
of 200 meters" when selective availability is enabled (and yes, they *can*
do it by geographic area by careful turning on/off on each orbit of each
satellite - devices are told the signal is degraded and can downvote that
signal.  So unless you're in the war zone, you'll just notice your device
reporting a lock on 2-3 fewer signals).

The upshot is that Grace Hopper told us about electricity moving a foot per
nanosecond - which means that the time-domain jitter introduced is all of
600 nanoseconds or so.  In other words, anybody using it to feed an NTP
server will hardly notice on the millisecond level.  If you're trying to
use NTP to get microsecond stability, you'll notice - but you have enough
*other* things to do correctly to do this that it shouldn't be an actual 
issue...


pgpCZR1NfsvBU.pgp
Description: PGP signature

Re: NIST NTP servers

[Josh Reynolds]

That would be a very poor idea, since a lot of the circuits the DoD
still uses to communicate with are ATM lines :)

On Tue, May 10, 2016 at 9:59 AM, Stephane Bortzmeyer  wrote:
> On Tue, May 10, 2016 at 10:52:28AM -0400,
>  valdis.kletni...@vt.edu  wrote
>  a message of 37 lines which said:
>
>> Note that they *do* have motivation to keep it working, simply
>> because so much of their *own* gear (from gear for individual
>> soldiers all the way to strategic bombers and aircraft carriers)
>> wants a working GPS signal...
>
> Yes, but they may switch it off for civilian use (by going encrypted,
> for instance) at any time, if it is better for *their* operations.
>
>

Re: NIST NTP servers

[Leo Bicknell]

In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:
> In search of stable, disparate stratum 1 NTP sources.

http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm

> We tried using “time.nist.gov” which returns varying round-robin addresses
> (as the link says), but Cisco IOS resolved the FQDN and embedded the
> numeric address in the “ntp server” config statement.

Depending on your hardware platform your Cisco Router is likely not
a great NTP server.  IOS is not designed for hyper-accuracy.

> After letting the new server config go through a few days of update cycles,
> the drift, offset and reachability stats are not anywhere as good as what
> the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.

The correct answer here is to run multiple NTP servers in your
network.  And by servers I mean real servers, with good quality
oscellators on the motherboard.  Then configure them to talk to
_many_ sources.  You need 4 sources of time minimum to redundantly
detect false tickers.  If you're serious about it then find ~10
Stratum 1 sources (ideally authenticated and from trusted entities),
one of which could be GPS as several have suggested.  You'll then
have high quality false ticker rejection.

Configure all of your devices to get NTP from the servers you run
using authentication.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpRuzcNumYGj.pgp
Description: PGP signature

Re: NIST NTP servers

[Stephane Bortzmeyer]

On Tue, May 10, 2016 at 10:52:28AM -0400,
 valdis.kletni...@vt.edu  wrote 
 a message of 37 lines which said:

> Note that they *do* have motivation to keep it working, simply
> because so much of their *own* gear (from gear for individual
> soldiers all the way to strategic bombers and aircraft carriers)
> wants a working GPS signal...

Yes, but they may switch it off for civilian use (by going encrypted,
for instance) at any time, if it is better for *their* operations.

RE: NIST NTP servers

[Chuck Church]

True, but I did mention verifying packet sources.  That needs to happen 
everywhere, and it's not hard to do.  Just getting everyone to do it is tough.

Chuck

-Original Message-
From: Allan Liska [mailto:al...@allan.org] 
Sent: Tuesday, May 10, 2016 10:40 AM
To: Chuck Church ; 'Majdi S. Abbas' ; 
nanog@nanog.org
Subject: RE: NIST NTP servers



On 5/10/2016 at 10:30 AM, "Chuck Church"  wrote:

>
>It doesn't really.  Granted there are a lot of CVEs coming out for NTP 
>the last year or so.  But I just don't think there are that many 
>attacks on it.
>It's just not worth the effort.  Changing time on devices is more an 
>annoyance than anything, and doesn't necessarily get you into a device.
>Sure you can hide your tracks a little by altering time in logs and 
>altering it back, but that's more of an in-depth nation-state kind of 
>attack, not going to be a script kiddie kind of thing.  Just follow the 
>best practices for verifying packet sources and NTP security itself, 
>and you should be ok.
>
>Chuck

I would argue that the fact the NTP can, and has been, be used in DDoS 
amplification attacks is a serious concern for using protocol going forward.



allan

Re: NIST NTP servers

[David Hubbard]

Ed, and anyone else reading this thread, I’m curious if you’ve looked at their 
authenticated NTP offering which uses different servers:

http://www.nist.gov/pml/div688/grp40/auth-ntp.cfm


We’re considering that but haven’t tried yet.

David




On 5/9/16, 11:01 PM, "NANOG on behalf of b f"  wrote:

>Hello List,
>
>
>In search of stable, disparate stratum 1 NTP sources.
>
>Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s
>NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi
>
>We tried using “time.nist.gov” which returns varying round-robin addresses
>(as the link says), but Cisco IOS resolved the FQDN and embedded the
>numeric address in the “ntp server” config statement.
>
>
>
>After letting the new server config go through a few days of update cycles,
>the drift, offset and reachability stats are not anywhere as good as what
>the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.
>
>
>I would greatly appreciate and feedback / advice, etc.
>
>
>Thanks!!!
>
>
>Ed

Re: NIST NTP servers

[Valdis . Kletnieks]

On Tue, 10 May 2016 16:39:54 +0200, Stephane Bortzmeyer said:

> You mean the GPS network is not managed by an external entity? With
> budget issues?
>
> http://www.schriever.af.mil/GPS

Note that they *do* have motivation to keep it working, simply because
so much of their *own* gear (from gear for individual soldiers all the
way to strategic bombers and aircraft carriers) wants a working GPS signal...


pgpNB7bU9HRAq.pgp
Description: PGP signature

Re: NIST NTP servers

[Stephane Bortzmeyer]

On Tue, May 10, 2016 at 06:48:52AM -0400,
 Steven Miano  wrote 
 a message of 41 lines which said:

> Going with an internal GPS/GLONASS/RADIO based S1 allows you to
> restrict incoming traffic and not rely on volunteers or external
> entities (which may undergo maintenance or budget issues).

You mean the GPS network is not managed by an external entity? With
budget issues?

http://www.schriever.af.mil/GPS

RE: NIST NTP servers

[Chuck Church]

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Majdi S. Abbas

>   So how does this stop from distributing time to their customers via
NTP?
>   GPS doesn't save the protocol, in particular where the S1 clocks
involved are embedded devices with rather coarse clocks and timestamping.
>   --msa

It doesn't really.  Granted there are a lot of CVEs coming out for NTP the
last year or so.  But I just don't think there are that many attacks on it.
It's just not worth the effort.  Changing time on devices is more an
annoyance than anything, and doesn't necessarily get you into a device.
Sure you can hide your tracks a little by altering time in logs and altering
it back, but that's more of an in-depth nation-state kind of attack, not
going to be a script kiddie kind of thing.  Just follow the best practices
for verifying packet sources and NTP security itself, and you should be ok.

Chuck

Re: NIST NTP servers

[Steven Miano]

NTP has vulnerabilities, so using an external source opens your networks
and infrastructure to disruptions.

Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict
incoming traffic and not rely on volunteers or external entities (which may
undergo maintenance or budget issues).

My preference is more so something akin to the GLN180PEX (I am not
affiliated or paid to endorse this product). It allows you to use commodity
hardware (like a decommissioned 1U or several preferably) and creation of
ones own reliable internal time source(s). Introducing black boxes into a
production (revenue generation or expected services by paying customers)
environment is undesirable.

>From there setting up NTPd, Chronyd, and PTPd is up to you.

Relying on satellites may seem like just another external reliance, but the
next life is proposing a design life of 12 years.

On Mon, May 9, 2016 at 11:12 PM, Majdi S. Abbas  wrote:

> On Tue, May 10, 2016 at 03:08:16AM +, Mel Beckman wrote:
> > NTP has vulnerabilities that make it generally unsuitable for
> > provider networks. I strongly recommend getting a GPS-based
> > time server. These are as cheap as $300. Here is one I use quite a bit:
>
> So how does this stop from distributing time to their
> customers via NTP?
>
> GPS doesn't save the protocol, in particular where the S1
> clocks involved are embedded devices with rather coarse clocks and
> timestamping.
>
> --msa
>



-- 
Miano, Steven M.
http://stevenmiano.com

Re: NIST NTP servers

[Majdi S. Abbas]

On Tue, May 10, 2016 at 03:08:16AM +, Mel Beckman wrote:
> NTP has vulnerabilities that make it generally unsuitable for 
> provider networks. I strongly recommend getting a GPS-based 
> time server. These are as cheap as $300. Here is one I use quite a bit:

So how does this stop from distributing time to their
customers via NTP?

GPS doesn't save the protocol, in particular where the S1 
clocks involved are embedded devices with rather coarse clocks and
timestamping.

--msa

Re: NIST NTP servers

[Spencer Ryan]

I would second the idea of using your own GPS appliance if possible.
On May 9, 2016 11:08 PM, "Mel Beckman"  wrote:

> NTP has vulnerabilities that make it generally unsuitable for provider
> networks. I strongly recommend getting a GPS-based time server. These are
> as cheap as $300. Here is one I use quite a bit:
>
> http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B002RC3Q4Q
>
> You’ll have a stratum 1 clock on site. Hard to beat.
>
>  -mel
>
> On May 9, 2016, at 8:01 PM, b f  freetexwat...@gmail.com>> wrote:
>
> Hello List,
>
>
> In search of stable, disparate stratum 1 NTP sources.
>
> Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s
> NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi
>
> We tried using “time.nist.gov” which returns
> varying round-robin addresses
> (as the link says), but Cisco IOS resolved the FQDN and embedded the
> numeric address in the “ntp server” config statement.
>
>
>
> After letting the new server config go through a few days of update cycles,
> the drift, offset and reachability stats are not anywhere as good as what
> the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.
>
>
> I would greatly appreciate and feedback / advice, etc.
>
>
> Thanks!!!
>
>
> Ed
>
>

Re: NIST NTP servers

[Mel Beckman]

NTP has vulnerabilities that make it generally unsuitable for provider 
networks. I strongly recommend getting a GPS-based time server. These are as 
cheap as $300. Here is one I use quite a bit:

http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B002RC3Q4Q

You’ll have a stratum 1 clock on site. Hard to beat.

 -mel

On May 9, 2016, at 8:01 PM, b f 
mailto:freetexwat...@gmail.com>> wrote:

Hello List,


In search of stable, disparate stratum 1 NTP sources.

Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s
NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi

We tried using “time.nist.gov” which returns varying 
round-robin addresses
(as the link says), but Cisco IOS resolved the FQDN and embedded the
numeric address in the “ntp server” config statement.



After letting the new server config go through a few days of update cycles,
the drift, offset and reachability stats are not anywhere as good as what
the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.


I would greatly appreciate and feedback / advice, etc.


Thanks!!!


Ed

NIST NTP servers

[b f]

Hello List,


In search of stable, disparate stratum 1 NTP sources.

Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s
NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi

We tried using “time.nist.gov” which returns varying round-robin addresses
(as the link says), but Cisco IOS resolved the FQDN and embedded the
numeric address in the “ntp server” config statement.



After letting the new server config go through a few days of update cycles,
the drift, offset and reachability stats are not anywhere as good as what
the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.


I would greatly appreciate and feedback / advice, etc.


Thanks!!!


Ed