Re: NIST NTP servers

[Harlan Stenn]

Harlan Stenn writes:
> Sharon Goldberg writes:
> > Well, if you really want to learn about the NTP servers a target is using
> > you can always just sent them a regular NTP timing query (mode 3) and just
> > read off the IP address in the reference ID field of the response (mode 4).
> 
> Unless the server is an IPv6 server.  This trick only works for IPv4.
> 
> And we have a fix for all of this that will be out soon.

Also, the attacker will need to know the correct origin timestamp for
the brief window where that attack will work, and even if this happens
either the client or the server will see syslog entries alerting to the
abuse (if folks are running new enough versions of ntpd).

H

Re: NIST NTP servers

[Harlan Stenn]

Sharon Goldberg writes:
> Well, if you really want to learn about the NTP servers a target is using
> you can always just sent them a regular NTP timing query (mode 3) and just
> read off the IP address in the reference ID field of the response (mode 4).

Unless the server is an IPv6 server.  This trick only works for IPv4.

And we have a fix for all of this that will be out soon.
-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: NIST NTP servers

[Valdis . Kletnieks]

On Wed, 11 May 2016 17:23:31 -0700, Eric Kuhnke said:

> average of $150/mo x 500 = $75,000

Id worry more about the fact that somebody is willing to spend $75K/mo to
attack me than the fact that it might be possible to wiggle my time base a bit.
At that point, you *really* have to worry about other, more productive attacks,
such as social engineering (including spear phishing and bribery), or obtaining
a 0-day against something in your network.

Remember that the FBI has basically admitted that the most effective means of
uncloaking a Tor user has been browser 0-days, even though decloaking by pwning
a majority of the servers is possible

(Seriously - if *you* had $75K/mo to attack somebody, what would *you* spend
it on?)



pgpaZojwN0Y_I.pgp
Description: PGP signature

Re: NIST NTP servers

[Eygene Ryabinkin]

Wed, May 11, 2016 at 05:20:28PM -0700, Scott Weeks wrote:
> --- m...@beckman.org wrote:
>> From: Mel Beckman 
>> 
>> Accurate time to the millisecond is pretty much 
>> essential for any network troubleshooting. Say 
>> you want to diagnose a SIP problem. You collect 
>> transaction logs from both phones, the VoIP 
>> gateway, and the PBX. Now you try to merge them 
>> to derive the sequence of events. You NEED 
>> millisecond accuracy.
>> 
> 
> If all logs are sent to a unix server that does 
> syslogd the log entries would go into the file
> in order no matter what timestamp is on them.

Modulo latencies and jitter from different machines to the log server.
Millisecond precision can be harmed by this, easily.  Especially by
jitter and order of millisecond here isn't something non-existent
in a long-distant networks.
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.

Re: NIST NTP servers

[Josh Reynolds]

maybe try with an odroid?
On May 11, 2016 8:45 PM, "Jon Meek"  wrote:

> A note on using a Raspberry Pi as a NTP server. In my limited home lab
> testing the RPi server had enough instability that Internet time sources
> were always preferred by my workstation after ntpd had been running for a
> while. Presumably this was due to the RPi's clock frequency drifting. At
> some point I will look at it again.
>
> If you do want to build your own Stratum 1 server you might want to glance
> at:
>
> https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf
>
> and the references there.
>
> I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but
> the test device was clearly not up to the job. At some point I hope to
> revisit this and do some more testing like I did for that poster. I'll add
> in a CDMA server and a dedicated WWVB receiver.
>
> Jon
>

Re: NIST NTP servers

[Jon Meek]

A note on using a Raspberry Pi as a NTP server. In my limited home lab
testing the RPi server had enough instability that Internet time sources
were always preferred by my workstation after ntpd had been running for a
while. Presumably this was due to the RPi's clock frequency drifting. At
some point I will look at it again.

If you do want to build your own Stratum 1 server you might want to glance
at:

https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf

and the references there.

I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but
the test device was clearly not up to the job. At some point I hope to
revisit this and do some more testing like I did for that poster. I'll add
in a CDMA server and a dedicated WWVB receiver.

Jon

Re: NIST NTP servers

[Sharon Goldberg]

With the caveat that if some of the servers are inside your own private
network then learning who the servers are might be less useful.

But this could be an issue for targets who use servers that are exclusively
on the public internet.

On Wed, May 11, 2016 at 3:15 PM, Sharon Goldberg  wrote:

> Well, if you really want to learn about the NTP servers a target is using
> you can always just sent them a regular NTP timing query (mode 3) and just
> read off the IP address in the reference ID field of the response (mode 4).
>
>
> Reference ID reveals the target that the client is sync'd to.
>
> If you do this over and over as the client changes the servers it sync's
> to, you learn all the servers.
>
> Or if you are really keen you can use our "kiss-of-death" attack to learn
> all the servers a client is willing to take time from. See sections V.B-V.C
> of our paper.
>
> https://eprint.iacr.org/2015/1020.pdf
>
> Sharon
>
>
>
> On Wed, May 11, 2016 at 3:07 PM, Florian Weimer  wrote:
>
>> * Chris Adams:
>>
>> > First, out of the box, if you use the public pool servers (default
>> > config), you'll typically get 4 random (more or less) servers from the
>> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
>> > high chance of guessing the servers your system is using.
>>
>> A determined attacker will just run servers in the official pool.
>>
>>
>
>
> --
> Sharon Goldberg
> Computer Science, Boston University
> http://www.cs.bu.edu/~goldbe
>



-- 
Sharon Goldberg
Computer Science, Boston University
http://www.cs.bu.edu/~goldbe

Re: NIST NTP servers

[Sharon Goldberg]

Well, if you really want to learn about the NTP servers a target is using
you can always just sent them a regular NTP timing query (mode 3) and just
read off the IP address in the reference ID field of the response (mode 4).


Reference ID reveals the target that the client is sync'd to.

If you do this over and over as the client changes the servers it sync's
to, you learn all the servers.

Or if you are really keen you can use our "kiss-of-death" attack to learn
all the servers a client is willing to take time from. See sections V.B-V.C
of our paper.

https://eprint.iacr.org/2015/1020.pdf

Sharon



On Wed, May 11, 2016 at 3:07 PM, Florian Weimer  wrote:

> * Chris Adams:
>
> > First, out of the box, if you use the public pool servers (default
> > config), you'll typically get 4 random (more or less) servers from the
> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > high chance of guessing the servers your system is using.
>
> A determined attacker will just run servers in the official pool.
>
>


-- 
Sharon Goldberg
Computer Science, Boston University
http://www.cs.bu.edu/~goldbe

Re: NIST NTP servers

[Lyndon Nerenberg]

> On May 11, 2016, at 5:42 PM, Scott Weeks  wrote:
> 
> Wouldn't the buffers empty in a FIFO manner?

They will empty in whatever order the implementation decides to write them.

But what's more important is the order in which the incoming packets are 
presented to the syslogd process. If you're listening on TCP connections, the 
receive order is very much determined by the strategy the syslogd 
implementation uses to read from FDs with available data.  I.e. elevator scan, 
lowest/highest first, circular queue, ...  In a threaded implementation, your 
reader workers, buffer writers, etc., are all at the mercy of the threading 
implementation; it's difficult to control thread dispatch ordering at that 
level of granularity.

--lyndon

Re: NIST NTP servers

[Gary E. Miller]

Yo Scott!

On Wed, 11 May 2016 17:42:34 -0700
"Scott Weeks"  wrote:

> > If all logs are sent to a unix server that does 
> > syslogd the log entries would go into the file
> > in order no matter what timestamp is on them.  
> 
> syslogd can have quite large buffers.
> ---
> 
> 
> Wouldn't the buffers empty in a FIFO manner?

Nope, each local syslog waits until its local buffer is full, or timed
out, then sends to the main syslog server.

The default flush_timeout() for syslog-ng is 10 seconds.  That is a long
time when debugging SIP.

http://www.syslog.org/syslog-ng/v2/

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpBtkPYjkTeV.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Scott Weeks]

--- g...@rellim.com wrote:
From: "Gary E. Miller" 

Yo Scott!

On Wed, 11 May 2016 17:20:28 -0700
"Scott Weeks"  wrote:

> If all logs are sent to a unix server that does 
> syslogd the log entries would go into the file
> in order no matter what timestamp is on them.

syslogd can have quite large buffers.
---


Wouldn't the buffers empty in a FIFO manner?

scott

Re: NIST NTP servers

[Gary E. Miller]

Yo Scott!

On Wed, 11 May 2016 17:20:28 -0700
"Scott Weeks"  wrote:

> If all logs are sent to a unix server that does 
> syslogd the log entries would go into the file
> in order no matter what timestamp is on them.

syslogd can have quite large buffers.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpByl5us85nK.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Eric Kuhnke]

Compared to the scale of the budget of small research projects run by
national intelligence agency sized organizations, you wouldn't have to be
very well funded to run a sizeable proportion of all tor exit nodes with
some degree of plausible deniability...

500 credit cards

500 unique bililng names/addresses and sets of contact info

spread 500 1U servers around the world in as many geographically unique
locations as you can find, with every dedicated hosting/colo company...

average of $150/mo x 500 = $75,000



On Wed, May 11, 2016 at 5:08 PM,  wrote:

> On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said:
> > * Chris Adams:
> >
> > > First, out of the box, if you use the public pool servers (default
> > > config), you'll typically get 4 random (more or less) servers from the
> > > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > > high chance of guessing the servers your system is using.
> >
> > A determined attacker will just run servers in the official pool.
>
> Such attacks have allegedly been attempted against Tor by certain
> very well funded adversaries.
>
> Thus my statement that if you're seeing that scale attack on your time
> sources, the fact that your time source is being attacked is the *least*
> of your problems...
>

Re: NIST NTP servers

[Scott Weeks]

--- m...@beckman.org wrote:
From: Mel Beckman 

Accurate time to the millisecond is pretty much 
essential for any network troubleshooting. Say 
you want to diagnose a SIP problem. You collect 
transaction logs from both phones, the VoIP 
gateway, and the PBX. Now you try to merge them 
to derive the sequence of events. You NEED 
millisecond accuracy.


If all logs are sent to a unix server that does 
syslogd the log entries would go into the file
in order no matter what timestamp is on them.

scott

Re: NIST NTP servers

[Valdis . Kletnieks]

On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said:
> * Chris Adams:
>
> > First, out of the box, if you use the public pool servers (default
> > config), you'll typically get 4 random (more or less) servers from the
> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > high chance of guessing the servers your system is using.
>
> A determined attacker will just run servers in the official pool.

Such attacks have allegedly been attempted against Tor by certain
very well funded adversaries.

Thus my statement that if you're seeing that scale attack on your time
sources, the fact that your time source is being attacked is the *least*
of your problems...


pgplgKIyYlw3x.pgp
Description: PGP signature

Re: NIST NTP servers

[Eric Kuhnke]

Cellular carriers also use GPS timing for many reasons that are not readily
apparent at the layer 3 router/IP/BGP network level. One big need is RF
related, back-to-back sector antenna frequency re-use with GPS synced
timing on the remote radio heads, such as an ABAB configuration on a tower
or rooftop site.

The same with some much less costly near consumer grade WISP radio
platforms and PTP radio systems nowadays.

In such a configuration the GPS timing signal from the local GPS receiver
(small cone shaped or puck antennas at the site) is actually the primary,
and whatever NTP-based GPS signal the network node might have access to is
secondary.


On Wed, May 11, 2016 at 12:10 PM, Mel Beckman  wrote:

> No, many cell carriers run their own completely independent timing
> networks. I support some head-ends where they have rubidium clocks and a
> T1-delivered time source. They do reference GPS, and many cell sites have
> GPS as a backup clock (you can see their conical antennas on the very top
> of the tower). But most cellular providers are very protective of their
> time sources. They’re also typically clocking SONET networks too, which
> requires BITS.
>
>  -mel
>
>
> JAshworth said:
> > CDMA and GSM are false diversity: both network types nodes *get their
> time*
> > from GPS, so far as I know.
>
>
> > On May 11, 2016, at 10:54 AM, valdis.kletni...@vt.edu wrote:
> >
> > On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said:
> >
> >> CDMA and GSM are false diversity: both network types nodes *get their
> time*
> >> from GPS, so far as I know.
> >
> > I'll make the fairly reasonable assumption that most readers of this
> list have
> > networks that span multiple buildings.
> >
> > If somebody is managing to figure out that you have a GPS in Building
> 37, and a
> > GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks
> at
> > other locations and getting close enough to all of them at the same time
> to
> > conduct a successful spoofing attack, all just to move your time source a
> > few seconds off
> >
> > ...  then the fact that GPS is spoofable is probably *NOT* your biggest
> > security problem.
> >
>
>

Re: TeamNANOG youtube video seeding

[Mikael Abrahamsson]

On Tue, 10 May 2016, james machado wrote:


First I am thrilled to see older Nanog meetings making it to youtube.

Having said that can the people putting up the files put the Nanog
meeting number in the title of the videos to make it easier to search
and determine relevance?


+1 from me. I also sent a message to the TeamNANOG youtube account with 
the same request.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: NIST NTP servers

[Mel Beckman]

No, many cell carriers run their own completely independent timing networks. I 
support some head-ends where they have rubidium clocks and a T1-delivered time 
source. They do reference GPS, and many cell sites have GPS as a backup clock 
(you can see their conical antennas on the very top of the tower). But most 
cellular providers are very protective of their time sources. They’re also 
typically clocking SONET networks too, which requires BITS.

 -mel


JAshworth said:
> CDMA and GSM are false diversity: both network types nodes *get their time* 
> from GPS, so far as I know.


> On May 11, 2016, at 10:54 AM, valdis.kletni...@vt.edu wrote:
> 
> On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said:
> 
>> CDMA and GSM are false diversity: both network types nodes *get their time*
>> from GPS, so far as I know.
> 
> I'll make the fairly reasonable assumption that most readers of this list have
> networks that span multiple buildings.
> 
> If somebody is managing to figure out that you have a GPS in Building 37, and 
> a
> GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at
> other locations and getting close enough to all of them at the same time to
> conduct a successful spoofing attack, all just to move your time source a
> few seconds off
> 
> ...  then the fact that GPS is spoofable is probably *NOT* your biggest
> security problem.
> 

Re: NIST NTP servers

[Florian Weimer]

* Chris Adams:

> First, out of the box, if you use the public pool servers (default
> config), you'll typically get 4 random (more or less) servers from the
> pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> high chance of guessing the servers your system is using.

A determined attacker will just run servers in the official pool.

Re: CALEA

[Ricky Beam]

On Tue, 10 May 2016 17:00:54 -0400, Brian Mengel  wrote:


AFAIK being able to do a lawful intercept on a specific, named,
individual's service has been a requirement for providers since 2007.


It's been required for longer than that. The telco I worked for over a  
decade ago didn't build the infrastructure until the FCC said they were  
going to stop funding upgrades. That really got 'em movin'. (suddenly  
"data services" people -- i.e. ME -- weren't redheaded stepchildren.)



have never heard of a provider, big or small, being called out for being
unable to provide this service when requested.


Where existing infrastructure is not already in place (read: T1/BRI/etc.),  
the telco can take up to 60 days to get that setup. I know more than one  
telco that used that grace period to actually setup CALEA in the first  
place.



did not perform intercepts routinely.


The historic published figures (i've not looked in years) suggest CALEA  
requests are statistically rare. The NC based telco I worked for had never  
received an order in the then ~40yr life of the company.


The mediation server needed to "mediate" between your customer  
aggregation box and the LEA is not inexpensive.


And also is not the telco's problem. Mediation is done by the LEA or 3rd  
party under contract to any number of agencies. For example, a telco tap  
order would mirror the control and voice traffic of a POTS line (T1/PRI  
channel, etc.) into a BRI or specific T1 channel. (dialup was later added,  
but wasn't required in my era, so we didn't support it.) We used to test  
that by tapping a tech's phone. Not having any mediation software, all I  
could do is "yeap, it's sending data" and listen to the voice channels on  
a t-berd.


--Ricky

Re: NIST NTP servers

[Lamar Owen]

On 05/11/2016 07:46 AM, Baldur Norddahl wrote:
But would you not need to actually spend three times $300 to get a 
good redundant solution?


While we are there, why not go all the way and get a rubidium standard 
with GPS sync? Anyone know of a (relatively) cheap solution with NTP 
output?


Ebay has several Symmetricom, Microsemi, Datum, Spectracom, and even 
Agilent solutions for prices from a few hundred US$ to a couple of 
thousand US$.  Even something like the Agilent Z3801, Z3805, or Z3816 
can be found for a few hundred US$.   New, these things are in the 
$10,000+ territory.  About the same range as mid-range ethernet gear.


I like our SSU2000, personally.

Re: NIST NTP servers

[Lamar Owen]

On 05/11/2016 12:05 AM, Joe Klein wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?


...

I remember it like it was only four years ago oh, wait

We have multiple sync sources ourselves, with a Symmetricom (formerly 
Datum) SSU2000 setup with a cesium PRS, a rubidium secondary, and an 
ovenized quartz for tertiary oscillators.  SSU2000 architecture is 
separate control and data planes, with time-sync on a different 
interface from the LAN-facing NTP NIC.  And the control plane is 
firewalled off from the main LAN.  An Agilent (now Symmetricom) Z3816 is 
secondary.


PC and SBC (RasPi, etc) oscillators are just not accurate enough for 
Stratum 1 standards; at best stratum 3 or 4, even when directly 
GPS-disciplined (stratum is NOT just a synonym for 'level' as a 
particular stratum really has stability, precision, and accuracy 
requirements).  WWV plus GPS; GPS as you may or may not be aware, is 
spoofable and is not as accurate as one might want.  Neither is WWV.


Good reference for time-nuts is, well, the 'time-n...@febo.com' mailing 
list.


(We're a radio astronomy observatory; accurate time and frequency 
standards are  a must here, especially as position accuracy of radio 
telescopes approach tens of arcseconds).

Re: NIST NTP servers

[Valdis . Kletnieks]

On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said:

> CDMA and GSM are false diversity: both network types nodes *get their time*
> from GPS, so far as I know.

I'll make the fairly reasonable assumption that most readers of this list have
networks that span multiple buildings.

If somebody is managing to figure out that you have a GPS in Building 37, and a
GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at
other locations and getting close enough to all of them at the same time to
conduct a successful spoofing attack, all just to move your time source a
few seconds off

...  then the fact that GPS is spoofable is probably *NOT* your biggest
security problem.



pgpYTudayH1Sq.pgp
Description: PGP signature

Re: NIST NTP servers

[Majdi S. Abbas]

On Wed, May 11, 2016 at 03:24:43PM +, Jay R. Ashworth wrote:
> We're all aware this project is underway, right?
> 
>   https://www.ntpsec.org/

Despite the name, I'm not aware of any significant protocol
changes.  It's just a recent fork of the reference implementation
minus the refclocks, which isn't particularly helpful if you /don't/
trust network time sources.

Long term, be looking at NTS:

https://datatracker.ietf.org/doc/draft-ietf-ntp-network-time-security/

In the meanwhile, I'd recommend something along the following
lines:

- Several nearby upstream servers configured per time server, per site
(As diversely as possible.)

- Diverse reference clocks (I run everything from WWV to GPS
  here.) providing authenticated time to your servers.

- That all your time servers in all sites be configured in an
authenticated full mesh of symmetric peers, allowing the other
sites to provide time to a site that has lost its upstream
servers or for whatever reason does not trust them at the moment.

And of course, ensure any hosts whose clocks you care about are
talking to at least a few of these, and preferably several.  I know the
common case configuration is either default/ntp-pool, or "we have two
time servers in this site and everything just chimes from them," but
neither is that great of a configuration.

--msa

Re: NIST NTP servers

[Jay R. Ashworth]

- Original Message -
> From: "Mel Beckman" 

> Read deeper into the thread and you'll find where I sourced inexpensive 
> RF-based
> NTP servers using CDMA, GSM, and even WWV. All radically different 
> technologies
> that are unlikely to have common failure modes. But yes, buying different
> brands can't hurt either.

CDMA and GSM are false diversity: both network types nodes *get their time* 
from GPS, so far as I know.

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: NIST NTP servers

[Jay R. Ashworth]

- Original Message -
> From: "Jared Mauch" 

>> Yes, and properly monitor your ntpd instances.
> 
> And upgrade them.
> 
> Some software distributors don’t ship modern software.  if you
> are using a distribution packaged ntpd it’s likely old and
> difficult to determine its lineage due to how it’s packaged.
> 
> If you’re using Redhat based systems consider using chrony
> instead, even the new beta fedora 24 uses 4.2.6 derived code
> vs 4.2.8

We're all aware this project is underway, right?

  https://www.ntpsec.org/

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: NIST NTP servers

[Scott Whyte]

On 5/10/16 21:05, Joe Klein wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
I believe this will become a stronger need over time, and apply to more 
than NTP: 
http://arstechnica.com/security/2016/02/using-ipv6-with-linux-youve-likely-been-visited-by-shodan-and-other-scanners/

6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  wrote:


I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

  -mel beckman


On May 10, 2016, at 5:18 PM, Chris Adams  wrote:

Once upon a time, Mel Beckman  said:

Boss: So how did a hacker get in and crash our accounting server, break

our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe Random Hacker isn't going to have a
high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the
common software won't step the clock more than IIRC 15 minutes.  Yes,
that can cause problems, but not the catastrophes of years in the future
or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
not so sure, and I haven't seen proof to the contrary.
--
Chris Adams 

RE: NIST NTP servers

[Chuck Church]

-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Leo Bicknell
>Sent: Wednesday, May 11, 2016 9:31 AM
>To: nanog@nanog.org
>Subject: Re: NIST NTP servers

>Personally, my network gets NTP from 14 stratum 1 sources right now.
>You, and the hacker, do not know which ones.  You have to guess at least
>8 to get me to move to your "hacked" time.  Good luck.

>Redundancy is the solution, not a new single point of failure.  GPS can be 
>part of the redundancy, not a sole solution.

This seems like the most reasonable advise.  If this truly becomes a concern, I 
would think IPS vendors could implement signatures to look for bad time.  Lots 
of ways to do this 
- look for a difference between the IPS realtime and NTP status versus the 
incoming packets.
- look for duplicate NTP responses, or responses that weren't requested 
- duplicate responses, but with differing TTLs, which might hint at one being 
spoofed.

Chuck

Re: NIST NTP servers

[Brandon Vincent]

GPS + a cesium or rubidium frequency standard is all you need.

Too expensive? Then time isn't important to your organization.

Re: NIST NTP servers

[Leo Bicknell]

In a message written on Wed, May 11, 2016 at 09:00:54AM -0500, Josh Reynolds 
wrote:
> I hope your receivers aren't all from a single source.

I have 4 each ACTS, GPS, and CDMA in my list, agumented with a pair
of PTP.  Amazingly right now all but 3 are within 2 microsconds of
each other, and those three outliers are 10 and 35 microseconds
off.  That's pretty impressive!

I didn't have to buy any of them, because various trustable entities
run those infrastructures.  Some of the trustable entites are the
same ones that send the time up to the GPS satellites. :)

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpKDSyi44ype.pgp
Description: PGP signature

Re: CALEA

[Leo Bicknell]

In a message written on Tue, May 10, 2016 at 03:00:59PM -0500, Josh Reynolds 
wrote:
> This is a large list that includes many Tier 1 network operators,
> government agencies,  and Fortune 500 network operators.
> 
> The silence should be telling.

NANOG has a strong self-selection for people who run core routing
devices and do things like BGP and peering negotiations with other
providers.

By contrast, CALEA requirements are generally all met by features
deployed at the customer-edge.  These groups are often a separate
silo from the backbone folks at the largest providers.

This is likely the wrong list for asking such questions, and the few
who do answer is likely to be smaller providers where people wear
multiple hats.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpWM43j2G20q.pgp
Description: PGP signature

Re: NIST NTP servers

[Mel Beckman]

Josh,

Read deeper into the thread and you'll find where I sourced inexpensive 
RF-based NTP servers using CDMA, GSM, and even WWV. All radically different 
technologies that are unlikely to have common failure modes. But yes, buying 
different brands can't hurt either. 

 -mel beckman

> On May 11, 2016, at 7:15 AM, Josh Reynolds  wrote:
> 
> I hope your receivers aren't all from a single source.
> 
> I was in Iraq when this (
> http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-on-gps/
> ) happened, which meant I had no GPS guided indirect fire assets for 2
> weeks.
> 
>> On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell  wrote:
>> In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman 
>> wrote:
>>> All because of misplaced trust in a tiny UDP packet that can worm its way 
>>> into your network from anywhere on the Internet.
>>> 
>>> I say you’re crazy if you don’t run a GPS-based NTP server, especially 
>>> given that they cost as little as $300 for very solid gear. Heck, get two 
>>> or three!
>> 
>> You're replacing one single point of failure with another.
>> 
>> Personally, my network gets NTP from 14 stratum 1 sources right now.
>> You, and the hacker, do not know which ones.  You have to guess at least
>> 8 to get me to move to your "hacked" time.  Good luck.
>> 
>> Redundancy is the solution, not a new single point of failure.  GPS
>> can be part of the redundancy, not a sole solution.
>> 
>> --
>> Leo Bicknell - bickn...@ufp.org
>> PGP keys at http://www.ufp.org/~bicknell/

Re: NIST NTP servers

[Mel Beckman]

Andreas,

Most data centers will require a remotely positioned NTP server, which is 
actually easier and cheaper than a remotely located active GPS antenna. I have 
placed the $300 commercial NTP servers in an environmental box on the roof, 
powering t by PoE, without problems. 

You don't need a redundant network either, nor redundant power. Just plunk down 
two of these gizmos, or as I suggested elsewhere, deploy one or more CDMA, GSM, 
or WWV-based clocks, for as much local infrastructure and signal source 
diversity as you like (I sourced some of these units earlier in the thread, all 
well less than $1K each. You pay more for diversity, but you get more too.

In response to the several DIYers on this thread: Anyone who thinks they're 
building a raspberry pi-based GPS NTP server for just $150 is kidding 
themselves. They forgot to include their labor, which is worth far more than 
the $300 for a commercial unit. The same goes for people who complain about 
even the minimal $300 price, forgetting that a commercial product has to pay 
for marketing, support, and make a profit. 

External NTP has two kinds of vulnerabilities: the ones we know, and the ones 
we don't. The ones we know are serious enough the pat the ones we don't should 
be considered with respect. Maybe diversity in Internet sources is a cure, 
maybe it isn't. But diverse RF sources is demonstrably more secure than the 
Internet.   

My point stands: secure external RF NTP sources are so plentiful that relying 
on Internet NTP is just plain crazy. 

 -mel beckman

> On May 11, 2016, at 7:12 AM, Andreas Ott  wrote:
> 
> Hi,
> 
>> Boss: That sounds expensive. How much are we talking?
>> IT guy: $300
> 
> Beware!
> 
> Over the past year we made engineering samples to deploy to datacenters.
> The goal was to use GPS and PPS to discipline ntpd appliances and serve 
> as stratum 1 to other NTP distribution servers without the $5k price tag
> of commercial NTP 1RU gear. We also deliberately not pursued the path of
> running antenna coax from the roof to a receiver, as that is not an
> option in all the datacenters where we need to deploy.
> 
> These appliances were built for lesss than $150 as 
> 
> (a) Raspberry-Pi with GPS receiver board
> 
> (b) Garmin 18(x) LVC with DB-9 to an "older whitebox server"
> 
> In my experience, most locations inside datacenters where you have good
> power and network connectivity have not "good enough" GPS signal reception
> due to servers emitting lots of RF noise in the range 1-2 GHz on the
> L-band. A brand new suite in the datacenter had OK GPS quality, but
> once we added 20+ racks with 40 servers each, GPS reception was pretty
> much gone. This hardware was an active antenna with less than 6 feet of
> cabling routed to the top of the network cabling rack. Most smartphones
> can run an app to show you the GPS signal on the phone, just walk around
> your datacenter and compare the signal.
> 
> The only workable solution was to move the GPS clock to a location
> where it had good GPS signal but neither redundant network nor conditioned
> power (aka. "on my desk near a south facing window"). It also works pretty 
> well "in my garage".
> 
> In places where GPS reception is good, you can achieve 10E-06 seconds
> accuracy over time even with cheap hardware. If you chose to run the DB-9
> NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor
> you can still get better than 10E-03 seconds accuracy.
> 
> 
> -andreas
> -- 
> Andreas Ott   (Time-Nut)   K6OTT   +1.408.431.8727   andr...@naund.org

Re: CALEA

[Brian Mengel]

AFAIK being able to do a lawful intercept on a specific, named,
individual's service has been a requirement for providers since 2007.  I
have never heard of a provider, big or small, being called out for being
unable to provide this service when requested.  I would be surprised if a
national broadband ISP with millions of subs did not have this ability and
did not perform intercepts routinely.  I would be surprised if a small town
providing it's own Internet access or small WISP serving a few hundred
customers went through the trouble and expense of being able to provide
this service.

The mediation server needed to "mediate" between your customer aggregation
box and the LEA is not inexpensive.  I believe there was talk about
"trusted third parties" providing mediation-as-a-service but I do not know
if any such entities exist.  The logistics of running a mediation server in
the cloud and being able to signal from the cloud to the aggregation box to
begin a mediation and ensuring that the data exported from the ISP to the
cloud to the LEA remained private would seem to be significant but not
insurmountable.



On Tue, May 10, 2016 at 4:11 PM, Josh Reynolds  wrote:

> The first rule of prism is...
>
>
> *silence*
>
>
> :)
>
> On Tue, May 10, 2016 at 3:04 PM, Christopher Morrow
>  wrote:
> >
> >
> > On Tue, May 10, 2016 at 4:00 PM, Josh Reynolds 
> wrote:
> >>
> >> This is a large list that includes many Tier 1 network operators,
> >> government agencies,  and Fortune 500 network operators
> >
> >
> > no one gets calea requests because prism gets all requests?
> >
>

Re: NIST NTP servers

[Josh Reynolds]

I hope your receivers aren't all from a single source.

I was in Iraq when this (
http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-on-gps/
) happened, which meant I had no GPS guided indirect fire assets for 2
weeks.

On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell  wrote:
> In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman 
> wrote:
>> All because of misplaced trust in a tiny UDP packet that can worm its way 
>> into your network from anywhere on the Internet.
>>
>> I say you’re crazy if you don’t run a GPS-based NTP server, especially given 
>> that they cost as little as $300 for very solid gear. Heck, get two or three!
>
> You're replacing one single point of failure with another.
>
> Personally, my network gets NTP from 14 stratum 1 sources right now.
> You, and the hacker, do not know which ones.  You have to guess at least
> 8 to get me to move to your "hacked" time.  Good luck.
>
> Redundancy is the solution, not a new single point of failure.  GPS
> can be part of the redundancy, not a sole solution.
>
> --
> Leo Bicknell - bickn...@ufp.org
> PGP keys at http://www.ufp.org/~bicknell/

Re: NIST NTP servers

[Andreas Ott]

Hi,

> Boss: That sounds expensive. How much are we talking?
> IT guy: $300

Beware!

Over the past year we made engineering samples to deploy to datacenters.
The goal was to use GPS and PPS to discipline ntpd appliances and serve 
as stratum 1 to other NTP distribution servers without the $5k price tag
of commercial NTP 1RU gear. We also deliberately not pursued the path of
running antenna coax from the roof to a receiver, as that is not an
option in all the datacenters where we need to deploy.

These appliances were built for lesss than $150 as 

(a) Raspberry-Pi with GPS receiver board

(b) Garmin 18(x) LVC with DB-9 to an "older whitebox server"

In my experience, most locations inside datacenters where you have good
power and network connectivity have not "good enough" GPS signal reception
due to servers emitting lots of RF noise in the range 1-2 GHz on the
L-band. A brand new suite in the datacenter had OK GPS quality, but
once we added 20+ racks with 40 servers each, GPS reception was pretty
much gone. This hardware was an active antenna with less than 6 feet of
cabling routed to the top of the network cabling rack. Most smartphones
can run an app to show you the GPS signal on the phone, just walk around
your datacenter and compare the signal.

The only workable solution was to move the GPS clock to a location
where it had good GPS signal but neither redundant network nor conditioned
power (aka. "on my desk near a south facing window"). It also works pretty 
well "in my garage".

In places where GPS reception is good, you can achieve 10E-06 seconds
accuracy over time even with cheap hardware. If you chose to run the DB-9
NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor
you can still get better than 10E-03 seconds accuracy.


-andreas
-- 
Andreas Ott   (Time-Nut)   K6OTT   +1.408.431.8727   andr...@naund.org

third party single pole administrator regimes

[Fletcher Kittredge]

This is outside plant related. Ignore if all you do is configure routers
(not that there is anything wrong with that.)

I would be interested in any network provider's experiences with third
party single pole administrator regulatory regimes, such as Connecticut's.
Please respond privately and I will summarize if there is any interest.

thanks

-- 
Fletcher Kittredge
GWI
www.gwi.net

RE: NIST NTP servers

[Allan Liska]

On 5/10/2016 at 10:30 AM, "Chuck Church"  wrote:

>
>It doesn't really.  Granted there are a lot of CVEs coming out for 
>NTP the
>last year or so.  But I just don't think there are that many 
>attacks on it.
>It's just not worth the effort.  Changing time on devices is more 
>an
>annoyance than anything, and doesn't necessarily get you into a 
>device.
>Sure you can hide your tracks a little by altering time in logs 
>and altering
>it back, but that's more of an in-depth nation-state kind of 
>attack, not
>going to be a script kiddie kind of thing.  Just follow the best 
>practices
>for verifying packet sources and NTP security itself, and you 
>should be ok.
>
>Chuck

I would argue that the fact the NTP can, and has been, be used in DDoS 
amplification attacks is a serious concern for using protocol going forward.



allan

Re: NIST NTP servers

[Ryan Harden]

_Everything_ has vulnerabilities and using _any_ external source opens your 
network and infrastructure to disruptions. NTP has been used for DDoS 
amplification attacks recently, but so has DNS and other well known/heavily 
used protocols.

With the right protections, syncing with an external NTP source is perfectly 
acceptable and safe.
Further, it’s generally a good idea to ‘peer’ (not just sync) your NTP servers 
with a few external sources. This removes the dependence on a single source and 
helps ensure that your time source agrees with the rest of the world. Peering 
requires interaction with the owners of the remote site, which establishes a 
basic level of trust that they’ll provide an accurate and stable service.

I’ve attached a diagram (sanitized) of what our NTP service will look like 
after an upcoming refresh.
All external sources are trusted and will be peered. All time devices peer with 
four other sources to ensure there is always a live source to sync/peer with.
A DNS record with round-robin is used for local clients to connect to the local 
Stratum 2 devices. The Stratum 1 GPS will not be directly accessible by users.

/Ryan

[cid:5676FF89-CBC8-42F7-84CE-69F431C23E48@int.ancker.net]

Ryan Harden
Research and Advanced Networking Architect
University of Chicago - ASN160
P: 773.834.5441

On May 10, 2016, at 5:48 AM, Steven Miano 
> wrote:

NTP has vulnerabilities, so using an external source opens your networks
and infrastructure to disruptions.

Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict
incoming traffic and not rely on volunteers or external entities (which may
undergo maintenance or budget issues).

My preference is more so something akin to the GLN180PEX (I am not
affiliated or paid to endorse this product). It allows you to use commodity
hardware (like a decommissioned 1U or several preferably) and creation of
ones own reliable internal time source(s). Introducing black boxes into a
production (revenue generation or expected services by paying customers)
environment is undesirable.

From there setting up NTPd, Chronyd, and PTPd is up to you.

Relying on satellites may seem like just another external reliance, but the
next life is proposing a design life of 12 years.

On Mon, May 9, 2016 at 11:12 PM, Majdi S. Abbas 
> wrote:

On Tue, May 10, 2016 at 03:08:16AM +, Mel Beckman wrote:
NTP has vulnerabilities that make it generally unsuitable for
provider networks. I strongly recommend getting a GPS-based
time server. These are as cheap as $300. Here is one I use quite a bit:

   So how does this stop from distributing time to their
customers via NTP?

   GPS doesn't save the protocol, in particular where the S1
clocks involved are embedded devices with rather coarse clocks and
timestamping.

   --msa




--
Miano, Steven M.
http://stevenmiano.com

VirginMedia AS5089

[Goodwin, Simon T via NANOG]

Can somebody contact me offline from VirginMedia UK regarding a BGP 
advertisement for 192.34.50.0/24 originating from AS1849 please

Thanks

Simon

Re: NIST NTP servers

[Leo Bicknell]

In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman 
wrote:
> All because of misplaced trust in a tiny UDP packet that can worm its way 
> into your network from anywhere on the Internet.
> 
> I say you’re crazy if you don’t run a GPS-based NTP server, especially given 
> that they cost as little as $300 for very solid gear. Heck, get two or three!

You're replacing one single point of failure with another.

Personally, my network gets NTP from 14 stratum 1 sources right now.
You, and the hacker, do not know which ones.  You have to guess at least
8 to get me to move to your "hacked" time.  Good luck.

Redundancy is the solution, not a new single point of failure.  GPS
can be part of the redundancy, not a sole solution.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpZ8nfasXwtV.pgp
Description: PGP signature

Re: NIST NTP servers

[Eygene Ryabinkin]

Tue, May 10, 2016 at 04:59:02PM +0200, Stephane Bortzmeyer wrote:
> On Tue, May 10, 2016 at 10:52:28AM -0400,
>  valdis.kletni...@vt.edu  wrote 
>  a message of 37 lines which said:
> 
> > Note that they *do* have motivation to keep it working, simply
> > because so much of their *own* gear (from gear for individual
> > soldiers all the way to strategic bombers and aircraft carriers)
> > wants a working GPS signal...
> 
> Yes, but they may switch it off for civilian use (by going encrypted,
> for instance) at any time, if it is better for *their* operations.

Civilian signals are already degraded in terms of accuracy (without
assisted GPS) and military ones are encrypted (but see [1]).  The
primary reason for encryption is, by the way, to address spoofing
issues for the mil people themselves, but it is also good not to arm
the potential enemy with the precise coordinates or to be able to
do spoofing for them.

And since many civilian, but nonetheless, vital orgs are using
civilian parts, it could be hard to shut it off without affecting some
parts of the infrastructure.  Not that nobody wants that, but it will
give an unneeded resonance and could lead to the terrible mess.

[1] http://www.gps.gov/technical/codeless/
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.

Re: NIST NTP servers

[Baldur Norddahl]

But would you not need to actually spend three times $300 to get a good 
redundant solution?


While we are there, why not go all the way and get a rubidium standard 
with GPS sync? Anyone know of a (relatively) cheap solution with NTP output?


https://en.wikipedia.org/wiki/Rubidium_standard

Regards,

Baldur

On 2016-05-11 09:24, Mel Beckman wrote:

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

  -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
> wrote:

Once upon a time, Mel Beckman > said:

Re: NIST NTP servers

[Steven Miano]

Building a S1 system with RaspberryPis would not fly in most of the
corporate/enterprise environments I've worked in (random 'appliances',
non-uniformity, and lack of support are all glaring issues).

Get a PCIe card with a BNC connector and dual power supplies for life in a
data center.

For home/hobby use a Garmin 18x LVC and any spare compute is a great
project: http://www.catb.org/gpsd/gpsd-time-service-howto.html

On Wed, May 11, 2016 at 6:47 AM, Dovid Bender  wrote:

> What about something like this?
> http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
> Has anyone used a Pi to create their own server?
>
>
> On Wed, May 11, 2016 at 3:24 AM, Mel Beckman  wrote:
>
> > Regarding Roland’s reference to time and position spoofing via a hacked
> > GPS signal, the hacker has to get physical line of sight to the victim’s
> > antenna in order to succeed with this attack. That’s likely within a few
> > blocks, if not within a few feet. And a rooftop antenna might require a
> > drone attack. And how does the drone get guidance without a reliable GPS
> > signal? :)
> >
> > Eric, I agree that sometimes a site can’t get a GPS signal, but in my
> > experience designing data centers, that’s still pretty rare. Many NTP
> > systems use an active GPS antenna that can be hundreds of feet away. But
> > you can always put the $300 NTP server in an outdoor enclosure and power
> it
> > with PoE.
> >
> > There’s always CDMA, GSM, and even WWV for a less-accurate plan B time
> > source.  Here’s a somewhat pricey ($700) CDMA gizmo I haven’t
> investigated
> > yet:
> >
> > http://www.beaglesoft.com/celsynhowworks.htm
> >
> > And their $400 WWV-based Stratum 1 time server:
> >
> > http://www.beaglesoft.com/radsynreceiver.htm
> >
> > So if you want non-Internet clock diversity, you can have clock
> diversity.
> > You just have to pay for it.
> >
> >  -mel
> >
> > On May 10, 2016, at 9:18 PM, Eric Kuhnke  eric.kuh...@gmail.com>> wrote:
> >
> > For quite some time, in debian the default configuration for the
> ntpd.conf
> > that ships with the package for the ntpd is to poll from four different,
> > semi-randomly assigned DNS pool based sources. I believe the same is true
> > for redhat/centos.
> >
> > In the event that one out of four sources is wildly wrong the ntpd will
> > ignore it.
> >
> > If people have routers/networking equipment inside their network that
> only
> > supports retrieving ntp from one IP address (or hostname) and have
> manually
> > configured it to request time from a single external source, not their
> own
> > internal ntpd that is <10ms away, bad things could definitely happen.
> >
> > It is worthwhile to have both polling from external sources via IP as
> well
> > as GPS sync. Many locations in a network have no hope of getting a GPS
> > signal or putting an antenna with a clear view to the sky, but may be on
> a
> > network segment that is <4ms away from many other nodes where you can
> > colocate a 1U box and GPS antenna.
> >
> > On Tue, May 10, 2016 at 9:05 PM, Joe Klein  jskl...@gmail.com>> wrote:
> >
> > Is this group aware of the incident with tock.usno.navy.mil &
> > tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost
> 12
> > years for the period of one hour, then return?
> >
> > The reasons were not fully explained, but the impact was global. Routers,
> > switches, power grids, phone systems, certificates, encryption, Kerberos,
> > logging and any tightly coupled transaction systems were impacted.
> >
> > So I began doing 'security research' on the topic (don't confuse me with
> > joe hacker), and discovered both interesting and terrifying issues,
> which I
> > will not disclose on an open forum.
> >
> > Needless to say, my suggestions are:
> > 1. Configure a trusted time source and good time stratum architecture for
> > your organization.
> > 2. When identifying your source of time, the majority of the technologies
> > can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
> > authentication.
> > 3. For distribution of time information inside your organization, ensure
> > your critical systems (Encryption, PKI, transactions, etc) are using your
> > redundant sources and authentication.
> > 4. Operating systems, programming languages, libraries, and applications
> > are sensitive to time changes and can fail in unexpected ways. Test them
> > before it's too late.
> > 5. Disallow internal system to seek NTP from other sources beyond your
> edge
> > routers.
> > 6. All core time systems should be monitored by your security team or
> SOC.
> >
> > One question, is this a topic anyone would find interested at a future
> > NANOG? Something like "Hacking and Defending time?".
> >
> >
> > Joe Klein
> > "Inveniam viam aut faciam"
> >
> > PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
> >
> > On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  

Re: NIST NTP servers

[Dovid Bender]

What about something like this?
http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
Has anyone used a Pi to create their own server?


On Wed, May 11, 2016 at 3:24 AM, Mel Beckman  wrote:

> Regarding Roland’s reference to time and position spoofing via a hacked
> GPS signal, the hacker has to get physical line of sight to the victim’s
> antenna in order to succeed with this attack. That’s likely within a few
> blocks, if not within a few feet. And a rooftop antenna might require a
> drone attack. And how does the drone get guidance without a reliable GPS
> signal? :)
>
> Eric, I agree that sometimes a site can’t get a GPS signal, but in my
> experience designing data centers, that’s still pretty rare. Many NTP
> systems use an active GPS antenna that can be hundreds of feet away. But
> you can always put the $300 NTP server in an outdoor enclosure and power it
> with PoE.
>
> There’s always CDMA, GSM, and even WWV for a less-accurate plan B time
> source.  Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated
> yet:
>
> http://www.beaglesoft.com/celsynhowworks.htm
>
> And their $400 WWV-based Stratum 1 time server:
>
> http://www.beaglesoft.com/radsynreceiver.htm
>
> So if you want non-Internet clock diversity, you can have clock diversity.
> You just have to pay for it.
>
>  -mel
>
> On May 10, 2016, at 9:18 PM, Eric Kuhnke > wrote:
>
> For quite some time, in debian the default configuration for the ntpd.conf
> that ships with the package for the ntpd is to poll from four different,
> semi-randomly assigned DNS pool based sources. I believe the same is true
> for redhat/centos.
>
> In the event that one out of four sources is wildly wrong the ntpd will
> ignore it.
>
> If people have routers/networking equipment inside their network that only
> supports retrieving ntp from one IP address (or hostname) and have manually
> configured it to request time from a single external source, not their own
> internal ntpd that is <10ms away, bad things could definitely happen.
>
> It is worthwhile to have both polling from external sources via IP as well
> as GPS sync. Many locations in a network have no hope of getting a GPS
> signal or putting an antenna with a clear view to the sky, but may be on a
> network segment that is <4ms away from many other nodes where you can
> colocate a 1U box and GPS antenna.
>
> On Tue, May 10, 2016 at 9:05 PM, Joe Klein > wrote:
>
> Is this group aware of the incident with tock.usno.navy.mil &
> tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
> years for the period of one hour, then return?
>
> The reasons were not fully explained, but the impact was global. Routers,
> switches, power grids, phone systems, certificates, encryption, Kerberos,
> logging and any tightly coupled transaction systems were impacted.
>
> So I began doing 'security research' on the topic (don't confuse me with
> joe hacker), and discovered both interesting and terrifying issues, which I
> will not disclose on an open forum.
>
> Needless to say, my suggestions are:
> 1. Configure a trusted time source and good time stratum architecture for
> your organization.
> 2. When identifying your source of time, the majority of the technologies
> can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
> authentication.
> 3. For distribution of time information inside your organization, ensure
> your critical systems (Encryption, PKI, transactions, etc) are using your
> redundant sources and authentication.
> 4. Operating systems, programming languages, libraries, and applications
> are sensitive to time changes and can fail in unexpected ways. Test them
> before it's too late.
> 5. Disallow internal system to seek NTP from other sources beyond your edge
> routers.
> 6. All core time systems should be monitored by your security team or SOC.
>
> One question, is this a topic anyone would find interested at a future
> NANOG? Something like "Hacking and Defending time?".
>
>
> Joe Klein
> "Inveniam viam aut faciam"
>
> PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
>
> On Tue, May 10, 2016 at 9:59 PM, Mel Beckman > wrote:
>
> I don't pretend to know all the ways a hacker can find out what nap
> servers a company uses, but I can envision a virus that could do that
> once
> behind a firewall. Every ntp response lists the current reference ntp
> server in the next higher stratum. There are many ways that process could
> harvest all ntp servers over time, and then pass the public IP back to a
> mother ship controller. It could be going on right now.
>
> My point is, when the fix is so cheap, why put up with this risk at all?
>
> -mel beckman
>
> On May 10, 2016, at 5:18 PM, Chris Adams > wrote:
>
> Once upon a time, Mel Beckman >
> said:
> 

Re: NIST NTP servers

[Mel Beckman]

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

 -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
> wrote:

Once upon a time, Mel Beckman > said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go
if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), 

Re: NIST NTP servers

[Mel Beckman]

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

 -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
> wrote:

Once upon a time, Mel Beckman > said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go
if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), 

Re: NIST NTP servers

[Mel Beckman]

Regarding Roland’s reference to time and position spoofing via a hacked GPS 
signal, the hacker has to get physical line of sight to the victim’s antenna in 
order to succeed with this attack. That’s likely within a few blocks, if not 
within a few feet. And a rooftop antenna might require a drone attack. And how 
does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my 
experience designing data centers, that’s still pretty rare. Many NTP systems 
use an active GPS antenna that can be hundreds of feet away. But you can always 
put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source.  
Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You 
just have to pay for it.

 -mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke 
> wrote:

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein 
> wrote:

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman 
> wrote:

I don't pretend to know all the ways a hacker can find out what nap
servers a company uses, but I can envision a virus that could do that
once
behind a firewall. Every ntp response lists the current reference ntp
server in the next higher stratum. There are many ways that process could
harvest all ntp servers over time, and then pass the public IP back to a
mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams 
> wrote:

Once upon a time, Mel Beckman > said:
Boss: So how did a hacker get in and crash our accounting server,
break
our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go
if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config),