Re: NTP Issues Today

2012-11-21 Thread Ask Bjørn Hansen
On Nov 20, 2012, at 13:00, Darius Jahandarie wrote: Hi everyone, I run the NTP Pool system - http://www.pool.ntp.org/ - so I have some opinions on some of this. :-) > But beyond that, I'm honestly rather curious what server selections > are a good idea. A first thought would be an adjacent co

Re: NTP Issues Today

2012-11-21 Thread Majdi S. Abbas
On Wed, Nov 21, 2012 at 10:41:01AM -0500, Jay Ashworth wrote: > "... against multiple [Stratum 1] sources..." > > Baby, if you've ever wondered... whether it matters whether your sources > are strat 1 or not, now you know -- since there's no real way to get > provenance on down-strat time sources

Re: NTP Issues Today

2012-11-21 Thread Jay Ashworth
- Original Message - > From: "Sid Rao" > We were synchronized against multiple sources. Unfortunately the Navy > NTP source contaminated multiple downstream sources. > > Unless you can trace all your sources, if these sources all have a > root source you will break. "... against multipl

Re: NTP Issues Today

2012-11-21 Thread Greg Ihnen
It sounds like the Navy and who ever else they partner with (NIST?) need some egress filtering on their NTP servers to catch and prevent events like this.

RE: NTP Issues Today

2012-11-21 Thread Chuck Church
-Original Message- >From: Jimmy Hess [mailto:mysi...@gmail.com] >Sent: Tuesday, November 20, 2012 7:50 PM >To: Van Wolfe >Cc: nanog@nanog.org >Subject: Re: NTP Issues Today >This _should_ have caused NTP to execute a panic shutdown, >instead of setting the c

Re: NTP Issues Today

2012-11-21 Thread Sid Rao
Guys: We were synchronized against multiple sources. Unfortunately the Navy NTP source contaminated multiple downstream sources. Unless you can trace all your sources, if these sources all have a root source you will break. Sid Rao | CTI Group | +1 (317) 262-4677 On Nov 21, 2012, at 8:01 AM

Re: NTP Issues Today

2012-11-21 Thread Neil Harris
On 21/11/12 12:34, Ryan Malayter wrote: On Nov 19, 2012, at 6:12 PM, "Scott Weeks" wrote: Lesson learned: Use more than one NTP source. The lesson is: use MORE THAN TWO diverse NTP sources. A man with two watches has no idea what the time it actually is. Per David Mills, from the discu

Re: NTP Issues Today

2012-11-21 Thread Ryan Malayter
On Nov 19, 2012, at 6:12 PM, "Scott Weeks" wrote: > Lesson learned: Use more than one NTP source. > The lesson is: use MORE THAN TWO diverse NTP sources. A man with two watches has no idea what the time it actually is.

Re: NTP Issues Today

2012-11-21 Thread Ryan Malayter
On Nov 19, 2012, at 6:12 PM, "Scott Weeks" wrote: > wbai...@satelliteintelligencegroup.com> > > Or you could just concede the fact that the navy is playing with time travel > again. > -- > > > To finish this thread off for the archives

Re: NTP Issues Today

2012-11-21 Thread Robert E. Seastrom
Blake Dunlap writes: > That's what happens when you just follow vendor recommendations blindly. If > you do follow that on vm's (which can actually be a good practice), make > sure they pull from your own time infrastructure, and not just the world at > large, and that those servers behave in a

Re: NTP Issues Today

2012-11-20 Thread George Herbert
As a reminder - time infrastructure is not recommended for virtualization. Make them physicals. On Tue, Nov 20, 2012 at 5:03 PM, Blake Dunlap wrote: > That's what happens when you just follow vendor recommendations blindly. If > you do follow that on vm's (which can actually be a good practice)

Re: NTP Issues Today

2012-11-20 Thread Blake Dunlap
That's what happens when you just follow vendor recommendations blindly. If you do follow that on vm's (which can actually be a good practice), make sure they pull from your own time infrastructure, and not just the world at large, and that those servers behave in a sane fashion with regard to time

Re: NTP Issues Today

2012-11-20 Thread Alvaro Pereira
Looks like something bad has happened: Behind the Random NTP Bizarreness of Incorrect Year Being Set https://isc.sans.edu/diary.html?n&storyid=14548 --- "A few people have written in within the past 18 hours about their NTP server/clients getting set to the year 2000. The cause of this behavior i

Re: NTP Issues Today

2012-11-20 Thread Damian Menscher
On Tue, Nov 20, 2012 at 4:49 PM, Jimmy Hess wrote: > On 11/19/12, Van Wolfe wrote: > > Did anyone else experience issues with NTP today? We had our server > > times update to the year 2000 at around 3:30 MT, then revert back to > 2012. > > Are you sure that you are actually using NTP to set you

Re: NTP Issues Today

2012-11-20 Thread Darius Jahandarie
On Tue, Nov 20, 2012 at 7:49 PM, Jimmy Hess wrote: > Are you sure that you are actually using NTP to set your clock? > For you to sync with 2000, you should have had multiple confused > peers from multiple time sources; possibly a false radio signal > > NTP by default has a panic threshold o

Re: NTP Issues Today

2012-11-20 Thread Jimmy Hess
On 11/19/12, Van Wolfe wrote: > Did anyone else experience issues with NTP today? We had our server > times update to the year 2000 at around 3:30 MT, then revert back to 2012. Are you sure that you are actually using NTP to set your clock? For you to sync with 2000, you should have had multipl

Re: Picking outside NTP servers (Re: NTP Issues Today)

2012-11-20 Thread Majdi S. Abbas
On Tue, Nov 20, 2012 at 04:53:39PM -0500, Jay Ashworth wrote: > For myself, I usually pick the first three in us.pool.ntp.org, tick and tock, > time.nist.gov, and a couple of regionally appropriate large universities. I'd advise going through the RR for a while, and pick servers close to y

Re: Picking outside NTP servers (Re: NTP Issues Today)

2012-11-20 Thread George Herbert
On Tue, Nov 20, 2012 at 1:53 PM, Jay Ashworth wrote: > > For myself, I usually pick the first three in us.pool.ntp.org, tick and tock, > time.nist.gov, and a couple of regionally appropriate large universities. As this week indicated, perhaps tick and tock are not sufficiently far apart to be

Picking outside NTP servers (Re: NTP Issues Today)

2012-11-20 Thread Jay Ashworth
- Original Message - > From: "Darius Jahandarie" > Choosing the first four servers is usually pretty straightforward: > *.CC.pool.ntp.org > > But beyond that, I'm honestly rather curious what server selections > are a good idea. A first thought would be an adjacent country, but > maybe t

Re: NTP Issues Today

2012-11-20 Thread Jared Mauch
On Nov 20, 2012, at 4:00 PM, Darius Jahandarie wrote: > Choosing the first four servers is usually pretty straightforward: > *.CC.pool.ntp.org > > But beyond that, I'm honestly rather curious what server selections > are a good idea. A first thought would be an adjacent country, but > maybe the

RE: [outages] NTP Issues Today

2012-11-20 Thread R. Benjamin Kessler
-Original Message- From: outages-boun...@outages.org [mailto:outages-boun...@outages.org] On Behalf Of Jeremy Chadwick Sent: Tuesday, November 20, 2012 10:38 AM To: Scott Voll Cc: Sid Rao; outages; nanog@nanog.org Subject: Re: [outages] NTP Issues Today I'm still waiting for someone wh

Re: NTP Issues Today

2012-11-20 Thread Mike Lyon
I usually use time.nist.gov. On Tue, Nov 20, 2012 at 1:00 PM, Darius Jahandarie wrote: > On Tue, Nov 20, 2012 at 3:15 PM, Leo Bicknell wrote: > > For small players, less than 4 sites, typically just use the NTP > > pool servers, configuring 4 per box minimum. If you want the same > > protection

Re: NTP Issues Today

2012-11-20 Thread Darius Jahandarie
On Tue, Nov 20, 2012 at 3:15 PM, Leo Bicknell wrote: > For small players, less than 4 sites, typically just use the NTP > pool servers, configuring 4 per box minimum. If you want the same > protection I just outlined in the paragraph before, make 4 of your > servers talk to the outside world, and

Re: NTP Issues Today

2012-11-20 Thread George Herbert
On Nov 20, 2012, at 11:39 AM, Jared Mauch wrote: . > > I've also been looking at an item like this: > > http://www.netburnerstore.com/ProductDetails.asp?ProductCode=PK70EX-NTP > > which is about $300 + misc parts. > > Should be well worth it to avoid a 'major outage' that some folks had wi

Re: NTP Issues Today

2012-11-20 Thread Leo Bicknell
In a message written on Tue, Nov 20, 2012 at 02:28:19PM -0500, Jay Ashworth wrote: > I'm curious, Leo, what your internal setup looks like. Do you have an > internal pair of masters, all slaved to those externals and one another, > with your machines homed to them? Full mesh? Or something else

Re: NTP Issues Today

2012-11-20 Thread Jared Mauch
On Nov 20, 2012, at 2:28 PM, Jay Ashworth wrote: > - Original Message - >> From: "Leo Bicknell" > >> To protect against two falseticking servers (tick and tock, as we saw on >> the 19th) you need _FIVE_ servers minimum configured if they are both in >> the list. More importantly, if yo

Re: NTP Issues Today

2012-11-20 Thread Jay Ashworth
- Original Message - > From: "Leo Bicknell" > To protect against two falseticking servers (tick and tock, as we saw on > the 19th) you need _FIVE_ servers minimum configured if they are both in > the list. More importantly, if you want to protect against a source > (GPS, CDMA, IRIG, WWIV,

Re: NTP Issues Today

2012-11-20 Thread Leo Bicknell
After some private replies, I'm going to reply to my own post with some information here. It appears many people don't understand how the NTP protocol works. I suspect many people have configured a "primary" and a "backup" NTP server on many of their devices. It turns out this is the _WORST_ pos

Re: NTP Issues Today

2012-11-20 Thread Seth Mattinen
On 11/19/12 6:08 PM, Wallace Keith wrote: > Just got paged with a pbx alarm that had 1970 as the year. By the time I > logged in , it was showing 2012. Using GPS for time and date. > I use GPS for my NTP server and didn't notice anything, but it's PPS disciplined after initial sync so it does

Re: [outages] NTP Issues Today

2012-11-20 Thread Colin Johnston
no idea, re sigterm cause checked firewall system logs and could not see cause from that either times are GMT Colin On 20 Nov 2012, at 17:05, Jeremy Chadwick wrote: > Colin, > > Signal 15 = SIGTERM, so something intentionally shut ntpd down on your > side. The logs I'd be interested in would

Re: NTP Issues Today

2012-11-20 Thread Steve Meuse
On Tue, Nov 20, 2012 at 11:38 AM, Leo Bicknell wrote: > > If your machines switched dates yesterday it probably means you're > NTP infrastructure is insufficiently peered and diversified. > If you take anything away from this thread, this is it -Steve

Re: [outages] NTP Issues Today

2012-11-20 Thread Colin Johnston
On 20 Nov 2012, at 15:38, Jeremy Chadwick wrote: > I'm still waiting for someone who was affected by this to provide > coherent logs from ntpd showing exactly when the time change happened. > Getting these, at least on an *IX system, is far from difficult folks. > from firewall ntp logs Nov 19

Re: NTP Issues Today

2012-11-20 Thread Leo Bicknell
In a message written on Mon, Nov 19, 2012 at 04:21:55PM -0700, Van Wolfe wrote: > Did anyone else experience issues with NTP today? We had our server > times update to the year 2000 at around 3:30 MT, then revert back to 2012. I'm surprised the various time geeks aren't all posting their logs, so

Re: NTP Issues Today

2012-11-20 Thread Sid Rao
>> From: Mark Andrews [mailto:ma...@isc.org] >> Sent: Monday, November 19, 2012 8:42 PM >> To: Van Wolfe >> Cc: nanog@nanog.org >> Subject: Re: NTP Issues Today >> >> >> In message >> >> , Van Wolfe writes: >>> Hello

Re: [outages] NTP Issues Today

2012-11-19 Thread Mike Lyon
gt; -george > > > > On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith > > wrote: > >> Just got paged with a pbx alarm that had 1970 as the year. By the time > I logged in , it was showing 2012. Using GPS for time and date. > >> > >>

Re: NTP Issues Today

2012-11-19 Thread George Herbert
howing 2012. Using GPS for time and date. > > -Original Message- > From: Mark Andrews [mailto:ma...@isc.org] > Sent: Monday, November 19, 2012 8:42 PM > To: Van Wolfe > Cc: nanog@nanog.org > Subject: Re: NTP Issues Today > > > In message > > , Van Wol

RE: NTP Issues Today

2012-11-19 Thread Wallace Keith
: NTP Issues Today In message , Van Wolfe writes: > Hello, > > Did anyone else experience issues with NTP today? We had our server > times update to the year 2000 at around 3:30 MT, then revert back to 2012. > > Thanks, > Van NTP should be immune from this sort of behav

Re: NTP Issues Today

2012-11-19 Thread Mark Andrews
In message , Van Wolfe writes: > Hello, > > Did anyone else experience issues with NTP today? We had our server > times update to the year 2000 at around 3:30 MT, then revert back to 2012. > > Thanks, > Van NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong

Re: NTP Issues Today

2012-11-19 Thread Warren Bailey
Or you could just concede the fact that the navy is playing with time travel again. From my Galaxy Note II, please excuse any mistakes. Original message From: Scott Weeks Date: 11/19/2012 3:52 PM (GMT-08:00) To: nanog@nanog.org Subject: Re: NTP Issues Today On 11

NTP Issues Today

2012-11-19 Thread Oscar Orosco
We had the same issue on our NTP server pointing to tick.usno.navy.mil. Set date back to year 2000. Date: Mon, 19 Nov 2012 16:21:55 -0700 From: Van Wolfe mailto:vanwo...@gmail.com>> To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: NTP Issues Today

Re: NTP Issues Today

2012-11-19 Thread Scott Weeks
--- wbai...@satelliteintelligencegroup.com wrote: From: Warren Bailey Or you could just concede the fact that the navy is playing with time travel again. -- To finish this thread off for the archives... Apparently something was up with

Re: NTP Issues Today

2012-11-19 Thread Scott Weeks
On 11/19/12 6:32 PM, "Scott Weeks" wrote: >--- vanwo...@gmail.com wrote: >From: Van Wolfe > >Did anyone else experience issues with NTP today? We had our server >times update to the year 2000 at around 3:30 MT, then revert back to 2012. >- >You need

Re: NTP Issues Today

2012-11-19 Thread Clay Haynes
Scott, I can confirm this had happened on one of my test servers - it was pointing to tick.usno.navy.mil and tock.usno.navy.mil at the time. - Clay On 11/19/12 6:32 PM, "Scott Weeks" wrote: > > >--- vanwo...@gmail.com wrote: >From: Van Wolfe > >Did anyone else experience issues with NTP tod

Re: NTP Issues Today

2012-11-19 Thread Scott Weeks
--- vanwo...@gmail.com wrote: From: Van Wolfe Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. - You need to provide more information. For example, what NTP

NTP Issues Today

2012-11-19 Thread Van Wolfe
Hello, Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. Thanks, Van