Re: [Wikitech-l] [Engineering] Upcoming leap second on Tue 30th

[Moritz Muhlenhoff]

Hi Gabriel,

We just talked about this on IRC, so just for other's benefit: With NTP's
> -x option we should be able to smear the adjustment (by slowing down the
> system clock temporarily) until the leap second is incorporated into the
> system time. This avoids non-monotonicity, which is important for systems
> that use time to capture causality. It would be great to apply the
> adjustment to all nodes of the cassandra cluster at once, so that their
> clocks are being slewed in lock-step.
>

Unfortunately this was broken in NTP 4.2.6 and only recently discovered:
http://bugs.ntp.org/show_bug.cgi?id=2745 (only fixed in current
development/pre-releases). Even if we would backport the fixes to our time
servers we'd run into problems, since the local time deviation in the
"normalisation period" wouldn't be consistent across the nodes of the
Cassandra cluster.

(chrony fully supports NTP smearing since the 2.0 release (27th April
2015), but that's also not a solution for the upcoming leap second).

We'll follow up in a separate mail how to best accomodate the Cassandra
cluster.

Cheers,
Moritz
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] Upcoming leap second on Tue 30th

[Jamison Lofthouse]

All staff members, in preparation for leap second 2015, should watch a
brief training on the other problems with time & timezones[1].

[1] https://www.youtube.com/watch?v=-5wpm-gesOY

On Thu, Jun 25, 2015 at 12:05 PM Gabriel Wicke  wrote:

> Alex and Moritz,
>
> thank you for taking care of this. These leap seconds are a real pain in
> the butt for time-based distributed systems, and I'm glad that we have a
> plan in place. I hope the movement to abolish leap seconds
> <
> https://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds
> >
> wins out in the end!
>
> On Thu, Jun 25, 2015 at 7:27 AM, Moritz Mühlenhoff 
> wrote:
> >
> > * On the 1st of July we'll re-enable NTP in batches. System clocks will
> > move forward by a second once NTP is started again,
> >
>
> To clarify: By default, system time will move *backwards* one second.
>
> We just talked about this on IRC, so just for other's benefit: With NTP's
> -x option we should be able to smear the adjustment (by slowing down the
> system clock temporarily) until the leap second is incorporated into the
> system time. This avoids non-monotonicity, which is important for systems
> that use time to capture causality. It would be great to apply the
> adjustment to all nodes of the cassandra cluster at once, so that their
> clocks are being slewed in lock-step.
>
> Gabriel
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] Upcoming leap second on Tue 30th

[Gabriel Wicke]

Alex and Moritz,

thank you for taking care of this. These leap seconds are a real pain in
the butt for time-based distributed systems, and I'm glad that we have a
plan in place. I hope the movement to abolish leap seconds

wins out in the end!

On Thu, Jun 25, 2015 at 7:27 AM, Moritz Mühlenhoff 
wrote:
>
> * On the 1st of July we'll re-enable NTP in batches. System clocks will
> move forward by a second once NTP is started again,
>

To clarify: By default, system time will move *backwards* one second.

We just talked about this on IRC, so just for other's benefit: With NTP's
-x option we should be able to smear the adjustment (by slowing down the
system clock temporarily) until the leap second is incorporated into the
system time. This avoids non-monotonicity, which is important for systems
that use time to capture causality. It would be great to apply the
adjustment to all nodes of the cassandra cluster at once, so that their
clocks are being slewed in lock-step.

Gabriel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l