Re: [Wikitech-l] [Engineering] Upcoming leap second on Tue 30th
Hi Gabriel, We just talked about this on IRC, so just for other's benefit: With NTP's -x option we should be able to smear the adjustment (by slowing down the system clock temporarily) until the leap second is incorporated into the system time. This avoids non-monotonicity, which is important for systems that use time to capture causality. It would be great to apply the adjustment to all nodes of the cassandra cluster at once, so that their clocks are being slewed in lock-step. Unfortunately this was broken in NTP 4.2.6 and only recently discovered: http://bugs.ntp.org/show_bug.cgi?id=2745 (only fixed in current development/pre-releases). Even if we would backport the fixes to our time servers we'd run into problems, since the local time deviation in the normalisation period wouldn't be consistent across the nodes of the Cassandra cluster. (chrony fully supports NTP smearing since the 2.0 release (27th April 2015), but that's also not a solution for the upcoming leap second). We'll follow up in a separate mail how to best accomodate the Cassandra cluster. Cheers, Moritz ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Upcoming leap second on Tue 30th
Alex and Moritz, thank you for taking care of this. These leap seconds are a real pain in the butt for time-based distributed systems, and I'm glad that we have a plan in place. I hope the movement to abolish leap seconds https://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds wins out in the end! On Thu, Jun 25, 2015 at 7:27 AM, Moritz Mühlenhoff mor...@wikimedia.org wrote: * On the 1st of July we'll re-enable NTP in batches. System clocks will move forward by a second once NTP is started again, To clarify: By default, system time will move *backwards* one second. We just talked about this on IRC, so just for other's benefit: With NTP's -x option we should be able to smear the adjustment (by slowing down the system clock temporarily) until the leap second is incorporated into the system time. This avoids non-monotonicity, which is important for systems that use time to capture causality. It would be great to apply the adjustment to all nodes of the cassandra cluster at once, so that their clocks are being slewed in lock-step. Gabriel ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Upcoming leap second on Tue 30th
All staff members, in preparation for leap second 2015, should watch a brief training on the other problems with time timezones[1]. [1] https://www.youtube.com/watch?v=-5wpm-gesOY On Thu, Jun 25, 2015 at 12:05 PM Gabriel Wicke gwi...@wikimedia.org wrote: Alex and Moritz, thank you for taking care of this. These leap seconds are a real pain in the butt for time-based distributed systems, and I'm glad that we have a plan in place. I hope the movement to abolish leap seconds https://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds wins out in the end! On Thu, Jun 25, 2015 at 7:27 AM, Moritz Mühlenhoff mor...@wikimedia.org wrote: * On the 1st of July we'll re-enable NTP in batches. System clocks will move forward by a second once NTP is started again, To clarify: By default, system time will move *backwards* one second. We just talked about this on IRC, so just for other's benefit: With NTP's -x option we should be able to smear the adjustment (by slowing down the system clock temporarily) until the leap second is incorporated into the system time. This avoids non-monotonicity, which is important for systems that use time to capture causality. It would be great to apply the adjustment to all nodes of the cassandra cluster at once, so that their clocks are being slewed in lock-step. Gabriel ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
