This seems to be a good opportunity to dig a bit deeper into certain operational aspects of NTP. Some things to be aware of:
How NTP Operates [1] It can take several minutes before ntpd will update the system time for the first time, while all processes are already started. This can be especially problematic for newly provisioned systems. You can speed up the time sync update by using the iburst keyword with the server configuration command [4]. It took 2-3 minutes on my testing VM without iburst to correct the time after startup. Definitely too long, as Cassandra would already have joined the cluster. With iburst it was only a few seconds. Adjustments will be done by either by stepping or slewing the clock. This can happen forward and backwards(!). Stepping will set the corrected value right away. Slewing will make adjustments in small increments of at most 0.5ms/s by speeding the clock up or slowing it down. It will take at least 2000 seconds to adjust the clock by slewing 1 second. * Time offsets < 128ms (default) will be slewed * Offsets > 128ms will be stepped unless -x flag is used. The threshold value can be changed with a tinker option. * Offsets > 1000ms will cause ntpd to fail and expect the administrator to fix the issue (potential hardware error) unless the -g flag is used. I think it’s fair to say that the –g options should be always set. I’m not fully sure about –x yet. Stepping the clock backwards is not a good idea of course. The best solution is probably to have –x set and create alerts on higher clock skews, that will prompt ops to resolve the situation manually. Leap second awareness Make sure your server is leap second aware in advance. You do not want to have the second corrected as part of a normal discrepancy detection process. Instead ntpd should be aware of the leap second in advance, so it can precisely schedule the adjustment. There're two ways to make your ntpd instance aware of the upcoming leap second. This may happen though the upstream ntp server, which may propagate the leap second one day in advance. But this doesn't have to be the case. You need to find out if the server pool is configured correctly for this. Another way to make your ntpd leap second aware is to use a custom file [2]. I had to modify the apparmor profile to make this work [3]. [1] http://doc.ntp.org/4.1.0/ntpd.htm [2] http://support.ntp.org/bin/view/Support/ConfiguringNTP#Section_6.14. [3] http://askubuntu.com/questions/571839/leapseconds-file-permission-denied [4] http://doc.ntp.org/4.1.0/confopt.htm Von: cass savy [mailto:casss...@gmail.com] Gesendet: Freitag, 15. Mai 2015 19:25 An: user@cassandra.apache.org Betreff: Leap sec Just curious to know on how you are preparing Prod C* clusters for leap sec. What are the workaorund other than upgrading kernel to 3.4+? Are you upgrading clusters to Java 7 or higher on client and C* servers?
