time changed from 2020-07-03 to 2022-05-18
Hello, Had to reboot the box. When it came up it got the wrong date. It looks like ntpd had a role here: (...) Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdc [SAT], not found in smartd database. Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdc [SAT], can't monitor Current_Pending_Sector count - no Attribute 197 Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdc [SAT], can't monitor Offline_Uncorrectable count - no Attribute 198 Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdc [SAT], is SMART capable. Adding to "monitor" list. Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd, type changed from 'scsi' to 'sat' Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd [SAT], opened Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd [SAT], Samsung SSD 860 QVO 2TB, S/N:S4CYNF0M520076M, WWN:5-002538-e40feda39, FW:RVQ11B6Q, 2.00 TB Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd [SAT], not found in smartd database. Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd [SAT], can't monitor Current_Pending_Sector count - no Attribute 197 Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd [SAT], can't monitor Offline_Uncorrectable count - no Attribute 198 Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/sdd [SAT], is SMART capable. Adding to "monitor" list. Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/nvme0, opened Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/nvme0, KINGSTON SKC2000M8250G, S/N:50026B72824168C0, FW:S2780101 Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/nvme0, is SMART capable. Adding to "monitor" list. Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/nvme1, opened Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/nvme1, KINGSTON SKC2000M8250G, S/N:50026B73324169CC, FW:S2780102 Jul 3 10:06:47 boombox smartd[2004]: Device: /dev/nvme1, is SMART capable. Adding to "monitor" list. Jul 3 10:06:47 boombox smartd[2004]: Monitoring 4 ATA/SATA, 0 SCSI/SAS and 2 NVMe devices May 18 10:06:48 boombox ntpd[2055]: CLOCK: time stepped by 59097600.478559 May 18 10:06:48 boombox ntpd[2055]: CLOCK: time changed from 2020-07-03 to 2022-05-18 May 18 10:06:48 boombox ntpd[2055]: INIT: MRU 10922 entries, 13 hash bits, 65536 bytes (cut) We're running a fairly recent git version of ntpsec: ntpsec-1.1.9-0.fc31.x86_64 on Fedora 31 on kernel.org 5.7.7. How can I avoid this from happening again? Please let me know. Kind regards, Udo ___ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
Re: time changed from 2020-07-03 to 2022-05-18
Udo van den Heuvel via devel writes: > May 18 10:06:48 boombox ntpd[2055]: CLOCK: time stepped by 59097600.478559 > May 18 10:06:48 boombox ntpd[2055]: CLOCK: time changed from 2020-07-03 to > 2022-05-18 That's why you don't trust the GPS time until you know it has a good lock and also why you shouldn't start ntpd with the option to accept such large clock steps. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra ___ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
Re: time changed from 2020-07-03 to 2022-05-18
On 03-07-2020 10:34, ASSI via devel wrote: > Udo van den Heuvel via devel writes: >> May 18 10:06:48 boombox ntpd[2055]: CLOCK: time stepped by 59097600.478559 >> May 18 10:06:48 boombox ntpd[2055]: CLOCK: time changed from 2020-07-03 to >> 2022-05-18 > > That's why you don't trust the GPS time until you know it has a good > lock and also why you shouldn't start ntpd with the option to accept > such large clock steps. So we remove `-g` option. Or replace by `-x`? But how could it get 'the wrong impression' in the first place? Kind regards, Udo ___ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
Re: time changed from 2020-07-03 to 2022-05-18
> May 18 10:06:48 boombox ntpd[2055]: CLOCK: time stepped by 59097600.478559 > May 18 10:06:48 boombox ntpd[2055]: CLOCK: time changed from 2020-07-03 to > 2022-05-18 > We're running a fairly recent git version of ntpsec: ntpsec-1.1.9-0.fc31.x86_6 > 4 on Fedora 31 on kernel.org 5.7.7. > How can I avoid this from happening again? That isn't enough info to figure out what happened. Somehow, ntpd thought the time was way off, and you had the -g switch on that allowed it to take big jumps. If you turn off the -g switch, ntpd will exit instead. You will have to start it again by hand. (Or maybe systemd will restart it, but that's another problem.) I'm guessing you have a GPS unit. (Or something similar, but GPS is the most common source of non-network time.) What sort of GPS unit? gpsd? ... Things should be setup such that servers from the network sanity check your GPS clock. What's in your ntp.conf? -- These are my opinions. I hate spam. ___ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
Re: time changed from 2020-07-03 to 2022-05-18
On 03-07-2020 15:00, Hal Murray wrote: >> How can I avoid this from happening again? > > That isn't enough info to figure out what happened. Somehow, ntpd thought > the > time was way off, and you had the -g switch on that allowed it to take big > jumps. If you turn off the -g switch, ntpd will exit instead. You will have > to start it again by hand. (Or maybe systemd will restart it, but that's > another problem.) Done removing -g. > I'm guessing you have a GPS unit. (Or something similar, but GPS is the most > common source of non-network time.) What sort of GPS unit? gpsd? ... Garmin gps18x on the serial port with USB power. With just $GPGGA and $GPRMC enabled, so where did it find that weird date? > Things should be setup such that servers from the network sanity check your > GPS clock. What's in your ntp.conf? driftfile /var/lib/ntp/drift nts key /etc/letsencrypt/keys/_key-certbot.pem nts cert /etc/letsencrypt/csr/_csr-certbot.pem restrict default nomodify nopeer noquery restrict -6 default nomodify nopeer noquery restrict 127.0.0.1 restrict -6 ::1 restrict 192.168.10.0 mask 255.255.255.0 nomodify refclock nmea unit 0 mode 7 flag3 1 flag2 0 flag1 0 time1 0.0006 time2 0.260 baud 4800 refclock pps unit 0 flag2 0 disable monitor server 192.168.10.98 minpoll 4 iburst server fd00:c0a8:a00:1::1 server time.cloudflare.com:1234 nts # TLS1.3 only server ntpmon.dcs1.biz nts server pi4.rellim.com nts server ntp1.glypnod.com nts server ntp2.glypnod.com nts server ntp.xs4all.nl server ntp2.xs4all.nl server ntp0.nl.net server ntp2.nl.net server keetweej.vanheusden.com server ntp.nmi.nl includefile /etc/ntp/crypto/pw keys /etc/ntp/keys statsdir /var/log/ntp/ logconfig =syncall -clockall Kind regards, Udo ___ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
Re: time changed from 2020-07-03 to 2022-05-18
On Fri, Jul 3, 2020, at 6:05 AM Udo van den Heuvel via devel wrote: > > On 03-07-2020 15:00, Hal Murray wrote: > >> How can I avoid this from happening again? > > > > That isn't enough info to figure out what happened. Somehow, ntpd thought > > the > > time was way off, and you had the -g switch on that allowed it to take big > > jumps. If you turn off the -g switch, ntpd will exit instead. You will > > have > > to start it again by hand. (Or maybe systemd will restart it, but that's > > another problem.) > > Done removing -g. > > > I'm guessing you have a GPS unit. (Or something similar, but GPS is the > > most > > common source of non-network time.) What sort of GPS unit? gpsd? ... > > Garmin gps18x on the serial port with USB power. > With just $GPGGA and $GPRMC enabled, so where did it find that weird date? It is too small to be a rollover issue I think. I might also suggest raising tos minclock and minsane to greater values on the premise that they may prevent a single clock from stepping your time so much. maybe 7 and 4 as opposed to 3 and 1 but I am probably wrong. ___ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
Re: time changed from 2020-07-03 to 2022-05-18
Udo van den Heuvel via devel writes: > Garmin gps18x on the serial port with USB power. > With just $GPGGA and $GPRMC enabled, so where did it find that weird date? If you had the clockstats file enabled, you would be able to see what message ntpd received at that point. Since you have RMC enabled, I guess that you got an RMC message into ntpd before it aquired a fix and if you didn't configure anything else, I guess it starts with whatever date the firmware was built. I think your GPS is supposed to store the last known position and time in Flash so this normally should not happen, but either that function isn't enabled or it decided that the information was unusable and did a full cold start. Since that firmware buils certainly was before the recorded build date of ntpd, it (unhelpfully in this case) decides it should pivot that date forward on the assumption you are suffering from GPS rollover issues (no it shouldn't do that unless configured, but that's how the implementation currently works). >> Things should be setup such that servers from the network sanity check your >> GPS clock. What's in your ntp.conf? > > refclock nmea unit 0 mode 7 flag3 1 flag2 0 flag1 0 time1 0.0006 time2 > 0.260 baud 4800 > refclock pps unit 0 flag2 0 This is a wierd config. You disable PPS processing on the NMEA, then enable kernel PPS discipline and also configure a PPS refclock? You should drop the PPS and enable PPS processing in NMEA instead, IMHO. Set up an udev rule to intialize the GPS correctly (preferrably with just the RMC message) and link the serial and PPS to /dev/gps0 and /dev/gpspps0. The Garmin has the RMI message that allows you to set the expected time and date, so using that during the device initialization to set it up with the current syste time & date should alleviate the problem with it starting up with the firmware build date and warping you into the future. Now, to prevent that from happening again: there are actually two systemd targets that deal with setting up the system time. You will want to use time-set.target to bring up the system time to a known good state and then time-sync.target to keep it there, which you ccan take advantage of by using two different configs for ntpd. The first one potentially steps the system clock, so you should not let any uncontrolled GPS time reach ntpd (I can generally rely on there being enough NTP servers on my local network to converge to the correct time) and then just exits. The second one then never needs to step the time since you start out very close, so you can configure the GPS there. If the GPS freaks out it will be marked as a falseticker and falls back to your network instead of warping system time and if ntpd itself freaks out it will be restarted. ntpsec-steptime.service --8<---cut here---start->8--- [Unit] Description=Network Time Service Documentation=man:ntpd(8) Wants=network.target ConditionCapability=CAP_SYS_TIME After=network.target nss-lookup.target Before=time-set.target Conflicts=systemd-timesyncd.service ntpd.service chrony.service ntp-steptime.service [Service] Type=oneshot PrivateTmp=true ExecStart=/usr/sbin/ntpd -q -g -N -u ntp:ntp -c /etc/ntp-step.conf TimeoutStartSec=60 RemainAfterExit=yes StandardOutput=null Restart=no [Install] WantedBy=time-set.target --8<---cut here---end--->8--- ntpsec-ntpd.service --8<---cut here---start->8--- [Unit] Description=Network Time Service Documentation=man:ntpd(8) Wants=network.target time-set.target ConditionCapability=CAP_SYS_TIME After=time-set.target network.target nss-lookup.target Before=time-sync.target Conflicts=systemd-timesyncd.service ntpd.service chrony.service [Service] Type=forking PIDFile=/run/ntp/ntpd.pid PrivateTmp=true ExecStart=/usr/sbin/ntpd -p /run/ntp/ntpd.pid -N -u ntp:ntp Restart=on-failure [Install] WantedBy=multi-user.target --8<---cut here---end--->8--- ntpsec-wait.service --8<---cut here---start->8--- [Unit] Description=Wait for ntpsec-ntpd to synchronize system clock Documentation=man:ntpwait(8) Requisite=ntpsec-ntpd.service Before=time-sync.target After=ntpsec-ntpd.service Conflicts=systemd-timesyncd.service ntp-wait.service ntpd.service chrony-wait.service chrony.service ConditionVirtualization=!container ConditionCapability=CAP_SYS_TIME Wants=time-sync.target [Service] Type=oneshot ExecStart=/usr/bin/ntpwait -v -s 1 -n 300 RemainAfterExit=yes StandardOutput=journal+console [Install] WantedBy=time-sync.target --8<---cut here---end--->8--- If you've set it up correctly these should have registered as follows: /etc/systemd/system/multi-user.target.wants/ntpsec-wait.service /etc/systemd/system/multi-user.target.wants/ntpsec-ntpd.service /etc/systemd/system/multi-user.target.wants/ntpsec-steptime.service /etc/systemd/system/tim
