Re: [Linuxptp-users] gPTP issues with ntpd
On Thu, Jul 21, 2022 at 02:13:29PM +0200, Jakub Raczyński wrote: > My interpretation is based only on ntpshmmon tool - it clearly shows that > ntpshm servo is only fed when PTP portState is set to Slave. So it is used > only to write to the segment, but phc2sys clearly controls when it is done. > Since ntpd does not communicate with phc2sys then whether it works or not is > only based on phc2sys. You said something about ntpd reset. I'm not sure what that is. If there was something removing the segment (e.g. a script using ipcrm) before phc2sys reverses the direction, it would create a new SHM, which wouldn't be monitored by ntpshmmon started earlier. > It seems you have great feature of phc2sys that allows to synchronize ntpshm > servo in Slave state and use CLOCK_REALTIME as Master, yet you say like it > does not exist. And again, I intend to use it that way. I am not trying to > read anything from ntpshm. ntpshm doesn't synchronize anything. It just provides data for another process to synchronize the clock. ntpd can synchronize CLOCK_REALTIME, but not a PHC, and even if it could, it wouldn't know when it should switch. You need two phc2sys instances. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
> 21.07.2022 13:06 Miroslav Lichvar wrote: > > > On Thu, Jul 21, 2022 at 12:41:54PM +0200, Jakub Raczyński wrote: > > > 21.07.2022 10:20 Miroslav Lichvar wrote: > > > No, that's not correct. Try running the ntpshmmon tool from gpsd to > > > see that phc2sys is writing new samples in both directions. > > > > You are mistaken, I did try ntpshmmon and it does write to ntpshm only in > > one direction as Slave. > > Well, that means there is something wrong with your setup. > > > As such, since data is not written to ntpshm, I assume that phc2sys does > > select direction incorrectly after ntpd reset when significant offset was > > present. Seems like servo reset issue that does not update direction or > > sets it incorrectly. I will probably debug it in following days but I had > > hoped you could be of assistance. > > Your interpretation is wrong. > > If you looked at the code, you would see that the ntpshm servo is only > writing to the segment. It doesn't know if there is anything reading > that data, or what happens with it. ntpd cannot communicate with > phc2sys over the SHM. I fail to understand that, what could be wrong? I have one-to-one device connection where i only change priority1 of devices to select which should be Slave or Master. My interpretation is based only on ntpshmmon tool - it clearly shows that ntpshm servo is only fed when PTP portState is set to Slave. So it is used only to write to the segment, but phc2sys clearly controls when it is done. Since ntpd does not communicate with phc2sys then whether it works or not is only based on phc2sys. It seems you have great feature of phc2sys that allows to synchronize ntpshm servo in Slave state and use CLOCK_REALTIME as Master, yet you say like it does not exist. And again, I intend to use it that way. I am not trying to read anything from ntpshm. Anyway, as said, I will look into it soon. Best regards Jakub Raczynski ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
On Thu, Jul 21, 2022 at 12:41:54PM +0200, Jakub Raczyński wrote: > > 21.07.2022 10:20 Miroslav Lichvar wrote: > > No, that's not correct. Try running the ntpshmmon tool from gpsd to > > see that phc2sys is writing new samples in both directions. > > You are mistaken, I did try ntpshmmon and it does write to ntpshm only in one > direction as Slave. Well, that means there is something wrong with your setup. > As such, since data is not written to ntpshm, I assume that phc2sys does > select direction incorrectly after ntpd reset when significant offset was > present. Seems like servo reset issue that does not update direction or sets > it incorrectly. I will probably debug it in following days but I had hoped > you could be of assistance. Your interpretation is wrong. If you looked at the code, you would see that the ntpshm servo is only writing to the segment. It doesn't know if there is anything reading that data, or what happens with it. ntpd cannot communicate with phc2sys over the SHM. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
> 21.07.2022 10:20 Miroslav Lichvar wrote: > > > On Wed, Jul 20, 2022 at 05:57:57PM +0200, Jakub Raczyński wrote: > > So this setup seems to be correct and from the phc2sys log I sent > > previously it seems to be. So it seems that phc2sys is correctly writing > > timestamps to ntpshm only when it is Slave. > > No, that's not correct. Try running the ntpshmmon tool from gpsd to > see that phc2sys is writing new samples in both directions. You are mistaken, I did try ntpshmmon and it does write to ntpshm only in one direction as Slave. When device switches to Master it stops writing to ntpshm. But from your reaction I assume this is unintended? Because it would work perfectly if not for the ntpd reset bug. > > > The only problem is caused by ntpd reset - it starts synchronizing to PTP > > (ntpshm) even if it shouldn't as is it Master. > > If you have multiple sources configured for ntpd, it could be > rejecting the SHM as a falseticker when the TAI-UTC correction in > phc2sys flips. On a restart/reset and depending on the polling > intervals, it could temporarily select the SHM source for > synchronization and later reject it again. > > Try it with SHM as the only source. That should make it obvious > that this cannot work. I did and yet again it works and switches direction of synchronization perfectly. Yet again, goal is to achieve ntpshm as clock source in Slave state and CLOCK_REALTIME as Master clock for the network. I do not expect ntpshm to be two-way synchronization clock. As such, since data is not written to ntpshm, I assume that phc2sys does select direction incorrectly after ntpd reset when significant offset was present. Seems like servo reset issue that does not update direction or sets it incorrectly. I will probably debug it in following days but I had hoped you could be of assistance. Best regards Jakub Raczynski ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
On Wed, Jul 20, 2022 at 05:57:57PM +0200, Jakub Raczyński wrote: > So this setup seems to be correct and from the phc2sys log I sent previously > it seems to be. So it seems that phc2sys is correctly writing timestamps to > ntpshm only when it is Slave. No, that's not correct. Try running the ntpshmmon tool from gpsd to see that phc2sys is writing new samples in both directions. > The only problem is caused by ntpd reset - it starts synchronizing to PTP > (ntpshm) even if it shouldn't as is it Master. If you have multiple sources configured for ntpd, it could be rejecting the SHM as a falseticker when the TAI-UTC correction in phc2sys flips. On a restart/reset and depending on the polling intervals, it could temporarily select the SHM source for synchronization and later reject it again. Try it with SHM as the only source. That should make it obvious that this cannot work. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
wrote: > > > > 20.07.2022 15:23 Miroslav Lichvar write: > > > > > > On Wed, Jul 20, 2022 at 03:12:39PM +0200, Jakub Raczyński wrote: > > > phc2sys[2031.823]: lan1 sys offset 4870 s0 freq +0 delay 1875 > > > phc2sys[2032.824]: lan1 sys offset 4915 s0 freq +0 delay 1875 > > > phc2sys[2033.824]: lan1 sys offset 5011 s0 freq +0 delay 1750 > > > phc2sys[2034.825]: port 360712.fffe.52efd6-1 changed state > > > phc2sys[2034.826]: reconfiguring after port state change > > > phc2sys[2034.828]: master clock not ready, waiting... > > > phc2sys[2035.828]: port 360712.fffe.52efd6-1 changed state > > > phc2sys[2035.830]: reconfiguring after port state change > > > phc2sys[2035.831]: selecting CLOCK_REALTIME for synchronization > > > phc2sys[2035.832]: selecting lan1 as the master clock > > > phc2sys[2035.833]: CLOCK_REALTIME phc offset -5136 s0 freq +0 > > > delay 1875 > > > phc2sys[2036.834]: CLOCK_REALTIME phc offset -4435 s0 freq +0 > > > delay 1875 > > > > > > So, as mentioned "That is not expected to work" but kinda did or seemed > > > like it. Would need more research and debugging what is actually > > > happening inside both servos. > > > > There is only one SHM segment and it's used by ntpd for > > synchronization of the system clock. When phc2sys reverses the > > direction, the offset will flip the sign, intending to synchronize the > > PHC to the system clock, but ntpd will still be using the data it > > receives to synchronize the system clock, which will cause a positive > > feedback loop and the clock will be steered away, stepped, or ntpd > > will give up depending on its configuration. > > > > -- > > Miroslav Lichvar > > Actually it didn't look like it - ntpd was giving up data from ntpshm > instantly after switching to Master and started using ntpshm data right after > switching to Slave. But as said, no idea what was going inside the servo. > > But anyway, I will stick to the proposed workaround. Thanks again. > > Jakub Raczynski > I want to apologize for many emails, but I do not understand and expressed myself poorly. When using phc2sys with: /usr/sbin/phc2sys -a -r -r -f /etc/ptp4l.cfg -E ntpshm -M 4 it mostly seems to work as I would expect - ntpd is not using data from PTP (ntpshm) when it is in Master state and us using data from ntpshm when Slave. The '-a -r -r' flag selects ntpshm as Slave servo only and CLOCK_REALTIME for Master. So this setup seems to be correct and from the phc2sys log I sent previously it seems to be. So it seems that phc2sys is correctly writing timestamps to ntpshm only when it is Slave. The only problem is caused by ntpd reset - it starts synchronizing to PTP (ntpshm) even if it shouldn't as is it Master. I have yet to debug whether it is phc2sys that does not set some flag for ntpd or if it is ntpd that clears some flag it probably shouldn't. But it seems like that phc2sys stays in mentioned "zombie servo" mode as it is synchronizing ntpshm to itself, even after killing ptp4l. I am merely asking for advice why is it issue. Why is ntpshm being fed timestamps even if it is in Master state? Is it not directly controlled by phc2sys? This should not require any workaround (although it would work, but seems a little excessive). Best regards Jakub Raczynski ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
> 20.07.2022 15:23 Miroslav Lichvar write: > > > On Wed, Jul 20, 2022 at 03:12:39PM +0200, Jakub Raczyński wrote: > > phc2sys[2031.823]: lan1 sys offset 4870 s0 freq +0 delay 1875 > > phc2sys[2032.824]: lan1 sys offset 4915 s0 freq +0 delay 1875 > > phc2sys[2033.824]: lan1 sys offset 5011 s0 freq +0 delay 1750 > > phc2sys[2034.825]: port 360712.fffe.52efd6-1 changed state > > phc2sys[2034.826]: reconfiguring after port state change > > phc2sys[2034.828]: master clock not ready, waiting... > > phc2sys[2035.828]: port 360712.fffe.52efd6-1 changed state > > phc2sys[2035.830]: reconfiguring after port state change > > phc2sys[2035.831]: selecting CLOCK_REALTIME for synchronization > > phc2sys[2035.832]: selecting lan1 as the master clock > > phc2sys[2035.833]: CLOCK_REALTIME phc offset -5136 s0 freq +0 > > delay 1875 > > phc2sys[2036.834]: CLOCK_REALTIME phc offset -4435 s0 freq +0 > > delay 1875 > > > > So, as mentioned "That is not expected to work" but kinda did or seemed > > like it. Would need more research and debugging what is actually happening > > inside both servos. > > There is only one SHM segment and it's used by ntpd for > synchronization of the system clock. When phc2sys reverses the > direction, the offset will flip the sign, intending to synchronize the > PHC to the system clock, but ntpd will still be using the data it > receives to synchronize the system clock, which will cause a positive > feedback loop and the clock will be steered away, stepped, or ntpd > will give up depending on its configuration. > > -- > Miroslav Lichvar Actually it didn't look like it - ntpd was giving up data from ntpshm instantly after switching to Master and started using ntpshm data right after switching to Slave. But as said, no idea what was going inside the servo. But anyway, I will stick to the proposed workaround. Thanks again. Jakub Raczynski ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
On Wed, Jul 20, 2022 at 03:12:39PM +0200, Jakub Raczyński wrote: > phc2sys[2031.823]: lan1 sys offset 4870 s0 freq +0 delay 1875 > phc2sys[2032.824]: lan1 sys offset 4915 s0 freq +0 delay 1875 > phc2sys[2033.824]: lan1 sys offset 5011 s0 freq +0 delay 1750 > phc2sys[2034.825]: port 360712.fffe.52efd6-1 changed state > phc2sys[2034.826]: reconfiguring after port state change > phc2sys[2034.828]: master clock not ready, waiting... > phc2sys[2035.828]: port 360712.fffe.52efd6-1 changed state > phc2sys[2035.830]: reconfiguring after port state change > phc2sys[2035.831]: selecting CLOCK_REALTIME for synchronization > phc2sys[2035.832]: selecting lan1 as the master clock > phc2sys[2035.833]: CLOCK_REALTIME phc offset -5136 s0 freq +0 delay > 1875 > phc2sys[2036.834]: CLOCK_REALTIME phc offset -4435 s0 freq +0 delay > 1875 > > So, as mentioned "That is not expected to work" but kinda did or seemed like > it. Would need more research and debugging what is actually happening inside > both servos. There is only one SHM segment and it's used by ntpd for synchronization of the system clock. When phc2sys reverses the direction, the offset will flip the sign, intending to synchronize the PHC to the system clock, but ntpd will still be using the data it receives to synchronize the system clock, which will cause a positive feedback loop and the clock will be steered away, stepped, or ntpd will give up depending on its configuration. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
wrote: > > > > 20.07.2022 13:33 Miroslav Lichvar wrote: > > > > > > On Wed, Jul 20, 2022 at 12:06:43PM +0200, Jakub Raczyński wrote: > > > I was trying to setup gPTP using linuxptp (ptp4l + phc2sys) that would > > > allow two way synchronization using ntpd. Setup without ntpd > > > (synchronizing CLOCK_REALTIME) seems to be working perfectly. > > > However with ntpd is that when in network there is no external gPTP > > > Master available and device does become Master itself it may synchronize > > > itself to its own shm memory. > > > > > > > > > phc2sys is run using: /usr/sbin/phc2sys -a -r -r -f /etc/ptp4l.cfg -E > > > ntpshm -M 4 > > > > That is not expected to work. phc2sys has only one servo and ntpshm > > can be used only in one direction. > > > > You would need a second phc2sys instance with an option to only synchronize > > the PHC, when the port is in master state. > > > > As a workaround, you could write a script that would monitor the state > > and start/stop phc2sys as needed. > > Best option would be if phc2sys could exist as Master-only (only '-r' flag > without '-a -r'). Such case could probably only useful for ntpshm servo that > can be used in Time Servers that use different sources of synchronization > (like GPS + PTP in our case). But as you said, right now only option seems to > be such workaround. > > Maybe such option would be worthy TODO? > Although, I would like to show one thing I accomplished with mentioned setup (phc2sys configuration). Other than that bug it seemed to have worked, although i have no idea what was really going inside the servo to judge if this was bug or intended. Example: - Switching from Slave to Master: phc2sys[1968.791]: CLOCK_REALTIME phc offset 644 s0 freq +0 delay 1750 phc2sys[1969.791]: CLOCK_REALTIME phc offset 512 s0 freq +0 delay 1875 phc2sys[1970.792]: port 360712.fffe.52efd6-1 changed state phc2sys[1970.793]: reconfiguring after port state change phc2sys[1970.794]: selecting lan1 for synchronization phc2sys[1970.794]: selecting CLOCK_REALTIME as the master clock phc2sys[1970.794]: lan1 sys offset -433 s0 freq +0 delay 1875 phc2sys[1971.795]: lan1 sys offset -364 s0 freq +0 delay 1875 phc2sys[1972.795]: lan1 sys offset -231 s0 freq +0 delay 1875 phc2sys[1973.796]: lan1 sys offset -205 s0 freq +0 delay 1875 - Switching from Slave to Master phc2sys[2031.823]: lan1 sys offset 4870 s0 freq +0 delay 1875 phc2sys[2032.824]: lan1 sys offset 4915 s0 freq +0 delay 1875 phc2sys[2033.824]: lan1 sys offset 5011 s0 freq +0 delay 1750 phc2sys[2034.825]: port 360712.fffe.52efd6-1 changed state phc2sys[2034.826]: reconfiguring after port state change phc2sys[2034.828]: master clock not ready, waiting... phc2sys[2035.828]: port 360712.fffe.52efd6-1 changed state phc2sys[2035.830]: reconfiguring after port state change phc2sys[2035.831]: selecting CLOCK_REALTIME for synchronization phc2sys[2035.832]: selecting lan1 as the master clock phc2sys[2035.833]: CLOCK_REALTIME phc offset -5136 s0 freq +0 delay 1875 phc2sys[2036.834]: CLOCK_REALTIME phc offset -4435 s0 freq +0 delay 1875 So, as mentioned "That is not expected to work" but kinda did or seemed like it. Would need more research and debugging what is actually happening inside both servos. Best regards Jakub Raczynski ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
> 20.07.2022 13:33 Miroslav Lichvar wrote: > > > On Wed, Jul 20, 2022 at 12:06:43PM +0200, Jakub Raczyński wrote: > > I was trying to setup gPTP using linuxptp (ptp4l + phc2sys) that would > > allow two way synchronization using ntpd. Setup without ntpd (synchronizing > > CLOCK_REALTIME) seems to be working perfectly. > > However with ntpd is that when in network there is no external gPTP Master > > available and device does become Master itself it may synchronize itself to > > its own shm memory. > > > > > > phc2sys is run using: /usr/sbin/phc2sys -a -r -r -f /etc/ptp4l.cfg -E > > ntpshm -M 4 > > That is not expected to work. phc2sys has only one servo and ntpshm > can be used only in one direction. > > You would need a second phc2sys instance with an option to only synchronize > the PHC, when the port is in master state. > > As a workaround, you could write a script that would monitor the state > and start/stop phc2sys as needed. Best option would be if phc2sys could exist as Master-only (only '-r' flag without '-a -r'). Such case could probably only useful for ntpshm servo that can be used in Time Servers that use different sources of synchronization (like GPS + PTP in our case). But as you said, right now only option seems to be such workaround. Maybe such option would be worthy TODO? > > While testing devices with gPTP we encountered, in my opinion, quite > > inconsequential behavior. Using different setups, I set following flags and > > had following outcome: > > > > > > gmCapable = 0 , slaveOnly = 0 -> OK > > gmCapable = 1 , slaveOnly = 0 -> OK > > gmCapable = 0 , slaveOnly = 1 -> Cannot mix 1588 slaveOnly with 802.1AS > > !gmCapable > > gmCapable = 1 , slaveOnly = 1 -> OK > > > > > > Frankly, I would expect combination of "gmCapable = 1 , slaveOnly = 1" to > > fail with than "gmCapable = 0 , slaveOnly = 1". > > > > > > > > I would like to ask for reasoning behind that combination block and not > > the other. Since SlaveOnly flag performs as expected, even for gPTP setup. > > I'm not sure. If slaveOnly is 1, the clock will not ever be a > grandmaster, i.e. not try to send sync messages. What would be > different with gmCapable of 0? > 'gmCapable 0 + slaveOnly 0' does send Announce messages but from other Slave devices we can only see "master clock not ready, waiting..." and they are locked into LISTENING state, which is quite different. So we were using 'slaveOnly 1' to not pollute network with excessive devices. Anyway, thanks for the help. Best regards Jakub Raczynski ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] gPTP issues with ntpd
On Wed, Jul 20, 2022 at 12:06:43PM +0200, Jakub Raczyński wrote: > I was trying to setup gPTP using linuxptp (ptp4l + phc2sys) that would allow > two way synchronization using ntpd. Setup without ntpd (synchronizing > CLOCK_REALTIME) seems to be working perfectly. > However with ntpd is that when in network there is no external gPTP Master > available and device does become Master itself it may synchronize itself to > its own shm memory. > > > phc2sys is run using: /usr/sbin/phc2sys -a -r -r -f /etc/ptp4l.cfg -E ntpshm > -M 4 That is not expected to work. phc2sys has only one servo and ntpshm can be used only in one direction. You would need a second phc2sys instance with an option to only synchronize the PHC, when the port is in master state. As a workaround, you could write a script that would monitor the state and start/stop phc2sys as needed. > While testing devices with gPTP we encountered, in my opinion, quite > inconsequential behavior. Using different setups, I set following flags and > had following outcome: > > > gmCapable = 0 , slaveOnly = 0 -> OK > gmCapable = 1 , slaveOnly = 0 -> OK > gmCapable = 0 , slaveOnly = 1 -> Cannot mix 1588 slaveOnly with 802.1AS > !gmCapable > gmCapable = 1 , slaveOnly = 1 -> OK > > > Frankly, I would expect combination of "gmCapable = 1 , slaveOnly = 1" to > fail with than "gmCapable = 0 , slaveOnly = 1". > > > > I would like to ask for reasoning behind that combination block and not the > other. Since SlaveOnly flag performs as expected, even for gPTP setup. I'm not sure. If slaveOnly is 1, the clock will not ever be a grandmaster, i.e. not try to send sync messages. What would be different with gmCapable of 0? -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
