Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hi, I've tried three different ConnectX-3 NICs now and they all behave the same. To rule out any issues with the GM I tried a Intel i210 as well and that is spot on with excellent sync. However, the Mellanox is not. It's almost as if there is a frequency correction happening inside the NIC every so often. Then the rms values are bad out of a sudden, getting better as frequency is adjusted to the GM and then, again, really bad. ptp4l[447.094]: rms 1580 max 6078 freq +1598846 +/- 4029 delay 983 +/- 3 ptp4l[448.091]: rms 1608 max 6080 freq +1598430 +/- 4117 delay 971 +/- 2 ptp4l[449.089]: rms 111 max 244 freq +1598218 +/- 180 delay 977 +/- 2 ptp4l[450.086]: rms 25 max 39 freq +1598474 +/- 23 delay 983 +/- 2 ptp4l[451.084]: rms 30 max 57 freq +1598512 +/- 23 delay 984 +/- 1 ptp4l[452.081]: rms 17 max 28 freq +1598504 +/- 14 delay 984 +/- 1 ptp4l[453.078]: rms 13 max 24 freq +1598507 +/- 15 delay 985 +/- 2 ptp4l[454.076]: rms 14 max 43 freq +1598510 +/- 26 delay 984 +/- 1 ptp4l[455.073]: rms7 max 16 freq +1598506 +/- 12 delay 985 +/- 1 ptp4l[456.071]: rms 11 max 18 freq +1598525 +/- 13 delay 983 +/- 1 ptp4l[457.068]: rms7 max 15 freq +1598519 +/- 14 delay 983 +/- 1 ptp4l[458.066]: rms6 max 14 freq +1598514 +/- 14 delay 984 +/- 1 ptp4l[459.063]: rms6 max 13 freq +1598519 +/- 14 delay 985 +/- 0 ptp4l[460.061]: rms 1563 max 5997 freq +1598869 +/- 3991 delay 982 +/- 3 Thanks Andre On 20/11/23 10:27, Andre Puschmann wrote: Hey, > How the GM side is configured? Are you writing system time to PHC > every second? If so, you can try make the phc free run. Without 1PPS > signal connecting to the phc or PTM enabled, it's not recommended to > set pmc's time by software, the jitter is quite big. I am not writing any time to the PHC. I just start ptp4l. Shouldn't that be enough to adjust to PHC to the GMs? > Is the GM and the client connected directly or through a switch? Try > connect them directly with an utp or fiber. The GM is directly connected to port0 of the NIC. And the GM is GPS synced. > Try the L2 transport. IIRC at least some Mellanox NICs performed > worse with UDP transport for some reason. This is already with L2. Meanwhile I tried with yet another ConnectX-3 card. This time a IBM branded with FW 2.42.5032 but the results are similar. One new thing I've observed is this: ptp4l[226.813]: rms 69 max 164 freq +1593151 +/- 131 delay 972 +/- 3 ptp4l[227.811]: rms 31 max 40 freq +1593342 +/- 23 delay 977 +/- 1 ptp4l[228.810]: rms 1607 max 6189 freq +1593762 +/- 4095 delay 979 +/- 2 ptp4l[229.816]: rms 23001617550970840 max 65058399023124016 freq -11106166 +/- 33598711 delay 970 +/- 4 ptp4l[230.695]: clockcheck: clock jumped backward or running slower than expected! ptp4l[230.695]: port 1 (enp1s0): SLAVE to UNCALIBRATED on SYNCHRONIZATION_FAULT ptp4l[230.821]: rms 65058399528662992 max 65058399969385488 freq -1 +/- 0 delay 9404287 +/- 6628445 ptp4l[231.825]: rms 65058400466045568 max 65058400899446072 freq -1 +/- 0 delay 15392460 +/- 2718826 ptp4l[232.828]: rms 65058401394850600 max 65058401833399432 freq -1 +/- 0 delay 12181777 +/- 2190097 ptp4l[233.831]: rms 65058402330812608 max 65058402771727544 freq -1 +/- 0 delay 16575665 +/- 1701941 RMS values were like before, but than suddenly increased and now don't go back. Thanks Andre On 19/11/23 22:07, Andre Puschmann wrote: Hey, I've been able to get my hands on a ConnectX-3 Pro card and have done some initial testing. The card indeed has a shared PHC for both ports so running ptp4l as BC or TC does indeed work without the jbod option. However, sync performance (i.e. rms values) for the downstream OCs isn't great. And in fact, even the Mellanox as a OC isn't giving great results - rms values jump a lot (and I've tried various PI value combinations). Is anyone else seeing this with Mlx cards as well? Could it be my model or the firmware? Here is the output of a OC config with the card: $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6 ptp4l[12737.960]: selected /dev/ptp0 as PTP clock ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1 ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447 ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4 ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay 951 +/- 14 ptp4l[12742.139]: rms 129 max 17
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hey, > How the GM side is configured? Are you writing system time to PHC > every second? If so, you can try make the phc free run. Without 1PPS > signal connecting to the phc or PTM enabled, it's not recommended to > set pmc's time by software, the jitter is quite big. I am not writing any time to the PHC. I just start ptp4l. Shouldn't that be enough to adjust to PHC to the GMs? > Is the GM and the client connected directly or through a switch? Try > connect them directly with an utp or fiber. The GM is directly connected to port0 of the NIC. And the GM is GPS synced. > Try the L2 transport. IIRC at least some Mellanox NICs performed > worse with UDP transport for some reason. This is already with L2. Meanwhile I tried with yet another ConnectX-3 card. This time a IBM branded with FW 2.42.5032 but the results are similar. One new thing I've observed is this: ptp4l[226.813]: rms 69 max 164 freq +1593151 +/- 131 delay 972 +/- 3 ptp4l[227.811]: rms 31 max 40 freq +1593342 +/- 23 delay 977 +/- 1 ptp4l[228.810]: rms 1607 max 6189 freq +1593762 +/- 4095 delay 979 +/- 2 ptp4l[229.816]: rms 23001617550970840 max 65058399023124016 freq -11106166 +/- 33598711 delay 970 +/- 4 ptp4l[230.695]: clockcheck: clock jumped backward or running slower than expected! ptp4l[230.695]: port 1 (enp1s0): SLAVE to UNCALIBRATED on SYNCHRONIZATION_FAULT ptp4l[230.821]: rms 65058399528662992 max 65058399969385488 freq -1 +/- 0 delay 9404287 +/- 6628445 ptp4l[231.825]: rms 65058400466045568 max 65058400899446072 freq -1 +/- 0 delay 15392460 +/- 2718826 ptp4l[232.828]: rms 65058401394850600 max 65058401833399432 freq -1 +/- 0 delay 12181777 +/- 2190097 ptp4l[233.831]: rms 65058402330812608 max 65058402771727544 freq -1 +/- 0 delay 16575665 +/- 1701941 RMS values were like before, but than suddenly increased and now don't go back. Thanks Andre On 19/11/23 22:07, Andre Puschmann wrote: Hey, I've been able to get my hands on a ConnectX-3 Pro card and have done some initial testing. The card indeed has a shared PHC for both ports so running ptp4l as BC or TC does indeed work without the jbod option. However, sync performance (i.e. rms values) for the downstream OCs isn't great. And in fact, even the Mellanox as a OC isn't giving great results - rms values jump a lot (and I've tried various PI value combinations). Is anyone else seeing this with Mlx cards as well? Could it be my model or the firmware? Here is the output of a OC config with the card: $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6 ptp4l[12737.960]: selected /dev/ptp0 as PTP clock ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1 ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447 ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4 ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay 951 +/- 14 ptp4l[12742.139]: rms 129 max 179 freq +268843 +/- 296 delay 963 +/- 11 ptp4l[12743.140]: rms 241 max 490 freq +268455 +/- 452 delay 948 +/- 1 ptp4l[12744.141]: rms 135 max 180 freq +268381 +/- 25 delay 947 +/- 1 ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay 950 +/- 1 ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay 935 +/- 7 ptp4l[12747.144]: rms 210 max 417 freq +268048 +/- 243 delay 942 +/- 3 ptp4l[12748.145]: rms 15 max 32 freq +268415 +/- 29 delay 947 +/- 2 ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay 950 +/- 1 ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay 942 +/- 4 Thanks Andre On 2/11/23 17:37, Jacob Keller wrote: On 11/2/2023 4:15 AM, Andre Puschmann wrote: Hi, On 2/11/23 4:11, James Clark wrote: I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), which has a shared PHC. You can get them for less than $50 on eBay/AliExpress. I had to upgrade the firmware on mine to get PTP support. I haven't yet tried it as a boundary clock. Excellent. This is very helpful James. I've ordered a MCX312A and B and will compare both here. I'll share my results here soon. If you have a chance please also share the firmware version you're currently using on your NIC. With my Intel NIC I could get the BC config working but I needed to set the twoStepFlag to 1. Otherwise I was getting this for both ports: ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range Yep, that would indicate the device doesn't support one-step
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hi, this morning I tried with the Mellanox OFED drivers v4.9-7.1.0 on Ubuntu 20.04 LTS. This was the last version I could get the Mellanox drivers compiled. Result, however, is the same: ptp4l[283.357]: selected /dev/ptp0 as PTP clock ptp4l[283.408]: port 1 (enp1s0): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[283.408]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[283.408]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[283.505]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1 ptp4l[283.758]: selected best master clock fcaf6a.fffe.02b447 ptp4l[283.758]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[284.536]: port 1 (enp1s0): minimum delay request interval 2^-4 ptp4l[284.946]: port 1 (enp1s0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[285.508]: rms 23255986572 max 37976909340 freq +182135 +/- 122816 delay -1439 +/- 1952 ptp4l[286.506]: rms 1677 max 6157 freq +267356 +/- 4283 delay 892 +/- 20 ptp4l[287.504]: rms 1590 max 6255 freq +268556 +/- 4011 delay 922 +/- 6 ptp4l[288.502]: rms 161 max 228 freq +268326 +/- 308 delay 936 +/- 7 ptp4l[289.500]: rms 208 max 231 freq +268746 +/- 29 delay 947 +/- 2 ptp4l[290.498]: rms 1570 max 6114 freq +268897 +/- 3989 delay 940 +/- 10 ptp4l[291.496]: rms 53 max 68 freq +268598 +/- 80 delay 944 +/- 3 ptp4l[292.502]: rms 1600 max 6172 freq +269013 +/- 4074 delay 947 +/- 4 ptp4l[293.507]: rms 99 max 235 freq +268431 +/- 173 delay 940 +/- 3 ptp4l[294.512]: rms 33 max 50 freq +268684 +/- 20 delay 946 +/- 2 Btw I am using a HP-branded MCX312B with FW version 2.42.5044 Thanks Andre On 19/11/23 22:07, Andre Puschmann wrote: Hey, I've been able to get my hands on a ConnectX-3 Pro card and have done some initial testing. The card indeed has a shared PHC for both ports so running ptp4l as BC or TC does indeed work without the jbod option. However, sync performance (i.e. rms values) for the downstream OCs isn't great. And in fact, even the Mellanox as a OC isn't giving great results - rms values jump a lot (and I've tried various PI value combinations). Is anyone else seeing this with Mlx cards as well? Could it be my model or the firmware? Here is the output of a OC config with the card: $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6 ptp4l[12737.960]: selected /dev/ptp0 as PTP clock ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1 ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447 ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4 ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay 951 +/- 14 ptp4l[12742.139]: rms 129 max 179 freq +268843 +/- 296 delay 963 +/- 11 ptp4l[12743.140]: rms 241 max 490 freq +268455 +/- 452 delay 948 +/- 1 ptp4l[12744.141]: rms 135 max 180 freq +268381 +/- 25 delay 947 +/- 1 ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay 950 +/- 1 ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay 935 +/- 7 ptp4l[12747.144]: rms 210 max 417 freq +268048 +/- 243 delay 942 +/- 3 ptp4l[12748.145]: rms 15 max 32 freq +268415 +/- 29 delay 947 +/- 2 ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay 950 +/- 1 ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay 942 +/- 4 Thanks Andre On 2/11/23 17:37, Jacob Keller wrote: On 11/2/2023 4:15 AM, Andre Puschmann wrote: Hi, On 2/11/23 4:11, James Clark wrote: I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), which has a shared PHC. You can get them for less than $50 on eBay/AliExpress. I had to upgrade the firmware on mine to get PTP support. I haven't yet tried it as a boundary clock. Excellent. This is very helpful James. I've ordered a MCX312A and B and will compare both here. I'll share my results here soon. If you have a chance please also share the firmware version you're currently using on your NIC. With my Intel NIC I could get the BC config working but I needed to set the twoStepFlag to 1. Otherwise I was getting this for both ports: ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range Yep, that would indicate the device doesn't support one-step mode. Sync quality wasn't great as expected though. I'll repeat with the Mellanox once I have them here. Thanks Andre For Intel NICs, the only products I am aware of which share PHC across the device are the E800 series devices. Pri
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hi, How the GM side is configured? Are you writing system time to PHC every second? If so, you can try make the phc free run. Without 1PPS signal connecting to the phc or PTM enabled, it's not recommended to set pmc's time by software, the jitter is quite big. Is the GM and the client connected directly or through a switch? Try connect them directly with an utp or fiber. Andre Puschmann 于2023年11月20日周一 05:21写道： > > Hey, > > I've been able to get my hands on a ConnectX-3 Pro card and have done > some initial testing. The card indeed has a shared PHC for both ports so > running ptp4l as BC or TC does indeed work without the jbod option. > > However, sync performance (i.e. rms values) for the downstream OCs isn't > great. And in fact, even the Mellanox as a OC isn't giving great results > - rms values jump a lot (and I've tried various PI value combinations). > > Is anyone else seeing this with Mlx cards as well? Could it be my model > or the firmware? > > Here is the output of a OC config with the card: > > $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6 > ptp4l[12737.960]: selected /dev/ptp0 as PTP clock > ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on > INIT_COMPLETE > ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on > INIT_COMPLETE > ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING > on INIT_COMPLETE > ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1 > ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447 > ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE > ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4 > ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on > MASTER_CLOCK_SELECTED > ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay 951 > +/- 14 > ptp4l[12742.139]: rms 129 max 179 freq +268843 +/- 296 delay 963 +/- 11 > ptp4l[12743.140]: rms 241 max 490 freq +268455 +/- 452 delay 948 +/- 1 > ptp4l[12744.141]: rms 135 max 180 freq +268381 +/- 25 delay 947 +/- 1 > ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay 950 > +/- 1 > ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay 935 > +/- 7 > ptp4l[12747.144]: rms 210 max 417 freq +268048 +/- 243 delay 942 +/- 3 > ptp4l[12748.145]: rms 15 max 32 freq +268415 +/- 29 delay 947 +/- 2 > ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay 950 > +/- 1 > ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay 942 > +/- 4 > > > Thanks > Andre > > > > > > On 2/11/23 17:37, Jacob Keller wrote: > > > > > > On 11/2/2023 4:15 AM, Andre Puschmann wrote: > >> Hi, > >> > >> On 2/11/23 4:11, James Clark wrote: > >>> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), > >>> which has a shared PHC. You can get them for less than $50 on > >>> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP > >>> support. I haven't yet tried it as a boundary clock. > >> > >> Excellent. This is very helpful James. I've ordered a MCX312A and B and > >> will compare both here. I'll share my results here soon. If you have a > >> chance please also share the firmware version you're currently using on > >> your NIC. > >> > >> With my Intel NIC I could get the BC config working but I needed to set > >> the twoStepFlag to 1. Otherwise I was getting this for both ports: > >> > >> ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range > >> > > > > Yep, that would indicate the device doesn't support one-step mode. > > > >> Sync quality wasn't great as expected though. I'll repeat with the > >> Mellanox once I have them here. > >> > >> Thanks > >> Andre > >> > > > > For Intel NICs, the only products I am aware of which share PHC across > > the device are the E800 series devices. Prior devices (E500, and E700, > > as well as the gigabit products) do share the same internal oscillator > > but due to the register interface each function has to setup its own clock. > > > > Thanks, > > Jake > > > > > > ___ > > Linuxptp-users mailing list > > Linuxptp-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/linuxptp-users > > -- > Andre Puschmann > > Software Radio Systems (SRS) > https://www.srs.io > an...@srs.io > > PGP/GnuPG key: 0x204A85DFEA324D58 > fingerprint: 3924 1C60 D52E 81A2 1F2E 0C9D 204A 85DF EA32 4D58 > > > > ___ > Linuxptp-users mailing list > Linuxptp-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linuxptp-users ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On Sun, Nov 19, 2023 at 10:07:50PM +0100, Andre Puschmann wrote: > Hey, > > I've been able to get my hands on a ConnectX-3 Pro card and have done some > initial testing. The card indeed has a shared PHC for both ports so running > ptp4l as BC or TC does indeed work without the jbod option. > > However, sync performance (i.e. rms values) for the downstream OCs isn't > great. And in fact, even the Mellanox as a OC isn't giving great results - > rms values jump a lot (and I've tried various PI value combinations). > > Is anyone else seeing this with Mlx cards as well? Could it be my model or > the firmware? Try the L2 transport. IIRC at least some Mellanox NICs performed worse with UDP transport for some reason. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hey, I've been able to get my hands on a ConnectX-3 Pro card and have done some initial testing. The card indeed has a shared PHC for both ports so running ptp4l as BC or TC does indeed work without the jbod option. However, sync performance (i.e. rms values) for the downstream OCs isn't great. And in fact, even the Mellanox as a OC isn't giving great results - rms values jump a lot (and I've tried various PI value combinations). Is anyone else seeing this with Mlx cards as well? Could it be my model or the firmware? Here is the output of a OC config with the card: $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6 ptp4l[12737.960]: selected /dev/ptp0 as PTP clock ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1 ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447 ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4 ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay 951 +/- 14 ptp4l[12742.139]: rms 129 max 179 freq +268843 +/- 296 delay 963 +/- 11 ptp4l[12743.140]: rms 241 max 490 freq +268455 +/- 452 delay 948 +/- 1 ptp4l[12744.141]: rms 135 max 180 freq +268381 +/- 25 delay 947 +/- 1 ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay 950 +/- 1 ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay 935 +/- 7 ptp4l[12747.144]: rms 210 max 417 freq +268048 +/- 243 delay 942 +/- 3 ptp4l[12748.145]: rms 15 max 32 freq +268415 +/- 29 delay 947 +/- 2 ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay 950 +/- 1 ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay 942 +/- 4 Thanks Andre On 2/11/23 17:37, Jacob Keller wrote: On 11/2/2023 4:15 AM, Andre Puschmann wrote: Hi, On 2/11/23 4:11, James Clark wrote: I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), which has a shared PHC. You can get them for less than $50 on eBay/AliExpress. I had to upgrade the firmware on mine to get PTP support. I haven't yet tried it as a boundary clock. Excellent. This is very helpful James. I've ordered a MCX312A and B and will compare both here. I'll share my results here soon. If you have a chance please also share the firmware version you're currently using on your NIC. With my Intel NIC I could get the BC config working but I needed to set the twoStepFlag to 1. Otherwise I was getting this for both ports: ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range Yep, that would indicate the device doesn't support one-step mode. Sync quality wasn't great as expected though. I'll repeat with the Mellanox once I have them here. Thanks Andre For Intel NICs, the only products I am aware of which share PHC across the device are the E800 series devices. Prior devices (E500, and E700, as well as the gigabit products) do share the same internal oscillator but due to the register interface each function has to setup its own clock. Thanks, Jake ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users -- Andre Puschmann Software Radio Systems (SRS) https://www.srs.io an...@srs.io PGP/GnuPG key: 0x204A85DFEA324D58 fingerprint: 3924 1C60 D52E 81A2 1F2E 0C9D 204A 85DF EA32 4D58 ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On 11/2/2023 4:15 AM, Andre Puschmann wrote: > Hi, > > On 2/11/23 4:11, James Clark wrote: >> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), >> which has a shared PHC. You can get them for less than $50 on >> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP >> support. I haven't yet tried it as a boundary clock. > > Excellent. This is very helpful James. I've ordered a MCX312A and B and > will compare both here. I'll share my results here soon. If you have a > chance please also share the firmware version you're currently using on > your NIC. > > With my Intel NIC I could get the BC config working but I needed to set > the twoStepFlag to 1. Otherwise I was getting this for both ports: > > ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range > Yep, that would indicate the device doesn't support one-step mode. > Sync quality wasn't great as expected though. I'll repeat with the > Mellanox once I have them here. > > Thanks > Andre > For Intel NICs, the only products I am aware of which share PHC across the device are the E800 series devices. Prior devices (E500, and E700, as well as the gigabit products) do share the same internal oscillator but due to the register interface each function has to setup its own clock. Thanks, Jake ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hi, On 2/11/23 4:11, James Clark wrote: I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), which has a shared PHC. You can get them for less than $50 on eBay/AliExpress. I had to upgrade the firmware on mine to get PTP support. I haven't yet tried it as a boundary clock. Excellent. This is very helpful James. I've ordered a MCX312A and B and will compare both here. I'll share my results here soon. If you have a chance please also share the firmware version you're currently using on your NIC. With my Intel NIC I could get the BC config working but I needed to set the twoStepFlag to 1. Otherwise I was getting this for both ports: ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range Sync quality wasn't great as expected though. I'll repeat with the Mellanox once I have them here. Thanks Andre ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On Thu, Nov 2, 2023 at 2:30 AM Andre Puschmann wrote: >Is there any other NIC > that you can recommend and/or know that has a shared PHC among it's ports? I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT), which has a shared PHC. You can get them for less than $50 on eBay/AliExpress. I had to upgrade the firmware on mine to get PTP support. I haven't yet tried it as a boundary clock. I also tried a secondhand dual-port Intel X550-T2, but that has a separate PHC for each port. James ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On 1/11/23 9:28, Miroslav Lichvar wrote: On Wed, Nov 01, 2023 at 09:04:12AM +0100, Andre Puschmann wrote: /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f /opt/ptp4l_cfg/bc.cfg -m -l 6 ptp4l[851.735]: selected /dev/ptp1 as PTP clock ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached ptp4l[851.735]: failed to open port enp1s0f1 failed to create a clock I am not sure where this is coming from but indeed the first interface seems PHC 1 assigned and the second PHC 0. Not sure if this is an issue. It is an issue. The two ports of the NIC don't share the same PHC, so there needs to be something keeping them in sync (phc2sys -a) and ptp4l needs to be told to not verify the PHC index numbers. That's the boundary_clock_jbod option. Don't expect best accuracy in this setup. Thanks Miroslav. I'll give this a try tomorrow. Is there any other NIC that you can recommend and/or know that has a shared PHC among it's ports? Thanks Andre ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On Wed, Nov 01, 2023 at 09:04:12AM +0100, Andre Puschmann wrote: > /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f /opt/ptp4l_cfg/bc.cfg -m > -l 6 > ptp4l[851.735]: selected /dev/ptp1 as PTP clock > ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch > ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached > ptp4l[851.735]: failed to open port enp1s0f1 > failed to create a clock > > I am not sure where this is coming from but indeed the first interface seems > PHC 1 assigned and the second PHC 0. Not sure if this is an issue. It is an issue. The two ports of the NIC don't share the same PHC, so there needs to be something keeping them in sync (phc2sys -a) and ptp4l needs to be told to not verify the PHC index numbers. That's the boundary_clock_jbod option. Don't expect best accuracy in this setup. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Hi everyone, I am picking up this thread as we're still facing some issues with getting the BC mode working on this two-port Intel 82599ES NIC. Cabling is still the same as in the original post - one port receives PTP sync from an external GM and should redistribute it over the second port to a client. We've previously used two ptp4linux instances, one for each interface but I am not sure that is the way to go. Indeed the clock_mode setting in the config file allows for BC which in turn requires two interfaces to be specified. However, doing so results in a PHC device mismatch: /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f /opt/ptp4l_cfg/bc.cfg -m -l 6 ptp4l[851.735]: selected /dev/ptp1 as PTP clock ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached ptp4l[851.735]: failed to open port enp1s0f1 failed to create a clock I am not sure where this is coming from but indeed the first interface seems PHC 1 assigned and the second PHC 0. Not sure if this is an issue. $ ethtool -T enp1s0f0 Time stamping parameters for enp1s0f0: Capabilities: hardware-transmit software-transmit hardware-receive software-receive software-system-clock hardware-raw-clock PTP Hardware Clock: 1 Hardware Transmit Timestamp Modes: off on Hardware Receive Filter Modes: none ptpv1-l4-sync ptpv1-l4-delay-req ptpv2-event I've further noticed the phc_index config param but it doesn't make a difference, resulting in the same mismatch error. Specifying the -p param, however, helps in getting over (this) error but I still don't get PTP packets on the output interface (tcpdump not showing anything). /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -p /dev/ptp0 -f /opt/ptp4l_cfg/bc.cfg -m -l6 ptp4l[1040.051]: selected /dev/ptp0 as PTP clock ptp4l[1040.051]: port 2 (enp1s0f0): taking /dev/ptp0 from the command line, not the attached ptp1 ptp4l[1040.097]: driver rejected most general HWTSTAMP filter ptp4l[1040.097]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range ptp4l[1040.132]: port 1 (enp1s0f1): INITIALIZING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED) ptp4l[1040.180]: driver rejected most general HWTSTAMP filter ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range ptp4l[1040.220]: port 2 (enp1s0f0): INITIALIZING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED) ptp4l[1040.220]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[1040.220]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE My questions are now whether or not the BC clock_mode is the way to go? And, if so, what do you think might be causing the issues we are seeing with the config [1] or setup? Thanks Andre [1] https://pastes.io/epvudxplcz On 3/10/23 15:41, Nils Fuerste wrote: Ah yeah, I didnt read well. That really improved the sync quality - thanks a lot for your advice! On 3/10/23 14:38, Miroslav Lichvar wrote: On Tue, Oct 03, 2023 at 02:33:32PM +0200, Nils Fuerste wrote: Thanks for your help! Unfortunately, setting those values doesn't work for me. I get the following errors: sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m -1.0 is an out of range value for option pi_proportional_const at line 73 failed to parse configuration file ./default-new-master.cfg It should be pi_proportional_exponent, not pi_proportional_const. $ sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m P is a malformed value for option pi_proportional_scale at line 75 failed to parse configuration file ./default-new-master.cfg P and I should be replaced with the values from the table. ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Ah yeah, I didnt read well. That really improved the sync quality - thanks a lot for your advice! On 3/10/23 14:38, Miroslav Lichvar wrote: On Tue, Oct 03, 2023 at 02:33:32PM +0200, Nils Fuerste wrote: Thanks for your help! Unfortunately, setting those values doesn't work for me. I get the following errors: sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m -1.0 is an out of range value for option pi_proportional_const at line 73 failed to parse configuration file ./default-new-master.cfg It should be pi_proportional_exponent, not pi_proportional_const. $ sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m P is a malformed value for option pi_proportional_scale at line 75 failed to parse configuration file ./default-new-master.cfg P and I should be replaced with the values from the table. ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On Tue, Oct 03, 2023 at 02:33:32PM +0200, Nils Fuerste wrote: > Thanks for your help! Unfortunately, setting those values doesn't work for > me. I get the following errors: > > sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m > -1.0 is an out of range value for option pi_proportional_const at line 73 > failed to parse configuration file ./default-new-master.cfg It should be pi_proportional_exponent, not pi_proportional_const. > $ sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m > P is a malformed value for option pi_proportional_scale at line 75 > failed to parse configuration file ./default-new-master.cfg P and I should be replaced with the values from the table. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Thanks for your help! Unfortunately, setting those values doesn't work for me. I get the following errors: sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m -1.0 is an out of range value for option pi_proportional_const at line 73 failed to parse configuration file ./default-new-master.cfg or $ sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m P is a malformed value for option pi_proportional_scale at line 75 failed to parse configuration file ./default-new-master.cfg I am on tag v4.1. Any ideas why this is not working? On 3/10/23 12:28, Miroslav Lichvar wrote: On Tue, Oct 03, 2023 at 11:45:49AM +0200, Nils Fuerste wrote: I have a 82599ES from Intel [1]. I found the paramters you were referring to but I am not sure how to adjust them. Can you give me some guidance for this if you get a chance? Try this: pi_proportional_exponent -1 pi_integral_exponent -1 pi_proportional_scale P pi_integral_scale I where P and I are from one line of this table: 0.316 0.01562 0.158 0.00391 0.079 0.00098 0.040 0.00024 0.020 0.6 This is the NTPv4 (RFC 5905) PLL at five different gains. If the sync interval is -4 (16 sync messages per second), try the middle pair first. ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On Tue, Oct 03, 2023 at 11:45:49AM +0200, Nils Fuerste wrote: > I have a 82599ES from Intel [1]. I found the paramters you were referring to > but I am not sure how to adjust them. Can you give me some guidance for this > if you get a chance? Try this: pi_proportional_exponent -1 pi_integral_exponent -1 pi_proportional_scale P pi_integral_scale I where P and I are from one line of this table: 0.316 0.01562 0.158 0.00391 0.079 0.00098 0.040 0.00024 0.020 0.6 This is the NTPv4 (RFC 5905) PLL at five different gains. If the sync interval is -4 (16 sync messages per second), try the middle pair first. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
Thanks for the quick answer! I have a 82599ES from Intel [1]. I found the paramters you were referring to but I am not sure how to adjust them. Can you give me some guidance for this if you get a chance? These are my current settings, basically the default.cfg: pi_proportional_const 0.0 pi_integral_const 0.0 pi_proportional_scale 0.0 pi_proportional_exponent -0.3 pi_proportional_norm_max 0.7 pi_integral_scale 0.0 pi_integral_exponent 0.4 pi_integral_norm_max 0.3 Thanks for the help! Best regards Nils [1] https://www.intel.la/content/www/xl/es/products/sku/41282/intel-82599es-10-gigabit-ethernet-controller/specifications.html On 2/10/23 11:12, Miroslav Lichvar wrote: On Mon, Oct 02, 2023 at 09:44:19AM +0200, Nils Fuerste wrote: Unfortunately, the sync on the server receiving the sync from the BC looks like this: ptp4l[3297.441]: master offset 42 s2 freq +9338 path delay 319 ptp4l[3298.441]: master offset -10 s2 freq +9298 path delay 319 ptp4l[3299.441]: master offset 19 s2 freq +9324 path delay 319 That doesn't look too bad. Is there a way to improve the configuration? I found the boundary_clock_jbod parameter but setting it to 1 didn't improve the sync. jbod is only needed when the ports don't share the same clock. It degrades the sync quality. Can someone provide a configuration for a simple BC setup with one two-port NIC? What is the sync quality I can expect? Any help is appreciated! Thanks in advance! It depends on the hardware. What NIC do you have? One thing you can try is to shorten the sync interval and decrease the PI constants, also on the client's port of the server to minimize the frequency noise transferred to the client. ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC
On Mon, Oct 02, 2023 at 09:44:19AM +0200, Nils Fuerste wrote: > Unfortunately, the sync on the server receiving the sync from the BC looks > like this: > > ptp4l[3297.441]: master offset 42 s2 freq +9338 path delay > 319 > ptp4l[3298.441]: master offset -10 s2 freq +9298 path delay > 319 > ptp4l[3299.441]: master offset 19 s2 freq +9324 path delay > 319 That doesn't look too bad. > Is there a way to improve the configuration? I found the boundary_clock_jbod > parameter but setting it to 1 didn't improve the sync. jbod is only needed when the ports don't share the same clock. It degrades the sync quality. > Can someone provide a > configuration for a simple BC setup with one two-port NIC? What is the sync > quality I can expect? Any help is appreciated! Thanks in advance! It depends on the hardware. What NIC do you have? One thing you can try is to shorten the sync interval and decrease the PI constants, also on the client's port of the server to minimize the frequency noise transferred to the client. -- Miroslav Lichvar ___ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
