This is a particularly curious problem.
The offset between B210’s of the “time after run” values after sample capture
has completed is very hard to explain…certainly from a hardware only
perspective.
What are the absolute time values reported? Are they at least approximately 5-6
seconds as you might expect from your python? Or some completely random value?
None of the UHD commands you use after “set_time_next_pps” should cause any
change in the relationship of the time of day counters and the PPS.
It makes sense that all samples have the expected RX metadata data, the capture
is triggered by an arithmetic comparison between the time-of-day counters and
the value you load into the H/W as a trigger using “set_start_time”, and the
counter value is read to add meta data to each packet of RX samples that is
assembled.
Superficially it would appear that somehow the time-of-day counters are
“corrupted” between the "time after align” values you read and before they
count to timespec(2.0)
(Completely random thought, but do you have the B210’s powered via USB or from
the 6V jacks? I’m presuming they are plugged into to adjacent USB ports on a
host?)
I should probably caution you, unrelated to this issue that the B210 isn't
designed so that multiple boards can be synced and used coherently unlike
N210/X310. Whilst the master clock
Is disciplined by the 10MHz, you are able to program it to arbitrary
frequencies that are not phase aligned between boards which means PPS edge
detect events as seen by the B210’s
may occur at different actual times, though this would only produce a time
misalignment of upto only 1 master clock period.
BTW Marcus the PPS bug I *think you are referring to (edge detect driven by
different synchronizers) was fixed before 3.9LTS was released and would have
manifested differently.
> On Feb 25, 2020, at 10:35 AM, Marcus D. Leech via USRP-users
> <usrp-users@lists.ettus.com> wrote:
>
> On 02/25/2020 09:33 AM, Michael Wentz via USRP-users wrote:
>> Marcus,
>>
>> Thanks for the suggestion. I added set_subdev_spec('A:A A:B') for the 2
>> channel case, but the results are the same.
>>
>> I also tried this code with a pair of X310s receiving. The results are
>> somewhat similar:
>> - Channel 0 (A:0) across 2 devices: no delay
>> - Channel 1 (B:0) across 2 devices: no delay
>> - Channels 0 and 1 (A:0 B:0) across 2 devices: channel 0 is in sync between
>> the two, but channel 1 is not - and channel 1 is not in sync with channel 0
>> on either device.
>> - Unlike with the B210s, the X310s both report the same time after streaming.
> Let us focus on the B210 first.
>
> I've been through your code, and I cannot find anything wrong with it. Now,
> I recall that there was a bug in earlier UHD with the two
> clocks in the B210 (each conceptual radio block in the FPGA includes its
> own time-of-day clock), and they wouldn't always get reset
> together, even with set_time_next_pps(), so there would be some skew.
>
> This, or something related may be the root cause of your issue. The
> engineer who most recently worked on B210 is being consulted
> on this, and hopefully myself or Sam Reiter will have an answer soon.
>
>
>>
>> I've done this in the past with many X310s (each with 2 channels) with
>> success, but that was with an older/pre-RFNoC UHD circa 2016. I'm using
>> pretty much the same code here. Anything else I can try to time sync
>> multiple devices?
>>
>> -Michael
>>
>> On Mon, Feb 24, 2020 at 9:07 PM Marcus D. Leech via USRP-users
>> <usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com>> wrote:
>> On 02/24/2020 05:10 PM, Michael Wentz via USRP-users wrote:
>>> Hi,
>>>
>>> I'm trying to synchronize multiple B210s to receive data at the same time.
>>> I'm only concerned about sample time alignment across devices, and am aware
>>> there is a phase offset that will need to be calibrated out separately.
>>> I've had success with 1 RX channel across 2 B210s, but something seems
>>> wrong when using 2 RX channels across 2 B210s.
>>>
>>> I've pasted a program below to help reproduce this issue. It uses 1 B210 to
>>> TX noise and 2 separate B210s to RX at the same time. There is a 40 dB
>>> attenuator on the output of the transmitting B210, then a 4 way splitter,
>>> and equal length cables to both receivers on the other 2 B210s.
>>> Additionally, the 2 receiving B210s get 10 MHz and PPS from a common
>>> OctoClock-G.
>>>
>>> Here are 10 observations of sample offsets between devices (1 MSPS rate):
>>> Only channel 0 on both RX: [0,0,0,0,0,0,0,0,0,0]
>>> Only channel 1 on both RX: [0,0,0,0,0,0,0,0,0,0]
>>> Channels 0 and 1 simultaneously on both RX:
>>> [14,1,-126,48,2,-88,17,-204,-96,2]
>>>
>>> In the 2 channel case both channels on one device are always aligned, but
>>> are offset from the channels on the other device. Both of the receiving
>>> B210s show the same time last PPS when asked before the streams start. The
>>> times in the RX metadata also match. But after the streams stop, the
>>> devices respond with different times - neither of which are correct - and
>>> the difference between them matches what I'm measuring.
>>>
>>> I'm using UHD 3.14.1.1 and GNU Radio 3.8 on RHEL 7.7. Any ideas?
>>>
>>> Thanks,
>>> Michael
>>>
>>> ----------
>>>
>>> #!/usr/bin/env python3
>>>
>>> from gnuradio import gr, uhd, analog, blocks
>>> import time
>>> import numpy as np
>>>
>>> tx_serial = '30D1***'
>>> rx_serials = ['3153***', '3153***']
>>> rx_channels = [0, 1]
>>>
>>> n_rx = len(rx_serials)
>>> n_chan = len(rx_channels)
>>>
>>> class top_block(gr.top_block):
>>> def __init__(self):
>>> gr.top_block.__init__(self)
>>>
>>> # transmit path
>>> noise_src = analog.noise_source_c(analog.GR_GAUSSIAN, 0.1)
>>> tx_strm_args = uhd.stream_args(cpu_format='fc32', channels=[0])
>>> tx = uhd.usrp_sink('serial=' + tx_serial, tx_strm_args)
>>> tx.set_clock_rate(16e6)
>>> tx.set_samp_rate(1e6)
>>> tx.set_center_freq(1e9)
>>> tx.set_gain(40)
>>> tx.set_antenna('TX/RX')
>>> self.connect(noise_src, tx)
>>>
>>> # receive path
>>> rx_strm_args = uhd.stream_args(cpu_format='fc32',
>>> channels=rx_channels)
>>> self.rx = [None]*n_rx
>>> head = [None]*n_rx*n_chan
>>> file_sink = [None]*n_rx*n_chan
>>> for i in range(n_rx):
>>> self.rx[i] = uhd.usrp_source('serial=' + rx_serials[i],
>>> rx_strm_args)
>>> self.rx[i].set_clock_rate(16e6)
>>> self.rx[i].set_samp_rate(1e6)
>>> self.rx[i].set_center_freq(1e9)
>>> self.rx[i].set_gain(40)
>>> self.rx[i].set_antenna('RX2')
>>> self.rx[i].set_time_source('external')
>>> self.rx[i].set_clock_source('external')
>>> for j in range(n_chan):
>>> ch = i*n_chan + j
>>> head[ch] = blocks.head(2*gr.sizeof_float, 10000)
>>> f = 'rx_%d%d.dat' % (i, j)
>>> file_sink[ch] = blocks.file_sink(2*gr.sizeof_float, f)
>>> file_sink[ch].set_unbuffered(True)
>>> self.connect((self.rx[i], j), head[ch], file_sink[ch])
>>>
>>> # make sure 10 MHz ref is locked
>>> for i in range(n_rx):
>>> while not self.rx[i].get_mboard_sensor('ref_locked').to_bool():
>>> print('RX %d waiting for 10 MHz ref lock...' % i)
>>> time.sleep(1)
>>> print('RX %d 10 MHz ref locked' % i)
>>>
>>> # time sync the two receivers
>>> for i in range(n_rx):
>>> t = self.rx[i].get_time_last_pps().get_real_secs()
>>> print('RX %d time before align: %r' % (i, t))
>>> time_last_pps = self.rx[0].get_time_last_pps().get_real_secs()
>>> while time_last_pps ==
>>> self.rx[0].get_time_last_pps().get_real_secs():
>>> time.sleep(0.1)
>>> for i in range(n_rx):
>>> self.rx[i].set_time_next_pps(uhd.time_spec_t(0))
>>> time_last_pps = self.rx[0].get_time_last_pps().get_real_secs()
>>> while time_last_pps ==
>>> self.rx[0].get_time_last_pps().get_real_secs():
>>> time.sleep(0.1)
>>> for i in range(n_rx):
>>> t = self.rx[i].get_time_last_pps().get_real_secs()
>>> print('RX %d time after align: %r' % (i, t))
>>>
>>> # collect in the future
>>> rx_time = uhd.time_spec_t(2)
>>> for i in range(n_rx):
>>> self.rx[i].set_start_time(rx_time)
>>> self.rx[i].set_recv_timeout(3)
>>>
>>> # run flowgraph
>>> tb = top_block()
>>> tb.start()
>>> time.sleep(5)
>>> tb.stop()
>>>
>>> # print time
>>> t = tb.rx[0].get_time_last_pps().get_real_secs()
>>> while t == tb.rx[0].get_time_last_pps().get_real_secs():
>>> time.sleep(0.1)
>>> for i in range(n_rx):
>>> t = tb.rx[i].get_time_last_pps().get_real_secs()
>>> print('RX %d time after run: %r' % (i, t))
>>>
>>> # cross-correlate receivers and print delay
>>> y = [None]*n_rx*n_chan
>>> for i in range(n_rx):
>>> for j in range(n_chan):
>>> ch = i*n_chan + j
>>> y[ch] = np.fromfile('rx_%d%d.dat' % (i, j), dtype=np.complex64)
>>> r = np.correlate(y[0], y[ch], mode='full')
>>> max_lag = np.max([len(y[0]), len(y[ch])]) - 1
>>> lags = np.arange(-max_lag, max_lag+1)
>>> d = lags[np.argmax(np.abs(r))]
>>> print('RX %d, channel %d: %d samples' % (i, j, d))
>>>
>>>
>>> _______________________________________________
>>> USRP-users mailing list
>>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com>
>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com>
>> IN the two channel case, you should probably set the subdev spec to:
>>
>> A:A A:B
>>
>> I didn't set a set_rx_subdev_spec() in the code.
>>
>>
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com>
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com>
>>
>>
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com>
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com>
>
> _______________________________________________
> USRP-users mailing list
> USRP-users@lists.ettus.com
> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
