Thanks Marcus. I've done a little more debugging with a simple C++ example
and the B210, and discovered that calling get_rx_stream() seems to be
slightly changing the device time by random amounts when 2 channels are
used. It doesn't matter if I actually issue a stream command or collect any
data. Maybe that will be helpful in reproducing/isolating the issue without
the three B210 setup I was using before.
-Michael
On Tue, Feb 25, 2020 at 1:35 PM Marcus D. Leech via USRP-users <
usrp-users@lists.ettus.com> wrote:
> On 02/25/2020 09:33 AM, Michael Wentz via USRP-users wrote:
>
> Marcus,
>
> Thanks for the suggestion. I added set_subdev_spec('A:A A:B') for the 2
> channel case, but the results are the same.
>
> I also tried this code with a pair of X310s receiving. The results are
> somewhat similar:
> - Channel 0 (A:0) across 2 devices: no delay
> - Channel 1 (B:0) across 2 devices: no delay
> - Channels 0 and 1 (A:0 B:0) across 2 devices: channel 0 is in sync
> between the two, but channel 1 is not - and channel 1 is not in sync with
> channel 0 on either device.
> - Unlike with the B210s, the X310s both report the same time after
> streaming.
>
> Let us focus on the B210 first.
>
> I've been through your code, and I cannot find anything wrong with it.
> Now, I recall that there was a bug in earlier UHD with the two
> clocks in the B210 (each conceptual radio block in the FPGA includes its
> own time-of-day clock), and they wouldn't always get reset
> together, even with set_time_next_pps(), so there would be some skew.
>
> This, or something related may be the root cause of your issue. The
> engineer who most recently worked on B210 is being consulted
> on this, and hopefully myself or Sam Reiter will have an answer soon.
>
>
>
> I've done this in the past with many X310s (each with 2 channels) with
> success, but that was with an older/pre-RFNoC UHD circa 2016. I'm using
> pretty much the same code here. Anything else I can try to time sync
> multiple devices?
>
> -Michael
>
> On Mon, Feb 24, 2020 at 9:07 PM Marcus D. Leech via USRP-users <
> usrp-users@lists.ettus.com> wrote:
>
>> On 02/24/2020 05:10 PM, Michael Wentz via USRP-users wrote:
>>
>> Hi,
>>
>> I'm trying to synchronize multiple B210s to receive data at the same
>> time. I'm only concerned about sample time alignment across devices, and am
>> aware there is a phase offset that will need to be calibrated out
>> separately. I've had success with 1 RX channel across 2 B210s, but
>> something seems wrong when using 2 RX channels across 2 B210s.
>>
>> I've pasted a program below to help reproduce this issue. It uses 1 B210
>> to TX noise and 2 separate B210s to RX at the same time. There is a 40 dB
>> attenuator on the output of the transmitting B210, then a 4 way splitter,
>> and equal length cables to both receivers on the other 2 B210s.
>> Additionally, the 2 receiving B210s get 10 MHz and PPS from a common
>> OctoClock-G.
>>
>> Here are 10 observations of sample offsets between devices (1 MSPS rate):
>> Only channel 0 on both RX: [0,0,0,0,0,0,0,0,0,0]
>> Only channel 1 on both RX: [0,0,0,0,0,0,0,0,0,0]
>> Channels 0 and 1 simultaneously on both
>> RX: [14,1,-126,48,2,-88,17,-204,-96,2]
>>
>> In the 2 channel case both channels on one device are always aligned, but
>> are offset from the channels on the other device. Both of the receiving
>> B210s show the same time last PPS when asked before the streams start. The
>> times in the RX metadata also match. But after the streams stop, the
>> devices respond with different times - neither of which are correct - and
>> the difference between them matches what I'm measuring.
>>
>> I'm using UHD 3.14.1.1 and GNU Radio 3.8 on RHEL 7.7. Any ideas?
>>
>> Thanks,
>> Michael
>>
>> ----------
>>
>> #!/usr/bin/env python3
>>
>> from gnuradio import gr, uhd, analog, blocks
>> import time
>> import numpy as np
>>
>> tx_serial = '30D1***'
>> rx_serials = ['3153***', '3153***']
>> rx_channels = [0, 1]
>>
>> n_rx = len(rx_serials)
>> n_chan = len(rx_channels)
>>
>> class top_block(gr.top_block):
>> def __init__(self):
>> gr.top_block.__init__(self)
>>
>> # transmit path
>> noise_src = analog.noise_source_c(analog.GR_GAUSSIAN, 0.1)
>> tx_strm_args = uhd.stream_args(cpu_format='fc32', channels=[0])
>> tx = uhd.usrp_sink('serial=' + tx_serial, tx_strm_args)
>> tx.set_clock_rate(16e6)
>> tx.set_samp_rate(1e6)
>> tx.set_center_freq(1e9)
>> tx.set_gain(40)
>> tx.set_antenna('TX/RX')
>> self.connect(noise_src, tx)
>>
>> # receive path
>> rx_strm_args = uhd.stream_args(cpu_format='fc32',
>> channels=rx_channels)
>> self.rx = [None]*n_rx
>> head = [None]*n_rx*n_chan
>> file_sink = [None]*n_rx*n_chan
>> for i in range(n_rx):
>> self.rx[i] = uhd.usrp_source('serial=' + rx_serials[i],
>> rx_strm_args)
>> self.rx[i].set_clock_rate(16e6)
>> self.rx[i].set_samp_rate(1e6)
>> self.rx[i].set_center_freq(1e9)
>> self.rx[i].set_gain(40)
>> self.rx[i].set_antenna('RX2')
>> self.rx[i].set_time_source('external')
>> self.rx[i].set_clock_source('external')
>> for j in range(n_chan):
>> ch = i*n_chan + j
>> head[ch] = blocks.head(2*gr.sizeof_float, 10000)
>> f = 'rx_%d%d.dat' % (i, j)
>> file_sink[ch] = blocks.file_sink(2*gr.sizeof_float, f)
>> file_sink[ch].set_unbuffered(True)
>> self.connect((self.rx[i], j), head[ch], file_sink[ch])
>>
>> # make sure 10 MHz ref is locked
>> for i in range(n_rx):
>> while not
>> self.rx[i].get_mboard_sensor('ref_locked').to_bool():
>> print('RX %d waiting for 10 MHz ref lock...' % i)
>> time.sleep(1)
>> print('RX %d 10 MHz ref locked' % i)
>>
>> # time sync the two receivers
>> for i in range(n_rx):
>> t = self.rx[i].get_time_last_pps().get_real_secs()
>> print('RX %d time before align: %r' % (i, t))
>> time_last_pps = self.rx[0].get_time_last_pps().get_real_secs()
>> while time_last_pps ==
>> self.rx[0].get_time_last_pps().get_real_secs():
>> time.sleep(0.1)
>> for i in range(n_rx):
>> self.rx[i].set_time_next_pps(uhd.time_spec_t(0))
>> time_last_pps = self.rx[0].get_time_last_pps().get_real_secs()
>> while time_last_pps ==
>> self.rx[0].get_time_last_pps().get_real_secs():
>> time.sleep(0.1)
>> for i in range(n_rx):
>> t = self.rx[i].get_time_last_pps().get_real_secs()
>> print('RX %d time after align: %r' % (i, t))
>>
>> # collect in the future
>> rx_time = uhd.time_spec_t(2)
>> for i in range(n_rx):
>> self.rx[i].set_start_time(rx_time)
>> self.rx[i].set_recv_timeout(3)
>>
>> # run flowgraph
>> tb = top_block()
>> tb.start()
>> time.sleep(5)
>> tb.stop()
>>
>> # print time
>> t = tb.rx[0].get_time_last_pps().get_real_secs()
>> while t == tb.rx[0].get_time_last_pps().get_real_secs():
>> time.sleep(0.1)
>> for i in range(n_rx):
>> t = tb.rx[i].get_time_last_pps().get_real_secs()
>> print('RX %d time after run: %r' % (i, t))
>>
>> # cross-correlate receivers and print delay
>> y = [None]*n_rx*n_chan
>> for i in range(n_rx):
>> for j in range(n_chan):
>> ch = i*n_chan + j
>> y[ch] = np.fromfile('rx_%d%d.dat' % (i, j), dtype=np.complex64)
>> r = np.correlate(y[0], y[ch], mode='full')
>> max_lag = np.max([len(y[0]), len(y[ch])]) - 1
>> lags = np.arange(-max_lag, max_lag+1)
>> d = lags[np.argmax(np.abs(r))]
>> print('RX %d, channel %d: %d samples' % (i, j, d))
>>
>>
>> _______________________________________________
>> USRP-users mailing
>> listUSRP-users@lists.ettus.comhttp://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>> IN the two channel case, you should probably set the subdev spec to:
>>
>> A:A A:B
>>
>> I didn't set a set_rx_subdev_spec() in the code.
>>
>>
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>
>
> _______________________________________________
> USRP-users mailing
> listUSRP-users@lists.ettus.comhttp://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
>
> _______________________________________________
> USRP-users mailing list
> USRP-users@lists.ettus.com
> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
