On 2022-09-23 11:50, Jim Palladino wrote:
I spent a fair amount of time trying to debug this. I wrote a custom
app using the RFNoC API and tried continuously changing the sample
rate (while streaming). I mostly bounced back and forth between
100Msps and 200Msps, with a master clock of 200MHz. To do this, I
would change the sample rate like this:
ddc_ctrl->set_output_rate(200000000, ACTIVE_CHAN);
I could always get it stop working after a certain amount of time.
Depending on my setup (adding sleeps in different parts of my or UHDs
code), that time could vary from seconds to more than a day. When it
stops working, UHD returns:
Error: RfnocError: OpTimeout: Control operation timed out waiting for
space in command buffer
I tried issuing other commands to the DDC controller, like changing
frequency. I could loop through those and other commands without ever
running into this issue. It seems specific to the set_output_rate
command. Adding time delays into the infinite loop, doesn't seem to
help -- I still run into the issue, eventually. I debugged enough to
find that where things get "stuck" is in the set_decim() method in
ddc_block_control.cpp. Part of the code looks like:
void set_decim(int decim, const size_t chan)
{
RFNOC_LOG_TRACE("Set decim to " << decim);
// Step 1: Calculate number of halfbands
uint32_t hb_enable = 0;
uint32_t cic_decim = decim;
while ((cic_decim % 2 == 0) and hb_enable < _num_halfbands) {
hb_enable++;
cic_decim /= 2;
}
// Step 2: Make sure we can handle the rest with the CIC
UHD_ASSERT_THROW(hb_enable <= _num_halfbands);
UHD_ASSERT_THROW(cic_decim > 0 and cic_decim <= _cic_max_decim);
const uint32_t decim_word = (hb_enable << 8) | cic_decim;
_ddc_reg_iface.poke32(SR_DECIM_ADDR, decim_word, chan);
// Rate change = M/N
_ddc_reg_iface.poke32(SR_N_ADDR, decim, chan);
_ddc_reg_iface.poke32(SR_M_ADDR, 1, chan);
// Configure time increment in ticks per M output samples
_ddc_reg_iface.poke32(SR_TIME_INCR_ADDR,
uint32_t(get_tick_rate()/get_output_rate(chan)), chan);
You can see a series of register pokes. I added debug statements in
between the pokes and found that the command buffer error occurs
almost always after:
_ddc_reg_iface.poke32(SR_N_ADDR, decim, chan);
Meaning that line will return, and I can see the debug statement I
print out after that line. So, it seems there isn't room for the SR_M
poke. Every once in a while, it would fail right after the SR_M poke
instead, but that is rare. Adding sleeps in the right place (like
after the SR_N poke) never seemed to fix the problem, although putting
a 1s sleep after the SR_N poke greatly extended the time before failure.
In ctrlport_endpoint.cpp, I tried changing DEFAULT_TIMEOUT from 1.0 to
10.0. The same issue still occurred, but I would see the code get
stuck right after the SR_N poke and then just sit there for 10 seconds
before outputting the "command buffer full" error.
In the FPGA, I tried a few things like increasing some buffer sizes,
watching some of the DDC related signals with an ILA, etc, but didn't
make any notable progress. I don't have time to go too far with that
right now.
Anyhow, it looks like, every once in a while, a DDC decimation change
puts something in a state where it stops working (no more data samples
are streaming) and the command buffer is full.
My setup is identical to Jason's from the original post (UHD 4.1.0.2,
Ubuntu 18.04, N320 with XG load.
If anyone has any more thoughts after reading this, it would be
appreciated.
Thanks,
Jim
------------------------------------------------------------------------
Thanks for all your work on this, Jim.
I've poked some Ettus/NI R&D folks on this, and hopefully can get a bit
more educated insight than I can provide--I don't
have an N320 myself, so I'd be guessing...
_______________________________________________
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-le...@lists.ettus.com
