It's not specific to N3xx. It can occur when you use the same bitstream and
change the crossbar routing between runs.
Wade
On Wed, Dec 16, 2020 at 4:42 PM Rob Kossler <rkoss...@nd.edu> wrote:
> Thanks Wade,
> That fixes it for my test case with 2 switchboards in series. I'll let
> you know if I see anything strange when testing with my custom blocks.
>
> By the way, does this bug only affect N3xx devices or all RFNoC devices?
> Rob
>
> On Wed, Dec 16, 2020 at 5:33 PM Wade Fife <wade.f...@ettus.com> wrote:
>
>> Rob,
>>
>> I think I found the source of the IQ swap issue you observed. This fixes
>> it for me:
>>
>> diff --git a/host/lib/rfnoc/mgmt_portal.cpp
>> b/host/lib/rfnoc/mgmt_portal.cpp
>> index a54efaaf7..7b09c540b 100644
>> --- a/host/lib/rfnoc/mgmt_portal.cpp
>> +++ b/host/lib/rfnoc/mgmt_portal.cpp
>> @@ -617,6 +617,24 @@ public:
>> _send_recv_mgmt_transaction(xport, cfg_xact);
>> }
>>
>> + // Build a management transaction to configure the destination
>> node
>> + {
>> + mgmt_payload cfg_xact;
>> + cfg_xact.set_header(my_epid, _protover, _chdr_w);
>> + _traverse_to_node(cfg_xact, dst_node_addr);
>> + mgmt_hop_t cfg_hop;
>> + // Configure buffer types
>> + cfg_hop.add_op(mgmt_op_t(mgmt_op_t::MGMT_OP_CFG_WR_REQ,
>> + mgmt_op_t::cfg_payload(REG_ISTRM_CTRL_STATUS,
>> + BUILD_CTRL_STATUS_WORD(false, false, BUFF_U64,
>> BUFF_U64, false))));
>> + // Return the packet back to us
>> + cfg_hop.add_op(mgmt_op_t(mgmt_op_t::MGMT_OP_RETURN));
>> + // Send the transaction and receive a response.
>> + // We don't care about the contents of the response.
>> + cfg_xact.add_hop(cfg_hop);
>> + _send_recv_mgmt_transaction(xport, cfg_xact);
>> + }
>> +
>> // Wait for stream configuration to finish on the HW side
>> _validate_stream_setup(xport, src_node_addr, timeout);
>>
>> Thanks,
>>
>> Wade
>>
>> On Mon, Dec 14, 2020 at 8:32 PM Wade Fife <wade.f...@ettus.com> wrote:
>>
>>> Hi Rob,
>>>
>>> I was able to reproduce the issue, so I'll see if I can get to the
>>> bottom of it. I'm glad you're able to work around it for now. Thanks for
>>> letting us know about it!
>>>
>>> Wade
>>>
>>> On Mon, Dec 14, 2020 at 8:59 AM Rob Kossler <rkoss...@nd.edu> wrote:
>>>
>>>> Hi Wade,
>>>> I tried the command to re-load the FPGA, but I couldn't communicate
>>>> with UHD nicely until I also added the command "systemctl restart
>>>> ursp-uhd". I am now able to reset the N310 to proper behavior when it gets
>>>> in this state.
>>>>
>>>> Regarding this issue of real/imag swapping, do you want me to create a
>>>> support request through supp...@ettus.com? Also, do you need me to
>>>> provide any more info or an example program / FPGA image? Note that now
>>>> that I know how to duplicate the issue, I believe I also know how to avoid
>>>> it. So, it is probably not a critical issue for me presently.
>>>>
>>>> Rob
>>>>
>>>> On Sat, Dec 12, 2020 at 9:47 AM Wade Fife <wade.f...@ettus.com> wrote:
>>>>
>>>>> Thanks for the updates. If you just want to reload the FPGA, try
>>>>> running "overlay rm n310 && overlay add n310" on the N310. This should
>>>>> simply reload the FPGA using the bistream already in the flash.
>>>>>
>>>>> Wade
>>>>>
>>>>> On Fri, Dec 11, 2020 at 6:45 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>>
>>>>>> Wade,
>>>>>> The following also fails using just 2 blocks and 2 attempts:
>>>>>> host_tx => Switchboard#0 => Switchboard#1 => host_rx // SUCCESS
>>>>>> host_tx => Switchboard#1 => Switchboard#0 => host_rx // FAILURE
>>>>>> (RX samples equal swapped I/Q of TX samples)
>>>>>>
>>>>>> In addition to wanting to get this issue fixed, I also have a
>>>>>> question about the best way to "reboot" the N310 which is what I need to
>>>>>> do
>>>>>> to fix the issue after it occurs. One way is to give the "reboot now"
>>>>>> linux command. But, this takes a long time. Another way is to reprogram
>>>>>> the FPGA image via uhd_image_loader. This is appreciably faster, but I'm
>>>>>> thinking that this is not a great idea if the flash memory has a limited
>>>>>> number of write cycles before failure. Is there another way to achieve a
>>>>>> reboot other than these two?
>>>>>> Rob
>>>>>>
>>>>>> On Fri, Dec 11, 2020 at 7:34 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>>>
>>>>>>> AHA! I duplicated the issue with the Switchboard image. The way to
>>>>>>> duplicate the issue is the run the following series of graphs:
>>>>>>> host_tx => Switchboard#0 => Switchboard#1 => host_rx // SUCCESS
>>>>>>> host_tx => Switchboard#2 => Switchboard#3 => host_rx // SUCCESS
>>>>>>> host_tx => Switchboard#0 => Switchboard#2 => host_rx // FAILURE
>>>>>>> (RX samples equal swapped I/Q of TX samples)
>>>>>>> Each of these 3 runs consists of running my application (similar to
>>>>>>> one of the examples such as rfnoc_rx_to_file) such that the UHD object
>>>>>>> is
>>>>>>> re-created each time. My guess is that you could use gnuradio to do it
>>>>>>> instead.
>>>>>>>
>>>>>>> My working theory is that the problem is caused by the fact that the
>>>>>>> Switchboard#2 input port was changed from being connected to the host in
>>>>>>> test case 2 to then be connected to another RFNoC block in test case 3.
>>>>>>> Or, I suppose it could be the output port on this block (same logic).
>>>>>>>
>>>>>>> If you want me to send you my FPGA image with the 4 Switchboard
>>>>>>> blocks, let me know.
>>>>>>> Rob
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 11, 2020 at 7:09 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>>>>
>>>>>>>> Hi Wade,
>>>>>>>> After thinking about it a bit, I would ignore the "negation" issue
>>>>>>>> for now. This may be a byproduct of I/Q swapping related to my custom
>>>>>>>> block.
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2020 at 6:34 PM Rob Kossler <rkoss...@nd.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Wade,
>>>>>>>>> Thanks for the info. I still do not know what's going on, but
>>>>>>>>> here are a few updates:
>>>>>>>>>
>>>>>>>>> - I built a new N310 image with 4 switchboards (1x1) and 1
>>>>>>>>> splitstream (1x2) blocks in addition to the default blocks. All
>>>>>>>>> of the
>>>>>>>>> additional blocks are connected to SEPs for dynamic linking. I
>>>>>>>>> tried my
>>>>>>>>> best to get bad behavior using a chain of 1 to 4 switchboards, but
>>>>>>>>> I could
>>>>>>>>> not duplicate any I/Q swapping or negation issues.
>>>>>>>>> - I went back to my custom image (that I described below) and
>>>>>>>>> found different behavior today (sometimes things worked OK). So,
>>>>>>>>> this got
>>>>>>>>> me to thinking that I am somehow getting the FPGA in a bad state
>>>>>>>>> such that
>>>>>>>>> a reboot (or FPGA re-flashing) fixes things. I have had some
>>>>>>>>> success with
>>>>>>>>> re-booting and things working properly. But, I still do not have
>>>>>>>>> a true
>>>>>>>>> handle on things and cannot adequately predict when things are
>>>>>>>>> going to
>>>>>>>>> succeed or fail.
>>>>>>>>> - One thing that has been constant is that I have never seen
>>>>>>>>> bad behavior when I only have 1 block in my graph (no matter which
>>>>>>>>> block I
>>>>>>>>> choose). Note that for all of my tests, the graph looks like
>>>>>>>>> this: host_tx
>>>>>>>>> => block_chain => host_rx, where block_chain is a sequential chain
>>>>>>>>> of 1 or
>>>>>>>>> more rfnoc blocks.
>>>>>>>>>
>>>>>>>>> I will send a follow up once I figure things out.
>>>>>>>>> Rob
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 11, 2020 at 6:22 PM Wade Fife <wade.f...@ettus.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Rob,
>>>>>>>>>>
>>>>>>>>>> The SEPs do have the ability to swap I and Q. This is because on
>>>>>>>>>> the host computer, I is usually in the lower bits and Q is in the
>>>>>>>>>> upper
>>>>>>>>>> bits of each 32-bit word, but in RFNoC, for historical reasons, I
>>>>>>>>>> goes in
>>>>>>>>>> the upper bits and Q in the lower bits. The software programs the IQ
>>>>>>>>>> swapping when it sets up the graph.
>>>>>>>>>>
>>>>>>>>>> I assume you're using dynamic connections (through the crossbar)
>>>>>>>>>> to control the number of FFTs the data is passed through, and not
>>>>>>>>>> static
>>>>>>>>>> connections? If that's the case then I wonder if software configures
>>>>>>>>>> IQ
>>>>>>>>>> swapping incorrectly in some configurations. I'll see if I can do
>>>>>>>>>> some
>>>>>>>>>> testing next week to confirm if it's working correctly.
>>>>>>>>>>
>>>>>>>>>> As for negation, I'm not aware of anywhere we do that off the top
>>>>>>>>>> of my head. Is that behavior block dependent? I'll see if I can find
>>>>>>>>>> anywhere this happens.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Wade
>>>>>>>>>>
>>>>>>>>>> On Thu, Dec 10, 2020 at 3:54 PM Rob Kossler via USRP-users <
>>>>>>>>>> usrp-users@lists.ettus.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> I am encountering very strange behavior with a custom FPGA image
>>>>>>>>>>> (N310). It appears that data streaming through SEPs can get swapped
>>>>>>>>>>> (I/Q)
>>>>>>>>>>> and/or negated. Is anyone at Ettus aware of anything that could
>>>>>>>>>>> cause
>>>>>>>>>>> this? Of course, the issue might be on my end, but I can't think
>>>>>>>>>>> of what
>>>>>>>>>>> it might be given that all of my custom blocks work as expected in
>>>>>>>>>>> isolation (if the block is the only block in graph).
>>>>>>>>>>>
>>>>>>>>>>> My custom image is the following:
>>>>>>>>>>>
>>>>>>>>>>> - default blocks of Radios, DDCs, DUCs (each 2x2 and
>>>>>>>>>>> statically connected as in default image)
>>>>>>>>>>> - custom blocks of two 1x1 windowed-fft blocks, two 1x1
>>>>>>>>>>> vector-avg blocks, and one 2x2 custom block. Note: each of these
>>>>>>>>>>> blocks is
>>>>>>>>>>> connected to its own SEP, so I can connect dynamically in any
>>>>>>>>>>> fashion.
>>>>>>>>>>>
>>>>>>>>>>> My test case is transmitting 8192 random samples from host to
>>>>>>>>>>> FFT block and then optionally through a 2nd FFT block before back
>>>>>>>>>>> to host.
>>>>>>>>>>> In the test case, the radios/DDCs/DUCs are not used.
>>>>>>>>>>>
>>>>>>>>>>> Here is what I observed:
>>>>>>>>>>>
>>>>>>>>>>> - If I only include 1 FFT block in my RFNoC graph, I get the
>>>>>>>>>>> expected results (the output from the FPGA matches what I
>>>>>>>>>>> calculate in
>>>>>>>>>>> Matlab for the FFT). This is true for either of the two FFT
>>>>>>>>>>> blocks.
>>>>>>>>>>> - If I include both FFT blocks in series, I can only match
>>>>>>>>>>> the FPGA output if I swap the I/Q values in between my Matlab
>>>>>>>>>>> FFTs.
>>>>>>>>>>> - Note: that this issue is not FFT-related as I can also
>>>>>>>>>>> duplicate this issue with the other blocks.
>>>>>>>>>>> - If I use 3 blocks in series (each through SEP), I need to
>>>>>>>>>>> negate certain data in order to get it to match the FPGA output
>>>>>>>>>>>
>>>>>>>>>>> My next step is likely to build a new image with Ettus-developed
>>>>>>>>>>> FIFOs to prove that the data is getting swapped/negated when 2 or
>>>>>>>>>>> more are
>>>>>>>>>>> used in series through SEPs.
>>>>>>>>>>>
>>>>>>>>>>> Let me know if you have any suggestions for other things to try.
>>>>>>>>>>>
>>>>>>>>>>> Rob
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> USRP-users mailing list
>>>>>>>>>>> USRP-users@lists.ettus.com
>>>>>>>>>>>
>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>>>>>>>
>>>>>>>>>>
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
