Re: [USRP-users] RfnocError: OpTimeout
Building the X310 FPGA does require a Xilinx Vivado license. It is not one of the FPGAs for which the free version of the tool works. You can find instructions for building the FPGA in the user manual: https://files.ettus.com/manual/md_usrp3_build_instructions.html Thanks, Wade On Fri, Nov 20, 2020 at 11:38 AM Dustin Widmann wrote: > Hi Wade, > > Increasing the size of the relevant buffers does sound interesting. > Unfortunately I'm not really familiar with the workflow for building > the FPGA bitstream. I'd be interested in trying it if you could point > me towards up-to-date documentation on doing it, so long as there > wouldn't be a major software purchase involved. > > Dustin > > On Wed, 2020-11-18 at 19:12 -0600, Wade Fife wrote: > > Dustin, > > > > It sounds like the software thinks the control port FIFO is filling > > up. Are you issuing a lot of timed commands, maybe far into the > > future? I wonder if issuing commands faster than they are being > > executed could cause the FIFO on the FPGA to fill up with commands. > > > > You could try increasing the timeout. Or, if you're comfortable with > > building the FPGA bitstream, you can try increasing the FIFO sizes on > > the FPGA. But either of these might just delay the inevitable. I'm > > not sure which block is causing the timeout, but these are the likely > > culprits if you want to try increasing the FIFO sizes: > > > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L147 > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L194 > > > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L131 > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L178 > > > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L131 > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L178 > > > > You might try doubling the number in each case then rebuild your FPGA > > image. > > > > Thanks, > > > > Wade > > > > > > On Wed, Nov 18, 2020 at 9:25 AM Dustin Widmann via USRP-users > > wrote: > > > Hi usrp-users, > > > > > > terminate called after throwing an instance of 'uhd::op_timeout' > > > what(): RfnocError: OpTimeout: Control operation timed out > > > waiting > > > for space in command buffer > > > > > > I've been getting the error above occasionally, usually after hours > > > of > > > operation. I've got a few questions about it: > > > * The error seems self explanatory, but why might it happen > > > sometimes > > > and not others? > > > * Are there any steps I can take to prevent the error from > > > occurring? > > > * Alternately, what would be the best way to catch and recover from > > > it? > > > > > > Relevant context: > > > * USRP X310 > > > * * ubx (using for 1x transmit) > > > * * twinrx (using for 2x phase synchronous rx, with lo sharing) > > > * UHD 4.0 C++ API, multiusrp > > > > > > Dustin > > > > > > > > > ___ > > > USRP-users mailing list > > > USRP-users@lists.ettus.com > > > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com > > > ___ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Re: [USRP-users] RfnocError: OpTimeout
Hi Wade, Increasing the size of the relevant buffers does sound interesting. Unfortunately I'm not really familiar with the workflow for building the FPGA bitstream. I'd be interested in trying it if you could point me towards up-to-date documentation on doing it, so long as there wouldn't be a major software purchase involved. Dustin On Wed, 2020-11-18 at 19:12 -0600, Wade Fife wrote: > Dustin, > > It sounds like the software thinks the control port FIFO is filling > up. Are you issuing a lot of timed commands, maybe far into the > future? I wonder if issuing commands faster than they are being > executed could cause the FIFO on the FPGA to fill up with commands. > > You could try increasing the timeout. Or, if you're comfortable with > building the FPGA bitstream, you can try increasing the FIFO sizes on > the FPGA. But either of these might just delay the inevitable. I'm > not sure which block is causing the timeout, but these are the likely > culprits if you want to try increasing the FIFO sizes: > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L147 > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L194 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L131 > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L178 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L131 > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L178 > > You might try doubling the number in each case then rebuild your FPGA > image. > > Thanks, > > Wade > > > On Wed, Nov 18, 2020 at 9:25 AM Dustin Widmann via USRP-users > wrote: > > Hi usrp-users, > > > > terminate called after throwing an instance of 'uhd::op_timeout' > > what(): RfnocError: OpTimeout: Control operation timed out > > waiting > > for space in command buffer > > > > I've been getting the error above occasionally, usually after hours > > of > > operation. I've got a few questions about it: > > * The error seems self explanatory, but why might it happen > > sometimes > > and not others? > > * Are there any steps I can take to prevent the error from > > occurring? > > * Alternately, what would be the best way to catch and recover from > > it? > > > > Relevant context: > > * USRP X310 > > * * ubx (using for 1x transmit) > > * * twinrx (using for 2x phase synchronous rx, with lo sharing) > > * UHD 4.0 C++ API, multiusrp > > > > Dustin > > > > > > ___ > > USRP-users mailing list > > USRP-users@lists.ettus.com > > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com ___ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Re: [USRP-users] RfnocError: OpTimeout
Hi Dustin, Based on your other post of the dsp_tune not appearing to have worked, it seems likely that this is indeed the case and all of your dsp_tune commands are getting backed up in the FIFO since they are never executed. Does this seem plausible? This of course doesn't explain why the commands are never getting executed, but it does seem to account for both behaviors you're seeing. Rob On Wed, Nov 18, 2020 at 7:37 PM Dustin Widmann via USRP-users < usrp-users@lists.ettus.com> wrote: > On Wed, 2020-11-18 at 19:12 -0600, Wade Fife wrote: > > Dustin, > > It sounds like the software thinks the control port FIFO is filling up. > Are you issuing a lot of timed commands, maybe far into the future? I > wonder if issuing commands faster than they are being executed could cause > the FIFO on the FPGA to fill up with commands. > > You could try increasing the timeout. Or, if you're comfortable with > building the FPGA bitstream, you can try increasing the FIFO sizes on the > FPGA. But either of these might just delay the inevitable. I'm not sure > which block is causing the timeout, but these are the likely culprits if > you want to try increasing the FIFO sizes: > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L147 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L194 > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L131 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L178 > > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L131 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L178 > > You might try doubling the number in each case then rebuild your FPGA > image. > > Thanks, > > Wade > > > On Wed, Nov 18, 2020 at 9:25 AM Dustin Widmann via USRP-users < > usrp-users@lists.ettus.com> wrote: > > Hi usrp-users, > > terminate called after throwing an instance of 'uhd::op_timeout' > what(): RfnocError: OpTimeout: Control operation timed out waiting > for space in command buffer > > I've been getting the error above occasionally, usually after hours of > operation. I've got a few questions about it: > * The error seems self explanatory, but why might it happen sometimes > and not others? > * Are there any steps I can take to prevent the error from occurring? > * Alternately, what would be the best way to catch and recover from it? > > Relevant context: > * USRP X310 > * * ubx (using for 1x transmit) > * * twinrx (using for 2x phase synchronous rx, with lo sharing) > * UHD 4.0 C++ API, multiusrp > > Dustin > > > ___ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com > > Hi Wade, > > I do issue timed commands, but never for far into the future. As of right > now, I: > * tune the transmitter immediately > * set the command time for 100ms in the future > * tune the receiver > * set the command time for 350ms in the future > * tune the receiver again > * wait 600ms > * collect some data > I tune twice, because tuning the twinrx actually requires more commands > than the buffer can hold, so if I don't, then I won't have a consistent > phase offset. This is done as needed and never done farther ahead of time > than described (though that could forseeably be useful towards my eventual > goal of making it be not super-slow). As of yet, I've only observed this > error after thousands of iterations. I'll try to increase the FIFO sizes > and do a build; thus far I had been using the FPGA image from the release. > Hopefully it will help. Thanks for the info Wade. > > Dustin > ___ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com > ___ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Re: [USRP-users] RfnocError: OpTimeout
On Wed, 2020-11-18 at 19:12 -0600, Wade Fife wrote: > Dustin, > > It sounds like the software thinks the control port FIFO is filling > up. Are you issuing a lot of timed commands, maybe far into the > future? I wonder if issuing commands faster than they are being > executed could cause the FIFO on the FPGA to fill up with commands. > > You could try increasing the timeout. Or, if you're comfortable with > building the FPGA bitstream, you can try increasing the FIFO sizes on > the FPGA. But either of these might just delay the inevitable. I'm > not sure which block is causing the timeout, but these are the likely > culprits if you want to try increasing the FIFO sizes: > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L147 > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L194 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L131 > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L178 > > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L131 > https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L178 > > You might try doubling the number in each case then rebuild your FPGA > image. > > Thanks, > > Wade > > > On Wed, Nov 18, 2020 at 9:25 AM Dustin Widmann via USRP-users > wrote: > > Hi usrp-users, > > > > terminate called after throwing an instance of 'uhd::op_timeout' > > what(): RfnocError: OpTimeout: Control operation timed out > > waiting > > for space in command buffer > > > > I've been getting the error above occasionally, usually after hours > > of > > operation. I've got a few questions about it: > > * The error seems self explanatory, but why might it happen > > sometimes > > and not others? > > * Are there any steps I can take to prevent the error from > > occurring? > > * Alternately, what would be the best way to catch and recover from > > it? > > > > Relevant context: > > * USRP X310 > > * * ubx (using for 1x transmit) > > * * twinrx (using for 2x phase synchronous rx, with lo sharing) > > * UHD 4.0 C++ API, multiusrp > > > > Dustin > > > > > > ___ > > USRP-users mailing list > > USRP-users@lists.ettus.com > > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com Hi Wade, I do issue timed commands, but never for far into the future. As of right now, I: * tune the transmitter immediately * set the command time for 100ms in the future * tune the receiver * set the command time for 350ms in the future * tune the receiver again * wait 600ms * collect some data I tune twice, because tuning the twinrx actually requires more commands than the buffer can hold, so if I don't, then I won't have a consistent phase offset. This is done as needed and never done farther ahead of time than described (though that could forseeably be useful towards my eventual goal of making it be not super-slow). As of yet, I've only observed this error after thousands of iterations. I'll try to increase the FIFO sizes and do a build; thus far I had been using the FPGA image from the release. Hopefully it will help. Thanks for the info Wade. Dustin ___ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Re: [USRP-users] RfnocError: OpTimeout
Dustin, It sounds like the software thinks the control port FIFO is filling up. Are you issuing a lot of timed commands, maybe far into the future? I wonder if issuing commands faster than they are being executed could cause the FIFO on the FPGA to fill up with commands. You could try increasing the timeout. Or, if you're comfortable with building the FPGA bitstream, you can try increasing the FIFO sizes on the FPGA. But either of these might just delay the inevitable. I'm not sure which block is causing the timeout, but these are the likely culprits if you want to try increasing the FIFO sizes: https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L147 https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/noc_shell_radio.v#L194 https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L131 https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_duc/noc_shell_duc.v#L178 https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L131 https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_ddc/noc_shell_ddc.v#L178 You might try doubling the number in each case then rebuild your FPGA image. Thanks, Wade On Wed, Nov 18, 2020 at 9:25 AM Dustin Widmann via USRP-users < usrp-users@lists.ettus.com> wrote: > Hi usrp-users, > > terminate called after throwing an instance of 'uhd::op_timeout' > what(): RfnocError: OpTimeout: Control operation timed out waiting > for space in command buffer > > I've been getting the error above occasionally, usually after hours of > operation. I've got a few questions about it: > * The error seems self explanatory, but why might it happen sometimes > and not others? > * Are there any steps I can take to prevent the error from occurring? > * Alternately, what would be the best way to catch and recover from it? > > Relevant context: > * USRP X310 > * * ubx (using for 1x transmit) > * * twinrx (using for 2x phase synchronous rx, with lo sharing) > * UHD 4.0 C++ API, multiusrp > > Dustin > > > ___ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com > ___ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
[USRP-users] RfnocError: OpTimeout
Hi usrp-users, terminate called after throwing an instance of 'uhd::op_timeout' what(): RfnocError: OpTimeout: Control operation timed out waiting for space in command buffer I've been getting the error above occasionally, usually after hours of operation. I've got a few questions about it: * The error seems self explanatory, but why might it happen sometimes and not others? * Are there any steps I can take to prevent the error from occurring? * Alternately, what would be the best way to catch and recover from it? Relevant context: * USRP X310 * * ubx (using for 1x transmit) * * twinrx (using for 2x phase synchronous rx, with lo sharing) * UHD 4.0 C++ API, multiusrp Dustin ___ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
