Hi Nate, I looked at the link you sent (performance tuning tips) and your email. Here are a few comments / questions:
- Regarding my initial question, what could be the cause of WORSE performance when I inserted the DmaFIFO in the receive chain of my RFNoC graph? Recall the "Radio->DDC->host" produces no errors, but "Radio->DDC->DmaFIFO->host" produces errors (timeouts) - Regarding "cpufrequtils" (from the performance tuning tips), I have run the suggestions on my 18.04 Ubuntu system (Xeon E5-1620v4 3.5GHz, 4-core/8-thread), but when I run cpufreq-info, there is often 1 or more CPUs that show up at 1.6 GHz or so (while the majority report ~3.6 GHz). It is not clear to me whether this utility is doing its job or not. - Regarding DPDK, I have tried to install it, but have had no success. The instructions say that after updating grub with "iommu=pt intel_iommu=on hugepages=2048", then "After you reboot, you should see /sys/kernel/iommu_groups populated". I do have such a folder, but it is empty so I'm not sure if this step was successful or not. Furthermore, I am unable to run the dpdk-devbind python script to bind the vfio-pci driver to my Intel X520-DA2 NIC (see error message below) - Regarding XFS vs EXT4, this is something I haven't tried yet, but plan to. I am completely unfamiliar with XFS. My SSD is actually 4 Samsung EVO 850 SATA SSDs in a software RAID-0 (using mdadm). If I copy a huge file from my RAM disk to the SSD, I am able to verify transfer rates greater than 1GB/s (I believe closer to 1.5GB/s). - Finally, regarding "D" (sequence errors), what is the possible cause? These are the most frustrating errors because their cause is a mystery to me. I fully expect that when my host PC is too slow to keep up with the torrent of data coming from the USRP that it should eventually backpressure all the way to the Radio which will then generate Overflows because it has no place to send the A/D data. So, if I was only seeing "O", it would make sense to me. But, the "D" makes no sense to me in my point-to-point direct connection between host and USRP. Do you know of any root cause for "D"? Thanks. Rob *DPDK error messages during dpdk-devbind.py* irisheyes0@irisheyes0-HP-Z440-Workstation:~$ /usr/share/dpdk/usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:00:19.0 'Ethernet Connection (2) I218-LM 15a0' if=eno1 drv=e1000e unused= *Active* 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens4f0 drv=ixgbe unused= 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens4f1 drv=ixgbe unused= 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens2f0 drv=ixgbe unused= 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens2f1 drv=ixgbe unused= Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> irisheyes0@irisheyes0-HP-Z440-Workstation:~$ sudo /usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci 01:00.0 [sudo] password for irisheyes0: Error - no supported modules(DPDK driver) are loaded irisheyes0@irisheyes0-HP-Z440-Workstation:~$ /usr/share/dpdk/usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:00:19.0 'Ethernet Connection (2) I218-LM 15a0' if=eno1 drv=e1000e unused= *Active* 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens4f0 drv=ixgbe unused= 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens4f1 drv=ixgbe unused= 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens2f0 drv=ixgbe unused= 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens2f1 drv=ixgbe unused= Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> irisheyes0@irisheyes0-HP-Z440-Workstation:~$ sudo modprobe vfio-pci irisheyes0@irisheyes0-HP-Z440-Workstation:~$ /usr/share/dpdk/usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:00:19.0 'Ethernet Connection (2) I218-LM 15a0' if=eno1 drv=e1000e unused=vfio-pci *Active* 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens4f0 drv=ixgbe unused=vfio-pci 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens4f1 drv=ixgbe unused=vfio-pci 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens2f0 drv=ixgbe unused=vfio-pci 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens2f1 drv=ixgbe unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> irisheyes0@irisheyes0-HP-Z440-Workstation:~$ sudo /usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci 01:00.0 Error: bind failed for 0000:01:00.0 - Cannot bind to driver vfio-pci Error: unbind failed for 0000:01:00.0 - Cannot open /sys/bus/pci/drivers //unbind irisheyes0@irisheyes0-HP-Z440-Workstation:~$ On Fri, Sep 6, 2019 at 6:02 PM Rob Kossler <rkoss...@nd.edu> wrote: > Hi Nate, > I'm using UHD 3.14.0.1. I am not using DPDK. > > Regarding the tuning, I think I was not clear in my email. I have no > trouble streaming to RAM disk using the standard Radio->DDC->host graph. I > mentioned that I was running 2x50MS/s, but I can go up to 2x200MS/s with > success. My issue is that after adding the DmaFIFO to the Rx chain, I got > timeouts (i.e., I suppose that the flow stopped for some reason) when > running the graph Radio->DDC->DmaFIFO->host. Even at 2x50MS/s. > > So, my question is: why is this happening? What is wrong with my plan to > insert the DmaFIFO in the Rx chain? What would possibly cause the > streaming to terminate such that my recv() loop times out (even with a 5s > timeout)? > > Rob > > > > On Fri, Sep 6, 2019 at 12:56 PM Ettus Research Support <supp...@ettus.com> > wrote: > >> Hi Rob, >> >> What version of UHD are you using? >> >> 2x RX 50 MS/s streams should work without much issue with a fast enough >> host, especially to a ram disk. >> >> Are you using DPDK? DPDK support for X3xx was recently added to UHD and >> will reduce the overhead on the host side, which can help quite a bit. Some >> anecdotal testing I've done with a N310, with the native UHD driver, to >> stream 2 channels full duplex, the minimum cpu freq I was able to run >> without any flow control errors was 3.8 GHz. Using DPDK, I was able to run >> 2x2 @ 125 MS/s with my CPU cores locked at 1.5 GHz with no flow control >> errors. Using DPDK, it's possible to stream 2x2 @ 200e6 on the X3xx with a >> SRAM FPGA image (it's not possible to TX at full rate using the native >> driver and DRAM based FPGA). >> >> You could try the few things listed here >> https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks >> >> One other bit to add, I've been able to stream 1 RX channel @ 200 MS/s >> straight to disk using a Intel 750 Series PCIe SSD until it was full (circa >> UHD 3.10.x). To do that, I had to use a sc16 host side data format and also >> use a XFS file system instead of EXT4. The host was a i7-4790k @ 4.4 GHz. I >> would recommend NVMe SSD drives now as they support faster rates than that >> PCIe SSD. >> >> >> Regards, >> Nate Temple >> >> On Fri, Sep 6, 2019 at 8:37 AM Rob Kossler via USRP-users < >> usrp-users@lists.ettus.com> wrote: >> >>> Hi, >>> As part of an effort to improve capability to store incoming receive >>> chain samples to files on my SSD without errors ('O' or 'D'), I decided to >>> wire an X310 noc graph to include the DmaFIFO. My thought was that the >>> DmaFIFO could better tolerate varying rates of sample consumption at the >>> OS. >>> >>> Before trying this by streaming to a file on my SSD, I first ran a test >>> which streamed to a RAM-based file (60 GB ram filesystem). I used an >>> X310/UBX160 with the default FPGA XG image and initiated a 2-channel >>> receive at 50MS/s (using my C++ app & UHD). To my surprise, I am getting >>> frequent "timeouts" on receive, but not always at the same time. In one >>> case, the receive worked successfully for 28 secs (2 ch, 50 MS/s). In >>> other cases, it timed out immediately or after several seconds. Note that >>> I can reliably run this same test without error if I remove the DmaFIFO. >>> >>> The following works fine: >>> RxRadio -> DDC -> host file (in RAM file system) >>> >>> The following times-out at random times: >>> RxRadio -> DDC -> DmaFIFO -> host file (in RAM file system) >>> >>> What could be the cause? Is there any reason the DmaFIFO shouldn't work >>> in the receive chain? >>> >>> Rob >>> _______________________________________________ >>> USRP-users mailing list >>> USRP-users@lists.ettus.com >>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>> >>
_______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com