Hi Guillermo, My experience with DPDK was the following: - very difficult to configure - once configured properly, streaming was exceptional (significant improvement compared to no DPDK). I was using Intel cards (likely X520-DA2 and XL710-QDA1). - lots of bad side effects related to DPDK taking over a dedicated CPU. I'm not sure why but at times even the mouse and keyboard would get so slow that the system became unusable - it occurred to me that maybe the mouse/keyboard were mistakenly being serviced by the CPU that was dedicated to DPDK - very strange. And, it wasn't just the mouse - other applications could be affected. The side effects were severe enough for me to abandon using DPDK on a regular basis. That said, I would consider it in your situation where you needed the performance.
I never experienced the troubles you are having where DPDK is working but performing poorly. In my case, if it worked at all, the performance was good. Sorry that I don't have any good suggestions for you. Whenever I am trying to capture high data rate receive streams, I save the data to files in a RAM file system. If I need these files permanently stored, I have had best success with running a separate utility that moves the saved files from RAM to storage. It doesn't seem like this approach should be necessary (as compared to multi threads within one process and/or keeping the data in shared memory rather than buffering in 'files') but perhaps my attempts at other approaches were faulty. Rob On Thu, Nov 18, 2021 at 5:03 AM Guillermo Ortas Delgado <g.or...@gmv.com> wrote: > Hi Marcus, > > > > Wow, good to know that I’m doing good. I thought I wasn’t the only one > working at these high rates. > > I guess there is the possibility to spread the load on another computer, > but I was interested in DPDK as it showed potential to accomplish my goal > more elegantly instead of just throwing more money at the problem. > > > > I’m still puzzled by DPDK dropping samples at even 25Msps on 2 channels. > > *@Rob Kossler: do you have any more input on this matter?* > > > > Best, > > Guillermo > > > > *De:* Marcus D. Leech [mailto:patchvonbr...@gmail.com] > *Enviado el:* 17 November 2021 17:43 > *Para:* Guillermo Ortas Delgado <g.or...@gmv.com>; Rob Kossler < > rkoss...@nd.edu> > *CC:* usrp-users@lists.ettus.com > *Asunto:* Re: [USRP-users] Re: DPDK drops samples at low rates > > > > On 2021-11-17 11:27, Guillermo Ortas Delgado wrote: > > Hi Marcus, thank you for your message. > > > > I do think the network layer is indeed the most significant part of this > challenging setup, let me illustrate: > > The platform I’m using is a server form-factor computer with a Xeon Silver > 4215R, 32GB of memory and no GPU. For storage, I have 4 SSDs of 3.2TB > mounted in RAID 0 providing a write capacity of 4GB/s, so that’s no issue. > I’m using all the optimizations mentioned here > <https://urldefense.com/v3/__https:/kb.ettus.com/Getting_Started_with_DPDK_and_UHD__;!!MvyJQugb!UJ8FfsPbUrq0LfNpjFLHfw9YyY1nkelByXNeHL_5zu_b0NDpEm1rMWVt0ZYW$> > and here > <https://urldefense.com/v3/__https:/kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks__;!!MvyJQugb!UJ8FfsPbUrq0LfNpjFLHfw9YyY1nkelByXNeHL_5zu_b0NDpEm1rMVnRj1fb$>, > so raw CPU power I think should be enough. > > Each of the four x310 USRPs is attached to a dedicated network card with > dual 10GbE ports, so network capacity is theoretically more than enough > (6.4Gb out of 10Gb maximum usage per interface). > > The scheme I’m trying to run is as follows: > > 1. USRP 1: 200 Msps on both channels > > 2. USRP 2: 200 Msps on both channels > > 3. USRP 3: 50 Msps on both channels > > 4. USRP 4: 50 Msps on both channels > > I’m streaming samples directly into shared memory, from which a separate > thread converts the samples to selectable 1/2/4/8/16 bits per sample and > stores them to the RAID 0 disk volume. Bit depth conversion is fast and > doesn’t seem to be the bottleneck. In fact, converting to 4 bits per sample > achieves better results than no conversion at all, forcing me to write the > full incoming 16 bits per sample. I launch a separate instance of my > program to store samples for each USRP, as I have observed this delivers > the best performance. > > With this, I’m able to run 4 channels at 200Msps and 2 channels at 50Msps. > But when I launch the last two channels at 50Msps the system can’t keep up > and the recording starts losing/dropping samples. > > I was able to run 4 channels at 184.32Msps and 4 channels at 46.08Msps for > a few seconds, but this is also not sustainable and samples are dropped > periodically. > > The application is very sensitive, so even a single sample lost or dropped > completely invalidates the recording. > > > > At these rates, the sheer amount of kernel systems calls seems to be the > most significant performance hit, that’s why I was looking at DPDK as a > potential solution. That being said, I’m able to sustain a solid stream > using the benchmark_rate program (discarding the samples) with 4 channels > at 200Msps and 4 channels at 50Msps without any drops/overflows. > > Do you have a sample program for high-performance/high-rate sample > streaming? The provided rx_samples_to_file is not nearly enough. > > What’s the preferred way to approach storing samples for maximum > performance? > > > > Thank you a lot and best regards, > > Guillermo > > 1Gsps is a totally *eye-watering* sample-rate for an ordinary computer to > "swallow" and write to disk. You are very likely at the very bleeding edge > with this, and > I'm not aware of anyone else doing work at these aggregate rates. The > fact that you are able to *both* "do stuff with the samples" AND write them > to a RAID > array at ~1Gsps is amazing. > > Do all USRPs have to be on the same computer for your application? Are > there opportunities to use a more distributed approach? > > > > *De:* Marcus D. Leech [mailto:patchvonbr...@gmail.com > <patchvonbr...@gmail.com>] > *Enviado el:* 17 November 2021 16:51 > *Para:* Guillermo Ortas Delgado <g.or...@gmv.com> <g.or...@gmv.com>; Rob > Kossler <rkoss...@nd.edu> <rkoss...@nd.edu> > *CC:* usrp-users@lists.ettus.com > *Asunto:* Re: [USRP-users] Re: DPDK drops samples at low rates > > > > On 2021-11-17 04:50, Guillermo Ortas Delgado wrote: > > Thanks for your message, I already have the mbuf size maxed out to 512 > (that’s the maximum value it will take). > > > > I have noticed that DPDPK v19.11 made great improvements to the BNXT > driver. Is there any chance to get UHD running with DPDK 19.11? Or even > better 20.11.3? > > Both are long-term support releases which are more mature and support > vector mode, which offers must better performance. > > Quote from DPDK 19.11: > “The BNXT PMD includes support for SSE vector mode on x86 platforms. > Vector provides *significantly improved performance* over the base > implementation” > > > > I already tried building UHD 4.1.0.4 with DPDK 19.11 by modifying the > makefile to accept this version, but the build fails. > > I would really appreciate it if you could add support for newer versions > of DPDK. > > > > Best, > > Guillermo > > > > > > P Please consider the environment before printing this e-mail. > > > > P Please consider the environment before printing this e-mail. >
_______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com To unsubscribe send an email to usrp-users-le...@lists.ettus.com
