Hi Keith,
Thank you very much for your advice. I will try to redesign the access to the mempool's. Regards, Andrei Comandatu On 02/26/2018 05:08 PM, Wiles, Keith wrote: > >> On Feb 26, 2018, at 8:36 AM, andrei <[email protected]> wrote: >> >> Hi, >> >> >> I had run into a deadlock by using DPDK, and I am out of ideas on how to >> debug the issue. >> >> >> Scenario: (The application is more complicated this is the simplified >> version) >> >> 4 mempools, 4 txBuffers, 3 threads, (CPU 4 cores; irrelevant). >> >> One thread extracts buffers(rte_mbuff) from a random memory(rte_mempool) >> pool and place them into a ring buffer. >> >> Other thread (Sender) extracts the buffers from the ring populates them >> with data and place them into a rte_eth_dev_tx_buffer by calling >> rte_eth_tx_buffer(); >> >> One thread (Flusher) is used as a flusher. It goes trough the >> rte_eth_dev_tx_buffer's and calls rte_eth_tx_buffer_flush. >> >> >> The deadlock occurs, in my opinion, when the Sender and the Flusher >> threads try to place back buffers into the same memory pool. >> >> This is a fragment of the core dump. >> >> >> Thread 2 (Thread 0x7f5932e69700 (LWP 14014)): >> #0 0x00007f59388e933a in common_ring_mp_enqueue () from >> /usr/local/lib/librte_mempool.so.2.1 >> #1 0x00007f59386b27e0 in ixgbe_xmit_pkts () from >> /usr/local/lib/librte_pmd_ixgbe.so.1.1 >> #2 0x00007f593d00aab7 in rte_eth_tx_burst (nb_pkts=<optimized out>, >> tx_pkts=<optimized out>, >> queue_id=0, port_id=<optimized out>) at >> /usr/local/include/dpdk/rte_ethdev.h:2858 >> #3 rte_eth_tx_buffer_flush (buffer=<optimized out>, queue_id=0, >> port_id=<optimized out>) >> at /usr/local/include/dpdk/rte_ethdev.h:3040 >> #4 rte_eth_tx_buffer (tx_pkt=<optimized out>, buffer=<optimized out>, >> queue_id=0, >> port_id=<optimized out>) at /usr/local/include/dpdk/rte_ethdev.h:3090 >> >> >> Thread 30 (Thread 0x7f5933175700 (LWP 13958)): >> #0 0x00007f59388e91cc in common_ring_mp_enqueue () from >> /usr/local/lib/librte_mempool.so.2.1 >> #1 0x00007f59386b27e0 in ixgbe_xmit_pkts () from >> /usr/local/lib/librte_pmd_ixgbe.so.1.1 >> #2 0x00007f593d007dfd in rte_eth_tx_burst (nb_pkts=<optimized out>, >> tx_pkts=0x7f587a410358, >> queue_id=<optimized out>, port_id=0 '\000') at >> /usr/local/include/dpdk/rte_ethdev.h:2858 >> #3 rte_eth_tx_buffer_flush (buffer=0x7f587a410340, queue_id=<optimized >> out>, port_id=0 '\000') >> at /usr/local/include/dpdk/rte_ethdev.h:3040 >> >> >> Questions: >> >> 1. I am using DPDK 17.02.01. Was a bug solved in newer releases >> that can be mapped to this behavior? >> >> 2. If two threads try to place buffers into the same pool, the >> operation should be synchronized by DPDK or by application? > Not sure this will help, but if you are running more then one thread per core > then you can have problems. I assume you have a mempool cache here and the > per lcore caches are not thread safe (If I remember correctly), the main > cache in the mempool is thread safe. If you are not using multiple threads > per lcore, then I expect something else is going on here as this is the > normal mode of operation for mempools. If you turn off the mempool caches you > should not see the problem, but then your performance will drop some. > > I had a similar problem and made sure only one thread puts or gets buffers > from the same mempool. > >> >> Thank you, >> >> Andrei Comandatu >> >> > Regards, > Keith >
