The following log message may appear after a slave is idle (or nearly idle) for a few minutes: "PMD: Failed to allocate LACP packet from pool".
Problem: All mbufs from a slave's private pool (used exclusively for transmitting LACPDUs) have been allocated and are still sitting in the device's tx descriptor ring and other cores' mempool caches. Solution: Ensure that each slaves' tx (LACPDU) mempool owns more than n-tx-queues * (n-tx-descriptors + per-core-mempool-flush-threshold) mbufs. Note that the LACP tx machine function is the only code that allocates from a slave's private pool. It runs in the context of the interrupt thread, and thus it has no mempool cache of its own. Signed-off-by: Robert Sanford <rsanford at akamai.com> --- drivers/net/bonding/rte_eth_bond_8023ad.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/drivers/net/bonding/rte_eth_bond_8023ad.c b/drivers/net/bonding/rte_eth_bond_8023ad.c index 2f7ae70..1207896 100644 --- a/drivers/net/bonding/rte_eth_bond_8023ad.c +++ b/drivers/net/bonding/rte_eth_bond_8023ad.c @@ -854,6 +854,8 @@ bond_mode_8023ad_activate_slave(struct rte_eth_dev *bond_dev, uint8_t slave_id) char mem_name[RTE_ETH_NAME_MAX_LEN]; int socket_id; unsigned element_size; + unsigned cache_size; + unsigned cache_flushthresh; uint32_t total_tx_desc; struct bond_tx_queue *bd_tx_q; uint16_t q_id; @@ -890,19 +892,21 @@ bond_mode_8023ad_activate_slave(struct rte_eth_dev *bond_dev, uint8_t slave_id) element_size = sizeof(struct slow_protocol_frame) + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM; + cache_size = RTE_MEMPOOL_CACHE_MAX_SIZE >= 32 ? + 32 : RTE_MEMPOOL_CACHE_MAX_SIZE; + cache_flushthresh = RTE_MEMPOOL_CALC_CACHE_FLUSHTHRESH(cache_size); /* The size of the mempool should be at least: * the sum of the TX descriptors + BOND_MODE_8023AX_SLAVE_TX_PKTS */ total_tx_desc = BOND_MODE_8023AX_SLAVE_TX_PKTS; for (q_id = 0; q_id < bond_dev->data->nb_tx_queues; q_id++) { bd_tx_q = (struct bond_tx_queue*)bond_dev->data->tx_queues[q_id]; - total_tx_desc += bd_tx_q->nb_tx_desc; + total_tx_desc += bd_tx_q->nb_tx_desc + cache_flushthresh; } snprintf(mem_name, RTE_DIM(mem_name), "slave_port%u_pool", slave_id); port->mbuf_pool = rte_mempool_create(mem_name, - total_tx_desc, element_size, - RTE_MEMPOOL_CACHE_MAX_SIZE >= 32 ? 32 : RTE_MEMPOOL_CACHE_MAX_SIZE, + total_tx_desc, element_size, cache_size, sizeof(struct rte_pktmbuf_pool_private), rte_pktmbuf_pool_init, NULL, rte_pktmbuf_init, NULL, socket_id, MEMPOOL_F_NO_SPREAD); -- 1.7.1