[dpdk-dev] KNI automatic IP configuration

[Olivier Deme]

Hi, sorry to resurrect this question.
I am now using a virtio network interface in a qemu VM.
Is it possible to use ifplugd to automatically setup the IP 
configuration of a KNI based interface?
Here is the output of lspci:
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:04.0 Ethernet controller: Red Hat, Inc Virtio network device
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
00:06.0 Ethernet controller: Red Hat, Inc Virtio network device

Thanks for your help.
Olivier.


On 05/04/15 02:08, Zhou, Danny wrote:
> e1000 driver supports many different NIC devices.
>
> Do you know your NIC device type or device_id?
>
>> -Original Message-
>> From: Olivier Deme [mailto:odeme at druidsoftware.com]
>> Sent: Sunday, April 05, 2015 1:03 AM
>> To: Zhou, Danny
>> Cc: dev at dpdk.org
>> Subject: Re: [dpdk-dev] KNI automatic IP configuration
>>
>> Currently I?m using the e1000 PMD driver.
>>
>> Thanks,
>> Olivier.
>>
>>> On 4 Apr 2015, at 12:51, Zhou, Danny  wrote:
>>>
>>> Which NIC device are you using now?
>>>
>>> KNI does not provide Ethtool support for all NICs supported by the 
>>> Poll-Mode Drivers.
>>>
>>>> -Original Message-
>>>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Olivier Deme
>>>> Sent: Friday, April 03, 2015 10:03 PM
>>>> To: dev at dpdk.org
>>>> Subject: [dpdk-dev] KNI automatic IP configuration
>>>>
>>>> Hi all,
>>>>
>>>> Is it possible on Linux to get the Linux network manager to automatically 
>>>> configure a KNI virtual network interface as soon as it
>>>> comes up?
>>>> I haven?t been able to get Fedora to apply the configuration under 
>>>> /etc/sysconfig/network-scripts/ifcfg-veth0 file upon
>> starting
>>>> my DPDK application.
>>>>
>>>> Executing ifup veth0 manually works but somehow the fedora network manager 
>>>> seems to have an issue to apply the if cfg
>> config
>>>> automatically.
>>>> In the system logs I can see something along the lines ?kni doesn?t 
>>>> support ethtool?. Is this related?
>>>>
>>>> I also tried ifplugd but without success.
>>>>
>>>> Many thanks for your help,
>>>>
>>>> Olivier.

-- 
*Olivier Dem?*
*Druid Software Ltd.*
*Tel: +353 1 202 1831*
*Email: odeme at druidsoftware.com <mailto:odeme at druidsoftware.com>*
*URL: http://www.druidsoftware.com*
Druid Software: Monetising enterprise small cells solutions.


Druid_Footer_Logo

[dpdk-dev] freeze with dpdk-2.0.0

[Olivier Deme]

Hi Changchun,

I confirm that blacklisting the first two network interfaces solve the 
problem.
Just to be clear, is it the case that it is not possible to have a DPDK 
application acquiring selectively virtio network devices without using 
the blacklisting (-b) option?

It seems a bit clumsy to have the application being configured with all 
network interfaces to be bound to DPDK.

Thanks,
Olivier.



On 15/04/15 04:45, Ouyang, Changchun wrote:
> Hi Oliver
>
>> -Original Message-
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Olivier Deme
>> Sent: Wednesday, April 15, 2015 1:16 AM
>> To: dev at dpdk.org
>> Subject: [dpdk-dev] freeze with dpdk-2.0.0
>>
>> Hi,
>>
>> I am trying to use DPDK-2.0.0 on Fedora 20 running in a qemu virtual machine.
>> After loading the uio and igb_uio module and setting up hugepages, I try to
>> run the helloworld demo application.
>> As soon as the helloworld displays the "hello from core 1, hello from core 0"
>> messages, I loose all network connectivity to the VM.
>>
>> If I try to run the helloworld application remotely the output freezes after
>> the following:
>> # ./build/helloworld -c 3 -n 2
>> EAL: Detected lcore 0 as core 0 on socket 0
>> EAL: Detected lcore 1 as core 0 on socket 0
>> EAL: Detected lcore 2 as core 0 on socket 0
>> EAL: Detected lcore 3 as core 0 on socket 0
>> EAL: Support maximum 128 logical core(s) by configuration.
>> EAL: Detected 4 lcore(s)
>> EAL: VFIO modules not all loaded, skip VFIO support...
>> EAL: Setting up memory...
>> EAL: Ask a virtual area of 0x700 bytes
>> EAL: Virtual area found at 0x7f273be0 (size = 0x700)
>> EAL: Ask a virtual area of 0x20 bytes
>> EAL: Virtual area found at 0x7f273ba0 (size = 0x20)
>> EAL: Ask a virtual area of 0x20 bytes
>> EAL: Virtual area found at 0x7f273b60 (size = 0x20)
>> EAL: Ask a virtual area of 0xc0 bytes
>> EAL: Virtual area found at 0x7f273a80 (size = 0xc0)
>> EAL: Requesting 64 pages of size 2MB from socket 0
>> EAL: TSC frequency is ~1995193 KHz
>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>> unreliable clock cycles !
>> EAL: Master lcore 0 is ready (tid=449e9900;cpuset=[0])
>> PMD: ENICPMD trace: rte_enic_pmd_init
>> EAL: lcore 1 is ready (tid=3a7ff700;cpuset=[1])
>>
>>
>> If I try to restart the network services, I get the error from syslog:
>> BUG: soft lockup - CPU#2 stuck for 22s!
>>
>>
>> The network devices exposed to the VM are 2 BCM5719 interfaces + 2
>> 82599ES interfaces:
>> 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719
>> Gigabit Ethernet PCIe (rev 01)
>> 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719
>> Gigabit Ethernet PCIe (rev 01)
>> 0d:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
>> Network Connection (rev 01)
>> 0d:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
>> Network Connection (rev 01)
>>
>>
>> Within the VM, lspci shows:
>> 00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
>> 00:04.0 Ethernet controller: Red Hat, Inc Virtio network device
>> 00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
>> 00:06.0 Ethernet controller: Red Hat, Inc Virtio network device
>>
>>
>> After loading the kernel modules, I bind the interfaces to DPDK with:
>> /root/dpdk-2.0.0/tools/dpdk_nic_bind.py --bind=igb_uio 00:05.0
>> /root/dpdk-2.0.0/tools/dpdk_nic_bind.py --bind=igb_uio 00:06.0
>>
>> Would you know what is causing this?
>>
> Maybe you can try with "--b 00:03.0 00:04.0" to force dpdk not to grab these 
> 2 virtio devices
> To see if this issue is resolved.
>
> For virtio dev, dpdk doesn't depend on uio/igb_uio any more after the single 
> virtio,
> It will also grab the virtio device to use it even without uio module except 
> for you specify it into black-list.
>
> Thanks
> Changchun
>

-- 
*Olivier Dem?*
*Druid Software Ltd.*
*Tel: +353 1 202 1831*
*Email: odeme at druidsoftware.com <mailto:odeme at druidsoftware.com>*
*URL: http://www.druidsoftware.com*
Druid Software: Monetising enterprise small cells solutions.


Druid_Footer_Logo

[dpdk-dev] freeze with dpdk-2.0.0

[Olivier Deme]

Hi,

I am trying to use DPDK-2.0.0 on Fedora 20 running in a qemu virtual 
machine.
After loading the uio and igb_uio module and setting up hugepages, I try 
to run the helloworld demo application.
As soon as the helloworld displays the "hello from core 1, hello from 
core 0" messages, I loose all network connectivity to the VM.

If I try to run the helloworld application remotely the output freezes 
after the following:
# ./build/helloworld -c 3 -n 2
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up memory...
EAL: Ask a virtual area of 0x700 bytes
EAL: Virtual area found at 0x7f273be0 (size = 0x700)
EAL: Ask a virtual area of 0x20 bytes
EAL: Virtual area found at 0x7f273ba0 (size = 0x20)
EAL: Ask a virtual area of 0x20 bytes
EAL: Virtual area found at 0x7f273b60 (size = 0x20)
EAL: Ask a virtual area of 0xc0 bytes
EAL: Virtual area found at 0x7f273a80 (size = 0xc0)
EAL: Requesting 64 pages of size 2MB from socket 0
EAL: TSC frequency is ~1995193 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using 
unreliable clock cycles !
EAL: Master lcore 0 is ready (tid=449e9900;cpuset=[0])
PMD: ENICPMD trace: rte_enic_pmd_init
EAL: lcore 1 is ready (tid=3a7ff700;cpuset=[1])


If I try to restart the network services, I get the error from syslog:
BUG: soft lockup - CPU#2 stuck for 22s!


The network devices exposed to the VM are 2 BCM5719 interfaces + 2 
82599ES interfaces:
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 
Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 
Gigabit Ethernet PCIe (rev 01)
0d:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
0d:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)


Within the VM, lspci shows:
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:04.0 Ethernet controller: Red Hat, Inc Virtio network device
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
00:06.0 Ethernet controller: Red Hat, Inc Virtio network device


After loading the kernel modules, I bind the interfaces to DPDK with:
/root/dpdk-2.0.0/tools/dpdk_nic_bind.py --bind=igb_uio 00:05.0
/root/dpdk-2.0.0/tools/dpdk_nic_bind.py --bind=igb_uio 00:06.0

Would you know what is causing this?

I haven't been able to try DPDK 1.8.0 because it doesn't compile on the 
latest Fedora Server 20.

Thanks for your help,
Olivier.


-- 
*Olivier Dem?*
*Druid Software Ltd.*
*Tel: +353 1 202 1831*
*Email: odeme at druidsoftware.com *
*URL: http://www.druidsoftware.com*
Druid Software: Monetising enterprise small cells solutions.


Druid_Footer_Logo

[dpdk-dev] KNI automatic IP configuration

[Olivier Deme]

Currently I?m using the e1000 PMD driver.

Thanks,
Olivier.

> On 4 Apr 2015, at 12:51, Zhou, Danny  wrote:
> 
> Which NIC device are you using now?
> 
> KNI does not provide Ethtool support for all NICs supported by the Poll-Mode 
> Drivers.
> 
>> -Original Message-
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Olivier Deme
>> Sent: Friday, April 03, 2015 10:03 PM
>> To: dev at dpdk.org
>> Subject: [dpdk-dev] KNI automatic IP configuration
>> 
>> Hi all,
>> 
>> Is it possible on Linux to get the Linux network manager to automatically 
>> configure a KNI virtual network interface as soon as it
>> comes up?
>> I haven?t been able to get Fedora to apply the configuration under 
>> /etc/sysconfig/network-scripts/ifcfg-veth0 file upon starting
>> my DPDK application.
>> 
>> Executing ifup veth0 manually works but somehow the fedora network manager 
>> seems to have an issue to apply the if cfg config
>> automatically.
>> In the system logs I can see something along the lines ?kni doesn?t support 
>> ethtool?. Is this related?
>> 
>> I also tried ifplugd but without success.
>> 
>> Many thanks for your help,
>> 
>> Olivier.
> 

[dpdk-dev] KNI automatic IP configuration

[Olivier Deme]

Hi all,

Is it possible on Linux to get the Linux network manager to automatically 
configure a KNI virtual network interface as soon as it comes up?
I haven?t been able to get Fedora to apply the configuration under 
/etc/sysconfig/network-scripts/ifcfg-veth0 file upon starting my DPDK 
application.

Executing ifup veth0 manually works but somehow the fedora network manager 
seems to have an issue to apply the if cfg config automatically.
In the system logs I can see something along the lines ?kni doesn?t support 
ethtool?. Is this related?

I also tried ifplugd but without success.

Many thanks for your help,

Olivier.

[dpdk-dev] Need some info on DPDK

[Olivier Deme]

It's all in the docs.
http://dpdk.org/doc/guides/prog_guide/libpcap_ring_based_poll_mode_drv.html
http://dpdk.org/doc/api/rte__ethdev_8h.html#a5686df2817980236f2c4f1cc72dd2c30



On 23/03/15 09:00, Shankari Vaidyalingam wrote:
> Hi
>
> Need some info on:
> (1) How to capture packets received on the DPDK interface?
> (2) Determine the MAC address corresponding to the PCI interface
>
>
> Regards
> Shankari.V

-- 
*Olivier Dem?*
*Druid Software Ltd.*
*Tel: +353 1 202 1831*
*Email: odeme at druidsoftware.com *
*URL: http://www.druidsoftware.com*
Druid Software: Monetising enterprise small cells solutions.

[dpdk-dev] [PATCH] kni:optimization of rte_kni_rx_burst

[Olivier Deme]

Hi Marc,

I think one of the observations is that currently the alloc_q grows very 
quickly to the maximum fifo size (1024).
The patch suggests fixing the alloc_q to a fix size and maybe make that 
size configurable in rte_kni_alloc or rte_kni_init.

It should then be up to the application to provision the mempool 
accordingly.
Currently the out of memory problem shows up if the mempool doesn't have 
1024 buffers per KNI.

Olivier.

On 25/02/15 12:38, Marc Sune wrote:
>
> On 25/02/15 13:24, Hemant at freescale.com wrote:
>> Hi OIivier
>>  Comments inline.
>> Regards,
>> Hemant
>>
>>> -Original Message-
>>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Olivier Deme
>>> Sent: 25/Feb/2015 5:44 PM
>>> To: dev at dpdk.org
>>> Subject: Re: [dpdk-dev] [PATCH] kni:optimization of rte_kni_rx_burst
>>>
>>> Thank you Hemant, I think there might be one issue left with the 
>>> patch though.
>>> The alloc_q must initially be filled with mbufs before getting mbuf 
>>> back on the
>>> tx_q.
>>>
>>> So the patch should allow rte_kni_rx_burst to check if alloc_q is 
>>> empty.
>>> If so, it should invoke kni_allocate_mbufs(kni, 0) (to fill the 
>>> alloc_q with
>>> MAX_MBUF_BURST_NUM mbufs)
>>>
>>> The patch for rte_kni_rx_burst would then look like:
>>>
>>> @@ -575,7 +575,7 @@ rte_kni_rx_burst(struct rte_kni *kni, struct 
>>> rte_mbuf
>>> **mbufs, unsigned num)
>>>
>>>/* If buffers removed, allocate mbufs and then put them into 
>>> alloc_q */
>>>if (ret)
>>> -kni_allocate_mbufs(kni);
>>> +  kni_allocate_mbufs(kni, ret);
>>> +  else if (unlikely(kni->alloc_q->write == kni->alloc_q->read))
>>> +  kni_allocate_mbufs(kni, 0);
>>>
>> [hemant]  This will introduce a run-time check.
>>
>> I missed to include the other change in the patch.
>>   I am doing it in kni_alloc i.e. initiate the alloc_q with default 
>> burst size.
>> kni_allocate_mbufs(ctx, 0);
>>
>> In a way, we are now suggesting to reduce the size of alloc_q to only 
>> default burst size.
>
> As an aside comment here, I think that we should allow to tweak the 
> userspace <-> kernel queue sizes (rx_q, tx_q, free_q and alloc_q) . 
> Whether this should be a build configuration option or a parameter to 
> rte_kni_init(), it is not completely clear to me, but I guess 
> rte_kni_init() is a better option.
>
> Having said that, the original mail from Hemant was describing that 
> KNI was giving an out-of-memory. This to me indicates that the pool is 
> incorrectly dimensioned. Even if KNI will not pre-allocate in the 
> alloc_q, or not completely, in the event of high load, you will get 
> this same "out of memory".
>
> We can reduce the usage of buffers by the KNI subsystem in kernel 
> space and in userspace, but the kernel will always need a small cache 
> of pre-allocated buffers (coming from user-space), since the KNI 
> kernel module does not know where to grab the packets from (which 
> pool). So my guess is that the dimensioning problem experienced by 
> Hemant would be the same, even with the proposed changes.
>
>>
>> Can we reach is situation, when the kernel is adding packets faster 
>> in tx_q than the application is able to dequeue?
>
> I think so. We cannot control much how the kernel will schedule the 
> KNI thread(s), specially if the # of threads in relation to the cores 
> is incorrect (not enough), hence we need at least a reasonable amount 
> of buffering to prevent early dropping to those "internal" burst side 
> effects.
>
> Marc
>
>>   alloc_q  can be empty in this case and kernel will be striving.
>>
>>> Olivier.
>>>
>>> On 25/02/15 11:48, Hemant Agrawal wrote:
>>>> From: Hemant Agrawal 
>>>>
>>>> if any buffer is read from the tx_q, MAX_BURST buffers will be 
>>>> allocated and
>>> attempted to be added to to the alloc_q.
>>>> This seems terribly inefficient and it also looks like the alloc_q 
>>>> will quickly fill
>>> to its maximum capacity. If the system buffers are low in number, it 
>>> will reach
>>> "out of memory" situation.
>>>> This patch allocates the number of buffers as many dequeued from tx_q.
>>>>
>>>> Signed-off-by: Hemant Agrawal 
>>>> ---
>>>>lib/librte_kni/rte_kni.c | 13 -
>>>>1 file changed, 8 

[dpdk-dev] [PATCH] kni:optimization of rte_kni_rx_burst

[Olivier Deme]

I guess it would be unusual but possible for the kernel to enqueue 
faster to tx_q than the application dequeues.
But that would also be possible with a real NIC, so I think it is 
acceptable for the kernel to have to drop egress packets in that case.


On 25/02/15 12:24, Hemant at freescale.com wrote:
> Hi OIivier
>Comments inline.
> Regards,
> Hemant
>
>> -Original Message-
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Olivier Deme
>> Sent: 25/Feb/2015 5:44 PM
>> To: dev at dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH] kni:optimization of rte_kni_rx_burst
>>
>> Thank you Hemant, I think there might be one issue left with the patch 
>> though.
>> The alloc_q must initially be filled with mbufs before getting mbuf back on 
>> the
>> tx_q.
>>
>> So the patch should allow rte_kni_rx_burst to check if alloc_q is empty.
>> If so, it should invoke kni_allocate_mbufs(kni, 0) (to fill the alloc_q with
>> MAX_MBUF_BURST_NUM mbufs)
>>
>> The patch for rte_kni_rx_burst would then look like:
>>
>> @@ -575,7 +575,7 @@ rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf
>> **mbufs, unsigned num)
>>
>>/* If buffers removed, allocate mbufs and then put them into alloc_q 
>> */
>>if (ret)
>> -kni_allocate_mbufs(kni);
>> +  kni_allocate_mbufs(kni, ret);
>> +  else if (unlikely(kni->alloc_q->write == kni->alloc_q->read))
>> +  kni_allocate_mbufs(kni, 0);
>>
> [hemant]  This will introduce a run-time check.
>
> I missed to include the other change in the patch.
>   I am doing it in kni_alloc i.e. initiate the alloc_q with default burst 
> size.
>   kni_allocate_mbufs(ctx, 0);
>
> In a way, we are now suggesting to reduce the size of alloc_q to only default 
> burst size.
>
> Can we reach is situation, when the kernel is adding packets faster in tx_q 
> than the application is able to dequeue?
>   alloc_q  can be empty in this case and kernel will be striving.
>
>> Olivier.
>>
>> On 25/02/15 11:48, Hemant Agrawal wrote:
>>> From: Hemant Agrawal 
>>>
>>> if any buffer is read from the tx_q, MAX_BURST buffers will be allocated and
>> attempted to be added to to the alloc_q.
>>> This seems terribly inefficient and it also looks like the alloc_q will 
>>> quickly fill
>> to its maximum capacity. If the system buffers are low in number, it will 
>> reach
>> "out of memory" situation.
>>> This patch allocates the number of buffers as many dequeued from tx_q.
>>>
>>> Signed-off-by: Hemant Agrawal 
>>> ---
>>>lib/librte_kni/rte_kni.c | 13 -
>>>1 file changed, 8 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/lib/librte_kni/rte_kni.c b/lib/librte_kni/rte_kni.c index
>>> 4e70fa0..4cf8e30 100644
>>> --- a/lib/librte_kni/rte_kni.c
>>> +++ b/lib/librte_kni/rte_kni.c
>>> @@ -128,7 +128,7 @@ struct rte_kni_memzone_pool {
>>>
>>>
>>>static void kni_free_mbufs(struct rte_kni *kni); -static void
>>> kni_allocate_mbufs(struct rte_kni *kni);
>>> +static void kni_allocate_mbufs(struct rte_kni *kni, int num);
>>>
>>>static volatile int kni_fd = -1;
>>>static struct rte_kni_memzone_pool kni_memzone_pool = { @@ -575,7
>>> +575,7 @@ rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf
>>> **mbufs, unsigned num)
>>>
>>> /* If buffers removed, allocate mbufs and then put them into alloc_q
>> */
>>> if (ret)
>>> -   kni_allocate_mbufs(kni);
>>> +   kni_allocate_mbufs(kni, ret);
>>>
>>> return ret;
>>>}
>>> @@ -594,7 +594,7 @@ kni_free_mbufs(struct rte_kni *kni)
>>>}
>>>
>>>static void
>>> -kni_allocate_mbufs(struct rte_kni *kni)
>>> +kni_allocate_mbufs(struct rte_kni *kni, int num)
>>>{
>>> int i, ret;
>>> struct rte_mbuf *pkts[MAX_MBUF_BURST_NUM]; @@ -620,7 +620,10
>> @@
>>> kni_allocate_mbufs(struct rte_kni *kni)
>>> return;
>>> }
>>>
>>> -   for (i = 0; i < MAX_MBUF_BURST_NUM; i++) {
>>> +   if (num == 0 || num > MAX_MBUF_BURST_NUM)
>>> +   num = MAX_MBUF_BURST_NUM;
>>> +
>>> +   for (i = 0; i < num; i++) {
>>> pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
>>> if (unlikely(pkts[i] == NULL)) {
>>> /* Out of me

[dpdk-dev] [PATCH] kni:optimization of rte_kni_rx_burst

[Olivier Deme]

Thank you Hemant, I think there might be one issue left with the patch 
though.
The alloc_q must initially be filled with mbufs before getting mbuf back 
on the tx_q.

So the patch should allow rte_kni_rx_burst to check if alloc_q is empty.
If so, it should invoke kni_allocate_mbufs(kni, 0)
(to fill the alloc_q with MAX_MBUF_BURST_NUM mbufs)

The patch for rte_kni_rx_burst would then look like:

@@ -575,7 +575,7 @@ rte_kni_rx_burst(struct rte_kni *kni, struct 
rte_mbuf **mbufs, unsigned num)

  /* If buffers removed, allocate mbufs and then put them into 
alloc_q */
  if (ret)
-kni_allocate_mbufs(kni);
+  kni_allocate_mbufs(kni, ret);
+  else if (unlikely(kni->alloc_q->write == kni->alloc_q->read))
+  kni_allocate_mbufs(kni, 0);


Olivier.

On 25/02/15 11:48, Hemant Agrawal wrote:
> From: Hemant Agrawal 
>
> if any buffer is read from the tx_q, MAX_BURST buffers will be allocated and 
> attempted to be added to to the alloc_q.
> This seems terribly inefficient and it also looks like the alloc_q will 
> quickly fill to its maximum capacity. If the system buffers are low in 
> number, it will reach "out of memory" situation.
>
> This patch allocates the number of buffers as many dequeued from tx_q.
>
> Signed-off-by: Hemant Agrawal 
> ---
>   lib/librte_kni/rte_kni.c | 13 -
>   1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/lib/librte_kni/rte_kni.c b/lib/librte_kni/rte_kni.c
> index 4e70fa0..4cf8e30 100644
> --- a/lib/librte_kni/rte_kni.c
> +++ b/lib/librte_kni/rte_kni.c
> @@ -128,7 +128,7 @@ struct rte_kni_memzone_pool {
>   
>   
>   static void kni_free_mbufs(struct rte_kni *kni);
> -static void kni_allocate_mbufs(struct rte_kni *kni);
> +static void kni_allocate_mbufs(struct rte_kni *kni, int num);
>   
>   static volatile int kni_fd = -1;
>   static struct rte_kni_memzone_pool kni_memzone_pool = {
> @@ -575,7 +575,7 @@ rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf 
> **mbufs, unsigned num)
>   
>   /* If buffers removed, allocate mbufs and then put them into alloc_q */
>   if (ret)
> - kni_allocate_mbufs(kni);
> + kni_allocate_mbufs(kni, ret);
>   
>   return ret;
>   }
> @@ -594,7 +594,7 @@ kni_free_mbufs(struct rte_kni *kni)
>   }
>   
>   static void
> -kni_allocate_mbufs(struct rte_kni *kni)
> +kni_allocate_mbufs(struct rte_kni *kni, int num)
>   {
>   int i, ret;
>   struct rte_mbuf *pkts[MAX_MBUF_BURST_NUM];
> @@ -620,7 +620,10 @@ kni_allocate_mbufs(struct rte_kni *kni)
>   return;
>   }
>   
> - for (i = 0; i < MAX_MBUF_BURST_NUM; i++) {
> + if (num == 0 || num > MAX_MBUF_BURST_NUM)
> + num = MAX_MBUF_BURST_NUM;
> +
> + for (i = 0; i < num; i++) {
>   pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
>   if (unlikely(pkts[i] == NULL)) {
>   /* Out of memory */
> @@ -636,7 +639,7 @@ kni_allocate_mbufs(struct rte_kni *kni)
>   ret = kni_fifo_put(kni->alloc_q, (void **)pkts, i);
>   
>   /* Check if any mbufs not put into alloc_q, and then free them */
> - if (ret >= 0 && ret < i && ret < MAX_MBUF_BURST_NUM) {MAX_MBUF_BURST_NUM
>
> + if (ret >= 0 && ret < i && ret < num) {
>   int j;
>   
>   for (j = ret; j < i; j++)

-- 
*Olivier Dem?*
*Druid Software Ltd.*
*Tel: +353 1 202 1831*
*Email: odeme at druidsoftware.com *
*URL: http://www.druidsoftware.com*
*Hall 7, stand 7F70.*
Druid Software: Monetising enterprise small cells solutions.

[dpdk-dev] rte_kni_rx_burst issues

[Olivier Deme]

All,

I know that an issue was already raised with regards to the efficiency 
of rte_kni_rx_burst but I think that there is more to it than previously 
discussed.

As previously pointed out rte_kni_rx_burst invokes kni_allocate_mbufs 
every single time.
In turn, it looks like kni_allocate_mbufs allocates 32 mbufs 
(MAX_MBUF_BURST_NUM) and attempt to enqueue these to the alloc_q.
If alloc_q is full (1024 buffers: KNI_FIFO_COUNT_MAX), 
kni_allocate_mbufs frees all buffers that couldn't be enqueued.

Further to the fact that this is very inefficient, it looks like 
invoking rte_kni_rx_burst in a loop is guaranteed to fill the alloc_q to 
its maximum capacity (1024) unless packets are read from the kernel 
faster than they are enqueued by rte_kni_rx_burst.

In my application, I hit the "Out of memory" error in kni_allocate_mbufs 
almost straight away because there is very little egress traffic from 
the kernel and my memory pool wasn't big enough to cater for the kni 
thread and other dpdk queues.

I would think the kni_allocate_mbufs should take a "buffer_count" 
parameter which is the number of desired mbufs to allocate and add to 
the alloc_q.
With this, rte_kni_rx_burst would be able to request allocating as many 
mbufs that are dequeued from the tx_q, so that the total number of 
buffers across alloc_q and tx_q remains constant.

I also noticed that none of these functions are declared inline.
This is not great as the thread that forwards packets between a NIC and 
the kernel may also be the same thread that forwards packets between 2 
NICs.
As such it would be better to avoid too many function calls to forward 
packets between NICs and kernel.

Kind Regards,
Olivier.