[dpdk-users] How to test l3fwd?

[Charlie Li]

Never mind - I figured it out.

On Wed, Jul 20, 2016 at 3:06 PM, Charlie Li  wrote:

> I am also attaching the full logs from l3fwd and l2fwd.
>
> On Wed, Jul 20, 2016 at 3:02 PM, Charlie Li  wrote:
>
>> Hello,
>>
>> My setup is dpdk-2.2.0 on Fedora 23 Server with kernel 4.5.7.
>>
>> I have been testing L2 throughput with l2fwd and an Ixia traffic
>> generator. It works as expected.
>>
>> Command: ./l2fwd -c 0xf -n 4 -- -p 0x3
>> Ixia traffic: MAC (Ethernet frames)
>>
>>
>> Now I am moving to test L3 throughput with l3fwd, but cannot start
>> traffic from Ixia
>>
>> Command: ./l3fwd -c 0xf -n 4 -- -p 0x3 --config="(0,0,2)(1,0,3)"
>> Ixia traffic: IPv4 (IP packets)
>>
>> My question is:
>>
>> What are the IP addresses of the two ports?
>>
>> "LPM: Adding route 0x01010100 / 24 (0)
>> LPM: Adding route 0x02010100 / 24 (1)"
>>
>> Does it mean the IP addresses are 1.1.1.0 (netmask 255.255.255.0) for
>> port0 and 2.1.1.0 (netmask 255.255.255.0) for port1?
>>
>> I set up the following two flows, but Ixia complains "unreachable"
>>
>> Flow1: Ixia PortA (1.1.1.100) -> DPDK Port0 (1.1.1.0) .(l3fwd)
>>  DPDK Port1 (2.1.1.0) -> Ixia PortB (2.1.1.100)
>>   Src IP: 1.1.1.100; Dst IP: 2.1.1.100; Gateway: 1.1.1.0
>>
>> Flow2: Ixia PortB (2.1.1.100) -> DPDK Port1 (2.1.1.0) .(l3fwd)
>>  DPDK Port0 (1.1.1.0) -> Ixia PortA (1.1.1.100)
>>   Src IP: 2.1.1.100; Dst IP: 1.1.1.100; Gateway: 2.1.1.0
>>
>> Thanks,
>> Charlie
>>
>>
>

[dpdk-users] pdump help

[Nishant Verma]

Hi,

Something is still creating the problem. just can't figure out.

Here is the scenario
Just inserted IGB UIO module, attached one 10G NIC to DPDK &
4096 hugepage.

---
trying to run l3fwd as primary and pducp as secondary  to capture packet
dump.
Earlier dpdk_pdump is crashing on either create_mp_ring_vdev:649 or
create_mp_ring_vdev:669.

Now i changed some parameters of pdump and l3fwd, change hugepages to 8192
and tried to run like this..
*./l3fwd -c 606 -n 4 --proc-type=primary  --socket-mem=1024 --file-prefix
l3f -- -p 0x1 --config="(0,0,1),(0,1,2)"*

Here is the log of app
"
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
PMD: bnxt_rte_pmd_init() called for (null)
EAL: PCI device :03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
EAL: PCI device :03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
EAL: PCI device :05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL: PCI device :05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
L3FWD: LPM or EM none selected, default LPM on
Initializing port 0 ... Creating queues: nb_rxq=2 nb_txq=4...
Address:0C:C4:7A:73:EF:14, Destination:02:00:00:00:00:00, Allocated mbuf
pool on socket 0
LPM: Adding route 0x01010100 / 24 (0)
LPM: Adding route IPV6 / 48 (0)
txq=1,0,0 txq=2,1,0 txq=9,2,0 txq=10,3,0

Skipping disabled port 1

Initializing rx queues on lcore 1 ... rxq=0,0,0
Initializing rx queues on lcore 2 ... rxq=0,1,0
Initializing rx queues on lcore 9 ...
Initializing rx queues on lcore 10 ...

"
It seems everything is good in this. So I tried to run pdump like


* ./dpdk_pdump -- --pdump
'port=0,queue=*,rx-dev=/tmp/rx-file.pcap,tx-dev=/tmp/tx-file.pcap,ring-size=1024,mbuf-size=20,total-num-mbufs=1025'*
Here is log for pdump

"
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL:   cannot connect to primary process!
EAL: VFIO support could not be initialized
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the
kernel.
EAL:This may cause issues with mapping memory into secondary processes
EAL: Could not open /dev/hugepages/rtemap_510
PANIC in rte_eal_init():
Cannot init memory
6: [./dpdk_pdump(_start+0x29) [0x4289b9]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)
[0x7fa22219aa40]]
4: [./dpdk_pdump(main+0x155) [0x42c417]]
3: [./dpdk_pdump(rte_eal_init+0xf14) [0x4603b4]]
2: [./dpdk_pdump(__rte_panic+0xc9) [0x424b59]]
1: [./dpdk_pdump(rte_dump_stack+0x2b) [0x4677fb]]
Aborted (core dumped)

"

It seems from log that their is some issue in memory between primary and
secondary, but when i tried to run one l2fwd as primary and one l3fwd as
secondary it works fine.


On Fri, Jul 22, 2016 at 7:21 AM, Pattan, Reshma 
wrote:

> Hi,
>
>
>
> I see some cleanup related bug in the dpdk_pdump tool, so until I publish
> fix for that,
>
> You need to restart  both primary and secondary process( work around)
> after you see any RING/VDEV failure issues.
>
>
>
> Thanks,
>
> Reshma
>
>
>
>
>
> *From:* Pattan, Reshma
> *Sent:* Friday, July 22, 2016 10:31 AM
> *To:* 'Nishant Verma' 
> *Cc:* users at dpdk.org
> *Subject:* RE: pdump help
>
>
>
> Hi,
>
>
>
> As I see ?RING: Cannot reserve memory?. Please allocate enough huge page
> memory and see what happens.
>
>
>
> Also once you encounter such error please restart  both primary and
> secondary process.
>
>
>
> Please paste me the full  error log, if you still face the issue.
>
>
>
> Thanks,
>
> Reshma
>
>
>
>
>
> *From:* Nishant Verma [mailto:vnish11 at gmail.com ]
> *Sent:* Thursday, July 21, 2016 9:10 PM
> *To:* Pattan, Reshma 
> *Cc:* users at dpdk.org
> *Subject:* Re: pdump help
>
>
>
> Hi Reshma,
>
>
>
> Tried things pointed by you(libpcap Installed & config parameter is set).
>
> But still things are not working.
>
>
>
> "
>
> root at ubuntu:/home/oneweb/dpdk/app/pdump# ./dpdk_pdump
> --proc-type=secondary -- --pdump
> 'device_id=03:00.0,queue=*,rx-dev=/tmp/rx-file.pcap,tx-dev=/tmp/tx-file.pcap'
>
> EAL: Detected 16 lcore(s)
>
> EAL: Probing VFIO support...
>
> EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the
> kernel.
>
> EAL:This may cause issues with mapping memory into secondary processes
>
> PMD: bnxt_rte_pmd_init() called for (null)
>
> EAL: PCI device :03:00.0 on NUMA socket 0
>
> EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
>
> EAL: PCI device :03:00.1 on NUMA socket 0
>
> EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
>
> PMD: eth_ixgbe_dev_init(): No TX queues configured yet. Using default TX
> function.
>
> EAL: PCI device :05:00.0 on NUMA socket 0
>
> EAL:   probe driver: 8086:1521 rte_igb_pmd
>
> EAL: PCI device :05:00.1 on NUMA socket 0
>
> EAL:   probe driver: 8086:1521 rte_igb_pmd
>
> RING: Cannot reserve memory
>
> EAL: Error - exiting with code: 1
>
>   Cause: File exists:create_mp_ring_vdev:649
>
> "
>
>
>
>
>
> Rgds,
>
> Nishant
>
>
>
>
>
> On Thu, Jul 21, 2016 at 5:16 AM, Pattan, Reshma 
> 

[dpdk-users] [dpdk-dev] em1000 driver lockup in KVM

[Stephen Hemminger]

On Fri, 22 Jul 2016 14:19:29 -0700
Prabahar Radhakrishnan  wrote:

> Hi,
>I was running a dpdk application with e1000 driver and I am facing a
> rare driver lockup condition.  Under this condition, what I am seeing is
> that the Receive side locks up.  The transmit side is fine.  When I execute
> "rte_eth_dev_stop()/rte_eth_dev_start()" from within the VM, the dpdk
> application recovers.  The lockup happens once in 2 or 3 days if I
> continuously ping an IP address serviced by the dpdk application.  Once it
> locks up, what I am noticing is that even though packets are coming in, the
> driver is not populating them and signaling the event.  In other words, in
> "eth_em_recv_pkts" call, I am not seeing "E1000_RXD_STAT_DD" being set for
> status, and the driver bails out.  If anyone has come across such
> condition, please let me know?  Appreciate the help.
> 
> Environment:  I have a VM running on top of Ubuntu/KVM.
> Application: Running dpdk-2.0.0.
> VM: uses e1000 driver.
> On the KVM side, our port is sitting in a bridge.  There are other VNFs
> that we use for testing and some of them are using virtio driver.
> 
> Thank You
> regards Prab

Save yourself lots of pain, use virtio not e1000 with DPDK.
The emulation of E1000 in QEMU is very limited and DPDK expects real hardware.
Lots of features and control are missing.

[dpdk-users] em1000 driver lockup in KVM

[Prabahar Radhakrishnan]

Hi,
   I was running a dpdk application with e1000 driver and I am facing a
rare driver lockup condition.  Under this condition, what I am seeing is
that the Receive side locks up.  The transmit side is fine.  When I execute
"rte_eth_dev_stop()/rte_eth_dev_start()" from within the VM, the dpdk
application recovers.  The lockup happens once in 2 or 3 days if I
continuously ping an IP address serviced by the dpdk application.  Once it
locks up, what I am noticing is that even though packets are coming in, the
driver is not populating them and signaling the event.  In other words, in
"eth_em_recv_pkts" call, I am not seeing "E1000_RXD_STAT_DD" being set for
status, and the driver bails out.  If anyone has come across such
condition, please let me know?  Appreciate the help.

Environment:  I have a VM running on top of Ubuntu/KVM.
Application: Running dpdk-2.0.0.
VM: uses e1000 driver.
On the KVM side, our port is sitting in a bridge.  There are other VNFs
that we use for testing and some of them are using virtio driver.

Thank You
regards Prab

[dpdk-users] Fwd: possible rte_ring issues: retransmission

[Сергей Львов]

-- Forwarded message --
From: CIDJEY 
Date: 2016-07-22 12:13 GMT+03:00
Subject: Fwd: possible rte_ring issues: retransmission
To: ?? ? 



-- Forwarded message --
From: CIDJEY 
Date: 2016-07-22 13:04 GMT+04:00
Subject: possible rte_ring issues: retransmission
To: users at dpdk.org


I'm sorry guys, I had to retransmit this message: my other mail domain
seems to be caught in the gmail spam filter, so I assumed all the guys
with gmail didn't see the message as well. Sorry if you see this
message twice.


Hi guys

I have a pipeline-like application - 3 modules (rx, worker, tx)
connected by rings.

Rx module gets the packtes, timestamps them and then sends them to the
worker via a ring. The ring has single producer (rx) and single
consumer (worker).
I've noticed that sometimes the worker gets some packets with
inconsistent timestamps: the timestamp of the next incoming packets is
less then the timestamp of the previous packet. Usually after that
follows another 2 or 3 packets with inconsistent timestamps. Then
everything get fine for some time. It happens quite often: every
1-2000 packets.

I've put a check function for following timestamps on the outgress and
ingress points of each module. It seems that on the outgress point of
the RX module (before the  rte_ring_enqueue_bulk  into the ring) the
timestamps are totally fine, but on the ingress point of the worker
(after rte_ring_dequeue_burst from the ring) some packets come in the
wrong order. It happens only between the bursts, the packets within
one burst are always consistent.

For example: the RX sent 20 packets in a bulk, the worker then
recieves 10 packets in one burst and 10 in another.
So sometimes I can see, that the first burst consists of the packets
1-2-3-4-5-6-7-8-11-12 and the second consists of the packets
9-10-13-14-15-16-17-18-19-20.
(Then again - on the out point of RX everything was perfectly fine)

And that wouldn't be that much of an issue, but my worker module is
highly sensitive to this sort of things. It totally breaks its logic.

Is that how the ring behaviour supposed to be? Am I missing something?
 Is there a way to force rings to deliver packets in the order they
were sent?

I'm using DPDK 2.2.0

Thanks in advance.


*Update:*

A little update, I hope it will clarify things a bit.
I've replaced timestamp checking with the counters, so here's the
actual example:

Before another burst the worker expects to get packet 410. Instead it
gets the burst of the next 6 packets: 412-413-414-415-416-417. The
counters on the RX tell me, that RX already sent 417 packets. So the
worker now misses two packets: 410-411. By the way, in the ring by
this exactly moment (if we can trust gdb) prod.tail == cons.tail, so
it seems there're no packets in the ring. It seems the 2 packets have
been just plain lost.

But after that if we proceed to the next dequeue_burst (also with the
gdb) we get another 32 packets, and the first two of them are the
missing 410-411. Oddly enough, after those two follow 450-451 and so
on, and on the NEXT dequeue burst we get another 32 packets: 418-449
(basically the ones that should be instead of the previous 32
packets)... I'm confused.

What is going on? Is it normal? How may I make sure the packets are
transferred in the right order?

Thanks

[dpdk-users] pdump help

[Pattan, Reshma]

Hi,

I see some cleanup related bug in the dpdk_pdump tool, so until I publish fix 
for that,
You need to restart  both primary and secondary process( work around) after you 
see any RING/VDEV failure issues.

Thanks,
Reshma


From: Pattan, Reshma
Sent: Friday, July 22, 2016 10:31 AM
To: 'Nishant Verma' 
Cc: users at dpdk.org
Subject: RE: pdump help

Hi,

As I see ?RING: Cannot reserve memory?. Please allocate enough huge page memory 
and see what happens.

Also once you encounter such error please restart  both primary and secondary 
process.

Please paste me the full  error log, if you still face the issue.

Thanks,
Reshma


From: Nishant Verma [mailto:vnis...@gmail.com]
Sent: Thursday, July 21, 2016 9:10 PM
To: Pattan, Reshma mailto:reshma.pattan at 
intel.com>>
Cc: users at dpdk.org
Subject: Re: pdump help

Hi Reshma,

Tried things pointed by you(libpcap Installed & config parameter is set).
But still things are not working.

"
root at ubuntu:/home/oneweb/dpdk/app/pdump# ./dpdk_pdump --proc-type=secondary 
-- --pdump 
'device_id=03:00.0,queue=*,rx-dev=/tmp/rx-file.pcap,tx-dev=/tmp/tx-file.pcap'
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the 
kernel.
EAL:This may cause issues with mapping memory into secondary processes
PMD: bnxt_rte_pmd_init() called for (null)
EAL: PCI device :03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
EAL: PCI device :03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
PMD: eth_ixgbe_dev_init(): No TX queues configured yet. Using default TX 
function.
EAL: PCI device :05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL: PCI device :05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
RING: Cannot reserve memory
EAL: Error - exiting with code: 1
  Cause: File exists:create_mp_ring_vdev:649
"


Rgds,
Nishant


On Thu, Jul 21, 2016 at 5:16 AM, Pattan, Reshma mailto:reshma.pattan at intel.com>> wrote:
Hi,

Pdump tool is designed based on PCAP PMD. So you need to install libpcap 
package and set CONFIG_RTE_LIBRTE_PMD_PCAP flag to ?y?
in config/common_base file and recompile the DPDK and run pdump tool. Please 
let me know if that fix the issue.

This dependency is mentioned in the  document 
?doc/guides/sample_app_ug/pdump.rst.?

Thanks,
Reshma

From: Nishant Verma [mailto:vnish11 at gmail.com]
Sent: Wednesday, July 20, 2016 8:08 PM
To: Pattan, Reshma mailto:reshma.pattan at 
intel.com>>
Subject: pdump help

Hi Reshma,

i am facing some issue in executing pdump. I have seen in logs that you 
recently updated the code. Please let me know if i am doing anything wrong.


app/pdump# ./dpdk_pdump -- --pdump 
'device_id=03:00.0,queue=*,rx-dev=/tmp/rx-1-file.pcap,tx-dev=tx-1-file.pcap'

This is the error i am getting.
Cause: File exists:create_mp_ring_vdev:649

?Thanks a lot.?

--
Rgds,
Nishant






--
Rgds,
Nishant

[dpdk-users] pdump help

[Pattan, Reshma]

Hi,

As I see ?RING: Cannot reserve memory?. Please allocate enough huge page memory 
and see what happens.

Also once you encounter such error please restart  both primary and secondary 
process.

Please paste me the full  error log, if you still face the issue.

Thanks,
Reshma


From: Nishant Verma [mailto:vnis...@gmail.com]
Sent: Thursday, July 21, 2016 9:10 PM
To: Pattan, Reshma 
Cc: users at dpdk.org
Subject: Re: pdump help

Hi Reshma,

Tried things pointed by you(libpcap Installed & config parameter is set).
But still things are not working.

"
root at ubuntu:/home/oneweb/dpdk/app/pdump# ./dpdk_pdump --proc-type=secondary 
-- --pdump 
'device_id=03:00.0,queue=*,rx-dev=/tmp/rx-file.pcap,tx-dev=/tmp/tx-file.pcap'
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the 
kernel.
EAL:This may cause issues with mapping memory into secondary processes
PMD: bnxt_rte_pmd_init() called for (null)
EAL: PCI device :03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
EAL: PCI device :03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:15ad rte_ixgbe_pmd
PMD: eth_ixgbe_dev_init(): No TX queues configured yet. Using default TX 
function.
EAL: PCI device :05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL: PCI device :05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
RING: Cannot reserve memory
EAL: Error - exiting with code: 1
  Cause: File exists:create_mp_ring_vdev:649
"


Rgds,
Nishant


On Thu, Jul 21, 2016 at 5:16 AM, Pattan, Reshma mailto:reshma.pattan at intel.com>> wrote:
Hi,

Pdump tool is designed based on PCAP PMD. So you need to install libpcap 
package and set CONFIG_RTE_LIBRTE_PMD_PCAP flag to ?y?
in config/common_base file and recompile the DPDK and run pdump tool. Please 
let me know if that fix the issue.

This dependency is mentioned in the  document 
?doc/guides/sample_app_ug/pdump.rst.?

Thanks,
Reshma

From: Nishant Verma [mailto:vnish11 at gmail.com]
Sent: Wednesday, July 20, 2016 8:08 PM
To: Pattan, Reshma mailto:reshma.pattan at 
intel.com>>
Subject: pdump help

Hi Reshma,

i am facing some issue in executing pdump. I have seen in logs that you 
recently updated the code. Please let me know if i am doing anything wrong.


app/pdump# ./dpdk_pdump -- --pdump 
'device_id=03:00.0,queue=*,rx-dev=/tmp/rx-1-file.pcap,tx-dev=tx-1-file.pcap'

This is the error i am getting.
Cause: File exists:create_mp_ring_vdev:649

?Thanks a lot.?

--
Rgds,
Nishant






--
Rgds,
Nishant

[dpdk-users] MLX4 on BSD?

[Bowden, Alex]

Hey,

>From what I can tell the PMD driver for MLX4 is for Linux only. Is this going 
>to change anytime soon?

Thanks,
Alex

[dpdk-users] possible rte_ring issues: update

[Alex]

A little update, I hope it will clarify things a bit.
I've replaced timestamp checking with the counters, so here's the actual 
example:

Before another burst the worker expects to get packet 410. Instead it gets the 
burst of the next 6 packets: 412-413-414-415-416-417. The counters on the RX 
tell me, that RX already sent 417 packets. So the worker now misses two 
packets: 410-411. By the way, in the ring by this exactly moment (if we can 
trust gdb) prod.tail == cons.tail, so it seems there're no packets in the ring. 
It seems the 2 packets have been just plain lost.

But after that if we proceed to the next dequeue_burst (also with the gdb) we 
get another 32 packets, and the first two of them are the missing 410-411. 
Oddly enough, after those two follow 450-451 and so on, and on the NEXT dequeue 
burst we get another 32 packets: 418-449 (basically the ones that should be 
instead of the previous 32 packets)... I'm confused.

What is going on? Is it normal? How may I make sure the packets are transferred 
in the right order?

Thanks


Hi guys

I have a pipeline-like application - 3 modules (rx, worker, tx) connected by 
rings.

Rx module gets the packtes, timestamps them and then sends them to the worker 
via a ring. The ring has single producer (rx) and single consumer (worker).
I've noticed that sometimes the worker gets some packets with inconsistent 
timestamps: the timestamp of the next incoming packets is less then the 
timestamp of the previous packet. Usually after that follows another 2 or 3 
packets with inconsistent timestamps. Then everything get fine for some time. 
It happens quite often: every 1-2000 packets.

I've put a check function for following timestamps on the outgress and ingress 
points of each module. It seems that on the outgress point of the RX module 
(before the  rte_ring_enqueue_bulk ?into the ring) the timestamps are totally 
fine, but on the ingress point of the worker (after rte_ring_dequeue_burst from 
the ring) some packets come in the wrong order. It happens only between the 
bursts, the packets within one burst are always consistent.

For example: the RX sent 20 packets in a bulk, the worker then recieves 10 
packets in one burst and 10 in another.
So sometimes I can see, that the first burst consists of the packets 
1-2-3-4-5-6-7-8-11-12 and the second consists of the packets 
9-10-13-14-15-16-17-18-19-20.
(Then again - on the out point of RX everything was perfectly fine)

And that wouldn't be that much of an issue, but my worker module is highly 
sensitive to this sort of things. It totally breaks its logic.

Is that how the ring behaviour supposed to be? Am I missing something? ?Is 
there a way to force rings to deliver packets in the order they were sent?

I'm using DPDK 2.2.0

Thanks in advance.