On 08/29/2018 11:22 PM, Wiles, Keith wrote: > >> On Aug 29, 2018, at 12:19 PM,
Saber Rezvani <irsa...@zoho.com> wrote: >> >> >> >> On 08/29/2018 01:39 AM,
Wiles, Keith wrote: >>>> On Aug 28, 2018, at 2:16 PM, Saber Rezvani
<irsa...@zoho.com> wrote: >>>> >>>> >>>> >>>> On 08/28/2018 11:39 PM, Wiles,
Keith wrote: >>>>> Which version of Pktgen? I just pushed a patch in 3.5.3 to
fix a performance problem. >>>> I use Pktgen verion 3.0.0, indeed it is O.k as
far as I have one core. (10 Gb/s) but when I increase the number of core (one
core per queue) then I loose some performance (roughly 8.5 Gb/s for 8-core). In
my scenario Pktgen shows it is generating at line rate, but receiving 8.5 Gb/s.
>>>> Is it because of Pktgen??? >>> Normally Pktgen can receive at line rate up
to 10G 64 byte frames, which means Pktgen should not be the problem. You can
verify that by looping the cable from one port to another on the pktgen machine
to create a external loopback. Then send traffic what ever you can send from
one port you should be able to receive those packets unless something is
configured wrong. >>> >>> Please send me the command line for pktgen. >>> >>>
>>> In pktgen if you have this config -m “[1-4:5-8].0” then you have 4 cores
sending traffic and 4 core receiving packets. >>> >>> In this case the TX cores
will be sending the packets on all 4 lcores to the same port. On the rx side
you have 4 cores polling 4 rx queues. The rx queues are controlled by RSS,
which means the RX traffic 5 tuples hash must divide the inbound packets across
all 4 queues to make sure each core is doing the same amount of work. If you
are sending only a single packet on the Tx cores then only one rx queue be
used. >>> >>> I hope that makes sense. >> I think there is a misunderstanding
of the problem. Indeed the problem is not the Pktgen. >> Here is my command -->
./app/app/x86_64-native-linuxapp-gcc/pktgen -c ffc0000 -n 4 -w 84:00.0 -w
84:00.1 --file-prefix pktgen_F2 --socket-mem 1000,2000,1000,1000 -- -T -P -m
"[18-19:20-21].0, [22:23].1" >> >> The problem is when I run the symmetric_mp
example for $numberOfProcesses=8 cores, then I have less throughput (roughly
8.4 Gb/s). but when I run it for $numberOfProcesses=3 cores throughput is 10G.
>> for i in `seq $numberOfProcesses`; >> do >> .... some calculation goes
here..... >> symmetric_mp -c $coremask -n 2 --proc-type=auto -w 0b:00.0 -w
0b:00.1 --file-prefix sm --socket-mem 4000,1000,1000,1000 -- -p 3
--num-procs=$numberOfProcesses --proc-id=$procid"; >> ..... >> done > Most NICs
have a limited amount of memory on the NIC and when you start to segment that
memory because you are using more queues it can effect performance. > > In one
of the NICs if you go over say 6 or 5 queues the memory per queue for Rx/Tx
packets starts to become a bottle neck as you do not have enough memory in the
Tx/Rx queues to hold enough packets. This can cause the NIC to drop Rx packets
because the host can not pull the data from the NIC or Rx ring on the host fast
enough. This seems to be the problem as the amount of time to process a packet
on the host has not changed only the amount of buffer space in the NIC as you
increase queues. > > I am not sure this is your issue, but I figured I would
state this point. What you said sounded logical, but is there away that I can
be sure? I mean are there some registers at NIC which show the number of packet
loss on NIC? or does DPDK have an API which shows the number of packet loss at
NIC level? > >> I am trying find out what makes this loss! >> >> >>>>>> On Aug
28, 2018, at 12:05 PM, Saber Rezvani <irsa...@zoho.com> wrote: >>>>>> >>>>>>
>>>>>> >>>>>> On 08/28/2018 08:31 PM, Stephen Hemminger wrote: >>>>>>> On Tue,
28 Aug 2018 17:34:27 +0430 >>>>>>> Saber Rezvani <irsa...@zoho.com> wrote:
>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> I have run
multi_process/symmetric_mp example in DPDK example directory. >>>>>>>> For a
one process its throughput is line rate but as I increase the >>>>>>>> number
of cores I see decrease in throughput. For example, If the number >>>>>>>> of
queues set to 4 and each queue assigns to a single core, then the >>>>>>>>
throughput will be something about 9.4. if 8 queues, then throughput >>>>>>>>
will be 8.5. >>>>>>>> >>>>>>>> I have read the following, but it was not
convincing. >>>>>>>> >>>>>>>>
http://mails.dpdk.org/archives/dev/2015-October/024960.html >>>>>>>> >>>>>>>>
>>>>>>>> I am eagerly looking forward to hearing from you, all. >>>>>>>>
>>>>>>>> >>>>>>>> Best wishes, >>>>>>>> >>>>>>>> Saber >>>>>>>> >>>>>>>>
>>>>>>> Not completely surprising. If you have more cores than packet line rate
>>>>>>> then the number of packets returned for each call to rx_burst will be
less. >>>>>>> With large number of cores, most of the time will be spent doing
reads of >>>>>>> PCI registers for no packets! >>>>>> Indeed pktgen says it is
generating traffic at line rate, but receiving less than 10 Gb/s. So, it that
case there should be something that causes the reduction in throughput :(
>>>>>> >>>>>> >>>>> Regards, >>>>> Keith >>>>> >>>> >>> Regards, >>> Keith >>>
>> Best regards, >> Saber > Regards, > Keith > Best regards, Saber
