Re: [vpp-dev] Question about acl match/permit behavior.

[Andrew Yourtchenko]

Hi Venkat,

Before doing ACL checks, acl-plugin checks the establshed sessions on
the given interface.

If an already established session is hit the action is set to "3" and
no further ACL check is done, which is what you see in your output.

For that case, the ACL# in the trace is a special-case "-1", same as
lc_index-1 - because we actually never get it to ACL lookup so do not
need it; and the rule# is the session index; and the trace_bits tells
us that this session is classifed to use timeout type "3", which is
defined in acl_timeout_e as "ACL_TIMEOUT_TCP_TRANSIENT" - which makes
sense, since you are sending the packets in one direction and thus the
session never establishes and the timeout type never changes to
"TCP_IDLE".

Of the packets you sent in total, 15002 packets  had hit the ACL#3
rule and caused the sessions to be created. The rest of the packets
had hit the established session entries and this is one of those that
you have captured.

At the time of taking the outputs the traffic was stopped and the
sessions were deleted by idle timeout (which for the transient state
is short).

Does this make sense ?

--a




On 9/14/20, Venkat  wrote:
> Hello Andrew,
>
> I am doing a simple test by sending TCP flows from Trex traffic generator.
> Traffic source is 16.0.0.1-16.0.0.100 and destination is
> 48.0.0.1-48.0.0.100.
> I am sending TCP traffic with CPS=50 from 100 clients to 100 servers via
> Trex http profile.
>
> I have a reflective ACL to permit the flow for the above src and dst.
> I expect the packet to hit acl index 3 rule 0 and get permitted.
> However, I see the following match in the packet trace which doesn't seem
> like it is hitting acl index 3 and rule 0 but hitting some other acl index
> (acl -1 and lc_index -1).
> What is the behavior here? Please provide some context here..
>
> 03:47:03:874624: acl-plugin-in-ip4-fa
>
> acl-plugin: lc_index: -1, sw_if_index 1, next index 1, action: 3, match: acl
> -1 rule 13309 trace_bits 80010303
>
> pkt info    43304310
> 0001010600502d14 0310
>
> lc_index 0 l3 ip4 16.0.0.67 -> 48.0.0.67 l4 lsb_of_sw_if_index 1 proto 6
> l4_is_input 1 l4_slow_path 0 l4_flags 0x01 port 11540 -> 80 tcp flags
> (valid) 10 rsvd 0
>
> DBGvpp# show acl-plugin acl
>
> acl-index 0 count 2 tag
> {8-sra-inbound-sasefwpro-nags-explicit-deny-8}
>
> 0: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
>
> 1: ipv6 deny src ::/0 dst ::/0 proto 0 sport 0 dport 0
>
> applied inbound on sw_if_index: 1
>
> applied outbound on sw_if_index:
>
> used in lookup context index: 1
>
> acl-index 1 count 6 tag
> {9-sra-inbound-sasefwpro-sase-default-fw-profile}
>
> 0: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 6 sport 0-65535
> dport 0-65535
>
> 1: ipv6 permit+reflect src ::/0 dst ::/0 proto 6 sport 0-65535 dport
> 0-65535
>
> 2: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 17 sport 0-65535
> dport 0-65535
>
> 3: ipv6 permit+reflect src ::/0 dst ::/0 proto 17 sport 0-65535 dport
> 0-65535
>
> 4: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535
> dport 0-65535
>
> 5: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535
> dport 0-65535
>
> applied inbound on sw_if_index: 1
>
> applied outbound on sw_if_index:
>
> used in lookup context index: 1
>
> acl-index 2 count 6 tag
> {9-sra-outbound-sasefwpro-sase-default-fw-profile}
>
> 0: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 6 sport 0-65535 dport
> 0-65535
>
> 1: ipv6 deny src ::/0 dst ::/0 proto 6 sport 0-65535 dport 0-65535
>
> 2: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 17 sport 0-65535 dport
> 0-65535
>
> 3: ipv6 deny src ::/0 dst ::/0 proto 17 sport 0-65535 dport 0-65535
>
> 4: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535 dport
> 0-65535
>
> 5: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535 dport
> 0-65535
>
> applied inbound on sw_if_index:
>
> applied outbound on sw_if_index: 1
>
> used in lookup context index: 0
>
> acl-index 3 count 1 tag {100-allow-48}
>
> 0: ipv4 permit+reflect src 16.0.0.0/16 dst 48.0.0.0/16 proto 0 sport 0 dport
> 0
>
> applied inbound on sw_if_index: 1
>
> used in lookup context index: 1
>
> DBGvpp#
>
> DBGvpp# show acl-plugin tables
>
> Stats counters enabled for interface ACLs: 0
>
> Mask-type entries:
>
> 0:    
>  0800 refcount 4
>
> 1:    
> 00ff 0e00 refcount 24
>
> 2:    
>  0800 refcount 2
>
> Mask-ready ACL representations
>
> acl-index 0 bitmask-ready layout
>
> applied lc_index list: 1
>
> 0:    
>   base mask index 0 acl 0 rule 0 action 0
>
> 1: 

[vpp-dev] vpp checkstyle failures

[Sergey Matov]

Greetings VPP community.

Recently I've created a small patch for counter.h core file.
For some reason several patchsets are failing with checkstyle issues.
https://gerrit.fd.io/r/c/vpp/+/28843
Fixing of the change by indent causes the entire file to be reworked since
the old file does not fit indent requirements. What is the best way to
process changes that fit the checkstyle but the original file does not?
Thanks in advance

-- 

Sergey Matov

-- 

t: +49 391 819099-0

--- enabling your networks
-

Travelping GmbH
Roentgenstraße 13
39108 Magdeburg
Germany

t: +49 391 819099-0
f: +49 391 819099-299

e: i...@travelping.com
w: https://www.travelping.com/
Company registration: Amtsgericht Stendal
Managing Director: Holger Winkelmann
Reg. No.: HRB 10578
VAT ID: DE236673780
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17396): https://lists.fd.io/g/vpp-dev/message/17396
Mute This Topic: https://lists.fd.io/mt/76861769/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp checkstyle failures

[Andrew Yourtchenko]

Hi Sergey,

have a look at 22963; the checkstyle job requires a particular version of 
checkstyle, it’s WIP in progress to update it, as well as to give us an option 
to selectively try out clang-format on some files.

However; I think this functionality of this patch has more fundamental 
question: it allows the use of the function that essentially renders the simple 
counter ambiguous in the way i see it as a user operationally:

If the counter value is 42 - does it mean it was incremented to 10042 and then 
decremented by 1 ? Or was it incremented by 142 and then 
decremented by 100 ?

The only sensible operation for something that is the *counter*, is to either 
increment or reset to zero in my view. You should not be allowed to “uncount” 
stuff.

However, if we are talking about a *gauge*, then it is a different story, and I 
suspect this may be the functionality you are after ? 
Like, the current number of connections tracked somewhere, etc ?

--a

>> On 15 Sep 2020, at 12:25, Sergey Matov  wrote:
> 
> Greetings VPP community.
> 
> Recently I've created a small patch for counter.h core file.
> For some reason several patchsets are failing with checkstyle issues.
> https://gerrit.fd.io/r/c/vpp/+/28843
> Fixing of the change by indent causes the entire file to be reworked since 
> the old file does not fit indent requirements. What is the best way to 
> process changes that fit the checkstyle but the original file does not?
> Thanks in advance
> 
> -- 
> Sergey Matov
> 
> -- 
> 
> t: +49 391 819099-0
> 
> --- enabling your networks 
> -
> 
> Travelping GmbH 
> Roentgenstraße 13
> 39108 Magdeburg
> Germany
> 
> 
> t: +49 391 819099-0
> f: +49 391 819099-299
> 
> e: i...@travelping.com
> w: https://www.travelping.com/
> 
> Company registration: Amtsgericht Stendal 
> Managing Director: Holger Winkelmann
> Reg. No.: HRB 10578
> VAT ID: DE236673780
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17397): https://lists.fd.io/g/vpp-dev/message/17397
Mute This Topic: https://lists.fd.io/mt/76861769/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp checkstyle failures

[Sergey Matov]

Hello Andrew. Thanks for quick response.

In a nutshell - yes, gauges are something I am also currently using to
track the number of entries in a bihash table(for example), however gauge
update will be done each time when the stats segment performs processing.
Idea was to track value more rapidly w/o increasing stats update interval.
I thought it worth to have it in upstream code.

On Tue, 15 Sep 2020 at 14:50, Andrew 👽 Yourtchenko 
wrote:

> Hi Sergey,
>
> have a look at 22963; the checkstyle job requires a particular version of
> checkstyle, it’s WIP in progress to update it, as well as to give us an
> option to selectively try out clang-format on some files.
>
> However; I think this functionality of this patch has more fundamental
> question: it allows the use of the function that essentially renders the
> simple counter ambiguous in the way i see it as a user operationally:
>
> If the counter value is 42 - does it mean it was incremented to 10042 and
> then decremented by 1 ? Or was it incremented by 142 and then
> decremented by 100 ?
>
> The only sensible operation for something that is the *counter*, is to
> either increment or reset to zero in my view. You should not be allowed to
> “uncount” stuff.
>
> However, if we are talking about a *gauge*, then it is a different story,
> and I suspect this may be the functionality you are after ?
> Like, the current number of connections tracked somewhere, etc ?
>
> --a
>
> On 15 Sep 2020, at 12:25, Sergey Matov 
> wrote:
>
> 
> Greetings VPP community.
>
> Recently I've created a small patch for counter.h core file.
> For some reason several patchsets are failing with checkstyle issues.
> https://gerrit.fd.io/r/c/vpp/+/28843
> Fixing of the change by indent causes the entire file to be reworked since
> the old file does not fit indent requirements. What is the best way to
> process changes that fit the checkstyle but the original file does not?
> Thanks in advance
>
> --
>
> Sergey Matov
>
> --
>
> t: +49 391 819099-0
>
> --- enabling your networks
> -
>
> Travelping GmbH
> Roentgenstraße 13
> 39108 Magdeburg
> Germany
>
> t: +49 391 819099-0
> f: +49 391 819099-299
>
> e: i...@travelping.com
> w: https://www.travelping.com/
> Company registration: Amtsgericht Stendal
> Managing Director: Holger Winkelmann
> Reg. No.: HRB 10578
> VAT ID: DE236673780
> 
>
>

-- 

Sergey Matov

-- 

t: +49 391 819099-0

--- enabling your networks
-

Travelping GmbH
Roentgenstraße 13
39108 Magdeburg
Germany

t: +49 391 819099-0
f: +49 391 819099-299

e: i...@travelping.com
w: https://www.travelping.com/
Company registration: Amtsgericht Stendal
Managing Director: Holger Winkelmann
Reg. No.: HRB 10578
VAT ID: DE236673780
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17398): https://lists.fd.io/g/vpp-dev/message/17398
Mute This Topic: https://lists.fd.io/mt/76861769/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Issue with adding new new node in between ip4-input n ip4-lookup

[Dave Barach via lists.fd.io]

Sounds like you may not have enabled the “test-node” feature on the rx 
sw_if_index. “show interface  features”... Note that if the packet comes 
from a bridge group, I suspect that you’ll need to enable the feature on the 
bvi vs the rx interface.

This is the kind of problem which “pcap dispatch trace ...” tends to help chase 
down.

D.

From: vpp-dev@lists.fd.io  On Behalf Of Caffeine Coder via 
lists.fd.io
Sent: Tuesday, September 15, 2020 2:56 AM
To: Vpp-dev 
Subject: [vpp-dev] Issue with adding new new node in between ip4-input n 
ip4-lookup

Hi
I am trying to add a new node to parse packet data after all vxlan decoding, 
ip4-input and before doing IP lookup. This code flow is is working fine for 
packets coming from vxlan-gpe/IPSec tunnels and not for vxlan.
Traffic is working fine except not going through this new "test-node".

The issue i am seeing with vxlan packet is: packet is going directly 
ip4-lookup. i.e from ip4-vxlan-bypass->vxlan4-input->l2 
(input,learn,fwd)->ip4-input->ip4_lookup. But I can't hook my node in between 
ip4-input and ip4_lookup.

If I do vxlan-gpe or IPSEC, i can hook the test node properly i.e. 
ip4-vxlan-gpe-bypass->vxlan4-gpe-input->ip4-input->test-node->ip4_lookup.

I tried adding and changing ".runs_before" and ".runs_after" in my test-node. 
Still I am not able to stitch my test-node, if traffic coming from  vxlan 
tunnels.
"show vlib graph" is very useful command to debug for not able to find what 
else might be missing for vxlan. I do see that vxlan packet going through lot 
of l2 nodes. Does that need any special config for making non vpp native l3 
nodes to work fine?

Any pointers will be helpful.

Thanks
Sam.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17399): https://lists.fd.io/g/vpp-dev/message/17399
Mute This Topic: https://lists.fd.io/mt/76859804/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp checkstyle failures

[Andrew Yourtchenko]

Hi Sergey,

The rationale behind this makes sense to me, 
though I still think it should be still a distinct new entity rather than 
“extending” the semantics of the existing one. 
We had about a year worth of rather disruptive injecting types in API 
so the users can distinguish a u32 IPv4 address from u32 Boolean and such - and 
I would rather avoid any moves in the opposite direction :-)

Now, with the rationale clarified -
I am not sure what the performance implications are of having such an entity 
(assuming the best case it’s widespread use - “gauge, but faster” - what’s not 
to like ?) - so maybe someone with more experience in that area can chime in. 
(I reacted mostly because I am involved with various release-y-infrastructure-y 
things that make checkstyle say “-1“ :-)

--a

>> On 15 Sep 2020, at 12:58, Sergey Matov  wrote:
> 
> Hello Andrew. Thanks for quick response.
> 
> In a nutshell - yes, gauges are something I am also currently using to track 
> the number of entries in a bihash table(for example), however gauge update 
> will be done each time when the stats segment performs processing. Idea was 
> to track value more rapidly w/o increasing stats update interval. I thought 
> it worth to have it in upstream code. 
> 
>> On Tue, 15 Sep 2020 at 14:50, Andrew 👽 Yourtchenko  
>> wrote:
>> Hi Sergey,
>> 
>> have a look at 22963; the checkstyle job requires a particular version of 
>> checkstyle, it’s WIP in progress to update it, as well as to give us an 
>> option to selectively try out clang-format on some files.
>> 
>> However; I think this functionality of this patch has more fundamental 
>> question: it allows the use of the function that essentially renders the 
>> simple counter ambiguous in the way i see it as a user operationally:
>> 
>> If the counter value is 42 - does it mean it was incremented to 10042 and 
>> then decremented by 1 ? Or was it incremented by 142 and then 
>> decremented by 100 ?
>> 
>> The only sensible operation for something that is the *counter*, is to 
>> either increment or reset to zero in my view. You should not be allowed to 
>> “uncount” stuff.
>> 
>> However, if we are talking about a *gauge*, then it is a different story, 
>> and I suspect this may be the functionality you are after ? 
>> Like, the current number of connections tracked somewhere, etc ?
>> 
>> --a
>> 
 On 15 Sep 2020, at 12:25, Sergey Matov  wrote:
>>> 
>>> Greetings VPP community.
>>> 
>>> Recently I've created a small patch for counter.h core file.
>>> For some reason several patchsets are failing with checkstyle issues.
>>> https://gerrit.fd.io/r/c/vpp/+/28843
>>> Fixing of the change by indent causes the entire file to be reworked since 
>>> the old file does not fit indent requirements. What is the best way to 
>>> process changes that fit the checkstyle but the original file does not?
>>> Thanks in advance
>>> 
>>> -- 
>>> Sergey Matov
>>> 
>>> -- 
>>> 
>>> t: +49 391 819099-0
>>> 
>>> --- enabling your networks 
>>> -
>>> 
>>> Travelping GmbH 
>>> Roentgenstraße 13
>>> 39108 Magdeburg
>>> Germany
>>> 
>>> 
>>> t: +49 391 819099-0
>>> f: +49 391 819099-299
>>> 
>>> e: i...@travelping.com
>>> w: https://www.travelping.com/
>>> 
>>> Company registration: Amtsgericht Stendal 
>>> Managing Director: Holger Winkelmann
>>> Reg. No.: HRB 10578
>>> VAT ID: DE236673780
>>> 
> 
> 
> -- 
> Sergey Matov
> 
> -- 
> 
> t: +49 391 819099-0
> 
> --- enabling your networks 
> -
> 
> Travelping GmbH 
> Roentgenstraße 13
> 39108 Magdeburg
> Germany
> 
> 
> t: +49 391 819099-0
> f: +49 391 819099-299
> 
> e: i...@travelping.com
> w: https://www.travelping.com/
> 
> Company registration: Amtsgericht Stendal 
> Managing Director: Holger Winkelmann
> Reg. No.: HRB 10578
> VAT ID: DE236673780
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17400): https://lists.fd.io/g/vpp-dev/message/17400
Mute This Topic: https://lists.fd.io/mt/76861769/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp checkstyle failures

[Ole Troan]

Hi Sergey,

> In a nutshell - yes, gauges are something I am also currently using to track 
> the number of entries in a bihash table(for example), however gauge update 
> will be done each time when the stats segment performs processing. Idea was 
> to track value more rapidly w/o increasing stats update interval. I thought 
> it worth to have it in upstream code. 

There are two types of gauges.
The one where you register with the stats segment, where the stats process 
periodically calls the registered function, inquiring the state of the gauge.
The other where the client code itself manipulates the gauge directly, just 
like it does for other counters.

For the latter case, can you not use vlib_set_simple_counter() directly?

Just wondering what a "decrementing" counter (or gauge) would mean??

Best regards,
Ole

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17401): https://lists.fd.io/g/vpp-dev/message/17401
Mute This Topic: https://lists.fd.io/mt/76861769/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Query regarding Scaling of CPU utilization with different number of workers

[siddarth rai]

Hello everyone,

I am using VPP 19.08 and running it in simple forwarding mode. It's
handling thousands of connections per second and over 8 Gbps of throughput.

I am curious to know what the general experience has been with CPU usage
per worker while increasing the number of workers and queues. I do not
expect the CPU usage to reduce linearly, but just want to know how much it
deviates from linear CPU usage.

In my experience, this deviation keeps on increasing with the number of
workers.

Regards,
Siddarth
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17402): https://lists.fd.io/g/vpp-dev/message/17402
Mute This Topic: https://lists.fd.io/mt/76867576/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Vijay Sampath]

Hi Florin,

In the 1 iperf connection test, I get different numbers every time I run.
When I ran today

- iperf and vpp in the same numa core as pci device: 50Gbps (although in
different runs I saw 30Gbps also)
- vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
- vpp and iperf in the other numa as pci device : 36Gbps

But these numbers vary from test to test. But I was never able to get
beyond 50G with 10connections with iperf on the other numa node. As I
mentioned in the previous email, when I repeat this test with Linux TCP as
the server, I am able to get 100G no matter which cores I start iperf on.

Thanks,

Vijay

On Mon, Sep 14, 2020 at 8:30 PM Florin Coras  wrote:

> Hi Vijay,
>
> In this sort of setup, with few connections, probably it’s inevitable to
> lose throughput because of the cross-numa memcpy. In your 1 iperf
> connection test, did you only change iperf’s numa or vpp’s worker as well?
>
> Regards,
> Florin
>
> On Sep 14, 2020, at 6:35 PM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> I ran some experiments by going cross numa, and see that I am not able to
> go beyond 50G. I tried with a different number of worker threads (5, 8 and
> 10), and going upto 10 iperf servers. I am attaching the show run output
> with 10 workers. When I run the same experiment in Linux, I don't see a
> difference in the bandwidth - iperf in both numa nodes are able to achieve
> 100G. Do you have any suggestions on other experiments to try?
>
> I also did try 1 iperf connection - and the bandwidth dropped from 33G to
> 23G for the same numa core vs different.
>
> Thanks,
>
> Vijay
>
> On Sat, Sep 12, 2020 at 2:40 PM Florin Coras 
> wrote:
>
>> Hi VIjay,
>>
>>
>> On Sep 12, 2020, at 12:06 PM, Vijay Sampath  wrote:
>>
>> Hi Florin,
>>
>> On Sat, Sep 12, 2020 at 11:44 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>>
>>> On Sep 12, 2020, at 10:06 AM, Vijay Sampath  wrote:
>>>
>>> Hi Florin,
>>>
>>> On Fri, Sep 11, 2020 at 11:23 PM Florin Coras 
>>> wrote:
>>>
 Hi Vijay,

 Quick replies inline.

 On Sep 11, 2020, at 7:27 PM, Vijay Sampath  wrote:

 Hi Florin,

 Thanks once again for looking at this issue. Please see inline:

 On Fri, Sep 11, 2020 at 2:06 PM Florin Coras 
 wrote:

> Hi Vijay,
>
> Inline.
>
> On Sep 11, 2020, at 1:08 PM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> Thanks for the response. Please see inline:
>
> On Fri, Sep 11, 2020 at 10:42 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Cool experiment. More inline.
>>
>> > On Sep 11, 2020, at 9:42 AM, Vijay Sampath 
>> wrote:
>> >
>> > Hi,
>> >
>> > I am using iperf3 as a client on an Ubuntu 18.04 Linux machine
>> connected to another server running VPP using 100G NICs. Both servers are
>> Intel Xeon with 24 cores.
>>
>> May I ask the frequency for those cores? Also what type of nic are
>> you using?
>>
>
> 2700 MHz.
>
>
> Probably this somewhat limits throughput per single connection
> compared to my testbed where the Intel cpu boosts to 4GHz.
>

 Please see below, I noticed an anomaly.


> The nic is a Pensando DSC100.
>
>
> Okay, not sure what to expect there. Since this mostly stresses the rx
> side, what’s the number of rx descriptors? Typically I test with 256, with
> more connections higher throughput you might need more.
>

 This is the default - comments seem to suggest that is 1024. I don't
 see any rx queue empty errors on the nic, which probably means there are
 sufficient buffers.


 Reasonable. Might want to try to reduce it down to 256 but performance
 will depend a lot on other things as well.

>>>
>>> This seems to help, but I do get rx queue empty nic drops. More below.
>>>
>>>
>>> That’s somewhat expected to happen either when 1) the peer tries to
>>> probe for more throughput and bursts a bit more than we can handle 2) a
>>> full vpp dispatch takes too long, which could happen because of the memcpy
>>> in tcp-established.
>>>
>>>
>>>

 > I am trying to push 100G traffic from the iperf Linux TCP client by
>> starting 10 parallel iperf connections on different port numbers and
>> pinning them to different cores on the sender side. On the VPP receiver
>> side I have 10 worker threads and 10 rx-queues in dpdk, and running 
>> iperf3
>> using VCL library as follows
>> >
>> > taskset 0x00400 sh -c
>> "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libvcl_ldpreload.so
>> VCL_CONFIG=/etc/vpp/vcl.conf iperf3 -s -4 -p 9000" &
>> > taskset 0x00800 sh -c
>> "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libvcl_ldpreload.so
>> VCL_CONFIG=/etc/vpp/vcl.conf iperf3 -s -4 -p 9001" &
>> > taskset 0x01000 sh -c "LD_PRELOAD=/usr/lib/x86_64
>> > ...
>> >
>> > MTU is set to 9216 everywhere, and TCP MSS set to 820

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Vijay Sampath]

Hi Florin,

I just realized that maybe in the VPP case there is an extra copy - once
from mbuf to shared fifo, and once from shared fifo to application buffer.
In Linux, there is probably just the copy from kernel space to user space.
Please correct me if I am wrong. If so, what I am doing is not an apples to
apples comparison.

Thanks,

Vijay

On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath  wrote:

> Hi Florin,
>
> In the 1 iperf connection test, I get different numbers every time I run.
> When I ran today
>
> - iperf and vpp in the same numa core as pci device: 50Gbps (although in
> different runs I saw 30Gbps also)
> - vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
> - vpp and iperf in the other numa as pci device : 36Gbps
>
> But these numbers vary from test to test. But I was never able to get
> beyond 50G with 10connections with iperf on the other numa node. As I
> mentioned in the previous email, when I repeat this test with Linux TCP as
> the server, I am able to get 100G no matter which cores I start iperf on.
>
> Thanks,
>
> Vijay
>
> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> In this sort of setup, with few connections, probably it’s inevitable to
>> lose throughput because of the cross-numa memcpy. In your 1 iperf
>> connection test, did you only change iperf’s numa or vpp’s worker as well?
>>
>> Regards,
>> Florin
>>
>> On Sep 14, 2020, at 6:35 PM, Vijay Sampath  wrote:
>>
>> Hi Florin,
>>
>> I ran some experiments by going cross numa, and see that I am not able to
>> go beyond 50G. I tried with a different number of worker threads (5, 8 and
>> 10), and going upto 10 iperf servers. I am attaching the show run output
>> with 10 workers. When I run the same experiment in Linux, I don't see a
>> difference in the bandwidth - iperf in both numa nodes are able to achieve
>> 100G. Do you have any suggestions on other experiments to try?
>>
>> I also did try 1 iperf connection - and the bandwidth dropped from 33G to
>> 23G for the same numa core vs different.
>>
>> Thanks,
>>
>> Vijay
>>
>> On Sat, Sep 12, 2020 at 2:40 PM Florin Coras 
>> wrote:
>>
>>> Hi VIjay,
>>>
>>>
>>> On Sep 12, 2020, at 12:06 PM, Vijay Sampath  wrote:
>>>
>>> Hi Florin,
>>>
>>> On Sat, Sep 12, 2020 at 11:44 AM Florin Coras 
>>> wrote:
>>>
 Hi Vijay,


 On Sep 12, 2020, at 10:06 AM, Vijay Sampath  wrote:

 Hi Florin,

 On Fri, Sep 11, 2020 at 11:23 PM Florin Coras 
 wrote:

> Hi Vijay,
>
> Quick replies inline.
>
> On Sep 11, 2020, at 7:27 PM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> Thanks once again for looking at this issue. Please see inline:
>
> On Fri, Sep 11, 2020 at 2:06 PM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Inline.
>>
>> On Sep 11, 2020, at 1:08 PM, Vijay Sampath 
>> wrote:
>>
>> Hi Florin,
>>
>> Thanks for the response. Please see inline:
>>
>> On Fri, Sep 11, 2020 at 10:42 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> Cool experiment. More inline.
>>>
>>> > On Sep 11, 2020, at 9:42 AM, Vijay Sampath 
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I am using iperf3 as a client on an Ubuntu 18.04 Linux machine
>>> connected to another server running VPP using 100G NICs. Both servers 
>>> are
>>> Intel Xeon with 24 cores.
>>>
>>> May I ask the frequency for those cores? Also what type of nic are
>>> you using?
>>>
>>
>> 2700 MHz.
>>
>>
>> Probably this somewhat limits throughput per single connection
>> compared to my testbed where the Intel cpu boosts to 4GHz.
>>
>
> Please see below, I noticed an anomaly.
>
>
>> The nic is a Pensando DSC100.
>>
>>
>> Okay, not sure what to expect there. Since this mostly stresses the
>> rx side, what’s the number of rx descriptors? Typically I test with 256,
>> with more connections higher throughput you might need more.
>>
>
> This is the default - comments seem to suggest that is 1024. I don't
> see any rx queue empty errors on the nic, which probably means there are
> sufficient buffers.
>
>
> Reasonable. Might want to try to reduce it down to 256 but performance
> will depend a lot on other things as well.
>

 This seems to help, but I do get rx queue empty nic drops. More below.


 That’s somewhat expected to happen either when 1) the peer tries to
 probe for more throughput and bursts a bit more than we can handle 2) a
 full vpp dispatch takes too long, which could happen because of the memcpy
 in tcp-established.



>
> > I am trying to push 100G traffic from the iperf Linux TCP client by
>>> starting 10 parallel iperf connections on different port numbers and
>>> pinning them to different cores on the sender side. On the VPP receiver
>

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Florin Coras]

Hi Vijay, 


> On Sep 15, 2020, at 8:54 AM, Vijay Sampath  wrote:
> 
> Hi Florin,
> 
> In the 1 iperf connection test, I get different numbers every time I run. 
> When I ran today
> 
> - iperf and vpp in the same numa core as pci device: 50Gbps (although in 
> different runs I saw 30Gbps also)

Is the throughput stable after the first second? If it keeps on fluctuating, 
something else might need to be investigated. If it’s stable, do you see any rx 
drops or tx drops on linux side when throughput is lower? 

> - vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
> - vpp and iperf in the other numa as pci device : 36Gbps

Reasonable, because in the first case the writer to and the reader from the 
fifo (vpp worker and vcl worker/iperf3) are on different numas. 

> 
> But these numbers vary from test to test. But I was never able to get beyond 
> 50G with 10connections with iperf on the other numa node. As I mentioned in 
> the previous email, when I repeat this test with Linux TCP as the server, I 
> am able to get 100G no matter which cores I start iperf on.

This is probably the cost of the extra memcpy. VPP delivers the data to iperf3 
in a shared memory fifo, but iperf3 then wants it copied into its own buffer. 
Unfortunately, we can’t optimize this without changing the apps. 

Regards,
Florin

> 
> Thanks,
> 
> Vijay
> 
> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> In this sort of setup, with few connections, probably it’s inevitable to lose 
> throughput because of the cross-numa memcpy. In your 1 iperf connection test, 
> did you only change iperf’s numa or vpp’s worker as well? 
> 
> Regards,
> Florin
> 
>> On Sep 14, 2020, at 6:35 PM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> I ran some experiments by going cross numa, and see that I am not able to go 
>> beyond 50G. I tried with a different number of worker threads (5, 8 and 10), 
>> and going upto 10 iperf servers. I am attaching the show run output with 10 
>> workers. When I run the same experiment in Linux, I don't see a difference 
>> in the bandwidth - iperf in both numa nodes are able to achieve 100G. Do you 
>> have any suggestions on other experiments to try?
>> 
>> I also did try 1 iperf connection - and the bandwidth dropped from 33G to 
>> 23G for the same numa core vs different.
>> 
>> Thanks,
>> 
>> Vijay
>> 
>> On Sat, Sep 12, 2020 at 2:40 PM Florin Coras > > wrote:
>> Hi VIjay, 
>> 
>> 
>>> On Sep 12, 2020, at 12:06 PM, Vijay Sampath >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> On Sat, Sep 12, 2020 at 11:44 AM Florin Coras >> > wrote:
>>> Hi Vijay, 
>>> 
>>> 
 On Sep 12, 2020, at 10:06 AM, Vijay Sampath >>> > wrote:
 
 Hi Florin,
 
 On Fri, Sep 11, 2020 at 11:23 PM Florin Coras >>> > wrote:
 Hi Vijay, 
 
 Quick replies inline. 
 
> On Sep 11, 2020, at 7:27 PM, Vijay Sampath  > wrote:
> 
> Hi Florin,
> 
> Thanks once again for looking at this issue. Please see inline:
> 
> On Fri, Sep 11, 2020 at 2:06 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Inline.
> 
>> On Sep 11, 2020, at 1:08 PM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> Thanks for the response. Please see inline:
>> 
>> On Fri, Sep 11, 2020 at 10:42 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> Cool experiment. More inline. 
>> 
>> > On Sep 11, 2020, at 9:42 AM, Vijay Sampath > > > wrote:
>> > 
>> > Hi,
>> > 
>> > I am using iperf3 as a client on an Ubuntu 18.04 Linux machine 
>> > connected to another server running VPP using 100G NICs. Both servers 
>> > are Intel Xeon with 24 cores.
>> 
>> May I ask the frequency for those cores? Also what type of nic are you 
>> using?
>> 
>> 2700 MHz. 
> 
> Probably this somewhat limits throughput per single connection compared 
> to my testbed where the Intel cpu boosts to 4GHz. 
>  
> Please see below, I noticed an anomaly. 
> 
> 
>> The nic is a Pensando DSC100.
> 
> Okay, not sure what to expect there. Since this mostly stresses the rx 
> side, what’s the number of rx descriptors? Typically I test with 256, 
> with more connections higher throughput you might need more. 
>  
> This is the default - comments seem to suggest that is 1024. I don't see 
> any rx queue empty errors on the nic, which probably means there are 
> sufficient buffers. 
 
 Reasonable. Might want to try to reduce it down to 256 but performance 
 will depend a lot on other things as well. 
>

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Florin Coras]

Hi Vijay, 

Yes, that is the case for this iperf3 test. The data is already in user space, 
and could be passed to the app in the shape of iovecs, to avoid the extra 
memcpy, but the app would need to be changed to have it release the memory 
whenever it’s done reading it. In case of iperf3 it would be on the spot, 
because it discards it. 

For completeness, note that we don’t currently have vcl apis to expose the fifo 
chunks as iovecs, but they shouldn’t be that difficult. 

Regards,
Florin

> On Sep 15, 2020, at 10:47 AM, Vijay Sampath  wrote:
> 
> Hi Florin,
> 
> I just realized that maybe in the VPP case there is an extra copy - once from 
> mbuf to shared fifo, and once from shared fifo to application buffer. In 
> Linux, there is probably just the copy from kernel space to user space. 
> Please correct me if I am wrong. If so, what I am doing is not an apples to 
> apples comparison.
> 
> Thanks,
> 
> Vijay
> 
> On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath  > wrote:
> Hi Florin,
> 
> In the 1 iperf connection test, I get different numbers every time I run. 
> When I ran today
> 
> - iperf and vpp in the same numa core as pci device: 50Gbps (although in 
> different runs I saw 30Gbps also)
> - vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
> - vpp and iperf in the other numa as pci device : 36Gbps
> 
> But these numbers vary from test to test. But I was never able to get beyond 
> 50G with 10connections with iperf on the other numa node. As I mentioned in 
> the previous email, when I repeat this test with Linux TCP as the server, I 
> am able to get 100G no matter which cores I start iperf on.
> 
> Thanks,
> 
> Vijay
> 
> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> In this sort of setup, with few connections, probably it’s inevitable to lose 
> throughput because of the cross-numa memcpy. In your 1 iperf connection test, 
> did you only change iperf’s numa or vpp’s worker as well? 
> 
> Regards,
> Florin
> 
>> On Sep 14, 2020, at 6:35 PM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> I ran some experiments by going cross numa, and see that I am not able to go 
>> beyond 50G. I tried with a different number of worker threads (5, 8 and 10), 
>> and going upto 10 iperf servers. I am attaching the show run output with 10 
>> workers. When I run the same experiment in Linux, I don't see a difference 
>> in the bandwidth - iperf in both numa nodes are able to achieve 100G. Do you 
>> have any suggestions on other experiments to try?
>> 
>> I also did try 1 iperf connection - and the bandwidth dropped from 33G to 
>> 23G for the same numa core vs different.
>> 
>> Thanks,
>> 
>> Vijay
>> 
>> On Sat, Sep 12, 2020 at 2:40 PM Florin Coras > > wrote:
>> Hi VIjay, 
>> 
>> 
>>> On Sep 12, 2020, at 12:06 PM, Vijay Sampath >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> On Sat, Sep 12, 2020 at 11:44 AM Florin Coras >> > wrote:
>>> Hi Vijay, 
>>> 
>>> 
 On Sep 12, 2020, at 10:06 AM, Vijay Sampath >>> > wrote:
 
 Hi Florin,
 
 On Fri, Sep 11, 2020 at 11:23 PM Florin Coras >>> > wrote:
 Hi Vijay, 
 
 Quick replies inline. 
 
> On Sep 11, 2020, at 7:27 PM, Vijay Sampath  > wrote:
> 
> Hi Florin,
> 
> Thanks once again for looking at this issue. Please see inline:
> 
> On Fri, Sep 11, 2020 at 2:06 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Inline.
> 
>> On Sep 11, 2020, at 1:08 PM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> Thanks for the response. Please see inline:
>> 
>> On Fri, Sep 11, 2020 at 10:42 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> Cool experiment. More inline. 
>> 
>> > On Sep 11, 2020, at 9:42 AM, Vijay Sampath > > > wrote:
>> > 
>> > Hi,
>> > 
>> > I am using iperf3 as a client on an Ubuntu 18.04 Linux machine 
>> > connected to another server running VPP using 100G NICs. Both servers 
>> > are Intel Xeon with 24 cores.
>> 
>> May I ask the frequency for those cores? Also what type of nic are you 
>> using?
>> 
>> 2700 MHz. 
> 
> Probably this somewhat limits throughput per single connection compared 
> to my testbed where the Intel cpu boosts to 4GHz. 
>  
> Please see below, I noticed an anomaly. 
> 
> 
>> The nic is a Pensando DSC100.
> 
> Okay, not sure what to expect there. Since this mostly stresses the rx 
> side, what’s the number of rx descriptors? Typically I test with 256, 
> with more connections higher throughput you mi

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Vijay Sampath]

Hi Florin,

Sure yes, and better still would be for the app to integrate directly with
VPP to even avoid the shared fifo copy, I assume. It's just that the VCL
library gives a quick way to get some benchmark numbers with existing
applications. Thanks for all the help. I have a much better idea now.

Thanks,

Vijay

On Tue, Sep 15, 2020 at 11:25 AM Florin Coras 
wrote:

> Hi Vijay,
>
> Yes, that is the case for this iperf3 test. The data is already in user
> space, and could be passed to the app in the shape of iovecs, to avoid the
> extra memcpy, but the app would need to be changed to have it release the
> memory whenever it’s done reading it. In case of iperf3 it would be on the
> spot, because it discards it.
>
> For completeness, note that we don’t currently have vcl apis to expose the
> fifo chunks as iovecs, but they shouldn’t be that difficult.
>
> Regards,
> Florin
>
> On Sep 15, 2020, at 10:47 AM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> I just realized that maybe in the VPP case there is an extra copy - once
> from mbuf to shared fifo, and once from shared fifo to application buffer.
> In Linux, there is probably just the copy from kernel space to user space.
> Please correct me if I am wrong. If so, what I am doing is not an apples to
> apples comparison.
>
> Thanks,
>
> Vijay
>
> On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath  wrote:
>
>> Hi Florin,
>>
>> In the 1 iperf connection test, I get different numbers every time I run.
>> When I ran today
>>
>> - iperf and vpp in the same numa core as pci device: 50Gbps (although in
>> different runs I saw 30Gbps also)
>> - vpp in the same numa core as pci device, iperf in the other numa :
>> 28Gbps
>> - vpp and iperf in the other numa as pci device : 36Gbps
>>
>> But these numbers vary from test to test. But I was never able to get
>> beyond 50G with 10connections with iperf on the other numa node. As I
>> mentioned in the previous email, when I repeat this test with Linux TCP as
>> the server, I am able to get 100G no matter which cores I start iperf on.
>>
>> Thanks,
>>
>> Vijay
>>
>> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> In this sort of setup, with few connections, probably it’s inevitable to
>>> lose throughput because of the cross-numa memcpy. In your 1 iperf
>>> connection test, did you only change iperf’s numa or vpp’s worker as well?
>>>
>>> Regards,
>>> Florin
>>>
>>> On Sep 14, 2020, at 6:35 PM, Vijay Sampath  wrote:
>>>
>>> Hi Florin,
>>>
>>> I ran some experiments by going cross numa, and see that I am not able
>>> to go beyond 50G. I tried with a different number of worker threads (5, 8
>>> and 10), and going upto 10 iperf servers. I am attaching the show run
>>> output with 10 workers. When I run the same experiment in Linux, I don't
>>> see a difference in the bandwidth - iperf in both numa nodes are able to
>>> achieve 100G. Do you have any suggestions on other experiments to try?
>>>
>>> I also did try 1 iperf connection - and the bandwidth dropped from 33G
>>> to 23G for the same numa core vs different.
>>>
>>> Thanks,
>>>
>>> Vijay
>>>
>>> On Sat, Sep 12, 2020 at 2:40 PM Florin Coras 
>>> wrote:
>>>
 Hi VIjay,


 On Sep 12, 2020, at 12:06 PM, Vijay Sampath  wrote:

 Hi Florin,

 On Sat, Sep 12, 2020 at 11:44 AM Florin Coras 
 wrote:

> Hi Vijay,
>
>
> On Sep 12, 2020, at 10:06 AM, Vijay Sampath 
> wrote:
>
> Hi Florin,
>
> On Fri, Sep 11, 2020 at 11:23 PM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Quick replies inline.
>>
>> On Sep 11, 2020, at 7:27 PM, Vijay Sampath 
>> wrote:
>>
>> Hi Florin,
>>
>> Thanks once again for looking at this issue. Please see inline:
>>
>> On Fri, Sep 11, 2020 at 2:06 PM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> Inline.
>>>
>>> On Sep 11, 2020, at 1:08 PM, Vijay Sampath 
>>> wrote:
>>>
>>> Hi Florin,
>>>
>>> Thanks for the response. Please see inline:
>>>
>>> On Fri, Sep 11, 2020 at 10:42 AM Florin Coras <
>>> fcoras.li...@gmail.com> wrote:
>>>
 Hi Vijay,

 Cool experiment. More inline.

 > On Sep 11, 2020, at 9:42 AM, Vijay Sampath 
 wrote:
 >
 > Hi,
 >
 > I am using iperf3 as a client on an Ubuntu 18.04 Linux machine
 connected to another server running VPP using 100G NICs. Both servers 
 are
 Intel Xeon with 24 cores.

 May I ask the frequency for those cores? Also what type of nic are
 you using?

>>>
>>> 2700 MHz.
>>>
>>>
>>> Probably this somewhat limits throughput per single connection
>>> compared to my testbed where the Intel cpu boosts to 4GHz.
>>>
>>
>> Please see below, I noticed an anomaly.
>>
>>
>>> The nic is a Pensando DSC100.
>>>
>>>
>>> Okay, not sure 

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Florin Coras]

Hi Vijay, 

Currently, builtin applications can only receive data from tcp in a session’s 
rx fifo. That’s a deliberate choice because, at scale, out of order data could 
end up consuming a lot of buffers, i.e., buffers are queued but cannot be 
consumed by the app until the gaps are filled. Still, builtin apps can avoid 
the extra memcpy vcl needs to do for traditional apps. 

Now, there have been talks and we have been considering the option of linking 
vlib buffers into the fifos (to avoid the memcpy) but there’s no ETA for that. 

Regards,
Florin

> On Sep 15, 2020, at 11:32 AM, Vijay Sampath  wrote:
> 
> Hi Florin,
> 
> Sure yes, and better still would be for the app to integrate directly with 
> VPP to even avoid the shared fifo copy, I assume. It's just that the VCL 
> library gives a quick way to get some benchmark numbers with existing 
> applications. Thanks for all the help. I have a much better idea now.
> 
> Thanks,
> 
> Vijay
> 
> On Tue, Sep 15, 2020 at 11:25 AM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Yes, that is the case for this iperf3 test. The data is already in user 
> space, and could be passed to the app in the shape of iovecs, to avoid the 
> extra memcpy, but the app would need to be changed to have it release the 
> memory whenever it’s done reading it. In case of iperf3 it would be on the 
> spot, because it discards it. 
> 
> For completeness, note that we don’t currently have vcl apis to expose the 
> fifo chunks as iovecs, but they shouldn’t be that difficult. 
> 
> Regards,
> Florin
> 
>> On Sep 15, 2020, at 10:47 AM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> I just realized that maybe in the VPP case there is an extra copy - once 
>> from mbuf to shared fifo, and once from shared fifo to application buffer. 
>> In Linux, there is probably just the copy from kernel space to user space. 
>> Please correct me if I am wrong. If so, what I am doing is not an apples to 
>> apples comparison.
>> 
>> Thanks,
>> 
>> Vijay
>> 
>> On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath > > wrote:
>> Hi Florin,
>> 
>> In the 1 iperf connection test, I get different numbers every time I run. 
>> When I ran today
>> 
>> - iperf and vpp in the same numa core as pci device: 50Gbps (although in 
>> different runs I saw 30Gbps also)
>> - vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
>> - vpp and iperf in the other numa as pci device : 36Gbps
>> 
>> But these numbers vary from test to test. But I was never able to get beyond 
>> 50G with 10connections with iperf on the other numa node. As I mentioned in 
>> the previous email, when I repeat this test with Linux TCP as the server, I 
>> am able to get 100G no matter which cores I start iperf on.
>> 
>> Thanks,
>> 
>> Vijay
>> 
>> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> In this sort of setup, with few connections, probably it’s inevitable to 
>> lose throughput because of the cross-numa memcpy. In your 1 iperf connection 
>> test, did you only change iperf’s numa or vpp’s worker as well? 
>> 
>> Regards,
>> Florin
>> 
>>> On Sep 14, 2020, at 6:35 PM, Vijay Sampath >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> I ran some experiments by going cross numa, and see that I am not able to 
>>> go beyond 50G. I tried with a different number of worker threads (5, 8 and 
>>> 10), and going upto 10 iperf servers. I am attaching the show run output 
>>> with 10 workers. When I run the same experiment in Linux, I don't see a 
>>> difference in the bandwidth - iperf in both numa nodes are able to achieve 
>>> 100G. Do you have any suggestions on other experiments to try?
>>> 
>>> I also did try 1 iperf connection - and the bandwidth dropped from 33G to 
>>> 23G for the same numa core vs different.
>>> 
>>> Thanks,
>>> 
>>> Vijay
>>> 
>>> On Sat, Sep 12, 2020 at 2:40 PM Florin Coras >> > wrote:
>>> Hi VIjay, 
>>> 
>>> 
 On Sep 12, 2020, at 12:06 PM, Vijay Sampath >>> > wrote:
 
 Hi Florin,
 
 On Sat, Sep 12, 2020 at 11:44 AM Florin Coras >>> > wrote:
 Hi Vijay, 
 
 
> On Sep 12, 2020, at 10:06 AM, Vijay Sampath  > wrote:
> 
> Hi Florin,
> 
> On Fri, Sep 11, 2020 at 11:23 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Quick replies inline. 
> 
>> On Sep 11, 2020, at 7:27 PM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> Thanks once again for looking at this issue. Please see inline:
>> 
>> On Fri, Sep 11, 2020 at 2:06 PM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> Inline.
>> 
>>> On Sep 11, 2020, at 1:08 PM, Vijay Sampath >> <

Re: [vpp-dev] Query regarding Scaling of CPU utilization with different number of workers

[Andrew Yourtchenko]

Why not  look at CSIT reports with config closest to yours ? 

They test various number of scenarios with varying number of cores... 

Else the question is waaay too general to say anything than “it depends”... 
(Imho). 

--a

> On 15 Sep 2020, at 17:39, siddarth rai  wrote:
> 
> 
> Hello everyone,
> 
> I am using VPP 19.08 and running it in simple forwarding mode. It's handling 
> thousands of connections per second and over 8 Gbps of throughput.
> 
> I am curious to know what the general experience has been with CPU usage per 
> worker while increasing the number of workers and queues. I do not expect the 
> CPU usage to reduce linearly, but just want to know how much it deviates from 
> linear CPU usage.
> 
> In my experience, this deviation keeps on increasing with the number of 
> workers.
> 
> Regards,
> Siddarth
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17409): https://lists.fd.io/g/vpp-dev/message/17409
Mute This Topic: https://lists.fd.io/mt/76867576/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] VPP 20.09 RC2 is tomorrow 16 sep : extraordinarily at *noon* UTC

[Andrew Yourtchenko]

Hi all,

Just a kind reminder that RC2 tagging is tomorrow 16 September. There needs to 
be some urgent work done on  the LF infra side to give us a bit more of 
breathing space, so I extraordinarily will have to do it earlier, at *noon 
UTC*, as opposed to 18:00 UTC.

I will do the tagging once I get all of the stable/2009 outstanding patches in 
- I did have a look at the fixes arriving to master and there were about 15 or 
so extra patches that the 20.09 would benefit as well from.

If you want your fix into RC2 and it’s not already cherry picked - please 
cherry-pick it to stable/2009 and  let me know before 11:00 UTC.

--a /* your friendly 20.09 release manager */-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17410): https://lists.fd.io/g/vpp-dev/message/17410
Mute This Topic: https://lists.fd.io/mt/76872599/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Vijay Sampath]

Hi Florin,

Got it. So what you are saying is that TCP applications cannot directly be
linked with VPP. They have to be a separate process and go through the VCL
library, although they can be optimized to avoid 1 extra memcpy. In future,
memcpy _may_ be avoided completely, but the applications have to still
reside as a separate process.

Thanks,

Vijay

On Tue, Sep 15, 2020 at 11:44 AM Florin Coras 
wrote:

> Hi Vijay,
>
> Currently, builtin applications can only receive data from tcp in a
> session’s rx fifo. That’s a deliberate choice because, at scale, out of
> order data could end up consuming a lot of buffers, i.e., buffers are
> queued but cannot be consumed by the app until the gaps are filled. Still,
> builtin apps can avoid the extra memcpy vcl needs to do for traditional
> apps.
>
> Now, there have been talks and we have been considering the option of
> linking vlib buffers into the fifos (to avoid the memcpy) but there’s no
> ETA for that.
>
> Regards,
> Florin
>
> On Sep 15, 2020, at 11:32 AM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> Sure yes, and better still would be for the app to integrate directly with
> VPP to even avoid the shared fifo copy, I assume. It's just that the VCL
> library gives a quick way to get some benchmark numbers with existing
> applications. Thanks for all the help. I have a much better idea now.
>
> Thanks,
>
> Vijay
>
> On Tue, Sep 15, 2020 at 11:25 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Yes, that is the case for this iperf3 test. The data is already in user
>> space, and could be passed to the app in the shape of iovecs, to avoid the
>> extra memcpy, but the app would need to be changed to have it release the
>> memory whenever it’s done reading it. In case of iperf3 it would be on the
>> spot, because it discards it.
>>
>> For completeness, note that we don’t currently have vcl apis to expose
>> the fifo chunks as iovecs, but they shouldn’t be that difficult.
>>
>> Regards,
>> Florin
>>
>> On Sep 15, 2020, at 10:47 AM, Vijay Sampath  wrote:
>>
>> Hi Florin,
>>
>> I just realized that maybe in the VPP case there is an extra copy - once
>> from mbuf to shared fifo, and once from shared fifo to application buffer.
>> In Linux, there is probably just the copy from kernel space to user space.
>> Please correct me if I am wrong. If so, what I am doing is not an apples to
>> apples comparison.
>>
>> Thanks,
>>
>> Vijay
>>
>> On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath  wrote:
>>
>>> Hi Florin,
>>>
>>> In the 1 iperf connection test, I get different numbers every time I
>>> run. When I ran today
>>>
>>> - iperf and vpp in the same numa core as pci device: 50Gbps (although in
>>> different runs I saw 30Gbps also)
>>> - vpp in the same numa core as pci device, iperf in the other numa :
>>> 28Gbps
>>> - vpp and iperf in the other numa as pci device : 36Gbps
>>>
>>> But these numbers vary from test to test. But I was never able to get
>>> beyond 50G with 10connections with iperf on the other numa node. As I
>>> mentioned in the previous email, when I repeat this test with Linux TCP as
>>> the server, I am able to get 100G no matter which cores I start iperf on.
>>>
>>> Thanks,
>>>
>>> Vijay
>>>
>>> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras 
>>> wrote:
>>>
 Hi Vijay,

 In this sort of setup, with few connections, probably it’s inevitable
 to lose throughput because of the cross-numa memcpy. In your 1 iperf
 connection test, did you only change iperf’s numa or vpp’s worker as well?

 Regards,
 Florin

 On Sep 14, 2020, at 6:35 PM, Vijay Sampath  wrote:

 Hi Florin,

 I ran some experiments by going cross numa, and see that I am not able
 to go beyond 50G. I tried with a different number of worker threads (5, 8
 and 10), and going upto 10 iperf servers. I am attaching the show run
 output with 10 workers. When I run the same experiment in Linux, I don't
 see a difference in the bandwidth - iperf in both numa nodes are able to
 achieve 100G. Do you have any suggestions on other experiments to try?

 I also did try 1 iperf connection - and the bandwidth dropped from 33G
 to 23G for the same numa core vs different.

 Thanks,

 Vijay

 On Sat, Sep 12, 2020 at 2:40 PM Florin Coras 
 wrote:

> Hi VIjay,
>
>
> On Sep 12, 2020, at 12:06 PM, Vijay Sampath 
> wrote:
>
> Hi Florin,
>
> On Sat, Sep 12, 2020 at 11:44 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>>
>> On Sep 12, 2020, at 10:06 AM, Vijay Sampath 
>> wrote:
>>
>> Hi Florin,
>>
>> On Fri, Sep 11, 2020 at 11:23 PM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> Quick replies inline.
>>>
>>> On Sep 11, 2020, at 7:27 PM, Vijay Sampath 
>>> wrote:
>>>
>>> Hi Florin,
>>>
>>> Thanks once again for looking at this issue. Please see inline:
>>>
>>> On Fri, Sep 11, 2020 at 2

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Florin Coras]

Hi Vijay, 

Oh, by no means. Builtin applications, i.e., applications that run within the 
vpp process, are definitely possible (see plugins/hs_apps/echo_client/server or 
the proxy). They run “on” the vpp workers and io/ctrl events are delivered by 
the session layer to those apps using callback functions. However, the session 
layer exchanges data with them using fifos, not vlib buffers. We might consider 
offering the option to improve that for low scale and high throughput 
scenarios, but that’s not possible today. 

Regards,
Florin

> On Sep 15, 2020, at 12:23 PM, Vijay Sampath  wrote:
> 
> Hi Florin,
> 
> Got it. So what you are saying is that TCP applications cannot directly be 
> linked with VPP. They have to be a separate process and go through the VCL 
> library, although they can be optimized to avoid 1 extra memcpy. In future, 
> memcpy _may_ be avoided completely, but the applications have to still reside 
> as a separate process.
> 
> Thanks,
> 
> Vijay
> 
> On Tue, Sep 15, 2020 at 11:44 AM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Currently, builtin applications can only receive data from tcp in a session’s 
> rx fifo. That’s a deliberate choice because, at scale, out of order data 
> could end up consuming a lot of buffers, i.e., buffers are queued but cannot 
> be consumed by the app until the gaps are filled. Still, builtin apps can 
> avoid the extra memcpy vcl needs to do for traditional apps. 
> 
> Now, there have been talks and we have been considering the option of linking 
> vlib buffers into the fifos (to avoid the memcpy) but there’s no ETA for 
> that. 
> 
> Regards,
> Florin
> 
>> On Sep 15, 2020, at 11:32 AM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> Sure yes, and better still would be for the app to integrate directly with 
>> VPP to even avoid the shared fifo copy, I assume. It's just that the VCL 
>> library gives a quick way to get some benchmark numbers with existing 
>> applications. Thanks for all the help. I have a much better idea now.
>> 
>> Thanks,
>> 
>> Vijay
>> 
>> On Tue, Sep 15, 2020 at 11:25 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> Yes, that is the case for this iperf3 test. The data is already in user 
>> space, and could be passed to the app in the shape of iovecs, to avoid the 
>> extra memcpy, but the app would need to be changed to have it release the 
>> memory whenever it’s done reading it. In case of iperf3 it would be on the 
>> spot, because it discards it. 
>> 
>> For completeness, note that we don’t currently have vcl apis to expose the 
>> fifo chunks as iovecs, but they shouldn’t be that difficult. 
>> 
>> Regards,
>> Florin
>> 
>>> On Sep 15, 2020, at 10:47 AM, Vijay Sampath >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> I just realized that maybe in the VPP case there is an extra copy - once 
>>> from mbuf to shared fifo, and once from shared fifo to application buffer. 
>>> In Linux, there is probably just the copy from kernel space to user space. 
>>> Please correct me if I am wrong. If so, what I am doing is not an apples to 
>>> apples comparison.
>>> 
>>> Thanks,
>>> 
>>> Vijay
>>> 
>>> On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath >> > wrote:
>>> Hi Florin,
>>> 
>>> In the 1 iperf connection test, I get different numbers every time I run. 
>>> When I ran today
>>> 
>>> - iperf and vpp in the same numa core as pci device: 50Gbps (although in 
>>> different runs I saw 30Gbps also)
>>> - vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
>>> - vpp and iperf in the other numa as pci device : 36Gbps
>>> 
>>> But these numbers vary from test to test. But I was never able to get 
>>> beyond 50G with 10connections with iperf on the other numa node. As I 
>>> mentioned in the previous email, when I repeat this test with Linux TCP as 
>>> the server, I am able to get 100G no matter which cores I start iperf on.
>>> 
>>> Thanks,
>>> 
>>> Vijay
>>> 
>>> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras >> > wrote:
>>> Hi Vijay, 
>>> 
>>> In this sort of setup, with few connections, probably it’s inevitable to 
>>> lose throughput because of the cross-numa memcpy. In your 1 iperf 
>>> connection test, did you only change iperf’s numa or vpp’s worker as well? 
>>> 
>>> Regards,
>>> Florin
>>> 
 On Sep 14, 2020, at 6:35 PM, Vijay Sampath >>> > wrote:
 
 Hi Florin,
 
 I ran some experiments by going cross numa, and see that I am not able to 
 go beyond 50G. I tried with a different number of worker threads (5, 8 and 
 10), and going upto 10 iperf servers. I am attaching the show run output 
 with 10 workers. When I run the same experiment in Linux, I don't see a 
 difference in the bandwidth - iperf in both numa nodes are able to achieve 
 100G. Do you have any suggestions on other experiments

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Vijay Sampath]

Hi Florin,

I got it now.

Also, I think you mentioned the support in the VCL library for the
application to read/write/free directly from fifo buffers is not yet
present, but can be added with little effort. Is that correct?

Thanks,

Vijay

On Tue, Sep 15, 2020 at 1:31 PM Florin Coras  wrote:

> Hi Vijay,
>
> Oh, by no means. Builtin applications, i.e., applications that run within
> the vpp process, are definitely possible (see
> plugins/hs_apps/echo_client/server or the proxy). They run “on” the vpp
> workers and io/ctrl events are delivered by the session layer to those apps
> using callback functions. However, the session layer exchanges data with
> them using fifos, not vlib buffers. We might consider offering the option
> to improve that for low scale and high throughput scenarios, but that’s not
> possible today.
>
> Regards,
> Florin
>
> On Sep 15, 2020, at 12:23 PM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> Got it. So what you are saying is that TCP applications cannot directly be
> linked with VPP. They have to be a separate process and go through the VCL
> library, although they can be optimized to avoid 1 extra memcpy. In future,
> memcpy _may_ be avoided completely, but the applications have to still
> reside as a separate process.
>
> Thanks,
>
> Vijay
>
> On Tue, Sep 15, 2020 at 11:44 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Currently, builtin applications can only receive data from tcp in a
>> session’s rx fifo. That’s a deliberate choice because, at scale, out of
>> order data could end up consuming a lot of buffers, i.e., buffers are
>> queued but cannot be consumed by the app until the gaps are filled. Still,
>> builtin apps can avoid the extra memcpy vcl needs to do for traditional
>> apps.
>>
>> Now, there have been talks and we have been considering the option of
>> linking vlib buffers into the fifos (to avoid the memcpy) but there’s no
>> ETA for that.
>>
>> Regards,
>> Florin
>>
>> On Sep 15, 2020, at 11:32 AM, Vijay Sampath  wrote:
>>
>> Hi Florin,
>>
>> Sure yes, and better still would be for the app to integrate directly
>> with VPP to even avoid the shared fifo copy, I assume. It's just that the
>> VCL library gives a quick way to get some benchmark numbers with existing
>> applications. Thanks for all the help. I have a much better idea now.
>>
>> Thanks,
>>
>> Vijay
>>
>> On Tue, Sep 15, 2020 at 11:25 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> Yes, that is the case for this iperf3 test. The data is already in user
>>> space, and could be passed to the app in the shape of iovecs, to avoid the
>>> extra memcpy, but the app would need to be changed to have it release the
>>> memory whenever it’s done reading it. In case of iperf3 it would be on the
>>> spot, because it discards it.
>>>
>>> For completeness, note that we don’t currently have vcl apis to expose
>>> the fifo chunks as iovecs, but they shouldn’t be that difficult.
>>>
>>> Regards,
>>> Florin
>>>
>>> On Sep 15, 2020, at 10:47 AM, Vijay Sampath  wrote:
>>>
>>> Hi Florin,
>>>
>>> I just realized that maybe in the VPP case there is an extra copy - once
>>> from mbuf to shared fifo, and once from shared fifo to application buffer.
>>> In Linux, there is probably just the copy from kernel space to user space.
>>> Please correct me if I am wrong. If so, what I am doing is not an apples to
>>> apples comparison.
>>>
>>> Thanks,
>>>
>>> Vijay
>>>
>>> On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath 
>>> wrote:
>>>
 Hi Florin,

 In the 1 iperf connection test, I get different numbers every time I
 run. When I ran today

 - iperf and vpp in the same numa core as pci device: 50Gbps (although
 in different runs I saw 30Gbps also)
 - vpp in the same numa core as pci device, iperf in the other numa :
 28Gbps
 - vpp and iperf in the other numa as pci device : 36Gbps

 But these numbers vary from test to test. But I was never able to get
 beyond 50G with 10connections with iperf on the other numa node. As I
 mentioned in the previous email, when I repeat this test with Linux TCP as
 the server, I am able to get 100G no matter which cores I start iperf on.

 Thanks,

 Vijay

 On Mon, Sep 14, 2020 at 8:30 PM Florin Coras 
 wrote:

> Hi Vijay,
>
> In this sort of setup, with few connections, probably it’s inevitable
> to lose throughput because of the cross-numa memcpy. In your 1 iperf
> connection test, did you only change iperf’s numa or vpp’s worker as well?
>
> Regards,
> Florin
>
> On Sep 14, 2020, at 6:35 PM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> I ran some experiments by going cross numa, and see that I am not able
> to go beyond 50G. I tried with a different number of worker threads (5, 8
> and 10), and going upto 10 iperf servers. I am attaching the show run
> output with 10 workers. When I run the same experiment in Linux, I don't
> see a difference

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Florin Coras]

Hi Vijay, 

Yes. Underneath, the fifos maintain a linked a list of chunks where the data is 
stored. VCL could provide pointers to those in the form of iovecs and another 
api to mark the data as consumed (implicitly release the chunks) once the app 
is done reading. But again, the apps would have to explicitly use these apis. 

Regards,
Florin

> On Sep 15, 2020, at 1:46 PM, Vijay Sampath  wrote:
> 
> Hi Florin,
> 
> I got it now.
> 
> Also, I think you mentioned the support in the VCL library for the 
> application to read/write/free directly from fifo buffers is not yet present, 
> but can be added with little effort. Is that correct?
> 
> Thanks,
> 
> Vijay
> 
> On Tue, Sep 15, 2020 at 1:31 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Oh, by no means. Builtin applications, i.e., applications that run within the 
> vpp process, are definitely possible (see plugins/hs_apps/echo_client/server 
> or the proxy). They run “on” the vpp workers and io/ctrl events are delivered 
> by the session layer to those apps using callback functions. However, the 
> session layer exchanges data with them using fifos, not vlib buffers. We 
> might consider offering the option to improve that for low scale and high 
> throughput scenarios, but that’s not possible today. 
> 
> Regards,
> Florin
> 
>> On Sep 15, 2020, at 12:23 PM, Vijay Sampath > > wrote:
>> 
>> Hi Florin,
>> 
>> Got it. So what you are saying is that TCP applications cannot directly be 
>> linked with VPP. They have to be a separate process and go through the VCL 
>> library, although they can be optimized to avoid 1 extra memcpy. In future, 
>> memcpy _may_ be avoided completely, but the applications have to still 
>> reside as a separate process.
>> 
>> Thanks,
>> 
>> Vijay
>> 
>> On Tue, Sep 15, 2020 at 11:44 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> Currently, builtin applications can only receive data from tcp in a 
>> session’s rx fifo. That’s a deliberate choice because, at scale, out of 
>> order data could end up consuming a lot of buffers, i.e., buffers are queued 
>> but cannot be consumed by the app until the gaps are filled. Still, builtin 
>> apps can avoid the extra memcpy vcl needs to do for traditional apps. 
>> 
>> Now, there have been talks and we have been considering the option of 
>> linking vlib buffers into the fifos (to avoid the memcpy) but there’s no ETA 
>> for that. 
>> 
>> Regards,
>> Florin
>> 
>>> On Sep 15, 2020, at 11:32 AM, Vijay Sampath >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> Sure yes, and better still would be for the app to integrate directly with 
>>> VPP to even avoid the shared fifo copy, I assume. It's just that the VCL 
>>> library gives a quick way to get some benchmark numbers with existing 
>>> applications. Thanks for all the help. I have a much better idea now.
>>> 
>>> Thanks,
>>> 
>>> Vijay
>>> 
>>> On Tue, Sep 15, 2020 at 11:25 AM Florin Coras >> > wrote:
>>> Hi Vijay, 
>>> 
>>> Yes, that is the case for this iperf3 test. The data is already in user 
>>> space, and could be passed to the app in the shape of iovecs, to avoid the 
>>> extra memcpy, but the app would need to be changed to have it release the 
>>> memory whenever it’s done reading it. In case of iperf3 it would be on the 
>>> spot, because it discards it. 
>>> 
>>> For completeness, note that we don’t currently have vcl apis to expose the 
>>> fifo chunks as iovecs, but they shouldn’t be that difficult. 
>>> 
>>> Regards,
>>> Florin
>>> 
 On Sep 15, 2020, at 10:47 AM, Vijay Sampath >>> > wrote:
 
 Hi Florin,
 
 I just realized that maybe in the VPP case there is an extra copy - once 
 from mbuf to shared fifo, and once from shared fifo to application buffer. 
 In Linux, there is probably just the copy from kernel space to user space. 
 Please correct me if I am wrong. If so, what I am doing is not an apples 
 to apples comparison.
 
 Thanks,
 
 Vijay
 
 On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath >>> > wrote:
 Hi Florin,
 
 In the 1 iperf connection test, I get different numbers every time I run. 
 When I ran today
 
 - iperf and vpp in the same numa core as pci device: 50Gbps (although in 
 different runs I saw 30Gbps also)
 - vpp in the same numa core as pci device, iperf in the other numa : 28Gbps
 - vpp and iperf in the other numa as pci device : 36Gbps
 
 But these numbers vary from test to test. But I was never able to get 
 beyond 50G with 10connections with iperf on the other numa node. As I 
 mentioned in the previous email, when I repeat this test with Linux TCP as 
 the server, I am able to get 100G no matter which cores I start iperf on.
 
 Thanks,
 
 Vijay
 
 On Mon, Sep 14, 2020 at 8:30 P

Re: [vpp-dev] 100G with iperf3 server using VCL library

[Vijay Sampath]

Hi Florin,

Sure got it. The options are clear now.

Thanks,

Vijay

On Tue, Sep 15, 2020 at 2:06 PM Florin Coras  wrote:

> Hi Vijay,
>
> Yes. Underneath, the fifos maintain a linked a list of chunks where the
> data is stored. VCL could provide pointers to those in the form of iovecs
> and another api to mark the data as consumed (implicitly release the
> chunks) once the app is done reading. But again, the apps would have to
> explicitly use these apis.
>
> Regards,
> Florin
>
> On Sep 15, 2020, at 1:46 PM, Vijay Sampath  wrote:
>
> Hi Florin,
>
> I got it now.
>
> Also, I think you mentioned the support in the VCL library for the
> application to read/write/free directly from fifo buffers is not yet
> present, but can be added with little effort. Is that correct?
>
> Thanks,
>
> Vijay
>
> On Tue, Sep 15, 2020 at 1:31 PM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Oh, by no means. Builtin applications, i.e., applications that run within
>> the vpp process, are definitely possible (see
>> plugins/hs_apps/echo_client/server or the proxy). They run “on” the vpp
>> workers and io/ctrl events are delivered by the session layer to those apps
>> using callback functions. However, the session layer exchanges data with
>> them using fifos, not vlib buffers. We might consider offering the option
>> to improve that for low scale and high throughput scenarios, but that’s not
>> possible today.
>>
>> Regards,
>> Florin
>>
>> On Sep 15, 2020, at 12:23 PM, Vijay Sampath  wrote:
>>
>> Hi Florin,
>>
>> Got it. So what you are saying is that TCP applications cannot
>> directly be linked with VPP. They have to be a separate process and go
>> through the VCL library, although they can be optimized to avoid 1 extra
>> memcpy. In future, memcpy _may_ be avoided completely, but the applications
>> have to still reside as a separate process.
>>
>> Thanks,
>>
>> Vijay
>>
>> On Tue, Sep 15, 2020 at 11:44 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> Currently, builtin applications can only receive data from tcp in a
>>> session’s rx fifo. That’s a deliberate choice because, at scale, out of
>>> order data could end up consuming a lot of buffers, i.e., buffers are
>>> queued but cannot be consumed by the app until the gaps are filled. Still,
>>> builtin apps can avoid the extra memcpy vcl needs to do for traditional
>>> apps.
>>>
>>> Now, there have been talks and we have been considering the option of
>>> linking vlib buffers into the fifos (to avoid the memcpy) but there’s no
>>> ETA for that.
>>>
>>> Regards,
>>> Florin
>>>
>>> On Sep 15, 2020, at 11:32 AM, Vijay Sampath  wrote:
>>>
>>> Hi Florin,
>>>
>>> Sure yes, and better still would be for the app to integrate directly
>>> with VPP to even avoid the shared fifo copy, I assume. It's just that the
>>> VCL library gives a quick way to get some benchmark numbers with existing
>>> applications. Thanks for all the help. I have a much better idea now.
>>>
>>> Thanks,
>>>
>>> Vijay
>>>
>>> On Tue, Sep 15, 2020 at 11:25 AM Florin Coras 
>>> wrote:
>>>
 Hi Vijay,

 Yes, that is the case for this iperf3 test. The data is already in user
 space, and could be passed to the app in the shape of iovecs, to avoid the
 extra memcpy, but the app would need to be changed to have it release the
 memory whenever it’s done reading it. In case of iperf3 it would be on the
 spot, because it discards it.

 For completeness, note that we don’t currently have vcl apis to expose
 the fifo chunks as iovecs, but they shouldn’t be that difficult.

 Regards,
 Florin

 On Sep 15, 2020, at 10:47 AM, Vijay Sampath  wrote:

 Hi Florin,

 I just realized that maybe in the VPP case there is an extra copy -
 once from mbuf to shared fifo, and once from shared fifo to application
 buffer. In Linux, there is probably just the copy from kernel space to user
 space. Please correct me if I am wrong. If so, what I am doing is not an
 apples to apples comparison.

 Thanks,

 Vijay

 On Tue, Sep 15, 2020 at 8:54 AM Vijay Sampath 
 wrote:

> Hi Florin,
>
> In the 1 iperf connection test, I get different numbers every time I
> run. When I ran today
>
> - iperf and vpp in the same numa core as pci device: 50Gbps (although
> in different runs I saw 30Gbps also)
> - vpp in the same numa core as pci device, iperf in the other numa :
> 28Gbps
> - vpp and iperf in the other numa as pci device : 36Gbps
>
> But these numbers vary from test to test. But I was never able to get
> beyond 50G with 10connections with iperf on the other numa node. As I
> mentioned in the previous email, when I repeat this test with Linux TCP as
> the server, I am able to get 100G no matter which cores I start iperf on.
>
> Thanks,
>
> Vijay
>
> On Mon, Sep 14, 2020 at 8:30 PM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> In

Re: [vpp-dev] Question about acl match/permit behavior.

[Venkat]

Thank you Andrew for taking the time and responding to my questions.
Much appreciated.


On Tue, Sep 15, 2020 at 2:01 AM Andrew 👽 Yourtchenko 
wrote:

> Hi Venkat,
>
> Before doing ACL checks, acl-plugin checks the establshed sessions on
> the given interface.
>
> If an already established session is hit the action is set to "3" and
> no further ACL check is done, which is what you see in your output.
>
> For that case, the ACL# in the trace is a special-case "-1", same as
> lc_index-1 - because we actually never get it to ACL lookup so do not
> need it; and the rule# is the session index; and the trace_bits tells
> us that this session is classifed to use timeout type "3", which is
> defined in acl_timeout_e as "ACL_TIMEOUT_TCP_TRANSIENT" - which makes
> sense, since you are sending the packets in one direction and thus the
> session never establishes and the timeout type never changes to
> "TCP_IDLE".
>
> Of the packets you sent in total, 15002 packets  had hit the ACL#3
> rule and caused the sessions to be created. The rest of the packets
> had hit the established session entries and this is one of those that
> you have captured.
>
> At the time of taking the outputs the traffic was stopped and the
> sessions were deleted by idle timeout (which for the transient state
> is short).
>
> Does this make sense ?
>
> --a
>
>
>
>
> On 9/14/20, Venkat  wrote:
> > Hello Andrew,
> >
> > I am doing a simple test by sending TCP flows from Trex traffic
> generator.
> > Traffic source is 16.0.0.1-16.0.0.100 and destination is
> > 48.0.0.1-48.0.0.100.
> > I am sending TCP traffic with CPS=50 from 100 clients to 100 servers via
> > Trex http profile.
> >
> > I have a reflective ACL to permit the flow for the above src and dst.
> > I expect the packet to hit acl index 3 rule 0 and get permitted.
> > However, I see the following match in the packet trace which doesn't seem
> > like it is hitting acl index 3 and rule 0 but hitting some other acl
> index
> > (acl -1 and lc_index -1).
> > What is the behavior here? Please provide some context here..
> >
> > 03:47:03:874624: acl-plugin-in-ip4-fa
> >
> > acl-plugin: lc_index: -1, sw_if_index 1, next index 1, action: 3, match:
> acl
> > -1 rule 13309 trace_bits 80010303
> >
> > pkt info   
> 43304310
> > 0001010600502d14 0310
> >
> > lc_index 0 l3 ip4 16.0.0.67 -> 48.0.0.67 l4 lsb_of_sw_if_index 1 proto 6
> > l4_is_input 1 l4_slow_path 0 l4_flags 0x01 port 11540 -> 80 tcp flags
> > (valid) 10 rsvd 0
> >
> > DBGvpp# show acl-plugin acl
> >
> > acl-index 0 count 2 tag
> > {8-sra-inbound-sasefwpro-nags-explicit-deny-8}
> >
> > 0: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
> >
> > 1: ipv6 deny src ::/0 dst ::/0 proto 0 sport 0 dport 0
> >
> > applied inbound on sw_if_index: 1
> >
> > applied outbound on sw_if_index:
> >
> > used in lookup context index: 1
> >
> > acl-index 1 count 6 tag
> > {9-sra-inbound-sasefwpro-sase-default-fw-profile}
> >
> > 0: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 6 sport 0-65535
> > dport 0-65535
> >
> > 1: ipv6 permit+reflect src ::/0 dst ::/0 proto 6 sport 0-65535 dport
> > 0-65535
> >
> > 2: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 17 sport
> 0-65535
> > dport 0-65535
> >
> > 3: ipv6 permit+reflect src ::/0 dst ::/0 proto 17 sport 0-65535 dport
> > 0-65535
> >
> > 4: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535
> > dport 0-65535
> >
> > 5: ipv4 permit+reflect src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535
> > dport 0-65535
> >
> > applied inbound on sw_if_index: 1
> >
> > applied outbound on sw_if_index:
> >
> > used in lookup context index: 1
> >
> > acl-index 2 count 6 tag
> > {9-sra-outbound-sasefwpro-sase-default-fw-profile}
> >
> > 0: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 6 sport 0-65535 dport
> > 0-65535
> >
> > 1: ipv6 deny src ::/0 dst ::/0 proto 6 sport 0-65535 dport 0-65535
> >
> > 2: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 17 sport 0-65535 dport
> > 0-65535
> >
> > 3: ipv6 deny src ::/0 dst ::/0 proto 17 sport 0-65535 dport 0-65535
> >
> > 4: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535 dport
> > 0-65535
> >
> > 5: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 1 sport 0-65535 dport
> > 0-65535
> >
> > applied inbound on sw_if_index:
> >
> > applied outbound on sw_if_index: 1
> >
> > used in lookup context index: 0
> >
> > acl-index 3 count 1 tag {100-allow-48}
> >
> > 0: ipv4 permit+reflect src 16.0.0.0/16 dst 48.0.0.0/16 proto 0 sport 0
> dport
> > 0
> >
> > applied inbound on sw_if_index: 1
> >
> > used in lookup context index: 1
> >
> > DBGvpp#
> >
> > DBGvpp# show acl-plugin tables
> >
> > Stats counters enabled for interface ACLs: 0
> >
> > Mask-type entries:
> >
> > 0:    
> >  0800 refcount 4
> >
> > 1:    

[vpp-dev] How does one port use Classify(ACL) to match multiple packets?

[yan mo]

Hi all,
 I want to match these two types of packets:
packets whose destination IP is 1.1.1.1;
ospf packets.

 I have two ideas, but they don't work:
1. Create a table and set the mask to l3 ip4 src proto, but according to the 
matching rules (packet&mask==match) when creating a session, I don't know how 
to configure the match rule;
2. Create two tables, table 1: mask is l3 ip4 src; table 2: mask is l3 ip4 
proto; and create corresponding sessions respectively. But the port only 
supports enable a table.

So what should I do? Does anyone have an idea, thanks in advance.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17417): https://lists.fd.io/g/vpp-dev/message/17417
Mute This Topic: https://lists.fd.io/mt/76880772/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-