Re: [vpp-dev] VPP / tcp_echo performance

[Florin Coras]

Hi Dom, 


> On Dec 12, 2019, at 12:29 PM, dch...@akouto.com wrote:
> 
> Hi Florin,
> 
> The saga continues, a little progress and more questions. In order to reduce 
> the variables, I am now only using VPP on one of the VMs: iperf3 server is 
> running on a VM with native Linux networking, and iperf3+VCL client running 
> on the second VM.

FC: Okay!

> 
> I've pasted the output from a few commands during this test run below and 
> have a few questions if you don't mind.
> The "show errors" command indicates "Tx packet drops (dpdk tx failure)". I 
> have done quite a bit of searching, found other mentions of this in other 
> threads but no tips as to where to look or hints on how it was / can be 
> solved. Any thoughts?
FC: The number of drops is not that large, so we can ignore for now. 
> I'm not really sure how to interpret the results of "show run" but nothing 
> jumps out at me, do you see anything useful in there?
FC: Nothing apart from the fact that one of vpp’s workers is moderately loaded 
(you’re still running 3 workers). 
> Some of the startup.conf options were not working for me, so I switched to 
> building from source (I chose to use tag v20.01-rc0 for some stability). 
> Still no luck with some of the options:
> When I try to use tcp { tso } I get this: 0: tcp_config_fn: unknown input ` 
> tso'
FC: You need to get “closer” to master HEAD. That tag was laid when 19.08 was 
released but tso support was merged afterwards. Typically our CI infra is good 
enough to keep things running so you might want to try master latest. 
> When I try to use num-mbufs in the dpdk section, I get 0: dpdk_config: 
> unknown input `num-mbufs 65535’
FC: This was deprecated at one point. The new stanza is "buffers { 
buffers-per-numa  }"
> 
> Do you know if these options are supported? I can't figure out a way to 
> increase mbufs since the above option does not work, and when I try to use 
> socket-mem (which according to the documentation is needed if there is a need 
> for a larger number of mbufs) I get this: dpdk_config:1408: socket-mem 
> argument is deprecated

FC: Yes, this was also deprecated. 

> 
> To answer some of your questions from your previous reply:
> I have indeed been using taaskset and watching CPU load with top to make sure 
> things are going where I expect them to go
> I am not trying to use jumbo buffers, increasing "default data-size" was just 
> an attempt to see if there would be a difference
> Thanks for the cubic congestion algo suggestion, made the change but no 
> improvement

FC: Understood! I guess that means we should try tso. I just tested it and it 
seems dpdk stanza needs an extra "dpdk {enable-tcp-udp-checksum}” apart from 
“dpdk { dev  { tso on } }”. Let me know if you hit any other issues with 
it. You’ll know that it’s running if you do “show session verbose 2” and you 
see “TSO" in the cfg flags, instead of “TSO off”. 

Regards, 
Florin
> Thank you for all the help, it is very much appreciated.
> 
> Regards,
> Dom
> 
> vpp# sh int
>   Name   IdxState  MTU (L3/IP4/IP6/MPLS) 
> Counter  Count
> GigabitEthernet0/3/0  1  up  9000/0/0/0 rx 
> packets   1642537
> rx bytes  
>  108676814
> tx 
> packets   5216493
> tx bytes  
> 7793319472
> drops 
>392
> ip4   
>1642178
> tx-error  
>475
> local00 down  0/0/0/0   drops 
>  1
> 
> vpp# sh err
>CountNode  Reason
>  1ip4-glean   ARP requests sent
>  7   dpdk-input   no error
>5216424  session-queue Packets transmitted
>  1tcp4-rcv-processPure ACKs received
>  2  tcp4-syn-sent SYN-ACKs received
>  7tcp4-establishedPackets pushed into rx fifo
>1619850tcp4-establishedPure ACKs received
>  22219tcp4-establishedDuplicate ACK
>  1tcp4-establishedResets received
> 62tcp4-establishedConnection closed
>  1tcp4-establishedFINs received
> 62   tcp4-output  Resets sent
>  2arp-reply   ARP replies sent
> 33ip4-input   unkn

Re: [vpp-dev] VPP / tcp_echo performance

[dchons]

Hi Florin,

The saga continues, a little progress and more questions. In order to reduce 
the variables, I am now only using VPP on one of the VMs: iperf3 server is 
running on a VM with native Linux networking, and iperf3+VCL client running on 
the second VM.

I've pasted the output from a few commands during this test run below and have 
a few questions if you don't mind.

* The "show errors" command indicates " *Tx packet drops (dpdk tx failure)* ". 
I have done quite a bit of searching, found other mentions of this in other 
threads but no tips as to where to look or hints on how it was / can be solved. 
Any thoughts?
* I'm not really sure how to interpret the results of "show run" but nothing 
jumps out at me, do you see anything useful in there?
* Some of the startup.conf options were not working for me, so I switched to 
building from source (I chose to use tag v20.01-rc0 for some stability). Still 
no luck with some of the options:

* When I try to use tcp { tso } I get this: *0:* *tcp_config_fn: unknown input 
` tso'*
* When I try to use num-mbufs in the dpdk section, I get *0: dpdk_config: 
unknown input `num-mbufs 65535'*

Do you know if these options are supported? I can't figure out a way to 
increase mbufs since the above option does not work, and when I try to use 
socket-mem (which according to the documentation is needed if there is a need 
for a larger number of mbufs) I get this: *dpdk_config:1408: socket-mem 
argument is deprecated*

To answer some of your questions from your previous reply:

* I have indeed been using taaskset and watching CPU load with top to make sure 
things are going where I expect them to go
* I am not trying to use jumbo buffers, increasing "default data-size" was just 
an attempt to see if there would be a difference
* Thanks for the cubic congestion algo suggestion, made the change but no 
improvement

Thank you for all the help, it is very much appreciated.

Regards,
Dom

*vpp# sh int*
Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          
Count
GigabitEthernet0/3/0              1      up          9000/0/0/0     rx packets  
             1642537
rx bytes               108676814
tx packets               5216493
tx bytes              7793319472
drops                        392
ip4                      1642178
tx-error                     475
local0                            0     down          0/0/0/0       drops       
                   1

*vpp# sh err*
Count                    Node                  Reason
1                ip4-glean               ARP requests sent
7               dpdk-input               no error
5216424              session-queue             Packets transmitted
1            tcp4-rcv-process            Pure ACKs received
2              tcp4-syn-sent             SYN-ACKs received
7            tcp4-established            Packets pushed into rx fifo
1619850            tcp4-established            Pure ACKs received
22219            tcp4-established            Duplicate ACK
1            tcp4-established            Resets received
62            tcp4-established            Connection closed
1            tcp4-established            FINs received
62               tcp4-output              Resets sent
2                arp-reply               ARP replies sent
33                ip4-input               unknown ip protocol
1                ip4-input               Multicast RPF check failed
1                ip4-glean               ARP requests sent
351                llc-input               unknown llc ssap/dsap
475         GigabitEthernet0/3/0-tx        Tx packet drops (dpdk tx failure)

*vpp# sh run*
Thread 0 vpp_main (lcore 7)
Time 94.7, average vectors/node 1.00, last 128 main loops 0.00 per node 0.00
vector rates in 0.e0, out 3.1669e-2, drop 1.0556e-2, punt 0.e0
Name                 State         Calls          Vectors        Suspends       
  Clocks       Vectors/Call
GigabitEthernet0/3/0-output      active                  3               3      
         0          3.29e4            1.00
GigabitEthernet0/3/0-tx          active                  3               3      
         0          3.73e4            1.00
acl-plugin-fa-cleaner-process  event wait                0               0      
         1          2.78e4            0.00
admin-up-down-process          event wait                0               0      
         1          2.24e3            0.00
api-rx-from-ring                any wait                 0               0      
        24          1.01e6            0.00
avf-process                    event wait                0               0      
         1          2.15e4            0.00
bfd-process                    event wait                0               0      
         1          1.49e4            0.00
bond-process                   event wait                0               0      
         1          1.43e4            0.00
dhcp-client-process             any wait                 0               0      
  

Re: [vpp-dev] Did anything ever make vpp's native ipsec stack (ia32) work with dpdk/phy nic?

[Chuan Han via Lists.Fd.Io]

Hi, Damjan,

It worked!

Thanks for helping!

I can see native ipsec stack works with aes-gcm-256 now.

00:44:05:465769: ip4-input-no-checksum
  IPSEC_ESP: 10.10.10.11 -> 10.10.10.10
tos 0x00, ttl 253, length 540, checksum 0x9387
fragment id 0x
00:44:05:465770: ipsec4-input-feature
  IPSEC_ESP: sa_id 2 spd 1 policy 0 spi 255128 (0x0003e498) seq 3478582380
00:44:05:465770: *esp4-decrypt*
  esp: crypto *aes-gcm-256* integrity none pkt-seq -816384916 sa-seq 0
sa-seq-hi 0

vpp# sh ver
vpp v20.01-rc0~735-gbfd7d294d built by root on esdn-lab at Wed Nov 27
19:34:36 UTC 2019
vpp# sh dpdk ver
DPDK Version: DPDK 19.08.0
DPDK EAL init args:   -c 5554 -n 4 --in-memory --log-level debug
--file-prefix vpp -w :1b:10.1 -w :19:00.1 --master-lcore 2
vpp#

8.5Gbps udp/tcp ixia traffic is happy now.

However, I did not see any performance improvement, i.e., still the same
8.5Gbps. Probably, native stack is more stable and well tested as you
mentioned before.

Thanks.
Chuan

On Thu, Dec 5, 2019 at 4:11 PM Damjan Marion  wrote:

>
> Hi Chuan,
>
> You need to specify salt for GCM to work in static config.
>
> i.e.
> ipsec sa add 1 spi 255129 esp crypto-key
> 0123456789012345678901234567890101234567890123456789012345678901 crypto-alg
> aes-gcm-256 salt 0x12345678
>
> LMK if this helps...
>
> --
> Damjan
>
>
> On 27 Nov 2019, at 15:16, Chuan Han  wrote:
>
> I switched cipher from aes-gcm to aes-cbc. native stack works. it seems
> the issue is related to aes-gcm cipher support in native stack?
> Probably some integration bug between aes-gcm and native stack?
>
> On Tue, Nov 19, 2019 at 10:42 AM Chuan Han via Lists.Fd.Io
>   wrote:
>
>> Hi, Damjan,
>>
>> See attachment for detailed logs. no.vdev.xxx files contain the log for
>> the case where vdev is commented out. v.dev.xxx files contain logs for the
>> case where vdev is enabled.
>>
>> I pinged srv-1 from srv-2, i.e., 172.16.2.2 -> 172.16.1.2.
>>
>> When vdev is removed, the srv-1 cannot decrypt the esp pkts. When vdev is
>> enabled, I can see srv-1 decrypted esp pkts and ping worked fine.
>>
>> Thanks.
>> Chuan
>>
>>
>> On Tue, Nov 19, 2019 at 2:08 AM Damjan Marion  wrote:
>>
>>> Hi Chuan,
>>>
>>> Please note that we have daily run of IPSec performance tests in CSIT
>>> with VPP running on the physical NIC with DPDK drivers.
>>> Also please note that every VPP patch is passing unit tests with IETF
>>> and NIST test encryption vectors.
>>>
>>> Other comments inline….
>>>
>>>
>>> > On 18 Nov 2019, at 23:48, Chuan Han via Lists.Fd.Io
>>>   wrote:
>>> >
>>> > Hi, vpp experts,
>>> >
>>> > I was told that vpp's native ipsec stack is stabler and more
>>> performant. We can enable it by commenting out the vdev line in dpdk
>>> stanza.
>>> >
>>> > However, when I did so, ipsec decryption failed.
>>> >
>>> > Ex:
>>> > # commenting out this line makes decryption fail.
>>> > vdev crypto_aesni_mb0,socket_id=0
>>>
>>> Have you validated that in your working case, packets are decrypted
>>> correctly?
>>> Can you share packet trace for both cases?
>>>
>>> >
>>> > Did anyone ever make native ipsec stack, i.e., ia32 work with dpdk/phy
>>> nic?
>>>
>>> yes, it is tested and working on the daily basis.
>>> >
>>> > The interesting thing is no matter whether I comment out the vdev line
>>> or not, ia32 is shown as the active crypto handler for aes-gcm-256. Does
>>> this mean ia32 is used by both cases?
>>> >
>>> > vpp# sh crypto engines
>>> > NamePrioDescription
>>> > ia32100 Intel IA32 ISA Optimized Crypto
>>> > ipsecmb 80  Intel(R) Multi-Buffer Crypto for IPsec
>>> Library 0.52.0
>>> > openssl 50  OpenSSL
>>> > vpp# sh crypto handlers
>>> > AlgoTypeActive  Candidates
>>> > (nil)
>>> > des-cbc encrypt openssl openssl
>>> > decrypt openssl openssl
>>> > 3des-cbcencrypt openssl openssl
>>> > decrypt openssl openssl
>>> > aes-128-cbc encrypt ia32ia32
>>> ipsecmb openssl
>>> > decrypt ia32ia32
>>> ipsecmb openssl
>>> > aes-192-cbc encrypt ia32ia32
>>> ipsecmb openssl
>>> > decrypt ia32ia32
>>> ipsecmb openssl
>>> > aes-256-cbc encrypt ia32ia32
>>> ipsecmb openssl
>>> > decrypt ia32ia32
>>> ipsecmb openssl
>>> > aes-128-ctr encrypt openssl openssl
>>> > decrypt openssl openssl
>>> > aes-192-ctr encrypt openssl openssl
>>> > decrypt openssl openssl
>>> > aes-256-

Re: [vpp-dev] How to add plugin in running VPP ?

[Dave Barach via Lists.Fd.Io]

As of today, plugins are loaded early enough in the vpp init sequence; 
everything which can be done in core engine code can also be done from plugins. 
They are not removed and they are not updated after vpp starts.

Making plugins removable or updateable would take considerable attention to 
detail, and doesn't seem worth the amount of work involved, especially with 
respect to plugin API changes.

Plugins would need "unload_prep" methods, with fine-grained controls to 
minimize unnecessary data-structure teardowns. Best to keep FIBs and session 
tables and so on intact if data structures match.

In my own production use of vpp, I expect a single vpp process to run for up to 
a year. Updates are scheduled in advance. The vpp update portion of the 
exercise typically takes about 10 seconds.

I'd be interested in hearing from the community: should we think about how to 
refactor the entire code base to support arbitrary plugin load / unload / 
reload scenarios? I don't think it's worth the amount of effort involved, but 
that just my view.

FWIW... Dave

From: vpp-dev@lists.fd.io  On Behalf Of chu.penghong
Sent: Thursday, December 12, 2019 3:03 AM
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] How to add plugin in running VPP ?


Hello,

I have read the page about how to add plugin in VPP : 
https://fd.io/docs/vpp/master/gettingstarted/developers/add_plugin.html. 
However, the page describes that plugins can be  added before VPP starts.  I 
wonder if VPP can add plugins when VPP is running.  I found  comments in  file  
src/vlib/unix/plugin.h :
"
  If an application calls vlib_load_new_plugins() -- possibly after
   * changing vlib_plugin_main.plugin_path / 
vlib_plugin_main.plugin_name_filter,
   * -- new plugins will be loaded. That, in turn, allows considerable
   * flexibility in terms of adding feature code or fixing bugs without
   * requiring the data-plane process to restart.
"
It seems that VPP can add plugins when VPP is running.  Can someone show me how 
to achive it ?

Thanks, chupenghong



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#14882): https://lists.fd.io/g/vpp-dev/message/14882
Mute This Topic: https://lists.fd.io/mt/68267897/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Coverity run FAILED as of 2019-12-12 14:00:17 UTC

[Noreply Jenkins]

Coverity run failed today.

Current number of outstanding issues are 2
Newly detected: 0
Eliminated: 0
More details can be found at  
https://scan.coverity.com/projects/fd-io-vpp/view_defects
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#14881): https://lists.fd.io/g/vpp-dev/message/14881
Mute This Topic: https://lists.fd.io/mt/68275061/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Coverity run FAILED as of 2019-12-12 14:00:11 UTC

[Noreply Jenkins]

Coverity run failed today.

Current number of outstanding issues are 2
Newly detected: 0
Eliminated: 0
More details can be found at  
https://scan.coverity.com/projects/fd-io-vpp/view_defects
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#14880): https://lists.fd.io/g/vpp-dev/message/14880
Mute This Topic: https://lists.fd.io/mt/68275057/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [csit-dev] [vpp-dev] Spurious API CRC failures

[Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via Lists.Fd.Io]

> automating_vpp_api_flag_day.rst

Reading that, I see various improvements I wanted to add,
but never found time to do it.
So, here [5] is a quick edit,
with more details on the CSIT side of the process,
and with links to existing changes as examples.

Vratko.

[5] https://gerrit.fd.io/r/c/csit/+/23966

From: csit-...@lists.fd.io  On Behalf Of Dave Wallace
Sent: Thursday, December 12, 2019 12:58 PM
To: Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco) ; 
Dave Barach (dbarach) ; csit-...@lists.fd.io; Maciek 
Konstantynowicz (mkonstan) ; Andrew Yourtchenko 
; Jan Gelety -X (jgelety - PANTHEON TECH SRO at Cisco) 

Cc: vpp-dev@lists.fd.io
Subject: Re: [csit-dev] [vpp-dev] Spurious API CRC failures

Vratko,

Thanks for the info.

Can you please add a section on the proper way to add/manage the collections in 
csit/resources/api/vpp/supported_crcs.yaml to 
csit/docs/automating_vpp_api_flag_day.rst?

-daw-
On 12/11/2019 11:01 AM, Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco) 
wrote:
> Unfortunately when I pushed the patch [2], it failed to pass the crc check

Added explanations to gerrit: [3], [4].

Vratko.

[3] 
https://gerrit.fd.io/r/c/csit/+/23926/1/resources/api/vpp/supported_crcs.yaml#34
[4] 
https://gerrit.fd.io/r/c/csit/+/23926/1/resources/api/vpp/supported_crcs.yaml#273

From: csit-...@lists.fd.io 
 On Behalf Of Dave Wallace
Sent: Wednesday, December 11, 2019 4:40 AM
To: Dave Barach (dbarach) ; 
csit-...@lists.fd.io; Maciek Konstantynowicz 
(mkonstan) ; Andrew Yourtchenko 
; Vratko Polak -X (vrpolak - 
PANTHEON TECH SRO at Cisco) ; Jan 
Gelety -X (jgelety - PANTHEON TECH SRO at Cisco) 

Cc: vpp-dev@lists.fd.io
Subject: Re: [csit-dev] [vpp-dev] Spurious API CRC failures

Correction inline...

On 12/10/2019 10:37 PM, Dave Wallace via Lists.Fd.Io wrote:
Jan/Vratko,

I spent the past several hours attempting to debug this issue.  When testing 
locally, using vpp master HEAD and csit oper-191209, I was able to reproduce 
the problem when running csit/resources/tools/integrated/check_crc.py

After attempting several iterations of reverting [0] and/or [1], I found that 
adding the crc's that were changed in [0] [1] to the main collection in 
csit/resources/api/vpp/supported_crcs.yaml would pass locally.  Unfortunately 
when I pushed the patch [2], it failed to pass the crc check in the 
csit-vpp-device-master-ubuntu1804-1n-skx verify job [2] (which I subsequently 
abandoned).

The changes in [1], don't look correct to me, since they are not related to 
gerrit 21706/17. I would think they either belong in their own collection or 
the main collection should be modified with the new crcs and not the ones in 
the "21706/17" collection.  I tried the latter experiment but the verify check 
failed locally.

At this point, it seems to me that there is a bug in the VppApiChecker but it 
is not clear to me where this root cause is when looking at the code. Or 
perhaps my local runtime environment is not correct.  I'll let you investigate 
further.

Hopefully this will help you resolve the issue quicker.

Thanks,
-daw-
[0] https://gerrit.fd.io/r/c/csit/+/23914
[1] https://gerrit.fd.io/r/c/csit/+/23921
[2] https://gerrit.fd.io/r/c/csit/+/23926

On 12/10/2019 9:03 AM, Dave Barach via Lists.Fd.Io wrote:
Folks,

This patch (among others) https://gerrit.fd.io/r/c/vpp/+/23625 changes zero 
APIs, but fails API CRC validation.

Please fix AYEC. We’re dead in the water.

Thanks... Dave

04:37:59 
/w/workspace/vpp-csit-verify-api-crc-master/src/tools/vppapigen/generate_json.py
04:38:10 Searching '/w/workspace/vpp-csit-verify-api-crc-master/src' for .api 
files.
04:38:10 json files written to: 
/w/workspace/vpp-csit-verify-api-crc-master/build-root/install-vpp-native/vpp/share/vpp/api/.
04:38:10 +++ python3 csit/resources/tools/integrated/check_crc.py
04:38:11 RuntimeError:
04:38:11 Incompatible API CRCs found in .api.json files:
04:38:11 {
04:38:11  "ip_address_details":"0xb1199745",
04:38:11  "ip_address_dump":"0x2d033de4",
04:38:11  "ip_neighbor_add_del":"0x105518b6",
04:38:11  "ip_route_add_del":"0xc1ff832d",
04:38:11  "ip_table_add_del":"0x0ffdaec0",
04:38:11  "sw_interface_ip6nd_ra_config":"0x3eb00b1c"
04:38:11 }
04:38:11 RuntimeError('Incompatible API CRCs found in .api.json files:\n{\n 
"ip_address_details":"0xb1199745",\n "ip_address_dump":"0x2d033de4",\n 
"ip_neighbor_add_del":"0x105518b6",\n "ip_route_add_del":"0xc1ff832d",\n 
"ip_table_add_del":"0x0ffdaec0",\n 
"sw_interface_ip6nd_ra_config":"0x3eb00b1c"\n}',)
04:38:11
04:38:11 @@@
04:38:11
04:38:11 VPP CSIT API CHECK FAIL!
04:38:11
04:38:11 This means the patch under test has missing messages,
04:38:11 or messages with unexpected CRCs compared to what

Re: [csit-dev] [vpp-dev] Spurious API CRC failures

[Dave Wallace]

Vratko,

Thanks for the info.

Can you please add a section on the proper way to add/manage the 
collections in csit/resources/api/vpp/supported_crcs.yaml to 
csit/docs/automating_vpp_api_flag_day.rst?


-daw-

On 12/11/2019 11:01 AM, Vratko Polak -X (vrpolak - PANTHEON TECH SRO at 
Cisco) wrote:


> Unfortunately when I pushed the patch [2], it failed to pass the crc 
check


Added explanations to gerrit: [3], [4].

Vratko.

[3] 
https://gerrit.fd.io/r/c/csit/+/23926/1/resources/api/vpp/supported_crcs.yaml#34


[4] 
https://gerrit.fd.io/r/c/csit/+/23926/1/resources/api/vpp/supported_crcs.yaml#273


*From:*csit-...@lists.fd.io  *On Behalf Of *Dave 
Wallace

*Sent:* Wednesday, December 11, 2019 4:40 AM
*To:* Dave Barach (dbarach) ; csit-...@lists.fd.io; 
Maciek Konstantynowicz (mkonstan) ; Andrew 
Yourtchenko ; Vratko Polak -X (vrpolak - PANTHEON 
TECH SRO at Cisco) ; Jan Gelety -X (jgelety - 
PANTHEON TECH SRO at Cisco) 

*Cc:* vpp-dev@lists.fd.io
*Subject:* Re: [csit-dev] [vpp-dev] Spurious API CRC failures

Correction inline...

On 12/10/2019 10:37 PM, Dave Wallace via Lists.Fd.Io wrote:

Jan/Vratko,

I spent the past several hours attempting to debug this issue. 
When testing locally, using vpp master HEAD and csit oper-191209,
I was able to reproduce the problem when running
csit/resources/tools/integrated/check_crc.py

After attempting several iterations of reverting [0] and/or [1], I
found that adding the crc's that were changed in [0] [1] to the
main collection in csit/resources/api/vpp/supported_crcs.yaml
would pass locally.  Unfortunately when I pushed the patch [2], it
failed to pass the crc check in the
csit-vpp-device-master-ubuntu1804-1n-skx verify job [2] (which I
subsequently abandoned).

The changes in [1], don't look correct to me, since they are not
related to gerrit 21706/17. I would think they either belong in
their own collection or the main collection should be modified
with the new crcs and not the ones in the "21706/17" collection. 
I tried the latter experiment but the verify check failed locally.

At this point, it seems to me that there is a bug in the
VppApiChecker but it is not clear to me where this root cause is
when looking at the code. Or perhaps my local runtime environment
is not correct.  I'll let you investigate further.

Hopefully this will help you resolve the issue quicker.

Thanks,
-daw-
[0] https://gerrit.fd.io/r/c/csit/+/23914
[1] https://gerrit.fd.io/r/c/csit/+/23921
[2] https://gerrit.fd.io/r/c/csit/+/23926

On 12/10/2019 9:03 AM, Dave Barach via Lists.Fd.Io wrote:

Folks,

This patch (among others) https://gerrit.fd.io/r/c/vpp/+/23625
changes zero APIs, but fails API CRC validation.

Please fix AYEC. We’re dead in the water.

Thanks... Dave

**


*04:37:59*/w/workspace/vpp-csit-verify-api-crc-master/src/tools/vppapigen/generate_json.py

*04:38:10*Searching
'/w/workspace/vpp-csit-verify-api-crc-master/src' for .api files.

*04:38:10*json files written to:

/w/workspace/vpp-csit-verify-api-crc-master/build-root/install-vpp-native/vpp/share/vpp/api/.

*04:38:10*+++ python3 csit/resources/tools/integrated/check_crc.py

*04:38:11*RuntimeError:

*04:38:11*Incompatible API CRCs found in .api.json files:

*04:38:11*{

*04:38:11* "ip_address_details":"0xb1199745",

*04:38:11* "ip_address_dump":"0x2d033de4",

*04:38:11* "ip_neighbor_add_del":"0x105518b6",

*04:38:11* "ip_route_add_del":"0xc1ff832d",

*04:38:11* "ip_table_add_del":"0x0ffdaec0",

*04:38:11* "sw_interface_ip6nd_ra_config":"0x3eb00b1c"

*04:38:11*}

*04:38:11*RuntimeError('Incompatible API CRCs found in
.api.json files:\n{\n "ip_address_details":"0xb1199745",\n
"ip_address_dump":"0x2d033de4",\n
"ip_neighbor_add_del":"0x105518b6",\n
"ip_route_add_del":"0xc1ff832d",\n
"ip_table_add_del":"0x0ffdaec0",\n
"sw_interface_ip6nd_ra_config":"0x3eb00b1c"\n}',)

*04:38:11*


*04:38:11*@@@

*04:38:11*

*04:38:11*VPP CSIT API CHECK FAIL!

*04:38:11*

*04:38:11*This means the patch under test has missing messages,

*04:38:11*or messages with unexpected CRCs compared to what
CSIT needs.

*04:38:11*Either this Change and/or its ancestors were editing
.api files,

*04:38:11*or your chain is not rebased upon the recent enough
VPP codebase.

*04:38:11*

*04:38:11*Please rebase the patch to see if that fixes the
problem.

*04:38:11*If that fails email csit-...@lists.fd.io
 for a new

*04:38:11*operational branch supporting the api changes.

*04:38:11

[vpp-dev] #vpp How can I make SR-IOV VF work in VPP in host ?

[jacicson1987]

create a VF with SR-IOV.
VPP take over VF *directly in host.*
Other hosts can not communicate with the VF in VPP because they can not get the 
VF neighbor.
But in VPP, it can ping other hosts by VF, then the hosts get the VF neighbor.
How can I make VF work like Phycics nic?

Test details as follows:

*1. Startup parameters:*
[root@localhost ~]# lsb_release -a
LSB Version:    
:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.7.1908 (Core)
Release:        7.7.1908
Codename:       Core

[root@localhost nete]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg00/Root rhgb quiet 
intel_iommu=on iommu=pt igb.max_vfs=1"
GRUB_DISABLE_RECOVERY="true"
[root@localhost nete]#

*2. VF interface created:
* enp5s0f1 vf 0 will be used. *
*
[root@localhost nete]# ip link show
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode 
DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp5s0f0:  mtu 1500 qdisc mq state DOWN 
mode DEFAULT group default qlen 1000
link/ether a0:36:9f:09:63:9c brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
5: eno1:  mtu 1500 qdisc mq state UP mode 
DEFAULT group default qlen 1000
link/ether ac:1f:6b:02:f5:c0 brd ff:ff:ff:ff:ff:ff
9: virbr0:  mtu 1500 qdisc noqueue state 
DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:a8:d0:4f brd ff:ff:ff:ff:ff:ff
10: virbr0-nic:  mtu 1500 qdisc pfifo_fast master virbr0 
state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:a8:d0:4f brd ff:ff:ff:ff:ff:ff
18: enp5s0f1:  mtu 1500 qdisc mq state UP mode 
DEFAULT group default qlen 1000
link/ether a0:36:9f:09:63:9d brd ff:ff:ff:ff:ff:ff
vf 0 MAC e2:d6:86:e3:d1:17, spoof checking off, link-state auto, trust on
19: enp1s0:  mtu 1500 qdisc mq state DOWN 
mode DEFAULT group default qlen 1000
link/ether 00:1b:21:bb:3d:00 brd ff:ff:ff:ff:ff:ff

*3. Change driver to vfio-pci:*
[root@localhost ~]# lsmod | grep vfio
vfio_pci               41412  0
vfio_iommu_type1       22440  0
vfio                   32657  2 vfio_iommu_type1,vfio_pci
irqbypass              13503  2 kvm,vfio_pci

[root@localhost nete]# dpdk-devbind.py -s

Network devices using DPDK-compatible driver

:05:10.0 'I350 Ethernet Controller Virtual Function' drv=vfio-pci 
unused=igbvf,uio_pci_generic
:05:10.1 'I350 Ethernet Controller Virtual Function' drv=vfio-pci 
unused=igbvf,uio_pci_generic
:0b:00.0 'I210 Gigabit Network Connection' drv=vfio-pci 
unused=igb,uio_pci_generic

Network devices using kernel driver
===
:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0 
drv=ixgbe unused=vfio-pci,uio_pci_generic
:05:00.0 'I350 Gigabit Network Connection' if=enp5s0f0 drv=igb 
unused=vfio-pci,uio_pci_generic
:05:00.1 'I350 Gigabit Network Connection' if=enp5s0f1 drv=igb 
unused=vfio-pci,uio_pci_generic
:0a:00.0 'I210 Gigabit Network Connection' if=eno1 drv=igb 
unused=vfio-pci,uio_pci_generic

Other network devices
=


Crypto devices using DPDK-compatible driver
===


Crypto devices using kernel driver
==


Other crypto devices


VPP take charge to

*4. start VPP:*
[root@localhost nete]# vppctl
___    _        _   _  ___
__/ __/ _ \  (_)__    | | / / _ \/ _ \
_/ _// // / / / _ \   | |/ / ___/ ___/
/_/ /(_)_/\___/   |___/_/  /_/

vpp# show int
Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          
Count
GigabitEthernetb/0/0              3     down         9000/0/0/0
VirtualFunctionEthernet5/10/0     1     down         9000/0/0/0
VirtualFunctionEthernet5/10/1     2      up          9000/0/0/0     rx packets  
                   1
rx bytes                      60
tx packets                     3
tx bytes                     278
drops                          1
local0                            0     down          0/0/0/0       drops       
                   1
vpp# show int addr
GigabitEthernetb/0/0 (dn):
VirtualFunctionEthernet5/10/0 (dn):
VirtualFunctionEthernet5/10/1 (up):
L3 10.2.21.210/24
L3 240e:ff:e000:c:a236:9fff:fe09:210d/64
local0 (dn):
vpp#

It seems VirtualFunctionEthernet5/10/1 can tx/rx packets.
*5. But another host can not ping it.*

[ckun@localhost ~]$ ping6 240e:ff:e000:c:a236:9fff:fe09:210d
PING 240e:ff:e000:c:a236:9fff:fe09:210d(240e:ff:e000:c:a236:9fff:fe09:210d) 56 
data bytes
>From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=1 Destination unreachable: 
>Address unreachable
>From 240e:ff:e000:c:4735:879f:4b07:76e icmp_

[vpp-dev] How to add plugin in running VPP ?

[chu.penghong]

Hello,


I have read the page about how to add plugin in VPP : 
https://fd.io/docs/vpp/master/gettingstarted/developers/add_plugin.html. 
However, the page describes that plugins can be  added before VPP starts.  I 
wonder if VPP can add plugins when VPP is running.  I found  comments in  file  
src/vlib/unix/plugin.h :

"
  If an application calls vlib_load_new_plugins() -- possibly after
   * changing vlib_plugin_main.plugin_path / 
vlib_plugin_main.plugin_name_filter,
   * -- new plugins will be loaded. That, in turn, allows considerable
   * flexibility in terms of adding feature code or fixing bugs without
   * requiring the data-plane process to restart.

"
It seems that VPP can add plugins when VPP is running.  Can someone show me how 
to achive it ?



Thanks, chupenghong
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#14875): https://lists.fd.io/g/vpp-dev/message/14875
Mute This Topic: https://lists.fd.io/mt/68267897/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-