Re: [vpp-dev] project call

[Christian Hopps]

"Damjan Marion via lists.fd.io"  writes:


Today on the project call, we were discussing about doing some changes on those 
calls.

Proposal from few of us who were attending is:

- we should make it less formal, stop doing readouts and focus on specific 
topics to discuss, if there is no topics to discuss we simply drop


By readouts do you mean status of various areas? I think the status reports are 
useful. I have 3 meetings scheduled in the time slot and so I haven't been able 
to attend for the most part; however being able to read minutes later is a way 
for me to not miss out on this useful information.


- we move to biweekly, same time, 2nd and 4th Tue each month


This sounds good to me. Perhaps with the frequency increase you'd like to cut 
down on the process (first point?) maybe just do readouts once a month then?


- to increase attendance we encourage people having something to discuss to 
announce that in advance on the mailing list



- we should preferably use this mailing list if we need to make any decisions


This makes sense, like IETF, you can use the meeting to raise/discuss issues 
and the list to make the decision on any questions that arise.

Thanks,
Chris.



What other people think?

Thanks,

—
Damjan









signature.asc
Description: PGP signature

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20477): https://lists.fd.io/g/vpp-dev/message/20477
Mute This Topic: https://lists.fd.io/mt/86936610/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

#notuseful [Re: [vpp-dev] #vppcom, #vnet #vpp]

[Christian Hopps]

People who have to sort through lots of email (like myself) do not open every 
email before marking it read or deleting it. Email with subject lines that give 
no clue to the content are instant deletes for me (and many others I'm sure) -- 
except in this case I decided to respond b/c for some reason I see a lot of 
these '#hashtag' subject lines on mail to the vpp-dev list for some bizarre 
reason.

Chris.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19626): https://lists.fd.io/g/vpp-dev/message/19626
Mute This Topic: https://lists.fd.io/mt/83734441/21656
Mute #notuseful:https://lists.fd.io/g/vpp-dev/mutehashtag/notuseful
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp in a non-previleged mode

[Christian Hopps]

Venumadhav Josyula  writes:


Hi Christian,

As mentioned in my previous email, we want to give Intel based ( 10G
) for the our packet processing so we need dpdk. With dpdk being
there i was trying to explore if we can run in non-previlege mode ...


If you are trying to take over PCI HW directly then you need to run privileged. 
You can get away with not running privileged when using virtual devices, but 
not actual physical HW.

Thanks,
Chris.



Thanks,
Regards,
Venu

On Fri, 18 Jun 2021 at 13:49, Christian Hopps 
wrote:


Venumadhav Josyula  writes:

> Hi Damien,
>
>>I’m asking because in vpp we have also native 
>> drivers for some NICs and paravirtualized >devices, and those
>> drivers are working in the >non-priv mode.

> You mean to say native NICs  para virtualized drivers ??? If
yes can
> share details.

What HW are you using is what he's asking. You may not need DPDK
at all, in which case you do not need to run vpp privileged.

Thanks,
Chris.

> So are you also implying that it might not be possible to run
vpp
> with dpdk in non-previlege mode ?, Which would be mean that
that pod
> need to have previlege mode
>
> Thanks
> Regards
> Venu
>
> On Fri, 18 Jun, 2021, 12:52 pm Damjan Marion, 
wrote:
>
>
>     I’m asking because in vpp we have also native drivers for
some
>     NICs and paravirtualized devices, and those drivers are
working
>     in the non-priv mode.
>
>     — 
>     Damjan
>
>     On 18.06.2021., at 09:12, Venumadhav Josyula <
vjosy...@gmail.com>
>     wrote:
>
>
>
>         Hi Damjan,
>
>         We need dpdk, the reason being that packets from the
NICs (
>         pollmode ) need to come inside our packet processing sw
(
>         GTPU). So we wanted to use dpdk for the same. Now we
wanted
>         to know the pod in which vpp is running in
non-previleged
>         mode.
>
>         Now we have questions
>         i) is it possible ?
>         ii) if yes how ?
>         Now we were looking at below link for examples, but no
>         luck... they non-previleged running vpp + dpdk had some
>         problems.
>         https://github.com/cncf/cnf-testbed/issues/291
>
>         Hence i am trying to check in the community.
>
>         Thanks,
>         Regards,
>         Venu
>
>
>         On Fri, 18 Jun 2021 at 12:20, Damjan Marion <
dmar...@me.com>
>         wrote:
>
>
>
>             Why do you need dpdk?
>
>             — 
>             Damjan
>
>
>                 On 18.06.2021., at 06:47, Venumadhav Josyula <
>                 vjosy...@gmail.com> wrote:
>
>
>                 Hi Christian,
    >
>                 Can you please share the exact steps please ?
>
>                 Thanks,
>                 Regards,
>                 Venu
>
>                 On Thu, 17 Jun 2021 at 21:25, Christian Hopps <
>                 cho...@chopps.org> wrote:
>
>
>                     "Venumadhav Josyula" 
writes:
>
>                     > Hi All,
>                     >
>                     > Can you run vpp + dpdk in non-privileged
mode ?
>                     This vpp running
>                     > inside pod as a cnf
>
>                     I did this at one point, IIRC I had to
disable
>                     something small bit of code in the
>                     dpdk_early_init that required root, but as
this
>                     code was only required to do something
directly
>                     with the HW later, it wasn't needed in the
>                     container/virtual case.
>
>                     Thanks,
>                     Chris.
>
>                     >
>                     > Thanks,
>                     > Regards,
>                     > Venu
>                     >
>                     >
>                     >
>                     >
>
>
>
>
>
>
>
>         
>
>





signature.asc
Description: PGP signature

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19614): https://lists.fd.io/g/vpp-dev/message/19614
Mute This Topic: https://lists.fd.io/mt/83551481/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp in a non-previleged mode

[Christian Hopps]

Venumadhav Josyula  writes:


Hi Damien,


I’m asking because in vpp we have also native 
drivers for some NICs and paravirtualized >devices, and those
drivers are working in the >non-priv mode.



You mean to say native NICs  para virtualized drivers ??? If yes can
share details.


What HW are you using is what he's asking. You may not need DPDK at all, in 
which case you do not need to run vpp privileged.

Thanks,
Chris.


So are you also implying that it might not be possible to run vpp
with dpdk in non-previlege mode ?, Which would be mean that that pod
need to have previlege mode

Thanks
Regards
Venu

On Fri, 18 Jun, 2021, 12:52 pm Damjan Marion,  wrote:


I’m asking because in vpp we have also native drivers for some
NICs and paravirtualized devices, and those drivers are working
in the non-priv mode.

— 
Damjan

On 18.06.2021., at 09:12, Venumadhav Josyula 
wrote:



Hi Damjan,

We need dpdk, the reason being that packets from the NICs (
pollmode ) need to come inside our packet processing sw (
GTPU). So we wanted to use dpdk for the same. Now we wanted
to know the pod in which vpp is running in non-previleged
mode.

Now we have questions
i) is it possible ?
ii) if yes how ?
Now we were looking at below link for examples, but no
luck... they non-previleged running vpp + dpdk had some
problems.
https://github.com/cncf/cnf-testbed/issues/291

Hence i am trying to check in the community.

Thanks,
Regards,
Venu


On Fri, 18 Jun 2021 at 12:20, Damjan Marion 
wrote:



Why do you need dpdk?

— 
Damjan


On 18.06.2021., at 06:47, Venumadhav Josyula <
vjosy...@gmail.com> wrote:


Hi Christian,

Can you please share the exact steps please ?

Thanks,
Regards,
Venu

On Thu, 17 Jun 2021 at 21:25, Christian Hopps <
cho...@chopps.org> wrote:


"Venumadhav Josyula"  writes:

> Hi All,
>
> Can you run vpp + dpdk in non-privileged mode ?
This vpp running
> inside pod as a cnf

I did this at one point, IIRC I had to disable
something small bit of code in the
dpdk_early_init that required root, but as this
code was only required to do something directly
with the HW later, it wasn't needed in the
container/virtual case.

Thanks,
Chris.

>
> Thanks,
> Regards,
> Venu
>
>
>
>















signature.asc
Description: PGP signature

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19609): https://lists.fd.io/g/vpp-dev/message/19609
Mute This Topic: https://lists.fd.io/mt/83551481/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp in a non-previleged mode

[Christian Hopps]

"Venumadhav Josyula"  writes:


Hi All,

Can you run vpp + dpdk in non-privileged mode ? This vpp running
inside pod as a cnf


I did this at one point, IIRC I had to disable something small bit of code in 
the dpdk_early_init that required root, but as this code was only required to 
do something directly with the HW later, it wasn't needed in the 
container/virtual case.

Thanks,
Chris.



Thanks,
Regards,
Venu








signature.asc
Description: PGP signature

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19602): https://lists.fd.io/g/vpp-dev/message/19602
Mute This Topic: https://lists.fd.io/mt/83551481/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] trex and vpp ipv6?

[Christian Hopps]

> On Feb 28, 2021, at 8:58 PM, hemant via lists.fd.io 
>  wrote:
> 
> Hi Chris,
> 
> I do have VPP and trex directly connected. Port 0 on trex sends packets to 
> VPP TenGig3/0/0 and VPP needs to switch the ingress packet to TenGig3/0/1 and 
> back to trex’s Port 1.  I am using stateless trex.
> 
> I did use MAC addresses in trex_cfg.yaml, but not “version : 2”.  I have also 
> tried a pccp file in but didn’t work.   The trex packet has reached VPP but 
> VPP fails to switch the packet arriving on TenGig3/0/0 to TenGig3/0/1.  I do 
> have static IPv6 neighbor configured on VPP to avoid VPP sending an NS.  
> However, VPP is still sending an NS to which trex does reply.  Soon as I use 
> service mode on trex and scan6, packets arrive at trex.  But service mode 
> can’t be used to test traffic.
> 
> I am using latest VPP downloaded from gerrit today and also using a latest 
> trex (2.88) downloaded from github.   I know VPP CSIT runs IPv6 tests and 
> CSIT uses trex.  It would e good to know what does CSIT use.
> 
> This is how I setup IPv6 on VPP using a script.
> 
> enable ip6 interface TenGigabitEthernet3/0/0
> enable ip6 interface TenGigabitEthernet3/0/1
> set int ip addr TenGigabitEthernet3/0/0 2001:420:380:dead:beef:d06:f001:1/128
> set int ip addr TenGigabitEthernet3/0/1 2001:420:380:dead:beef:d06:f002:1/128
> set int state TenGigabitEthernet3/0/0 up
> set int state TenGigabitEthernet3/0/1 up
> ip6 nd TenGigabitEthernet3/0/0 ra-suppress
> ip6 nd TenGigabitEthernet3/0/1 ra-suppress
> set ip neighbor TenGigabitEthernet3/0/0 2001:420:380:dead:beef:d06:f001:2 
> 00:25:90:eb:e8:ec
> set ip neighbor TenGigabitEthernet3/0/1 2001:420:380:dead:beef:d06:f002:2 
> 00:25:90:eb:e8:ed
> ip route add 2001:420:380:dead:beef:d06:f001:2/128 via 
> 2001:420:380:dead:beef:d06:f001:1 TenGigabitEthernet3/0/0
> ip route add 2001:420:380:dead:beef:d06:f002:2/128 via 
> 2001:420:380:dead:beef:d06:f002:1 TenGigabitEthernet3/0/1

I believe your "via" is wrong. It should point to trex not your local IP. Try:

ip route add 2001:420:380:dead:beef:d06:f001:2/128 via TenGigabitEthernet3/0/0

Although I'd maybe try a non-"p2p" style configuration using /64 shared 
prefixes for each interface.

Also, I think your going to need routes that match the actual traffic your 
generating from trex.

Thanks,
Chris.

> clear run
> 
> thanks,
> 
> Hemant
> 
> From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>  <mailto:vpp-dev@lists.fd.io>> On Behalf Of Christian Hopps
> Sent: Sunday, February 28, 2021 7:52 PM
> To: hem...@mnkcg.com <mailto:hem...@mnkcg.com>
> Cc: Christian Hopps mailto:cho...@chopps.org>>; 
> vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
> Subject: Re: [vpp-dev] trex and vpp ipv6?
> 
> I've got this to work. Try using MAC addresses instead of IP in your 
> trex_cfg.yaml
> 
> - port_limit: 2
>   version   : 2
>   low_end   : false
>   interfaces: ["3b:00.0", "3b:00.1"]
>   port_info :
>  - src_mac  : b8:59:9f:xx:xx:xx
>dest_mac : b8:59:9f:yy:yy:yy
>  - src_mac  : b8:59:9f:xx:xx:xx
>dest_mac : b8:59:9f:yy:yy:yy
> 
> (replace xx and yy)
> 
> I use python client code that constructs IPv6 packets using scapy, there's 
> some other trick for using IPv4 pcap files if that's what you use, google 
> will help there. I also use static ipv6 neighbors in VPP to point back to 
> trex, but I don't know if that's needed.
> 
> Thanks,
> Chris.
> 
> 
> On Feb 27, 2021, at 8:52 PM, hemant via lists.fd.io <http://lists.fd.io/> 
> mailto:hemant=mnkcg@lists.fd.io>> wrote:
> 
> Folks,
> 
> If anyone could get ipv6 traffic testing work between a VPP node and trex, 
> could you please how you made trex work? I have tried a few things including 
> setting up static neighbors on the vpp node and trex does not show any mac 
> unresolved or any other issue.  But trex is unable to send any packets to the 
> VPP node.
> 
> I could make trex work with IPv4 and the VPP node using “ip” and default_gw” 
> in trex yaml file.
> 
> Thanks,
> 
> Hemant
> 
> 
> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18821): https://lists.fd.io/g/vpp-dev/message/18821
Mute This Topic: https://lists.fd.io/mt/80964994/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] trex and vpp ipv6?

[Christian Hopps]

I've got this to work. Try using MAC addresses instead of IP in your 
trex_cfg.yaml

- port_limit: 2
  version   : 2
  low_end   : false
  interfaces: ["3b:00.0", "3b:00.1"]
  port_info :
 - src_mac  : b8:59:9f:xx:xx:xx
   dest_mac : b8:59:9f:yy:yy:yy
 - src_mac  : b8:59:9f:xx:xx:xx
   dest_mac : b8:59:9f:yy:yy:yy

(replace xx and yy)

I use python client code that constructs IPv6 packets using scapy, there's some 
other trick for using IPv4 pcap files if that's what you use, google will help 
there. I also use static ipv6 neighbors in VPP to point back to trex, but I 
don't know if that's needed.

Thanks,
Chris.

> On Feb 27, 2021, at 8:52 PM, hemant via lists.fd.io 
>  wrote:
> 
> Folks,
> 
> If anyone could get ipv6 traffic testing work between a VPP node and trex, 
> could you please how you made trex work? I have tried a few things including 
> setting up static neighbors on the vpp node and trex does not show any mac 
> unresolved or any other issue.  But trex is unable to send any packets to the 
> VPP node.
> 
> I could make trex work with IPv4 and the VPP node using “ip” and default_gw” 
> in trex yaml file.
> 
> Thanks,
> 
> Hemant
> 
> 



signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18818): https://lists.fd.io/g/vpp-dev/message/18818
Mute This Topic: https://lists.fd.io/mt/80964994/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VPP crash while allocation buffer in Azure with MLX5

[Christian Hopps]

No I don't think it was a build issue. I think it doesn't work. :)

I don't have the cycles currently to circle back and check again, perhaps in 
the future.

Thanks,
Chris.

> On Jan 5, 2021, at 3:11 AM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
>> Isn't dpdk/connectx-5 broken in 20.09?
> 
> I am not sure - I did not test it, but if you refer to the build issue, it 
> was caused by meson and this has been reverted from 20.09 if I am not 
> mistaken (and should be fixed in master).
> 
> Best
> ben
> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18473): https://lists.fd.io/g/vpp-dev/message/18473
Mute This Topic: https://lists.fd.io/mt/79422123/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VPP crash while allocation buffer in Azure with MLX5

[Christian Hopps]

Isn't dpdk/connectx-5 broken in 20.09?

Thanks,
Chris.

> On Jan 4, 2021, at 9:02 AM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi,
> 
> VPP 19.08 is no longer supported. Can you try with VPP 20.09 instead?
> 
> Best
> ben
> 
>> -Original Message-
>> From: vpp-dev@lists.fd.io  > > On Behalf Of Himanshu
>> Rakshit
>> Sent: lundi 4 janvier 2021 12:42
>> To: vpp-dev@lists.fd.io 
>> Subject: [vpp-dev] VPP crash while allocation buffer in Azure with MLX5
>> 
>> Hi All,
>> 
>> We are observing the the following crash in VPP while running in Azure
>> with MLX5
>> 
>> Program received signal SIGABRT, Aborted.
>> 0x7f8c2387c387 in raise () from /lib64/libc.so.6
>> (gdb) bt
>> #0  0x7f8c2387c387 in raise () from /lib64/libc.so.6
>> #1  0x7f8c2387da78 in abort () from /lib64/libc.so.6
>> #2  0x0040755a in os_panic () at /home/vpp/src/vpp/vnet/main.c:355
>> #3  0x7f8c2461eb39 in debugger () at /home/vpp/src/vppinfra/error.c:84
>> #4  0x7f8c2461ef08 in _clib_error (how_to_die=2, function_name=0x0,
>> line_number=0, fmt=0x7f8bdffa7df8 "%s:%d (%s) assertion `%s' fails")
>>at /home/vpp/src/vppinfra/error.c:143
>> #5  0x7f8bdfd38823 in vlib_get_buffer_index (vm=0x7f8c25479d80
>> , p=0x17f19ed00)
>>at /home/vpp/src/vlib/buffer_funcs.h:261
>> #6  0x7f8bdfd38b87 in vlib_get_buffer_indices_with_offset
>> (vm=0x7f8c25479d80 , b=0x7f8be48c5fc8,
>> bi=0x7f8be4a1c514,
>>count=2, offset=128) at /home/vpp/src/vlib/buffer_funcs.h:322
>> #7  0x7f8bdfd3ae2d in dpdk_device_input (vm=0x7f8c25479d80
>> , dm=0x7f8be06968e0 , xd=0x7f8be494e240,
>>node=0x7f8be3afae80, thread_index=0, queue_id=0) at
>> /home/vpp/src/plugins/dpdk/device/node.c:371
>> #8  0x7f8bdfd3b362 in dpdk_input_node_fn_avx2 (vm=0x7f8c25479d80
>> , node=0x7f8be3afae80, f=0x0)
>>at /home/vpp/src/plugins/dpdk/device/node.c:469
>> #9  0x7f8c251d6c42 in dispatch_node (vm=0x7f8c25479d80
>> , node=0x7f8be3afae80, type=VLIB_NODE_TYPE_INPUT,
>>dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
>> last_time_stamp=265582086742909)
>>at /home/vpp/src/vlib/main.c:1209
>> #10 0x7f8c251d8d94 in vlib_main_or_worker_loop (vm=0x7f8c25479d80
>> , is_main=1)
>>at /home/vpp/src/vlib/main.c:1781
>> #11 0x7f8c251d98a9 in vlib_main_loop (vm=0x7f8c25479d80
>> )
>>at /home/vpp/src/vlib/main.c:1930
>> #12 0x7f8c251da571 in vlib_main (vm=0x7f8c25479d80 ,
>> input=0x7f8be2695fb0)
>>at /home/vpp/src/vlib/main.c:2147
>> #13 0x7f8c25240533 in thread0 (arg=140239897599360) at
>> /home/vpp/src/vlib/unix/main.c:640
>> #14 0x7f8c2463f9b0 in clib_calljmp ()
>>   from /home/vpp/build-root/install-vpp_debug-
>> native/vpp/lib/libvppinfra.so.19.08.1
>> #15 0x7ffc224c65e0 in ?? ()
>> #16 0x7f8c25240aa9 in vlib_unix_main (argc=67, argv=0x855030)
>>at /home/vpp/src/vlib/unix/main.c:710
>> #17 0x00406ece in main (argc=67, argv=0x855030) at
>> /home/vpp/src/vpp/vnet/main.c:280
>> 
>> Details:
>> VPP Version: 19.08
>> DPDK version: 19.05
>> Linux Version: CentOS Linux release 7.8.2003 (Core)
>> Kernel version: 3.10.0-1127.19.1.el7.x86_64
>> Step Followed: https://fd.io/docs/vpp/master/usecases/vppinazure.html
>> 
>> Interfaces are coming up properly, normal ping is working fine but when we
>> are trying to run more data we are running into this issue.
>> 
>> 
>> More Info:
>> - 1G hge pages is configured
>> - We can see the following error in vpp show logging
>> 
>> [root@exe91-fpm vpp]# lspci
>> :00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX
>> Host bridge (AGP disabled) (rev 03)
>> :00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev
>> 01)
>> :00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev
>> 01)
>> :00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
>> :00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V
>> virtual VGA
>> 042d:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx Virtual Function] (rev 80)
>> 1d94:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx Virtual Function] (rev 80)
>> 
>> int
>>  Name   IdxState  MTU (L3/IP4/IP6/MPLS)
>> Counter  Count
>> FiftyGigabitEthernet2 1  up  2000/0/0/0 rx
>> packets 1
>>rx
>> bytes  42
>>tx
>> packets 1
>>tx
>> bytes  42
>>drops
>> 1
>> FiftyGigabitEthernet3 2  up  9000/0/0/0 rx
>> packets 9
>> 

Re: [vpp-dev] Problem with native avf

[Christian Hopps]

> On Dec 7, 2020, at 5:16 PM, Damjan Marion  wrote:
> 
>> On 07.12.2020., at 22:55, Christian Hopps  wrote:
> 
> I just bumped it to 2.13.10 and added REMAKE_INITRD=“yes” into debian/dkms.in 
> so that should be fixed:
> 
> $ lsinitramfs /boot/initrd.img | grep i40e
> usr/lib/modules/5.8.0-31-generic/updates/dkms/i40e.ko

Updated, and works.

> 
>> 
>>> also, I asked for your crete interface config….
>> 
>> create interface avf :65:0a.0

I switched this to also include "rx-queue-size 2048" to see if it helped with 
my other problem, but it didn't.

>> 
>> As mentioned above, when I rmmod/modprobe the new driver I don't hit the lut 
>> error anymore.
>> 
>> Now I need to figure out why I see such bad performance (tons of rx 
>> discards) using these interfaces (when using 2 of them), but not when using 
>> 1 avf VF and the other interface is a 10G i520 nic (dpdk driver).
>> 
>> Bad Side:
>> 
>> $ docker-compose exec p1 vppctl show hard
>>  NameIdx   Link  Hardware
>> avf-0/65/2/0   2 up   avf-0/65/2/0
>>  Link speed: 10 Gbps
>>  Ethernet address 02:41:0d:0d:0d:0b
>>  flags: initialized admin-up vaddr-dma link-up rx-interrupts
>>  offload features: l2 adv-link-speed vlan rx-polling rss-pf
>>  num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
>>  speed
>>  stats:
>>rx bytes 1520728906
>>rx unicast   184910147
>>rx discards  129083610
>>tx bytes 4226915088
>>tx unicast   184349288
>> avf-0/65/a/0   1 up   avf-0/65/a/0
>>  Link speed: 10 Gbps
>>  Ethernet address 02:41:0b:0b:0b:0b
>>  flags: initialized admin-up vaddr-dma link-up rx-interrupts
>>  offload features: l2 adv-link-speed vlan rx-polling rss-pf
>>  num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
>>  speed
>>  stats:
>>rx bytes 3781507000
>>rx unicast   93533516
>>rx broadcast 3
>>rx discards  73223358
>>tx bytes 2212324998
>>tx unicast   13714424
>> ipsec0 3 up   ipsec0
>>  Link speed: unknown
>>  IPSec
>> local0 0down  local0
>>  Link speed: unknown
>>  local
>> loop0  4 up   loop0
>>  Link speed: unknown
>>  Ethernet address de:ad:00:00:00:00
>> 
>> Good Side:
>> 
>> $ docker-compose exec p2 vppctl show hard
>>  NameIdx   Link  Hardware
>> TenGigabitEthernet17/0/0   1 up   TenGigabitEthernet17/0/0
>>  Link speed: 10 Gbps
>>  Ethernet address f8:f2:1e:3c:15:ec
>>  Intel 82599
>>carrier up full duplex mtu 9206
>>flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum 
>> rx-ip4-cksum
>>Devargs:
>>rx: queues 1 (max 128), desc 1024 (min 32 max 4096 align 8)
>>tx: queues 6 (max 64), desc 1024 (min 32 max 4096 align 8)
>>pci: device 8086:10fb subsystem 8086:0003 address :17:00.00 numa 0
>>max rx packet len: 15872
>>promiscuous: unicast off all-multicast on
>>vlan offload: strip off filter off qinq off
>>rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro
>>   macsec-strip vlan-filter vlan-extend jumbo-frame 
>> scatter
>>   security keep-crc rss-hash
>>rx offload active: ipv4-cksum jumbo-frame scatter
>>tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
>>   tcp-tso macsec-insert multi-segs security
>>tx offload active: udp-cksum tcp-cksum multi-segs
>>rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp
>>   ipv6-udp ipv6-ex ipv6
>>rss active:none
>>tx burst function: ixgbe_xmit_pkts
>>rx burst function: ixgbe_recv_scattered_pkts_vec
>> 
>>tx frames ok   183386223
>>tx bytes ok 277646741622
>>rx frames ok   182834391
>>rx bytes ok 276811244995
>>extended stats:
>>  rx_good_packets  182834488
>>  tx_good_packets  183386223
>>  rx_good_bytes 276811391853
>>  tx_good_byt

Re: [vpp-dev] Problem with native avf

[Christian Hopps]

> On Dec 7, 2020, at 3:02 PM, Damjan Marion  wrote:
> 
> 
> 
>> On 07.12.2020., at 20:41, Christian Hopps > <mailto:cho...@chopps.org>> wrote:
>> 
>> 
>> 
>>> On Dec 7, 2020, at 1:44 PM, Damjan Marion >> <mailto:damjan.mar...@gmail.com>> wrote:
>>> 
>>>> 
>>>> On 07.12.2020., at 17:02, Christian Hopps >>> <mailto:cho...@chopps.org>> wrote:
>>>> 
>>> 
>>> please send me output of: extras/scripts/lsnet script and exact “create 
>>> interface avf” commands you use….
>> 
>> PCI Address  MAC address   Device NameDriver StateSpeed  
>> Port Type
>>  = == ==  == 
>> 
>> :65:00.0 40:a6:b7:4b:62:08 enp101s0f0 i40e   down 1Mb/s  
>> Direct Attach Copper
>> :65:00.1 40:a6:b7:4b:62:09 enp101s0f1 i40e   down 1Mb/s  
>> Direct Attach Copper
>> :65:00.2 40:a6:b7:4b:62:0a enp101s0f2 i40e   down 1Mb/s  
>> Direct Attach Copper
>> :65:00.3 40:a6:b7:4b:62:0b enp101s0f3 i40e   down 1Mb/s  
>> Direct Attach Copper
>> :b3:00.0 00:e0:8d:7e:1f:36 enp179s0f0 ixgbe  down Unknown!   
>> Direct Attach Copper
>> :b3:00.1 00:e0:8d:7e:1f:37 enp179s0f1 ixgbe  down Unknown!   
>> Direct Attach Copper
>> :01:00.0 a0:42:3f:3c:f8:ee enp1s0f0   ixgbe  up   1Mb/s  
>> Twisted Pair
>> :01:00.1 a0:42:3f:3c:f8:ef enp1s0f1   ixgbe  down Unknown!   
>> Twisted Pair
>> :17:00.0 f8:f2:1e:3c:15:ec enp23s0f0  ixgbe  down Unknown!   
>> Direct Attach Copper
>> :17:00.1 f8:f2:1e:3c:15:ed enp23s0f1  ixgbe  down Unknown!   
>> Direct Attach Copper
>> :01:10.0 52:bf:27:59:df:50 eth0   ixgbevfdown Unknown!   
>> Other
>> :01:10.2 ee:24:0b:0c:93:3f eth1   ixgbevfdown Unknown!   
>> Other
>> :01:10.4 9e:8e:ce:da:38:f5 eth2   ixgbevfdown Unknown!   
>> Other
>> :01:10.6 2a:f4:a2:ea:4c:5d eth3   ixgbevfdown Unknown!   
>> Other
>> 
> 
> I dont see your VFs on the list?Do you have them created?
> Do you see them with “lspci | grep Ether”?

[ The VFs are not in the list b/c they are created as part of the automation 
that also launches vpp, I ran this before running that b/c once vpp is up the 
VFs also don't show up in that list b/c they have been rebound. :) ]

First, things work with 2.12.; however, 2.12 does not load on reboot, I must 
rmmod and modprobe after rebooting to get the 2.12 driver. I do have another 
problem though (mentioned at end)...

> also, I asked for your crete interface config….

create interface avf :65:0a.0

As mentioned above, when I rmmod/modprobe the new driver I don't hit the lut 
error anymore.

Now I need to figure out why I see such bad performance (tons of rx discards) 
using these interfaces (when using 2 of them), but not when using 1 avf VF and 
the other interface is a 10G i520 nic (dpdk driver).

Bad Side:

$ docker-compose exec p1 vppctl show hard
  NameIdx   Link  Hardware
avf-0/65/2/0   2 up   avf-0/65/2/0
  Link speed: 10 Gbps
  Ethernet address 02:41:0d:0d:0d:0b
  flags: initialized admin-up vaddr-dma link-up rx-interrupts
  offload features: l2 adv-link-speed vlan rx-polling rss-pf
  num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
  speed
  stats:
rx bytes 1520728906
rx unicast   184910147
rx discards  129083610
tx bytes 4226915088
tx unicast   184349288
avf-0/65/a/0   1 up   avf-0/65/a/0
  Link speed: 10 Gbps
  Ethernet address 02:41:0b:0b:0b:0b
  flags: initialized admin-up vaddr-dma link-up rx-interrupts
  offload features: l2 adv-link-speed vlan rx-polling rss-pf
  num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
  speed
  stats:
rx bytes 3781507000
rx unicast   93533516
rx broadcast 3
rx discards  73223358
tx bytes 2212324998
tx unicast   13714424
ipsec0 3 up   ipsec0
  Link speed: unknown
  IPSec
local0 0down  local0
  Link speed: unknown
  local
loop0  4 up   loop0
  Link speed: unknown
  Ethernet address de:ad:00:00:00:00

Good Side:

$ docker-compose exec p2 vppctl show hard
  NameIdx   Link  Hardware
TenGigabitEthernet17/0/0   1 up   TenGigabitEthernet17/0/0
  Link speed

Re: [vpp-dev] Problem with native avf

[Christian Hopps]

> On Dec 7, 2020, at 1:44 PM, Damjan Marion  wrote:
> 
>> 
>> On 07.12.2020., at 17:02, Christian Hopps  wrote:
>> 
> 
> please send me output of: extras/scripts/lsnet script and exact “create 
> interface avf” commands you use….

PCI Address  MAC address   Device NameDriver StateSpeed  
Port Type
 = == ==  == 

:65:00.0 40:a6:b7:4b:62:08 enp101s0f0 i40e   down 1Mb/s  
Direct Attach Copper
:65:00.1 40:a6:b7:4b:62:09 enp101s0f1 i40e   down 1Mb/s  
Direct Attach Copper
:65:00.2 40:a6:b7:4b:62:0a enp101s0f2 i40e   down 1Mb/s  
Direct Attach Copper
:65:00.3 40:a6:b7:4b:62:0b enp101s0f3 i40e   down 1Mb/s  
Direct Attach Copper
:b3:00.0 00:e0:8d:7e:1f:36 enp179s0f0 ixgbe  down Unknown!   
Direct Attach Copper
:b3:00.1 00:e0:8d:7e:1f:37 enp179s0f1 ixgbe  down Unknown!   
Direct Attach Copper
:01:00.0 a0:42:3f:3c:f8:ee enp1s0f0   ixgbe  up   1Mb/s  
Twisted Pair
:01:00.1 a0:42:3f:3c:f8:ef enp1s0f1   ixgbe  down Unknown!   
Twisted Pair
:17:00.0 f8:f2:1e:3c:15:ec enp23s0f0  ixgbe  down Unknown!   
Direct Attach Copper
:17:00.1 f8:f2:1e:3c:15:ed enp23s0f1  ixgbe  down Unknown!   
Direct Attach Copper
:01:10.0 52:bf:27:59:df:50 eth0   ixgbevfdown Unknown!   
Other
:01:10.2 ee:24:0b:0c:93:3f eth1   ixgbevfdown Unknown!   
Other
:01:10.4 9e:8e:ce:da:38:f5 eth2   ixgbevfdown Unknown!   
Other
:01:10.6 2a:f4:a2:ea:4c:5d eth3   ixgbevfdown Unknown!   
Other


> 
>> 
>>> What kernel and driver version do you use?
>> 
>> Host Config:
>> 
>> $ cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=20.10
>> DISTRIB_CODENAME=groovy
>> DISTRIB_DESCRIPTION="Ubuntu 20.10"
>> $ uname -a
>> Linux labnh 5.8.0-31-generic #33-Ubuntu SMP Mon Nov 23 18:44:54 UTC 2020 
>> x86_64 x86_64 x86_64 GNU/Linux
>> 
>> Docker Config (compiled and run inside):
>> 
>> root@p1:/# cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=18.04
>> DISTRIB_CODENAME=bionic
>> DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
>> 
>>> 
>>> Have you tried latest PF driver from intel?
>> 
>> No; however, I am running such a new ubuntu (5.8 kernel) so I was hoping 
>> that was sufficient.
> 
> Agree, still, please, do me a favour and try with latest, so I know i’m 
> looking at the same thing.

I did the build and installed the deb, I also updated the firmware on the NIC; 
however,

$ dpkg -l | grep i40
ii  i40e-dkms   2.12.6  
all  Intel i40e adapter driver

$ sudo ethtool -i enp101s0f0
driver: i40e
version: 2.8.20-k
firmware-version: 8.15 0x80009621 1.2829.0
expansion-rom-version:
bus-info: :65:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

It doesn't seem to have used it. Is there something else I need to do to have 
it use the dkms driver?

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18273): https://lists.fd.io/g/vpp-dev/message/18273
Mute This Topic: https://lists.fd.io/mt/78779141/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Problem with native avf

[Christian Hopps]

> On Dec 7, 2020, at 9:57 AM, Damjan Marion  wrote:
> 
> 
> Anything in dmesg output on PF side?

VF setup script running

> [  168.765633] i40e :65:00.2: FW LLDP is enabled
> [  168.765635] i40e :65:00.2: Allocating 1 VFs.
> [  168.870431] pci :65:0a.0: [8086:154c] type 00 class 0x02
> [  168.870441] pci :65:0a.0: enabling Extended Tags
> [  168.870609] pci :65:0a.0: Adding to iommu group 101
> [  168.881032] iavf: Intel(R) Ethernet Adaptive Virtual Function Network 
> Driver - version 3.2.3-k
> [  168.881033] Copyright (c) 2013 - 2018 Intel Corporation.
> [  168.881136] iavf :65:0a.0: enabling device ( -> 0002)
> [  168.925644] iavf :65:0a.0: Device is still in reset (-16), retrying
> [  168.971405] i40e :65:00.2: Setting MAC 02:41:0b:0b:0b:0b on VF 0
> [  169.061729] i40e :65:00.2: Bring down and up the VF interface to make 
> this change effective.
> [  169.154400] i40e :65:00.2: VF 0 is now trusted
> [  169.332527] i40e :65:00.0: FW LLDP is enabled
> [  169.332528] i40e :65:00.0: Allocating 1 VFs.
> [  169.438431] pci :65:02.0: [8086:154c] type 00 class 0x02
> [  169.438441] pci :65:02.0: enabling Extended Tags
> [  169.438588] pci :65:02.0: Adding to iommu group 102
> [  169.438668] iavf :65:02.0: enabling device ( -> 0002)
> [  169.483272] iavf :65:02.0: Device is still in reset (-16), retrying
> [  169.539244] i40e :65:00.0: Setting MAC 02:41:0d:0d:0d:0b on VF 0
> [  169.629728] i40e :65:00.0: Bring down and up the VF interface to make 
> this change effective.
> [  169.722387] i40e :65:00.0: VF 0 is now trusted
> [  169.902916] i40e :65:00.3: FW LLDP is enabled
> [  169.902917] i40e :65:00.3: Allocating 1 VFs.
> [  170.010432] pci :65:0e.0: [8086:154c] type 00 class 0x02
> [  170.010441] pci :65:0e.0: enabling Extended Tags
> [  170.010573] pci :65:0e.0: Adding to iommu group 103
> [  170.010641] iavf :65:0e.0: enabling device ( -> 0002)
> [  170.055217] iavf :65:0e.0: Device is still in reset (-16), retrying
> [  170.110941] i40e :65:00.3: Setting MAC 02:42:0c:0c:0c:0c on VF 0
> [  170.201737] i40e :65:00.3: Bring down and up the VF interface to make 
> this change effective.
> [  170.294398] i40e :65:00.3: VF 0 is now trusted

VPP starts running

> [  178.922603] ixgbe :17:00.0: complete
> [  178.928006] i40e :65:00.0 enp101s0f0: NIC Link is Down
> [  179.186499] vfio-pci :17:00.0: enabling device (0142 -> 0143)
> [  180.105670] i40e :65:00.0 enp101s0f0: NIC Link is Up, 10 Gbps Full 
> Duplex, Flow Control: None
> [  180.247071] vfio-pci :65:0a.0: enabling device ( -> 0002)
> [  180.841280] i40e :65:00.2: VF 0 failed opcode 24, retval: -5
> [  181.721005] vfio-pci :65:0e.0: enabling device ( -> 0002)
> [  182.304338] i40e :65:00.3: VF 0 failed opcode 24, retval: -5

> What kernel and driver version do you use?

Host Config:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
DISTRIB_DESCRIPTION="Ubuntu 20.10"
$ uname -a
Linux labnh 5.8.0-31-generic #33-Ubuntu SMP Mon Nov 23 18:44:54 UTC 2020 x86_64 
x86_64 x86_64 GNU/Linux

Docker Config (compiled and run inside):

root@p1:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

> 
> Have you tried latest PF driver from intel?

No; however, I am running such a new ubuntu (5.8 kernel) so I was hoping that 
was sufficient.

Thanks,
Chris.

> 
> —
> Damjan
> 
>> On 07.12.2020., at 15:42, Christian Hopps  wrote:
>> 
>> I'm hitting a problem with the native AVF driver. The issue does not seem to 
>> exist when using the DPDK driver. This is on stable/2009, with the DPDK bump 
>> to 2008 reverted (b/c 2008 doesn't work with mellanox anymore apparently).
>> 
>> I have an x710 card configured with a 4x10G breakout cable. The code that is 
>> failing is
>> 
>> if ((ad->feature_bitmap & VIRTCHNL_VF_OFFLOAD_RSS_PF) &&
>> (error = avf_op_config_rss_lut (vm, ad)))
>>   return error;
>> 
>> Here's the debug log:
>> 
>> 2020/12/07 14:19:25:639 debug  avf:65:0a.0: 
>> request_queues: num_queue_pairs 6
>> 2020/12/07 14:19:25:779 debug  avf:65:0a.0: version: 
>> major 1 minor 1
>> 2020/12/07 14:19:25:779 debug  avf:65:0a.0: 
>> get_vf_reqources: bitmap 0xb00a1
>> 2020/12/07 14:19:25:789 debug  avf:65:0a.0: 
>> get_vf_reqources: num_vsis 1 num_queue_pairs 6 max_vectors 5 max_mtu 0 
>> vf_offload_flags 0xb000 rss_key_size 5

[vpp-dev] Problem with native avf

[Christian Hopps]

I'm hitting a problem with the native AVF driver. The issue does not seem to 
exist when using the DPDK driver. This is on stable/2009, with the DPDK bump to 
2008 reverted (b/c 2008 doesn't work with mellanox anymore apparently).

I have an x710 card configured with a 4x10G breakout cable. The code that is 
failing is

  if ((ad->feature_bitmap & VIRTCHNL_VF_OFFLOAD_RSS_PF) &&
  (error = avf_op_config_rss_lut (vm, ad)))
return error;

Here's the debug log:

2020/12/07 14:19:25:639 debug  avf:65:0a.0: request_queues: 
num_queue_pairs 6
2020/12/07 14:19:25:779 debug  avf:65:0a.0: version: major 
1 minor 1
2020/12/07 14:19:25:779 debug  avf:65:0a.0: 
get_vf_reqources: bitmap 0xb00a1
2020/12/07 14:19:25:789 debug  avf:65:0a.0: 
get_vf_reqources: num_vsis 1 num_queue_pairs 6 max_vectors 5 max_mtu 0 
vf_offload_flags 0xb000 rss_key_size 52 rss_lut_size 64
2020/12/07 14:19:25:789 debug  avf:65:0a.0: 
get_vf_reqources_vsi[0]: vsi_id 18 num_queue_pairs 6 vsi_type 6 qset_handle 6 
default_mac_addr 02:41:0b:0b:0b:0b
2020/12/07 14:19:25:789 debug  avf:65:0a.0: 
disable_vlan_stripping
2020/12/07 14:19:25:800 debug  avf:65:0a.0: config_rss_lut: 
vsi_id 18 rss_lut_size 64 lut 
0x
2020/12/07 14:19:25:906 erravf1313:13:13.0: error: 
avf_send_to_pf: error [v_opcode = 24, v_retval -5]

If I comment out the 2 calls to VIRTCHNL_CF_OFFLOAD_RSS_PF, the device 
initializes.

Is there something about the breakout cable configuration that needs special 
attention from this code?

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18264): https://lists.fd.io/g/vpp-dev/message/18264
Mute This Topic: https://lists.fd.io/mt/78779141/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Christian Hopps]

We actually had to revert the DPDK 20.08 use to continue to have this work in 
20.09.

Isn't it a bit unexpected to have a DPDK supported NIC not work though? I'd 
think we'd want this fixed, is there any idea why it's broken?

Thanks,
Chris.

> On Nov 20, 2020, at 2:47 PM, Mohammed Hawari  wrote:
> 
> Hi Ashish,
> 
> The DPDK plugin for mlx5 NICs is not supported by the current code on master. 
> We strongly encourage you to try the rdma plugin, which offers a native VPP 
> driver for mlx5 NICs without relying on DPDK. The use of this driver is 
> documented in  src/plugins/rdma/rdma_doc.md.
> 
> Best regards
> 
> Mohammed
> 
>> On 19 Nov 2020, at 13:18, ashish.sax...@hsc.com 
>>  wrote:
>> 
>> Hi vpp devs,
>> 
>> I have been trying to run vpp on latest master branch with mlx 5 enables on 
>> centos 8.2 machine .
>> I have downloaded and installed the mellanox driver from 
>> url:https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed 
>>  and 
>> installed the following dependencies:
>> #yum install perl-Term-ANSIColor
>> #yum install tcl tk python36
>> #./mlnxofedinstall --upstream-libs --dpdk --without-fw-update
>> #/etc/init.d/openibd restart
>> 
>> I cloned the latest master branch (v21.01-rc0) (git clone -b master 
>> https://gerrit.fd.io/r/vpp )
>> I enabled mlx5 from file:build/external/packages/dpdk.mk
>> DPDK_MLX5_PMD?= y
>> DPDK_MLX5_COMMON_PMD ?= y
>> 
>> Then I followed these steps to compile vpp :
>> 
>> #make wipe-release
>> #make install-dep
>> #make install-ext-deps
>> #make build-release
>> # make pkg-rpm
>> #rpm -ivh *.rpm
>> 
>> The compilation went fine , but got the following issue while installing the 
>> rpm files:
>> #rpm -ivh *.rpm
>> error: Failed dependencies:
>>  python(abi) = 2.7 is needed by 
>> vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
>>  python-setuptools is needed by 
>> vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
>> 
>> I changed line 143 on extras/rpm/vpp.spec: From "python-setuptools" to " 
>> python3-setuptools".
>> Then , I  proceeded with re-compile, and was able to install vpp 
>> successfully after that.
>> 
>> But when starting vpp, it kept crashing with error below:
>>
>> Nov  6 15:01:44 sim01 systemd[1]: Starting Vector Packet Processing 
>> Process...
>> Nov  6 15:01:44 sim01 systemd[1]: Started Vector Packet Processing Process.
>> Nov  6 15:01:44 sim01 vpp[1557233]: /usr/bin/vpp: symbol lookup error: 
>> /usr/lib/vpp_plugins/dpdk_plugin.so: undefined symbol: ibv_fork_init
>> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Main process exited, 
>> code=exited, status=127/n/a
>> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Failed with result 
>> 'exit-code'.
>> Nov  6 15:01:48 sim01 systemd[1]: Stopped Vector Packet Processing Process.
>> 
>> How can I compile vpp with mlx so that it didn't crash .
>> 
>> Thanks and Regards,
>> Ashish
>> 
>> 
>> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18127): https://lists.fd.io/g/vpp-dev/message/18127
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: Handoff design issues [Re: RES: RES: [vpp-dev] Increasing NAT worker handoff frame queue size NAT_FQ_NELTS to avoid congestion drops?]

[Christian Hopps]

Just checked out the patch; you are compressing the frames on the receiving 
thread side. I didn't realize (i.e., look) that the code was copying the buffer 
indexes into a new frame anyway, as it is, I think this is a good fix!

Thanks,
Chris.

> On Nov 16, 2020, at 4:20 AM, Klement Sekera -X (ksekera - PANTHEON TECH SRO 
> at Cisco)  wrote:
> 
> That’s exactly what my patch improves. Coalescing small groups of packets 
> waiting in the handoff queue into a full(er) frame allows the downstream node 
> to do more “V” and achieve better performance. And that’s also what I’ve seen 
> when testing the patch.
> 
> Thanks,
> Klement
> 
> ps. in case you missed the link: https://gerrit.fd.io/r/c/vpp/+/28980
> 
>> On 13 Nov 2020, at 22:47, Christian Hopps  wrote:
>> 
>> FWIW, I too have hit this issue. Basically VPP is designed to process a 
>> packet from rx to tx in the same thread. When downstream nodes run slower, 
>> the upstream rx node doesn't run, so the vector size in each frame naturally 
>> increases, and then the downstream nodes can benefit from "V" (i.e., 
>> processing multiple packets in one go).
>> 
>> This back-pressure from downstream does not occur when you hand-off from a 
>> fast thread to a slower thread, so you end up with many single packet frames 
>> and fill your hand-off queue.
>> 
>> The quick fix one tries then is to increase the queue size; however, this is 
>> not a great solution b/c you are still not taking advantage of the "V" in 
>> VPP. To really fit this back into the original design one needs to somehow 
>> still be creating larger vectors in the hand-off frames.
>> 
>> TBH I think the right solution here is to not hand-off frames, and instead 
>> switch to packet queues and then on the handed-off side the frames would get 
>> constructed from packet queues (basically creating another polling input 
>> node but on the new thread).
>> 
>> Thanks,
>> Chris.
>> 
>>> On Nov 13, 2020, at 12:21 PM, Marcos - Mgiga  wrote:
>>> 
>>> Understood. And what path did you take in order to analyse and monitor 
>>> vector rates ? Is there some specific command or log ?
>>> 
>>> Thanks
>>> 
>>> Marcos
>>> 
>>> -Mensagem original-
>>> De: vpp-dev@lists.fd.io  Em nome de ksekera via []
>>> Enviada em: sexta-feira, 13 de novembro de 2020 14:02
>>> Para: Marcos - Mgiga 
>>> Cc: Elias Rudberg ; vpp-dev@lists.fd.io
>>> Assunto: Re: RES: [vpp-dev] Increasing NAT worker handoff frame queue size 
>>> NAT_FQ_NELTS to avoid congestion drops?
>>> 
>>> Not completely idle, more like medium load. Vector rates at which I saw 
>>> congestion drops were roughly 40 for thread doing no work (just handoffs - 
>>> I hardcoded it this way for test purpose), and roughly 100 for thread 
>>> picking the packets doing NAT.
>>> 
>>> What got me into infra investigation was the fact that once I was hitting 
>>> vector rates around 255, I did see packet drops, but no congestion drops.
>>> 
>>> HTH,
>>> Klement
>>> 
>>>> On 13 Nov 2020, at 17:51, Marcos - Mgiga  wrote:
>>>> 
>>>> So you mean that this situation ( congestion drops) is most likely to 
>>>> occur when the system in general is idle than when it is processing a 
>>>> large amount of traffic?
>>>> 
>>>> Best Regards
>>>> 
>>>> Marcos
>>>> 
>>>> -Mensagem original-
>>>> De: vpp-dev@lists.fd.io  Em nome de Klement
>>>> Sekera via lists.fd.io Enviada em: sexta-feira, 13 de novembro de 2020
>>>> 12:15
>>>> Para: Elias Rudberg 
>>>> Cc: vpp-dev@lists.fd.io
>>>> Assunto: Re: [vpp-dev] Increasing NAT worker handoff frame queue size 
>>>> NAT_FQ_NELTS to avoid congestion drops?
>>>> 
>>>> Hi Elias,
>>>> 
>>>> I’ve already debugged this and came to the conclusion that it’s the infra 
>>>> which is the weak link. I was seeing congestion drops at mild load, but 
>>>> not at full load. Issue is that with handoff, there is uneven workload. 
>>>> For simplicity’s sake, just consider thread 1 handing off all the traffic 
>>>> to thread 2. What happens is that for thread 1, the job is much easier, it 
>>>> just does some ip4 parsing and then hands packet to thread 2, which 
>>>> actually does the heavy lifting of hash inserts/lookups/translation etc. 
>>>> 6

Re: [vpp-dev] Frequently updated gauge (scalar)?

[Christian Hopps]

Yeah, I ended up doing this. Would be nice to have access to a "double" vs 
"u64" type, but using vlib_set_counter works well enough for me.

Thanks,
Chris.

> On Nov 13, 2020, at 5:16 PM, otr...@employees.org wrote:
> 
> Hi Christian,
> 
> Aboslutely. The periodic gauge callback is purely there for sources that 
> don't provide the counters themselves.
> E.g. used for polling mempry heaps, free buffers and so on.
> 
> You can just use a normal counter (counter.c) for your high frequency gauge.
> 
> Cheers,
> Ole
> 
>> On 13 Nov 2020, at 20:20, Christian Hopps  wrote:
>> 
>> I have need to track a frequently updated value (pps rate during congestion 
>> control), but I need this to be close to very frequently updated (basically 
>> whenever I change the value which is based on RTT). It seems the current 
>> gauge code you supply a callback which updates the counter (at a default of 
>> every 10 seconds).
>> 
>> Is there something about the scalar stat that's different from the counter 
>> stat that requires this infrequent updating? Is it possible to "register" a 
>> scalar state that I can update very frequently instead directly (instead of 
>> supplying a callback)?
>> 
>> Thanks,
>> Chris.
>> 
>> 
>> 
> 



signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18029): https://lists.fd.io/g/vpp-dev/message/18029
Mute This Topic: https://lists.fd.io/mt/78237093/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Handoff design issues [Re: RES: RES: [vpp-dev] Increasing NAT worker handoff frame queue size NAT_FQ_NELTS to avoid congestion drops?]

[Christian Hopps]

FWIW, I too have hit this issue. Basically VPP is designed to process a packet 
from rx to tx in the same thread. When downstream nodes run slower, the 
upstream rx node doesn't run, so the vector size in each frame naturally 
increases, and then the downstream nodes can benefit from "V" (i.e., processing 
multiple packets in one go).

This back-pressure from downstream does not occur when you hand-off from a fast 
thread to a slower thread, so you end up with many single packet frames and 
fill your hand-off queue.

The quick fix one tries then is to increase the queue size; however, this is 
not a great solution b/c you are still not taking advantage of the "V" in VPP. 
To really fit this back into the original design one needs to somehow still be 
creating larger vectors in the hand-off frames.

TBH I think the right solution here is to not hand-off frames, and instead 
switch to packet queues and then on the handed-off side the frames would get 
constructed from packet queues (basically creating another polling input node 
but on the new thread).

Thanks,
Chris.

> On Nov 13, 2020, at 12:21 PM, Marcos - Mgiga  wrote:
> 
> Understood. And what path did you take in order to analyse and monitor vector 
> rates ? Is there some specific command or log ?
> 
> Thanks
> 
> Marcos
> 
> -Mensagem original-
> De: vpp-dev@lists.fd.io  Em nome de ksekera via []
> Enviada em: sexta-feira, 13 de novembro de 2020 14:02
> Para: Marcos - Mgiga 
> Cc: Elias Rudberg ; vpp-dev@lists.fd.io
> Assunto: Re: RES: [vpp-dev] Increasing NAT worker handoff frame queue size 
> NAT_FQ_NELTS to avoid congestion drops?
> 
> Not completely idle, more like medium load. Vector rates at which I saw 
> congestion drops were roughly 40 for thread doing no work (just handoffs - I 
> hardcoded it this way for test purpose), and roughly 100 for thread picking 
> the packets doing NAT.
> 
> What got me into infra investigation was the fact that once I was hitting 
> vector rates around 255, I did see packet drops, but no congestion drops.
> 
> HTH,
> Klement
> 
>> On 13 Nov 2020, at 17:51, Marcos - Mgiga  wrote:
>> 
>> So you mean that this situation ( congestion drops) is most likely to occur 
>> when the system in general is idle than when it is processing a large amount 
>> of traffic?
>> 
>> Best Regards
>> 
>> Marcos
>> 
>> -Mensagem original-
>> De: vpp-dev@lists.fd.io  Em nome de Klement
>> Sekera via lists.fd.io Enviada em: sexta-feira, 13 de novembro de 2020
>> 12:15
>> Para: Elias Rudberg 
>> Cc: vpp-dev@lists.fd.io
>> Assunto: Re: [vpp-dev] Increasing NAT worker handoff frame queue size 
>> NAT_FQ_NELTS to avoid congestion drops?
>> 
>> Hi Elias,
>> 
>> I’ve already debugged this and came to the conclusion that it’s the infra 
>> which is the weak link. I was seeing congestion drops at mild load, but not 
>> at full load. Issue is that with handoff, there is uneven workload. For 
>> simplicity’s sake, just consider thread 1 handing off all the traffic to 
>> thread 2. What happens is that for thread 1, the job is much easier, it just 
>> does some ip4 parsing and then hands packet to thread 2, which actually does 
>> the heavy lifting of hash inserts/lookups/translation etc. 64 element queue 
>> can hold 64 frames, one extreme is 64 1-packet frames, totalling 64 packets, 
>> other extreme is 64 255-packet frames, totalling ~16k packets. What happens 
>> is this: thread 1 is mostly idle and just picking a few packets from NIC and 
>> every one of these small frames creates an entry in the handoff queue. Now 
>> thread 2 picks one element from the handoff queue and deals with it before 
>> picking another one. If the queue has only 3-packet or 10-packet elements, 
>> then thread 2 can never really get into what VPP excels in - bulk processing.
>> 
>> Q: Why doesn’t it pick as many packets as possible from the handoff queue?
>> A: It’s not implemented.
>> 
>> I already wrote a patch for it, which made all congestion drops which I saw 
>> (in above synthetic test case) disappear. Mentioned patch 
>> https://gerrit.fd.io/r/c/vpp/+/28980 is sitting in gerrit.
>> 
>> Would you like to give it a try and see if it helps your issue? We
>> shouldn’t need big queues under mild loads anyway …
>> 
>> Regards,
>> Klement
>> 
>>> On 13 Nov 2020, at 16:03, Elias Rudberg  wrote:
>>> 
>>> Hello VPP experts,
>>> 
>>> We are using VPP for NAT44 and we get some "congestion drops", in a
>>> situation where we think VPP is far from overloaded in general. Then
>>> we started to investigate if it would help to use a larger handoff
>>> frame queue size. In theory at least, allowing a longer queue could
>>> help avoiding drops in case of short spikes of traffic, or if it
>>> happens that some worker thread is temporarily busy for whatever
>>> reason.
>>> 
>>> The NAT worker handoff frame queue size is hard-coded in the
>>> NAT_FQ_NELTS macro in src/plugins/nat/nat.h where the current value
>>> is 64. The idea is that putting a larger value there 

[vpp-dev] Frequently updated gauge (scalar)?

[Christian Hopps]

I have need to track a frequently updated value (pps rate during congestion 
control), but I need this to be close to very frequently updated (basically 
whenever I change the value which is based on RTT). It seems the current gauge 
code you supply a callback which updates the counter (at a default of every 10 
seconds).

Is there something about the scalar stat that's different from the counter stat 
that requires this infrequent updating? Is it possible to "register" a scalar 
state that I can update very frequently instead directly (instead of supplying 
a callback)?

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18023): https://lists.fd.io/g/vpp-dev/message/18023
Mute This Topic: https://lists.fd.io/mt/78237093/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Latest master, dpdk and mlx5 failing

[Christian Hopps]

I just noticed this error I'm getting

net_mlx4: cannot load glue library: 
/home/chopps/w/vpp/build-root/install-vpp-native/external/lib/dpdk/pmds-20.0.3-glue/librte_pmd_mlx4_glue.so.18.02.0:
 cannot open shared object file: No such file or directory
net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core 
libraries (libibverbs, libmlx4)
common_mlx5: Cannot load glue library: 
/home/chopps/w/vpp/build-root/install-vpp-native/external/lib/dpdk/pmds-20.0.3-glue/librte_pmd_mlx5_glue.so.20.02.0:
 cannot open shared object file: No such file or directory
common_mlx5: Cannot initialize MLX5 common due to missing run-time dependency 
on rdma-core libraries (libibverbs, libmlx5)

And:

(default) [17:50:17 ...]$ find ../build-root/*-vpp-native/external -name 
'*glue*'
../build-root/build-vpp-native/external/src-dpdk/drivers/common/mlx5/linux/mlx5_glue.c
../build-root/build-vpp-native/external/src-dpdk/drivers/common/mlx5/linux/mlx5_glue.h
../build-root/build-vpp-native/external/src-dpdk/drivers/net/mlx4/mlx4_glue.c
../build-root/build-vpp-native/external/src-dpdk/drivers/net/mlx4/mlx4_glue.h
../build-root/build-vpp-native/external/build-dpdk/drivers/common/mlx5/linux/librte_pmd_mlx5_glue.so
../build-root/build-vpp-native/external/build-dpdk/drivers/common/mlx5/linux/b7c1ada@@rte_pmd_mlx5_glue@sha
../build-root/build-vpp-native/external/build-dpdk/drivers/common/mlx5/linux/b7c1ada@@rte_pmd_mlx5_glue@sha/mlx5_glue.c.o
../build-root/build-vpp-native/external/build-dpdk/drivers/common/mlx5/linux/librte_pmd_mlx5_glue.so.20.02.0
../build-root/build-vpp-native/external/build-dpdk/drivers/net/mlx4/librte_pmd_mlx4_glue.so
../build-root/build-vpp-native/external/build-dpdk/drivers/net/mlx4/8672f8e@@rte_pmd_mlx4_glue@sha
../build-root/build-vpp-native/external/build-dpdk/drivers/net/mlx4/8672f8e@@rte_pmd_mlx4_glue@sha/mlx4_glue.c.o
../build-root/build-vpp-native/external/build-dpdk/drivers/net/mlx4/librte_pmd_mlx4_glue.so.18.02.0
../build-root/install-vpp-native/external/lib/dpdk/pmds-20.0.3-glue
(default) [17:50:24 ...]

So it looks like the glue is being built but not installed?

Thanks,
Chris.

> On Oct 2, 2020, at 12:21 PM, Matthew Smith  wrote:
> 
> Hi Mohammed,
> 
> I think it will only affect builds where DPDK_MLX5_PMD=y is set, but I cannot 
> say for sure. The scripts/configurations I build with always set that flag, 
> so I have not tried to generate a build without it set recently.
> 
> -Matt
> 
> 
> On Fri, Oct 2, 2020 at 12:57 AM Mohammed HAWARI  <mailto:momohaw...@gmail.com>> wrote:
> Hello Chris, Matthew,
> 
> Thanks for raising that issue. Just to be clear and better understand, does 
> the problem occur with the default config, i.e., without trying to compile 
> any MLX driver in DPDK? Or does it only appear when setting DPDK_MLX5_PMD=y ?
> Thanks
> Best regards
> Mohammed
>> On 1 Oct 2020, at 22:33, Matthew Smith via lists.fd.io <http://lists.fd.io/> 
>> mailto:mgsmith=netgate@lists.fd.io>> 
>> wrote:
>> 
>> Hi Chris,
>> 
>> I did this in my local build:
>> 
>> diff --git a/build/external/packages/dpdk.mk <http://dpdk.mk/> 
>> b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> index 49761cd56..a30ffd2ac 100644
>> --- a/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> +++ b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> @@ -139,6 +139,7 @@ DPDK_MESON_ARGS = \
>> -Dtests=false \
>> "-Ddisable_drivers=$(DPDK_DRIVERS_DISABLED)" \
>> "-Ddisable_libs=$(DPDK_LIBS_DISABLED)" \
>> +   -Dibverbs_link=dlopen \
>> -Db_pie=true \
>> -Dmachine=$(DPDK_MACHINE) \
>> --buildtype=$(DPDK_BUILD_TYPE)
>> 
>> If I try to submit it upstream, I would probably do something nicer using a 
>> DPDK_ environment variable, but for the moment this got me past 
>> that error. I have not actually tested with an mlx5 device yet, so I don't 
>> know if something additional will be required in order to forward packets 
>> via an mlx5 NIC, but it did fix the error you pasted and allow 
>> dpdk_plugin.so to be loaded.
>> 
>> -Matt
>> 
>> 
>> On Thu, Oct 1, 2020 at 2:10 PM Christian Hopps > <mailto:cho...@chopps.org>> wrote:
>> I've rebased my local branch on the latest master and dpdk is failing to 
>> load now b/c
>> 
>> 2020/10/01 18:31:54:514 errplugin/load
>> /home/chopps/w/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins/dpdk_plugin.so:
>>  undefined symbol: ibv_fork_init
>> 
>> I noticed that the dpdk build system has been changed is there something I 
>> need to do to get it to link properly with the required libraries now?
>> 
>>

Re: [vpp-dev] Latest master, dpdk and mlx5 failing

[Christian Hopps]

> On Oct 2, 2020, at 1:07 PM, Damjan Marion via lists.fd.io 
>  wrote:
> 
> 
> May be a good time to consider moving to rdma/native implementation.
> What are the gaps for you?

We have some conditional chained decap code based on the underlying dpdk mbufs 
to do buffer indirection -- it's not overly pretty but it works :) Also, 
although we haven't tested with it yet, we want to use async crypto offload, 
and once we pay the mbuf penalty for that ... *shrug*

When using copy (rather than chain) and inline crypto, native drivers are an 
option.

Thanks,
Chris.

> 
> 
> —
> Damjan
> 
>> On 02.10.2020., at 19:01, Christian Hopps > <mailto:cho...@chopps.org>> wrote:
>> 
>> FWIW, I got this building and running; however, since the bump to 20.08 
>> VPP+DPDK no longer finds the mlx5 interfaces.
>> 
>> 2020/10/02 16:58:53:176 warn   dpdk   EAL init args: -c 3f -n 4 
>> --in-memory --file-prefix vpp -w :21:00.1 -w :21:00.0 --master-lcore >> 0
>> 2020/10/02 16:58:53:508 notice dpdk   DPDK drivers found no 
>> Ethernet devices...
>> 
>> Those devices are found when dpdk 20.05 is used instead.
>> 
>>> On Oct 2, 2020, at 12:21 PM, Matthew Smith via lists.fd.io 
>>> <http://lists.fd.io/> >> <mailto:mgsmith=netgate@lists.fd.io>> wrote:
>>> 
>>> Hi Mohammed,
>>> 
>>> I think it will only affect builds where DPDK_MLX5_PMD=y is set, but I 
>>> cannot say for sure. The scripts/configurations I build with always set 
>>> that flag, so I have not tried to generate a build without it set recently.
>>> 
>>> -Matt
>>> 
>>> 
>>> On Fri, Oct 2, 2020 at 12:57 AM Mohammed HAWARI >> <mailto:momohaw...@gmail.com>> wrote:
>>> Hello Chris, Matthew,
>>> 
>>> Thanks for raising that issue. Just to be clear and better understand, does 
>>> the problem occur with the default config, i.e., without trying to compile 
>>> any MLX driver in DPDK? Or does it only appear when setting DPDK_MLX5_PMD=y 
>>> ?
>>> Thanks
>>> Best regards
>>> Mohammed
>>>> On 1 Oct 2020, at 22:33, Matthew Smith via lists.fd.io 
>>>> <http://lists.fd.io/> >>> <mailto:mgsmith=netgate@lists.fd.io>> wrote:
>>>> 
>>>> Hi Chris,
>>>> 
>>>> I did this in my local build:
>>>> 
>>>> diff --git a/build/external/packages/dpdk.mk <http://dpdk.mk/> 
>>>> b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>>>> index 49761cd56..a30ffd2ac 100644
>>>> --- a/build/external/packages/dpdk.mk <http://dpdk.mk/>
>>>> +++ b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>>>> @@ -139,6 +139,7 @@ DPDK_MESON_ARGS = \
>>>> -Dtests=false \
>>>> "-Ddisable_drivers=$(DPDK_DRIVERS_DISABLED)" \
>>>> "-Ddisable_libs=$(DPDK_LIBS_DISABLED)" \
>>>> +   -Dibverbs_link=dlopen \
>>>> -Db_pie=true \
>>>> -Dmachine=$(DPDK_MACHINE) \
>>>> --buildtype=$(DPDK_BUILD_TYPE)
>>>> 
>>>> If I try to submit it upstream, I would probably do something nicer using 
>>>> a DPDK_ environment variable, but for the moment this got me 
>>>> past that error. I have not actually tested with an mlx5 device yet, so I 
>>>> don't know if something additional will be required in order to forward 
>>>> packets via an mlx5 NIC, but it did fix the error you pasted and allow 
>>>> dpdk_plugin.so to be loaded.
>>>> 
>>>> -Matt
>>>> 
>>>> 
>>>> On Thu, Oct 1, 2020 at 2:10 PM Christian Hopps >>> <mailto:cho...@chopps.org>> wrote:
>>>> I've rebased my local branch on the latest master and dpdk is failing to 
>>>> load now b/c
>>>> 
>>>> 2020/10/01 18:31:54:514 errplugin/load
>>>> /home/chopps/w/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins/dpdk_plugin.so:
>>>>  undefined symbol: ibv_fork_init
>>>> 
>>>> I noticed that the dpdk build system has been changed is there something I 
>>>> need to do to get it to link properly with the required libraries now?
>>>> 
>>>> I did change "n" in
>>>> 
>>>>   DPDK_MLX5_PMD?= n
>>>>   DPDK_MLX5_COMMON_PMD ?= n
>>>> 
>>>> to "y" to try and get MLX5 PMD to build.
>>>> 
>>>> I also added
>>>> 
>>>>   vpp_uses_dpdk_mlx5_pmd = yes
>>>> 
>>>> to build-data/platforms/vpp.mk <http://vpp.mk/>
>>>> 
>>>> I also tried adding:
>>>> 
>>>>   vpp_uses_dpdk_ibverbs_link_dlopen = yes
>>>> 
>>>> which didn't fix the problem.
>>>> 
>>>> Any suggestions on how to fix this?
>>>> 
>>>> Thanks,
>>>> Chris.
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17633): https://lists.fd.io/g/vpp-dev/message/17633
Mute This Topic: https://lists.fd.io/mt/77247865/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Latest master, dpdk and mlx5 failing

[Christian Hopps]

FWIW, I got this building and running; however, since the bump to 20.08 
VPP+DPDK no longer finds the mlx5 interfaces.

2020/10/02 16:58:53:176 warn   dpdk   EAL init args: -c 3f -n 4 
--in-memory --file-prefix vpp -w :21:00.1 -w :21:00.0 --master-lcore 0
2020/10/02 16:58:53:508 notice dpdk   DPDK drivers found no 
Ethernet devices...

Those devices are found when dpdk 20.05 is used instead.

> On Oct 2, 2020, at 12:21 PM, Matthew Smith via lists.fd.io 
>  wrote:
> 
> Hi Mohammed,
> 
> I think it will only affect builds where DPDK_MLX5_PMD=y is set, but I cannot 
> say for sure. The scripts/configurations I build with always set that flag, 
> so I have not tried to generate a build without it set recently.
> 
> -Matt
> 
> 
> On Fri, Oct 2, 2020 at 12:57 AM Mohammed HAWARI  <mailto:momohaw...@gmail.com>> wrote:
> Hello Chris, Matthew,
> 
> Thanks for raising that issue. Just to be clear and better understand, does 
> the problem occur with the default config, i.e., without trying to compile 
> any MLX driver in DPDK? Or does it only appear when setting DPDK_MLX5_PMD=y ?
> Thanks
> Best regards
> Mohammed
>> On 1 Oct 2020, at 22:33, Matthew Smith via lists.fd.io <http://lists.fd.io/> 
>> mailto:mgsmith=netgate@lists.fd.io>> 
>> wrote:
>> 
>> Hi Chris,
>> 
>> I did this in my local build:
>> 
>> diff --git a/build/external/packages/dpdk.mk <http://dpdk.mk/> 
>> b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> index 49761cd56..a30ffd2ac 100644
>> --- a/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> +++ b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> @@ -139,6 +139,7 @@ DPDK_MESON_ARGS = \
>> -Dtests=false \
>> "-Ddisable_drivers=$(DPDK_DRIVERS_DISABLED)" \
>> "-Ddisable_libs=$(DPDK_LIBS_DISABLED)" \
>> +   -Dibverbs_link=dlopen \
>> -Db_pie=true \
>> -Dmachine=$(DPDK_MACHINE) \
>> --buildtype=$(DPDK_BUILD_TYPE)
>> 
>> If I try to submit it upstream, I would probably do something nicer using a 
>> DPDK_ environment variable, but for the moment this got me past 
>> that error. I have not actually tested with an mlx5 device yet, so I don't 
>> know if something additional will be required in order to forward packets 
>> via an mlx5 NIC, but it did fix the error you pasted and allow 
>> dpdk_plugin.so to be loaded.
>> 
>> -Matt
>> 
>> 
>> On Thu, Oct 1, 2020 at 2:10 PM Christian Hopps > <mailto:cho...@chopps.org>> wrote:
>> I've rebased my local branch on the latest master and dpdk is failing to 
>> load now b/c
>> 
>> 2020/10/01 18:31:54:514 errplugin/load
>> /home/chopps/w/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins/dpdk_plugin.so:
>>  undefined symbol: ibv_fork_init
>> 
>> I noticed that the dpdk build system has been changed is there something I 
>> need to do to get it to link properly with the required libraries now?
>> 
>> I did change "n" in
>> 
>>   DPDK_MLX5_PMD?= n
>>   DPDK_MLX5_COMMON_PMD ?= n
>> 
>> to "y" to try and get MLX5 PMD to build.
>> 
>> I also added
>> 
>>   vpp_uses_dpdk_mlx5_pmd = yes
>> 
>> to build-data/platforms/vpp.mk <http://vpp.mk/>
>> 
>> I also tried adding:
>> 
>>   vpp_uses_dpdk_ibverbs_link_dlopen = yes
>> 
>> which didn't fix the problem.
>> 
>> Any suggestions on how to fix this?
>> 
>> Thanks,
>> Chris.
>> 
>> 
>> 
>> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17631): https://lists.fd.io/g/vpp-dev/message/17631
Mute This Topic: https://lists.fd.io/mt/77247865/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Latest master, dpdk and mlx5 failing

[Christian Hopps]

I've rebased my local branch on the latest master and dpdk is failing to load 
now b/c

2020/10/01 18:31:54:514 errplugin/load
/home/chopps/w/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins/dpdk_plugin.so:
 undefined symbol: ibv_fork_init

I noticed that the dpdk build system has been changed is there something I need 
to do to get it to link properly with the required libraries now?

I did change "n" in

  DPDK_MLX5_PMD?= n
  DPDK_MLX5_COMMON_PMD ?= n

to "y" to try and get MLX5 PMD to build.

I also added

  vpp_uses_dpdk_mlx5_pmd = yes

to build-data/platforms/vpp.mk

I also tried adding:

  vpp_uses_dpdk_ibverbs_link_dlopen = yes

which didn't fix the problem.

Any suggestions on how to fix this?

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17626): https://lists.fd.io/g/vpp-dev/message/17626
Mute This Topic: https://lists.fd.io/mt/77247865/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] dpdk 20.08

[Christian Hopps]

With the bump to dpdk 20.08, vpp (through DPDK) is not finding any interfaces 
on my systems with connectx-5 and connectx-6 NICs. Is this working for anyone 
else and the mellanox cards?

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17287): https://lists.fd.io/g/vpp-dev/message/17287
Mute This Topic: https://lists.fd.io/mt/76437772/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Why drop NULL crypto alg tunnel packets?

[Christian Hopps]

Hi,

Regarding: https://gerrit.fd.io/r/c/vpp/+/20593

I don't understand why the fix for writing the trailer in the wrong place for 
NULL crypto alg was to create a drop node for NULL crypto (and drop the 
packets) vs fixing the trailer writing bug. Was this a stop-gap fix?

I certainly use a crypto/integ NULL configurations for debug and development 
(e.g., to test performance on systems w/o HW crypto acceleration), and feel 
this is a valid configuration.

Thanks,
Chris.



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17281): https://lists.fd.io/g/vpp-dev/message/17281
Mute This Topic: https://lists.fd.io/mt/76345436/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] dpdk crypto dev and socket id / numa problem

[Christian Hopps]

Hi,

So there's a problem in the dpdk crypto code I think. In dpdk's ipsec.c in 
crypto_scan_devs() each crypto device is assigned a numa domain number

  dev->numa = rte_cryptodev_socket_id (i);

Which I figured out meant I need to allocate a vdev cryptodev per numa (for 
numa domains I will use) specifying the numa value as the socket id.

When dpdk does the crypto_auto_placement assigning queue pairs (resources) to 
each worker thread, it doesn't pay any attention to this numa value.

I think it might should only allocate resources (queue pairs) from crypto 
devices that co-locate with that workers nuam domain. A per numa rte_mempool is 
allocated and shared by all crypto devices associated with that numa domain.

I hit a problem b/c I wasn't creating a cryptodev for numa node 2 
(socket_id=2), but the worker was running on that numa node (2), so I SIGSEGV'd 
in crypto_alloc_ops() b/c it's mempool was NULL; however, even if the user does 
allocated enough devices (with enough queue pairs to cover all workers for each 
crypto device) the wrong cryptodev will be used by a given worker if it doesn't 
share it's numa domain (and so the memory will not be local)

I have a simple fix for this I could submit.

However, getting this right might be even more important when the crypto device 
is an actual physical device (e.g., QAT) and it's PCI root is located in a 
given numa domain. Here you would want to *prefer* to use the device associate 
with the numa node, however, I think you might want worker threads running in 
other numa domains to fallback to using a non-local crypto offload device if 
there was no numa local one.

It seems important for high performance setups where one has a NIC and QAT card 
per socket/pci bus to be choosing the correct (local) crypto offload device, 
but for less perfect multi-socket configurations one should probably still be 
able to use a singular QAT card. :)

Thoughts?

Thanks,
Chris.






signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17280): https://lists.fd.io/g/vpp-dev/message/17280
Mute This Topic: https://lists.fd.io/mt/76343919/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Question on performance tests..

[Christian Hopps]

> On Aug 17, 2020, at 10:52 AM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi Christian,
> 
> I am CC'ing csit-dev but here are some elements.
> 
> AFAIK, CSIT results are not trustworthy for Mellanox NIC yet unfortunately: 
> T-Rex suffers from "stretch duration" when using Mellanox NIC to inject 
> traffic.
> Here is my understanding of what this means: when asked to send at X packets 
> in Y seconds (hence at a rate of X/Y pps), T-Rex effectively takes Y + Z 
> seconds. CSIT figures are based on the programmed rate (X/Y) and not the real 
> rate (X/(Y+Z)).
> In a nutshell: we are not really sure where we're at with Mellanox card in 
> CSIT.

This is very interesting. I believe I am seeing this stretch from trex as well; 
however, it happens with i710 as well as a cx5 cards in the trex tester.

Basically I tell trex to start then after the expected amount of time (for 
which I'm doing other sampling) I tell the trex client to wait for traffic to 
complete. I would expect this to return immediately as I've been timing things. 
Instead it is taking longer (perhaps much longer) depending on how high the 
rate was that I tried to send.

  c.start(ports=check_ports, mult=mult, duration=duration)
  [wait duration seconds doing sampling]
  c.wait_on_traffic(rx_delay_ms=100)

Thanks,
Chris.

> Your figures look similar to what I get on my setup (~20Mpps bi-directional 
> for IPv4 forwarding).
> 
> Best
> ben
> 
>> -Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Christian
>> Hopps
>> Sent: lundi 17 août 2020 15:00
>> To: vpp-dev 
>> Cc: Christian Hopps 
>> Subject: [vpp-dev] Question on performance tests..
>> 
>> [sent to csit-dev but perhaps since I'm not subscribed it didn't post]
>> 
>> I'm looking at this performance graph:
>> 
>> https://docs.fd.io/csit/master/trending/trending/ip4-2n-clx-cx556a.html
>> 
>> Is this a uni-directional or a bi-directional test?
>> 
>> There is a huge performance gain between June and July it seems. Do we
>> know which change this was? I'm on a current vpp master and I am only
>> getting about 9Mpps+ bidirectional (18Mpps total) with IPv4 forwarding of
>> 64 octet packets with 100GE Cx5 cards and Trex.
>> 
>> Thanks,
>> Chris.
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17252): https://lists.fd.io/g/vpp-dev/message/17252
Mute This Topic: https://lists.fd.io/mt/76242714/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Question on performance tests..

[Christian Hopps]

[sent to csit-dev but perhaps since I'm not subscribed it didn't post]

I'm looking at this performance graph:

 https://docs.fd.io/csit/master/trending/trending/ip4-2n-clx-cx556a.html

Is this a uni-directional or a bi-directional test?

There is a huge performance gain between June and July it seems. Do we know 
which change this was? I'm on a current vpp master and I am only getting about 
9Mpps+ bidirectional (18Mpps total) with IPv4 forwarding of 64 octet packets 
with 100GE Cx5 cards and Trex.

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17247): https://lists.fd.io/g/vpp-dev/message/17247
Mute This Topic: https://lists.fd.io/mt/76242714/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] address sanitizer and unit tests not working

[Christian Hopps]

> On Aug 12, 2020, at 10:12 AM, Benoit Ganne (bganne)  wrote:
> 
> Hi Christian,
> 
> Thanks for the head's up.
> 
>> I think maybe the right thing to do here is create a custom cmake build
>> rule to stage the entire python package in build-root/build*
> 
> Alternatively, do you know if we can add an arbitrary file to be packaged by 
> setup.py? If we could just tell ssetup.py 'take also this absolute filename' 
> that would allow us to generate the vpp_ffi.py in build-root and left the 
> rest unchanged...

Not exaclty. I think if you made the generated file part of a sub-pkg maybe you 
could use this?

https://docs.python.org/3/distutils/setupscript.html#listing-whole-packages 


Alternatively, is there a way to figure out if ASAN is required by the ffi 
dlopen at runtime? Then we wouldn't need to generate different source files.

Thanks,
Chris.

> 
> Best
> ben
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17203): https://lists.fd.io/g/vpp-dev/message/17203
Mute This Topic: https://lists.fd.io/mt/75917538/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] address sanitizer and unit tests not working

[Christian Hopps]

FWIW I looked at this briefly but started rat-holing trying to better integrate 
the python packaging (which I am somewhat familiar with) with cmake (which I 
know almost nothing about).

I think maybe the right thing to do here is create a custom cmake build rule to 
stage the entire python package in build-root/build* doing the pattern 
substitution on vpp_ffi.py.in as well as setup.py (to set a package_dir to 
point to the new staging location). Then the install rule would run "python 
setup.py install" inside that staging build dir with the arguments to install 
in the build-root/install* location (this latter rule is already there). This 
gets away from modifying files inside the source directory, as not modifying 
source directory files seems the expected behavior when using cmake.

I'm not sure of the cmake magic to set dependencies correctly so that the 
custom build rule to copy the files gets invoked whenever a source python file 
changes, or ASAN is disabled or enabled, and how to carry those dependencies 
forward to the install rule.

Thanks,
Chris.

> On Aug 7, 2020, at 12:01 PM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
>> If I clear the runner (gitlab) cache it causes a full rebuild, and then I
>> don't see the problem.
>> Shouldn't the generated file be going under build-root? I ask b/c it's
>> showing up in git status now. Maybe this also affects the cmake
>> dependency?
> 
> Hmm great point, I'll see how to fix that.
> 
> Best
> ben
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17196): https://lists.fd.io/g/vpp-dev/message/17196
Mute This Topic: https://lists.fd.io/mt/75917538/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] address sanitizer and unit tests not working

[Christian Hopps]

> On Aug 7, 2020, at 11:34 AM, Benoit Ganne (bganne)  wrote:
> 
>> I'm seeing some issues with src/vpp-api/python/vpp_papi/vpp_ffi.py not
>> being generated using the provided fix in my CI test runs. I think it has
>> something to do with an old build being present in the ci runner.
> 
> Yep, I ran into this once too but I am not sure which build dep is the issue.
> However, it should only happened once right after the patch application, and 
> should be solved with a cleanup: 'make rebuild' (or 'make wipe' etc.). 
> Subsequent incremental builds should not be affected anymore.
> Or do you see it happening over and over again?

If I clear the runner (gitlab) cache it causes a full rebuild, and then I don't 
see the problem.

Shouldn't the generated file be going under build-root? I ask b/c it's showing 
up in git status now. Maybe this also affects the cmake dependency?

Thanks,
Chris.

> Best
> ben
> 
>> ...
>> make[2]: Leaving directory '/builds/4gc3WVUZ/0/oss/vpp/test/ext'
>> Traceback (most recent call last):
>>  File "/builds/4gc3WVUZ/0/oss/vpp/build-root/build-
>> test/src/sanity_run_vpp.py", line 7, in 
>>from framework import VppDiedError, VppTestCase, KeepAliveReporter
>>  File "/builds/4gc3WVUZ/0/oss/vpp/test/framework.py", line 33, in
>> 
>>from vpp_papi.vpp_stats import VPPStats
>>  File "/builds/4gc3WVUZ/0/oss/vpp/src/vpp-
>> api/python/vpp_papi/vpp_stats.py", line 5, in 
>>import vpp_papi.vpp_ffi as vpp_ffi
>> ModuleNotFoundError: No module named 'vpp_papi.vpp_ffi'
>> ***
>> * Sanity check failed, cannot run vpp
>> ***
>> 
>> 
>>  On Aug 3, 2020, at 3:19 AM, Benoit Ganne (bganne) > <mailto:bga...@cisco.com <mailto:bga...@cisco.com>> > wrote:
>> 
>>  Hi Christian,
>> 
>>  you need https://gerrit.fd.io/r/c/vpp/+/27268 
>> <https://gerrit.fd.io/r/c/vpp/+/27268> or you can recompile
>> with gcc instead.
>>  There are a couple of crashes still, here are some fixes you might
>> want to apply first:
>>  - https://gerrit.fd.io/r/c/vpp/+/27962 
>> <https://gerrit.fd.io/r/c/vpp/+/27962>
>>  - https://gerrit.fd.io/r/c/vpp/+/27963 
>> <https://gerrit.fd.io/r/c/vpp/+/27963>
>>  - https://gerrit.fd.io/r/c/vpp/+/27959 
>> <https://gerrit.fd.io/r/c/vpp/+/27959>
>> 
>>  Best
>>  ben
>> 
>> 
>> 
>>  -Original Message-
>>  From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> 
>> <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>>  > d...@lists.fd.io <mailto:d...@lists.fd.io> <mailto:vpp-dev@lists.fd.io 
>> <mailto:vpp-dev@lists.fd.io>> > On Behalf Of Christian
>>  Hopps
>>  Sent: samedi 1 août 2020 00:54
>>  To: vpp-dev mailto:vpp-dev@lists.fd.io> 
>> <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>>
>>> 
>>  Cc: Christian Hopps > <mailto:cho...@chopps.org>
>> <mailto:cho...@chopps.org <mailto:cho...@chopps.org>> >
>>  Subject: [vpp-dev] address sanitizer and unit tests not
>> working
>> 
>>  Hi,
>> 
>>  I'm trying to run the unit test with address sanitizer enabled
>> but it's
>>  failing to run on the latest master for me. This is building
>> on ubuntu
>>  18.04 with clang-9 installed, the build used the same
>> sanitizer make
>>  option too.
>> 
>>  Any suggestions?
>> 
>>  Thanks,
>>  Chris.
>> 
>>  [18:51:50 dak:/var/build/vpp]$ make test-debug
>> VPP_EXTRA_CMAKE_ARGS=-
>>  DVPP_ENABLE_SANITIZE_ADDR=ON
>>  make -C /var/build/vpp/build-root PLATFORM=vpp TAG=vpp_debug
>> vpp-install
>>  make[1]: Entering directory '/var/build/vpp/build-root'
>>   Arch for platform 'vpp' is native 
>>   Finding source for external 
>>   Makefile fragment found in /var/build/vpp/build-
>>  data/packages/external.mk 
>>   Source found in /var/build/vpp/build 
>>   Arch for platform 'vpp' is native 
>>   Finding source for vpp 
>>   Makefile fragment found in /var/build/vpp/b

Re: [vpp-dev] address sanitizer and unit tests not working

[Christian Hopps]

I'm seeing some issues with src/vpp-api/python/vpp_papi/vpp_ffi.py not being 
generated using the provided fix in my CI test runs. I think it has something 
to do with an old build being present in the ci runner.

...
make[2]: Leaving directory '/builds/4gc3WVUZ/0/oss/vpp/test/ext'
Traceback (most recent call last):
  File 
"/builds/4gc3WVUZ/0/oss/vpp/build-root/build-test/src/sanity_run_vpp.py", line 
7, in 
from framework import VppDiedError, VppTestCase, KeepAliveReporter
  File "/builds/4gc3WVUZ/0/oss/vpp/test/framework.py", line 33, in 
from vpp_papi.vpp_stats import VPPStats
  File "/builds/4gc3WVUZ/0/oss/vpp/src/vpp-api/python/vpp_papi/vpp_stats.py", 
line 5, in 
import vpp_papi.vpp_ffi as vpp_ffi
ModuleNotFoundError: No module named 'vpp_papi.vpp_ffi'
***
* Sanity check failed, cannot run vpp
***

> On Aug 3, 2020, at 3:19 AM, Benoit Ganne (bganne)  wrote:
> 
> Hi Christian,
> 
> you need https://gerrit.fd.io/r/c/vpp/+/27268 or you can recompile with gcc 
> instead.
> There are a couple of crashes still, here are some fixes you might want to 
> apply first:
> - https://gerrit.fd.io/r/c/vpp/+/27962
> - https://gerrit.fd.io/r/c/vpp/+/27963
> - https://gerrit.fd.io/r/c/vpp/+/27959
> 
> Best
> ben
> 
>> -----Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Christian
>> Hopps
>> Sent: samedi 1 août 2020 00:54
>> To: vpp-dev 
>> Cc: Christian Hopps 
>> Subject: [vpp-dev] address sanitizer and unit tests not working
>> 
>> Hi,
>> 
>> I'm trying to run the unit test with address sanitizer enabled but it's
>> failing to run on the latest master for me. This is building on ubuntu
>> 18.04 with clang-9 installed, the build used the same sanitizer make
>> option too.
>> 
>> Any suggestions?
>> 
>> Thanks,
>> Chris.
>> 
>> [18:51:50 dak:/var/build/vpp]$ make test-debug VPP_EXTRA_CMAKE_ARGS=-
>> DVPP_ENABLE_SANITIZE_ADDR=ON
>> make -C /var/build/vpp/build-root PLATFORM=vpp TAG=vpp_debug vpp-install
>> make[1]: Entering directory '/var/build/vpp/build-root'
>>  Arch for platform 'vpp' is native 
>>  Finding source for external 
>>  Makefile fragment found in /var/build/vpp/build-
>> data/packages/external.mk 
>>  Source found in /var/build/vpp/build 
>>  Arch for platform 'vpp' is native 
>>  Finding source for vpp 
>>  Makefile fragment found in /var/build/vpp/build-data/packages/vpp.mk
>> 
>>  Source found in /var/build/vpp/src 
>>  Configuring external: nothing to do 
>>  Building external: nothing to do 
>>  Installing external: nothing to do 
>>  Configuring vpp: nothing to do 
>>  Building vpp in /var/build/vpp/build-root/build-vpp_debug-native/vpp
>> 
>> ninja: no work to do.
>>  Installing vpp: nothing to do 
>> make[1]: Leaving directory '/var/build/vpp/build-root'
>> make -C test VPP_BUILD_DIR=/var/build/vpp/build-root/build-vpp_debug-
>> native VPP_BIN=/var/build/vpp/build-root/install-vpp_debug-
>> native/vpp/bin/vpp VPP_PLUGIN_PATH=/var/build/vpp/build-root/install-
>> vpp_debug-native/vpp/lib/vpp_plugins:/var/build/vpp/build-root/install-
>> vpp_debug-native/vpp/lib64/vpp_plugins
>> VPP_TEST_PLUGIN_PATH=/var/build/vpp/build-root/install-vpp_debug-
>> native/vpp/lib/vpp_api_test_plugins:/var/build/vpp/build-root/install-
>> vpp_debug-native/vpp/lib64/vpp_api_test_plugins
>> VPP_INSTALL_PATH=/var/build/vpp/build-root/install-vpp_debug-native/
>> LD_LIBRARY_PATH=/var/build/vpp/build-root/install-vpp_debug-
>> native/vpp/lib/:/var/build/vpp/build-root/install-vpp_debug-
>> native/vpp/lib64/ EXTENDED_TESTS= PYTHON= OS_ID=ubuntu
>> RND_SEED=1596235925.011464 CACHE_OUTPUT= test
>> make[1]: Entering directory '/var/build/vpp/test'
>> ls: cannot access '/var/build/vpp/src/plugins/sctp/test/*.py': No such
>> file or directory
>> make -C ext test-apps
>> make[2]: Entering directory '/var/build/vpp/test/ext'
>> cc -o /var/build/vpp/build-root/build-test/vapi_test/vapi_c_test -
>> std=gnu99 -g -Wall -lstdc++ -pthread -I/var/build/vpp/src -
>> I/var/build/vpp/build-root/install-vpp_debug-native//vpp/include -
>> I/var/build/vpp/build-root/build-test/vapi_test
>> /var/build/vpp/test/ext/vapi_c_test.c -L/var/build/vpp/build-root/install-
>> vpp_debug-native//vpp/lib -lvppinfra -lvlibmemoryclient -lsvm -lpthread -
>> lcheck -lrt -lm -lvapiclient -lsubunit
>> /var/

[vpp-dev] address sanitizer and unit tests not working

[Christian Hopps]

Hi,

I'm trying to run the unit test with address sanitizer enabled but it's failing 
to run on the latest master for me. This is building on ubuntu 18.04 with 
clang-9 installed, the build used the same sanitizer make option too.

Any suggestions?

Thanks,
Chris.

[18:51:50 dak:/var/build/vpp]$ make test-debug 
VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON
make -C /var/build/vpp/build-root PLATFORM=vpp TAG=vpp_debug vpp-install
make[1]: Entering directory '/var/build/vpp/build-root'
 Arch for platform 'vpp' is native 
 Finding source for external 
 Makefile fragment found in /var/build/vpp/build-data/packages/external.mk 

 Source found in /var/build/vpp/build 
 Arch for platform 'vpp' is native 
 Finding source for vpp 
 Makefile fragment found in /var/build/vpp/build-data/packages/vpp.mk 
 Source found in /var/build/vpp/src 
 Configuring external: nothing to do 
 Building external: nothing to do 
 Installing external: nothing to do 
 Configuring vpp: nothing to do 
 Building vpp in /var/build/vpp/build-root/build-vpp_debug-native/vpp 
ninja: no work to do.
 Installing vpp: nothing to do 
make[1]: Leaving directory '/var/build/vpp/build-root'
make -C test VPP_BUILD_DIR=/var/build/vpp/build-root/build-vpp_debug-native 
VPP_BIN=/var/build/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp 
VPP_PLUGIN_PATH=/var/build/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins:/var/build/vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_plugins
 
VPP_TEST_PLUGIN_PATH=/var/build/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_api_test_plugins:/var/build/vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_api_test_plugins
 VPP_INSTALL_PATH=/var/build/vpp/build-root/install-vpp_debug-native/ 
LD_LIBRARY_PATH=/var/build/vpp/build-root/install-vpp_debug-native/vpp/lib/:/var/build/vpp/build-root/install-vpp_debug-native/vpp/lib64/
 EXTENDED_TESTS= PYTHON= OS_ID=ubuntu RND_SEED=1596235925.011464 CACHE_OUTPUT= 
test
make[1]: Entering directory '/var/build/vpp/test'
ls: cannot access '/var/build/vpp/src/plugins/sctp/test/*.py': No such file or 
directory
make -C ext test-apps
make[2]: Entering directory '/var/build/vpp/test/ext'
cc -o /var/build/vpp/build-root/build-test/vapi_test/vapi_c_test -std=gnu99 -g 
-Wall -lstdc++ -pthread -I/var/build/vpp/src 
-I/var/build/vpp/build-root/install-vpp_debug-native//vpp/include 
-I/var/build/vpp/build-root/build-test/vapi_test 
/var/build/vpp/test/ext/vapi_c_test.c 
-L/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib -lvppinfra 
-lvlibmemoryclient -lsvm -lpthread -lcheck -lrt -lm -lvapiclient -lsubunit
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_store_n'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_load8'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_load2'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_load4'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_set_shadow_f8'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_stack_malloc_2'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_unregister_globals'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_stack_malloc_4'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_stack_free_5'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_register_globals'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_set_shadow_00'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_stack_malloc_0'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_store2'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_store4'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_report_store8'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_handle_no_return'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_allocas_unpoison'
/var/build/vpp/build-root/install-vpp_debug-native//vpp/lib/libvppinfra.so: 
undefined reference to `__asan_poison_memory_region'

Re: [vpp-dev] debugging corrupted frame arguments

[Christian Hopps]

> On Jul 23, 2020, at 4:09 PM, Dave Barach via lists.fd.io 
>  wrote:
> 
[ swapped the order of my reply :) ]

> Without having all of the source code available and a reasonable way to repro 
> the issue, it's going to be quite hard to help you find the culprit.

Yes, I realize, so was really only looking for general debug tools to help me 
track it down, e.g., like Benoit's suggestion of AddressSanitizer. Which I have 
running with the latest SIGSEGVs that I put below.

> What is the invalid buffer index value? How many elements are in the frame? 
> Is it always the same frame element which takes a lightning hit?

Sorry for late response It's been tough to gather this time around. I have 2 
instances that hit around a similar time:

vpp 1:

(gdb) frame
#1  0x74df20ba in esp_encrypt_inline (vm=0x7fffb4a41040, 
node=0x7fffb4d95f00, frame=0x7fffb4a63ac0, is_ip6=0, is_tun=0, async_next=1) at 
/home/chopps/w/vpp/src/vnet/ipsec/esp_encrypt.c:628
628   p = vlib_buffer_get_current (b[1]);
(gdb) p *frame
$40 = {
  frame_flags = 6,
  flags = 0,
  scalar_size = 0 '\000',
  vector_size = 4 '\004',
  n_vectors = 5,
  arguments = 0x7fffb4a63ac8 "\376\376\376\376\376\376\376\376\344\317", 

}
(gdb) p n_left
$41 = 3
(gdb) p/x *from@5
$42 = {0xf2cfe4, 0xe8252e, 0xfb7161, 0xb4d0b42a, 0x72a90}
(gdb) p/x *from@10
$43 = {0xf2cfe4, 0xe8252e, 0xfb7161, 0xb4d0b42a, 0x72a90, 0xe41942, 0xe41dd3, 
0xe4ebf7, 0xe4c8bd, 0xf18a5f}

vpp 2:

(gdb) frame
#1  0x74df20ba in esp_encrypt_inline (vm=0x7fffb4a41040, 
node=0x7fffb4b530c0, frame=0x7fffb48205c0, is_ip6=0, is_tun=0, async_next=1) at 
/home/chopps/w/vpp/src/vnet/ipsec/esp_encrypt.c:628
628   p = vlib_buffer_get_current (b[1]);
(gdb) p *frame
$12 = {
  frame_flags = 6,
  flags = 0,
  scalar_size = 0 '\000',
  vector_size = 4 '\004',
  n_vectors = 7,
  arguments = 0x7fffb48205c8 "\376\376\376\376\376\376\376\376\332\354\377"
}
(gdb) p n_left
$13 = 5
(gdb) p/x *from@7
$14 = {0xffecda, 0xe50030, 0xf0baed, 0x7442defe, 0x727ba, 0x219c, 0x0}
(gdb) p/x *from@10
$15 = {0xffecda, 0xe50030, 0xf0baed, 0x7442defe, 0x727ba, 0x219c, 0x0, 
0xe4369d, 0xeb2e1f, 0xfe172a}
(gdb)

In both cases above the corruption starts in the 4th element, it also appears 
to run to the end of the from array (5th and 7th element respectively), but not 
any further (that's why I printed @10 for both froms).

I'm going to rerun this at this point, but this time I am going to validate the 
from array on entry to esp_encrypt_inline to see if the corruption is happening 
before or during that functions execution.

The other variation I can try is to run the tunnel a bit faster and slower to 
increase the per call packet count to see if the corruption moves away from the 
4th element.

Thanks,
Chris.

> 
> D.
> -Original Message-
> From: vpp-dev@lists.fd.io  On Behalf Of Christian Hopps
> Sent: Thursday, July 23, 2020 1:10 PM
> To: vpp-dev 
> Cc: Christian Hopps 
> Subject: [vpp-dev] debugging corrupted frame arguments
> 
> I have a very intermittent memory corruption occurring in the buffer indices 
> passed in a nodes frame (encryption node).
> 
> Basically one of the indices is clearly not a vadlid buffer index, and this 
> is leading to a SIGSEGV when the code attempts to use the buffer.
> 
> I see that vlib_frame_ts are allocated from the main heap, so I'm wondering 
> is there any heap/alloc debugging I can enable to help figure out what is 
> corrupting this vlib_frame_t?
> 
> FWIW what seems weird is that I am validating the indices in 
> vlib_put_frame_to_node() (I changed the validation code to actually resolve 
> the buffer index), so apparently the vector is being corrupted between the 
> node that creates and puts the vlib_frame_t, and the pending node being 
> dispatched. The heap appears to be per CPU as well, this all makes things odd 
> since the pending node should be running immediately after the node that put 
> the valid frame there. So it's hard to imagine what could be corrupting this 
> memory in between the put and the dispatch.
> 
> Thanks,
> Chris.
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17082): https://lists.fd.io/g/vpp-dev/message/17082
Mute This Topic: https://lists.fd.io/mt/75750436/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] debugging corrupted frame arguments

[Christian Hopps]

I have a very intermittent memory corruption occurring in the buffer indices 
passed in a nodes frame (encryption node).

Basically one of the indices is clearly not a vadlid buffer index, and this is 
leading to a SIGSEGV when the code attempts to use the buffer.

I see that vlib_frame_ts are allocated from the main heap, so I'm wondering is 
there any heap/alloc debugging I can enable to help figure out what is 
corrupting this vlib_frame_t?

FWIW what seems weird is that I am validating the indices in 
vlib_put_frame_to_node() (I changed the validation code to actually resolve the 
buffer index), so apparently the vector is being corrupted between the node 
that creates and puts the vlib_frame_t, and the pending node being dispatched. 
The heap appears to be per CPU as well, this all makes things odd since the 
pending node should be running immediately after the node that put the valid 
frame there. So it's hard to imagine what could be corrupting this memory in 
between the put and the dispatch.

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17070): https://lists.fd.io/g/vpp-dev/message/17070
Mute This Topic: https://lists.fd.io/mt/75750436/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] AMD Epyc and vpp.

[Christian Hopps]

Thanks,ChbrisOn Jul 22, 2020, at 11:32 AM, Benoit Ganne (bganne) via lists.fd.io  wrote:I tried setting that but didn't notice an issue, perhaps it's not an IObottleneck.Could you share the output of: - clear run/show run (it is important to clear 1st to capture "live" stats) - show err - show threads - show buffers - show pciWhile traffic is flowing?To give some ideas, a VPP user reported ~65Mpps with a CX5 on Epyc with 8 workers, but YMMV.Bestben
[15:50:04 cmf-tm-3:~]$ vppctl clear run; sleep 11; vppctl show run; vppctl show 
threads; vppctl show cpu; vppctl show buffers ; vppctl show pci
Thread 0 vpp_main (lcore 0)
Time 11.0, 10 sec internal node vector rate 0.00
  vector rates in 0.e0, out 0.e0, drop 0.e0, punt 0.e0
 Name State Calls  Vectors
Suspends  Packet-Clocks   Vectors/Call
api-rx-from-ringany wait 0   0  
 1  1.91e40.00   0
dpdk-processany wait 0   0  
 4  4.84e60.00   0
fib-walkany wait 0   0  
 5  3.49e30.00   0
ikev2-manager-process   any wait 0   0  
11  5.01e30.00   0
ip4-full-reassembly-expire-wal  any wait 0   0  
 1  4.41e30.00   0
ip6-full-reassembly-expire-wal  any wait 0   0  
 1  6.00e30.00   0
ip6-icmp-neighbor-discovery-ev  any wait 0   0  
11  2.89e30.00   0
statseg-collector-process   time wait0   0  
 1  4.46e40.00   0
unix-cli-local:12active  1   0  
 2  3.41e90.00   0
unix-cli-new-sessionany wait 0   0  
 3  2.24e30.00   0
unix-epoll-input polling306904   0  
 0  1.07e50.00   0
---
Thread 1 vpp_wk_0 (lcore 1)
Time 11.0, 10 sec internal node vector rate 1.94
  vector rates in 1.4592e6, out 7.2961e5, drop 0.e0, punt 0.e0
 Name State Calls  Vectors
Suspends  Packet-Clocks   Vectors/Call
dpdk-crypto-inputpolling  24469809   0  
 0  1.19e20.00 100
dpdk-input   polling  24469809 8030066  
 0  1.07e3 .33 100
ethernet-input   active4149970 8030066  
 0  3.25e21.93   0
ip4-input-no-checksumactive4149970 8030066  
 0  1.37e21.93   0
ip4-lookup   active4149970 8030066  
 0  1.33e21.93   0
ip4-midchain active4149970 8030066  
 0  1.52e21.93   0
iptfs-encap4-tun active4149970 8030066  
 0  8.18e21.93 102
iptfs-pacer  polling  24469809 8030067  
 0  7.80e2 .33 100
---
Thread 2 vpp_wk_1 (lcore 2)
Time 11.0, 10 sec internal node vector rate 18.13
  vector rates in 1.4274e6, out 7.1372e5, drop 0.e0, punt 0.e0
 Name State Calls  Vectors
Suspends  Packet-Clocks   Vectors/Call
dpdk-crypto-inputpolling433068 7855124  
 0  3.27e3   18.14 100
dpdk-esp4-decryptactive 433068 7855134  
 0  1.39e2   18.14   0
dpdk-esp4-decrypt-post   active 433068 7855124  
 0  1.06e2   18.14   0
dpdk-input   polling433068 7855134  
 0  1.95e2   18.14 100
ethernet-input   active 433068 7855134  
 0  7.64e1   18.14   0
ip4-input-no-checksumactive 433068 7855134  
 0  5.57e1   18.14   0
ip4-localactive 433068 7855134  
 0  5.00e1   18.14   0
ip4-lookup   active 433068 7855134  
 0 

Re: [vpp-dev] AMD Epyc and vpp.

[Christian Hopps]

I tried that, but

> On Jul 22, 2020, at 1:07 AM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi Christian,
> 
> Everything else being correct (VPP NUMA core placement vs NIC etc) and if you 
> see a bottleneck on IO, you might need to set the NIC as 'Preferred IO' in 
> the BIOS.


I tried setting that but didn't notice an issue, perhaps it's not an IO 
bottleneck. All the IO is to a single PCIe gen3 100G connectx-5 card, in both 
cases (although the aggregate bandwidth over 4 cores is no higher than 40G and 
no higher than 10G per core).

Shuold I change the compiler options? The standard cpu flags seem to be:

  set(CMAKE_C_FLAGS "-march=corei7 -mtune=corei7-avx ${CMAKE_C_FLAGS}")

with variants added to that. I think for Epyc the march/mtune can be set to 
znver2, but I'm not sure how that plays with the avx2 and avx512 variants later 
(avx2 is supported by epyc but not avx512). I tried creating a znver2 variant 
but I think I'm missing something for that to actually work.

Thanks,
Chris.

> 
> Best
> Ben
> 
>> -Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Christian
>> Hopps
>> Sent: mercredi 22 juillet 2020 02:34
>> To: vpp-dev 
>> Cc: Christian Hopps 
>> Subject: [vpp-dev] AMD Epyc and vpp.
>> 
>> Hi vpp-dev,
>> 
>> Has anyone done performance analysis with the new AMD epyc processors and
>> VPP?
>> 
>> Just naively running my normal build shows a 3GHz Epyc machine under-
>> performing a 2.1GHz intel xeon.
>> 
>> Thanks,
>> Chris.
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17036): https://lists.fd.io/g/vpp-dev/message/17036
Mute This Topic: https://lists.fd.io/mt/75716056/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] AMD Epyc and vpp.

[Christian Hopps]

Hi vpp-dev,

Has anyone done performance analysis with the new AMD epyc processors and VPP?

Just naively running my normal build shows a 3GHz Epyc machine under-performing 
a 2.1GHz intel xeon.

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17023): https://lists.fd.io/g/vpp-dev/message/17023
Mute This Topic: https://lists.fd.io/mt/75716056/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Why not break the check flow when the first error occurs of ip4_inout()?

[Christian Hopps]

Shouldn't the version check come first and the code avoid further checks if it 
fails? Probably unlikely to fail though given execution has reached "ip4_input" 
:)

Thanks,
Chris.

> On Jul 21, 2020, at 7:12 AM, Dave Barach via lists.fd.io 
>  wrote:
> 
> Branches hurt performance more than arithmetic and/or conditional moves.
> 
> From: vpp-dev@lists.fd.io   > On Behalf Of "??
> Sent: Monday, July 20, 2020 10:17 PM
> To: vpp-dev mailto:vpp-dev@lists.fd.io>>
> Subject: [vpp-dev] Why not break the check flow when the first error occurs 
> of ip4_inout()?
> 
> Hi all,
> 
> When I'm reading the ip4_input.c code, I'm confused about the following code.
> 
> vnet/vnet/ip/ip4_input.c:ip4_input_inline()
> {
>  ...
> error0 = error1 = IP4_ERROR_NONE;
> 
> /* Punt packets with options. */
> error0 = (ip0->ip_version_and_header_length & 0xf) != 5 ? 
> IP4_ERROR_OPTIONS : error0;
> error1 = (ip1->ip_version_and_header_length & 0xf) != 5 ? 
> IP4_ERROR_OPTIONS : error1;
> 
> /* Version != 4?  Drop it. */
> error0 = (ip0->ip_version_and_header_length >> 4) != 4 ? 
> IP4_ERROR_VERSION : error0;
> error1 = (ip1->ip_version_and_header_length >> 4) != 4 ? 
> IP4_ERROR_VERSION : error1;
> 
> /* Verify header checksum. */
> 
> error0 = 0x != ip_csum_fold (sum0) ? IP4_ERROR_BAD_CHECKSUM : 
> error0;
> error1 = 0x != ip_csum_fold (sum1) ? IP4_ERROR_BAD_CHECKSUM : 
> error1;
> 
> /* Drop fragmentation offset 1 packets. */
> 
> error0 = ip4_get_fragment_offset (ip0) == 1 ? 
> IP4_ERROR_FRAGMENT_OFFSET_ONE : error0;
> error1 = ip4_get_fragment_offset (ip1) == 1 ? 
> IP4_ERROR_FRAGMENT_OFFSET_ONE : error1;
> 
> /* TTL < 1? Drop it. */
> error0 = (ip0->ttl < 1 && cast0 == VNET_UNICAST) ? 
> IP4_ERROR_TIME_EXPIRED : error0;
> error1 = (ip1->ttl < 1 && cast1 == VNET_UNICAST) ? 
> IP4_ERROR_TIME_EXPIRED : error1;
> 
> /* Verify lengths. */
> error0 = ip_len0 < sizeof (ip0[0]) ? IP4_ERROR_TOO_SHORT : error0;
> error1 = ip_len1 < sizeof (ip1[0]) ? IP4_ERROR_TOO_SHORT : error1;
> 
> p0->error = error_node->errors[error0];
> p1->error = error_node->errors[error1];
> 
> if (PREDICT_FALSE(error0 != IP4_ERROR_NONE))
> {
> if (error0 == IP4_ERROR_TIME_EXPIRED) {
> icmp4_error_set_vnet_buffer(p0, ICMP4_time_exceeded,
> ICMP4_time_exceeded_ttl_exceeded_in_transit, 0);
> next0 = IP4_INPUT_NEXT_ICMP_ERROR;
> } else
> next0 = error0 != IP4_ERROR_OPTIONS ? IP4_INPUT_NEXT_DROP 
> : IP4_INPUT_NEXT_PUNT;
> }
> if (PREDICT_FALSE(error1 != IP4_ERROR_NONE))
> {
> if (error1 == IP4_ERROR_TIME_EXPIRED) {
> icmp4_error_set_vnet_buffer(p1, ICMP4_time_exceeded,
> ICMP4_time_exceeded_ttl_exceeded_in_transit, 0);
> next1 = IP4_INPUT_NEXT_ICMP_ERROR;
> } else
> next1 = error1 != IP4_ERROR_OPTIONS ? IP4_INPUT_NEXT_DROP 
> : IP4_INPUT_NEXT_PUNT;
> }
> }
> 
> ---
> 
> 1. why not break the check flow when the first error occurs? for example:
>  error0 = (ip0->ip_version_and_header_length & 0xf) != 5 ? 
> IP4_ERROR_OPTIONS : error0;
>  if (error0 != IP4_ERROR_NONE) {goto error_handle;}
> 
> 2. suppose all the results of checking are wrong, I think at the end, we 
> could only get the last error rather than the first error.
>  options error, version error, ...and length error, so the last error is 
> IP4_ERROR_TOO_SHORT
> 
> Is there any optimization concept behind this？ Can I improe(hope it will 
> improve :-)） the code like the following, before we check the next error, 
> let's check wether the error has occured:
> 
>   /* init error to none */
>   error0 = error1 = IP4_ERROR_NONE;
> 
>   /* check the first error */
>   error0 = (ip0->ip_version_and_header_length & 0xf) != 5 ? IP4_ERROR_OPTIONS 
> : error0;
> 
>   error0 = error0 != IP4_ERROR_NONE? error0 : 
> ((ip0->ip_version_and_header_length >> 4) != 4 ? IP4_ERROR_VERSION : error0);
>   error0 = error0 != IP4_ERROR_NONE? error0 : (0x != ip_csum_fold (sum0) 
> ? IP4_ERROR_BAD_CHECKSUM : error0);
>   ...
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17014): https://lists.fd.io/g/vpp-dev/message/17014
Mute This Topic: https://lists.fd.io/mt/75696670/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: 

[vpp-dev] Regarding new ipsec interface patch

[Christian Hopps]

Hi vpp-dev,

In the interest of expressing the community support that Neale asked for

  https://gerrit.fd.io/r/c/vpp/+/27795

Looks good to me.

It may need some minor additional development to polish some rough edges, but I 
have been able to use it with a few IPTFS specific modifications to implement 
the same route-based functionality that I had in 1908.

As was asked in the last VPP dev meeting, the risk of collateral damage is 
extremely low, and it would be good to get it merged to master now to expose 
this in any case (the changes are minimal outside the new ipsec_itf.[ch] files).

So my recommendation is to merge now to get good soak time prior to 2009 (LTS) 
being cut.

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17000): https://lists.fd.io/g/vpp-dev/message/17000
Mute This Topic: https://lists.fd.io/mt/75679898/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [tsc] [vpp-dev] Replacing master/slave nomenclature

[Christian Hopps]

> On Jul 14, 2020, at 1:20 PM, St Leger, Jim  wrote:
> 
> I believe the DPDK community converged on:
> master/slave lcore -> initial/worker lcore

VPP is ok here I think with "main" and "worker".

> blacklist/whitelist -> blocklist/allowlist

That one feels a bit clunky to me. I wonder why they didn't go for something 
more natural like

  nouns: blocked/allowed
  verbs: block/allow

The terms blacklist/whitelist can be a nouns or verbs, and I suspect they are 
often not implemented as an actual list data structure, so trying to keep the 
"list" suffix seems an unnecessary carryover (and sounds clunky IMHO). :)

Thanks,
Chris.

> 
> Full community discussion: 
> https://mails.dpdk.org/archives/dev/2020-June/thread.html#169337
> 
> Jim
> 
> -Original Message-
> From: vpp-dev@lists.fd.io  On Behalf Of Jerome Tollet 
> via lists.fd.io
> Sent: Tuesday, July 14, 2020 10:10 AM
> To: Chris Luke ; Steven Luong (sluong) 
> ; Dave Barach (dbarach) ; Kinsella, Ray 
> ; Stephen Hemminger ; 
> vpp-dev@lists.fd.io; t...@lists.fd.io; Ed Warnicke (eaw) 
> Subject: Re: [tsc] [vpp-dev] Replacing master/slave nomenclature
> 
> Hi Chris,
> I suspect it would be good to align on the new bond nomenclature coming from 
> other projects. DPDK and Linux are probably starting points we should 
> consider IMO.
> Jerome
> 
> Le 14/07/2020 18:45, « t...@lists.fd.io au nom de Chris Luke » 
>  a écrit :
> 
>It is subjective and contextualized. But in this case, if making the 
> effort to correct a wrong, why stop half way?
> 
>Chris.
> 
>-Original Message-
>From: vpp-dev@lists.fd.io  On Behalf Of Jerome Tollet 
> via lists.fd.io
>Sent: Tuesday, July 14, 2020 12:37
>To: Steven Luong (sluong) ; Dave Barach (dbarach) 
> ; Kinsella, Ray ; Stephen Hemminger 
> ; vpp-dev@lists.fd.io; t...@lists.fd.io; Ed 
> Warnicke (eaw) 
>Subject: [EXTERNAL] Re: [vpp-dev] Replacing master/slave nomenclature
> 
>Hi Steven,
>Please note that per this proposition,  
> https://urldefense.com/v3/__https://lkml.org/lkml/2020/7/4/229__;!!CQl3mcHX2A!QdLdxm4rtZW-mFe5jt_qzEpx-_X2KWnqvjyEl-7Py41jsEV7FrnEw0lTNcF8LdfUzQ$
>  , slave must be avoided but master can be kept.
>Maybe master/member or master/secondary could be options too.
>Jerome
> 
>Le 14/07/2020 18:32, « vpp-dev@lists.fd.io au nom de steven luong via 
> lists.fd.io »  a 
> écrit :
> 
>I am in the process of pushing a patch to replace master/slave with 
> aggregator/member for the bonding.
> 
>Steven
> 
>On 7/13/20, 4:44 AM, "vpp-dev@lists.fd.io on behalf of Dave Barach via 
> lists.fd.io"  
> wrote:
> 
>+1, especially since our next release will be supported for a 
> year, and API name changes are involved...
> 
>-Original Message-
>From: Kinsella, Ray 
>Sent: Monday, July 13, 2020 6:01 AM
>To: Dave Barach (dbarach) ; Stephen Hemminger 
> ; vpp-dev@lists.fd.io; t...@lists.fd.io; Ed 
> Warnicke (eaw) 
>Subject: Re: [vpp-dev] Replacing master/slave nomenclature
> 
>Hi Stephen,
> 
>I agree, I don't think we should ignore this.
>Ed - I suggest we table a discussion at the next FD.io TSC?
> 
>Ray K
> 
>On 09/07/2020 17:05, Dave Barach via lists.fd.io wrote:
>> Looping in the technical steering committee...
>> 
>> -Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Stephen 
>> Hemminger
>> Sent: Thursday, July 2, 2020 7:02 PM
>> To: vpp-dev@lists.fd.io
>> Subject: [vpp-dev] Replacing master/slave nomenclature
>> 
>> Is the VPP project addressing the use of master/slave nomenclature in the 
>> code base, documentation and CLI?  We are doing this for DPDK and it would 
>> be good if the replacement wording used in DPDK matched the wording used in 
>> FD.io projects.
>> 
>> Particularly problematic is the use of master/slave in bonding.
>> This seems to be a leftover from Linux, since none of the commercial 
>> products use that terminology and it is not present in 802.1AX standard.
>> 
>> The IEEE and IETF are doing an across the board look at these terms in 
>> standards.
>> 
>> 
>> 
>> 
> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16958): https://lists.fd.io/g/vpp-dev/message/16958
Mute This Topic: https://lists.fd.io/mt/75503531/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ipsec interface revisted.

[Christian Hopps]

Hi Neale,

Thanks very much for this!

I will try and get this tested ASAP (it will require a bit of work to port to 
master to test).

> For rx have you added a ip-protocol dispatch in the decypt? If so we can 
> upstream that change in this patch if you like.

That is a very interesting idea, and is aligned with our current push in the 
IETF to get the IPTFS next header value allocated from the IP protocols IANA 
registry. :)

Alas, for now I have a custom patch in the v4, v6, check in esp decrypt (vnet 
and dpdk) looking for IPTFS protocol.

For the most part I used a set of callbacks named "tfs" I added to ipsec to 
hook in. In this particular case though it's hard-coded for now.

Thanks,
Chris.

> On Jul 6, 2020, at 4:57 AM, Neale Ranns via lists.fd.io 
>  wrote:
> 
> 
> 
> From: Christian Hopps 
> Date: Friday 26 June 2020 at 12:13
> To: "Neale Ranns (nranns)" 
> Cc: Christian Hopps , vpp-dev 
> Subject: Re: [vpp-dev] ipsec interface revisted.
> 
> 
> 
> 
>> On Jun 26, 2020, at 4:22 AM, Neale Ranns (nranns)  wrote:
>> 
>> Hi Chris,
>> 
>> As far as I'm concerned, it's your plugin, you can add whatever 
>> functionality you need. If you separate the new interface type out into 
>> another plugin, so it can be used without your feature, then the community 
>> will benefit twice.  Let's just make sure we document the whys and hows of 
>> each model.
>> 
>> My preferred outcome though would be to try and find a way for your feature 
>> to work with the existing model. If you'd like to explore those 
>> possibilities perhaps we could discuss code specifics.
> 
> Hi Neale,
> 
> I can try some code specifics.
> 
> A major part of IPTFS is the constructing of the inner ESP payload, and 
> emitting this with specific timing disassociated with the carried user 
> traffic as described previously (and documented in 
> https://tools.ietf.org/html/draft-ietf-ipsecme-iptfs-01).
> 
> I'm going to leave out optimizations that may re-use buffers etc.. to keep 
> things simple..
> 
> This IPTFS machinery receives and terminates the user (protected) packet 
> buffer through the graph. IPTFS encapsulates this user packet into 1 or more 
> IPTFS payloads along with other user traffic to the same SA. It then paces 
> the output of these IPTFS payloads on a disconnected output worker. These 
> IPTFS payloads are passed on to the esp_encrypt for adding ESP and the outer 
> tunnel encap.
> 
> So: e.g.,
> 
> No IPTFS, SA tunnel mode.
>  [IP-A]   [IP-A] 
> [IPsec-IP|ESP|IP-A]
>   worker A: [IP-A] -> ipsec-output -> esp-encrypt [outer-encap added] -> 
> ip-lookup ... -> "if-tx"
> 
>  [IP-B]   [IP-B] 
> [IPsec-IP|ESP|IP-B]
>   worker A: [IP-A] -> ipsec-output -> esp-encrypt [outer-encap added] -> 
> ip-lookup ... -> "if-tx"
> 
>  [IP-C]   [IP-C] 
> [IPsec-IP|ESP|IP-C]
>   worker A: [IP-A] -> ipsec-output -> esp-encrypt [outer-encap added] -> 
> ip-lookup ... -> "if-tx"
> 
> 
> IPTFS:
> 
>   [IP-A]   [IP-A] [TFS|IP-A]
>   worker A: ipsec-output -> iptfs-encap-enq -> encapsualte with other user 
> traffic construct in PACEQ
>   [IP-B]   [IP-B] [TFS|IP-A|IP-B]
>   worker A: ipsec-output -> iptfs-encap-enq -> encapsualte with other user 
> traffic construct in PACEQ
>   [IP-C]   [IP-C] 
> [TFS|IP-A|IP-B|IP-C]
>   worker A: ipsec-output -> iptfs-encap-enq -> encapsualte with other user 
> traffic construct in PACEQ
>  [TFS|IP-A|IP-B|IP-C]
>   worker A: PACEQ -> iptfs-pacer (timed) enqueue to OUTQ
> 
> 
>[TFS|IP-A|IP-B|IP-C]   [TFS|IP-A|IP-B|IP-C]
>   [IPsec-IP|ESP|TFS|IP-A|IP-B|IP-C]
>   worker B: OUTQ -> iptfs_output (timed) -> esp-encrypt [outer-encap added] 
> ->  ip-lookup ... -> "if-tx"
> 
> 
> Code Details (where does IPTFS hook):
> 
> SA Policy Directed
> 
> in ipsec_output_inline
> 
>   if (p0->policy == IPSEC_POLICY_ACTION_PROTECT)
> { ...
>   if (sa->protocol == IPSEC_PROTOCOL_ESP)
> {
>   if (ipsec_sa_is_IPTFS (sa))
> next_node_index = im->tfs_encap_node_index;
>   else if (is_ipv6)
> next_node_index = im->esp6_encrypt_node_index;
>   else
> next_node_index = im->esp4_encrypt_node_index;
> 
> Interface Directed
> 
>

Re: [vpp-dev] name filter fix in 1908

[Christian Hopps]

https://gerrit.fd.io/r/c/vpp/+/27761

> On Jun 30, 2020, at 11:23 AM, Andrew Yourtchenko  wrote:
> 
> Hi Christian,
> 
> Ok so given it’s a fix, feel free to cherry-pick into 19.08 without the API 
> version change, I will merge it.
> 
> --a
> 
>> On 30 Jun 2020, at 15:52, Ole Troan  wrote:
>> 
>> Hi Christian,
>> 
>> Changing the patch is fine.
>> The only use of version apart from human indications is for the is api 
>> experimental or not by checkstyle-api
>> Cheers
>> Ole
>> 
>>> On 30 Jun 2020, at 15:38, Christian Hopps  wrote:
>>> 
>>> I noticed that it was the "patch" version number that changed. Semver 
>>> claims the patch number indicates backwards compatible changes -- not sure 
>>> how VPP API is using the "patch" number though. So when I resolved the 
>>> conflict locally after cherry picking I just changed 3.1.1 to 3.1.2 to keep 
>>> the spirit of the change. Reverting the version change upstream is another 
>>> solution too, if that helps with the API crc/signature stuff. I don't fully 
>>> comprehended the VPP API versioning, so better someone else makes that call.
>>> 
>>> Thanks,
>>> Chris.
>>> 
>>>>> On Jun 30, 2020, at 8:04 AM, Dave Barach (dbarach)  
>>>>> wrote:
>>>> 
>>>> Dear Chris,
>>>> 
>>>> In looking at the patch, I have a question: the API version number changed 
>>>> despite the fact that the API itself was unchanged.
>>>> 
>>>> Should we revert the API version number bump and then cherry-pick to 19.08?
>>>> 
>>>> Looping in Ole for an opinion...
>>>> 
>>>> Thanks... Dave
>>>> 
>>>> -Original Message-
>>>> From: vpp-dev@lists.fd.io  On Behalf Of Christian 
>>>> Hopps
>>>> Sent: Tuesday, June 30, 2020 3:10 AM
>>>> To: vpp-dev 
>>>> Cc: Christian Hopps 
>>>> Subject: [vpp-dev] name filter fix in 1908
>>>> 
>>>> Could this fix: https://gerrit.fd.io/r/c/vpp/+/23140
>>>> 
>>>> be pulled into stable/1908?
>>>> 
>>>> It applies clean after adapting the version number change to be compatible 
>>>> with 1908 branch.
>>>> 
>>>> Thanks,
>>>> Chris.
>>>> 
>>> 
>> 
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16881): https://lists.fd.io/g/vpp-dev/message/16881
Mute This Topic: https://lists.fd.io/mt/75209217/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Strongswan VPP integration.

[Christian Hopps]

Hi folks,

I adapted some github public changes made a couple years back for VPP 18.04 to 
finish them off and bring them to 19.08 (20.01, 20.05). However, in order to 
contribute these back to Strongswan they would like an MIT license. There was 
no license on the new files in github, however, they started (at least 
template-wise) from existing GPLd files in Strongswan. These original files the 
strongswan maintainers have a right to license differently which is why they 
ask for MIT license for new contributions even if originally from their GPLd 
files.

Anyway, the original changes were developed by Matus Fabian (while he was 
working at Pantheon Tech I believe), so could someone put me in touch with 
someone at Pantheon that might be able to help resolve this issue for me?

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16871): https://lists.fd.io/g/vpp-dev/message/16871
Mute This Topic: https://lists.fd.io/mt/75261689/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] name filter fix in 1908

[Christian Hopps]

I noticed that it was the "patch" version number that changed. Semver claims 
the patch number indicates backwards compatible changes -- not sure how VPP API 
is using the "patch" number though. So when I resolved the conflict locally 
after cherry picking I just changed 3.1.1 to 3.1.2 to keep the spirit of the 
change. Reverting the version change upstream is another solution too, if that 
helps with the API crc/signature stuff. I don't fully comprehended the VPP API 
versioning, so better someone else makes that call.

Thanks,
Chris.

> On Jun 30, 2020, at 8:04 AM, Dave Barach (dbarach)  wrote:
> 
> Dear Chris,
> 
> In looking at the patch, I have a question: the API version number changed 
> despite the fact that the API itself was unchanged.
> 
> Should we revert the API version number bump and then cherry-pick to 19.08?
> 
> Looping in Ole for an opinion...
> 
> Thanks... Dave
> 
> -Original Message-
> From: vpp-dev@lists.fd.io  On Behalf Of Christian Hopps
> Sent: Tuesday, June 30, 2020 3:10 AM
> To: vpp-dev 
> Cc: Christian Hopps 
> Subject: [vpp-dev] name filter fix in 1908
> 
> Could this fix: https://gerrit.fd.io/r/c/vpp/+/23140
> 
> be pulled into stable/1908?
> 
> It applies clean after adapting the version number change to be compatible 
> with 1908 branch.
> 
> Thanks,
> Chris.
> 



signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16846): https://lists.fd.io/g/vpp-dev/message/16846
Mute This Topic: https://lists.fd.io/mt/75209217/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] name filter fix in 1908

[Christian Hopps]

Could this fix: https://gerrit.fd.io/r/c/vpp/+/23140

be pulled into stable/1908?

It applies clean after adapting the version number change to be compatible with 
1908 branch.

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16842): https://lists.fd.io/g/vpp-dev/message/16842
Mute This Topic: https://lists.fd.io/mt/75209217/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ipsec interface revisted.

[Christian Hopps]

> On Jun 26, 2020, at 4:22 AM, Neale Ranns (nranns)  wrote:
> 
> Hi Chris, 
> 
> As far as I'm concerned, it's your plugin, you can add whatever functionality 
> you need. If you separate the new interface type out into another plugin, so 
> it can be used without your feature, then the community will benefit twice.  
> Let's just make sure we document the whys and hows of each model.
> 
> My preferred outcome though would be to try and find a way for your feature 
> to work with the existing model. If you'd like to explore those possibilities 
> perhaps we could discuss code specifics. 

Hi Neale,

I can try some code specifics.

A major part of IPTFS is the constructing of the inner ESP payload, and 
emitting this with specific timing disassociated with the carried user traffic 
as described previously (and documented in 
https://tools.ietf.org/html/draft-ietf-ipsecme-iptfs-01).

I'm going to leave out optimizations that may re-use buffers etc.. to keep 
things simple..

This IPTFS machinery receives and terminates the user (protected) packet buffer 
through the graph. IPTFS encapsulates this user packet into 1 or more IPTFS 
payloads along with other user traffic to the same SA. It then paces the output 
of these IPTFS payloads on a disconnected output worker. These IPTFS payloads 
are passed on to the esp_encrypt for adding ESP and the outer tunnel encap.

So: e.g., 

No IPTFS, SA tunnel mode.
 [IP-A]   [IP-A] 
[IPsec-IP|ESP|IP-A] 
  worker A: [IP-A] -> ipsec-output -> esp-encrypt [outer-encap added] -> 
ip-lookup ... -> "if-tx"

 [IP-B]   [IP-B] 
[IPsec-IP|ESP|IP-B] 
  worker A: [IP-A] -> ipsec-output -> esp-encrypt [outer-encap added] -> 
ip-lookup ... -> "if-tx"

 [IP-C]   [IP-C] 
[IPsec-IP|ESP|IP-C] 
  worker A: [IP-A] -> ipsec-output -> esp-encrypt [outer-encap added] -> 
ip-lookup ... -> "if-tx"


IPTFS:

  [IP-A]   [IP-A] [TFS|IP-A] 
  worker A: ipsec-output -> iptfs-encap-enq -> encapsualte with other user 
traffic construct in PACEQ
  [IP-B]   [IP-B] [TFS|IP-A|IP-B] 
  worker A: ipsec-output -> iptfs-encap-enq -> encapsualte with other user 
traffic construct in PACEQ
  [IP-C]   [IP-C] 
[TFS|IP-A|IP-B|IP-C]
  worker A: ipsec-output -> iptfs-encap-enq -> encapsualte with other user 
traffic construct in PACEQ
 [TFS|IP-A|IP-B|IP-C]
  worker A: PACEQ -> iptfs-pacer (timed) enqueue to OUTQ
   
  
   [TFS|IP-A|IP-B|IP-C]   [TFS|IP-A|IP-B|IP-C]  
[IPsec-IP|ESP|TFS|IP-A|IP-B|IP-C]
  worker B: OUTQ -> iptfs_output (timed) -> esp-encrypt [outer-encap added] ->  
ip-lookup ... -> "if-tx"


Code Details (where does IPTFS hook):

SA Policy Directed

in ipsec_output_inline

  if (p0->policy == IPSEC_POLICY_ACTION_PROTECT)
{ ...
  if (sa->protocol == IPSEC_PROTOCOL_ESP)
{
  if (ipsec_sa_is_IPTFS (sa))
next_node_index = im->tfs_encap_node_index;
  else if (is_ipv6)
next_node_index = im->esp6_encrypt_node_index;
  else
next_node_index = im->esp4_encrypt_node_index;

Interface Directed

  [iptfs init code]

  ipsec_add_feature ("ip4-output", "iptfs-encap4-tun",
 >encap4_tun_feature_index);
 
  [tunnel create code]

  In ipsec_tunnel_feature_set

  Instead of:

  arc = vnet_get_feature_arc_index ("ip4-output");
  vnet_feature_enable_disable_with_index (arc, esp4_feature_index,
  t->sw_if_index, enable,
  >output_sa_index,
  sizeof (t->output_sa_index));

  A callback is made to the TFS code which does:

  arc = vnet_get_feature_arc_index ("ip4-output");
  vnet_feature_enable_disable_with_index (
  arc, tfsm->encap4_tun_feature_index, t->sw_if_index, enable,
  >output_sa_index, sizeof (t->output_sa_index));

This latter Interface Directed code is what has been removed from VPP. The 
packets I receive on this path were not not (and should not) be already 
pre-encapsulated with any outer tunnel IP header, or I'm going to have to 
immediately remove this encap before placing the user traffic in the the TFS 
payload (see above). Having the overhead of adding an IP header, then 
immediately removing it simply to get traffic directed to my IPTFS encap 
routine is not a reasonable replacement.

So the "new model" in VPP is that I have to add a non-encapsulating interface 
to direct the user traffic to me for TFS processing (i.e., unencapsulated). 
This interface is going to bear a striking resemblance to the one that was just 
removed from 19.08 VPP. :) 

Thanks,

Re: [vpp-dev] ipsec interface revisted.

[Christian Hopps]

Hi Neale,

It's maybe worth pointing this out: using policy based IPsec continues to work 
fine for me. What I had and lost is route based IPsec, i.e., a destination 
interface that directs traffic to an SA *without trying to 
"partially-implement" the tunnel mode SA functionality*.

The new code is making the assumption that a tunnel mode SA's encap function 
can be 1-1 replaced by an upstream IPIP encapsulating function. This assumption 
is what is breaking me b/c my ESP payload is not a simple single IP packet.

Each of my ESP payloads is an aggregate of IP packets sometimes padded 
sometimes fractional. These payloads are emitted on a fixed timed interval 
(which determines if there is padding or not) and encapsulated only upon being 
emitted. *Specifically* this cannot be driven by the users (inner) traffic or 
we've failed -- all of this is the function the IPTFS SA implements. All of 
this is coded and working for policy directed tunnel mode SAs. Having to 
somehow replicate this functionality in two places (one place for policy based, 
and then in some new tightly coupled encapsulating custom interface type) to 
support the vpp post 19.08 optimization couldn't be seen as being "cleaner" by 
any subjective measure I hope.

So, in order to make things work post 19.08 I'm going to have to (re-)introduce 
a new "iptfs" interface type to VPP, so that I can have traffic directed to my 
SA w/o any encapsulation being done beforehand. If the goal was to avoid 
requiring per-use case interface types I don't think it worked for me. 
Basically I'm going to have to put back what was removed, maybe with a few 
refinements since I'm there already. Will that be acceptable when I eventually 
try and upstream the code? There's also the distinct possibility that we can't 
get this extra work funded in which case it would then only be available 
externally based on 19.08 which would be a shame as this work is based on the 
what is being done in ipsecme WG of the IETF. I would guess this would be seen 
as a poor outcome by others too.

FWIW the GRE use-case was, and remains, a transport mode use of IPsec, so no 
extra assumptions are being made for that use case. The new code probably is an 
improvement for GRE with transport mode IPsec, but for the purpose of 
discussing IPsec tunnel mode functionality it's a bit distracting.

Thanks,
Chris.

> On Jun 23, 2020, at 8:51 AM, Neale Ranns (nranns)  wrote:
> 
> 
> Hi Chris,
> 
> On 22/06/2020 13:09, "Christian Hopps"  wrote:
> 
>> 
>> - It operates directly with the IPsec tunnel mode and transport mode SAs 
>> without needing to mangle the internal definition of SA tunnel into 
>> transport mode.
> 
>Do you have any comments on this point? This is what I was talking about 
> when I said a "cleaner abstraction".
> 
> 'cleaner' is a subjective term and subjective analysis is subject to bias; my 
> biases are clear. That's why I tried to frame this discussion with an 
> objective comparison of use case. First in what we can't do with ipip/gre 
> then in what we 'can't' do with xfrms.  More later..
> 
>> - CRITICAL: Its use does not impose any encapsulation on the traffic itself.
>> 
>> Critical implies that without this some use cases are impossible. What is it 
>> that cannot be done with the tunnel model?
> 
>- You can't tunnel non-singular IP traffic with IPIP (my use case requires 
> this) IOW "ipip-only" model assumes the ESP payload is a single IP packet 
> that will be wrapped with an outer IP packet.
> 
> More on this below...
> 
>- You can't do route based transport mode IPsec using ipip (my use case 
> doesn't require this).
> 
> That's true, but is that a use case? Route based VPNs apply to transit 
> traffic, as well as locally generated, and one shouldn't send transit traffic 
> using a transport mode SA, i.e. without encap that ensures it returns.
> 
>Both of these speak to the "more powerful abstraction" point.
> 
>- The xfrm interface allows for route-based selection, but also allows for 
> additional policy restrictions. This might be possible with IPIP model, but 
> again it doesn't seem as clean/obvious.
> 
> VPP supports policy based VPNs using an SPD attached to an interface. Any 
> interface type could be used.
> 
>There's a reason that linux went with this approach as their improvement 
> on the VTI interface, rather than just re-using IPIP tunnel after 
> re-interpreting tunnel mode SAs into transport mode SAs like the new VPP code 
> does. The latter makes assumptions about the IPsec tunnel mode payload and 
> generally feels like a restricted "works for me/good enough" solution, rather 
> than a clean abstraction one can build new use cases on (as I am trying to 

Re: [vpp-dev] ipsec interface revisted.

[Christian Hopps]

> On Jun 22, 2020, at 4:11 AM, Neale Ranns via lists.fd.io 
>  wrote:
> 
>
>
> From:  on behalf of Christian Hopps 
> Date: Thursday 18 June 2020 at 18:20
> To: vpp-dev 
> Cc: Christian Hopps 
> Subject: [vpp-dev] ipsec interface revisted.
>
> Hi,
> 
> So to revisit this topic from a different angle. I believe VPP needs 
> something like the xfrm linux interface [1]. If I understand things correctly 
> this actually provides what was useful (but more-so) with old ipsec interface 
> functionality that has been lost. It is also a much cleaner/more powerful 
> abstraction than the current "ipip tunnel with transport mode SA" trick that 
> replaced the old ipsec interface functionality.
>
> As you say, replaced by ipip/gre tunnel interface, which shows the 
> functionality has not been lost. Route based VPNs are still supported.
> 
> 
> The idea is to have an interface that one can use as a result of a FIB 
> lookup. SAs can be attached to this interface. The replacement for the old 
> ipsec interface functionality that was removed after 19.08, is that you 
> attach a simple *tunnel mode* SA with "accept all" policy to the xfrm 
> interface. You can also attach more complex policies if you care to, but the 
> common and highly efficient case will be "accept all".
> 
> The win here is that:
> 
>  - FIB Fast: You get an interface that can be the result of the forwarding 
> lookup
>   . highly efficient especially with common zero cost match all policy
>   . compare to adding complex ip policy directly to all your ingress 
> interfaces (expensive).
>  - You have one interface for both IPv4 and IPv6
>
> All of which are available with the ipip/gre tunnel model.

Yes.

> 
>  - It operates directly with the IPsec tunnel mode and transport mode SAs 
> without needing to mangle the internal definition of SA tunnel into transport 
> mode.

Do you have any comments on this point? This is what I was talking about when I 
said a "cleaner abstraction".

>  - CRITICAL: Its use does not impose any encapsulation on the traffic itself.
>
> Critical implies that without this some use cases are impossible. What is it 
> that cannot be done with the tunnel model?

- You can't tunnel non-singular IP traffic with IPIP (my use case requires 
this) IOW "ipip-only" model assumes the ESP payload is a single IP packet that 
will be wrapped with an outer IP packet.

- You can't do route based transport mode IPsec using ipip (my use case doesn't 
require this).

Both of these speak to the "more powerful abstraction" point.

- The xfrm interface allows for route-based selection, but also allows for 
additional policy restrictions. This might be possible with IPIP model, but 
again it doesn't seem as clean/obvious.

There's a reason that linux went with this approach as their improvement on the 
VTI interface, rather than just re-using IPIP tunnel after re-interpreting 
tunnel mode SAs into transport mode SAs like the new VPP code does. The latter 
makes assumptions about the IPsec tunnel mode payload and generally feels like 
a restricted "works for me/good enough" solution, rather than a clean 
abstraction one can build new use cases on (as I am trying to do).

In my case I want to insert myself into a place in the graph that appears to no 
longer exist. IPTFS is a tunnel that carriers multiple or fractional inner IP 
packets per outer IP packet -- this simply does not map to an IPIP interface, 
where it would easily map to an XFRM style interface. So it definitely is a 
loss of functionality.

Additionally I'm worried there may be even newer changes being made in VPP that 
are making further assumptions (restrictions) based on the new "ipip-only" 
model (something to do with rewrites/adjacencies maybe? -- my understanding 
here is minimal). It's been hard to justify, time-wise, tracking changes in 
code that seems engineered to not work for me. :)

Thanks,
Chris.

>
> /neale
> 
> 
> Thoughts?
> 
> Thanks,
> Chris.
> 
> [1] - 
> https://wiki.strongswan.org/projects/strongswan/wiki/RouteBasedVPN#XFRM-Interfaces-on-Linux
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16780): https://lists.fd.io/g/vpp-dev/message/16780
Mute This Topic: https://lists.fd.io/mt/74962223/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] ipsec interface revisted.

[Christian Hopps]

Hi,

So to revisit this topic from a different angle. I believe VPP needs something 
like the xfrm linux interface [1]. If I understand things correctly this 
actually provides what was useful (but more-so) with old ipsec interface 
functionality that has been lost. It is also a much cleaner/more powerful 
abstraction than the current "ipip tunnel with transport mode SA" trick that 
replaced the old ipsec interface functionality.

The idea is to have an interface that one can use as a result of a FIB lookup. 
SAs can be attached to this interface. The replacement for the old ipsec 
interface functionality that was removed after 19.08, is that you attach a 
simple *tunnel mode* SA with "accept all" policy to the xfrm interface. You can 
also attach more complex policies if you care to, but the common and highly 
efficient case will be "accept all".

The win here is that:

 - FIB Fast: You get an interface that can be the result of the forwarding 
lookup
  . highly efficient especially with common zero cost match all policy
  . compare to adding complex ip policy directly to all your ingress interfaces 
(expensive).
 - You have one interface for both IPv4 and IPv6
 - It operates directly with the IPsec tunnel mode and transport mode SAs 
without needing to mangle the internal definition of SA tunnel into transport 
mode.
 - CRITICAL: Its use does not impose any encapsulation on the traffic itself.

Thoughts?

Thanks,
Chris.

[1] - 
https://wiki.strongswan.org/projects/strongswan/wiki/RouteBasedVPN#XFRM-Interfaces-on-Linux-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16754): https://lists.fd.io/g/vpp-dev/message/16754
Mute This Topic: https://lists.fd.io/mt/74962223/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

> On May 11, 2020, at 11:03 AM, Christian Hopps  wrote:
> 
>> On May 11, 2020, at 9:56 AM, Neale Ranns (nranns)  wrote:
>> 
>> On 11/05/2020 14:28, "Christian Hopps"  wrote:
> 
>> 
>>   Is it *really* that big a deal to have a logical interface represent a 
>> tunnel mode SA? It actually seems a fairly elegant to me. Instead of tossing 
>> it out, why not just clean up the objectionable API pieces? The current 
>> changes can continue to be used for the GRE tunnel case, so the current work 
>> is not lost either.
>> 
>> We seem to have come full circle... there is a logical interface, it's 
>> called ipipX. Clearly the name of the interface is not important.
>> The logical interface shouldn't represent the SA, it should represent the 
>> peering, and it's the peering that defines the encap. I say this because SAs 
>> come and go as rekeying occurs, but the peering remains. You don't want to 
>> delete your interface and recreate all your routes, each time you rekey. In 
>> fact there's a good test; if the peering addresses can 
>> regularly/reasonably/etc change during a rekey, then the ipip case is 
>> definitely worse, since one would need to create a new ipip tunnel and IMO 
>> that would justify a different interface type.
>> 
>> What would be preferable/more efficient for your feature is if the encap 
>> were applied after your feature is run, but do I really need a new interface 
>> type just for that? let me see what I can do.

Do you think a fix will available before the next LTS release?

Thanks,
Chris.

> 
> Fantastic. :)
> 
> Thanks,
> Chris.
> 
>> 
>> /neale
>> 
>> 
>>   Thanks,
>>   Chris.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16751): https://lists.fd.io/g/vpp-dev/message/16751
Mute This Topic: https://lists.fd.io/mt/74027328/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Interesting backtrace in 1908

[Christian Hopps]

FWIW, I cherry picked this w/o looking too close and hit the bug again, looked 
closer, this is not the fix. :)

The change for me is to the stack size for startup config node in 
src/vlib/unix/main.c

Thanks,
Chris.

> On Jun 6, 2020, at 4:34 AM, Andrew Yourtchenko  wrote:
> 
> https://gerrit.fd.io/r/c/vpp/+/27457
> 
> It was in that bunch of “catch up cherrypicks” that I was working on last 
> week.
> 
> --a
> 
>> On 5 Jun 2020, at 23:36, Dave Barach via lists.fd.io 
>>  wrote:
>> 
>> Dear Chris,
>> 
>> Whew, that just made my weekend a lot happier. 
>> 
>> I'll look into why the relevant patch didn't make it back into 19.08 - it 
>> will now! - unfortunately "stuff happens..."
>> 
>> Thanks for confirming... Dave
>> 
>> -Original Message-----
>> From: Christian Hopps  
>> Sent: Friday, June 5, 2020 5:09 PM
>> To: Dave Barach (dbarach) 
>> Cc: Christian Hopps ; vpp-dev 
>> Subject: Re: [vpp-dev] Interesting backtrace in 1908
>> 
>> Bingo.
>> 
>> In fact in 19.08 the value is left as 0 which defaults to 15. I took it from 
>> 20 down to 15, starting successfully until I reached 15 which then hit the 
>> problem (both with the arp path and the other).
>> 
>> Thanks for the help finding this!
>> 
>> Chris.
>> 
>>> On Jun 5, 2020, at 4:52 PM, Dave Barach via lists.fd.io 
>>>  wrote:
>>> 
>>> Hmmm. That begins to smell like an undetected stack overflow. To test that 
>>> theory: s/18/20/ below: 
>>> 
>>> /* *INDENT-OFF* */
>>> VLIB_REGISTER_NODE (startup_config_node,static) = {
>>>   .function = startup_config_process,
>>>   .type = VLIB_NODE_TYPE_PROCESS,
>>>   .name = "startup-config-process",
>>>   .process_log2_n_stack_bytes = 18,
>>> };
>>> /* *INDENT-ON* */
>>> 
>>> It's entirely possible that compiling -O0 blows the stack, especially if 
>>> you end up 75 miles deep in fib code.
>>> 
>>> Dave
>>> 
>>> -Original Message-
>>> From: Christian Hopps 
>>> Sent: Friday, June 5, 2020 4:28 PM
>>> To: Dave Barach (dbarach) 
>>> Cc: Christian Hopps ; vpp-dev 
>>> Subject: Re: [vpp-dev] Interesting backtrace in 1908
>>> 
>>> 
>>> 
>>>> On Jun 5, 2020, at 2:10 PM, Dave Barach via lists.fd.io 
>>>>  wrote:
>>>> 
>>>> Step 1 is to make the silly-looking sibling recursion in 
>>>> vlib_node_add_next_with_slot(...) disappear. I’m on it...
>>>> 
>>>> Just to ask, can you repro w/ master/latest?
>>> 
>>> I will try and do this.
>>> 
>>> In the meantime I moved the arp configs to later in my startup config (this 
>>> is actually built by a test script) and immediately hit another sigsegv in 
>>> startup. This one is in infra but is going through my code initialization, 
>>> but also rooted in startup config processing... It's also in memcpy code, 
>>> which is making me suspicious now.
>>> 
>>> Again, I've changed "-O2" to "-O0" in the cmake vpp.mk package, when I 
>>> change it back to -O2 I do not hit either bug. So I'm now wondering if 
>>> there is something wrong with doing this, like do I need to do something 
>>> else as well?
>>> 
>>> What I'm going for is not to have CLIB_DEBUG defined, but still have useful 
>>> levels of debugabillity to do RCA on a much (millions of packets) later 
>>> problem I have.
>>> 
>>> modified   build-data/platforms/vpp.mk
>>> @@ -35,13 +35,21 @@ vpp_debug_TAG_CFLAGS = -O0 -DCLIB_DEBUG 
>>> $(vpp_common_cflags)  vpp_debug_TAG_CXXFLAGS = -O0 -DCLIB_DEBUG 
>>> $(vpp_common_cflags)  vpp_debug_TAG_LDFLAGS = -O0 -DCLIB_DEBUG 
>>> $(vpp_common_cflags)
>>> 
>>> -vpp_TAG_CFLAGS = -O2 $(vpp_common_cflags) -vpp_TAG_CXXFLAGS = -O2 
>>> $(vpp_common_cflags) -vpp_TAG_LDFLAGS = -O2 $(vpp_common_cflags) -pie
>>> +vpp_TAG_CFLAGS = -O0 $(vpp_common_cflags) vpp_TAG_CXXFLAGS = -O0
>>> +$(vpp_common_cflags) vpp_TAG_LDFLAGS = -O0 $(vpp_common_cflags) -pie
>>> 
>>> The new backtrace I'm seeing is:
>>> 
>>> Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
>>> 0x75cbe8b1 in clib_mov16 (dst=0x6d986a57 >> access memory at address 0x6d986a57>, src=0x2d8f076c2d08 
>>> ) at 
>>> /var/build/vpp/src/vppinfra/memcpy_sse3.h:56
>>> 
>>> (gdb

Re: [vpp-dev] Interesting backtrace in 1908

[Christian Hopps]

Bingo.

In fact in 19.08 the value is left as 0 which defaults to 15. I took it from 20 
down to 15, starting successfully until I reached 15 which then hit the problem 
(both with the arp path and the other).

Thanks for the help finding this!

Chris.

> On Jun 5, 2020, at 4:52 PM, Dave Barach via lists.fd.io 
>  wrote:
> 
> Hmmm. That begins to smell like an undetected stack overflow. To test that 
> theory: s/18/20/ below: 
> 
> /* *INDENT-OFF* */
> VLIB_REGISTER_NODE (startup_config_node,static) = {
>.function = startup_config_process,
>.type = VLIB_NODE_TYPE_PROCESS,
>.name = "startup-config-process",
>.process_log2_n_stack_bytes = 18,
> };
> /* *INDENT-ON* */
> 
> It's entirely possible that compiling -O0 blows the stack, especially if you 
> end up 75 miles deep in fib code.
> 
> Dave
> 
> -Original Message-
> From: Christian Hopps  
> Sent: Friday, June 5, 2020 4:28 PM
> To: Dave Barach (dbarach) 
> Cc: Christian Hopps ; vpp-dev 
> Subject: Re: [vpp-dev] Interesting backtrace in 1908
> 
> 
> 
>> On Jun 5, 2020, at 2:10 PM, Dave Barach via lists.fd.io 
>>  wrote:
>> 
>> Step 1 is to make the silly-looking sibling recursion in 
>> vlib_node_add_next_with_slot(...) disappear. I’m on it...
>> 
>> Just to ask, can you repro w/ master/latest?
> 
> I will try and do this.
> 
> In the meantime I moved the arp configs to later in my startup config (this 
> is actually built by a test script) and immediately hit another sigsegv in 
> startup. This one is in infra but is going through my code initialization, 
> but also rooted in startup config processing... It's also in memcpy code, 
> which is making me suspicious now.
> 
> Again, I've changed "-O2" to "-O0" in the cmake vpp.mk package, when I change 
> it back to -O2 I do not hit either bug. So I'm now wondering if there is 
> something wrong with doing this, like do I need to do something else as well?
> 
> What I'm going for is not to have CLIB_DEBUG defined, but still have useful 
> levels of debugabillity to do RCA on a much (millions of packets) later 
> problem I have.
> 
> modified   build-data/platforms/vpp.mk
> @@ -35,13 +35,21 @@ vpp_debug_TAG_CFLAGS = -O0 -DCLIB_DEBUG 
> $(vpp_common_cflags)  vpp_debug_TAG_CXXFLAGS = -O0 -DCLIB_DEBUG 
> $(vpp_common_cflags)  vpp_debug_TAG_LDFLAGS = -O0 -DCLIB_DEBUG 
> $(vpp_common_cflags)
> 
> -vpp_TAG_CFLAGS = -O2 $(vpp_common_cflags) -vpp_TAG_CXXFLAGS = -O2 
> $(vpp_common_cflags) -vpp_TAG_LDFLAGS = -O2 $(vpp_common_cflags) -pie
> +vpp_TAG_CFLAGS = -O0 $(vpp_common_cflags) vpp_TAG_CXXFLAGS = -O0 
> +$(vpp_common_cflags) vpp_TAG_LDFLAGS = -O0 $(vpp_common_cflags) -pie
> 
> The new backtrace I'm seeing is:
> 
> Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
> 0x75cbe8b1 in clib_mov16 (dst=0x6d986a57  memory at address 0x6d986a57>, src=0x2d8f076c2d08  access memory at address 0x2d8f076c2d08>) at 
> /var/build/vpp/src/vppinfra/memcpy_sse3.h:56
> 
> (gdb) bt
> #0  0x75cbe8b1 in clib_mov16 (dst=0x6d986a57  access memory at address 0x6d986a57>, src=0x2d8f076c2d08  Cannot access memory at address 0x2d8f076c2d08>) at 
> /var/build/vpp/src/vppinfra/memcpy_sse3.h:56
> #1  0x75cbe910 in clib_mov32 (dst=0x7fffb90bea90 "", 
> src=0x7fffafa01fd0 "iptfs-refill-zpool sa_index %d before %d requested %d 
> head %d tail %d") at /var/build/vpp/src/vppinfra/memcpy_sse3.h:66
> #2  0x75cbe951 in clib_mov64 (dst=0x7fffb90bea90 "", 
> src=0x7fffafa01fd0 "iptfs-refill-zpool sa_index %d before %d requested %d 
> head %d tail %d") at /var/build/vpp/src/vppinfra/memcpy_sse3.h:73
> #3  0x75cbed5a in clib_memcpy_fast (dst=0x7fffb90bea90, 
> src=0x7fffafa01fd0, n=5) at /var/build/vpp/src/vppinfra/memcpy_sse3.h:273
> #4  0x75cc5e72 in do_percent (_s=0x7fffb8141a58, fmt=0x75dc7734 
> "%s%c", va=0x7fffb8141bc8) at /var/build/vpp/src/vppinfra/format.c:341
> #5  0x75cc6564 in va_format (s=0x0, fmt=0x75dc7734 "%s%c", 
> va=0x7fffb8141bc8) at /var/build/vpp/src/vppinfra/format.c:404
> #6  0x75cc6810 in format (s=0x0, fmt=0x75dc7734 "%s%c") at 
> /var/build/vpp/src/vppinfra/format.c:428
> #7  0x75cace7d in elog_event_type_register (em=0x7656c7a8 
> , t=0x7fffb9058300) at 
> /var/build/vpp/src/vppinfra/elog.c:173
> #8  0x7fffaf9b8ba4 in elog_event_data_inline (cpu_time=3193258542505306, 
> track=0x7fffb9093f98, type=0x7fffafc08880 , em=0x7656c7a8 
> ) at /var/build/vpp/src/vppinfra/elog.h:310
> #9  elog_data_inline (track=0x7fffb9093f98, type=0x7fffafc08880 , 
>

Re: [vpp-dev] Interesting backtrace in 1908

[Christian Hopps]

;, fp=0x76fea017 
) at 
/var/build/vpp/src/vlibmemory/vlib_api.c:571
#18 vl_api_rpc_call_main_thread (fp=0x76fea017 
, data=0x7fffb8145480 "\001", 
data_length=600) at /var/build/vpp/src/vlibmemory/vlib_api.c:602
#19 0x76fea06a in ipsec_add_del_tunnel_if (args=0x7fffb8145480) at 
/var/build/vpp/src/vnet/ipsec/ipsec_if.c:216
#20 0x76fde020 in create_ipsec_tunnel_command_fn (vm=0x7656c400 
, input=0x7fffb8146ed0, cmd=0x7fffb82e29a8) at 
/var/build/vpp/src/vnet/ipsec/ipsec_cli.c:857
#21 0x762385f2 in vlib_cli_dispatch_sub_commands (vm=0x7656c400 
, cm=0x7656c630 , 
input=0x7fffb8146ed0, parent_command_index=772) at 
/var/build/vpp/src/vlib/cli.c:649
#22 0x762383a3 in vlib_cli_dispatch_sub_commands (vm=0x7656c400 
, cm=0x7656c630 , 
input=0x7fffb8146ed0, parent_command_index=10) at 
/var/build/vpp/src/vlib/cli.c:609
#23 0x762383a3 in vlib_cli_dispatch_sub_commands (vm=0x7656c400 
, cm=0x7656c630 , 
input=0x7fffb8146ed0, parent_command_index=0) at 
/var/build/vpp/src/vlib/cli.c:609
#24 0x762391c3 in vlib_cli_input (vm=0x7656c400 , 
input=0x7fffb8146ed0, function=0x0, function_arg=0) at 
/var/build/vpp/src/vlib/cli.c:750
#25 0x76344666 in startup_config_process (vm=0x7656c400 
, rt=0x7fffb813e000, f=0x0) at 
/var/build/vpp/src/vlib/unix/main.c:367
#26 0x76297fa0 in vlib_process_bootstrap (_a=140736274672448) at 
/var/build/vpp/src/vlib/main.c:1472
#27 0x75ce23d8 in clib_calljmp () from 
target:/var/build/vpp/build-root/install-vpp-native/vpp/lib/libvppinfra.so.19.08.2
#28 0x7fffb7a8a7e0 in ?? ()
#29 0x762981ec in vlib_process_startup (f=0x7fffb7f71c74, 
p=0x7fffb813e000, vm=0x7fffb8f498c8) at /var/build/vpp/src/vlib/main.c:1494
#30 dispatch_process (vm=0x7fffb87ef9c0, p=0x30, f=0x30, 
last_time_stamp=140736297535488) at /var/build/vpp/src/vlib/main.c:1539
#31 0x7629fa4b in vlib_main_or_worker_loop (is_main=1, 
vm=0x7656c400 ) at /var/build/vpp/src/vlib/main.c:1914
#32 vlib_main_loop (vm=0x7656c400 ) at 
/var/build/vpp/src/vlib/main.c:1931
#33 0x762a6cef in vlib_main (vm=0x7656c400 , 
input=0x7fffb7a8bfb0) at /var/build/vpp/src/vlib/main.c:2148
#34 0x76345ad6 in thread0 (arg=140737326269440) at 
/var/build/vpp/src/vlib/unix/main.c:649
#35 0x75ce23d8 in clib_calljmp () from 
target:/var/build/vpp/build-root/install-vpp-native/vpp/lib/libvppinfra.so.19.08.2
#36 0x7fffcf60 in ?? ()
#37 0x76346b02 in vlib_unix_main (argc=69, argv=0x7fffe4e8) at 
/var/build/vpp/src/vlib/unix/main.c:719
#38 0xb801 in main (argc=69, argv=0x7fffe4e8) at 
/var/build/vpp/src/vpp/vnet/main.c:280
(gdb) fr 4
#4  0x75cc5e72 in do_percent (_s=0x7fffb8141a58, fmt=0x75dc7734 
"%s%c", va=0x7fffb8141bc8) at /var/build/vpp/src/vppinfra/format.c:341
341   vec_add (s, cstring, len);

(gdb) p cstring
$13 = 0x7fffafa01fd0 "iptfs-refill-zpool sa_index %d before %d requested %d 
head %d tail %d"
(gdb) p len
$14 = 69
(gdb) p len
$15 = 69

# I believe the vec_add has just done a _vec_resize b/c of this:
(gdb) fr 4
#4  0x75cc5e72 in do_percent (_s=0x7fffb8141a58, fmt=0x75dc7734 
"%s%c", va=0x7fffb8141bc8) at /var/build/vpp/src/vppinfra/format.c:341
341   vec_add (s, cstring, len);
(gdb) p _s
$21 = (u8 **) 0x7fffb8141a58
(gdb) p *_s
$22 = (u8 *) 0x0
(gdb) p s
$23 = (u8 *) 0x7fffb90bea90 ""
(gdb) p *((vec_header_t *)s - 1)
$24 = {len = 69, dlmalloc_header_offset = 0, vector_data = 0x7fffb90bea90 ""}

So everything should be peachy for clib_memcpy_fast, but something goes 
horribly wrong when it gets to clib_mov16.

#1  0x75cbe910 in clib_mov32 (dst=0x7fffb90bea90 "", src=0x7fffafa01fd0 
"iptfs-refill-zpool sa_index %d before %d requested %d head %d tail %d") at 
/var/build/vpp/src/vppinfra/memcpy_sse3.h:66
66clib_mov16 ((u8 *) dst + 0 * 16, (const u8 *) src + 0 * 16);
(gdb) p dst
$27 = (u8 *) 0x7fffb90bea90 ""
(gdb) p src
$28 = (const u8 *) 0x7fffafa01fd0 "iptfs-refill-zpool sa_index %d before %d 
requested %d head %d tail %d"
(gdb) down
#0  0x000075cbe8b1 in clib_mov16 (dst=0x6d986a57 , src=0x2d8f076c2d08 ) at 
/var/build/vpp/src/vppinfra/memcpy_sse3.h:56
56  {


Thanks,
Chris.

>
> Thanks... Dave
>
> From: vpp-dev@lists.fd.io  On Behalf Of Christian Hopps
> Sent: Friday, June 5, 2020 1:29 PM
> To: vpp-dev 
> Cc: Christian Hopps 
> Subject: [vpp-dev] Interesting backtrace in 1908
>
> I'm wondering if maybe this SIGSEGV/backtrace might be related to the other 
> recently reported problem with the FIB and barrier code? The workers are at 
> the barrier when the SIGSEGV happens, but maybe they aren't when they need to 
> be earlier on?
> 
> In this case I've compiled w/o CLIB_DEBUG set, but with compiler flags set to 
> -O0 instead of -

[vpp-dev] Interesting backtrace in 1908

[Christian Hopps]

I'm wondering if maybe this SIGSEGV/backtrace might be related to the other 
recently reported problem with the FIB and barrier code? The workers are at the 
barrier when the SIGSEGV happens, but maybe they aren't when they need to be 
earlier on?

In this case I've compiled w/o CLIB_DEBUG set, but with compiler flags set to 
-O0 instead of -O2 (trying to debug another problem that occurs much later).

This is being hit (apparently) when my startup config is adding a static arp 
entry (included below the backtrace)

I've sync'd code to the 2 recent commits past 19.08.02 as well as cherry 
picking the fix from Dave for the counter resize issue in the FIB.

I can try and put together a more in depth bug report (or try an RCA it 
myself), but I'm wondering if something might be easily identified from this 
backtrace w/o doing a bunch more work.

Thanks,
Chris.

(gdb) info thre
  Id   Target Id Frame
* 1Thread 83.83 "vpp_main" 0x75ccbb11 in clib_memcpy_fast 
(dst=0x1400, src=0x4500e239, n=936751150609465344) at 
/var/build/vpp/src/vppinfra/memcpy_sse3.h:187
  2Thread 83.86 "eal-intr-thread" 0x759a3bb7 in epoll_wait 
(epfd=epfd@entry=15, events=events@entry=0x7fff9e8dbe10, 
maxevents=maxevents@entry=1, timeout=timeout@entry=-1) at 
../sysdeps/unix/sysv/linux/epoll_wait.c:30
  3Thread 83.87 "vpp_wk_0" 0x76290c90 in 
vlib_worker_thread_barrier_check () at /var/build/vpp/src/vlib/threads.h:430
  4Thread 83.88 "vpp_wk_1" 0x76290c9a in 
vlib_worker_thread_barrier_check () at /var/build/vpp/src/vlib/threads.h:430
  5Thread 83.89 "vpp_wk_2" 0x76290c9f in 
vlib_worker_thread_barrier_check () at /var/build/vpp/src/vlib/threads.h:430

(gdb) bt
#0  0x75ccbb11 in clib_memcpy_fast (dst=0x1400, 
src=0x4500e239, n=936751150609465344) at 
/var/build/vpp/src/vppinfra/memcpy_sse3.h:187
#1  0x75cd49a8 in lookup (v=0x7fffb7c793c8, key=615, op=SET, 
new_value=0x7fffb7c04880, old_value=0x0) at 
/var/build/vpp/src/vppinfra/hash.c:611
#2  0x75cd6217 in _hash_set3 (v=0x7fffb7c793c8, key=615, 
value=0x7fffb7c04880, old_value=0x0) at /var/build/vpp/src/vppinfra/hash.c:840
#3  0x762b0b28 in vlib_node_add_next_with_slot (vm=0x7656c400 
, node_index=522, next_node_index=615, slot=3) at 
/var/build/vpp/src/vlib/node.c:241
#4  0x762b102b in vlib_node_add_next_with_slot (vm=0x7656c400 
, node_index=521, next_node_index=615, slot=3) at 
/var/build/vpp/src/vlib/node.c:253
#5  0x762b102b in vlib_node_add_next_with_slot (vm=0x7656c400 
, node_index=520, next_node_index=615, slot=3) at 
/var/build/vpp/src/vlib/node.c:253
#6  0x762b102b in vlib_node_add_next_with_slot (vm=0x7656c400 
, node_index=519, next_node_index=615, slot=3) at 
/var/build/vpp/src/vlib/node.c:253
#7  0x762b102b in vlib_node_add_next_with_slot (vm=0x7656c400 
, node_index=274, next_node_index=615, slot=3) at 
/var/build/vpp/src/vlib/node.c:253
#8  0x762b102b in vlib_node_add_next_with_slot (vm=0x7656c400 
, node_index=523, next_node_index=615, slot=3) at 
/var/build/vpp/src/vlib/node.c:253
#9  0x776e120a in vlib_node_add_next (next_node=615, node=523, 
vm=0x7656c400 ) at 
/var/build/vpp/src/vlib/node_funcs.h:1109
#10 adj_nbr_update_rewrite_internal (adj=0x7fffb821e5c0, 
adj_next_index=IP_LOOKUP_NEXT_REWRITE, this_node=523, next_node=615, 
rewrite=0x0) at /var/build/vpp/src/vnet/adj/adj_nbr.c:488
#11 0x776e0c34 in adj_nbr_update_rewrite (adj_index=2, 
flags=ADJ_NBR_REWRITE_FLAG_COMPLETE, rewrite=0x7fffb89f4d60 "\002") at 
/var/build/vpp/src/vnet/adj/adj_nbr.c:314
#12 0x76f695b3 in arp_mk_complete (ai=2, e=0x7fffb7f116a8) at 
/var/build/vpp/src/vnet/ethernet/arp.c:385
#13 0x76f696be in arp_mk_complete_walk (ai=2, ctx=0x7fffb7f116a8) at 
/var/build/vpp/src/vnet/ethernet/arp.c:430
#14 0x776e15c0 in adj_nbr_walk_nh4 (sw_if_index=1, addr=0x7fffb7f116ac, 
cb=0x76f69696 , ctx=0x7fffb7f116a8) at 
/var/build/vpp/src/vnet/adj/adj_nbr.c:624
#15 0x76f6a436 in arp_update_adjacency (vnm=0x77b47e80 , 
sw_if_index=1, ai=2) at /var/build/vpp/src/vnet/ethernet/arp.c:540
#16 0x76b30f27 in ethernet_update_adjacency (vnm=0x77b47e80 
, sw_if_index=1, ai=2) at 
/var/build/vpp/src/vnet/ethernet/interface.c:210
#17 0x77706ceb in vnet_update_adjacency_for_sw_interface 
(vnm=0x77b47e80 , sw_if_index=1, ai=2) at 
/var/build/vpp/src/vnet/adj/rewrite.c:187
#18 0x776e0b18 in adj_nbr_add_or_lock (nh_proto=FIB_PROTOCOL_IP4, 
link_type=VNET_LINK_IP4, nh_addr=0x7fffb821f200, sw_if_index=1) at 
/var/build/vpp/src/vnet/adj/adj_nbr.c:252
#19 0x776bff09 in fib_path_attached_next_hop_get_adj 
(path=0x7fffb821f1e8, link=VNET_LINK_IP4) at 
/var/build/vpp/src/vnet/fib/fib_path.c:668
#20 0x776bff50 in fib_path_attached_next_hop_set (path=0x7fffb821f1e8) 
at /var/build/vpp/src/vnet/fib/fib_path.c:682
#21 

Re: [vpp-dev] ixge and rdma drivers

[Christian Hopps]

Looking forward at new purchases. I was trying to find the list of native 
supported PCI devices. The obvious ones I see are avf and rdma. So Intel 710 
(now 810?) virtual function and mellanox connectx-(456?)

That's not a lot to choose from, but the mellanox choice is pretty good one I 
suppose if it works well (e.g. good support for flow director type stuff etc).

I'm starting to think that I need to probably stick with DPDK though, I'm also 
looking into the intel QAT card which gets me back into mbufs overhead anyway. 
This is a bit sad since the vlib+mbuf overhead is eating bandwidth and/or pps.

Thanks,
Chris.



> On Jun 2, 2020, at 7:59 AM, Damjan Marion via lists.fd.io 
>  wrote:
> 
> 
> Hi Christian,
> 
> ixgbe driver is deprecated, that code is very old, from days when VPP was not 
> open source.
> That driver was used for Intel Niantic family (82599, x5x0) of NICs which are 
> those days replaced by  Intel Fortville (x710, xl710, xvv710, x722).
> For fortville and soon for columbiaville NICs (e810), you can use native AVF 
> driver….
> 
> You need to create virtual function first, as AVF driver works only with VFs. 
> There is script which does the job in extras/scripts/vfctl
> 
> 
> i.e.
> 
> $ extras/scripts/vfctl create :61:00.1 1
> 
> Virtual Functions:
> ID PCI Addr PCI IDDriver   MAC Addr  Config
> 0  :61:06.0 8086:37cd vfio-pci  spoof checking off, link-state auto, 
> trust on
> 
> And the in in VPP:
> 
> vpp# create interface avf :61:06.0 name eth0 
> 
> 
> 
>> On 2 Jun 2020, at 09:40, Christian Hopps  wrote:
>> 
>> Hi vpp-dev,
>> 
>> I've been contemplating trying to use native drivers in place of DPDK with 
>> the understanding that I may be paying a ~20% penalty by using DPDK. So I 
>> went to try things out, but had some trouble. The systems in paticular I'm 
>> interested in have 10GE intel NICs in them which I believe would be 
>> supported by the ixge driver. I noticed that this driver has been marked 
>> deprecated in VPP though. Is there a replacement or is DPDK required for 
>> this NIC?
>> 
>> I also have systems that have mlx5 (and eventually will have connectx-6 
>> cards). These cards appear to be supported by the rdma native driver. I was 
>> able to create the interfaces and saw TX packets but no RX.  Is this driver 
>> considered stable and usable in 19.08 (and if not which release would it be 
>> consider so)?
>> 
>> Thanks,
>> Chris.
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16632): https://lists.fd.io/g/vpp-dev/message/16632
Mute This Topic: https://lists.fd.io/mt/74623336/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ixge and rdma drivers

[Christian Hopps]

Ok. Thanks for the info.

FWIW, apparently x520-2 is still somewhat common b/c that's what I've got from 
new dell servers from mid-last year. Will try and get the newer cards in the 
future.

Thanks,
Chris.

> On Jun 2, 2020, at 7:59 AM, Damjan Marion via lists.fd.io 
>  wrote:
> 
> 
> Hi Christian,
> 
> ixgbe driver is deprecated, that code is very old, from days when VPP was not 
> open source.
> That driver was used for Intel Niantic family (82599, x5x0) of NICs which are 
> those days replaced by  Intel Fortville (x710, xl710, xvv710, x722).
> For fortville and soon for columbiaville NICs (e810), you can use native AVF 
> driver….
> 
> You need to create virtual function first, as AVF driver works only with VFs. 
> There is script which does the job in extras/scripts/vfctl
> 
> 
> i.e.
> 
> $ extras/scripts/vfctl create :61:00.1 1
> 
> Virtual Functions:
> ID PCI Addr PCI IDDriver   MAC Addr  Config
> 0  :61:06.0 8086:37cd vfio-pci  spoof checking off, link-state auto, 
> trust on
> 
> And the in in VPP:
> 
> vpp# create interface avf :61:06.0 name eth0 
> 
> 
> 
>> On 2 Jun 2020, at 09:40, Christian Hopps  wrote:
>> 
>> Hi vpp-dev,
>> 
>> I've been contemplating trying to use native drivers in place of DPDK with 
>> the understanding that I may be paying a ~20% penalty by using DPDK. So I 
>> went to try things out, but had some trouble. The systems in paticular I'm 
>> interested in have 10GE intel NICs in them which I believe would be 
>> supported by the ixge driver. I noticed that this driver has been marked 
>> deprecated in VPP though. Is there a replacement or is DPDK required for 
>> this NIC?
>> 
>> I also have systems that have mlx5 (and eventually will have connectx-6 
>> cards). These cards appear to be supported by the rdma native driver. I was 
>> able to create the interfaces and saw TX packets but no RX.  Is this driver 
>> considered stable and usable in 19.08 (and if not which release would it be 
>> consider so)?
>> 
>> Thanks,
>> Chris.
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16616): https://lists.fd.io/g/vpp-dev/message/16616
Mute This Topic: https://lists.fd.io/mt/74623336/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] ixge and rdma drivers

[Christian Hopps]

Hi vpp-dev,

I've been contemplating trying to use native drivers in place of DPDK with the 
understanding that I may be paying a ~20% penalty by using DPDK. So I went to 
try things out, but had some trouble. The systems in paticular I'm interested 
in have 10GE intel NICs in them which I believe would be supported by the ixge 
driver. I noticed that this driver has been marked deprecated in VPP though. Is 
there a replacement or is DPDK required for this NIC?

I also have systems that have mlx5 (and eventually will have connectx-6 cards). 
These cards appear to be supported by the rdma native driver. I was able to 
create the interfaces and saw TX packets but no RX.  Is this driver considered 
stable and usable in 19.08 (and if not which release would it be consider so)?

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16606): https://lists.fd.io/g/vpp-dev/message/16606
Mute This Topic: https://lists.fd.io/mt/74623336/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] unix-epoll-input in workers

[Christian Hopps]

Who runs in interrupt mode? :)

Ok. Well it can sit in gerrit as a starting point for future pickup I guess.

Thanks,
Chris.

> On May 20, 2020, at 4:11 PM, Damjan Marion via lists.fd.io 
>  wrote:
> 
> 
> No, as another important function of that node is to fall asleep when there 
> is no interfaces in polling mode. Interface queues can be dynamically 
> assigned to different workers so there will be lot of messing around to make 
> this working.
> 
>> On 20 May 2020, at 21:51, Christian Hopps  wrote:
>> 
>> Would this work?
>> 
>> https://gerrit.fd.io/r/c/vpp/+/27186
>> 
>> Thanks,
>> Chris.
>> 
>>> On May 20, 2020, at 1:44 PM, Christian Hopps  wrote:
>>> 
>>> 
>>> 
>>>> On May 20, 2020, at 11:39 AM, Damjan Marion via lists.fd.io 
>>>>  wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 20 May 2020, at 16:29, Christian Hopps  wrote:
>>>>> 
>>>>> 
>>>>>> On May 20, 2020, at 9:42 AM, Damjan Marion via lists.fd.io 
>>>>>>  wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On 20 May 2020, at 14:38, Christian Hopps  wrote:
>>>>>>> 
>>>>>>> I'm wondering why I have unix-epoll-input in my worker threads "show 
>>>>>>> runtime" results. Couldn't it selectively disable/enable itself based 
>>>>>>> on whether it actually had any work to do (things to poll)? I'm aware 
>>>>>>> it modifies its behavior when there are other polling nodes running, 
>>>>>>> but it still is taking time to run occasionally, and I'm not sure why.
>>>>>> 
>>>>>> It. is needed there for handling interrupts, and probably nobody 
>>>>>> bothered to spend time on adding some logic to turn it off if there is 
>>>>>> no work for him as it’s impact on performance is negligible.
>>>>> 
>>>>> Ok, thanks. For me its running about 1550ns per call which, if I did my 
>>>>> math right, represents ~92 packets (min 46 octets == 84 ethernet octets) 
>>>>> at 40GE and ~23 packets on a 10GE.
>>>> 
>>>> so your core does 40Gb linerate with 64 byte packets :)
>>> 
>>> No. :)
>>> 
>>> I'm tyring to get the best I can from imix on the user side, and 1500 octet 
>>> (full MTU) fixed interval sending on the secure tunnel side. I wouldn't 
>>> expect the current processors to be able to handle small packets line rate 
>>> on 100G interfaces, in any case.
>>> 
>>>> My math is:
>>>> 
>>>> - Best case we do around 100 clock cycles per packet.
>>>> - If cpu is running at 2.5GHz that means 40ns per packet.
>>>> - in 1550ns we can process ~39 packets.
>>>> - 100 clock cycles per packet means 25Mpps @ 2.5GHz
>>>> 
>>>> 39 packets less processed out of 25M means performance impact is 0,000156%.
>>> 
>>> 1550ns is a per call stat I believe. It seems to average around ~252 calls 
>>> per second for me so thats 9828 packets. If those are 9k packets that's 
>>> 707Mbps of bandwidth (9828*9000*8=707616000). Even at 1500 it represents 
>>> 118 Mbps (117936000). Looking at the code though it's basically being 
>>> called every 1024 (1025?) loops.
>>> 
>>> My main issue though is that people are going to be paying attention to 
>>> timing with my application (IPTFS), so odd blips in packet arrival will be 
>>> noticed, and then have to be explained and justified.
>>> 
>>> I'll try disabling it with a hack for now. :)
>>> 
>>> Thanks,
>>> Chris.
>>> 
>>>> Makes sense?
>>>> 
>>>> — 
>>>> Damjan
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16467): https://lists.fd.io/g/vpp-dev/message/16467
Mute This Topic: https://lists.fd.io/mt/74348786/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] unix-epoll-input in workers

[Christian Hopps]

Would this work?

https://gerrit.fd.io/r/c/vpp/+/27186

Thanks,
Chris.

> On May 20, 2020, at 1:44 PM, Christian Hopps  wrote:
> 
> 
> 
>> On May 20, 2020, at 11:39 AM, Damjan Marion via lists.fd.io 
>>  wrote:
>> 
>> 
>> 
>>> On 20 May 2020, at 16:29, Christian Hopps  wrote:
>>> 
>>> 
>>>> On May 20, 2020, at 9:42 AM, Damjan Marion via lists.fd.io 
>>>>  wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 20 May 2020, at 14:38, Christian Hopps  wrote:
>>>>> 
>>>>> I'm wondering why I have unix-epoll-input in my worker threads "show 
>>>>> runtime" results. Couldn't it selectively disable/enable itself based on 
>>>>> whether it actually had any work to do (things to poll)? I'm aware it 
>>>>> modifies its behavior when there are other polling nodes running, but it 
>>>>> still is taking time to run occasionally, and I'm not sure why.
>>>> 
>>>> It. is needed there for handling interrupts, and probably nobody bothered 
>>>> to spend time on adding some logic to turn it off if there is no work for 
>>>> him as it’s impact on performance is negligible.
>>> 
>>> Ok, thanks. For me its running about 1550ns per call which, if I did my 
>>> math right, represents ~92 packets (min 46 octets == 84 ethernet octets) at 
>>> 40GE and ~23 packets on a 10GE.
>> 
>> so your core does 40Gb linerate with 64 byte packets :)
> 
> No. :)
> 
> I'm tyring to get the best I can from imix on the user side, and 1500 octet 
> (full MTU) fixed interval sending on the secure tunnel side. I wouldn't 
> expect the current processors to be able to handle small packets line rate on 
> 100G interfaces, in any case.
> 
>> My math is:
>> 
>> - Best case we do around 100 clock cycles per packet.
>> - If cpu is running at 2.5GHz that means 40ns per packet.
>> - in 1550ns we can process ~39 packets.
>> - 100 clock cycles per packet means 25Mpps @ 2.5GHz
>> 
>> 39 packets less processed out of 25M means performance impact is 0,000156%.
> 
> 1550ns is a per call stat I believe. It seems to average around ~252 calls 
> per second for me so thats 9828 packets. If those are 9k packets that's 
> 707Mbps of bandwidth (9828*9000*8=707616000). Even at 1500 it represents 118 
> Mbps (117936000). Looking at the code though it's basically being called 
> every 1024 (1025?) loops.
> 
> My main issue though is that people are going to be paying attention to 
> timing with my application (IPTFS), so odd blips in packet arrival will be 
> noticed, and then have to be explained and justified.
> 
> I'll try disabling it with a hack for now. :)
> 
> Thanks,
> Chris.
> 
>> Makes sense?
>> 
>> — 
>> Damjan
>> 
>> 
>> 
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16465): https://lists.fd.io/g/vpp-dev/message/16465
Mute This Topic: https://lists.fd.io/mt/74348786/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] unix-epoll-input in workers

[Christian Hopps]

> On May 20, 2020, at 11:39 AM, Damjan Marion via lists.fd.io 
>  wrote:
> 
> 
> 
>> On 20 May 2020, at 16:29, Christian Hopps  wrote:
>> 
>> 
>>> On May 20, 2020, at 9:42 AM, Damjan Marion via lists.fd.io 
>>>  wrote:
>>> 
>>> 
>>> 
>>>> On 20 May 2020, at 14:38, Christian Hopps  wrote:
>>>> 
>>>> I'm wondering why I have unix-epoll-input in my worker threads "show 
>>>> runtime" results. Couldn't it selectively disable/enable itself based on 
>>>> whether it actually had any work to do (things to poll)? I'm aware it 
>>>> modifies its behavior when there are other polling nodes running, but it 
>>>> still is taking time to run occasionally, and I'm not sure why.
>>> 
>>> It. is needed there for handling interrupts, and probably nobody bothered 
>>> to spend time on adding some logic to turn it off if there is no work for 
>>> him as it’s impact on performance is negligible.
>> 
>> Ok, thanks. For me its running about 1550ns per call which, if I did my math 
>> right, represents ~92 packets (min 46 octets == 84 ethernet octets) at 40GE 
>> and ~23 packets on a 10GE.
> 
> so your core does 40Gb linerate with 64 byte packets :)

No. :)

I'm tyring to get the best I can from imix on the user side, and 1500 octet 
(full MTU) fixed interval sending on the secure tunnel side. I wouldn't expect 
the current processors to be able to handle small packets line rate on 100G 
interfaces, in any case.

> My math is:
> 
> - Best case we do around 100 clock cycles per packet.
> - If cpu is running at 2.5GHz that means 40ns per packet.
> - in 1550ns we can process ~39 packets.
> - 100 clock cycles per packet means 25Mpps @ 2.5GHz
> 
> 39 packets less processed out of 25M means performance impact is 0,000156%.

1550ns is a per call stat I believe. It seems to average around ~252 calls per 
second for me so thats 9828 packets. If those are 9k packets that's 707Mbps of 
bandwidth (9828*9000*8=707616000). Even at 1500 it represents 118 Mbps 
(117936000). Looking at the code though it's basically being called every 1024 
(1025?) loops.

My main issue though is that people are going to be paying attention to timing 
with my application (IPTFS), so odd blips in packet arrival will be noticed, 
and then have to be explained and justified.

I'll try disabling it with a hack for now. :)

Thanks,
Chris.

> Makes sense?
> 
> — 
> Damjan
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16463): https://lists.fd.io/g/vpp-dev/message/16463
Mute This Topic: https://lists.fd.io/mt/74348786/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] unix-epoll-input in workers

[Christian Hopps]

> On May 20, 2020, at 9:42 AM, Damjan Marion via lists.fd.io 
>  wrote:
> 
> 
> 
>> On 20 May 2020, at 14:38, Christian Hopps  wrote:
>> 
>> I'm wondering why I have unix-epoll-input in my worker threads "show 
>> runtime" results. Couldn't it selectively disable/enable itself based on 
>> whether it actually had any work to do (things to poll)? I'm aware it 
>> modifies its behavior when there are other polling nodes running, but it 
>> still is taking time to run occasionally, and I'm not sure why.
> 
> It. is needed there for handling interrupts, and probably nobody bothered to 
> spend time on adding some logic to turn it off if there is no work for him as 
> it’s impact on performance is negligible.

Ok, thanks. For me its running about 1550ns per call which, if I did my math 
right, represents ~92 packets (min 46 octets == 84 ethernet octets) at 40GE and 
~23 packets on a 10GE.

Thanks,
Chris.


> 
> — 
> Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16459): https://lists.fd.io/g/vpp-dev/message/16459
Mute This Topic: https://lists.fd.io/mt/74348786/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] unix-epoll-input in workers

[Christian Hopps]

I'm wondering why I have unix-epoll-input in my worker threads "show runtime" 
results. Couldn't it selectively disable/enable itself based on whether it 
actually had any work to do (things to poll)? I'm aware it modifies its 
behavior when there are other polling nodes running, but it still is taking 
time to run occasionally, and I'm not sure why.

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16457): https://lists.fd.io/g/vpp-dev/message/16457
Mute This Topic: https://lists.fd.io/mt/74348786/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: (Q about fixing endianness bugs in handlers) Re: [vpp-dev] Proposal for VPP binary API stability

[Christian Hopps]

> On May 16, 2020, at 9:52 AM, Andrew  Yourtchenko  wrote:
> 
> On 5/16/20, Dave Barach (dbarach)  wrote:
>> API messages in network byte order made sense 10 years ago when I worked
>> with a mixed x86_64 / ppc32 system. As Damjan points out, API
>> interoperability between big-endian and little-endian systems is a boutique
>> use-case these days.
>> 
>> Timing is key. We won’t be able to cherry-pick API message handler fixes
>> across an endian-order flag-day. If we decide to switch to native byte
>> order, we’d better switch right before we pull our next LTS release.
> 
> Do I understand it right that the idea here would be to flip the
> endianness right before pulling stable/2009 branch ?

If this is done right before (or near) the branch pull for 2009 then I think 
the release candidate should be left to bake longer than normal, and API users 
encouraged to port/test to the release candidate. Obviously any bugs found 
would then be fixed on the RC. Maybe ask for a couple/few stakeholders to sign 
up to do this and then wait a reasonable amount of time for them to sign-off?

I know we use the binary APIs, I believe Netgate does as well. I'm sure there 
are others too (might be good to collect a list of these folks if one doesn't 
exist yet).

If these changes were coupled with the (more important IMO) change i've been 
(ceaselessly :) asking for in the API (non-callback sync functions) I would be 
more than willing to sign up for this.

Thanks,
Chris.

> 
> --a
> 
> 
>> 
>> FWIW... Dave
>> 
>> From: vpp-dev@lists.fd.io  On Behalf Of Damjan Marion
>> via lists.fd.io
>> Sent: Saturday, May 16, 2020 7:23 AM
>> To: Andrew Yourtchenko 
>> Cc: Christian Hopps ; Ole Troan ;
>> vpp-dev ; Jerome Tollet (jtollet) 
>> Subject: Re: (Q about fixing endianness bugs in handlers) Re: [vpp-dev]
>> Proposal for VPP binary API stability
>> 
>> 
>> Knowing that even hard-core big-endian systems like PowerPC decided to
>> switch to Little endian and that situation where binary api will be
>> exchanged between BE and LE systems is extremely small, maybe we should
>> consider removing endian conversion from APIs and simply represent data
>> in the native form. Looks like this is fertile ground for new bugs…
>> 
>> Thoughts?
>> 
>> —
>> Damjan
>> 
>> 
>>> On 15 May 2020, at 16:20, Andrew Yourtchenko
>>> mailto:ayour...@gmail.com>> wrote:
>>> 
>>> There's a very interesting couple of gerrit changes that just came in
>>> today that is worth discussing,
>>> and they are a perfect case study to further clarify the process - so
>>> I tweaked the subject accordingly..
>>> The API message itself is relatively minor, but regardless of what is
>>> agreed that makes a good case study.
>>> 
>>> Backstory:
>>> 
>>> Once upon a time on Aug 20 2019, commit 053204ab changed
>>> sw_interface_set_rx_mode.mode from u8 to an enum, but an htonl
>>> conversion
>>> function didn't make it there (enums are u32 by default as far as I can
>>> see).
>>> 
>>> This was after the 19.08 branch pull, and it wasn't ever tackled, so
>>> this (buggy) behavior ended being in 20.01, 20.01.1, and in the
>>> current 20.05-rc1.
>>> 
>>> Fast forward a bit, today I was pinged about the two changes - one for
>>> master, one for stable/2001:
>>> 
>>> https://gerrit.fd.io/r/c/vpp/+/26879 - in master - forces the enum to be a
>>> u8
>>> 
>>> https://gerrit.fd.io/r/c/vpp/+/26915 - in stable/2001 - adds the
>>> htonl() call and changes the existing ("buggy")
>>> behavior in the 20.01.2 - thus would silently break any API consumers
>>> that coded against the previous "buggy" behavior.
>>> 
>>> And then we have a question open about stable/2005, which "by the
>>> letter" can potentially accept only the second approach, since it is
>>> past the API freeze.
>>> 
>>> Additional bit: this API is not touched in make test, so this bug had
>>> slipped through.
>>> 
>>> So there are the following options:
>>> 
>>> 1) Merge both patches, and treat the 20.05 similar to 20.01, thus
>>> making a "silent change" in both, but making the master look closer to
>>> a 19.08 format.
>>> 
>>> 2) Leave the 20.05 and 20.01 alone with the "buggy" behavior, and
>>> merge the master patch that shrinks the enum down to 1 byte
>>> 
>>> 3) Merge the 20.01 and cherry-pick it to

Re: (Q about fixing endianness bugs in handlers) Re: [vpp-dev] Proposal for VPP binary API stability

[Christian Hopps]

" behavior.
>>> 
>>> And then we have a question open about stable/2005, which "by the
>>> letter" can potentially accept only the second approach, since it is
>>> past the API freeze.
>>> 
>>> Additional bit: this API is not touched in make test, so this bug had
>>> slipped through.
>>> 
>>> So there are the following options:
>>> 
>>> 1) Merge both patches, and treat the 20.05 similar to 20.01, thus
>>> making a "silent change" in both, but making the master look closer to
>>> a 19.08 format.
>>> 
>>> 2) Leave the 20.05 and 20.01 alone with the "buggy" behavior, and
>>> merge the master patch that shrinks the enum down to 1 byte
>>> 
>>> 3) Merge the 20.01 and cherry-pick it to master and 2005 - fixing the
>>> endianness of the u32 enum everywhere, but making an effective "silent
>>> change" in 20.01&20.05
>>> 
>>> 4)  merge the patch in master that shrinks the enum down to one byte,
>>> and cherry-pick it to 20.01 and 20.05 - thus breaking the contract of
>>> "no api changes" but at least this gets to be explicitly visible early
>>> on.
>>> 
>>> 5) under the current proposal, if the API message is defined as
>>> "production" then there would need to be a new "in-progress" message
>>> in master with either of the two fixes, the buggy message would need
>>> to be marked as "deprecated".  And under the current proposal by the
>>> 20.09 the "in-progress" message would become "production", the current
>>> message would be shown as "deprecated",  to be deleted in 21.01.
>>> 
>>> So, the feedback that I would appreciate on the above:
>>> 
>>> 1) the least worst course of action "right now", for this particular
>>> issue. I discussed this with Ole and we have different opinions, so I
>>> would like the actual API users to chime in. Please tell me which is
>>> the least worst option from your point of view :-)
>>> 
>>> 2) What is the best course of action in the future. Note, this is also
>>> the simpler case in that there is a way to trigger a CRC-affecting
>>> change by forcing the enum to be a u8. What would have been the best
>>> course of action if it was simply a missing ntohl() for a field that
>>> noone complained about for 1.5 releases. Can we assume that no-one
>>> used that API and "fix the representation" ?
>>> 
>>> 3) would there be an interest in having a sort of registry of "who
>>> wants to know about things related to each API" - so that if a bug
>>> like this arises that requires a behavior change to fix, I could poll
>>> the affected folks and maybe be able to get away with (1) or (3) from
>>> above ?
>>> 
>>> And a tweak to the process (and potentially tooling) with regards to
>>> going to "production API status":
>>> 
>>> An API that is not touched during a "make test" can not be moved
>>> beyond the "in-progress" status. Adding the unit test implies the
>>> commitment from the API contributor to follow the option (5) above
>>> even for "trivial endianness bugs".
>>> 
>>> Thoughts ?
>>> 
>>> --a
>>> 
>>> 
>>> 
>>> 
>>>> On 5/14/20, Christian Hopps  wrote:
>>>> API stability is borderline critical for a project like VPP I think. Yes, 
>>>> it
>>>> can be used stand-alone, but real value is added by building products on 
>>>> top
>>>> of it.
>>>> 
>>>> Also important is having an API framework that allows for
>>>> backward-compatible changes to the API for making improvements.
>>>> 
>>>> I'm not sure if VPP has the latter, but if there's too much pain with your
>>>> proposed rules I think the latter should addressed rather than sacrificing
>>>> the former. So I support your rules.
>>>> 
>>>> Thanks,
>>>> Chris.
>>>> 
>>>>> On May 14, 2020, at 8:28 AM, Ole Troan  wrote:
>>>>> 
>>>>> Andrew and I have discussed a process around API changes with the goal to
>>>>> limit the impact on VPP consumers.
>>>>> One big painpoint in tracking VPP releases for users of the VPP binary has
>>>>> been API

Re: (Q about fixing endianness bugs in handlers) Re: [vpp-dev] Proposal for VPP binary API stability

[Christian Hopps]

 2) What is the best course of action in the future. Note, this is also
> the simpler case in that there is a way to trigger a CRC-affecting
> change by forcing the enum to be a u8. What would have been the best
> course of action if it was simply a missing ntohl() for a field that
> noone complained about for 1.5 releases. Can we assume that no-one
> used that API and "fix the representation" ?
> 
> 3) would there be an interest in having a sort of registry of "who
> wants to know about things related to each API" - so that if a bug
> like this arises that requires a behavior change to fix, I could poll
> the affected folks and maybe be able to get away with (1) or (3) from
> above ?
> 
> And a tweak to the process (and potentially tooling) with regards to
> going to "production API status":
> 
> An API that is not touched during a "make test" can not be moved
> beyond the "in-progress" status. Adding the unit test implies the
> commitment from the API contributor to follow the option (5) above
> even for "trivial endianness bugs".
> 
> Thoughts ?
> 
> --a
> 
> 
> 
> 
> On 5/14/20, Christian Hopps  wrote:
>> API stability is borderline critical for a project like VPP I think. Yes, it
>> can be used stand-alone, but real value is added by building products on top
>> of it.
>> 
>> Also important is having an API framework that allows for
>> backward-compatible changes to the API for making improvements.
>> 
>> I'm not sure if VPP has the latter, but if there's too much pain with your
>> proposed rules I think the latter should addressed rather than sacrificing
>> the former. So I support your rules.
>> 
>> Thanks,
>> Chris.
>> 
>>> On May 14, 2020, at 8:28 AM, Ole Troan  wrote:
>>> 
>>> Andrew and I have discussed a process around API changes with the goal to
>>> limit the impact on VPP consumers.
>>> One big painpoint in tracking VPP releases for users of the VPP binary has
>>> been API changes.
>>> 
>>> We want to ensure:
>>> - A production API never changes.
>>> 
>>> - A production API can be deprecated with one release notice.
>>> E.g. show_version_2 is introduced in 20.01 and show_version_1 is marked as
>>> deprecated.
>>> show_version_1 is marked with "option delete_by = "v20.05";
>>> 
>>> - APIs marked as experimental / not released can change with no notice.
>>> Added support for option status = "in_progress";
>>> 
>>> This patch:
>>> https://gerrit.fd.io/r/c/vpp/+/26881
>>> 
>>> has a tool "crcchecker" that we intend to run as part of the verify job.
>>> It will block modifications to existing production APIs.
>>> make checkstyle-api
>>> It also supports dumping API manifests and make API comparisions across
>>> releases/revisions/commits.
>>> 
>>> A production API is defined by having .api semantic major version > 0.
>>> New messages can override that by setting status="in_progress".
>>> 
>>> The consequence of this is that developers must maintain two (or more)
>>> versions of an API for some time.
>>> This is obviously painful, but at least it aligns cost and benefit better
>>> than when we forced the API users to do it.
>>> 
>>> Looking back from 17.01 onwards, it looks like we on average have done
>>> about 50 backwards incompatible changes per release.
>>> That's ignoring the work on API typing, which have resulted in
>>> significantly more changes.
>>> 
>>> The hope is that we can start v20.09 development with this process in
>>> place.
>>> 
>>> What do people think?
>>> 
>>> Best regards,
>>> Ole
>> 
>> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16408): https://lists.fd.io/g/vpp-dev/message/16408
Mute This Topic: https://lists.fd.io/mt/74228149/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Performance on the Macchiatobin

[Christian Hopps]

We've also seen good numbers from these boxes; however, we have had issues with 
connecting them back-to-back (they don't work that way the interfaces flap), 
they seem to work when connected through a *non-marvel* based switch. IOW 
there's something weird going on with the marvel chip talking to itself (or the 
parts they use).

Then we experienced problems with the HW encryption offload, getting into 
non-recoverable failures. This could be sw bugs either in DPDK or in the 
marvell mvsam support code, or perhaps in the HW, who knows.

Not seeing any recent development and being unable to get support from anyone 
for the boxes caused us to stop using them. It's a shame the performance was 
fantastic with the encryption HW so it was a hard choice to make.

If you have time to debug these issues then they might be good HW, otherwise ...

Thanks,
Chris.



> On May 14, 2020, at 2:07 PM, Govindarajan Mohandoss 
>  wrote:
> 
> Hi Shmuel,
>  We measured IPv4 forwarding performance in the beginning of 2019 (with VPP 
> Master branch) and got 5.77 MPPS with single core @ 1.6 GHz. We used Marvell 
> I/O Plugin (PMD) and not DPDK. We don’t have the latest performance numbers.  
> Currently, our Macchiatobin boards are not functional.  Please contact Nitin 
> Saxena for latest updates.
> 
> Thanks
> Govind
> 
>> -Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Shmuel H. via
>> lists.fd.io
>> Sent: Thursday, May 14, 2020 9:53 AM
>> To: vpp-dev@lists.fd.io
>> Subject: [vpp-dev] Performance on the Macchiatobin
>> 
>> Hi,
>> 
>> I have noticed that there has been some work on the Macchiatobin on vpp
>> (as a part of the aarach64 project).
>> 
>> However, I have not managed to find any performance results for the
>> Macchiatobin as it is not a part of CSIT.
>> 
>> I have found this related(?) [1] JIRA task from 2019/2018, but with no actual
>> results.
>> 
>> If someone has some information about what performance I should expect
>> from the Macchiatobin, I would appreciate it a lot.
>> 
>> [1]: https://jira.fd.io/browse/VPP-1267
>> 
>> Regards,
>> --
>> - Shmuel Hazan
>> 
>> mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16392): https://lists.fd.io/g/vpp-dev/message/16392
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Well would be a nice tool if one could use it while developing with changes to 
the API, but the changed API issue (an added function, which *should* be 
backward compatible) and go packaging nightmare with trying to regenerate the 
vpp API inside dependent packages (vpp-agent which then breaks cn-infra, etc..) 
appear to be insurmountable in a reasonable amount of time (a few hours)

Too bad, looked interesting. I'm going to try and back out my API changes and 
see if that works as a stop-gap, but otherwise will have to punt.

Thanks,
Chris.

> On May 14, 2020, at 7:24 AM, Christian Hopps  wrote:
> 
> Has anyone ever tried to build this with a development version (where APIs 
> have changed)?
> 
> After some go fmt.Printf debugging I see that I have 3 incompatible API calls 
> with vpp1908 (.01) binapi. It's really not obvious since i'm not a go 
> packaging expert how to update everything. I did find some info on generating 
> a vpp go api, but how to do that with the downloaded go module dependencies 
> (or how to update those to include my development version) is not obvious.
> 
> Thanks,
> Chris.
> 
>> On May 14, 2020, at 5:27 AM, Christian Hopps  wrote:
>> 
>> Yes, looking at that now. Should have googled first :)
>> 
>> I think the only thing missing here is the usage bar to present "idleness" 
>> per-worker/core. But this is a great start.
>> 
>> Thanks,
>> Chris.
>> 
>>> On May 14, 2020, at 5:25 AM, Benoit Ganne (bganne) via lists.fd.io 
>>>  wrote:
>>> 
>>> Hi Chris,
>>> 
>>>> Has anyone contemplated the possibility of a "vpptop" like utility for
>>>> VPP? The thought crossed my mind while I was using htop to help debug some
>>>> performance problems I've been having that aren't directly related to vpp
>>>> processing.
>>> 
>>> Something like this https://github.com/PantheonTechnologies/vpptop ?
>>> 
>>> ben
>>> 
>> 
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16385): https://lists.fd.io/g/vpp-dev/message/16385
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Proposal for VPP binary API stability

[Christian Hopps]

API stability is borderline critical for a project like VPP I think. Yes, it 
can be used stand-alone, but real value is added by building products on top of 
it.

Also important is having an API framework that allows for backward-compatible 
changes to the API for making improvements.

I'm not sure if VPP has the latter, but if there's too much pain with your 
proposed rules I think the latter should addressed rather than sacrificing the 
former. So I support your rules.

Thanks,
Chris.

> On May 14, 2020, at 8:28 AM, Ole Troan  wrote:
> 
> Andrew and I have discussed a process around API changes with the goal to 
> limit the impact on VPP consumers.
> One big painpoint in tracking VPP releases for users of the VPP binary has 
> been API changes.
> 
> We want to ensure:
> - A production API never changes.
> 
> - A production API can be deprecated with one release notice.
> E.g. show_version_2 is introduced in 20.01 and show_version_1 is marked as 
> deprecated.
> show_version_1 is marked with "option delete_by = "v20.05";
> 
> - APIs marked as experimental / not released can change with no notice.
> Added support for option status = "in_progress";
> 
> This patch:
> https://gerrit.fd.io/r/c/vpp/+/26881
> 
> has a tool "crcchecker" that we intend to run as part of the verify job.
> It will block modifications to existing production APIs.
> make checkstyle-api
> It also supports dumping API manifests and make API comparisions across 
> releases/revisions/commits.
> 
> A production API is defined by having .api semantic major version > 0.
> New messages can override that by setting status="in_progress".
> 
> The consequence of this is that developers must maintain two (or more) 
> versions of an API for some time.
> This is obviously painful, but at least it aligns cost and benefit better 
> than when we forced the API users to do it.
> 
> Looking back from 17.01 onwards, it looks like we on average have done about 
> 50 backwards incompatible changes per release.
> That's ignoring the work on API typing, which have resulted in significantly 
> more changes.
> 
> The hope is that we can start v20.09 development with this process in place.
> 
> What do people think?
> 
> Best regards,
> Ole

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16381): https://lists.fd.io/g/vpp-dev/message/16381
Mute This Topic: https://lists.fd.io/mt/74203477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Has anyone ever tried to build this with a development version (where APIs have 
changed)?

After some go fmt.Printf debugging I see that I have 3 incompatible API calls 
with vpp1908 (.01) binapi. It's really not obvious since i'm not a go packaging 
expert how to update everything. I did find some info on generating a vpp go 
api, but how to do that with the downloaded go module dependencies (or how to 
update those to include my development version) is not obvious.

Thanks,
Chris.

> On May 14, 2020, at 5:27 AM, Christian Hopps  wrote:
> 
> Yes, looking at that now. Should have googled first :)
> 
> I think the only thing missing here is the usage bar to present "idleness" 
> per-worker/core. But this is a great start.
> 
> Thanks,
> Chris.
> 
>> On May 14, 2020, at 5:25 AM, Benoit Ganne (bganne) via lists.fd.io 
>>  wrote:
>> 
>> Hi Chris,
>> 
>>> Has anyone contemplated the possibility of a "vpptop" like utility for
>>> VPP? The thought crossed my mind while I was using htop to help debug some
>>> performance problems I've been having that aren't directly related to vpp
>>> processing.
>> 
>> Something like this https://github.com/PantheonTechnologies/vpptop ?
>> 
>> ben
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16379): https://lists.fd.io/g/vpp-dev/message/16379
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Yes, looking at that now. Should have googled first :)

I think the only thing missing here is the usage bar to present "idleness" 
per-worker/core. But this is a great start.

Thanks,
Chris.

> On May 14, 2020, at 5:25 AM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi Chris,
> 
>> Has anyone contemplated the possibility of a "vpptop" like utility for
>> VPP? The thought crossed my mind while I was using htop to help debug some
>> performance problems I've been having that aren't directly related to vpp
>> processing.
> 
> Something like this https://github.com/PantheonTechnologies/vpptop ?
> 
> ben
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16376): https://lists.fd.io/g/vpp-dev/message/16376
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Has anyone contemplated the possibility of a "vpptop" like utility for VPP? The 
thought crossed my mind while I was using htop to help debug some performance 
problems I've been having that aren't directly related to vpp processing.

I'm thinking that vpptop could present a dashboard like top does showing the 
(top?) node run times, etc -- with the ability to narrow to per-core. A good 
starting point for the column data would be show runtime output.

More importantly maybe, it would also present per-core usage bars giving a 
"graphical" representation of "idleness" per core so one could quickly know how 
efficiently things were running on each core, and thus if workload was 
distributed correctly.

For memory it could show a graphical bar depicting buffer usage as well.

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16373): https://lists.fd.io/g/vpp-dev/message/16373
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

> On May 11, 2020, at 9:56 AM, Neale Ranns (nranns)  wrote:
> 
> 
> 
> On 11/05/2020 14:28, "Christian Hopps"  wrote:
> 
> 
> 
>> On May 11, 2020, at 7:50 AM, Neale Ranns (nranns)  wrote:
>> 
>> 
>> 
>> From:  on behalf of Christian Hopps 
>> Date: Sunday 10 May 2020 at 14:33
>> To: "Neale Ranns (nranns)" 
>> Cc: Christian Hopps , vpp-dev 
>> Subject: Re: [vpp-dev] IPsec tunnel interfaces?
>> 
>>> On May 9, 2020, at 7:23 AM, Neale Ranns via lists.fd.io 
>>>  wrote:
>>> 
>>> 
>>> 
>>> Hi Chris,
>>> 
>>> 
>>>> Are there other properties of a tunnel mode SA that you need that are lost 
>>>> with this approach?
>>> 
>>> I need to use tunnel mode SAs provided by IKEv2. Transport mode is an 
>>> optional (normally on-the-wire IKEv2 negotiated) feature of IPsec. These 
>>> tunnel mode SAs will have IPTFS enabled on them, and that functionality is 
>>> only defined for IPsec tunnel mode SAs.
>>> 
>>> 
>>> The only difference in VPP between a transport and tunnel mode SA is the 
>>> presence of the encap. So I think it’s fair to say that what you need is an 
>>> interface to interact with the L[23] system, ‘encap’ to describe how to 
>>> encap/decap packets (i.e. what to copy from overlay/underlay (DSCP, ECN, 
>>> etc) and an SA (for the algo set);
>>> 
>>>  Interface + encap + SA
>>> 
>>> VPP doesn’t model encap separately. So it’s a question of where you add the 
>>> parenthesis.
>>> 
>>>  (interface + encap) + SA = ipip tunnel + transport mode SA
>>> 
>>> Or
>>> 
>>>  Interface + (encap + SA) = ipsec dedicated interface + tunnel mode SA
>>> 
>>> In both cases the same information is available, it’s just modelled 
>>> differently. The first model is used since it reuses the 
>>> types/functionality that VPP already has to support other use case, without 
>>> the need for a dedicated interface type. Is it not possible for you to work 
>>> with the first model, or is it less convenient?
>> 
>> SO, I have implemented 
>> https://tools.ietf.org/html/draft-ietf-ipsecme-iptfs-01 in VPP 19.08. The 
>> functionality is working as specified in the draft using tunnel mode SAs.
>> 
>> Conceptually what happens (commonly) is this:
>> 
>> 
>> Pkt   Pkt Single IPsec 
>> Tunnel Pkt
>> ---   --- 
>> --
>> [UA]..[Un] ---> user-intf [ VPP ] sa-tunnel-intf ---> [IP(SATunnel) + ESP + 
>> [UA]..[Un][Pad]]
>> 
>> 
>> The encpasulation *has* to occur at the SA tunnel point, not 
>> pre-encapsulated by a generic IP-IP interface with a transport mode SA 
>> attached to it downstream.
>> 
>> This is the condition I don’t fully understand….
>> 
>> 
>> The key here is that there is not a 1-1 mapping of user IP packets to IPsec 
>> packets. FWIW, this isn't just a problem for this particular IPTFS 
>> technology, there are other simple cases (e.g., sending only pad IPsec 
>> packets for limited traffic flow confidentiality) where there is not a 1-1 
>> mapping between user IP packets and SA tunnel mode packets.
>> 
>> Now, re-architecting IPTFS to exist outside of IPsec so that it could be a 
>> new generic IP tunnel technology is certainly a fun idea (topic for another 
>> thread), it's just not an option, or relevant to the functionality that 
>> appears to have been lost in VPP.
>> 
>> Here's a packet trace for how this works (incoming ping):
>> 
>> USER-SIDE:
>> 
>> 00:00:08:845351: dpdk-input
>>  ...
>>  ICMP echo_request checksum 0xaea9
>> 00:00:08:845366: ethernet-input
>> 00:00:08:845382: ip4-input-no-checksum
>>  ICMP: 11.11.11.253 -> 12.12.12.12
>>  ICMP echo_request checksum 0xaea9
>> 00:00:08:845389: ip4-lookup
>>  ICMP: 11.11.11.253 -> 12.12.12.12
>>  ICMP echo_request checksum 0xaea9
>> 00:00:08:845396: ip4-midchain
>>ICMP: 11.11.11.253 -> 12.12.12.12
>>ICMP echo_request checksum 0xaea9
>> 00:00:08:845402: iptfs-encap4-tun
>>  sa_index: 1
>> 
>> At this point in old code, the packet does not have the tunnel encap added, 
>> it new code it does.
> 
>Right, so this is the problem, at this point -- pre-ipsec encapsulation -- 
> in the arc I am collecting (aggregating multiple us

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

> On May 11, 2020, at 7:50 AM, Neale Ranns (nranns)  wrote:
> 
>
>
> From:  on behalf of Christian Hopps 
> Date: Sunday 10 May 2020 at 14:33
> To: "Neale Ranns (nranns)" 
> Cc: Christian Hopps , vpp-dev 
> Subject: Re: [vpp-dev] IPsec tunnel interfaces?
>
> > On May 9, 2020, at 7:23 AM, Neale Ranns via lists.fd.io 
> >  wrote:
> > 
> >
> > 
> > Hi Chris,
> > 
> > 
> > > Are there other properties of a tunnel mode SA that you need that are 
> > > lost with this approach?
> > 
> > I need to use tunnel mode SAs provided by IKEv2. Transport mode is an 
> > optional (normally on-the-wire IKEv2 negotiated) feature of IPsec. These 
> > tunnel mode SAs will have IPTFS enabled on them, and that functionality is 
> > only defined for IPsec tunnel mode SAs.
> > 
> > 
> > The only difference in VPP between a transport and tunnel mode SA is the 
> > presence of the encap. So I think it’s fair to say that what you need is an 
> > interface to interact with the L[23] system, ‘encap’ to describe how to 
> > encap/decap packets (i.e. what to copy from overlay/underlay (DSCP, ECN, 
> > etc) and an SA (for the algo set);
> > 
> >   Interface + encap + SA
> > 
> > VPP doesn’t model encap separately. So it’s a question of where you add the 
> > parenthesis.
> > 
> >   (interface + encap) + SA = ipip tunnel + transport mode SA
> > 
> > Or
> > 
> >   Interface + (encap + SA) = ipsec dedicated interface + tunnel mode SA
> > 
> > In both cases the same information is available, it’s just modelled 
> > differently. The first model is used since it reuses the 
> > types/functionality that VPP already has to support other use case, without 
> > the need for a dedicated interface type. Is it not possible for you to work 
> > with the first model, or is it less convenient?
> 
> SO, I have implemented 
> https://tools.ietf.org/html/draft-ietf-ipsecme-iptfs-01 in VPP 19.08. The 
> functionality is working as specified in the draft using tunnel mode SAs.
> 
> Conceptually what happens (commonly) is this:
> 
>   
> Pkt   Pkt Single IPsec 
> Tunnel Pkt
> ---   --- 
> --
> [UA]..[Un] ---> user-intf [ VPP ] sa-tunnel-intf ---> [IP(SATunnel) + ESP + 
> [UA]..[Un][Pad]]
> 
> 
> The encpasulation *has* to occur at the SA tunnel point, not pre-encapsulated 
> by a generic IP-IP interface with a transport mode SA attached to it 
> downstream.
> 
> This is the condition I don’t fully understand….
> 
> 
> The key here is that there is not a 1-1 mapping of user IP packets to IPsec 
> packets. FWIW, this isn't just a problem for this particular IPTFS 
> technology, there are other simple cases (e.g., sending only pad IPsec 
> packets for limited traffic flow confidentiality) where there is not a 1-1 
> mapping between user IP packets and SA tunnel mode packets.
> 
> Now, re-architecting IPTFS to exist outside of IPsec so that it could be a 
> new generic IP tunnel technology is certainly a fun idea (topic for another 
> thread), it's just not an option, or relevant to the functionality that 
> appears to have been lost in VPP.
> 
> Here's a packet trace for how this works (incoming ping):
> 
> USER-SIDE:
> 
> 00:00:08:845351: dpdk-input
>   ...
>   ICMP echo_request checksum 0xaea9
> 00:00:08:845366: ethernet-input
> 00:00:08:845382: ip4-input-no-checksum
>   ICMP: 11.11.11.253 -> 12.12.12.12
>   ICMP echo_request checksum 0xaea9
> 00:00:08:845389: ip4-lookup
>   ICMP: 11.11.11.253 -> 12.12.12.12
>   ICMP echo_request checksum 0xaea9
> 00:00:08:845396: ip4-midchain
> ICMP: 11.11.11.253 -> 12.12.12.12
> ICMP echo_request checksum 0xaea9
> 00:00:08:845402: iptfs-encap4-tun
>   sa_index: 1
> 
> At this point in old code, the packet does not have the tunnel encap added, 
> it new code it does.

Right, so this is the problem, at this point -- pre-ipsec encapsulation -- in 
the arc I am collecting (aggregating multiple user IP packets) into a single 
payload to send over the IPsec SA tunnel. I can't have IP-IP packets here.

It's not OK to have to strip off superfluous IP headers (thus having paid a 
price to have them added, only to be immediately removed in the next node in 
the graph) from all the user packets just so I can get back to the 
functionality I had before 20.01, that's not the same behavior. :)

> AGGREGATING AND QUEUEING OCCURS - The packet is encapsulated (along with any 
> others currently waiting) into the next-to-be-sent IPTFS pac

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

Just a general followup in case tl;dr

The point of traffic flow security/confidentiality is to hide the effects of 
the user traffic on the secure tunnel output. With IPTFS there is a complete 
decoupling so that one cannot infer anything about the encapsulated traffic 
from the IPsec tunnel data packets (when they are sent, how big they are, etc), 
there are other (more complex) methods for obfuscation that may or may not be 
better. 

In any case, having the user data traffic directly determine the IPsec tunnel 
behavior is inherently less secure than if it doesn't, so the idea that one 
could find an equivalence between a tunnel mode SA and a transport mode SA 
doesn't work here. For some uses of IPsec it may work, but not always.

In order to make transport mode work (e.g., to support the Cisco-preferred, 
thus common, GRE+transport IPsec config) we are going to have to do more 
standards work and probably put various limitations etc on the types of user IP 
that can be input (so that we can cache and replicate that information at will) 
-- but that does not represent an equivalence.

Thanks,
Chris.

> On May 10, 2020, at 8:33 AM, Christian Hopps  wrote:
> 
>> On May 9, 2020, at 7:23 AM, Neale Ranns via lists.fd.io 
>>  wrote:
>> 
>> 
>> 
>> Hi Chris,
>> 
>> 
>>> Are there other properties of a tunnel mode SA that you need that are lost 
>>> with this approach?
>> 
>> I need to use tunnel mode SAs provided by IKEv2. Transport mode is an 
>> optional (normally on-the-wire IKEv2 negotiated) feature of IPsec. These 
>> tunnel mode SAs will have IPTFS enabled on them, and that functionality is 
>> only defined for IPsec tunnel mode SAs.
>> 
>> 
>> The only difference in VPP between a transport and tunnel mode SA is the 
>> presence of the encap. So I think it’s fair to say that what you need is an 
>> interface to interact with the L[23] system, ‘encap’ to describe how to 
>> encap/decap packets (i.e. what to copy from overlay/underlay (DSCP, ECN, 
>> etc) and an SA (for the algo set);
>> 
>>  Interface + encap + SA
>> 
>> VPP doesn’t model encap separately. So it’s a question of where you add the 
>> parenthesis.
>> 
>>  (interface + encap) + SA = ipip tunnel + transport mode SA
>> 
>> Or
>> 
>>  Interface + (encap + SA) = ipsec dedicated interface + tunnel mode SA
>> 
>> In both cases the same information is available, it’s just modelled 
>> differently. The first model is used since it reuses the types/functionality 
>> that VPP already has to support other use case, without the need for a 
>> dedicated interface type. Is it not possible for you to work with the first 
>> model, or is it less convenient?
> 
> SO, I have implemented 
> https://tools.ietf.org/html/draft-ietf-ipsecme-iptfs-01 in VPP 19.08. The 
> functionality is working as specified in the draft using tunnel mode SAs.
> 
> Conceptually what happens (commonly) is this:
> 
> 
> Pkt   Pkt Single IPsec 
> Tunnel Pkt
> ---   --- 
> --
> [UA]..[Un] ---> user-intf [ VPP ] sa-tunnel-intf ---> [IP(SATunnel) + ESP + 
> [UA]..[Un][Pad]]
> 
> 
> The encpasulation *has* to occur at the SA tunnel point, not pre-encapsulated 
> by a generic IP-IP interface with a transport mode SA attached to it 
> downstream.
> 
> The key here is that there is not a 1-1 mapping of user IP packets to IPsec 
> packets. FWIW, this isn't just a problem for this particular IPTFS 
> technology, there are other simple cases (e.g., sending only pad IPsec 
> packets for limited traffic flow confidentiality) where there is not a 1-1 
> mapping between user IP packets and SA tunnel mode packets.
> 
> Now, re-architecting IPTFS to exist outside of IPsec so that it could be a 
> new generic IP tunnel technology is certainly a fun idea (topic for another 
> thread), it's just not an option, or relevant to the functionality that 
> appears to have been lost in VPP.
> 
> Here's a packet trace for how this works (incoming ping):
> 
> USER-SIDE:
> 
> 00:00:08:845351: dpdk-input
>  ...
>  ICMP echo_request checksum 0xaea9
> 00:00:08:845366: ethernet-input
> 00:00:08:845382: ip4-input-no-checksum
>  ICMP: 11.11.11.253 -> 12.12.12.12
>  ICMP echo_request checksum 0xaea9
> 00:00:08:845389: ip4-lookup
>  ICMP: 11.11.11.253 -> 12.12.12.12
>  ICMP echo_request checksum 0xaea9
> 00:00:08:845396: ip4-midchain
>ICMP: 11.11.11.253 -> 12.12.12.12
>ICMP echo_request checksum 0xaea9
> 00:00:08:845402: iptfs-encap4-tun
>  sa_index: 1
> 
> 
> AGGREGA

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

etup the code I add myself as a feature.

  VNET_FEATURE_INIT (iptfs_encap4_tun_feat_node, static) = {
.arc_name = "ip4-output",
.node_name = "iptfs-encap4-tun",
.runs_before = VNET_FEATURES ("esp4-encrypt-tun", "dpdk-esp4-encrypt-tun"),
  };

  ipsec_add_feature ("ip4-output", "iptfs-encap4-tun",
 >encap4_tun_feature_index);
 
then inside ipsec_tunnel_feature_set I do (in a callback, but whatever):

  vnet_feature_enable_disable_with_index (
  arc, tfsm->encap4_tun_feature_index, t->sw_if_index, enable,
  >output_sa_index, sizeof (t->output_sa_index));

To enable the IPTFS feature on the ipsec* interface arc.

If I have missed a simple way to do this with the new code, I'm all ears and 
thankful for help. :)

Thanks,
Chris.


> 
> /neale
> 
>
> 
> 
> There will be future work in IETF/ipsecme to enable a form of transport mode 
> support in IPTFS to handle the Cisco-preferred GRE with transport mode IPsec 
> configuration, but that is not available today, and obviously won't be the 
> only option standardized.
> 
> Thanks,
> Chris.
> 
> 
> > /neale
> > 
> >
> > 
> >
> > 
> > 
> > Thanks,
> > Chris.
> > 
> > 
> > > 
> > >I did read through the Wiki and it seems that this change was 
> > > motivated by wanting to cleanup the IPsec API, but that hardly seems like 
> > > justification to eliminate the efficient use of an entire variant of 
> > > commonly used IPsec functionality.
> > > 
> > > Cleaning up the API was one motivation. It was a pain that each time we 
> > > added new attributes to SA creation (like ESN, UDP, algo=foo) (for use 
> > > with the SPD) we had to make similar changes to both the ipsec and 
> > > ipsec_gre create APIs. The other motivation was removing 2 interface 
> > > types that did exactly the same as the existing ipip and gre tunnels (and 
> > > the same goes for their APIs too, like how do I configure what DCSP, ECN, 
> > > DF to copy on encap/decap).
> > > 
> > > /neale
> > > 
> > > 
> > > 
> > >Could we bring back the functionality using more "acceptable to the 
> > > project" APIs or something?
> > > 
> > >Thanks,
> > >Chris.
> > > 
> > >> 
> > >> /neale
> > >> 
> > >> 
> > >> From:  on behalf of Christian Hopps 
> > >> 
> > >> Date: Wednesday 6 May 2020 at 14:32
> > >> To: vpp-dev 
> > >> Cc: Christian Hopps 
> > >> Subject: [vpp-dev] IPsec tunnel interfaces?
> > >> 
> > >> Hi, vpp-dev,
> > >> 
> > >> Post 19.08 seems to have removed IPsec logical interfaces.
> > >> 
> > >> One cannot always use transport mode IPsec.
> > >> 
> > >> How can I get the efficiency of route based (FIB) IPsec w/o transport 
> > >> mode? Adding superfluous encapsulations (wasting bandwidth) to replace 
> > >> this (seemingly lost, hope not) functionality is not an option.
> > >> 
> > >> Thanks,
> > >> Chris.
> > >> 
> > 
> > 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16297): https://lists.fd.io/g/vpp-dev/message/16297
Mute This Topic: https://lists.fd.io/mt/74027328/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

On May 8, 2020, at 3:57 AM, Neale Ranns via lists.fd.io 
 wrote:
> 
>
>
> From:  on behalf of Christian Hopps 
> Date: Thursday 7 May 2020 at 23:27
> To: "Neale Ranns (nranns)" 
> Cc: Christian Hopps , vpp-dev 
> Subject: Re: [vpp-dev] IPsec tunnel interfaces?
>
> 
> 
> > On May 7, 2020, at 1:41 PM, Neale Ranns (nranns)  wrote:
> > 
> > 
> > Hi Chris,
> > 
> > On 07/05/2020 16:55, "Christian Hopps"  wrote:
> > 
> > 
> > 
> >> On May 7, 2020, at 8:15 AM, Neale Ranns (nranns)  wrote:
> >> 
> >> 
> >> Hi Chris,
> >> 
> >> They were replaced by ipip interfaces protected by SAs:
> >>  https://wiki.fd.io/view/VPP/IPSec#Tunnel_Mode
> >> 
> >> the tunnel always adds encap. You can configure your SA to add additional 
> >> encap if you want.
> > 
> >Ok, but that's not a replacement for the old functionality, right? The 
> > old functionality had the SA tunnel represented as an unnumbered interface 
> > that could be routed onto efficiently using the FIB. The unnumbered 
> > interface used the SA tunnel endpoint addresses.
> > 
> > It is intended to be a replacement. If it's not then there's a problem and 
> > I'll address it.
> 
> Great. :)
> 
> > 
> > Both old an new have one critical thing in common, there's an interface and 
> > an inbound and outbound SA.
> > The old functionality had a ip-in-ip interface tightly coupled to two SAs, 
> > so tight that there was a dedicated interface type, ipsecX, and the 
> > interface and SAs were configured at the same time. In the new model that 
> > coupling is loose; there's a ipip interface and its associated SAs, they 
> > are configured independently and bound together.
> > If you made the ipsecX unnumbered you now make the ipipX unnumbered. If you 
> > had routes through ipsecX, now you route through IpipX.
> > 
> >The wiki shows the that SA tunnel mode is re-encapsulating the already 
> > encapsulated IP-IP tunnel traffic. So now I have 3 IP headers instead of 
> > the 2 IP headers as before?
> > 
> > If an SA is in transport mode it does not add headers, if it's in tunnel 
> > mode it does. When a packet egresses the tunnel it is encapped by that 
> > tunnel, if you don't want another set of IP headers then configure the SA 
> > that protects the tunnel to be in transport mode. 
> 
> So with the old functionality the interface was unnumbered, and used the SA 
> endpoint IPs for the it's encapsulating IP header. Using it with SA in tunnel 
> mode did not produce 3 IP headers. That's what we need to fix I guess, it 
> should only produce 2 IP headers, the user's and the SA tunnel IP header.
> 
> You still have the choice to make the ipip interface unnumbered, or you can 
> assign it a subnet. Or if it’s an L2 interface (like the old ipsec-gre was 
> and the new protected GRE interface can be) you can add it to a BD. The end 
> points of the tunnel are your choice.
> 
> 
> 
> >Putting aside the wasted IP header bandwidth for the moment though, I 
> > don't understand what's actually supposed to be happening here. What does 
> > the configuration look like? I have an SA with endpoints 
> > (Local-IP,Remote-IP) and I have user traffic with (User-SIP,User-DIP). 
> > Previously I had an unnumbered interface that used the SAs 
> > (Local-IP,Remote-IP) for it's IP header. I then routed traffic for 
> > (User-DIP) over that unnumbered interface. How does one configure that with 
> > this ipip tunnel replacement?
> > 
> > create ipip src Local-IP dst remote-IP
> > set int unnum ipip0 use Eth0
> > ipsec sa add id 5 [crypto .. ] [integ ..] mode=transport
> > ipsec sa add id 6 [crypto .. ] [integ ..] mode=transport
> > ipsec tun protect ipip0 in 5 out 6
> > set int state ipip0 up
> > 
> > ip route User-DIP/32 via ipip0
> > 
> > does that get you what you need?
> 
> No, using transport mode IPsec is not an option. It must be an SA in standard 
> tunnel mode. And it can't have 3 IP headers. :)
> 
> if your original packet that is routed into the tunnel was:
> 
>   [ IP (user-DIP, user-SIP) | TCP | Payload ]
> 
> Then the above configuration will give you:
> 
>  [ MAC | [ IP (tun-DIP, tun-SIP ) | ESP | IP (User-DIP, User-SIP) | TCP | 
> Payload ]
> 
> If you didn’t have protection on the tunnel interface, you’d get
> 
>  [ MAC | [ IP (tun-DIP, tun-SIP ) | IP (User-DIP, User-SIP) | TCP | Payload ]
> 
> So the SA acts like transport mode, i.e. it’s inserting the ESP header 
> between the tunnel IP h

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

> On May 7, 2020, at 1:41 PM, Neale Ranns (nranns)  wrote:
> 
> 
> Hi Chris,
> 
> On 07/05/2020 16:55, "Christian Hopps"  wrote:
> 
> 
> 
>> On May 7, 2020, at 8:15 AM, Neale Ranns (nranns)  wrote:
>> 
>> 
>> Hi Chris,
>> 
>> They were replaced by ipip interfaces protected by SAs:
>>  https://wiki.fd.io/view/VPP/IPSec#Tunnel_Mode
>> 
>> the tunnel always adds encap. You can configure your SA to add additional 
>> encap if you want.
> 
>Ok, but that's not a replacement for the old functionality, right? The old 
> functionality had the SA tunnel represented as an unnumbered interface that 
> could be routed onto efficiently using the FIB. The unnumbered interface used 
> the SA tunnel endpoint addresses.
> 
> It is intended to be a replacement. If it's not then there's a problem and 
> I'll address it.

Great. :)

> 
> Both old an new have one critical thing in common, there's an interface and 
> an inbound and outbound SA.
> The old functionality had a ip-in-ip interface tightly coupled to two SAs, so 
> tight that there was a dedicated interface type, ipsecX, and the interface 
> and SAs were configured at the same time. In the new model that coupling is 
> loose; there's a ipip interface and its associated SAs, they are configured 
> independently and bound together.
> If you made the ipsecX unnumbered you now make the ipipX unnumbered. If you 
> had routes through ipsecX, now you route through IpipX.
> 
>The wiki shows the that SA tunnel mode is re-encapsulating the already 
> encapsulated IP-IP tunnel traffic. So now I have 3 IP headers instead of the 
> 2 IP headers as before?
> 
> If an SA is in transport mode it does not add headers, if it's in tunnel mode 
> it does. When a packet egresses the tunnel it is encapped by that tunnel, if 
> you don't want another set of IP headers then configure the SA that protects 
> the tunnel to be in transport mode. 

So with the old functionality the interface was unnumbered, and used the SA 
endpoint IPs for the it's encapsulating IP header. Using it with SA in tunnel 
mode did not produce 3 IP headers. That's what we need to fix I guess, it 
should only produce 2 IP headers, the user's and the SA tunnel IP header.

>Putting aside the wasted IP header bandwidth for the moment though, I 
> don't understand what's actually supposed to be happening here. What does the 
> configuration look like? I have an SA with endpoints (Local-IP,Remote-IP) and 
> I have user traffic with (User-SIP,User-DIP). Previously I had an unnumbered 
> interface that used the SAs (Local-IP,Remote-IP) for it's IP header. I then 
> routed traffic for (User-DIP) over that unnumbered interface. How does one 
> configure that with this ipip tunnel replacement?
> 
> create ipip src Local-IP dst remote-IP
> set int unnum ipip0 use Eth0
> ipsec sa add id 5 [crypto .. ] [integ ..] mode=transport
> ipsec sa add id 6 [crypto .. ] [integ ..] mode=transport
> ipsec tun protect ipip0 in 5 out 6
> set int state ipip0 up
> 
> ip route User-DIP/32 via ipip0
> 
> does that get you what you need?

No, using transport mode IPsec is not an option. It must be an SA in standard 
tunnel mode. And it can't have 3 IP headers. :)

Thanks,
Chris.


> 
>I did read through the Wiki and it seems that this change was motivated by 
> wanting to cleanup the IPsec API, but that hardly seems like justification to 
> eliminate the efficient use of an entire variant of commonly used IPsec 
> functionality.
> 
> Cleaning up the API was one motivation. It was a pain that each time we added 
> new attributes to SA creation (like ESN, UDP, algo=foo) (for use with the 
> SPD) we had to make similar changes to both the ipsec and ipsec_gre create 
> APIs. The other motivation was removing 2 interface types that did exactly 
> the same as the existing ipip and gre tunnels (and the same goes for their 
> APIs too, like how do I configure what DCSP, ECN, DF to copy on encap/decap).
> 
> /neale
> 
> 
> 
>Could we bring back the functionality using more "acceptable to the 
> project" APIs or something?
> 
>Thanks,
>Chris.
> 
>> 
>> /neale
>> 
>> 
>> From:  on behalf of Christian Hopps 
>> Date: Wednesday 6 May 2020 at 14:32
>> To: vpp-dev 
>> Cc: Christian Hopps 
>> Subject: [vpp-dev] IPsec tunnel interfaces?
>> 
>> Hi, vpp-dev,
>> 
>> Post 19.08 seems to have removed IPsec logical interfaces.
>> 
>> One cannot always use transport mode IPsec.
>> 
>> How can I get the efficiency of route based (FIB) IPsec w/o transport mode? 
>> Adding superfluous encapsulations (wasting bandwidth) to replace this 
>> (seemingly lost, hope not) functionality is not an option.
>> 
>> Thanks,
>> Chris.
>> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16277): https://lists.fd.io/g/vpp-dev/message/16277
Mute This Topic: https://lists.fd.io/mt/74027328/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

> On May 7, 2020, at 8:15 AM, Neale Ranns (nranns)  wrote:
> 
>
> Hi Chris,
>
> They were replaced by ipip interfaces protected by SAs:
>   https://wiki.fd.io/view/VPP/IPSec#Tunnel_Mode
>
> the tunnel always adds encap. You can configure your SA to add additional 
> encap if you want.

Ok, but that's not a replacement for the old functionality, right? The old 
functionality had the SA tunnel represented as an unnumbered interface that 
could be routed onto efficiently using the FIB. The unnumbered interface used 
the SA tunnel endpoint addresses.

The wiki shows the that SA tunnel mode is re-encapsulating the already 
encapsulated IP-IP tunnel traffic. So now I have 3 IP headers instead of the 2 
IP headers as before?

Putting aside the wasted IP header bandwidth for the moment though, I don't 
understand what's actually supposed to be happening here. What does the 
configuration look like? I have an SA with endpoints (Local-IP,Remote-IP) and I 
have user traffic with (User-SIP,User-DIP). Previously I had an unnumbered 
interface that used the SAs (Local-IP,Remote-IP) for it's IP header. I then 
routed traffic for (User-DIP) over that unnumbered interface. How does one 
configure that with this ipip tunnel replacement?

I did read through the Wiki and it seems that this change was motivated by 
wanting to cleanup the IPsec API, but that hardly seems like justification to 
eliminate the efficient use of an entire variant of commonly used IPsec 
functionality.

Could we bring back the functionality using more "acceptable to the project" 
APIs or something?

Thanks,
Chris.

>
> /neale
>
>
> From:  on behalf of Christian Hopps 
> Date: Wednesday 6 May 2020 at 14:32
> To: vpp-dev 
> Cc: Christian Hopps 
> Subject: [vpp-dev] IPsec tunnel interfaces?
>
> Hi, vpp-dev,
> 
> Post 19.08 seems to have removed IPsec logical interfaces.
> 
> One cannot always use transport mode IPsec.
> 
> How can I get the efficiency of route based (FIB) IPsec w/o transport mode? 
> Adding superfluous encapsulations (wasting bandwidth) to replace this 
> (seemingly lost, hope not) functionality is not an option.
> 
> Thanks,
> Chris.
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16272): https://lists.fd.io/g/vpp-dev/message/16272
Mute This Topic: https://lists.fd.io/mt/74027328/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] IPsec tunnel interfaces?

[Christian Hopps]

Hi, vpp-dev,

Post 19.08 seems to have removed IPsec logical interfaces.

One cannot always use transport mode IPsec.

How can I get the efficiency of route based (FIB) IPsec w/o transport mode? 
Adding superfluous encapsulations (wasting bandwidth) to replace this 
(seemingly lost, hope not) functionality is not an option.

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16247): https://lists.fd.io/g/vpp-dev/message/16247
Mute This Topic: https://lists.fd.io/mt/74027328/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] worker barrier state

[Christian Hopps]

> On May 4, 2020, at 3:59 AM, Neale Ranns via lists.fd.io 
>  wrote:
> 
> 
> Hi Chris,
> 
> With SAs there are two scenarios to consider for inflight packets
> 1) the SA is unlinked
> 2) the SA is deleted.
> 
> We've talked at length about how to deal with 2).
> By 'unlinked' I mean that whatever config dictated that an SA be used has now 
> gone (like tunnel protection or SPD policy). An inflight packet that is 
> processed by an IPSec node would (by either scheme we discussed for 1)) 
> retrieve the SA do encrypt/decrypt and then attempt to send the packet on its 
> merry way; this is the point at which it could fail. I say 'could' because it 
> depends on how the unlink affected the vlib graph. In today's tunnel 
> protection esp_encrpyt does vnet_feature_next(), this is not going to end 
> well once esp_encrypt is no longer a feature on the arc. In tomorrow's tunnel 
> protection we'll change that:
>   https://gerrit.fd.io/r/c/vpp/+/26265
> and it should be safe. But, what if the next API removes the tunnel whilst 
> there are still inflight packets? Is it still safe? Is it still correct to 
> send encrypted tunnelled packets?

It's safe to send in-flight encrypted packets for an SA that was deleted after 
they were encrypted, this reduces to buffering. After the SA is deleted new 
packets won't follow that path (they won't have the index+gen associate with 
them at all b/c of policy). If the user configures things such that packets 
which used to be encrypted should no longer encrypted, well then that's what 
they asked for for *newly arived* packets.

The security issue to be careful of here would be to have in-flight packets 
that progressed through valid policy such that they were "in the tunnel" and 
waiting to be encrypted, and then send them un-encrypted. Hopefully the change 
you reference above does not do that.

> 
> I think I'm coming round to the opinion that the safest way to approach this 
> is to ensure that if the SA can be found, whatever state it is in (created, 
> unlinked or deleted) then it needs to have a flag that states whether it 
> should be used or the packet dropped. We'd update this state when the SA is 
> [un]linked, with the barrier held.

Why does the SA object needs to be kept around? The indexes generation number 
being unequal is enough to say the object is deleted. Per the concern I 
mentioned above, we should make sure the packet doesn't *forget* that its been 
associated with that an index+generation number (i.e., it's in the tunnel), but 
that's orthogonal to whether the SA state itself is still in the pool.

> 
> On a somewhat related topic, you probably saw:
>  https://gerrit.fd.io/r/c/vpp/+/26811
> as an example of getting MP safe APIs wrong.

I made a similar change locally back when we started talking about this. :)

Thanks,
Chris.



> /neale
> 
> On 24/04/2020 16:34, "Christian Hopps"  wrote:
> 
> 
>Hi Neale,
> 
>    Comments also inline...
> 
>Neale Ranns (nranns)  writes:
> 
>> Hi Chris,
>> 
>> Comments inline...
>> 
>> On 15/04/2020 15:14, "Christian Hopps"  wrote:
>> 
>>Hi Neale,
>> 
>>I agree that something like 4, is probably the correct approach. I had a 
>> side-meeting with some of the ARM folks (Govind and Honnappa), and we 
>> thought using a generation number for the state rather than just waiting 
>> "long-enough" to recycle it could work. The generation number would be the 
>> atomic value associated with the state. So consider this API:
>> 
>> - MP-safe pools store generation numbers alongside each object.
>> - When you allocate a new object from the pool you get an index and 
>> generation number.
>> - When storing the object index you also save the generation number.
>> - When getting a pointer to the object you pass the API the index and 
>> generation number and it will return NULL if the generation number did not 
>> match the one stored with the object in the pool.
>> - When you delete a pool object its generation number is incremented 
>> (with barrier).
>> 
>>The size of the generation number needs to be large enough to guarantee 
>> there is no wrap with objects still in the system that have stored the 
>> generation number. Technically this is a "long-enough" aspect of the scheme. 
>> :) One could imagine using less than 64 bits for the combination of index 
>> and generation, if that was important.
>> 
>> It's a good scheme, I like it.
>> I assume the pool indices would be 64 bit and the separation between vector 
>> index and generation would be hidden from the user. Maybe a 32 bit value 
>> would suff

Re: [vpp-dev] 11th hour changes and code reviews

[Christian Hopps]

> On May 4, 2020, at 3:46 PM, Andrew Yourtchenko  wrote:
> More inline:
>> On 4 May 2020, at 19:20, Paul Vinciguerra  wrote:
>> 
>> I agree that the change I submitted is a monster.  Honestly, there are 2-5 
>> git-friendly changes that could be refactored out of it.  Its written and 
>> used by me, but it's not worth investing the time to make mergeable if there 
>> is no desire for it.
> 
> That’s why it’s handy to discuss the big changes that touch the APIs upfront, 
> with the end users. Just last week I saw a mail on how the past year there 
> were so many API changes, which make life harder. VPP’s reason for existence 
> (at least in my view) is its users. 
> 
> This change reshuffles the messages and changes semantics for a bunch of 
> calls for the sole purpose of architectural purity. Purity is beautiful but 
> the users deserve a heads up at least.
> 
>>   I only submitted it at all because I had mentioned last week to someone 
>> that I had essentially stopped contributing and I was asked if I could be 
>> enticed to start re-submitting code.  There is no intent on my part to 
>> subvert the release. 
>> 
>> Everyone complains about the tests.  I was "asked" last year to clean up the 
>> tests.  It's not that code hasn't been written to clean it up.  The reality 
>> is that code becomes a monster when simple refactorings aren't merged.
> 
> The tests need proper debugging first, *before* doing the refactorings. The 
> tests still can not run with high concurrent number of jobs. (N~48).
> The tests still fail intermittently.
> 
> This is what folks mean when when complaining. Rearranging the calling 
> convention doesn’t help this in the slightest, just adds unnecessary churn.
> 
>> 
>> If you all tell me you want the changes,  I'm glad to clean it up.
> 
> That change contains multiple pieces which you yourself described.
> 
> - dump by sw_if_index ? Sure. But again, not across the entire code base but 
> component by component (if there are many). Reviewed by the maintainer of the 
> component.

The fact that this might be a bit painful to do in parts is a nice feedback to 
the pain that the change causes downstream. :)

I like consistent APIs as much as the next person, maybe more so, but 
non-backward compatible changes in order to tidy up the API cause pain and 
maybe aren't so great.

That said, adding filter functionality is good, so if there wasn't the ability 
to filter on sw_if_index before, I'd vote for it.

Thanks,
Chris.

> 
> - changing the message structure ? Again, if there are multiple components, 
> it should not be hard making small changes that would be trivial to review 
> (though before that, like i said, it would be nice to prove the users and 
> subsystem maintainers what they think of the changes). Also how do we ensure 
> the “old” inconsistency doesn’t resurrect in the new messages ? If there is 
> no automated check against that, this is just adding churn.
> 
> - internal refactorings that have zero user impact. Why not. But again, if 
> you are going across components, each commit should be within a component, 
> and reviewed/committed by the maintainer of the component.
> 
> —a
> 
>> 
>> If, in the meantime, someone wants to +2 Neale's change [2], that would be 
>> great!
>> 
>> [0] https://gerrit.fd.io/r/c/vpp/+/26833
>> [1] https://lists.fd.io/g/csit-dev/message/4009
>> [2] https://gerrit.fd.io/r/c/vpp/+/26820/4 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16236): https://lists.fd.io/g/vpp-dev/message/16236
Mute This Topic: https://lists.fd.io/mt/73980355/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] route lookup api

[Christian Hopps]

After multiple amended commits to fix python style errors, a new version with 
v4/v6 API tests has been uploaded.

https://gerrit.fd.io/r/c/vpp/+/26829

Thanks,
Chris.

> On May 2, 2020, at 5:25 PM, Christian Hopps  wrote:
> 
> I was able to do a minimal test on it so I submitted, but please review as 
> well. :)
> 
> https://gerrit.fd.io/r/c/vpp/+/26829
> 
> Thanks,
> Chris.
> 
>> On Apr 28, 2020, at 12:08 PM, Jon Loeliger via lists.fd.io 
>>  wrote:
>> 
>> Chris,
>> 
>> If you can shoot me the patch, as is, I will rebase and submit it?
>> 
>> jdl
>> 
>> 
>> On Mon, Apr 27, 2020 at 7:27 PM Dave Wallace  wrote:
>> Gentle reminder that the VPP 20.05 API freeze is next Wed, May 6, 2020, so 
>> please don't delay to long if this is required for 20.05.
>> 
>> Thanks,
>> -daw- (wearing the friendly VPP release manager hat)
>> 
>> On 4/27/2020 5:02 PM, Christian Hopps wrote:
>>>> On Apr 27, 2020, at 11:44 AM, Jon Loeliger via lists.fd.io 
>>>> 
>>>> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Wed, Feb 19, 2020 at 4:47 AM Christian Hopps 
>>>> 
>>>> wrote:
>>>> 
>>>> 
>>>>> On Feb 19, 2020, at 2:02 AM, Neale Ranns via Lists.Fd.Io 
>>>>> 
>>>>> wrote:
>>>>> 
>>>>> 
>>>>> Hi Chris,
>>>>> 
>>>>> Adding an API to dump a single route would be a welcome addition to the 
>>>>> API.
>>>>> 
>>>> Ok, I'll do that then.
>>>> 
>>>> Hey Chris,
>>>> 
>>>> Did you ever get this API enhancement written?  If so, is there a Gerrit 
>>>> for it?
>>>> 
>>> I have this written/tested, but it lives in my 19.08 (product) branch right 
>>> now. I might have near the end of the week to cherry pick it to master and 
>>> test it there.
>>> 
>>> Thanks,
>>> Chris.
>>> 
>>> 
>>>> Thanks,
>>>> jdl
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16230): https://lists.fd.io/g/vpp-dev/message/16230
Mute This Topic: https://lists.fd.io/mt/71382541/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] route lookup api

[Christian Hopps]

I was able to do a minimal test on it so I submitted, but please review as 
well. :)

https://gerrit.fd.io/r/c/vpp/+/26829

Thanks,
Chris.

> On Apr 28, 2020, at 12:08 PM, Jon Loeliger via lists.fd.io 
>  wrote:
> 
> Chris,
> 
> If you can shoot me the patch, as is, I will rebase and submit it?
> 
> jdl
> 
> 
> On Mon, Apr 27, 2020 at 7:27 PM Dave Wallace  wrote:
> Gentle reminder that the VPP 20.05 API freeze is next Wed, May 6, 2020, so 
> please don't delay to long if this is required for 20.05.
> 
> Thanks,
> -daw- (wearing the friendly VPP release manager hat)
> 
> On 4/27/2020 5:02 PM, Christian Hopps wrote:
>>> On Apr 27, 2020, at 11:44 AM, Jon Loeliger via lists.fd.io 
>>> 
>>>  wrote:
>>> 
>>> 
>>> 
>>> On Wed, Feb 19, 2020 at 4:47 AM Christian Hopps 
>>> 
>>>  wrote:
>>> 
>>> 
>>>> On Feb 19, 2020, at 2:02 AM, Neale Ranns via Lists.Fd.Io 
>>>> 
>>>>  wrote:
>>>> 
>>>>
>>>> Hi Chris,
>>>>
>>>> Adding an API to dump a single route would be a welcome addition to the 
>>>> API.
>>>> 
>>> Ok, I'll do that then.
>>> 
>>> Hey Chris,
>>> 
>>> Did you ever get this API enhancement written?  If so, is there a Gerrit 
>>> for it?
>>> 
>> I have this written/tested, but it lives in my 19.08 (product) branch right 
>> now. I might have near the end of the week to cherry pick it to master and 
>> test it there.
>> 
>> Thanks,
>> Chris.
>> 
>> 
>>> Thanks,
>>> jdl
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16223): https://lists.fd.io/g/vpp-dev/message/16223
Mute This Topic: https://lists.fd.io/mt/71382541/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

bug in api generation [Re: [vpp-dev] packed enums]

[Christian Hopps]

The fallout is a bit worse than first anticipated.

I converted my code to use the vl_api_*_t_endian functions rather than doing 
the hton*/ntoh* by hand (since a bunch of them were now doing the wrong thing), 
and discovered another bigger problem. Unions are not handled correctly by the 
generated endian functions e.g.,:

static inline void vl_api_punt_t_endian (vl_api_punt_t *a)
{
int i __attribute__((unused));
vl_api_punt_type_t_endian(>type);
vl_api_punt_union_t_endian(>punt);
}

static inline void vl_api_punt_union_t_endian (vl_api_punt_union_t *a)
{
int i __attribute__((unused));
vl_api_punt_exception_t_endian(>exception);
vl_api_punt_l4_t_endian(>l4);
vl_api_punt_ip_proto_t_endian(>ip_proto);
}

The second function there is flipping bits in the same memory space (union) 
multiple times in different ways according to each structure in the union.

Anything that has a union in the API will suffer this same bug; the recent enum 
size changes to very common types (address family, ip proto, etc) merely cause 
this bug to be hit more ofetn.

Thanks,
Chris.

> On May 2, 2020, at 8:20 AM, Christian Hopps  wrote:
> 
> Some recent changes related to packed enums in the API have broken already 
> written code for low level API use. I think the main commit that did this was 
> the one that added __packed__ to the generated enum declaration, but there 
> are the related changes that add the enum size declarations I guess. Those 
> appear to be percolating slowly (e.g., in ip_types.api), although version 
> numbers are not being changed inline (e.g., "enum address_family" went from 
> default u32 to u8 but the ip_types.api version didn't change)
> 
> I understand that the API needs to be able to change, but I'm using like 
> maybe 7 or 8 API calls and I now have conditional code for seemingly every 
> release since 19.04. It's frustrating.
> 
> The project encourages running the latest code by moving support away from 
> old releases quickly, but then is making it hard to follow with API churn.
> 
> That said I would not be being bitten by this particular change I think if I 
> were using the higher level API; however, I have resisted this b/c that API 
> with synchronous calls requiring a callback for results is ... inelegant, to 
> put it nicely. :)
> 
> Thanks,
> Chris.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16222): https://lists.fd.io/g/vpp-dev/message/16222
Mute This Topic: https://lists.fd.io/mt/73940014/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] packed enums

[Christian Hopps]

Some recent changes related to packed enums in the API have broken already 
written code for low level API use. I think the main commit that did this was 
the one that added __packed__ to the generated enum declaration, but there are 
the related changes that add the enum size declarations I guess. Those appear 
to be percolating slowly (e.g., in ip_types.api), although version numbers are 
not being changed inline (e.g., "enum address_family" went from default u32 to 
u8 but the ip_types.api version didn't change)

I understand that the API needs to be able to change, but I'm using like maybe 
7 or 8 API calls and I now have conditional code for seemingly every release 
since 19.04. It's frustrating.

The project encourages running the latest code by moving support away from old 
releases quickly, but then is making it hard to follow with API churn.

That said I would not be being bitten by this particular change I think if I 
were using the higher level API; however, I have resisted this b/c that API 
with synchronous calls requiring a callback for results is ... inelegant, to 
put it nicely. :)

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16220): https://lists.fd.io/g/vpp-dev/message/16220
Mute This Topic: https://lists.fd.io/mt/73933468/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] route lookup api

[Christian Hopps]

> On Apr 27, 2020, at 11:44 AM, Jon Loeliger via lists.fd.io 
>  wrote:
> 
> 
> 
> On Wed, Feb 19, 2020 at 4:47 AM Christian Hopps  wrote:
> 
> > On Feb 19, 2020, at 2:02 AM, Neale Ranns via Lists.Fd.Io 
> >  wrote:
> > 
> >
> > Hi Chris,
> >
> > Adding an API to dump a single route would be a welcome addition to the API.
> 
> Ok, I'll do that then.
> 
> Hey Chris,
> 
> Did you ever get this API enhancement written?  If so, is there a Gerrit for 
> it?

I have this written/tested, but it lives in my 19.08 (product) branch right 
now. I might have near the end of the week to cherry pick it to master and test 
it there.

Thanks,
Chris.

> Thanks,
> jdl
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16164): https://lists.fd.io/g/vpp-dev/message/16164
Mute This Topic: https://lists.fd.io/mt/71382541/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] worker barrier state

[Christian Hopps]

Hi Neale,

Comments also inline...

Neale Ranns (nranns)  writes:


Hi Chris,

Comments inline...

On 15/04/2020 15:14, "Christian Hopps"  wrote:

Hi Neale,

I agree that something like 4, is probably the correct approach. I had a side-meeting 
with some of the ARM folks (Govind and Honnappa), and we thought using a generation 
number for the state rather than just waiting "long-enough" to recycle it could 
work. The generation number would be the atomic value associated with the state. So 
consider this API:

 - MP-safe pools store generation numbers alongside each object.
 - When you allocate a new object from the pool you get an index and 
generation number.
 - When storing the object index you also save the generation number.
 - When getting a pointer to the object you pass the API the index and 
generation number and it will return NULL if the generation number did not 
match the one stored with the object in the pool.
 - When you delete a pool object its generation number is incremented (with 
barrier).

The size of the generation number needs to be large enough to guarantee there is no 
wrap with objects still in the system that have stored the generation number. Technically 
this is a "long-enough" aspect of the scheme. :) One could imagine using less 
than 64 bits for the combination of index and generation, if that was important.

It's a good scheme, I like it.
I assume the pool indices would be 64 bit and the separation between vector 
index and generation would be hidden from the user. Maybe a 32 bit value would 
suffice in most cases, but why skimp...


I was thinking to keep the index and generation number separate at the most 
basic API, to allow for selecting the size of the each independently and for 
efficient storage. I'm thinking for some applications one might want to do 
something like

cacheline_packed_struct {
   ...
   u32 foo_index;
   u32 bar_index;
   u16 foo_gen;
   u16 bar_gen;
   ...
};

a set of general purpose macros could be created for combining the 2 values 
into a single integer value though.


The advantage over just waiting N seconds to recycle the index is that the 
system scales better, i.e., if you just wait N seconds to reuse, and are 
creating and deleting objects at a significant rate, your pool can blow up in 
the N seconds of time. With the generation number this is not a problem as you 
can re-use the object immediately. Another advantage is that you don't have to 
have the timer logic (looping per pool or processing all pools) to free up old 
indices.

Yes, for my time based scheme, the size of the pool will become dependent on 
some integration over a rate of change, which is not deterministic, which is 
not great, but I don't suppose all APIs are subject to large churn.
With the generation scheme the pool always requires more memory, since you're 
storing a generation value for each index, but being a deterministic size (even 
though probably bigger), I'd probably take that.
I wouldn't use timer logic in my scheme. I'd make the pool's free-list a fifo (as 
opposed to the stack it is today) and each entry in the list has the index and the 
time it was added. If t_now - t_head < t_wrap I can pull from the free-list, 
else the pool needs to grow.


FWIW, I think the space used for the timestamp would be similar to the space 
used for the generation number, probably it's a wash.


The generation number scheme will still need the thread barrier to 
increment the generation number to make sure no-one is using the object in 
parallel. But this is a common problem with deleting non-reference-counted 
shared state I believe.

I don't think you strictly need the barrier, you can still use a 
make-before-break update. One downside of the generation approach is that nodes 
that try and fetch the state using the index will get NULL, so the only option 
is to drop, as opposed to what the make-before-break change determined. Mind 
you, this is probably fine for most practical purposes. Again if we're talking 
SAs, then at this point the SA is decoupled from the graph (i.e. it's no longer 
protecting the tunnel or it's not linked to a policy in the SPD), so drop is 
all we can do anyway.


You need the barrier to make sure no in-flight packets are concurrently using 
the object the index/gen pointed at. Strictly speaking, you could increment the 
generation number w/o the barrier, but then you have to hold the barrier during 
free/re-use of the pool object. The beauty of the generation number is that you 
only hold the barrier while you increment it, and the free/re-use of the object 
is done outside the barrier.


I see it as a trade-off between a cost for every packet forwarded versus how 
many may be dropped during API calls. I wouldn't want the scheme employed to 
ensure safe delete to affect the overall packet through put - most of the time 
I'm not changing the state...

Now we have a fe

[vpp-dev] Packet processing time.

[Christian Hopps]

The recent discussion on reference counting and barrier timing has got me 
interested in packet processing time. I realize there's a way to use "show 
runtime" along with knowledge of the arc a packet follows, but I'm curious if 
something more straight-forward has been attempted where packets are 
timestamped on ingress (or creation) and stats are collected on egress 
(transmission)?

I also have an unrelated interest in hooking into the graph 
immediate-post-transmission -- I'd like to adjust an input queue size only when 
the packet that enqueued on it is actually transmitted on the wire, and not 
just handed off downstream on the arc -- this would be a likely the same place 
packet stat collection might occur. :)

Thanks,
Chris.


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16110): https://lists.fd.io/g/vpp-dev/message/16110
Mute This Topic: https://lists.fd.io/mt/73114130/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Checkstyle script not work in ubuntu

[Christian Hopps]

+1 for clang format.

Regarding the in tree .clang-format, I had to use my own .clang_format settings 
though as the VPP/C++ version has different tab defaults from the indent 
default currently used in VPP (never use and 4 space vs always use and 8 space).

Thanks,
Chris.

> On Apr 18, 2020, at 7:55 AM, Klement Sekera via lists.fd.io 
>  wrote:
> 
> clang-format can be tuned to emulate indent - it’s not 100% perfect match, 
> but I’ve been using it for some time to format multi-line macros, e.g. 
> pool_foreach and it’s been doing a pretty good job. Config file for that is 
> already in vpp source tree (vpp/.clang-format) and used as default for cpp 
> code formatting.
> 
>> On 18 Apr 2020, at 13:49, Dave Barach via lists.fd.io 
>>  wrote:
>> 
>> +1, this seems like a viable scheme to me.
>> 
>> We’ll need to configure the underlying indent engine so that newly-indented 
>> code looks as much like the rest of the code as possible.
>> 
>> The result below wouldn’t preclude automatic cherry-picking, but it would 
>> make everyone’s head explode, particularly if one’s favorite code editor 
>> likes to “fix” such things:
>> 
>> if (a)
>>  {
>>b = 13;
>>c = 12;
>>/* new code */
>>if(d) {
>>e=this_is_new();
>>}
>>/* end new code */
>>  }
>> 
>> Thanks... Dave
>> 
>> From: Damjan Marion  
>> Sent: Saturday, April 18, 2020 5:51 AM
>> To: Andrew Yourtchenko 
>> Cc: Dave Barach (dbarach) ; Zhang Yuwei 
>> ; vpp-dev@lists.fd.io
>> Subject: Re: [vpp-dev] Checkstyle script not work in ubuntu
>> 
>> 
>> And this is example of script, which just formats modified lines, instead of 
>> re-formating whole file, as we do today.
>> With something like this, we can introduce new indent or even move to 
>> clang-format without the need to reformat old code….
>> 
>> https://github.com/llvm-mirror/clang/blob/master/tools/clang-format/clang-format-diff.py
>> 
>> — 
>> Damjan
>> 
>> 
>> On 18 Apr 2020, at 11:00, Damjan Marion via lists.fd.io 
>>  wrote:
>> 
>> 
>> If we decided to stick with old indent, which i still disagree that is right 
>> thing to do, can you just compile indent all the time and 
>> modify path so /opt/vpp/…/bin/ comes first. I really don’t like one more 
>> option in the top level Makefile.
>> 
>> — 
>> Damjan
>> 
>> 
>> On 18 Apr 2020, at 10:29, Andrew Yourtchenko  wrote:
>> 
>> I made https://gerrit.fd.io/r/#/c/vpp/+/22963/ that you can try and see how 
>> it works for you.
>> 
>> It allows to install the “correct” version of indent into the build tree, so 
>> the rest of the system is unaffected.
>> 
>> --a
>> 
>> 
>> On 11 Apr 2020, at 14:04, Dave Barach via lists.fd.io 
>>  wrote:
>> 
>> 
>> The script works fine. You have the wrong version of gnu indent installed. 
>> This is the version you need:
>> 
>> $ indent --version
>> GNU indent 2.2.11
>> 
>> From: vpp-dev@lists.fd.io  On Behalf Of Zhang Yuwei
>> Sent: Saturday, April 11, 2020 1:04 AM
>> To: vpp-dev@lists.fd.io
>> Subject: [vpp-dev] Checkstyle script not work in ubuntu
>> 
>> Hi Guys,
>>I find checkstyle script doesn’t work normally in ubuntu 
>> sometimes that I run make fixstyle in ubuntu and submit the code to gerrit 
>> but still fail in checkstyle step. I need to move to centos to make it work, 
>> can anybody check this? Thanks a lot.
>> 
>> Regards,
>> Yuwei
>> 
>> 
>> 
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16107): https://lists.fd.io/g/vpp-dev/message/16107
Mute This Topic: https://lists.fd.io/mt/72939086/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] worker barrier state

[Christian Hopps]

Perhaps useful noting that our meeting was about a month ago, and had focused 
on the similar "4" like approach (albeit using the generation number) 
independently as well.

I'd still like to get some more timing information on barrier timings vs 
ref-counting though. Are a few clocks for reference counting too much or are 
they a worthwhile cost to avoiding the expensive barrier?

Are there any published numbers available on the barrier timing? I'm rather 
worried about the barrier and how long it can stop all packet processing. I 
believe worst case is the longest running node in the system time + the amount 
of time the barrier is held (i.e., whatever is going on while inside the 
barrier) + time to fix the graph if it was changed.

Even if the graph isn't changing, and the code in the barrier is fast, you 
still have to wait for the slowest node in the graph to complete as well, and 
that could be a very long time to not be processing packets.

Unless I'm mistaken, b/c the barrier must wait for the slowest node, and then 
that node (or another similarly clocked node) must make up for not running, 
your maximum loss free processing rate is always less than 1/2 the send 
capability of your slowest node. Stated another way, 1/2 a VLIB_FRAME_SIZE (128 
normally) worth of packet time on your slowest node must be reserved for the 
barrier or you are dropping packets.

Thanks,
Chris.

> On Apr 15, 2020, at 9:13 AM, Christian Hopps  wrote:
> 
> Hi Neale,
> 
> I agree that something like 4, is probably the correct approach. I had a 
> side-meeting with some of the ARM folks (Govind and Honnappa), and we thought 
> using a generation number for the state rather than just waiting 
> "long-enough" to recycle it could work. The generation number would be the 
> atomic value associated with the state. So consider this API:
> 
> - MP-safe pools store generation numbers alongside each object.
> - When you allocate a new object from the pool you get an index and 
> generation number.
> - When storing the object index you also save the generation number.
> - When getting a pointer to the object you pass the API the index and 
> generation number and it will return NULL if the generation number did not 
> match the one stored with the object in the pool.
> - When you delete a pool object its generation number is incremented (with 
> barrier).
> 
> The size of the generation number needs to be large enough to guarantee there 
> is no wrap with objects still in the system that have stored the generation 
> number. Technically this is a "long-enough" aspect of the scheme. :) One 
> could imagine using less than 64 bits for the combination of index and 
> generation, if that was important.
> 
> The advantage over just waiting N seconds to recycle the index is that the 
> system scales better, i.e., if you just wait N seconds to reuse, and are 
> creating and deleting objects at a significant rate, your pool can blow up in 
> the N seconds of time. With the generation number this is not a problem as 
> you can re-use the object immediately. Another advantage is that you don't 
> have to have the timer logic (looping per pool or processing all pools) to 
> free up old indices.
> 
> The generation number scheme will still need the thread barrier to increment 
> the generation number to make sure no-one is using the object in parallel. 
> But this is a common problem with deleting non-reference-counted shared state 
> I believe.
> 
> When you mentioned packet counters, that's really a reference count I guess. 
> The trade-off here seems to me to be 2 cache-line-invalidates per packet 
> (once on ingress once on egress) for the counter vs a barrier hit (all packet 
> processing stops) per delete of the state. For your setup that you measured 
> the packet counter solution how long does it spend from the barrier sync 
> request to release (i.e., how long is the system not processing packets)?
> 
> Thanks,
> Chris.
> 
>> On Apr 15, 2020, at 5:38 AM, Neale Ranns (nranns)  wrote:
>> 
>> 
>> Hi Chris,
>> 
>> Firstly, apologies for the lengthy delay. 
>> 
>> When I say 'state' in the following I'm referring to some object[s] that are 
>> used to forward packets. 
>> 
>> I'd classify the possible solution space as:
>> 1) maintain per-packet counters for the state to indicate how many packets 
>> currently refer to that state.
>>Pros; we know exactly when the state is no longer required and can be 
>> safely removed.
>>Cons; significant per-packet cost, similar to maintaining counters. For 
>> reference, on my [aging] system enabling adjacency counters takes 
>> ip4-rewrite from 2.52e1 to 3.49e1 clocks. The wait times could be large 
>> 

Re: [vpp-dev] worker barrier state

[Christian Hopps]

irst and second steps are very much dependent on the type 
> of state we're talking about. For SAs for example, unlinking the SA from the 
> lookup data-structure is accomplished using a separate API from the SA 
> delete*. The final step we can easily accomplish with a new version of the 
> pool allocator whose free-list prevents reuse for say 5 seconds (an age in DP 
> terms).
> 
> Thoughts?
> 
> /neale
> 
> * I note that a SA delete is already (optimistically) marked MP safe, which 
> assumes the system flushes inbetween these API calls.
> 
> 
> 
> 
> On 26/03/2020 16:09, "Christian Hopps"  wrote:
> 
> 
> 
>> On Mar 25, 2020, at 1:39 PM, Dave Barach via Lists.Fd.Io 
>>  wrote:
>> 
>> Vlib_main_t *vm->main_loop_count.
>> 
>> One trip around the main loop accounts for all per-worker local graph edges 
>> / acyclic graph behaviors. 
>> 
>> As to the magic number E (not to be confused with e): repeatedly handing off 
>> packets from thread to thread seems like a bad implementation strategy. The 
>> packet tracer will tell you how many handoffs are involved in a certain 
>> path, as will a bit of code inspection.
> 
>No, it would not be a good implementation strategy. :)
> 
>However, I was looking at trying to code this in an upstreamable way, and 
> I didn't think I got to make assumptions about how others might wire things 
> together. I suppose we could just define a maximum number of handoffs and 
> then if users violated that number they would need to increase it?
> 
>> Neale has some experience with this scenario, maybe he can share some 
>> thoughts...
> 
>Hoping so. :)
> 
>I noticed that crypto engine handoffs were added to the non-dpdk ipsec 
> encrypt/decrypt in master, which seems somewhat relevant.
> 
>Thanks,
>Chris.
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16085): https://lists.fd.io/g/vpp-dev/message/16085
Mute This Topic: https://lists.fd.io/mt/72542383/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] n_vectors...

[Christian Hopps]

> On Mar 31, 2020, at 9:20 AM, Elias Rudberg  wrote:
> 
> Hi Chris and Dave,
> 
> Thanks for bringing this up, and thanks for explaining!
> 
> I agree with Chris that this is confusing, it makes it much more
> difficult to understand the code.

As someone who UTSL as a first, second and third setp, I tend to agree. It made 
things harder then they probably needed to be when I was trying to grok the 
gestalt.

Lot of history though, hard to change (or even ask) for something like this.

A happy start for me would be if "show runtime" didn't call packets "Vectors". 
And, if someone repurposed vlib for non-packets they could change "Packets" to 
"Elements" or whatever they want :) Of course I did this in our local branch, 
but wasn't planning on trying to upstream.

Thanks,
Chris.

> 
> Perhaps this is the kind of thing that doesn't matter much to those who
> are already familiar with the code, while at the same time it matters a
> lot for newcomers. If you want to lower the threshold for new people to
> be able to come in and understand the code and possibly contribute,
> then I think it would be a good idea to fix this even if it means
> changing many lines of code. It could be argued that the fact that
> "n_vectors" exists in so many places makes it even more important to
> have a reasonable name for it. One way could be to start with renaming
> things in some of the main data structures like those in vlib/node.h
> and vlib/threads.h and such places, and the changes the compiler will
> force as a result of that.
> 
> Best regards,
> Elias
> 
> 
> On Tue, 2020-03-31 at 00:45 +, Dave Barach via Lists.Fd.Io wrote:
>> Hmmm, yeah. Been at this for years, I can’t really remember when we
>> settled on e.g. n_vectors vs. n_vector_elts or some such.
>> 
>> In new code, it’s perfectly fair to use whatever names seem fit for
>> purpose.
>> 
>> Vlib would be happy doing image processing, or any other kind of
>> vector processing. There’s no law which says that frames need to have
>> 32-bit elements. Each node decides.
>> 
>> FWIW... Dave
>> 
>> From: vpp-dev@lists.fd.io  On Behalf Of
>> Christian Hopps
>> Sent: Monday, March 30, 2020 8:07 PM
>> To: vpp-dev 
>> Cc: Christian Hopps 
>> Subject: [vpp-dev] n_vectors...
>> 
>> Something has always bothered me about my understanding of VPPs use
>> of the term "vector" and "vectors". When I think of Vector Packet
>> Processing I think of processing a vector (array) of packets in a
>> single call to a node. The code, though, then seems to refer to the
>> individual packets as "vectors" when it uses field names like
>> "n_vectors" to refer to the number of buffers in a frame, or when
>> "show runtime" talks about "vectors per call", when I think it's
>> really talking about "packets/buffers per call" (and my mind wants to
>> think that it's always *1* vector/frame of packets per call by
>> design).
>> 
>> I find this confusing, and so I thought I'd ask if there was some
>> meaning here I'm missing?
>> 
>> Thanks,
>> Chris.
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15959): https://lists.fd.io/g/vpp-dev/message/15959
Mute This Topic: https://lists.fd.io/mt/72667316/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] n_vectors...

[Christian Hopps]

Something has always bothered me about my understanding of VPPs use of the term 
"vector" and "vectors". When I think of Vector Packet Processing I think of 
processing a vector (array) of packets in a single call to a node. The code, 
though, then seems to refer to the individual packets as "vectors" when it uses 
field names like "n_vectors" to refer to the number of buffers in a frame, or 
when "show runtime" talks about "vectors per call", when I think it's really 
talking about "packets/buffers per call" (and my mind wants to think that it's 
always *1* vector/frame of packets per call by design).

I find this confusing, and so I thought I'd ask if there was some meaning here 
I'm missing?

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15950): https://lists.fd.io/g/vpp-dev/message/15950
Mute This Topic: https://lists.fd.io/mt/72667316/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp_papi not updated

[Christian Hopps]

Right but that's for running the unit tests right? I'm not running unit tests, 
I'm actually just using the python API in another application that 
controls/monitors VPP.

Thanks,
Chris.

> On Mar 27, 2020, at 6:45 PM, Paul Vinciguerra  
> wrote:
> 
> From the test/Makefile:
>   $(PYTHON_INTERP) -m pip install -e $(PAPI_PYTHON_SRC_DIR)"
> 
> On Fri, Mar 27, 2020 at 6:42 PM Christian Hopps  wrote:
> Hmm, at least on 19.08 doing the papi-wipe/wipe-papi is only removing things 
> under test. I'm running an external python app (not under test) and it's 
> looking in the install directory which isn't updated after the wipe-papi.
> 
> I found a hack/workaround which is to touch the CMakeLists.txt file in 
> src/vpp-api/python, though. That CMakeLists.txt just has an install command 
> so I guess it's not setting up any dependencies?
> 
> Thanks,
> Chris.
> 
> > On Mar 27, 2020, at 6:30 PM, Paul Vinciguerra  
> > wrote:
> > 
> > try 'test-wipe-papi'. 
> > You may need:
> > 
> > https://gerrit.fd.io/r/c/vpp/+/26177
> > 
> > On Fri, Mar 27, 2020 at 6:26 PM Christian Hopps  wrote:
> > [on 19.08]
> > 
> > I'm seeing something odd building. When I "make build" after changing 
> > something in "src/vpp-api/python/vpp_papi/*.py" it does not get reinstalled 
> > in the build-root. In fact if I delete 
> > build-root/install-vpp_debug-native/lib64/python3.6/site-packages/vpp_papi* 
> > it also does not get reinstalled with a "make build".
> > 
> > I guess vpp_papi is not fully integrated into the cmake build system? Short 
> > of fixing that, is there a way to force it to reinstall?
> > 
> > Thanks,
> > Chris.
> > 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15914): https://lists.fd.io/g/vpp-dev/message/15914
Mute This Topic: https://lists.fd.io/mt/72599377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp_papi not updated

[Christian Hopps]

Hmm, at least on 19.08 doing the papi-wipe/wipe-papi is only removing things 
under test. I'm running an external python app (not under test) and it's 
looking in the install directory which isn't updated after the wipe-papi.

I found a hack/workaround which is to touch the CMakeLists.txt file in 
src/vpp-api/python, though. That CMakeLists.txt just has an install command so 
I guess it's not setting up any dependencies?

Thanks,
Chris.

> On Mar 27, 2020, at 6:30 PM, Paul Vinciguerra  
> wrote:
> 
> try 'test-wipe-papi'. 
> You may need:
> 
> https://gerrit.fd.io/r/c/vpp/+/26177
> 
> On Fri, Mar 27, 2020 at 6:26 PM Christian Hopps  wrote:
> [on 19.08]
> 
> I'm seeing something odd building. When I "make build" after changing 
> something in "src/vpp-api/python/vpp_papi/*.py" it does not get reinstalled 
> in the build-root. In fact if I delete 
> build-root/install-vpp_debug-native/lib64/python3.6/site-packages/vpp_papi* 
> it also does not get reinstalled with a "make build".
> 
> I guess vpp_papi is not fully integrated into the cmake build system? Short 
> of fixing that, is there a way to force it to reinstall?
> 
> Thanks,
> Chris.
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15912): https://lists.fd.io/g/vpp-dev/message/15912
Mute This Topic: https://lists.fd.io/mt/72599377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] vpp_papi not updated

[Christian Hopps]

[on 19.08]

I'm seeing something odd building. When I "make build" after changing something 
in "src/vpp-api/python/vpp_papi/*.py" it does not get reinstalled in the 
build-root. In fact if I delete 
build-root/install-vpp_debug-native/lib64/python3.6/site-packages/vpp_papi* it 
also does not get reinstalled with a "make build".

I guess vpp_papi is not fully integrated into the cmake build system? Short of 
fixing that, is there a way to force it to reinstall?

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15910): https://lists.fd.io/g/vpp-dev/message/15910
Mute This Topic: https://lists.fd.io/mt/72599377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] worker barrier state

[Christian Hopps]

> On Mar 25, 2020, at 1:39 PM, Dave Barach via Lists.Fd.Io 
>  wrote:
> 
> Vlib_main_t *vm->main_loop_count.
> 
> One trip around the main loop accounts for all per-worker local graph edges / 
> acyclic graph behaviors. 
> 
> As to the magic number E (not to be confused with e): repeatedly handing off 
> packets from thread to thread seems like a bad implementation strategy. The 
> packet tracer will tell you how many handoffs are involved in a certain path, 
> as will a bit of code inspection.

No, it would not be a good implementation strategy. :)

However, I was looking at trying to code this in an upstreamable way, and I 
didn't think I got to make assumptions about how others might wire things 
together. I suppose we could just define a maximum number of handoffs and then 
if users violated that number they would need to increase it?

> Neale has some experience with this scenario, maybe he can share some 
> thoughts...

Hoping so. :)

I noticed that crypto engine handoffs were added to the non-dpdk ipsec 
encrypt/decrypt in master, which seems somewhat relevant.

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15883): https://lists.fd.io/g/vpp-dev/message/15883
Mute This Topic: https://lists.fd.io/mt/72542383/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] worker barrier state

[Christian Hopps]

I'm not clear on what you mean by table add/del, but I can give you the 
scenario I'm concerned with.

I have a packet P input and it has some state S associated with it.

The API wants to delete state S. When is it safe?

Say P's arc from input to output contains E edges. Each node on the arc could 
conceivably handoff packet P to another worker for processing. So if I read 
things correctly I need to wait at least E laps until I know for sure that P is 
out of the system, and S is safe to delete.

Q: How do I know what value E is?

I am not in control of all nodes along a P's arc and how they might handoff 
packets, and the graph is not acyclic so I couldn't even use a max value like 
the total number of nodes in the graph for E as the packet may loop back.

Q: Which lap counter am I looking at?

As you point out each vlib_main_t has it's own counter (main_loop_count?) so I 
think I have to record every workers main_loop_count in the state S and wait 
for every counter to  be +E before deleting S.

Thanks for the help!

Chris.

> On Mar 25, 2020, at 12:15 PM, Dave Barach (dbarach)  wrote:
> 
> +1. 
>
> View any metadata subject to table add/del accidents with suspicion. There is 
> a safe delete paradigm: each vlib_main_t has a “lap counter”.  When deleting 
> table entries: atomically update table entries. Record the lap counter and 
> wait until all worker threads have completed a lap. Then, delete (or 
> pool_put) the underlying data structure.
>
> Dave
>
>
> From: vpp-dev@lists.fd.io  On Behalf Of Damjan Marion 
> via Lists.Fd.Io
> Sent: Wednesday, March 25, 2020 12:10 PM
> To: Christian Hopps 
> Cc: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] worker barrier state
>
> 
> 
> > On 25 Mar 2020, at 16:01, Christian Hopps  wrote:
> > 
> > Is it supposed to be the case that no packets are inflight (*) in the graph 
> > when the worker barrier is held?
> > 
> > I think perhaps MP unsafe API code is assuming this.
> > 
> > I also think that the frame queues used by handoff code violate this 
> > assumption.
> > 
> > Can someone with deep VPP knowledge clarify this for me? :)
> 
> 
> correct, there is small chance that frame is enqueued right before worker 
> hits barrier…
> 
> — 
> Damjan
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15873): https://lists.fd.io/g/vpp-dev/message/15873
Mute This Topic: https://lists.fd.io/mt/72542383/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] worker barrier state

[Christian Hopps]

Is it supposed to be the case that no packets are inflight (*) in the graph 
when the worker barrier is held?

I think perhaps MP unsafe API code is assuming this.

I also think that the frame queues used by handoff code violate this assumption.

Can someone with deep VPP knowledge clarify this for me? :)

Thanks,
Chris.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15867): https://lists.fd.io/g/vpp-dev/message/15867
Mute This Topic: https://lists.fd.io/mt/72542383/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-