Re: [vpp-dev] Linking DPDK libs to plugins

[bjeremy32]

I believe it was just ring that they cared about.

-Original Message-
From: Damjan Marion  
Sent: Monday, October 25, 2021 11:08 AM
To: bjerem...@gmail.com
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Linking DPDK libs to plugins


Ok, i’m affraid that to implement this you will need to introduce lot of mess.
At the end probably will be easier to implement that functionality natively.

Which exact implementation of the dpdk mempool you are looking to use (ring, 
stack, bucket, ...)?

— 
Damjan



> On 25.10.2021., at 17:39,   wrote:
> 
> Hi Damjan,
> 
> Thanks for the reply
> 
> Here are the  details:
> 
> 1. We want to use only the rte_mempool infrastructure for lockless global 
> memory pools. We will not be using any mbuf infrastructure from dpdk
> 2. We want to use this infra across our multiple plugins
> 3. We want to be able to include rte_mempool data structures from our 
> multiple header files (.h files )
> 4. We want to be able to make calls to rte_mempool apis from our source code 
> ( .c files )
> 
> -Original Message-
> From: Damjan Marion  
> Sent: Monday, October 25, 2021 5:22 AM
> To: bjerem...@gmail.com
> Cc: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] Linking DPDK libs to plugins
> 
> 
> 
> 
>> On 25.10.2021., at 01:13, bjerem...@gmail.com wrote:
>> 
>> Greetings,
>> 
>> Let me preface this by saying that I really do not know much about the CMake 
>> utility. But I am trying to see if there is a way to make the DPDK libs 
>> accessible to other plugins (aside from the dpdk plugin) that are in their 
>> own project/subdirectory similar. I am working with v20.05 currently 
>> (although we are upgrading to 21.06 if that make a difference).
>> 
>> Initially it was suggested to me that I could just add a couple lines 
>> to my CMakeLists to link the dpdk_plugin.so to my own plugin.. but I 
>> have not been able to get this to work.. It never seems to recognize 
>> the path to the .so, even if I give the absolute path
>> 
>> set(DPDK_PLUGIN_LINK_FLAGS "${DPDK_PLUGIN_LINK_FLAGS} -L > plugins> -ldpdk_plugin.so")
>> 
>> add_vpp_plugin(my_plugin
>> ….
>>  LINK_FLAGS
>>  “${ DPDK_PLUGIN_LINK_FLAGS }”
>> 
>> Another approach suggested was to maybe use dlsym to dynamically load 
>> symbols… Anyway, I was thinking that someone has to have had done this 
>> before, or maybe have more of a clue as to how to do this then I currently 
>> do.
>> 
> 
> 
> 
> Please note that VPP is not DPDK application, DPDK is just optional device 
> driver layer for us.
> 
> Even if you manage to get your plugin linked against DPDK libs, there is no 
> guarantee that you will be able to use all dpdk data structures. Most obvious 
> example, rte_mbuf structure for a packet buffer may not be populated for you.
> 
> Also use of DPDK comes with a performance cost, we need to copy buffer 
> metadata left and right on both RX and TX side.
> 
> What specific DPDK library you would like to use? We may have alternative 
> proposal….
> 
> —
> Damjan
> 
> 



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20388): https://lists.fd.io/g/vpp-dev/message/20388
Mute This Topic: https://lists.fd.io/mt/86565585/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Linking DPDK libs to plugins

[bjeremy32]

Hi Damjan,

Thanks for the reply

Here are the  details:

1. We want to use only the rte_mempool infrastructure for lockless global 
memory pools. We will not be using any mbuf infrastructure from dpdk
2. We want to use this infra across our multiple plugins
3. We want to be able to include rte_mempool data structures from our multiple 
header files (.h files )
4. We want to be able to make calls to rte_mempool apis from our source code ( 
.c files )

-Original Message-
From: Damjan Marion  
Sent: Monday, October 25, 2021 5:22 AM
To: bjerem...@gmail.com
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Linking DPDK libs to plugins




> On 25.10.2021., at 01:13, bjerem...@gmail.com wrote:
> 
> Greetings,
>  
> Let me preface this by saying that I really do not know much about the CMake 
> utility. But I am trying to see if there is a way to make the DPDK libs 
> accessible to other plugins (aside from the dpdk plugin) that are in their 
> own project/subdirectory similar. I am working with v20.05 currently 
> (although we are upgrading to 21.06 if that make a difference).
>  
> Initially it was suggested to me that I could just add a couple lines 
> to my CMakeLists to link the dpdk_plugin.so to my own plugin.. but I 
> have not been able to get this to work.. It never seems to recognize 
> the path to the .so, even if I give the absolute path
>  
> set(DPDK_PLUGIN_LINK_FLAGS "${DPDK_PLUGIN_LINK_FLAGS} -L  plugins> -ldpdk_plugin.so")
>  
> add_vpp_plugin(my_plugin
> ….
>   LINK_FLAGS
>   “${ DPDK_PLUGIN_LINK_FLAGS }”
>  
> Another approach suggested was to maybe use dlsym to dynamically load 
> symbols… Anyway, I was thinking that someone has to have had done this 
> before, or maybe have more of a clue as to how to do this then I currently do.
>  



Please note that VPP is not DPDK application, DPDK is just optional device 
driver layer for us.

Even if you manage to get your plugin linked against DPDK libs, there is no 
guarantee that you will be able to use all dpdk data structures. Most obvious 
example, rte_mbuf structure for a packet buffer may not be populated for you.

Also use of DPDK comes with a performance cost, we need to copy buffer metadata 
left and right on both RX and TX side.

What specific DPDK library you would like to use? We may have alternative 
proposal….

—
Damjan



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20381): https://lists.fd.io/g/vpp-dev/message/20381
Mute This Topic: https://lists.fd.io/mt/86565585/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Linking DPDK libs to plugins

[bjeremy32]

Greetings,

 

Let me preface this by saying that I really do not know much about the CMake
utility. But I am trying to see if there is a way to make the DPDK libs
accessible to other plugins (aside from the dpdk plugin) that are in their
own project/subdirectory similar. I am working with v20.05 currently
(although we are upgrading to 21.06 if that make a difference).

 

Initially it was suggested to me that I could just add a couple lines to my
CMakeLists to link the dpdk_plugin.so to my own plugin.. but I have not been
able to get this to work.. It never seems to recognize the path to the .so,
even if I give the absolute path

 

set(DPDK_PLUGIN_LINK_FLAGS "${DPDK_PLUGIN_LINK_FLAGS} -L  -ldpdk_plugin.so")

 

add_vpp_plugin(my_plugin

..

  LINK_FLAGS

  "${ DPDK_PLUGIN_LINK_FLAGS }"

 

Another approach suggested was to maybe use dlsym to dynamically load
symbols. Anyway, I was thinking that someone has to have had done this
before, or maybe have more of a clue as to how to do this then I currently
do.

 

 

 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20374): https://lists.fd.io/g/vpp-dev/message/20374
Mute This Topic: https://lists.fd.io/mt/86565585/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

I think I was going to try and write something more acceptable to be merged, 
but if you end up getting it working, then you may want to offer that as a 
solution to the community.

 

From: vpp-dev@lists.fd.io  On Behalf Of Ramkumar
Sent: Thursday, December 17, 2020 3:56 PM
To: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 

Thanks bjeremy32 for sharing the patch! 
As mentioned by Dave, I have implemented load calculation in a plugin. I can 
try to reevaluate my implementation with yours as reference. Will let you know 
if I get it working. 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18397): https://lists.fd.io/g/vpp-dev/message/18397
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

If you want to refactor it in a way to fit the code architecture better, feel 
free.

 

From: vpp-dev@lists.fd.io  On Behalf Of Dave Barach
Sent: Thursday, December 17, 2020 1:43 PM
To: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 

As the sole remaining original vpp author and the area maintainer, I have 
abundant experience dealing with src/vlib/{main.c, threads.c}. 

 

In every case to date, we’ve been able to leverage existing callback hooks, or 
to add callback hooks so that folks can do what they need to do (e.g. from 
plugins) without modifying the core engine.

 

Might we manage to take that approach here? It’s in everyone’s interest to do 
so...

 

Thanks... Dave

 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18394): https://lists.fd.io/g/vpp-dev/message/18394
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

Ramkumar,

 

Attached are the patches to 20.05 to implement the cpu usage… I think this was 
all that was changed. You may want to hand merge them instead of applying the 
patches though, It may be ok… but im not 100% other lines were not touched.

 

You will be able to view cpu usage with:  ‘watch vppctl show threads’. 

 

As I said, it seems to work rather well for us. We run in various 
configurations. But your particular case may be different and require some 
other tweaking, but this just illustrates the idea… 

 

1.  Grouped no handoff - all rcv queues handled by a single core, no handoff
2.  Grouped with handoff
3.  Separated cores with handoff – each separate core handles a rcv queue, 
handoffs work to a separate thread
4.  Separated cores no handoff

 

In all cases, it seems the cpu usage metric is fairly accurate. We notice as 
pps increases for us packet start dropping at about 90%. And for worker threads 
(not in poll mode) we could compare the cpu usage using htop with our patch… it 
was within 1-2% which was good enough…

 

 

From: vpp-dev@lists.fd.io  On Behalf Of bjeremy32 via 
lists.fd.io
Sent: Wednesday, December 16, 2020 11:14 AM
To: 'Ramkumar B' 
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 

If it will help, I will share my patch as soon as I get authorization from my 
corporate overlords.

 

From: Ramkumar B mailto:ramukmar1...@gmail.com> > 
Sent: Wednesday, December 16, 2020 11:08 AM
To: bjerem...@gmail.com <mailto:bjerem...@gmail.com> 
Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> 
Subject: Re: [vpp-dev] Core Load Calculation

 




I got your idea. Thanks for taking time to clarify. I kind of tried the same 
but in vain. I think I will look into this solution again.

 

On Wed, Dec 16, 2020 at 10:21 PM mailto:bjerem...@gmail.com> > wrote:

In dispatch_node, before we call node->function any accumulated clock cycles 
are counted toward idle, after the node->function returns if ( 
(VLIB_NODE_TYPE_PRE_INPUT || VLIB_NODE_TYPE_INPUT) && number of packets 
processed == 0) then we count it as idle, else we count it as busy.

 

 

From: Ramkumar B mailto:ramukmar1...@gmail.com> > 
Sent: Wednesday, December 16, 2020 10:27 AM
To: bjerem...@gmail.com <mailto:bjerem...@gmail.com> 
Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> 
Subject: Re: [vpp-dev] Core Load Calculation

 

The processing of the rx queue is counted toward IDLE

Please explain what kind of processing. We are not doing any such rx processing 
separately. For us, the ring_dequeue_burst's cycles is negligible compared to 
processing cycles.

 

On Wed, Dec 16, 2020 at 8:54 PM mailto:bjerem...@gmail.com> > wrote:

Nope… the math works out… nothing goes to 0… we are incrementing both IDLE and 
BUSY. The processing of the rx queue is counted toward IDLE… In a completely 
idle system, the cpu usage falls to 0… as traffic ramps up it will go further 
positive based on pps, and as it goes to idle again it will go down to 0…

 

 

From: Damjan Marion mailto:dmar...@me.com> > 
Sent: Wednesday, December 16, 2020 3:00 AM
To: bjerem...@gmail.com <mailto:bjerem...@gmail.com> 
Cc: Nick Zavaritsky mailto:nick.zavarit...@emnify.com> >; ramukmar1...@gmail.com 
<mailto:ramukmar1...@gmail.com> ; vpp-dev@lists.fd.io 
<mailto:vpp-dev@lists.fd.io> 
Subject: Re: [vpp-dev] Core Load Calculation

 

 

 

On 16.12.2020., at 05:45, bjerem...@gmail.com <mailto:bjerem...@gmail.com>  
wrote:

 

What we did was to count clock cycles (timestamps) spent processing the nodes, 
call this busy, and count the clock cycles spent elsewhere, call this idle. And 
then simply output (busy/(busy + idle). We did this per thread and stored the 
calculations globally. We then outputted the calculations on the ‘vppctl show 
thread’ cli.

 

This doesnt make lot of sense. Each input node is spending cycles to process rx 
queue, and out of that processing there may be zero or non-zero packets. So in 
completelly idle system according to your math result will be far away from 0.

 

 

 

This created a nice cpu usage metric that seems pretty accurate for io and 
worker threads…

 

What do you mean by io threads?

 

We compared this to something like htop for the worker threads and its within 
~1%, the io cores seem the same as we can see the usage go to about 80-90% when 
we start dropping packets. We are able to calculate the approx. PPS for a given 
system configuration.

 

It actually wasn’t a lot of effort. I think only three files were touched. 1. A 
new header file o create the global data structure to hold the clock cycle 
counts and associated macros to init and update the structures. 2. Vlib/main.c 
to instrument the main loop and collect the busy/idle clock cycles 3. Vlib 
threads_cli.c as show threads fn was used to output the cpu usage per thread.

 

From: vpp-dev@lists.fd.io <mailto:vpp-de

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

If it will help, I will share my patch as soon as I get authorization from my 
corporate overlords.

 

From: Ramkumar B  
Sent: Wednesday, December 16, 2020 11:08 AM
To: bjerem...@gmail.com
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 




I got your idea. Thanks for taking time to clarify. I kind of tried the same 
but in vain. I think I will look into this solution again.

 

On Wed, Dec 16, 2020 at 10:21 PM mailto:bjerem...@gmail.com> > wrote:

In dispatch_node, before we call node->function any accumulated clock cycles 
are counted toward idle, after the node->function returns if ( 
(VLIB_NODE_TYPE_PRE_INPUT || VLIB_NODE_TYPE_INPUT) && number of packets 
processed == 0) then we count it as idle, else we count it as busy.

 

 

From: Ramkumar B mailto:ramukmar1...@gmail.com> > 
Sent: Wednesday, December 16, 2020 10:27 AM
To: bjerem...@gmail.com  
Cc: vpp-dev@lists.fd.io  
Subject: Re: [vpp-dev] Core Load Calculation

 

The processing of the rx queue is counted toward IDLE

Please explain what kind of processing. We are not doing any such rx processing 
separately. For us, the ring_dequeue_burst's cycles is negligible compared to 
processing cycles.

 

On Wed, Dec 16, 2020 at 8:54 PM mailto:bjerem...@gmail.com> > wrote:

Nope… the math works out… nothing goes to 0… we are incrementing both IDLE and 
BUSY. The processing of the rx queue is counted toward IDLE… In a completely 
idle system, the cpu usage falls to 0… as traffic ramps up it will go further 
positive based on pps, and as it goes to idle again it will go down to 0…

 

 

From: Damjan Marion mailto:dmar...@me.com> > 
Sent: Wednesday, December 16, 2020 3:00 AM
To: bjerem...@gmail.com  
Cc: Nick Zavaritsky mailto:nick.zavarit...@emnify.com> >; ramukmar1...@gmail.com 
 ; vpp-dev@lists.fd.io 
 
Subject: Re: [vpp-dev] Core Load Calculation

 

 

 

On 16.12.2020., at 05:45, bjerem...@gmail.com   
wrote:

 

What we did was to count clock cycles (timestamps) spent processing the nodes, 
call this busy, and count the clock cycles spent elsewhere, call this idle. And 
then simply output (busy/(busy + idle). We did this per thread and stored the 
calculations globally. We then outputted the calculations on the ‘vppctl show 
thread’ cli.

 

This doesnt make lot of sense. Each input node is spending cycles to process rx 
queue, and out of that processing there may be zero or non-zero packets. So in 
completelly idle system according to your math result will be far away from 0.

 

 

 

This created a nice cpu usage metric that seems pretty accurate for io and 
worker threads…

 

What do you mean by io threads?

 

We compared this to something like htop for the worker threads and its within 
~1%, the io cores seem the same as we can see the usage go to about 80-90% when 
we start dropping packets. We are able to calculate the approx. PPS for a given 
system configuration.

 

It actually wasn’t a lot of effort. I think only three files were touched. 1. A 
new header file o create the global data structure to hold the clock cycle 
counts and associated macros to init and update the structures. 2. Vlib/main.c 
to instrument the main loop and collect the busy/idle clock cycles 3. Vlib 
threads_cli.c as show threads fn was used to output the cpu usage per thread.

 

From: vpp-dev@lists.fd.io   mailto:vpp-dev@lists.fd.io> > On Behalf Of Nick Zavaritsky
Sent: Tuesday, December 15, 2020 1:38 PM
To: ramukmar1...@gmail.com  
Cc: vpp-dev@lists.fd.io  
Subject: Re: [vpp-dev] Core Load Calculation

 

We’ve been using flamegraphs quite successfully to see how much leeway is there.

 

Ex: 
https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg

 

On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io   
mailto:ramukmar1998=gmail@lists.fd.io> 
> wrote:
 

Hello All,

 

I'm trying to calculate VPP core's load. I completely understand about the 
polling cores' 100% CPU usage. My requirement is different where I need to 
calculate the core's load based on how much more PPS it can handle before a 
packet drop occurs in the queue.

 

The vec/call is a good indicator of load, but it does not increase linearly 
with PPS. This is because of the VPP's self balancing behaviour where the cost 
per packet reduces as load increases. It would be a great help if anyone can 
point out factors to calculate load.

 

Thanks and regards, 

Ramkumar Balu

 




 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18377): https://lists.fd.io/g/vpp-dev/message/18377
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

In dispatch_node, before we call node->function any accumulated clock cycles 
are counted toward idle, after the node->function returns if ( 
(VLIB_NODE_TYPE_PRE_INPUT || VLIB_NODE_TYPE_INPUT) && number of packets 
processed == 0) then we count it as idle, else we count it as busy.

 

 

From: Ramkumar B  
Sent: Wednesday, December 16, 2020 10:27 AM
To: bjerem...@gmail.com
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 

The processing of the rx queue is counted toward IDLE

Please explain what kind of processing. We are not doing any such rx processing 
separately. For us, the ring_dequeue_burst's cycles is negligible compared to 
processing cycles.

 

On Wed, Dec 16, 2020 at 8:54 PM mailto:bjerem...@gmail.com> > wrote:

Nope… the math works out… nothing goes to 0… we are incrementing both IDLE and 
BUSY. The processing of the rx queue is counted toward IDLE… In a completely 
idle system, the cpu usage falls to 0… as traffic ramps up it will go further 
positive based on pps, and as it goes to idle again it will go down to 0…

 

 

From: Damjan Marion mailto:dmar...@me.com> > 
Sent: Wednesday, December 16, 2020 3:00 AM
To: bjerem...@gmail.com  
Cc: Nick Zavaritsky mailto:nick.zavarit...@emnify.com> >; ramukmar1...@gmail.com 
 ; vpp-dev@lists.fd.io 
 
Subject: Re: [vpp-dev] Core Load Calculation

 

 

 

On 16.12.2020., at 05:45, bjerem...@gmail.com   
wrote:

 

What we did was to count clock cycles (timestamps) spent processing the nodes, 
call this busy, and count the clock cycles spent elsewhere, call this idle. And 
then simply output (busy/(busy + idle). We did this per thread and stored the 
calculations globally. We then outputted the calculations on the ‘vppctl show 
thread’ cli.

 

This doesnt make lot of sense. Each input node is spending cycles to process rx 
queue, and out of that processing there may be zero or non-zero packets. So in 
completelly idle system according to your math result will be far away from 0.

 

 

 

This created a nice cpu usage metric that seems pretty accurate for io and 
worker threads…

 

What do you mean by io threads?

 

We compared this to something like htop for the worker threads and its within 
~1%, the io cores seem the same as we can see the usage go to about 80-90% when 
we start dropping packets. We are able to calculate the approx. PPS for a given 
system configuration.

 

It actually wasn’t a lot of effort. I think only three files were touched. 1. A 
new header file o create the global data structure to hold the clock cycle 
counts and associated macros to init and update the structures. 2. Vlib/main.c 
to instrument the main loop and collect the busy/idle clock cycles 3. Vlib 
threads_cli.c as show threads fn was used to output the cpu usage per thread.

 

From: vpp-dev@lists.fd.io   mailto:vpp-dev@lists.fd.io> > On Behalf Of Nick Zavaritsky
Sent: Tuesday, December 15, 2020 1:38 PM
To: ramukmar1...@gmail.com  
Cc: vpp-dev@lists.fd.io  
Subject: Re: [vpp-dev] Core Load Calculation

 

We’ve been using flamegraphs quite successfully to see how much leeway is there.

 

Ex: 
https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg





On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io   
mailto:ramukmar1998=gmail@lists.fd.io> 
> wrote:
 

Hello All,

 

I'm trying to calculate VPP core's load. I completely understand about the 
polling cores' 100% CPU usage. My requirement is different where I need to 
calculate the core's load based on how much more PPS it can handle before a 
packet drop occurs in the queue.

 

The vec/call is a good indicator of load, but it does not increase linearly 
with PPS. This is because of the VPP's self balancing behaviour where the cost 
per packet reduces as load increases. It would be a great help if anyone can 
point out factors to calculate load.

 

Thanks and regards, 

Ramkumar Balu








 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18372): https://lists.fd.io/g/vpp-dev/message/18372
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

Also.. forgot to mention… if a node doesn’t process any packets… it is counted 
as idle, else busy

 

From: bjerem...@gmail.com  
Sent: Wednesday, December 16, 2020 9:25 AM
To: 'Damjan Marion' 
Cc: 'Nick Zavaritsky' ; ramukmar1...@gmail.com; 
vpp-dev@lists.fd.io
Subject: RE: [vpp-dev] Core Load Calculation

 

Nope… the math works out… nothing goes to 0… we are incrementing both IDLE and 
BUSY. The processing of the rx queue is counted toward IDLE… In a completely 
idle system, the cpu usage falls to 0… as traffic ramps up it will go further 
positive based on pps, and as it goes to idle again it will go down to 0…

 

 

From: Damjan Marion mailto:dmar...@me.com> > 
Sent: Wednesday, December 16, 2020 3:00 AM
To: bjerem...@gmail.com  
Cc: Nick Zavaritsky mailto:nick.zavarit...@emnify.com> >; ramukmar1...@gmail.com 
 ; vpp-dev@lists.fd.io 
 
Subject: Re: [vpp-dev] Core Load Calculation

 

 

 

On 16.12.2020., at 05:45, bjerem...@gmail.com   
wrote:

 

What we did was to count clock cycles (timestamps) spent processing the nodes, 
call this busy, and count the clock cycles spent elsewhere, call this idle. And 
then simply output (busy/(busy + idle). We did this per thread and stored the 
calculations globally. We then outputted the calculations on the ‘vppctl show 
thread’ cli.

 

This doesnt make lot of sense. Each input node is spending cycles to process rx 
queue, and out of that processing there may be zero or non-zero packets. So in 
completelly idle system according to your math result will be far away from 0.

 

 

 

This created a nice cpu usage metric that seems pretty accurate for io and 
worker threads…

 

What do you mean by io threads?

 

We compared this to something like htop for the worker threads and its within 
~1%, the io cores seem the same as we can see the usage go to about 80-90% when 
we start dropping packets. We are able to calculate the approx. PPS for a given 
system configuration.

 

It actually wasn’t a lot of effort. I think only three files were touched. 1. A 
new header file o create the global data structure to hold the clock cycle 
counts and associated macros to init and update the structures. 2. Vlib/main.c 
to instrument the main loop and collect the busy/idle clock cycles 3. Vlib 
threads_cli.c as show threads fn was used to output the cpu usage per thread.

 

From: vpp-dev@lists.fd.io   mailto:vpp-dev@lists.fd.io> > On Behalf Of Nick Zavaritsky
Sent: Tuesday, December 15, 2020 1:38 PM
To: ramukmar1...@gmail.com  
Cc: vpp-dev@lists.fd.io  
Subject: Re: [vpp-dev] Core Load Calculation

 

We’ve been using flamegraphs quite successfully to see how much leeway is there.

 

Ex: 
https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg





On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io   
mailto:ramukmar1998=gmail@lists.fd.io> 
> wrote:
 

Hello All,

 

I'm trying to calculate VPP core's load. I completely understand about the 
polling cores' 100% CPU usage. My requirement is different where I need to 
calculate the core's load based on how much more PPS it can handle before a 
packet drop occurs in the queue.

 

The vec/call is a good indicator of load, but it does not increase linearly 
with PPS. This is because of the VPP's self balancing behaviour where the cost 
per packet reduces as load increases. It would be a great help if anyone can 
point out factors to calculate load.

 

Thanks and regards, 

Ramkumar Balu








 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18369): https://lists.fd.io/g/vpp-dev/message/18369
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

Nope… the math works out… nothing goes to 0… we are incrementing both IDLE and 
BUSY. The processing of the rx queue is counted toward IDLE… In a completely 
idle system, the cpu usage falls to 0… as traffic ramps up it will go further 
positive based on pps, and as it goes to idle again it will go down to 0…

 

 

From: Damjan Marion  
Sent: Wednesday, December 16, 2020 3:00 AM
To: bjerem...@gmail.com
Cc: Nick Zavaritsky ; ramukmar1...@gmail.com; 
vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 

 





On 16.12.2020., at 05:45, bjerem...@gmail.com   
wrote:

 

What we did was to count clock cycles (timestamps) spent processing the nodes, 
call this busy, and count the clock cycles spent elsewhere, call this idle. And 
then simply output (busy/(busy + idle). We did this per thread and stored the 
calculations globally. We then outputted the calculations on the ‘vppctl show 
thread’ cli.

 

This doesnt make lot of sense. Each input node is spending cycles to process rx 
queue, and out of that processing there may be zero or non-zero packets. So in 
completelly idle system according to your math result will be far away from 0.

 





 

This created a nice cpu usage metric that seems pretty accurate for io and 
worker threads…

 

What do you mean by io threads?





We compared this to something like htop for the worker threads and its within 
~1%, the io cores seem the same as we can see the usage go to about 80-90% when 
we start dropping packets. We are able to calculate the approx. PPS for a given 
system configuration.

 

It actually wasn’t a lot of effort. I think only three files were touched. 1. A 
new header file o create the global data structure to hold the clock cycle 
counts and associated macros to init and update the structures. 2. Vlib/main.c 
to instrument the main loop and collect the busy/idle clock cycles 3. Vlib 
threads_cli.c as show threads fn was used to output the cpu usage per thread.

 

From: vpp-dev@lists.fd.io   mailto:vpp-dev@lists.fd.io> > On Behalf Of Nick Zavaritsky
Sent: Tuesday, December 15, 2020 1:38 PM
To: ramukmar1...@gmail.com  
Cc: vpp-dev@lists.fd.io  
Subject: Re: [vpp-dev] Core Load Calculation

 

We’ve been using flamegraphs quite successfully to see how much leeway is there.

 

Ex: 
https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg






On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io   
mailto:ramukmar1998=gmail@lists.fd.io> 
> wrote:
 

Hello All,

 

I'm trying to calculate VPP core's load. I completely understand about the 
polling cores' 100% CPU usage. My requirement is different where I need to 
calculate the core's load based on how much more PPS it can handle before a 
packet drop occurs in the queue.

 

The vec/call is a good indicator of load, but it does not increase linearly 
with PPS. This is because of the VPP's self balancing behaviour where the cost 
per packet reduces as load increases. It would be a great help if anyone can 
point out factors to calculate load.

 

Thanks and regards, 

Ramkumar Balu









 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18366): https://lists.fd.io/g/vpp-dev/message/18366
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Core Load Calculation

[bjeremy32]

What we did was to count clock cycles (timestamps) spent processing the nodes, 
call this busy, and count the clock cycles spent elsewhere, call this idle. And 
then simply output (busy/(busy + idle). We did this per thread and stored the 
calculations globally. We then outputted the calculations on the ‘vppctl show 
thread’ cli.

 

This created a nice cpu usage metric that seems pretty accurate for io and 
worker threads… We compared this to something like htop for the worker threads 
and its within ~1%, the io cores seem the same as we can see the usage go to 
about 80-90% when we start dropping packets. We are able to calculate the 
approx. PPS for a given system configuration.

 

It actually wasn’t a lot of effort. I think only three files were touched. 1. A 
new header file o create the global data structure to hold the clock cycle 
counts and associated macros to init and update the structures. 2. Vlib/main.c 
to instrument the main loop and collect the busy/idle clock cycles 3. Vlib 
threads_cli.c as show threads fn was used to output the cpu usage per thread.

 

From: vpp-dev@lists.fd.io  On Behalf Of Nick Zavaritsky
Sent: Tuesday, December 15, 2020 1:38 PM
To: ramukmar1...@gmail.com
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Core Load Calculation

 

We’ve been using flamegraphs quite successfully to see how much leeway is there.

 

Ex: 
https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg





On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io 
mailto:ramukmar1998=gmail@lists.fd.io> 
> wrote:
 

Hello All,

 

I'm trying to calculate VPP core's load. I completely understand about the 
polling cores' 100% CPU usage. My requirement is different where I need to 
calculate the core's load based on how much more PPS it can handle before a 
packet drop occurs in the queue.

 

The vec/call is a good indicator of load, but it does not increase linearly 
with PPS. This is because of the VPP's self balancing behaviour where the cost 
per packet reduces as load increases. It would be a great help if anyone can 
point out factors to calculate load.

 

Thanks and regards, 

Ramkumar Balu






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18348): https://lists.fd.io/g/vpp-dev/message/18348
Mute This Topic: https://lists.fd.io/mt/78980301/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-