Re: [vpp-dev] Core Load Calculation
Thanks bjeremy32 for sharing the patch! As mentioned by Dave, I have implemented load calculation in a plugin. I can try to reevaluate my implementation with yours as reference. Will let you know if I get it working. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18395): https://lists.fd.io/g/vpp-dev/message/18395 Mute This Topic: https://lists.fd.io/mt/78980301/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Core Load Calculation
I got your idea. Thanks for taking time to clarify. I kind of tried the same but in vain. I think I will look into this solution again. On Wed, Dec 16, 2020 at 10:21 PM wrote: > In dispatch_node, before we call node->function any accumulated clock > cycles are counted toward idle, after the node->function returns if ( > (VLIB_NODE_TYPE_PRE_INPUT || VLIB_NODE_TYPE_INPUT) && number of packets > processed == 0) then we count it as idle, else we count it as busy. > > > > > > *From:* Ramkumar B > *Sent:* Wednesday, December 16, 2020 10:27 AM > *To:* bjerem...@gmail.com > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > The processing of the rx queue is counted toward IDLE > > Please explain what kind of processing. We are not doing any such rx > processing separately. For us, the ring_dequeue_burst's cycles > is negligible compared to processing cycles. > > > > On Wed, Dec 16, 2020 at 8:54 PM wrote: > > Nope… the math works out… nothing goes to 0… we are incrementing both IDLE > and BUSY. The processing of the rx queue is counted toward IDLE… In a > completely idle system, the cpu usage falls to 0… as traffic ramps up it > will go further positive based on pps, and as it goes to idle again it will > go down to 0… > > > > > > *From:* Damjan Marion > *Sent:* Wednesday, December 16, 2020 3:00 AM > *To:* bjerem...@gmail.com > *Cc:* Nick Zavaritsky ; ramukmar1...@gmail.com; > vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > > > > > On 16.12.2020., at 05:45, bjerem...@gmail.com wrote: > > > > What we did was to count clock cycles (timestamps) spent processing the > nodes, call this busy, and count the clock cycles spent elsewhere, call > this idle. And then simply output (busy/(busy + idle). We did this per > thread and stored the calculations globally. We then outputted the > calculations on the ‘vppctl show thread’ cli. > > > > This doesnt make lot of sense. Each input node is spending cycles to > process rx queue, and out of that processing there may be zero or non-zero > packets. So in completelly idle system according to your math result will > be far away from 0. > > > > > > > > This created a nice cpu usage metric that seems pretty accurate for io and > worker threads… > > > > What do you mean by io threads? > > > > We compared this to something like htop for the worker threads and its > within ~1%, the io cores seem the same as we can see the usage go to about > 80-90% when we start dropping packets. We are able to calculate the approx. > PPS for a given system configuration. > > > > It actually wasn’t a lot of effort. I think only three files were touched. > 1. A new header file o create the global data structure to hold the clock > cycle counts and associated macros to init and update the structures. 2. > Vlib/main.c to instrument the main loop and collect the busy/idle clock > cycles 3. Vlib threads_cli.c as show threads fn was used to output the cpu > usage per thread. > > > > *From:* vpp-dev@lists.fd.io *On Behalf Of *Nick > Zavaritsky > *Sent:* Tuesday, December 15, 2020 1:38 PM > *To:* ramukmar1...@gmail.com > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > We’ve been using flamegraphs quite successfully to see how much leeway is > there. > > > > Ex: > https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg > > > > On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io < > ramukmar1998=gmail@lists.fd.io> wrote: > > > Hello All, > > > > I'm trying to calculate VPP core's load. I completely understand about the > polling cores' 100% CPU usage. My requirement is different where I need to > calculate the core's load based on how much more PPS it can handle before a > packet drop occurs in the queue. > > > > The vec/call is a good indicator of load, but it does not increase > linearly with PPS. This is because of the VPP's self balancing behaviour > where the cost per packet reduces as load increases. It would be a great > help if anyone can point out factors to calculate load. > > > > Thanks and regards, > > Ramkumar Balu > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18375): https://lists.fd.io/g/vpp-dev/message/18375 Mute This Topic: https://lists.fd.io/mt/78980301/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Core Load Calculation
Yeah idle is when the input node doesn't get packets while polling. Other nodes will not be scheduled without packets. And as I said, the rx cycles is negligible.. On Wed, Dec 16, 2020 at 10:00 PM wrote: > Also.. forgot to mention… if a node doesn’t process any packets… it is > counted as idle, else busy > > > > *From:* bjerem...@gmail.com > *Sent:* Wednesday, December 16, 2020 9:25 AM > *To:* 'Damjan Marion' > *Cc:* 'Nick Zavaritsky' ; > ramukmar1...@gmail.com; vpp-dev@lists.fd.io > *Subject:* RE: [vpp-dev] Core Load Calculation > > > > Nope… the math works out… nothing goes to 0… we are incrementing both IDLE > and BUSY. The processing of the rx queue is counted toward IDLE… In a > completely idle system, the cpu usage falls to 0… as traffic ramps up it > will go further positive based on pps, and as it goes to idle again it will > go down to 0… > > > > > > *From:* Damjan Marion > *Sent:* Wednesday, December 16, 2020 3:00 AM > *To:* bjerem...@gmail.com > *Cc:* Nick Zavaritsky ; ramukmar1...@gmail.com; > vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > > > > > On 16.12.2020., at 05:45, bjerem...@gmail.com wrote: > > > > What we did was to count clock cycles (timestamps) spent processing the > nodes, call this busy, and count the clock cycles spent elsewhere, call > this idle. And then simply output (busy/(busy + idle). We did this per > thread and stored the calculations globally. We then outputted the > calculations on the ‘vppctl show thread’ cli. > > > > This doesnt make lot of sense. Each input node is spending cycles to > process rx queue, and out of that processing there may be zero or non-zero > packets. So in completelly idle system according to your math result will > be far away from 0. > > > > > > > > This created a nice cpu usage metric that seems pretty accurate for io and > worker threads… > > > > What do you mean by io threads? > > > > We compared this to something like htop for the worker threads and its > within ~1%, the io cores seem the same as we can see the usage go to about > 80-90% when we start dropping packets. We are able to calculate the approx. > PPS for a given system configuration. > > > > It actually wasn’t a lot of effort. I think only three files were touched. > 1. A new header file o create the global data structure to hold the clock > cycle counts and associated macros to init and update the structures. 2. > Vlib/main.c to instrument the main loop and collect the busy/idle clock > cycles 3. Vlib threads_cli.c as show threads fn was used to output the cpu > usage per thread. > > > > *From:* vpp-dev@lists.fd.io *On Behalf Of *Nick > Zavaritsky > *Sent:* Tuesday, December 15, 2020 1:38 PM > *To:* ramukmar1...@gmail.com > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > We’ve been using flamegraphs quite successfully to see how much leeway is > there. > > > > Ex: > https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg > > > > On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io < > ramukmar1998=gmail@lists.fd.io> wrote: > > > Hello All, > > > > I'm trying to calculate VPP core's load. I completely understand about the > polling cores' 100% CPU usage. My requirement is different where I need to > calculate the core's load based on how much more PPS it can handle before a > packet drop occurs in the queue. > > > > The vec/call is a good indicator of load, but it does not increase > linearly with PPS. This is because of the VPP's self balancing behaviour > where the cost per packet reduces as load increases. It would be a great > help if anyone can point out factors to calculate load. > > > > Thanks and regards, > > Ramkumar Balu > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18371): https://lists.fd.io/g/vpp-dev/message/18371 Mute This Topic: https://lists.fd.io/mt/78980301/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Core Load Calculation
The processing of the rx queue is counted toward IDLE Please explain what kind of processing. We are not doing any such rx processing separately. For us, the ring_dequeue_burst's cycles is negligible compared to processing cycles. On Wed, Dec 16, 2020 at 8:54 PM wrote: > Nope… the math works out… nothing goes to 0… we are incrementing both IDLE > and BUSY. The processing of the rx queue is counted toward IDLE… In a > completely idle system, the cpu usage falls to 0… as traffic ramps up it > will go further positive based on pps, and as it goes to idle again it will > go down to 0… > > > > > > *From:* Damjan Marion > *Sent:* Wednesday, December 16, 2020 3:00 AM > *To:* bjerem...@gmail.com > *Cc:* Nick Zavaritsky ; ramukmar1...@gmail.com; > vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > > > > > On 16.12.2020., at 05:45, bjerem...@gmail.com wrote: > > > > What we did was to count clock cycles (timestamps) spent processing the > nodes, call this busy, and count the clock cycles spent elsewhere, call > this idle. And then simply output (busy/(busy + idle). We did this per > thread and stored the calculations globally. We then outputted the > calculations on the ‘vppctl show thread’ cli. > > > > This doesnt make lot of sense. Each input node is spending cycles to > process rx queue, and out of that processing there may be zero or non-zero > packets. So in completelly idle system according to your math result will > be far away from 0. > > > > > > > > This created a nice cpu usage metric that seems pretty accurate for io and > worker threads… > > > > What do you mean by io threads? > > > > We compared this to something like htop for the worker threads and its > within ~1%, the io cores seem the same as we can see the usage go to about > 80-90% when we start dropping packets. We are able to calculate the approx. > PPS for a given system configuration. > > > > It actually wasn’t a lot of effort. I think only three files were touched. > 1. A new header file o create the global data structure to hold the clock > cycle counts and associated macros to init and update the structures. 2. > Vlib/main.c to instrument the main loop and collect the busy/idle clock > cycles 3. Vlib threads_cli.c as show threads fn was used to output the cpu > usage per thread. > > > > *From:* vpp-dev@lists.fd.io *On Behalf Of *Nick > Zavaritsky > *Sent:* Tuesday, December 15, 2020 1:38 PM > *To:* ramukmar1...@gmail.com > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Core Load Calculation > > > > We’ve been using flamegraphs quite successfully to see how much leeway is > there. > > > > Ex: > https://gist.githack.com/mejedi/d5d094df63faba66d413a677fcef26e3/raw/95294d36c4b180ba6741d793bf345041b00af48e/g.svg > > > > > On 15 Dec 2020, at 19:53, Ramkumar B via lists.fd.io < > ramukmar1998=gmail@lists.fd.io> wrote: > > > Hello All, > > > > I'm trying to calculate VPP core's load. I completely understand about the > polling cores' 100% CPU usage. My requirement is different where I need to > calculate the core's load based on how much more PPS it can handle before a > packet drop occurs in the queue. > > > > The vec/call is a good indicator of load, but it does not increase > linearly with PPS. This is because of the VPP's self balancing behaviour > where the cost per packet reduces as load increases. It would be a great > help if anyone can point out factors to calculate load. > > > > Thanks and regards, > > Ramkumar Balu > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18368): https://lists.fd.io/g/vpp-dev/message/18368 Mute This Topic: https://lists.fd.io/mt/78980301/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Core Load Calculation
Hello All, I'm trying to calculate VPP core's load. I completely understand about the polling cores' 100% CPU usage. My requirement is different where I need to calculate the core's load based on how much more PPS it can handle before a packet drop occurs in the queue. The vec/call is a good indicator of load, but it does not increase linearly with PPS. This is because of the VPP's self balancing behaviour where the cost per packet reduces as load increases. It would be a great help if anyone can point out factors to calculate load. Thanks and regards, Ramkumar Balu -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18344): https://lists.fd.io/g/vpp-dev/message/18344 Mute This Topic: https://lists.fd.io/mt/78980301/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
