Re: [vpp-dev] VPP 2005 is crashing on stopping the VCL applications #vpp-hoststack

[Dave Barach via lists.fd.io]

You can press the “cherrypick” button as easily as Florin... Hint...

From: vpp-dev@lists.fd.io  On Behalf Of Raj Kumar
Sent: Monday, August 17, 2020 5:09 PM
To: Ayush Gautam 
Cc: Florin Coras ; vpp-dev 
Subject: Re: [vpp-dev] VPP 2005 is crashing on stopping the VCL applications 
#vpp-hoststack

Hi Florin,
Can we please have the fix[1] on "stable/2005" branch.

[1] https://gerrit.fd.io/r/c/vpp/+/28173

 Thanks,
-Raj

On Wed, Aug 5, 2020 at 7:30 PM Raj Kumar via lists.fd.io 
mailto:gmail@lists.fd.io>> wrote:
Hi Florin,
Yes , this[1] fixed the issue.

thanks,
-Raj

On Wed, Aug 5, 2020 at 1:57 AM Florin Coras 
mailto:fcoras.li...@gmail.com>> wrote:
Hi Raj,

Does this [1] fix the issue?

Regards,
Florin

[1] https://gerrit.fd.io/r/c/vpp/+/28173


On Aug 4, 2020, at 8:24 AM, Raj Kumar 
mailto:raj.gauta...@gmail.com>> wrote:

Hi Florin,
I tried vppcom_epoll_wait() on 2 different  servers by using a simple 
application ( with only 1 worker thread) . But, still vppcom_epoll_wait() is 
not being timed out if I do not use use-mq-eventfd .
Here are the server details -
server 1  -  Red hat 7.5  , vpp release - 20.01
server 2 - Centos 8.1  , vpp release - 20.05

thanks,
-Raj


On Tue, Aug 4, 2020 at 10:24 AM Florin Coras 
mailto:fcoras.li...@gmail.com>> wrote:
Hi Raj,

Interesting. For some reason, the message queue’s underlying 
pthread_cond_timedwait does not work in your setup. Not sure what to make of 
that, unless maybe you’re trying to epoll wait from multiple threads that share 
the same message queue. That is not supported since each thread must have its 
own message queue, i.e., all threads that call epoll should be registered as 
workers. Alternatively, some form of locking or vls, instead of vcl, should be 
used.

The downside to switching the message queue to eventfd notifications, instead 
of mutext/condvar, is that waits are inefficient, i.e., they act pretty much 
like spinlocks. Do keep that in mind.

Regards,
Florin


On Aug 4, 2020, at 6:37 AM, Raj Kumar 
mailto:raj.gauta...@gmail.com>> wrote:

Hi Florin,
After adding use-mq-eventfd in VCL configuration, it is working as expected.
Thanks! for your help.

vcl {
  rx-fifo-size 400
  tx-fifo-size 400
  app-scope-local
  app-scope-global
  use-mq-eventfd
  api-socket-name /tmp/vpp-api.sock
}

thanks,
-Raj

On Tue, Aug 4, 2020 at 12:08 AM Florin Coras 
mailto:fcoras.li...@gmail.com>> wrote:
Hi Raj,

Glad to hear that issue is solved. What vcl config are you running? Did you 
configure use-mq-eventd?

Regards,
Florin


On Aug 3, 2020, at 8:33 PM, Raj Kumar 
mailto:raj.gauta...@gmail.com>> wrote:

Hi Florin,
This issue is resolved now.  In my application, on receiving the kill signal 
main thread was sending phread_cancel() to the child thread because of that 
child thread was not exiting gracefully.
I have one question; it seems that vppcom_epoll_wait(epfd, rcvEvents, 
MAX_RETURN_EVENTS, 6.0) is not returning after timed out if the timeout 
value is a non zero value. It timed out only if the timeout value is 0.
The issue that I am facing is that if there is no traffic at all ( receiver is 
just listening on the connections ) then the worker thread is not exiting as it 
is blocked by vppcom_epoll_wait().

Thanks,
-Raj



On Wed, Jul 29, 2020 at 11:23 PM Florin Coras 
mailto:fcoras.li...@gmail.com>> wrote:
Hi Raj,

In that case it should work. Just from the trace lower it’s hard to figure out 
what exactly happened. Also, keep in mind that vcl is not thread safe, so make 
sure you’re not trying to share sessions or allow two workers to  interact with 
the message queue(s) at the same time.

Regards,
Florin


On Jul 29, 2020, at 8:17 PM, Raj Kumar 
mailto:raj.gauta...@gmail.com>> wrote:

Hi Florin,
I am using kill  to stop the application. But , the application has a kill 
signal handler and after receiving the signal it is exiting gracefully.
About vppcom_app_exit, I think this function is registered with atexit() inside 
vppcom_app_create() so it should call when the application exits.
Even, I also tried this vppcom_app_exit() explicitly before exiting the 
application but still I am seeing the same issue.

My application is a multithreaded application. Can you please suggest some 
cleanup functions ( vppcom functions) that  I should call before exiting a 
thread and the main application for a proper cleanup.
I also tried vppcom_app_destroy() before exiting the main application but still 
I am seeing the same issue.

thanks,
-Raj

On Wed, Jul 29, 2020 at 5:34 PM Florin Coras 
mailto:fcoras.li...@gmail.com>> wrote:
Hi Raj,

Does stopping include a call to vppcom_app_exit or killing the applications? If 
the latter, the apps might be killed with some mutexes/spinlocks held. For now, 
we only support the former.

Regards,
Florin

> On Jul 29, 2020, at 1:49 PM, Raj Kumar 
> mailto:raj.gauta...@gmail.com>> wrote:
>
> Hi,
> In my UDP application , I am using VPP host stack to receive packets and 
> memIf to transmit packets. 

Re: [vpp-dev] VPP 2005 is crashing on stopping the VCL applications #vpp-hoststack

[Raj Kumar]

Hi Florin,
Can we please have the fix[1] on "stable/2005" branch.

[1] https://gerrit.fd.io/r/c/vpp/+/28173

 Thanks,
-Raj

On Wed, Aug 5, 2020 at 7:30 PM Raj Kumar via lists.fd.io  wrote:

> Hi Florin,
> Yes , this[1] fixed the issue.
>
> thanks,
> -Raj
>
> On Wed, Aug 5, 2020 at 1:57 AM Florin Coras 
> wrote:
>
>> Hi Raj,
>>
>> Does this [1] fix the issue?
>>
>> Regards,
>> Florin
>>
>> [1] https://gerrit.fd.io/r/c/vpp/+/28173
>>
>> On Aug 4, 2020, at 8:24 AM, Raj Kumar  wrote:
>>
>> Hi Florin,
>> I tried vppcom_epoll_wait() on 2 different  servers by using a simple
>> application ( with only 1 worker thread) . But, still vppcom_epoll_wait()
>> is not being timed out if I do not use use-mq-eventfd .
>> Here are the server details -
>> server 1  -  Red hat 7.5  , vpp release - 20.01
>> server 2 - Centos 8.1  , vpp release - 20.05
>>
>> thanks,
>> -Raj
>>
>>
>> On Tue, Aug 4, 2020 at 10:24 AM Florin Coras 
>> wrote:
>>
>>> Hi Raj,
>>>
>>> Interesting. For some reason, the message queue’s underlying
>>> pthread_cond_timedwait does not work in your setup. Not sure what to make
>>> of that, unless maybe you’re trying to epoll wait from multiple threads
>>> that share the same message queue. That is not supported since each thread
>>> must have its own message queue, i.e., all threads that call epoll should
>>> be registered as workers. Alternatively, some form of locking or vls,
>>> instead of vcl, should be used.
>>>
>>> The downside to switching the message queue to eventfd notifications,
>>> instead of mutext/condvar, is that waits are inefficient, i.e., they act
>>> pretty much like spinlocks. Do keep that in mind.
>>>
>>> Regards,
>>> Florin
>>>
>>> On Aug 4, 2020, at 6:37 AM, Raj Kumar  wrote:
>>>
>>> Hi Florin,
>>> After adding use-mq-eventfd in VCL configuration, it is working as
>>> expected.
>>> Thanks! for your help.
>>>
>>> vcl {
>>>   rx-fifo-size 400
>>>   tx-fifo-size 400
>>>   app-scope-local
>>>   app-scope-global
>>>   use-mq-eventfd
>>>   api-socket-name /tmp/vpp-api.sock
>>> }
>>>
>>> thanks,
>>> -Raj
>>>
>>> On Tue, Aug 4, 2020 at 12:08 AM Florin Coras 
>>> wrote:
>>>
 Hi Raj,

 Glad to hear that issue is solved. What vcl config are you running? Did
 you configure use-mq-eventd?

 Regards,
 Florin

 On Aug 3, 2020, at 8:33 PM, Raj Kumar  wrote:

 Hi Florin,
 This issue is resolved now.  In my application, on receiving the kill
 signal main thread was sending phread_cancel() to the child thread
 because of that child thread was not exiting gracefully.
 I have one question; it seems that vppcom_epoll_wait(epfd, rcvEvents,
 MAX_RETURN_EVENTS, 6.0) is not returning after timed out if the timeout
 value is a non zero value. It timed out only if the timeout value is 0.
 The issue that I am facing is that if there is no traffic at all (
 receiver is just listening on the connections ) then the worker thread is
 not exiting as it is blocked by vppcom_epoll_wait().

 Thanks,
 -Raj



 On Wed, Jul 29, 2020 at 11:23 PM Florin Coras 
 wrote:

> Hi Raj,
>
> In that case it should work. Just from the trace lower it’s hard to
> figure out what exactly happened. Also, keep in mind that vcl is not 
> thread
> safe, so make sure you’re not trying to share sessions or allow two 
> workers
> to  interact with the message queue(s) at the same time.
>
> Regards,
> Florin
>
> On Jul 29, 2020, at 8:17 PM, Raj Kumar  wrote:
>
> Hi Florin,
> I am using kill  to stop the application. But , the application
> has a kill signal handler and after receiving the signal it is exiting
> gracefully.
> About vppcom_app_exit, I think this function is registered with
> atexit() inside vppcom_app_create() so it should call when the application
> exits.
> Even, I also tried this vppcom_app_exit() explicitly before exiting
> the application but still I am seeing the same issue.
>
> My application is a multithreaded application. Can you please suggest
> some cleanup functions ( vppcom functions) that  I should call before
> exiting a thread and the main application for a proper cleanup.
> I also tried vppcom_app_destroy() before exiting the main application
> but still I am seeing the same issue.
>
> thanks,
> -Raj
>
> On Wed, Jul 29, 2020 at 5:34 PM Florin Coras 
> wrote:
>
>> Hi Raj,
>>
>> Does stopping include a call to vppcom_app_exit or killing the
>> applications? If the latter, the apps might be killed with some
>> mutexes/spinlocks held. For now, we only support the former.
>>
>> Regards,
>> Florin
>>
>> > On Jul 29, 2020, at 1:49 PM, Raj Kumar 
>> wrote:
>> >
>> > Hi,
>> > In my UDP application , I am using VPP host stack to receive
>> packets and memIf to transmit packets. There are a total 6 

Re: [vpp-dev] Question on performance tests..

[Benoit Ganne (bganne) via lists.fd.io]

Hi Christian,

I am CC'ing csit-dev but here are some elements.

AFAIK, CSIT results are not trustworthy for Mellanox NIC yet unfortunately: 
T-Rex suffers from "stretch duration" when using Mellanox NIC to inject traffic.
Here is my understanding of what this means: when asked to send at X packets in 
Y seconds (hence at a rate of X/Y pps), T-Rex effectively takes Y + Z seconds. 
CSIT figures are based on the programmed rate (X/Y) and not the real rate 
(X/(Y+Z)).
In a nutshell: we are not really sure where we're at with Mellanox card in CSIT.

Your figures look similar to what I get on my setup (~20Mpps bi-directional for 
IPv4 forwarding).

Best
ben

> -Original Message-
> From: vpp-dev@lists.fd.io  On Behalf Of Christian
> Hopps
> Sent: lundi 17 août 2020 15:00
> To: vpp-dev 
> Cc: Christian Hopps 
> Subject: [vpp-dev] Question on performance tests..
> 
> [sent to csit-dev but perhaps since I'm not subscribed it didn't post]
> 
> I'm looking at this performance graph:
> 
>  https://docs.fd.io/csit/master/trending/trending/ip4-2n-clx-cx556a.html
> 
> Is this a uni-directional or a bi-directional test?
> 
> There is a huge performance gain between June and July it seems. Do we
> know which change this was? I'm on a current vpp master and I am only
> getting about 9Mpps+ bidirectional (18Mpps total) with IPv4 forwarding of
> 64 octet packets with 100GE Cx5 cards and Trex.
> 
> Thanks,
> Chris.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17249): https://lists.fd.io/g/vpp-dev/message/17249
Mute This Topic: https://lists.fd.io/mt/76242714/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Jenkins.fd.io is currently wedged -- LF ticket was opened last night

[Dave Wallace]

Folks,

Sorry for the late notice. I forgot to send this out on Friday.

Vexxhost was doing maintenance/upgrades on their infrastructure and they 
changed something in the service catalog to point to a new hostname.  
This was cached by Jenkins and caused the lockup.


Jenkins was restarted without clearing the build queue which resolved 
the issue.  The size of the build queue had nothing to do with the 
jenkins lockup which was confirmed by deleting the stable/1908 merge 
jobs from the queue prior several hours prior to the restart.


Thanks,
-daw-

On 8/14/20 7:56 AM, Dave Wallace via lists.fd.io wrote:
[0] 
https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-20423 



On 8/14/2020 7:56 AM, Dave Wallace wrote:

Folks,

While merging the stream of stable/1908 cherry-picks for the 19.08.3 
release, jenkins.fd.io locked up with ~300 merge jobs in the build 
queue. A ticket [0] was opened last night when the stoppage was 
detected and jenkins will likely need to be rebooted after some 
investigation.


Thank you for your patience.
-daw-






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17248): https://lists.fd.io/g/vpp-dev/message/17248
Mute This Topic: https://lists.fd.io/mt/76244437/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Question on performance tests..

[Christian Hopps]

[sent to csit-dev but perhaps since I'm not subscribed it didn't post]

I'm looking at this performance graph:

 https://docs.fd.io/csit/master/trending/trending/ip4-2n-clx-cx556a.html

Is this a uni-directional or a bi-directional test?

There is a huge performance gain between June and July it seems. Do we know 
which change this was? I'm on a current vpp master and I am only getting about 
9Mpps+ bidirectional (18Mpps total) with IPv4 forwarding of 64 octet packets 
with 100GE Cx5 cards and Trex.

Thanks,
Chris.


signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17247): https://lists.fd.io/g/vpp-dev/message/17247
Mute This Topic: https://lists.fd.io/mt/76242714/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vapi and counters for interfaces

[Ole Troan]

Hi Stanislav,

> I want to get counters for interfaces through the VAPI. Is it possible? I 
> can't find any methods to do it.
> 
> I am using C++ VAPI headers, but any references with C would be helpful too.
> 
> Or I would be glad if anyone would tell me the proper way to collect counters 
> for all interfaces in VPP.

VPP telemetry is exposed directly through a shared memory segment.
(Not using the binary VPP API).
You can access this directly via src/vpp-api/client/stat_client.h
or use the C++ wrapper in extras/vom/vom/stat_reader.hpp

Best regards,
Ole-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17246): https://lists.fd.io/g/vpp-dev/message/17246
Mute This Topic: https://lists.fd.io/mt/76205779/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-