[vpp-dev] Is there any Linux FD to poll for VCL message

[Satya Murthy]

Hi ,

We have a TCP application trying integrate with VPP-VCL framework.

Our application has its own dispatch loop with epoll and we would like to know 
if VCL framework has any linux fd ( like an eventfd for the entire svm message 
queue ) that we can add into our epoll to poll for VCL session messages.

Once we get an asynchronous indication that a message has arrived in the VCL 
svm message queue, we can call vppcom_epoll_wait() function to read the 
messages for sessions and handle them accordingly.

Any inputs on how we can achieve this?

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15750): https://lists.fd.io/g/vpp-dev/message/15750
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Florin Coras]

Hi Murthy, 

Yes it does, although we’re guilty of not having it properly documented. 

The message queue used between vpp and a vcl worker can do both mutex/condvar 
and eventfd notifications. The former is the default but you can switch to 
eventfds by adding to vcl.conf "use-mq-eventfd”. You can then use 
vppcom_worker_mqs_epfd to retrieve a vcl worker's epoll fd (it’s an epoll fd 
for historic reasons) which you should be able to nest into your own linux 
epoll fd. 

Note that you’ll also need to force memfd segments for vpp’s message queues, 
i.e., session { evt_qs_memfd_seg }, and use the socket transport for binary 
api, i.e., in vpp’s startup.conf add "socksvr { /path/to/api.sock }" and in 
vcl.conf "api-socket-name /path/to/api.sock”. 

Regards,
Florin

> On Mar 12, 2020, at 4:40 AM, Satya Murthy  wrote:
> 
> Hi ,
> 
> We have a TCP application trying integrate with VPP-VCL framework.
> 
> Our application has its own dispatch loop with epoll and we would like to 
> know if VCL framework has any linux fd ( like an eventfd for the entire svm 
> message queue ) that we can add into our epoll to poll for VCL session 
> messages.
> 
> Once we get an asynchronous indication that a message has arrived in the VCL 
> svm message queue, we can call vppcom_epoll_wait() function to read the 
> messages for sessions and handle them accordingly. 
> 
> Any inputs on how we can achieve this?
> 
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15760): https://lists.fd.io/g/vpp-dev/message/15760
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Satya Murthy]

Hi Florin,

Thank you very much for the inputs.
These are very difficult to understand unless we go through the code in detail.
Today, Whole day, I was trying to follow your instructions and get this working 
by looking at the code as well.
However, I am not fully successful.
Before going further, I would like to get my understanding clear, so that it 
will form basis for my debugging.

Here are couple of questions:
1)
==
The message queue used between vpp and a vcl worker can do both mutex/condvar 
and eventfd notifications. The former is the default but you can switch to 
eventfds by adding to vcl.conf "use-mq-eventfd”. You can then use 
vppcom_worker_mqs_epfd to retrieve a vcl worker's epoll fd (it’s an epoll fd 
for historic reasons) which you should be able to nest into your own linux 
epoll fd.
==
The message queues between VPP and VCL can be present either in "shared memory" 
(or) "memfd segments".
For eventfd to work, the queues need to be present in the "memfd segments".
Is this correct understanding ?

2)
==
Note that you’ll also need to force memfd segments for vpp’s message queues, 
i.e., session { evt_qs_memfd_seg }, and use the socket transport for binary 
api, i.e., in vpp’s startup.conf add "socksvr { /path/to/api.sock }" and in 
vcl.conf "api-socket-name /path/to/api.sock”.
==
I didnt understand the reason for moving the binary api to sockets.
Is this due to shm/memfd wont be used at the same time ?

3)
In a nut shell:

VCL-APP   VPP
VCL-APP  <---Binary Api via LinuxDomain 
Sockets--> VPP

We will have two api clients with this model. One is shared memory client and 
other is a socket client.

Is my understanding correct ?
--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15771): https://lists.fd.io/g/vpp-dev/message/15771
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Florin Coras]

Hi Murthy, 

Inline. 

> On Mar 13, 2020, at 4:54 AM, Satya Murthy  wrote:
> 
> Hi Florin,
> 
> Thank you very much for the inputs.
> These are very difficult to understand unless we go through the code in 
> detail.
> Today, Whole day, I was trying to follow your instructions and get this 
> working by looking at the code as well.

I’d recommend starting here 
https://wiki.fd.io/images/9/9c/Vpp-hoststack-kc-eu19.pdf

> However, I am not fully successful. 
> Before going further, I would like to get my understanding clear, so that it 
> will form basis for my debugging.
> 
> Here are couple of questions:
> 1) 
> ==
> The message queue used between vpp and a vcl worker can do both mutex/condvar 
> and eventfd notifications. The former is the default but you can switch to 
> eventfds by adding to vcl.conf "use-mq-eventfd”. You can then use 
> vppcom_worker_mqs_epfd to retrieve a vcl worker's epoll fd (it’s an epoll fd 
> for historic reasons) which you should be able to nest into your own linux 
> epoll fd. 
> ==
> The message queues between VPP and VCL can be present either in "shared 
> memory" (or) "memfd segments".
> For eventfd to work, the queues need to be present in the "memfd segments".
> Is this correct understanding ?

FC: VCL workers and vpp use message queues to exchange io/ctrl messages and 
fifos to exchange data. Both are allocated in shared memory segments which can 
be of two flavors, POSIX shm segments (shm_open) or memfd based (memfd 
exchanged over a unix socket). For the latter to work, we use the binary api’s 
socket transport to exchange the fds.

If configured to use eventds for message queue signaling, vpp must exchange 
those eventfds with the vcl workers. For that, we also use the binary api’s 
socket. That should explain why the binary api’s socket transport is needed. 
Note also that this is just a configuration from vcl consumer perspective, it 
should not otherwise affect the app. 

> 
> 2) 
> ==
> Note that you’ll also need to force memfd segments for vpp’s message queues, 
> i.e., session { evt_qs_memfd_seg }, and use the socket transport for binary 
> api, i.e., in vpp’s startup.conf add "socksvr { /path/to/api.sock }" and in 
> vcl.conf "api-socket-name /path/to/api.sock”. 
> ==
> I didnt understand the reason for moving the binary api to sockets.
> Is this due to shm/memfd wont be used at the same time ? 

FC: I hope it’s clear now why you need to move to the binary api’s socket 
transport. 

> 
> 3) 
> In a nut shell:
> 
> VCL-APP   -> VPP
> VCL-APP  <---Binary Api via LinuxDomain 
> Sockets--> VPP
> 
> We will have two api clients with this model. One is shared memory client and 
> other is a socket client.

FC: If you mean to point out that there are two channels from vcl to vpp, 
that’s correct. The first is the one described above, but note that the message 
queues are not bidirectional. VPP has another set of message queues the apps 
use to enqueue notifications towards vpp.

As for the second channel, the binary api, it’s used to 1) attach vcl to the 
session layer, 2) its socket is used for exchanging fds (memfds and eventfds) 
3) sometimes for exchanging configuration. 

But again, apart from configuration changes, this should be completely 
transparent to vcl consumers. 

Regards,
Florin

> 
> Is my understanding correct ? 
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15778): https://lists.fd.io/g/vpp-dev/message/15778
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Satya Murthy]

Hi Florin,

Thanks a lot for the detailed explanation. This kind of gives an overview of 
this area, which really helps in our integration.

Just one more question:
We are planning to remove the concept of vcl worker in our worker, as our app 
is a single threaded app and will not be multi-threaded at any point of time in 
future as well. Hope this is doable and does not pose any restrictions in any 
of the VCL s/w layers.

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15780): https://lists.fd.io/g/vpp-dev/message/15780
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Florin Coras]

Hi Murthy, 

Glad it helps! By construction, when vcl is initialized (vppcom_app_create()) 
only one worker is initialized. As long as you don’t register other workers 
(vppcom_worker_register), you don’t really have to worry about anything else 
and there are no performance penalties.

It’s important to remember that vcl is not thread safe by itself, i.e., workers 
will never take any locks exactly because we want them to be as light weight as 
possible. If the application does not want to manage locking/forking, it should 
use vcl_locked (vls) but that’s an entirely different topic. 

Regards,
Florin

> On Mar 13, 2020, at 9:49 AM, Satya Murthy  wrote:
> 
> Hi Florin,
> 
> Thanks a lot for the detailed explanation. This kind of gives an overview of 
> this area, which really helps in our integration.
> 
> Just one more question:
> We are planning to remove the concept of vcl worker in our worker, as our app 
> is a single threaded app and will not be multi-threaded at any point of time 
> in future as well. Hope this is doable and does not pose any restrictions in 
> any of the VCL s/w layers.
>
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15781): https://lists.fd.io/g/vpp-dev/message/15781
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Satya Murthy]

Hi Florin,

Over the weekend, I went through the document that you mentioned and it gave me 
a good overview. Thanks for pointing to that doc.
However, my task of integrating mqs->epfd into our main dispatch loop still 
seems to be needing more finer details of the code.

With this respect, I have following few queries. It would be great if you can 
help with this.
If this path poses more effort/risk in integration, we may want to switch to 
host interface approach ( which has its own issues though :(  )

1) Terminology mapping.
In the VCL application framework, is the following mapping a correct 
understanding (with some open items) ?

TCP-Application => app with a specific client index
Worker          => thread in the tcp-application
Session         => tcp end point
mq in worker    => mapped to ?
mq_evt_conn     => mapped to ?

2) Howcome vppcom_epoll_create is creating a new session. Bit confusing here.

3) Is the TCP server listen_fd also can be integrated into our application 
dispatch loop ?

4) How to get hold of mqs->epfd to put into our dispatcher?
I see that the vcl_worker_t is inside vcl_private.c and .h which are not 
accessible by the application directly.
Also, i dont see a direct need of having vcl_test_worker* framework in our app. 
Hope this is fine.

5) why can't the TCP application directly use vcl_worker_t data structure ?

6) Is there a sample tcp server example that has the server implementation with 
linux epoll instead of vppcom_epoll system.

7) Do you think this approach of integration will be more complex and hence do 
you suggest us moving to LDP (or) host interface approach ?
Please let us know.

Thanks in advance for your time.

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15788): https://lists.fd.io/g/vpp-dev/message/15788
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Is there any Linux FD to poll for VCL message

[Florin Coras]

Hi Murthy, 

Inline.

> On Mar 16, 2020, at 12:35 AM, Satya Murthy  wrote:
> 
> Hi Florin,
>
> Over the weekend, I went through the document that you mentioned and it gave 
> me a good overview. Thanks for pointing to that doc.
> However, my task of integrating mqs->epfd into our main dispatch loop still 
> seems to be needing more finer details of the code.

Glad it helped. 

>
> With this respect, I have following few queries. It would be great if you can 
> help with this.
> If this path poses more effort/risk in integration, we may want to switch to 
> host interface approach ( which has its own issues though :(  )

This does need more effort but as long as you understand what you’re doing it 
should be fine. 

>
> 1) Terminology mapping.
> In the VCL application framework, is the following mapping a correct 
> understanding (with some open items) ?
>
> TCP-Application => app with a specific client index

I’d call the app an application that needs a tcp connection. And yes, it will 
have a client index in vpp. 

> Worker  => thread in the tcp-application 

For your use case, yes. New threads are not implicitly mapped to new workers. 
 
> Session => tcp end point

Session in vcl is equivalent to an fd. The actual tcp endpoint is in the tcp 
transport layer which sits under the session layer in vpp.

> mq in worker=> mapped to ?

There’s no mapping here. The message queue is the “channel” on which vcl 
receives io/ctrl notifications from vpp. 

> mq_evt_conn => mapped to ?

Applications don’t have access to this. They only have access to the “mqs epoll 
fd”, i.e., mqs_epfd. Previously, the workers could’ve had more fds in their 
mqs_epfd but today we can only have 1, the mq’s eventfd, if one was configured. 
Initially, this mqs_epfd was used to epoll all of the fds in it, now it’s used 
to epoll the mq’s eventfd. In the future, this could be converted to a normal 
fd, but it should be transparent for the app. 

>
> 2) Howcome vppcom_epoll_create is creating a new session. Bit confusing here.

That’s like creating a new fd, i.e., you’re given a “session handle” in return. 
Why would it be confusing? 

>
> 3) Is the TCP server listen_fd also can be integrated into our application 
> dispatch loop ? 

Not sure I understood the question. Are you asking if you can integrate a 
listener fd in a vcl epoll fd? Then, yes. 

>
> 4) How to get hold of mqs->epfd to put into our dispatcher? 
>I see that the vcl_worker_t is inside vcl_private.c and .h which are not 
> accessible by the application directly.
>Also, i dont see a direct need of having vcl_test_worker* framework in our 
> app. Hope this is fine.

vcl_test_worker is a test toy app, so you really don’t need it. Applications 
should only use vppcom.h and the function you’re looking for is 
vppcom_worker_mqs_epfd. That will return the worker’s mqs_epfd which you can 
afterwards EPOLL_CTL_ADD to your linux epoll fd. 

>
> 5) why can't the TCP application directly use vcl_worker_t data structure ?

Because we don’t want applications to interact/depend on vcl internals. Also, 
the goal with vcl is to have a posix-like api. You should not need more than 
the session handles and the apis exposed in vppcom.h

>   
> 6) Is there a sample tcp server example that has the server implementation 
> with linux epoll instead of vppcom_epoll system.

A simple one, no, as far as I know. This mechanism was used in the envoy 
integration done here [1], but that’s not probably the easiest thing to follow. 

[1] https://github.com/sbelair2/envoy/tree/vpp_integration 


>
> 7) Do you think this approach of integration will be more complex and hence 
> do you suggest us moving to LDP (or) host interface approach ?
>Please let us know.

It will be slightly more complicated because you’ll be “manually” tracking the 
mq’s activity. But in terms of actual extra code needed, you only need to make 
sure that whenever you get an epoll event on the mqs_epfd, you call 
vppcom_epoll_wait() with the vcl epoll fd to which you’ve added your app’s 
sessions (probably including the listen_fd you mentioned above). 

>
> Thanks in advance for your time.

Hope it helps. 

Regards, 
Florin

>
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15791): https://lists.fd.io/g/vpp-dev/message/15791
Mute This Topic: https://lists.fd.io/mt/71899986/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-