Hi Raj, Inline.
> On May 16, 2020, at 2:30 PM, Raj Kumar <raj.gauta...@gmail.com> wrote: > > Hi Florin, > > I am using VPP 20.05 rc0 . Should I upgrade it ? FC: Not necessarily, as long as it’s relatively recent, i.e., it includes all of the recent udp updates. > > Thanks! for providing the patch, i will try it on Monday. Actually, I am > testing in a controlled environment where I can not change the VPP libraries. > I will try it on my server. FC: Sounds good. Let me know how it goes! > > On the UDP connection; yes, the error EINPROGRESS was there because I am > using non-blocking connection. Now, I am ignoring this error. > Sometime , VPP crashes when I kills my application ( not gracefully) even > when there is a single connection . FC: That might have to do with the app dying such that 1) it does not detach from vpp (e.g., sigkill and atexit function is not executed) 2) it dies with the message queue mutex held and 3) vpp tries to enqueue more events before detecting that it crashed (~30s). > > The good part is that now I am able to move connections on different cores by > connecting it on receiving the first packet and then re-binding the socket to > listen. > Basically, this approach works but I have not tested it thoroughly. However , > I am still in favor of using the UDPC connection. FC: If you have enough logic in your app to emulate a handshake, i.e., always have the client send a few bytes and wait for a reply from the server before opening a new connection, then this approach is probably more flexible from core placement perspective. The patch tries to emulate the old udpc with udp (udpc in vpp was confusing for consumers). You might get away with listening from multiple vcl workers on the same udp ip:port pair and have vpp load balance accepts between them, but I’ve never tested that. You can do this only with udp listeners that have been initialized as connected (so only with the patch). > > Btw, in trace logs I see some ssvm_delete related error when re-binding the > connection. FC: I think it’s fine. Going over the interactions step by step to see if understand what you’re doing (and hopefully help you understand what vpp does underneath). > > vpp# sh session verbose > Connection State Rx-f > Tx-f > [0:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:6677-LISTEN 0 0 > Thread 0: active sessions 1 > > Connection State Rx-f > Tx-f > [1:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:6677-OPENED 0 0 > Thread 1: active sessions 1 > > Connection State Rx-f > Tx-f > [2:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:6677-OPENED 0 0 > Thread 2: active sessions 1 > Thread 3: no sessions > Thread 4: no sessions > > VCL<24434>: configured VCL debug level (2) from VCL_DEBUG! > VCL<24434>: using default heapsize 268435456 (0x10000000) > VCL<24434>: allocated VCL heap = 0x7f7f18d1b010, size 268435456 (0x10000000) > VCL<24434>: using default configuration. > vppcom_connect_to_vpp:487: vcl<24434:0>: app (udp6_rx) connecting to VPP api > (/vpe-api)... > vppcom_connect_to_vpp:502: vcl<24434:0>: app (udp6_rx) is connected to VPP! > vppcom_app_create:1200: vcl<24434:0>: sending session enable > vppcom_app_create:1208: vcl<24434:0>: sending app attach > vppcom_app_create:1217: vcl<24434:0>: app_name 'udp6_rx', my_client_index 0 > (0x0) FC: Added worker 0 > vppcom_connect_to_vpp:487: vcl<24434:1>: app (udp6_rx-wrk-1) connecting to > VPP api (/vpe-api)... > vppcom_connect_to_vpp:502: vcl<24434:1>: app (udp6_rx-wrk-1) is connected to > VPP! > vcl_worker_register_with_vpp:262: vcl<24434:1>: added worker 1 > vl_api_app_worker_add_del_reply_t_handler:235: vcl<94:-1>: worker 1 > vpp-worker 1 added FC: Adding worker 1 > vppcom_epoll_create:2558: vcl<24434:1>: Created vep_idx 0 > vppcom_session_create:1279: vcl<24434:1>: created session 1 > vppcom_session_bind:1426: vcl<24434:1>: session 1 handle 16777217: binding to > local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port 6677, proto UDP > vppcom_session_listen:1458: vcl<24434:1>: session 16777217: sending vpp > listen request... > vcl_session_bound_handler:607: vcl<24434:1>: session 1 [0x0]: listen > succeeded! > vppcom_epoll_ctl:2658: vcl<24434:1>: EPOLL_CTL_ADD: vep_sh 16777216, sh > 16777217, events 0x1, data 0xffffffff! FC: Listened on session 1 and added it to epoll session 0 > udpRxThread started!!! ... rx port = 6677 > Waiting for a client to connect on port 6677 ... > vppcom_session_connect:1742: vcl<24434:1>: session handle 16777217 > (STATE_CLOSED): connecting to peer IPv6 2001:5b0:ffff:700:b883:31f:29e:9886 > port 40300 proto UDP FC: I guess at this point you got data on the listener so you now try to connect it to the peer. > vppcom_epoll_ctl:2696: vcl<24434:1>: EPOLL_CTL_MOD: vep_sh 16777216, sh > 16777217, events 0x2011, data 0x1000001! > vppcom_session_create:1279: vcl<24434:1>: created session 2 > vppcom_session_bind:1426: vcl<24434:1>: session 2 handle 16777218: binding to > local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port 6677, proto UDP > vppcom_session_listen:1458: vcl<24434:1>: session 16777218: sending vpp > listen request... FC: Request to create new listener. > vcl_session_app_add_segment_handler:855: vcl<24434:1>: mapped new segment > '24177-2' size 134217728 FC: This is probably the connects segment manager segment that was just created with the first segment in it. > vcl_session_connected_handler:505: vcl<24434:1>: session 1 [0x100000000] > connected! rx_fifo 0x224051c80, refcnt 1, tx_fifo 0x224051b80, refcnt 1 FC: Connect for previous listener (session 1) succeeded. > vcl_session_app_add_segment_handler:855: vcl<24434:1>: mapped new segment > '24177-3' size 134217728 FC: This is the new listener’s first segment manager segment. So session 2 has segment 24177-3 associated to it. > vcl_session_bound_handler:607: vcl<24434:1>: session 2 [0x0]: listen > succeeded! > vppcom_epoll_ctl:2658: vcl<24434:1>: EPOLL_CTL_ADD: vep_sh 16777216, sh > 16777218, events 0x1, data 0xffffffff! FC: Listen succeeded on session 2 and it was added to the vep group. > vcl_session_migrated_handler:674: vcl<24434:1>: Migrated 0x100000000 to > thread 2 0x200000000 FC: You got new data on the connected session (session 1) and the session was migrated to the rss selected thread in vpp. > new connecton > vppcom_session_connect:1742: vcl<24434:1>: session handle 16777218 > (STATE_CLOSED): connecting to peer IPv6 2001:5b0:ffff:700:b883:31f:29e:9886 > port 60725 proto UDP FC: Connecting session 2 (the latest listener) > vppcom_epoll_ctl:2696: vcl<24434:1>: EPOLL_CTL_MOD: vep_sh 16777216, sh > 16777218, events 0x2011, data 0x1000002! > vppcom_session_create:1279: vcl<24434:1>: created session 3 > vppcom_session_bind:1426: vcl<24434:1>: session 3 handle 16777219: binding to > local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port 6677, proto UDP > vppcom_session_listen:1458: vcl<24434:1>: session 16777219: sending vpp > listen request... FC: Trying to listen on a new session (session 3) > ssvm_delete_shm:205: unlink segment '24177-3': No such file or directory > (errno 2) FC: This is okay, I think, because vpp already deleted the shm segment (so there’s nothing left to delete). You might want to consider using memfd segments (although it involves a bit of configuration like here [1]). [1] https://wiki.fd.io/view/VPP/HostStack/LDP/iperf > vcl_segment_detach:467: vcl<24434:1>: detached segment 3 handle 0 > vcl_session_app_del_segment_handler:863: vcl<24434:1>: Unmapped segment: 0 FC: Because session 2 stopped listening, the underlying segment manager (all listeners have a segment manager in vpp) was removed. VPP forced vcl to unmap the segment as well. > vcl_session_connected_handler:505: vcl<24434:1>: session 2 [0x100000000] > connected! rx_fifo 0x224051a80, refcnt 1, tx_fifo 0x224051980, refcnt 1 > vcl_session_app_add_segment_handler:855: vcl<24434:1>: mapped new segment > '24177-4' size 134217728 > vcl_session_bound_handler:607: vcl<24434:1>: session 3 [0x0]: listen > succeeded! > vppcom_epoll_ctl:2658: vcl<24434:1>: EPOLL_CTL_ADD: vep_sh 16777216, sh > 16777219, events 0x1, data 0xffffffff! FC: New listener (session 3) got new segment (and segment manager) and it was added to epoll group. Regards, Florin > > thanks, > -Raj > > > On Sat, May 16, 2020 at 2:23 PM Florin Coras <fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>> wrote: > Hi Raj, > > Assuming you are trying to open more than one connected udp session, does > this [1] solve the problem (note it's untested)? > > To reproduce legacy behavior, this allows you to listen on VPPCOM_PROTO_UDPC > but that is now converted by vcl into a udp listen that propagates with a > “connected” flag to vpp. That should result in a udp listener that behaves > like an “old” udpc listener. > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/27111 > <https://gerrit.fd.io/r/c/vpp/+/27111> > >> On May 16, 2020, at 10:56 AM, Florin Coras via lists.fd.io >> <http://lists.fd.io/> <fcoras.lists=gmail....@lists.fd.io >> <mailto:fcoras.lists=gmail....@lists.fd.io>> wrote: >> >> Hi Raj, >> >> Are you using master latest/20.05 rc1 or something older? The fact that >> you’re getting a -115 (EINPROGRESS) suggests you might’ve marked the >> connection as “non-blocking” although you created it as blocking. If that’s >> so, the return value is not an error. >> >> Also, how if vpp crashing? Are you by chance trying to open a lot of udp >> connections back to back? >> >> Regards, >> Florin >> >>> On May 16, 2020, at 10:23 AM, Raj Kumar <raj.gauta...@gmail.com >>> <mailto:raj.gauta...@gmail.com>> wrote: >>> >>> Hi Florin, >>> I tried to connect on receiving the first UDP packet . But, it did not >>> work. I am getting error -115 in the application and VPP is crashing. >>> >>> This is something I tried in the code (udp receiver) - >>> sockfd = vppcom_session_create(VPPCOM_PROTO_UDP, 0); >>> rv_vpp = vppcom_session_bind (sockfd, &endpt); >>> if (FD_ISSET(session_idx, &readfds)) >>> { >>> n = vppcom_session_recvfrom(sockfd, (char *)buffer, MAXLINE, 0, >>> &client); >>> if(first_pkt) >>> rv_vpp = vppcom_session_connect (sockfd, &client); >>> //Here getting rv_vpp as -115 >>> } >>> Please let me know if I am doing something wrong. >>> >>> Here are the traces - >>> >>> VCL<16083>: configured VCL debug level (2) from VCL_DEBUG! >>> VCL<16083>: using default heapsize 268435456 (0x10000000) >>> VCL<16083>: allocated VCL heap = 0x7fd255ed2010, size 268435456 (0x10000000) >>> VCL<16083>: using default configuration. >>> vppcom_connect_to_vpp:487: vcl<16083:0>: app (udp6_rx) connecting to VPP >>> api (/vpe-api)... >>> vppcom_connect_to_vpp:502: vcl<16083:0>: app (udp6_rx) is connected to VPP! >>> vppcom_app_create:1200: vcl<16083:0>: sending session enable >>> vppcom_app_create:1208: vcl<16083:0>: sending app attach >>> vppcom_app_create:1217: vcl<16083:0>: app_name 'udp6_rx', my_client_index 0 >>> (0x0) >>> >>> vppcom_connect_to_vpp:487: vcl<16083:1>: app (udp6_rx-wrk-1) connecting to >>> VPP api (/vpe-api)... >>> vppcom_connect_to_vpp:502: vcl<16083:1>: app (udp6_rx-wrk-1) is connected >>> to VPP! >>> vl_api_app_worker_add_del_reply_t_handler:235: vcl<94:-1>: worker 1 >>> vpp-worker 1 added >>> vcl_worker_register_with_vpp:262: vcl<16083:1>: added worker 1 >>> vppcom_session_create:1279: vcl<16083:1>: created session 0 >>> vppcom_session_bind:1426: vcl<16083:1>: session 0 handle 16777216: binding >>> to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port 6677, proto >>> UDP >>> vppcom_session_listen:1458: vcl<16083:1>: session 16777216: sending vpp >>> listen request... >>> vcl_session_bound_handler:607: vcl<16083:1>: session 0 [0x0]: listen >>> succeeded! >>> vppcom_session_connect:1742: vcl<16083:1>: session handle 16777216 >>> (STATE_CLOSED): connecting to peer IPv6 2001:5b0:ffff:700:b883:31f:29e:9886 >>> port 51190 proto UDP >>> udpRxThread started!!! ... rx port = 6677vppcom_session_connect() failed >>> ... -115 >>> vcl_session_cleanup:1300: vcl<16083:1>: session 0 [0x0] closing >>> vcl_worker_cleanup_cb:190: vcl<94:-1>: cleaned up worker 1 >>> vl_client_disconnect:309: peer unresponsive, give up >>> >>> thanks, >>> -Raj >>> >>> >>> On Fri, May 15, 2020 at 8:10 PM Florin Coras <fcoras.li...@gmail.com >>> <mailto:fcoras.li...@gmail.com>> wrote: >>> Hi Raj, >>> >>> There are no explicit vcl apis that allow a udp listener to be switched to >>> connected mode. We might decide to do this at one point through a new bind >>> api (non-posix like) since we do support this for builtin applications. >>> >>> However, you now have the option of connecting a bound session. That is, on >>> the first received packet on a udp listener, you can grab the peer’s >>> address and connect it. Iperf3 in udp mode, which is part of our make test >>> infra, does exactly that. Subsequently, it re-binds the port to accept more >>> connections. Would that work for you? >>> >>> Regards, >>> Florin >>> >>>> On May 15, 2020, at 4:06 PM, Raj Kumar <raj.gauta...@gmail.com >>>> <mailto:raj.gauta...@gmail.com>> wrote: >>>> >>>> Thanks! Florin, >>>> >>>> OK, I understood that I need to change my application to use UDP socket >>>> and then use vppcom_session_connect(). >>>> This is fine for the UDP client ( sender) . >>>> >>>> But ,in UDP Server ( receiver) , I am not sure how to use the >>>> vppcom_session_connect(). . >>>> I am using vppcom_session_listen() to listen on the connections and then >>>> calling vppcom_session_accept() to accept a new connection. >>>> >>>> With UDPC, I was able to utilize the RSS ( receiver side scaling) feature >>>> to move the received connections on the different cores /threads. >>>> >>>> Just want to confirm if I can achieve the same with UDP. >>>> >>>> I will change my application and will update you about the result. >>>> >>>> Thanks, >>>> -Raj >>>> >>>> >>>> On Fri, May 15, 2020 at 5:17 PM Florin Coras <fcoras.li...@gmail.com >>>> <mailto:fcoras.li...@gmail.com>> wrote: >>>> Hi Raj, >>>> >>>> We removed udpc transport in vpp. I’ll push a patch that removes it from >>>> vcl as well. >>>> >>>> Calling connect on a udp connection will give you connected semantics now. >>>> Let me know if that solves the issue for you. >>>> >>>> Regards, >>>> Florin >>>> >>>> >>>>> On May 15, 2020, at 12:15 PM, Raj Kumar <raj.gauta...@gmail.com >>>>> <mailto:raj.gauta...@gmail.com>> wrote: >>>>> >>>>> Hi, >>>>> I am getting segmentation fault in VPP when using VCL VPPCOM_PROTO_UDPC >>>>> socket. This issue is observed with both UDP sender and UDP receiver >>>>> application. >>>>> >>>>> However, both UDP sender and receiver works fine with VPPCOM_PROTO_UDP. >>>>> >>>>> Here is the stack trace - >>>>> >>>>> (gdb) bt >>>>> #0 0x0000000000000000 in ?? () >>>>> #1 0x00007ffff775da59 in session_open_vc (app_wrk_index=1, >>>>> rmt=0x7fffb5e34cc0, opaque=0) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session.c:1217 >>>>> #2 0x00007ffff7779257 in session_mq_connect_handler (data=0x7fffb676e7a8) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_node.c:138 >>>>> #3 0x00007ffff7780f48 in session_event_dispatch_ctrl >>>>> (elt=0x7fffb643f51c, wrk=0x7fffb650a640) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session.h:262 >>>>> #4 session_queue_node_fn (vm=<optimized out>, node=<optimized out>, >>>>> frame=<optimized out>) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_node.c:1409 >>>>> #5 0x00007ffff6b214c1 in dispatch_node (last_time_stamp=<optimized out>, >>>>> frame=0x0, dispatch_state=VLIB_NODE_STATE_POLLING, >>>>> type=VLIB_NODE_TYPE_INPUT, node=0x7fffb5a9a980, vm=0x7ffff6d7c200 >>>>> <vlib_global_main>) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1235 >>>>> #6 vlib_main_or_worker_loop (is_main=1, vm=0x7ffff6d7c200 >>>>> <vlib_global_main>) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1815 >>>>> #7 vlib_main_loop (vm=0x7ffff6d7c200 <vlib_global_main>) at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1990 >>>>> #8 vlib_main (vm=<optimized out>, vm@entry=0x7ffff6d7c200 >>>>> <vlib_global_main>, input=input@entry=0x7fffb5e34fa0) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:2236 >>>>> #9 0x00007ffff6b61756 in thread0 (arg=140737334723072) at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/unix/main.c:658 >>>>> #10 0x00007ffff602fc0c in clib_calljmp () from /lib64/libvppinfra.so.20.05 >>>>> #11 0x00007fffffffd1e0 in ?? () >>>>> #12 0x00007ffff6b627ed in vlib_unix_main (argc=<optimized out>, >>>>> argv=<optimized out>) >>>>> at >>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/unix/main.c:730 >>>>> >>>>> Earlier , I tested this functionality with VPP 20.01 release with the >>>>> following patches and it worked perfectly. >>>>> https://gerrit.fd.io/r/c/vpp/+/24332 >>>>> <https://gerrit.fd.io/r/c/vpp/+/24332> >>>>> https://gerrit.fd.io/r/c/vpp/+/24334 >>>>> <https://gerrit.fd.io/r/c/vpp/+/24334> >>>>> https://gerrit.fd.io/r/c/vpp/+/24462 >>>>> <https://gerrit.fd.io/r/c/vpp/+/24462> >>>>> >>>>> Thanks, >>>>> -Raj >>>>> >>>> >>> >> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16436): https://lists.fd.io/g/vpp-dev/message/16436 Mute This Topic: https://lists.fd.io/mt/74234856/21656 Mute #vpp-hoststack: https://lists.fd.io/mk?hashtag=vpp-hoststack&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
