Hi Damjan, Thanks for your quick response. Will try tapv2.
Are there any recommendations / best_practices for vpp to send/receive packets to/from a socket based application running side by side? Will a veth pair be any better than tapv2 in this case? -R > On Jan 24, 2019, at 11:43 PM, Damjan Marion <dmar...@me.com> wrote: > > > Don't use tapcli code, it is outdated and I just submitted patch to deprecate > it (as agreed few months ago on the community call). > > https://gerrit.fd.io/r/17073 > > tapv2 (create interface tap) should be able to run on worker, but don't > expect magic, it's jus tap.... > > >> On 25 Jan 2019, at 08:14, ranadip....@gmail.com wrote: >> >> Hello, >> >> I am trying to use vpp with 4 worker threads. >> Client is sending packet to server through vpp. This packet comes to vpp, >> and gets picked up from a VF. VPP sends this packet to the another >> user-space application over a tapcli interface. >> The user-space application processes the packet, and then sends it back to >> vpp. Vpp forwards the packet to the server. >> >> I am using vpp with 4 worker threads. >> >> Without the traffic, four worker threads are all showing ~100% cpu usage >> (which is expected). However, with traffic, the vpp main thread is also >> showing ~100% cpu usage. I do not want vpp main thread to reach 100% cpu. >> With the limited knowledge that I have on vpp, it seems like main thread is >> doing lots of processing in the tapcli-rx node. Is there a way to keep vpp >> main relatively free? Is tap interface the correct choice to communicate >> between vpp and another user-space application? If not, what should be used? >> >> I have tried this on both vpp 18.07 and 18.10. >> >> vpp# show run >> Thread 0 vpp_main (lcore 2) >> Time 916.9, average vectors/node 40.16, last 128 main loops 0.00 per node >> 0.00 >> vector rates in 1.3178e5, out 1.3178e5, drop 1.3087e-2, punt 0.0000e0 >> Name State Calls Vectors >> Suspends Clocks Vectors/Call >> VirtualFunctionEthernet3/10/1- active 3008704 120840447 >> 0 1.36e1 40.16 >> VirtualFunctionEthernet3/10/1- active 3008704 120840447 >> 0 1.02e2 40.16 >> acl-plugin-fa-cleaner-process event wait 0 0 >> 1 1.85e4 0.00 >> admin-up-down-process event wait 0 0 >> 1 1.20e3 0.00 >> api-rx-from-ring any wait 0 0 >> 50 7.80e4 0.00 >> arp-input active 1 1 >> 0 2.39e6 1.00 >> avf-process event wait 0 0 >> 1 2.14e3 0.00 >> bfd-process event wait 0 0 >> 1 1.84e3 0.00 >> bond-process event wait 0 0 >> 1 1.44e3 0.00 >> cdp-process any wait 0 0 >> 1 1.38e7 0.00 >> dhcp-client-process any wait 0 0 >> 10 1.02e4 0.00 >> dhcp6-client-cp-process any wait 0 0 >> 1 1.07e3 0.00 >> dhcp6-pd-client-cp-process any wait 0 0 >> 1 9.28e2 0.00 >> dhcp6-pd-reply-publisher-proce event wait 0 0 >> 1 8.88e2 0.00 >> dhcp6-reply-publisher-process event wait 0 0 >> 1 1.02e3 0.00 >> dns-resolver-process any wait 0 0 >> 1 2.18e3 0.00 >> dpdk-ipsec-process done 1 0 >> 0 1.29e5 0.00 >> dpdk-process any wait 0 0 >> 306 5.73e4 0.00 >> error-drop active 11 12 >> 0 5.73e3 1.09 >> ethernet-input active 3008710 120840454 >> 0 3.78e1 40.16 >> fib-walk any wait 0 0 >> 459 3.03e3 0.00 >> flow-report-process any wait 0 0 >> 1 6.92e2 0.00 >> flowprobe-timer-process any wait 0 0 >> 1 3.09e3 0.00 >> igmp-timer-process event wait 0 0 >> 1 2.18e3 0.00 >> ikev2-manager-process any wait 0 0 >> 916 2.46e3 0.00 >> ioam-export-process any wait 0 0 >> 1 1.10e3 0.00 >> ip-neighbor-scan-process any wait 0 0 >> 16 7.03e3 0.00 >> ip-route-resolver-process any wait 0 0 >> 10 8.13e3 0.00 >> ip4-glean active 2 2 >> 0 9.34e3 1.00 >> ip4-input active 3008704 120840447 >> 0 4.03e1 40.16 >> ip4-lookup active 3008704 120840447 >> 0 3.67e1 40.16 >> ip4-reassembly-expire-walk any wait 0 0 >> 92 3.96e3 0.00 >> ip4-rewrite active 3008702 120840445 >> 0 3.08e1 40.16 >> ip6-icmp-neighbor-discovery-ev any wait 0 0 >> 916 2.00e3 0.00 >> ip6-input active 5 6 >> 0 1.17e3 1.20 >> ip6-not-enabled active 5 6 >> 0 5.45e2 1.20 >> ip6-reassembly-expire-walk any wait 0 0 >> 92 4.30e3 0.00 >> l2fib-mac-age-scanner-process event wait 0 0 >> 1 1.78e3 0.00 >> lacp-process event wait 0 0 >> 1 1.41e7 0.00 >> lisp-retry-service any wait 0 0 >> 459 4.11e3 0.00 >> lldp-process event wait 0 0 >> 1 1.31e7 0.00 >> memif-process event wait 0 0 >> 1 4.28e3 0.00 >> nat-det-expire-walk done 1 0 >> 0 1.23e3 0.00 >> nat64-expire-walk event wait 0 0 >> 1 1.46e4 0.00 >> rd-cp-process any wait 0 0 >> 1 9.16e2 0.00 >> send-dhcp6-client-message-proc any wait 0 0 >> 1 2.88e3 0.00 >> send-dhcp6-pd-client-message-p any wait 0 0 >> 1 1.02e3 0.00 >> send-garp-na-process event wait 0 0 >> 1 2.88e4 0.00 >> send-rs-process any wait 0 0 >> 1 1.24e3 0.00 >> startup-config-process done 1 0 >> 1 3.75e3 0.00 >> tapcli-rx interrupt wa 5382120 120840506 >> 0 2.42e3 22.45 >> udp-ping-process any wait 0 0 >> 1 5.70e3 0.00 >> unix-cli-local:6 active 6 0 >> 757 3.46e15 0.00 >> unix-epoll-input polling 16312904 0 >> 0 6.94e4 0.00 >> vhost-user-process any wait 0 0 >> 1 1.32e4 0.00 >> vhost-user-send-interrupt-proc any wait 0 0 >> 1 1.03e3 0.00 >> vpe-link-state-process event wait 0 0 >> 3 4.95e3 0.00 >> vpe-oam-process any wait 0 0 >> 450 2.86e3 0.00 >> vxlan-gpe-ioam-export-process any wait 0 0 >> 1 8.76e2 0.00 >> wildcard-ip4-arp-publisher-pro event wait 0 0 >> 1 1.09e3 0.00 >> --------------- >> Thread 1 vpp_wk_0 (lcore 3) >> Time 916.9, average vectors/node 1.43, last 128 main loops 0.00 per node 0.00 >> vector rates in 3.2809e4, out 3.2809e4, drop 5.6599e-1, punt 0.0000e0 >> Name State Calls Vectors >> Suspends Clocks Vectors/Call >> arp-input active 518 518 >> 0 7.74e3 1.00 >> dpdk-input polling 3205687180 30084768 >> 0 4.31e4 0.00 >> error-drop active 519 519 >> 0 1.28e3 1.00 >> ethernet-input active 518 518 >> 0 1.73e3 1.00 >> ip4-glean active 1 1 >> 0 6.62e4 1.00 >> ip4-input-no-checksum active 20966739 30084250 >> 0 3.64e2 1.43 >> ip4-lookup active 20966739 30084250 >> 0 2.46e2 1.43 >> ip4-rewrite active 20966738 30084249 >> 0 1.93e2 1.43 >> tapcli-0-output active 20966739 30084250 >> 0 2.03e2 1.43 >> tapcli-0-tx active 20966739 30084250 >> 0 1.29e4 1.43 >> --------------- >> Thread 2 vpp_wk_1 (lcore 4) >> Time 916.9, average vectors/node 1.45, last 128 main loops 0.00 per node 0.00 >> vector rates in 3.2819e4, out 3.2819e4, drop 0.0000e0, punt 0.0000e0 >> Name State Calls Vectors >> Suspends Clocks Vectors/Call >> dpdk-input polling 3062149112 30094113 >> 0 4.24e4 0.00 >> ip4-input-no-checksum active 20687846 30094113 >> 0 3.67e2 1.45 >> ip4-lookup active 20687846 30094113 >> 0 2.59e2 1.45 >> ip4-rewrite active 20687846 30094113 >> 0 2.03e2 1.45 >> tapcli-0-output active 20687846 30094113 >> 0 2.11e2 1.45 >> tapcli-0-tx active 20687846 30094113 >> 0 1.31e4 1.45 >> --------------- >> Thread 3 vpp_wk_2 (lcore 5) >> Time 916.9, average vectors/node 1.46, last 128 main loops 0.00 per node 0.00 >> vector rates in 3.2774e4, out 3.2774e4, drop 2.4428e-1, punt 0.0000e0 >> Name State Calls Vectors >> Suspends Clocks Vectors/Call >> dpdk-input polling 2997474079 30052436 >> 0 4.33e4 .01 >> error-drop active 224 224 >> 0 1.57e3 1.00 >> ip4-drop active 224 224 >> 0 8.69e2 1.00 >> ip4-input-no-checksum active 20609931 30052436 >> 0 3.74e2 1.46 >> ip4-lookup active 20609931 30052436 >> 0 2.64e2 1.46 >> ip4-rewrite active 20609737 30052212 >> 0 2.01e2 1.46 >> tapcli-0-output active 20609737 30052212 >> 0 2.15e2 1.46 >> tapcli-0-tx active 20609737 30052212 >> 0 1.31e4 1.46 >> --------------- >> Thread 4 vpp_wk_3 (lcore 6) >> Time 916.9, average vectors/node 1.45, last 128 main loops 0.00 per node 0.00 >> vector rates in 3.2811e4, out 3.2811e4, drop 0.0000e0, punt 0.0000e0 >> Name State Calls Vectors >> Suspends Clocks Vectors/Call >> dpdk-input polling 3135905643 30086296 >> 0 4.30e4 0.00 >> ip4-input-no-checksum active 20714610 30086296 >> 0 3.74e2 1.45 >> ip4-lookup active 20714610 30086296 >> 0 2.55e2 1.45 >> ip4-rewrite active 20714610 30086296 >> 0 1.97e2 1.45 >> tapcli-0-output active 20714610 30086296 >> 0 2.04e2 1.45 >> tapcli-0-tx active 20714610 30086296 >> 0 1.31e4 1.45 >> >> Here is one of the packets traced on vpp main thread. >> >> vpp# show trace >> ------------------- Start of thread 0 vpp_main ------------------- >> Packet 1 >> >> 00:11:26:605715: tapcli-rx >> tapcli-0 >> 00:11:26:605725: ethernet-input >> IP4: c2:aa:32:4f:d8:1c -> 02:fe:d9:0f:5c:f4 >> 00:11:26:605741: ip4-input >> TCP: 10.0.1.1 -> 172.18.60.74 >> tos 0x00, ttl 64, length 60, checksum 0x475f >> fragment id 0x0000, flags DONT_FRAGMENT >> TCP: 82 -> 60818 >> seq. 0xcbbfa586 ack 0x36bb48c6 >> flags 0x12 SYN ACK, tcp header: 40 bytes >> window 28960, checksum 0xab2b >> 00:11:26:605748: ip4-lookup >> fib 0 dpo-idx 3 flow hash: 0x00000000 >> TCP: 10.0.1.1 -> 172.18.60.74 >> tos 0x00, ttl 64, length 60, checksum 0x475f >> fragment id 0x0000, flags DONT_FRAGMENT >> TCP: 82 -> 60818 >> seq. 0xcbbfa586 ack 0x36bb48c6 >> flags 0x12 SYN ACK, tcp header: 40 bytes >> window 28960, checksum 0xab2b >> 00:11:26:605754: ip4-rewrite >> tx_sw_if_index 1 dpo-idx 3 : ipv4 via 172.18.60.74 >> VirtualFunctionEthernet3/10/1: mtu:9000 ac1f6b17f379da78bd634ef70800 flow >> hash: 0x00000000 >> 00000000: ac1f6b17f379da78bd634ef708004500003c000040003f06485f0a000101ac12 >> 00000020: 3c4a0052ed92cbbfa58636bb48c6a0127120ab2b0000020405b40402 >> 00:11:26:605755: VirtualFunctionEthernet3/10/1-output >> VirtualFunctionEthernet3/10/1 >> IP4: da:78:bd:63:4e:f7 -> ac:1f:6b:17:f3:79 >> TCP: 10.0.1.1 -> 172.18.60.74 >> tos 0x00, ttl 63, length 60, checksum 0x485f >> fragment id 0x0000, flags DONT_FRAGMENT >> TCP: 82 -> 60818 >> seq. 0xcbbfa586 ack 0x36bb48c6 >> flags 0x12 SYN ACK, tcp header: 40 bytes >> window 28960, checksum 0xab2b >> 00:11:26:605758: VirtualFunctionEthernet3/10/1-tx >> VirtualFunctionEthernet3/10/1 tx queue 0 >> buffer 0x23e5a0: current data 0, length 74, free-list 0, clone-count 0, >> totlen-nifb 0, trace 0x0 >> l2-hdr-offset 0 l3-hdr-offset 14 >> PKT MBUF: port 65535, nb_segs 1, pkt_len 74 >> buf_len 2176, data_len 74, ol_flags 0x0, data_off 128, phys_addr >> 0xe7996880 >> packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >> rss 0x0 fdir.hi 0x0 fdir.lo 0x0 >> IP4: da:78:bd:63:4e:f7 -> ac:1f:6b:17:f3:79 >> TCP: 10.0.1.1 -> 172.18.60.74 >> tos 0x00, ttl 63, length 60, checksum 0x485f >> fragment id 0x0000, flags DONT_FRAGMENT >> TCP: 82 -> 60818 >> seq. 0xcbbfa586 ack 0x36bb48c6 >> flags 0x12 SYN ACK, tcp header: 40 bytes >> window 28960, checksum 0xab2b >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#11999): https://lists.fd.io/g/vpp-dev/message/11999 >> Mute This Topic: https://lists.fd.io/mt/29534956/675642 >> Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480514 >> Group Owner: vpp-dev+ow...@lists.fd.io >> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dmar...@me.com] >> -=-=-=-=-=-=-=-=-=-=-=- > > -- > Damjan >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12012): https://lists.fd.io/g/vpp-dev/message/12012 Mute This Topic: https://lists.fd.io/mt/29534956/21656 Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
