Hi, Sorry, I don’t work on OVS, so I never tested pdump with OVS. I remembered, Ciara testing earlier versions of pdump with OVS in the past.
Thanks, Reshma > -----Original Message----- > From: David Marchand <[email protected]> > Sent: Thursday, November 14, 2019 4:41 PM > To: Pattan, Reshma <[email protected]> > Cc: [email protected]; Kevin Traynor <[email protected]>; > Aaron Conole <[email protected]>; Loftus, Ciara > <[email protected]>; Flavio Leitner <[email protected]>; Ilya Maximets > <[email protected]>; Stokes, Ian <[email protected]> > Subject: Re: [PATCH] dpdk: Deprecate pdump support. > > Hello Reshma, > > Has pdump been tested (recently) with OVS? > > > On Mon, Nov 11, 2019 at 7:53 PM Ilya Maximets <[email protected]> > wrote: > > > > The conventional way for packet dumping in OVS is to use ovs-tcpdump > > that works via traffic mirroring. DPDK pdump could probably be used > > for some lower level debugging, but it is not commonly used for > > various reasons. > > > > There are lots of limitations for using this functionality in practice. > > Most of them connected with running secondary pdump process and > memory > > layout issues like requirement to disable ASLR in kernel. > > More details are available in DPDK guide: > > https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html#multi- > p > > rocess-limitations > > > > Beside the functional limitations it's also hard to use this > > functionality correctly. User must be sure that OVS and pdump utility > > are running on different CPU cores, which is hard because non-PMD > > threads could float over available CPU cores. This or any other > > misconfiguration will likely lead to crash of the pdump utility or/and > > OVS. > > > > Another problem is that the user must actually have this special pdump > > utility in a system and it might be not available in distributions. > > > > This change disables pdump support by default introducing special > > configuration option '--enable-dpdk-pdump'. Deprecation warnings will > > be shown to users on configuration and in runtime. > > > > Claiming to completely remove this functionality from OVS in one of > > the next releases. > > > > Signed-off-by: Ilya Maximets <[email protected]> > > Acked-by: Aaron Conole <[email protected]> > > - Recompiled from scratch, on OVS master (before this patch) with dpdk > 18.11.2. > > other_config : {dpdk-init="true", pmd-cpu-mask="0x00008002"} > > 2 physical ports, 2 vhost ports. > > 2019-11-14T14:13:09.596Z|00018|dpdk|INFO|Using DPDK 18.11.2 2019-11- > 14T14:13:09.596Z|00019|dpdk|INFO|DPDK Enabled - initializing... > 2019-11-14T14:13:09.596Z|00020|dpdk|INFO|No vhost-sock-dir provided - > defaulting to //var/run/openvswitch 2019-11- > 14T14:13:09.596Z|00021|dpdk|INFO|IOMMU support for vhost-user-client > disabled. > 2019-11-14T14:13:09.596Z|00022|dpdk|INFO|POSTCOPY support for vhost- > user-client disabled. > 2019-11-14T14:13:09.596Z|00023|dpdk|INFO|Per port memory for DPDK > devices disabled. > 2019-11-14T14:13:09.596Z|00024|dpdk|INFO|EAL ARGS: ovs-vswitchd -- > socket-mem 1024 --socket-limit 1024 -l 0. > 2019-11-14T14:13:09.600Z|00025|dpdk|INFO|EAL: Detected 28 lcore(s) > 2019-11-14T14:13:09.600Z|00026|dpdk|INFO|EAL: Detected 1 NUMA nodes > 2019-11-14T14:13:09.602Z|00027|dpdk|INFO|EAL: Multi-process socket > /var/run/openvswitch/dpdk/rte/mp_socket > 2019-11-14T14:13:09.618Z|00028|dpdk|INFO|EAL: Probing VFIO support... > 2019-11-14T14:13:09.618Z|00029|dpdk|INFO|EAL: VFIO support initialized > 2019-11-14T14:13:14.612Z|00030|dpdk|INFO|EAL: PCI device 0000:01:00.0 on > NUMA socket 0 > 2019-11-14T14:13:14.612Z|00031|dpdk|INFO|EAL: probe driver: > 8086:10fb net_ixgbe > 2019-11-14T14:13:14.613Z|00032|dpdk|INFO|EAL: using IOMMU type 1 > (Type 1) > 2019-11-14T14:13:14.744Z|00033|dpdk|INFO|EAL: Ignore mapping IO port > bar(2) > 2019-11-14T14:13:15.090Z|00034|dpdk|INFO|EAL: PCI device 0000:01:00.1 on > NUMA socket 0 > 2019-11-14T14:13:15.090Z|00035|dpdk|INFO|EAL: probe driver: > 8086:10fb net_ixgbe > 2019-11-14T14:13:15.199Z|00036|dpdk|INFO|EAL: Ignore mapping IO port > bar(2) > 2019-11-14T14:13:15.530Z|00037|dpdk|INFO|EAL: PCI device 0000:07:00.0 on > NUMA socket 0 > 2019-11-14T14:13:15.530Z|00038|dpdk|INFO|EAL: probe driver: > 8086:1521 net_e1000_igb > 2019-11-14T14:13:15.530Z|00039|dpdk|INFO|EAL: PCI device 0000:07:00.1 on > NUMA socket 0 > 2019-11-14T14:13:15.530Z|00040|dpdk|INFO|EAL: probe driver: > 8086:1521 net_e1000_igb > ... > 2019-11-14T14:13:15.802Z|00042|dpdk|INFO|DPDK pdump packet capture > enabled 2019-11-14T14:13:15.803Z|00043|dpdk|INFO|DPDK Enabled - > initialized > > - Attached a gdb to ovs-vswitchd. > > - Started pdump: > # sudo -u openvswitch XDG_RUNTIME_DIR=/var/run/openvswitch > ./v18.11.2/app/dpdk-pdump -- --pdump > 'port=0,queue=*,rx-dev=/tmp/pkts.pcap' > EAL: Detected 28 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Multi-process socket > /var/run/openvswitch/dpdk/rte/mp_socket_83791_549cdfd05e328e > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:10fb net_ixgbe > EAL: using IOMMU type 1 (Type 1) > EAL: PCI device 0000:01:00.1 on NUMA socket 0 > EAL: probe driver: 8086:10fb net_ixgbe > EAL: PCI device 0000:07:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > EAL: PCI device 0000:07:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > Port 3 MAC: 02 70 63 61 70 00 > > - Sent one packet to the first physical port from my tgen > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7f4840659700 (LWP 84336)] > bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe5af50, > bd=0x14fdad880) at > /root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:190 > 190 objptr = bucket_stack_pop(bd->buckets[rte_lcore_id()]); > (gdb) bt > #0 bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe5af50, > bd=0x14fdad880) at > /root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:190 > #1 bucket_dequeue (mp=<optimized out>, obj_table=0x14fe5af50, > n=<optimized out>) at > /root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:288 > #2 0x00000000004eeeef in rte_mempool_ops_dequeue_bulk (n=251, > obj_table=0x14fe5af50, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:657 > #3 __mempool_generic_get (cache=0x14fe5af40, n=1, > obj_table=0x7f4840656fe0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:1363 > #4 rte_mempool_generic_get (cache=0x14fe5af40, n=1, > obj_table=0x7f4840656fe0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:1426 > #5 rte_mempool_get_bulk (n=1, obj_table=0x7f4840656fe0, > mp=0x14fe2dac0) at /root/dpdk/v18.11.2/include/rte_mempool.h:1459 > #6 rte_mempool_get (obj_p=0x7f4840656fe0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:1485 > #7 rte_mbuf_raw_alloc (mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mbuf.h:1078 > #8 rte_pktmbuf_alloc (mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mbuf.h:1331 > #9 pdump_pktmbuf_copy (mp=0x14fe2dac0, m=0x1509ea100) at > /root/dpdk/lib/librte_pdump/rte_pdump.c:99 > #10 pdump_copy (pkts=<optimized out>, nb_pkts=<optimized out>, > user_params=<optimized out>) at > /root/dpdk/lib/librte_pdump/rte_pdump.c:151 > #11 0x00000000004eff31 in pdump_rx (port=<optimized out>, > qidx=<optimized out>, pkts=<optimized out>, nb_pkts=<optimized out>, > max_pkts=<optimized out>, user_params=<optimized out>) > at /root/dpdk/lib/librte_pdump/rte_pdump.c:172 > #12 0x00000000009f25fa in rte_eth_rx_burst (nb_pkts=32, > rx_pkts=0x7f4840657110, queue_id=0, port_id=0) at > /usr/local/include/dpdk/rte_ethdev.h:3888 > #13 netdev_dpdk_rxq_recv (rxq=0x1501bd940, batch=0x7f4840657100, > qfill=0x0) at ../lib/netdev-dpdk.c:2287 > #14 0x000000000093dab1 in netdev_rxq_recv (rx=<optimized out>, > batch=batch@entry=0x7f4840657100, qfill=<optimized out>) at > ../lib/netdev.c:724 > #15 0x0000000000911694 in dp_netdev_process_rxq_port > (pmd=pmd@entry=0x7f484065a010, rxq=0x2711ae0, port_no=3) at > ../lib/dpif-netdev.c:4268 > #16 0x0000000000911af9 in pmd_thread_main (f_=<optimized out>) at > ../lib/dpif-netdev.c:5526 > #17 0x000000000099355d in ovsthread_wrapper (aux_=<optimized out>) at > ../lib/ovs-thread.c:383 > #18 0x00007f4875402dd5 in start_thread (arg=0x7f4840659700) at > pthread_create.c:307 > #19 0x00007f4874920ead in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > (gdb) p bd->buckets[rte_lcore_id()] > $2 = (struct bucket_stack *) 0x0 > > > We can see a "bucket" mempool. > ovs is using "ring_mp_mc" default mempool. > So maybe something unaligned here. > > > On OVS side, we can see the pdump code expects that this mempool has > some resource initialised for a lcore 15. > Not sure who is responsible for doing this part, primary ? secondary ? > > The pdump application in 18.11.2 has a hardwired core mask as 0x1. > If I shoot this (backporting a commit that did this) and start pdump on lcores > 0, 1 and 15 (to mimic OVS running on master core 0 + lcore 1 and 15), I get a > segfault a little bit later, but again with an uninitialised resource on OVS > side. > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7ff795324700 (LWP 176548)] > bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe30bd0, > bd=0x14fdad880) at > /root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:194 > 194 rc = rte_ring_dequeue(bd->shared_bucket_ring, > (gdb) bt > #0 bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe30bd0, > bd=0x14fdad880) at > /root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:194 > #1 bucket_dequeue (mp=<optimized out>, obj_table=0x14fe30bd0, > n=<optimized out>) at > /root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:288 > #2 0x00000000004eeeef in rte_mempool_ops_dequeue_bulk (n=251, > obj_table=0x14fe30bd0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:657 > #3 __mempool_generic_get (cache=0x14fe30bc0, n=1, > obj_table=0x7ff795321fe0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:1363 > #4 rte_mempool_generic_get (cache=0x14fe30bc0, n=1, > obj_table=0x7ff795321fe0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:1426 > #5 rte_mempool_get_bulk (n=1, obj_table=0x7ff795321fe0, > mp=0x14fe2dac0) at /root/dpdk/v18.11.2/include/rte_mempool.h:1459 > #6 rte_mempool_get (obj_p=0x7ff795321fe0, mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mempool.h:1485 > #7 rte_mbuf_raw_alloc (mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mbuf.h:1078 > #8 rte_pktmbuf_alloc (mp=0x14fe2dac0) at > /root/dpdk/v18.11.2/include/rte_mbuf.h:1331 > #9 pdump_pktmbuf_copy (mp=0x14fe2dac0, m=0x1509ea100) at > /root/dpdk/lib/librte_pdump/rte_pdump.c:99 > #10 pdump_copy (pkts=<optimized out>, nb_pkts=<optimized out>, > user_params=<optimized out>) at > /root/dpdk/lib/librte_pdump/rte_pdump.c:151 > #11 0x00000000004eff31 in pdump_rx (port=<optimized out>, > qidx=<optimized out>, pkts=<optimized out>, nb_pkts=<optimized out>, > max_pkts=<optimized out>, user_params=<optimized out>) > at /root/dpdk/lib/librte_pdump/rte_pdump.c:172 > #12 0x00000000009f25fa in rte_eth_rx_burst (nb_pkts=32, > rx_pkts=0x7ff795322110, queue_id=0, port_id=0) at > /usr/local/include/dpdk/rte_ethdev.h:3888 > #13 netdev_dpdk_rxq_recv (rxq=0x1501bd940, batch=0x7ff795322100, > qfill=0x0) at ../lib/netdev-dpdk.c:2287 > #14 0x000000000093dab1 in netdev_rxq_recv (rx=<optimized out>, > batch=batch@entry=0x7ff795322100, qfill=<optimized out>) at > ../lib/netdev.c:724 > #15 0x0000000000911694 in dp_netdev_process_rxq_port > (pmd=pmd@entry=0x7ff795325010, rxq=0x2376b30, port_no=3) at > ../lib/dpif-netdev.c:4268 > #16 0x0000000000911af9 in pmd_thread_main (f_=<optimized out>) at > ../lib/dpif-netdev.c:5526 > #17 0x000000000099355d in ovsthread_wrapper (aux_=<optimized out>) at > ../lib/ovs-thread.c:383 > #18 0x00007ff7cd81ddd5 in start_thread (arg=0x7ff795324700) at > pthread_create.c:307 > #19 0x00007ff7ccd3bead in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > (gdb) p bd->shared_bucket_ring > $1 = (struct rte_ring *) 0x0 > > > > -- > David Marchand _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
