Ravi,

I only have an SSE machine (Ivy Bridge) and DPDK is using ring mempool as far 
as I can tell from gdb. You are using AVX2 which I don't have one to try it to 
see whether Octeontx mempool is the default mempool for AVX2. What do you put 
in dpdk in the host startup.conf? What is the output for show dpdk version?

Steven

On 6/5/18, 1:40 PM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur" 
<vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:

    Hi Damjan,
    
    I am not intentional using it. I am running VPP on a x86 Ubuntu server.
    
    uname -a
    4.9.77.2-rt61 #1 SMP PREEMPT RT Tue May 15 20:36:51 UTC 2018 x86_64
    x86_64 x86_64 GNU/Linux
    
    Thanks.
    
    On Tue, Jun 5, 2018 at 1:10 PM, Damjan Marion <dmar...@me.com> wrote:
    > Dear Ravi,
    >
    > Currently we don't support Octeon TX mempool. Are you intentionally using
    > it?
    >
    > Regards,
    >
    > Damjan
    >
    > On 5 Jun 2018, at 21:46, Ravi Kerur <rke...@gmail.com> wrote:
    >
    > Steven,
    >
    > I managed to get Tx/Rx rings setup with 1GB hugepages. However, when I
    > assign an IP address to both vhost-user/virtio interfaces and initiate
    > a ping VPP crashes.
    >
    > Any other mechanism available to test Tx/Rx path between Vhost and
    > Virtio? Details below.
    >
    >
    > *******On host*******
    > vpp#show vhost-user VirtualEthernet0/0/0
    > Virtio vhost-user interfaces
    > Global:
    >  coalesce frames 32 time 1e-3
    >  number of rx virtqueues in interrupt mode: 0
    > Interface: VirtualEthernet0/0/0 (ifindex 3)
    > virtio_net_hdr_sz 12
    > features mask (0xffffffffffffffff):
    > features (0x110008000):
    >   VIRTIO_NET_F_MRG_RXBUF (15)
    >   VIRTIO_F_INDIRECT_DESC (28)
    >   VIRTIO_F_VERSION_1 (32)
    >  protocol features (0x0)
    >
    > socket filename /var/run/vpp/sock3.sock type server errno "Success"
    >
    > rx placement:
    >   thread 0 on vring 1, polling
    > tx placement: lock-free
    >   thread 0 on vring 0
    >
    > Memory regions (total 1)
    > region fd    guest_phys_addr    memory_size        userspace_addr
    > mmap_offset        mmap_addr
    > ====== ===== ================== ================== ==================
    > ================== ==================
    >  0     26    0x00007f54c0000000 0x0000000040000000 0x00007f54c0000000
    > 0x0000000000000000 0x00007faf00000000
    >
    > Virtqueue 0 (TX)
    >  qsz 256 last_avail_idx 0 last_used_idx 0
    >  avail.flags 1 avail.idx 256 used.flags 1 used.idx 0
    >  kickfd 27 callfd 24 errfd -1
    >
    > Virtqueue 1 (RX)
    >  qsz 256 last_avail_idx 0 last_used_idx 0
    >  avail.flags 1 avail.idx 0 used.flags 1 used.idx 0
    >  kickfd 28 callfd 25 errfd -1
    >
    >
    > vpp#set interface ip address VirtualEthernet0/0/0 10.1.1.1/24
    >
    > ************On container**********
    > vpp# show interface VirtioUser0/0/0
    >              Name               Idx       State          Counter
    >    Count
    > VirtioUser0/0/0                   1         up
    > vpp#
    > vpp# set interface ip address VirtioUser0/0/0 10.1.1.2/24
    > vpp#
    > vpp# ping 10.1.1.1
    >
    > Statistics: 5 sent, 0 received, 100% packet loss
    > vpp#
    >
    >
    > ************Host vpp crash with following backtrace******************
    > Continuing.
    >
    > Program received signal SIGSEGV, Segmentation fault.
    > octeontx_fpa_bufpool_alloc (handle=0)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:57
    > 57        return (void *)(uintptr_t)fpavf_read64((void *)(handle +
    > (gdb) bt
    > #0  octeontx_fpa_bufpool_alloc (handle=0)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:57
    > #1  octeontx_fpavf_dequeue (mp=0x7fae7fc9ab40, obj_table=0x7fb04d868880,
    > n=528)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:98
    > #2  0x00007fb04b73bdef in rte_mempool_ops_dequeue_bulk (n=528,
    > obj_table=<optimized out>,
    >    mp=0x7fae7fc9ab40)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:492
    > #3  __mempool_generic_get (cache=<optimized out>, n=<optimized out>,
    > obj_table=<optimized out>,
    >    mp=<optimized out>)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1271
    > #4  rte_mempool_generic_get (cache=<optimized out>, n=<optimized out>,
    >    obj_table=<optimized out>, mp=<optimized out>)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1306
    > #5  rte_mempool_get_bulk (n=528, obj_table=<optimized out>,
    > mp=0x7fae7fc9ab40)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1339
    > #6  dpdk_buffer_fill_free_list_avx2 (vm=0x7fb08ec69480
    > <vlib_global_main>, fl=0x7fb04cb2b100,
    >    min_free_buffers=<optimized out>)
    >    at 
/var/venom/rk-vpp-1804/vpp/build-data/../src/plugins/dpdk/buffer.c:228
    > #7  0x00007fb08e5046ea in vlib_buffer_alloc_from_free_list (index=0
    > '\000', n_buffers=514,
    >    buffers=0x7fb04cb8ec58, vm=0x7fb08ec69480 <vlib_global_main>)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/buffer_funcs.h:306
    > #8  vhost_user_if_input (mode=<optimized out>, node=0x7fb04d0f5b80,
    > qid=<optimized out>,
    >    vui=0x7fb04d87523c, vum=0x7fb08e9b9560 <vhost_user_main>,
    >    vm=0x7fb08ec69480 <vlib_global_main>)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1644
    > #9  vhost_user_input (f=<optimized out>, node=<optimized out>,
    > vm=<optimized out>)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
    > #10 vhost_user_input_avx2 (vm=<optimized out>, node=<optimized out>,
    > frame=<optimized out>)
    >    at
    > 
/var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
    > #11 0x00007fb08ea166b3 in dispatch_node (last_time_stamp=<optimized
    > out>, frame=0x0,
    >    dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
    > node=0x7fb04d0f5b80,
    >    vm=0x7fb08ec69480 <vlib_global_main>)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:988
    > #12 vlib_main_or_worker_loop (is_main=1, vm=0x7fb08ec69480
    > <vlib_global_main>)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1505
    > #13 vlib_main_loop (vm=0x7fb08ec69480 <vlib_global_main>)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1633
    > #14 vlib_main (vm=vm@entry=0x7fb08ec69480 <vlib_global_main>,
    > input=input@entry=0x7fb04d077fa0)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1787
    > #15 0x00007fb08ea4d683 in thread0 (arg=140396286350464)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/unix/main.c:568
    > #16 0x00007fb08dbe15d8 in clib_calljmp ()
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vppinfra/longjmp.S:110
    > #17 0x00007fff0726d370 in ?? ()
    > ---Type <return> to continue, or q <return> to quit---
    > #18 0x00007fb08ea4e3da in vlib_unix_main (argc=<optimized out>,
    > argv=<optimized out>)
    >    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/unix/main.c:632
    > #19 0x0000001900000000 in ?? ()
    > #20 0x000000e700000000 in ?? ()
    > #21 0x0000000000000831 in ?? ()
    > #22 0x00007fb08e9aac00 in ?? () from 
/usr/lib/x86_64-linux-gnu/libvnet.so.0
    >
    > **************Vhost-user debugs on host**********
    > Jun  5 19:23:35 [18916]: vhost_user_socksvr_accept_ready:1294: New
    > client socket for vhost interface 3, fd 23
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:995: if 3 msg
    > VHOST_USER_SET_OWNER
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:847: if 3 msg
    > VHOST_USER_GET_FEATURES - reply 0x000000015c628000
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:995: if 3 msg
    > VHOST_USER_SET_OWNER
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1004: if 3 msg
    > VHOST_USER_SET_VRING_CALL 0
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1004: if 3 msg
    > VHOST_USER_SET_VRING_CALL 1
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:852: if 3 msg
    > VHOST_USER_SET_FEATURES features 0x0000000110008000
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:877: if 3 msg
    > VHOST_USER_SET_MEM_TABLE nregions 1
    > Jun  5 19:23:35[18916]: vhost_user_socket_read:916: map memory region
    > 0 addr 0 len 0x40000000 fd 26 mapped 0x7faf00000000 page_sz 0x40000000
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:932: if 3 msg
    > VHOST_USER_SET_VRING_NUM idx 0 num 256
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1096: if 3 msg
    > VHOST_USER_SET_VRING_BASE idx 0 num 0
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:943: if 3 msg
    > VHOST_USER_SET_VRING_ADDR idx 0
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1037: if 3 msg
    > VHOST_USER_SET_VRING_KICK 0
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:932: if 3 msg
    > VHOST_USER_SET_VRING_NUM idx 1 num 256
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1096: if 3 msg
    > VHOST_USER_SET_VRING_BASE idx 1 num 0
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:943: if 3 msg
    > VHOST_USER_SET_VRING_ADDR idx 1
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1037: if 3 msg
    > VHOST_USER_SET_VRING_KICK 1
    > Jun  5 19:23:35 [18916]: vhost_user_socket_read:1211: if 3
    > VHOST_USER_SET_VRING_ENABLE: enable queue 0
    > Jun  5 19:23:35[18916]: vhost_user_socket_read:1211: if 3
    > VHOST_USER_SET_VRING_ENABLE: enable queue 1
    >
    > Thanks.
    >
    > On Tue, Jun 5, 2018 at 11:31 AM, Steven Luong (sluong) <slu...@cisco.com>
    > wrote:
    >
    > Ravi,
    >
    > In order to use dpdk virtio_user, you need 1GB huge page.
    >
    > Steven
    >
    > On 6/5/18, 11:17 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >    Hi Steven,
    >
    >    Connection is the problem. I don't see memory regions setup correctly.
    >    Below are some details. Currently I am using 2MB hugepages.
    >
    >    (1) Create vhost-user server
    >    debug vhost-user on
    >    vpp# create vhost socket /var/run/vpp/sock3.sock server
    >    VirtualEthernet0/0/0
    >    vpp# set interface state VirtualEthernet0/0/0 up
    >    vpp#
    >    vpp#
    >
    >    (2) Instantiate a container
    >    docker run -it --privileged -v
    >    /var/run/vpp/sock3.sock:/var/run/usvhost1 -v
    >    /dev/hugepages:/dev/hugepages dpdk-app-vpp:latest
    >
    >    (3) Inside the container run EAL/DPDK virtio with following startup 
conf.
    >    unix {
    >      nodaemon
    >      log /var/log/vpp/vpp.log
    >      full-coredump
    >      cli-listen /run/vpp/cli.sock
    >      gid vpp
    >    }
    >
    >    api-trace {
    >      on
    >    }
    >
    >    api-segment {
    >      gid vpp
    >    }
    >
    >    dpdk {
    >            no-pci
    >            vdev virtio_user0,path=/var/run/usvhost1
    >    }
    >
    >    Following errors are seen due to 2MB hugepages and I think DPDK
    >    requires "--single-file-segments" option.
    >
    >    /usr/bin/vpp[19]: dpdk_config:1275: EAL init args: -c 1 -n 4 --no-pci
    >    --vdev virtio_user0,path=/var/run/usvhost1 --huge-dir
    >    /run/vpp/hugepages --file-prefix vpp --master-lcore 0 --socket-mem
    >    64,64
    >    /usr/bin/vpp[19]: dpdk_config:1275: EAL init args: -c 1 -n 4 --no-pci
    >    --vdev virtio_user0,path=/var/run/usvhost1 --huge-dir
    >    /run/vpp/hugepages --file-prefix vpp --master-lcore 0 --socket-mem
    >    64,64
    >    EAL: 4 hugepages of size 1073741824 reserved, but no mounted hugetlbfs
    >    found for that size
    >    EAL: VFIO support initialized
    >    get_hugepage_file_info(): Exceed maximum of 8
    >    prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
    >    DPDK physical memory layout:
    >
    >
    >    Second test case>
    >    (1) and (2) are same as above. I run VPP inside a container with
    >    following startup config
    >
    >    unix {
    >      nodaemon
    >      log /var/log/vpp/vpp.log
    >      full-coredump
    >      cli-listen /run/vpp/cli.sock
    >      gid vpp
    >    }
    >
    >    api-trace {
    >      on
    >    }
    >
    >    api-segment {
    >      gid vpp
    >    }
    >
    >    dpdk {
    >            no-pci
    >            single-file-segments
    >            vdev virtio_user0,path=/var/run/usvhost1
    >    }
    >
    >
    >    VPP fails to start with
    >    plugin.so
    >    vpp[19]: dpdk_config: unknown input `single-file-segments no-pci vd...'
    >    vpp[19]: dpdk_config: unknown input `single-file-segments no-pci vd...'
    >
    >    [1]+  Done                    /usr/bin/vpp -c /etc/vpp/startup.conf
    >    root@867dc128b544:~/dpdk#
    >
    >
    >    show version (on both host and container).
    >    vpp v18.04-rc2~26-gac2b736~b45 built by root on 34a554d1c194 at Wed
    >    Apr 25 14:53:07 UTC 2018
    >    vpp#
    >
    >    Thanks.
    >
    >    On Tue, Jun 5, 2018 at 9:23 AM, Steven Luong (sluong) 
<slu...@cisco.com>
    > wrote:
    >
    > Ravi,
    >
    > Do this
    >
    > 1. Run VPP native vhost-user in the host. Turn on debug "debug vhost-user
    > on".
    > 2. Bring up the container with the vdev virtio_user commands that you have
    > as before
    > 3. show vhost-user in the host and verify that it has a shared memory
    > region. If not, the connection has a problem. Collect the show vhost-user
    > and debug vhost-user and send them to me and stop. If yes, proceed with 
step
    > 4.
    > 4. type "trace vhost-user-input 100" in the host
    > 5. clear error, and clear interfaces in the host and the container.
    > 6. do the ping from the container.
    > 7. Collect show error, show trace, show interface, and show vhost-user in
    > the host. Collect show error and show interface in the container. Put 
output
    > in github and provide a link to view. There is no need to send a large 
file.
    >
    > Steven
    >
    > On 6/4/18, 5:50 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >    Hi Steven,
    >
    >    Thanks for your help. I am using vhost-user client (VPP in container)
    >    and vhost-user server (VPP in host). I thought it should work.
    >
    >    create vhost socket /var/run/vpp/sock3.sock server (On host)
    >
    >    create vhost socket /var/run/usvhost1 (On container)
    >
    >    Can you please point me to a document which shows how to create VPP
    >    virtio_user interfaces or static configuration in
    >    /etc/vpp/startup.conf?
    >
    >    I have used following declarations in /etc/vpp/startup.conf
    >
    >    # vdev virtio_user0,path=/var/run/vpp/sock3.sock,mac=52:54:00:00:04:01
    >    # vdev virtio_user1,path=/var/run/vpp/sock4.sock,mac=52:54:00:00:04:02
    >
    >    but it doesn't work.
    >
    >    Thanks.
    >
    >    On Mon, Jun 4, 2018 at 3:57 PM, Steven Luong (sluong) 
<slu...@cisco.com>
    > wrote:
    >
    > Ravi,
    >
    > VPP only supports vhost-user in the device mode. In your example, the 
host,
    > in device mode, and the container also in device mode do not make a happy
    > couple. You need one of them, either the host or container, running in
    > driver mode using the dpdk vdev virtio_user command in startup.conf. So 
you
    > need something like this
    >
    > (host) VPP native vhost-user ----- (container) VPP DPDK vdev virtio_user
    >                          -- or --
    > (host) VPP DPDK vdev virtio_user ---- (container) VPP native vhost-user
    >
    > Steven
    >
    > On 6/4/18, 3:27 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >    Hi Steven
    >
    >    Though crash is not happening anymore, there is still an issue with Rx
    >    and Tx. To eliminate whether it is testpmd or vpp, I decided to run
    >
    >    (1) VPP vhost-user server on host-x
    >    (2) Run VPP in a container on host-x and vhost-user client port
    >    connecting to vhost-user server.
    >
    >    Still doesn't work. Details below. Please let me know if something is
    >    wrong in what I am doing.
    >
    >
    >    (1) VPP vhost-user as a server
    >    (2) VPP in a container virtio-user or vhost-user client
    >
    >    (1) Create vhost-user server socket on VPP running on host.
    >
    >    vpp#create vhost socket /var/run/vpp/sock3.sock server
    >    vpp#set interface state VirtualEthernet0/0/0 up
    >    show vhost-user VirtualEthernet0/0/0 descriptors
    >    Virtio vhost-user interfaces
    >    Global:
    >    coalesce frames 32 time 1e-3
    >    number of rx virtqueues in interrupt mode: 0
    >    Interface: VirtualEthernet0/0/0 (ifindex 3)
    >    virtio_net_hdr_sz 0
    >    features mask (0xffffffffffffffff):
    >    features (0x0):
    >    protocol features (0x0)
    >
    >    socket filename /var/run/vpp/sock3.sock type server errno "Success"
    >
    >    rx placement:
    >    tx placement: spin-lock
    >    thread 0 on vring 0
    >
    >    Memory regions (total 0)
    >
    >    vpp# set interface ip address VirtualEthernet0/0/0 192.168.1.1/24
    >    vpp#
    >
    >    (2) Instantiate a docker container to run VPP connecting to 
sock3.server
    > socket.
    >
    >    docker run -it --privileged -v
    >    /var/run/vpp/sock3.sock:/var/run/usvhost1 -v
    >    /dev/hugepages:/dev/hugepages dpdk-app-vpp:latest
    >    root@4b1bd06a3225:~/dpdk#
    >    root@4b1bd06a3225:~/dpdk# ps -ef
    >    UID PID PPID C STIME TTY TIME CMD
    >    root 1 0 0 21:39 ? 00:00:00 /bin/bash
    >    root 17 1 0 21:39 ? 00:00:00 ps -ef
    >    root@4b1bd06a3225:~/dpdk#
    >
    >    root@8efda6701ace:~/dpdk# ps -ef | grep vpp
    >    root 19 1 39 21:41 ? 00:00:03 /usr/bin/vpp -c /etc/vpp/startup.conf
    >    root 25 1 0 21:41 ? 00:00:00 grep --color=auto vpp
    >    root@8efda6701ace:~/dpdk#
    >
    >    vpp#create vhost socket /var/run/usvhost1
    >    vpp#set interface state VirtualEthernet0/0/0 up
    >    vpp#show vhost-user VirtualEthernet0/0/0 descriptors
    >    Virtio vhost-user interfaces
    >    Global:
    >    coalesce frames 32 time 1e-3
    >    number of rx virtqueues in interrupt mode: 0
    >    Interface: VirtualEthernet0/0/0 (ifindex 1)
    >    virtio_net_hdr_sz 0
    >    features mask (0xffffffffffffffff):
    >    features (0x0):
    >    protocol features (0x0)
    >
    >    socket filename /var/run/usvhost1 type client errno "Success"
    >
    >    rx placement:
    >    tx placement: spin-lock
    >    thread 0 on vring 0
    >
    >    Memory regions (total 0)
    >
    >    vpp#
    >
    >    vpp# set interface ip address VirtualEthernet0/0/0 192.168.1.2/24
    >    vpp#
    >
    >    vpp# ping 192.168.1.1
    >
    >    Statistics: 5 sent, 0 received, 100% packet loss
    >    vpp#
    >
    >    On Thu, May 31, 2018 at 2:30 PM, Steven Luong (sluong) 
<slu...@cisco.com>
    > wrote:
    >
    > show interface and look for the counter and count columns for the
    > corresponding interface.
    >
    > Steven
    >
    > On 5/31/18, 1:28 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >    Hi Steven,
    >
    >    You made my day, thank you. I didn't realize different dpdk versions
    >    (vpp -- 18.02.1 and testpmd -- from latest git repo (probably 18.05)
    >    could be the cause of the problem, I still dont understand why it
    >    should as virtio/vhost messages are meant to setup tx/rx rings
    >    correctly?
    >
    >    I downloaded dpdk 18.02.1 stable release and at least vpp doesn't
    >    crash now (for both vpp-native and dpdk vhost interfaces). I have one
    >    question is there a way to read vhost-user statistics counter (Rx/Tx)
    >    on vpp? I only know
    >
    >    'show vhost-user <intf>' and 'show vhost-user <intf> descriptors'
    >    which doesn't show any counters.
    >
    >    Thanks.
    >
    >    On Thu, May 31, 2018 at 11:51 AM, Steven Luong (sluong)
    >    <slu...@cisco.com> wrote:
    >
    > Ravi,
    >
    > For (1) which works, what dpdk version are you using in the host? Are you
    > using the same dpdk version as VPP is using? Since you are using VPP 
latest,
    > I think it is 18.02. Type "show dpdk version" at the VPP prompt to find 
out
    > for sure.
    >
    > Steven
    >
    > On 5/31/18, 11:44 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >    Hi Steven,
    >
    >    i have tested following scenarios and it basically is not clear why
    >    you think DPDK is the problem? Is it possible VPP and DPDK use
    >    different virtio versions?
    >
    >    Following are the scenarios I have tested
    >
    >    (1) testpmd/DPDK vhost-user (running on host) and testpmd/DPDK
    >    virito-user (in a container) -- can send and receive packets
    >    (2) VPP-native vhost-user (running on host) and testpmd/DPDK
    >    virtio-user (in a container) -- VPP crashes and it is in VPP code
    >    (3) VPP-DPDK vhost user (running on host) and testpmd/DPDK virtio-user
    >    (in a container) -- VPP crashes and in DPDK
    >
    >    Thanks.
    >
    >    On Thu, May 31, 2018 at 10:12 AM, Steven Luong (sluong)
    >    <slu...@cisco.com> wrote:
    >
    > Ravi,
    >
    > I've proved my point -- there is a problem in the way that you invoke
    > testpmd. The shared memory region that it passes to the device is not
    > accessible from the device. I don't know what the correct options are that
    > you need to use. This is really a question for dpdk.
    >
    > As a further exercise, you could remove VPP in the host and instead run
    > testpmd in device mode using "--vdev
    > net_vhost0,iface=/var/run/vpp/sock1.sock" option. I bet you testpmd in the
    > host will crash in the same place. I hope you can find out the answer from
    > dpdk and tell us about it.
    >
    > Steven
    >
    > On 5/31/18, 9:31 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
    > <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
    >
    >    Hi Steven,
    >
    >    Thank you for your help, I removed sock1.sock and sock2.sock,
    >    restarted vpp, atleast interfaces get created. However, when I start
    >    dpdk/testpmd inside the container it crashes as well. Below are some
    >    details. I am using vpp code from latest repo.
    >
    >    (1) On host
    >    show interface
    >                  Name               Idx       State          Counter
    >        Count
    >    VhostEthernet2                    3        down
    >    VhostEthernet3                    4        down
    >    VirtualFunctionEthernet4/10/4     1        down
    >    VirtualFunctionEthernet4/10/6     2        down
    >    local0                            0        down
    >    vpp#
    >    vpp# set interface state VhostEthernet2 up
    >    vpp# set interface state VhostEthernet3 up
    >    vpp#
    >    vpp# set interface l2 bridge VhostEthernet2 1
    >    vpp# set interface l2 bridge VhostEthernet3 1
    >    vpp#
    >
    >    (2) Run tespmd inside the container
    >    docker run -it --privileged -v
    >    /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
    >    /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
    >    /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -l 16-19
    >    -n 4 --log-level=8 -m 64 --no-pci
    >    --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
    >    --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
    >    -i
    >    EAL: Detected 28 lcore(s)
    >    EAL: Detected 2 NUMA nodes
    >    EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
    >    EAL: 8192 hugepages of size 2097152 reserved, but no mounted hugetlbfs
    >    found for that size
    >    EAL: Probing VFIO support...
    >    EAL: VFIO support initialized
    >    EAL: Setting up physically contiguous memory...
    >    EAL: locking hot plug lock memory...
    >    EAL: primary init32...
    >    Interactive-mode selected
    >    Warning: NUMA should be configured manually by using
    >    --port-numa-config and --ring-numa-config parameters along with
    >    --numa.
    >    testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
    >    size=2176, socket=0
    >    testpmd: preferred mempool ops selected: ring_mp_mc
    >    testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=171456,
    >    size=2176, socket=1
    >    testpmd: preferred mempool ops selected: ring_mp_mc
    >    Port 0 is now not stopped
    >    Port 1 is now not stopped
    >    Please stop the ports first
    >    Done
    >    testpmd>
    >
    >    (3) VPP crashes with the same issue but inside dpdk code
    >
    >    (gdb) cont
    >    Continuing.
    >
    >    Program received signal SIGSEGV, Segmentation fault.
    >    [Switching to Thread 0x7ffd0d08e700 (LWP 41257)]
    >    rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
    >    out>, mbuf_pool=0x7fe17fc883c0,
    >        pkts=pkts@entry=0x7fffb671ebc0, count=count@entry=32)
    >        at
    > 
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
    >    1504        free_entries = *((volatile uint16_t *)&vq->avail->idx) -
    >    (gdb) bt
    >    #0  rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
    > out>,
    >        mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
    >    count=count@entry=32)
    >        at
    > 
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
    >    #1  0x00007fffb4718e6f in eth_vhost_rx (q=0x7fe17fbbdd80,
    > bufs=0x7fffb671ebc0,
    >        nb_bufs=<optimized out>)
    >        at
    > 
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/net/vhost/rte_eth_vhost.c:410
    >    #2  0x00007fffb441cb7c in rte_eth_rx_burst (nb_pkts=256,
    >    rx_pkts=0x7fffb671ebc0, queue_id=0,
    >        port_id=3) at
    >
    > 
/var/venom/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_ethdev.h:3635
    >    #3  dpdk_device_input (queue_id=0, thread_index=<optimized out>,
    >    node=0x7fffb732c700,
    >        xd=0x7fffb7337240, dm=<optimized out>, vm=0x7fffb6703340)
    >        at /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:477
    >    #4  dpdk_input_node_fn_avx2 (vm=<optimized out>, node=<optimized out>,
    >    f=<optimized out>)
    >        at /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:658
    >    #5  0x00007ffff7954d35 in dispatch_node
    >    (last_time_stamp=12531752723928016, frame=0x0,
    >        dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
    >    node=0x7fffb732c700,
    >        vm=0x7fffb6703340) at
    > /var/venom/vpp/build-data/../src/vlib/main.c:988
    >    #6  vlib_main_or_worker_loop (is_main=0, vm=0x7fffb6703340)
    >        at /var/venom/vpp/build-data/../src/vlib/main.c:1507
    >    #7  vlib_worker_loop (vm=0x7fffb6703340) at
    >    /var/venom/vpp/build-data/../src/vlib/main.c:1641
    >    #8  0x00007ffff6ad25d8 in clib_calljmp ()
    >        at /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
    >    #9  0x00007ffd0d08ddb0 in ?? ()
    >    #10 0x00007fffb4436edd in eal_thread_loop (arg=<optimized out>)
    >        at
    > 
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
    >    #11 0x0000000000000000 in ?? ()
    >    (gdb) frame 0
    >    #0  rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
    > out>,
    >        mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
    >    count=count@entry=32)
    >        at
    > 
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
    >    1504        free_entries = *((volatile uint16_t *)&vq->avail->idx) -
    >    (gdb) p vq
    >    $1 = (struct vhost_virtqueue *) 0x7fc3ffc84b00
    >    (gdb) p vq->avail
    >    $2 = (struct vring_avail *) 0x7ffbfff98000
    >    (gdb) p *$2
    >    Cannot access memory at address 0x7ffbfff98000
    >    (gdb)
    >
    >
    >    Thanks.
    >
    >    On Thu, May 31, 2018 at 12:09 AM, Steven Luong (sluong)
    >    <slu...@cisco.com> wrote:
    >
    > Sorry, I was expecting to see two VhostEthernet interfaces like this. 
Those
    > VirtualFunctionEthernet are your physical interfaces.
    >
    > sh int
    >              Name               Idx       State          Counter
    > Count
    > VhostEthernet0                    1         up
    > VhostEthernet1                    2         up
    > local0                            0        down
    > DBGvpp#
    >
    > You have to first manually remove /var/run/vpp/sock1.sock and
    > /var/run/vpp/sock2.sock before you start vpp on the host. dpdk does not 
like
    > it if they already existed. If you successfully create VhostEthernet
    > interface, try to send some traffic through it to see if it crashes or 
not.
    >
    > Steven
    >
    > On 5/30/18, 9:17 PM, "vpp-dev@lists.fd.io on behalf of Steven Luong
    > (sluong)" <vpp-dev@lists.fd.io on behalf of slu...@cisco.com> wrote:
    >
    >    Ravi,
    >
    >    I don't think you can declare (2) works fine yet. Please bring up the
    > dpdk vhost-user interfaces and try to send some traffic between them to
    > exercise the shared memory region from dpdk virtio-user which may be
    > "questionable".
    >
    >        VirtualFunctionEthernet4/10/4     1        down
    >        VirtualFunctionEthernet4/10/6     2        down
    >
    >    Steven
    >
    >    On 5/30/18, 4:41 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >        Hi Steve,
    >
    >        Thank you for your inputs, I added feature-mask to see if it helps 
in
    >        setting up queues correctly, it didn't so I will remove it. I have
    >        tried following combination
    >
    >        (1) VPP->vhost-user (on host) and DPDK/testpmd->virtio-user (in a
    >        container)  -- VPP crashes
    >        (2) DPDK/testpmd->vhost-user (on host) and 
DPDK/testpmd->virtio-user
    >        (in a container) -- works fine
    >
    >        To use DPDK vhost-user inside VPP, I defined configuration in
    >        startup.conf as mentioned by you and it looks as follows
    >
    >        unix {
    >          nodaemon
    >          log /var/log/vpp/vpp.log
    >          full-coredump
    >          cli-listen /run/vpp/cli.sock
    >          gid vpp
    >        }
    >
    >        api-segment {
    >          gid vpp
    >        }
    >
    >        cpu {
    >                main-core 1
    >                corelist-workers 6-9
    >        }
    >
    >        dpdk {
    >                dev 0000:04:10.4
    >                dev 0000:04:10.6
    >                uio-driver vfio-pci
    >                vdev net_vhost0,iface=/var/run/vpp/sock1.sock
    >                vdev net_vhost1,iface=/var/run/vpp/sock2.sock
    >                huge-dir /dev/hugepages_1GB
    >                socket-mem 2048,2048
    >        }
    >
    >        From VPP logs
    >        dpdk: EAL init args: -c 3c2 -n 4 --vdev
    >        net_vhost0,iface=/var/run/vpp/sock1.sock --vdev
    >        net_vhost1,iface=/var/run/vpp/sock2.sock --huge-dir
    > /dev/hugepages_1GB
    >        -w 0000:04:10.4 -w 0000:04:10.6 --master-lcore 1 --socket-mem
    >        2048,2048
    >
    >        However, VPP doesn't create interface at all
    >
    >        vpp# show interface
    >                      Name               Idx       State          Counter
    >            Count
    >        VirtualFunctionEthernet4/10/4     1        down
    >        VirtualFunctionEthernet4/10/6     2        down
    >        local0                            0        down
    >
    >        since it is a static mapping I am assuming it should be created,
    > correct?
    >
    >        Thanks.
    >
    >        On Wed, May 30, 2018 at 3:43 PM, Steven Luong (sluong)
    > <slu...@cisco.com> wrote:
    >
    > Ravi,
    >
    > First and foremost, get rid of the feature-mask option. I don't know what
    > 0x40400000 does for you. If that does not help, try testing it with dpdk
    > based vhost-user instead of VPP native vhost-user to make sure that they 
can
    > work well with each other first. To use dpdk vhost-user, add a vdev 
command
    > in the startup.conf for each vhost-user device that you have.
    >
    > dpdk { vdev net_vhost0,iface=/var/run/vpp/sock1.sock }
    >
    > dpdk based vhost-user interface is named VhostEthernet0, VhostEthernet1,
    > etc. Make sure you use the right interface name to set the state to up.
    >
    > If dpdk based vhost-user does not work with testpmd either, it looks like
    > some problem with the way that you invoke testpmd.
    >
    > If dpdk based vhost-user works well with the same testpmd device driver 
and
    > not vpp native vhost-user, I can set up something similar to yours to look
    > into it.
    >
    > The device driver, testpmd, is supposed to pass the shared memory region 
to
    > VPP for TX/RX queues. It looks like VPP vhost-user might have run into a
    > bump there with using the shared memory (txvq->avail).
    >
    > Steven
    >
    > PS. vhost-user is not an optimum interface for containers. You may want to
    > look into using memif if you don't already know about it.
    >
    >
    > On 5/30/18, 2:06 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
    >
    >    I am not sure what is wrong with the setup or a bug in vpp, vpp
    >    crashes with vhost<-->virtio communication.
    >
    >    (1) Vhost-interfaces are created and attached to bridge-domain as 
follows
    >
    >    create vhost socket /var/run/vpp/sock1.sock server feature-mask
    > 0x40400000
    >    create vhost socket /var/run/vpp/sock2.sock server feature-mask
    > 0x40400000
    >    set interface state VirtualEthernet0/0/0 up
    >    set interface state VirtualEthernet0/0/1 up
    >
    >    set interface l2 bridge VirtualEthernet0/0/0 1
    >    set interface l2 bridge VirtualEthernet0/0/1 1
    >
    >
    >    (2) DPDK/testpmd is started in a container to talk to vpp/vhost-user
    >    interface as follows
    >
    >    docker run -it --privileged -v
    >    /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
    >    /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
    >    /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
    >    4 --log-level=9 -m 64 --no-pci --single-file-segments
    >    --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
    >    --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
    >    -i
    >
    >    (3) show vhost-user VirtualEthernet0/0/1
    >    Virtio vhost-user interfaces
    >    Global:
    >      coalesce frames 32 time 1e-3
    >      number of rx virtqueues in interrupt mode: 0
    >    Interface: VirtualEthernet0/0/1 (ifindex 4)
    >    virtio_net_hdr_sz 10
    >     features mask (0x40400000):
    >     features (0x0):
    >      protocol features (0x0)
    >
    >     socket filename /var/run/vpp/sock2.sock type server errno "Success"
    >
    >     rx placement:
    >     tx placement: spin-lock
    >       thread 0 on vring 0
    >       thread 1 on vring 0
    >       thread 2 on vring 0
    >       thread 3 on vring 0
    >       thread 4 on vring 0
    >
    >     Memory regions (total 1)
    >     region fd    guest_phys_addr    memory_size        userspace_addr
    >    mmap_offset        mmap_addr
    >     ====== ===== ================== ================== ==================
    >    ================== ==================
    >      0     55    0x00007ff7c0000000 0x0000000040000000 0x00007ff7c0000000
    >    0x0000000000000000 0x00007ffbc0000000
    >
    >    vpp# show vhost-user VirtualEthernet0/0/0
    >    Virtio vhost-user interfaces
    >    Global:
    >      coalesce frames 32 time 1e-3
    >      number of rx virtqueues in interrupt mode: 0
    >    Interface: VirtualEthernet0/0/0 (ifindex 3)
    >    virtio_net_hdr_sz 10
    >     features mask (0x40400000):
    >     features (0x0):
    >      protocol features (0x0)
    >
    >     socket filename /var/run/vpp/sock1.sock type server errno "Success"
    >
    >     rx placement:
    >     tx placement: spin-lock
    >       thread 0 on vring 0
    >       thread 1 on vring 0
    >       thread 2 on vring 0
    >       thread 3 on vring 0
    >       thread 4 on vring 0
    >
    >     Memory regions (total 1)
    >     region fd    guest_phys_addr    memory_size        userspace_addr
    >    mmap_offset        mmap_addr
    >     ====== ===== ================== ================== ==================
    >    ================== ==================
    >      0     51    0x00007ff7c0000000 0x0000000040000000 0x00007ff7c0000000
    >    0x0000000000000000 0x00007ffc00000000
    >
    >    (4) vpp stack trace
    >    Program received signal SIGSEGV, Segmentation fault.
    >    [Switching to Thread 0x7ffd0e090700 (LWP 46570)]
    >    0x00007ffff7414642 in vhost_user_if_input
    >    (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
    >        node=0x7fffb76bab00, qid=<optimized out>, vui=0x7fffb6739700,
    >        vum=0x7ffff78f4480 <vhost_user_main>, vm=0x7fffb672a9c0)
    >        at
    > /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
    >    1596      if (PREDICT_FALSE (txvq->avail->flags & 0xFFFE))
    >    (gdb) bt
    >    #0  0x00007ffff7414642 in vhost_user_if_input
    >    (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
    >        node=0x7fffb76bab00, qid=<optimized out>, vui=0x7fffb6739700,
    >        vum=0x7ffff78f4480 <vhost_user_main>, vm=0x7fffb672a9c0)
    >        at
    > /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
    >    #1  vhost_user_input (f=<optimized out>, node=<optimized out>,
    >    vm=<optimized out>)
    >        at
    > /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
    >    #2  vhost_user_input_avx2 (vm=<optimized out>, node=<optimized out>,
    >    frame=<optimized out>)
    >        at
    > /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
    >    #3  0x00007ffff7954d35 in dispatch_node
    >    (last_time_stamp=12391212490024174, frame=0x0,
    >        dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
    >    node=0x7fffb76bab00,
    >        vm=0x7fffb672a9c0) at
    > /var/venom/vpp/build-data/../src/vlib/main.c:988
    >    #4  vlib_main_or_worker_loop (is_main=0, vm=0x7fffb672a9c0)
    >        at /var/venom/vpp/build-data/../src/vlib/main.c:1507
    >    #5  vlib_worker_loop (vm=0x7fffb672a9c0) at
    >    /var/venom/vpp/build-data/../src/vlib/main.c:1641
    >    #6  0x00007ffff6ad25d8 in clib_calljmp ()
    >        at /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
    >    #7  0x00007ffd0e08fdb0 in ?? ()
    >    #8  0x00007fffb4436edd in eal_thread_loop (arg=<optimized out>)
    >        at
    > 
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
    >    #9  0x0000000000000000 in ?? ()
    >    (gdb) frame 0
    >    #0  0x00007ffff7414642 in vhost_user_if_input
    >    (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
    >        node=0x7fffb76bab00, qid=<optimized out>, vui=0x7fffb6739700,
    >        vum=0x7ffff78f4480 <vhost_user_main>, vm=0x7fffb672a9c0)
    >        at
    > /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
    >    1596      if (PREDICT_FALSE (txvq->avail->flags & 0xFFFE))
    >    (gdb) p txvq
    >    $1 = (vhost_user_vring_t *) 0x7fffb6739ac0
    >    (gdb) p *txvq
    >    $2 = {cacheline0 = 0x7fffb6739ac0 "?", qsz_mask = 255, last_avail_idx
    >    = 0, last_used_idx = 0,
    >      n_since_last_int = 0, desc = 0x7ffbfff97000, avail = 0x7ffbfff98000,
    >    used = 0x7ffbfff99000,
    >      int_deadline = 0, started = 1 '\001', enabled = 0 '\000', log_used = 0
    > '\000',
    >      cacheline1 = 0x7fffb6739b00 "????\n", errfd = -1, callfd_idx = 10,
    >    kickfd_idx = 14,
    >      log_guest_addr = 0, mode = 1}
    >    (gdb) p *(txvq->avail)
    >    Cannot access memory at address 0x7ffbfff98000
    >    (gdb)
    >
    >    On Tue, May 29, 2018 at 10:47 AM, Ravi Kerur <rke...@gmail.com> wrote:
    >
    > Steve,
    >
    > Thanks for inputs on debugs and gdb. I am using gdb on my development
    > system to debug the issue. I would like to have reliable core
    > generation on the system on which I don't have access to install gdb.
    > I installed corekeeper and it still doesn't generate core. I am
    > running vpp inside a VM (VirtualBox/vagrant), not sure if I need to
    > set something inside vagrant config file.
    >
    > dpkg -l corekeeper
    > Desired=Unknown/Install/Remove/Purge/Hold
    > |
    > 
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    > |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    > ||/ Name                 Version         Architecture    Description
    > 
+++-====================-===============-===============-==============================================
    > ii  corekeeper           1.6             amd64           enable core
    > files and report crashes to the system
    >
    > Thanks.
    >
    > On Tue, May 29, 2018 at 9:38 AM, Steven Luong (sluong) <slu...@cisco.com>
    > wrote:
    >
    > Ravi,
    >
    > I install corekeeper and the core file is kept in /var/crash. But why not
    > use gdb to attach to the VPP process?
    > To turn on VPP vhost-user debug, type "debug vhost-user on" at the VPP
    > prompt.
    >
    > Steven
    >
    > On 5/29/18, 9:10 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
    > <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
    >
    >    Hi Marco,
    >
    >
    >    On Tue, May 29, 2018 at 6:30 AM, Marco Varlese <mvarl...@suse.de> 
wrote:
    >
    > Ravi,
    >
    > On Sun, 2018-05-27 at 12:20 -0700, Ravi Kerur wrote:
    >
    > Hello,
    >
    > I have a VM(16.04.4 Ubuntu x86_64) with 2 cores and 4G RAM. I have
    > installed VPP successfully on it. Later I have created vhost-user
    > interfaces via
    >
    > create vhost socket /var/run/vpp/sock1.sock server
    > create vhost socket /var/run/vpp/sock2.sock server
    > set interface state VirtualEthernet0/0/0 up
    > set interface state VirtualEthernet0/0/1 up
    >
    > set interface l2 bridge VirtualEthernet0/0/0 1
    > set interface l2 bridge VirtualEthernet0/0/1 1
    >
    > I then run 'DPDK/testpmd' inside a container which will use
    > virtio-user interfaces using the following command
    >
    > docker run -it --privileged -v
    > /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
    > /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
    > /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
    > 4 --log-level=9 -m 64 --no-pci --single-file-segments
    > --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:01:00:01:01:01
    > --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:01:00:01:01:02 --
    > -i
    >
    > VPP Vnet crashes with following message
    >
    > May 27 11:44:00 localhost vnet[6818]: received signal SIGSEGV, PC
    > 0x7fcca4620187, faulting address 0x7fcb317ac000
    >
    > Questions:
    > I have 'ulimit -c unlimited' and /etc/vpp/startup.conf has
    > unix {
    >  nodaemon
    >  log /var/log/vpp/vpp.log
    >  full-coredump
    >  cli-listen /run/vpp/cli.sock
    >  gid vpp
    > }
    >
    > But I couldn't locate corefile?
    >
    > The location of the coredump file depends on your system configuration.
    >
    > Please, check "cat /proc/sys/kernel/core_pattern"
    >
    > If you have systemd-coredump in the output of the above command, then 
likely
    > the
    > location of the coredump files is "/var/lib/systemd/coredump/"
    >
    > You can also change the location of where your system places the coredump
    > files:
    > echo '/PATH_TO_YOU_LOCATION/core_%e.%p' | sudo tee
    > /proc/sys/kernel/core_pattern
    >
    > See if that helps...
    >
    >
    >    Initially '/proc/sys/kernel/core_pattern' was set to 'core'. I changed
    >    it to 'systemd-coredump'. Still no core generated. VPP crashes
    >
    >    May 29 08:54:34 localhost vnet[4107]: received signal SIGSEGV, PC
    >    0x7f0167751187, faulting address 0x7efff43ac000
    >    May 29 08:54:34 localhost systemd[1]: vpp.service: Main process
    >    exited, code=killed, status=6/ABRT
    >    May 29 08:54:34 localhost systemd[1]: vpp.service: Unit entered failed
    > state.
    >    May 29 08:54:34 localhost systemd[1]: vpp.service: Failed with result
    > 'signal'.
    >
    >
    >    cat /proc/sys/kernel/core_pattern
    >    systemd-coredump
    >
    >
    >    ulimit -a
    >    core file size          (blocks, -c) unlimited
    >    data seg size           (kbytes, -d) unlimited
    >    scheduling priority             (-e) 0
    >    file size               (blocks, -f) unlimited
    >    pending signals                 (-i) 15657
    >    max locked memory       (kbytes, -l) 64
    >    max memory size         (kbytes, -m) unlimited
    >    open files                      (-n) 1024
    >    pipe size            (512 bytes, -p) 8
    >    POSIX message queues     (bytes, -q) 819200
    >    real-time priority              (-r) 0
    >    stack size              (kbytes, -s) 8192
    >    cpu time               (seconds, -t) unlimited
    >    max user processes              (-u) 15657
    >    virtual memory          (kbytes, -v) unlimited
    >    file locks                      (-x) unlimited
    >
    >    cd /var/lib/systemd/coredump/
    >    root@localhost:/var/lib/systemd/coredump# ls
    >    root@localhost:/var/lib/systemd/coredump#
    >
    >
    > (2) How to enable debugs? I have used 'make build' but no additional
    > logs other than those shown below
    >
    >
    > VPP logs from /var/log/syslog is shown below
    > cat /var/log/syslog
    > May 27 11:40:28 localhost vpp[6818]: vlib_plugin_early_init:361:
    > plugin path /usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: abf_plugin.so (ACL based Forwarding)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: acl_plugin.so (Access Control Lists)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device
    > Plugin)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:191: Loaded
    > plugin: cdp_plugin.so
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: flowprobe_plugin.so (Flow per Packet)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: gbp_plugin.so (Group Based Policy)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: gtpu_plugin.so (GTPv1-U)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: igmp_plugin.so (IGMP messaging)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: ioam_plugin.so (Inbound OAM)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:117: Plugin
    > disabled (default): ixge_plugin.so
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: l2e_plugin.so (L2 Emulation)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: lacp_plugin.so (Link Aggregation Control Protocol)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: lb_plugin.so (Load Balancer)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: memif_plugin.so (Packet Memory Interface (experimetal))
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: nat_plugin.so (Network Address Translation)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: pppoe_plugin.so (PPPoE)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
    > May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: srv6as_plugin.so (Static SRv6 proxy)
    > May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
    > May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
    > May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
    > plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: dpdk_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: udp_ping_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: lb_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: ioam_export_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: ioam_trace_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin:
    > vxlan_gpe_ioam_export_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: ioam_vxlan_gpe_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
    > load_one_vat_plugin:67: Loaded plugin: ioam_pot_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: flowprobe_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: stn_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: nat_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: udp_ping_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: pppoe_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: lacp_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: acl_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: ioam_export_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: ioam_trace_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: vxlan_gpe_ioam_export_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: gtpu_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: cdp_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: ioam_vxlan_gpe_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: memif_test_plugin.so
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
    > Loaded plugin: ioam_pot_test_plugin.so
    > May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]: dpdk: EAL
    > init args: -c 1 -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore
    > 0 --socket-mem 256,0
    > May 27 11:40:29 localhost /usr/bin/vpp[6818]: dpdk: EAL init args: -c
    > 1 -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore 0
    > --socket-mem 256,0
    > May 27 11:40:29 localhost vnet[6818]: dpdk_ipsec_process:1019: not
    > enough DPDK crypto resources, default to OpenSSL
    > May 27 11:43:19 localhost vnet[6818]: show vhost-user: unknown input 
`detail
    > May 27 11:44:00 localhost vnet[6818]: received signal SIGSEGV, PC
    > 0x7fcca4620187, faulting address 0x7fcb317ac000
    > May 27 11:44:00 localhost systemd[1]: vpp.service: Main process
    > exited, code=killed, status=6/ABRT
    > May 27 11:44:00 localhost systemd[1]: vpp.service: Unit entered failed
    > state.
    > May 27 11:44:00 localhost systemd[1]: vpp.service: Failed with result
    > 'signal'.
    > May 27 11:44:00 localhost systemd[1]: vpp.service: Service hold-off
    > time over, scheduling restart
    >
    >
    >    Thanks,
    >    Ravi
    >
    >
    >
    > Thanks.
    >
    > Cheers,
    > Marco
    >
    >
    >
    >
    > --
    > Marco V
    >
    > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
    > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    > 
    >
    >
    
    
    
    


-=-=-=-=-=-=-=-=-=-=-=-
Links:

You receive all messages sent to this group.

View/Reply Online (#9539): https://lists.fd.io/g/vpp-dev/message/9539
View All Messages In Topic (28): https://lists.fd.io/g/vpp-dev/topic/20346431
Mute This Topic: https://lists.fd.io/mt/20346431/21656
New Topic: https://lists.fd.io/g/vpp-dev/post

Change Your Subscription: https://lists.fd.io/g/vpp-dev/editsub/21656
Group Home: https://lists.fd.io/g/vpp-dev
Contact Group Owner: vpp-dev+ow...@lists.fd.io
Terms of Service: https://lists.fd.io/static/tos
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
Email sent to: arch...@mail-archive.com
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to