On Fri, Dec 2, 2022 at 10:20 AM Maxime Coquelin
<[email protected]> wrote:
>
>
>
> On 12/2/22 15:59, Mike Pattrick wrote:
> > On Fri, Dec 2, 2022 at 5:37 AM Maxime Coquelin
> > <[email protected]> wrote:
> >>
> >>
> >>
> >> On 12/2/22 11:09, David Marchand wrote:
> >>> On Wed, Nov 30, 2022 at 9:30 PM Ilya Maximets <[email protected]> wrote:
> >>>>>>>> Shouldn't this be 0x7f instead?
> >>>>>>>> 0x3f doesn't enable bit #6, which is responsible for dumping
> >>>>>>>> shared huge pages. Or am I missing something?
> >>>>>>>
> >>>>>>> That's a good point, the hugepage may or may not be private. I'll send
> >>>>>>> in a new one.
> >>>>>>
> >>>>>> OK. One thing to think about though is that we'll grab
> >>>>>> VM memory, I guess, in case we have vhost-user ports.
> >>>>>> So, the core dump size can become insanely huge.
> >>>>>>
> >>>>>> The downside of not having them is inability to inspect
> >>>>>> virtqueues and stuff in the dump.
> >>>>>
> >>>>> Did you consider madvise()?
> >>>>>
> >>>>> MADV_DONTDUMP (since Linux 3.4)
> >>>>> Exclude from a core dump those pages in the range
> >>>>> specified by addr and length. This is useful in applications that
> >>>>> have large areas of memory that are known not to be useful in a core
> >>>>> dump. The effect of MADV_DONT‐
> >>>>> DUMP takes precedence over the bit mask that is set via
> >>>>> the /proc/[pid]/coredump_filter file (see core(5)).
> >>>>>
> >>>>> MADV_DODUMP (since Linux 3.4)
> >>>>> Undo the effect of an earlier MADV_DONTDUMP.
> >>>>
> >>>> I don't think OVS actually knows location of particular VM memory
> >>>> pages that we do not need. And dumping virtqueues and stuff is,
> >>>> probably, the point of this patch (?).
> >>>>
> >>>> vhost-user library might have a better idea on which particular parts
> >>>> of the memory guest may use for virtqueues and buffers, but I'm not
> >>>> 100% sure.
> >>>
> >>> Yes, distinguishing hugepages of interest is a problem.
> >>>
> >>> Since v20.05, DPDK mem allocator takes care of excluding (unused)
> >>> hugepages from dump.
> >>> So with this OVS patch, if we catch private and shared hugepages,
> >>> "interesting" DPDK hugepages will get dumped, which is useful for
> >>> debugging post mortem.
> >>>
> >>> Adding Maxime, who will have a better idea of what is possible for the
> >>> guest mapping part.
> >>>
> >>>
> >>
> >> I wonder if we could do a MADV_DONTDUMP on all the guest memory at mmap
> >> time, then there are two cases:
> >> a. vIOMMU = OFF. In this case we could do MADV_DODUMP on virtqueues
> >> memory. Doing so, we would have the rings memory, but not their buffers
> >> (except if they are located on same hugepages).
> >> b. vIOMMU = ON. In this case we could do MADV_DODUMP on IOTLB_UPDATE
> >> new entries and MADV_DONTDUMP on invalidated entries. Doing so we will
> >> get both vrings and their buffers the backend is allowed to access.
> >>
> >> I can prepare a PoC quickly if someone is willing to experiment.
> >
> > A big motivation for this patch is DPDK becomes a big black hole in
> > coredumps, preventing even netdev structures from being enumerated.
> >
> > I threw together a small PoC for this yesterday, only marking vhost
> > mmaps as DONTDUMP. The result was that a simple OVS configuration with
> > one vhost attached to an 8gb VM dropped the zstd compressed coredump
> > including shared huge pages from 486.4M to 19.8M. The resulting
> > coredump was also significantly more debuggable, all internal OVS
> > datastructures became available as well as many DPDK ones.
> >
> > I'll look at cleaning things up and submitting to the DPDK mailing list.
>
> Thanks for doing the PoC, that will definitely help debugging!
>
> I have quickly drafted a patch doing the MADV_DODUMP on the area of
> interest for Vhost as described in my initial reply, could you have a
> try with it? (disclaimer: not even build tested)
Dump with shared huge pages
Thu 2022-12-01 13:07:33 EST 213904 0 0 SIGSEGV present
/usr/sbin/ovs-vswitchd 486.4M
uncompressed size: 15,047,430,144
Dump with vhost mmaps set to dontdump
Thu 2022-12-01 13:23:02 EST 217253 0 0 SIGSEGV present
/usr/sbin/ovs-vswitchd 19.8M
uncompressed size: 4,310,011,904
Dump with Maxime patch
Fri 2022-12-02 11:34:16 EST 232661 0 0 SIGSEGV present
/usr/sbin/ovs-vswitchd 19.7M
uncompressed size: 4,310,007,808
Not sure where the 4kb went from yesterday to today
-M
>
> Maxime
>
> =========================================================================
>
> diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c
> index 6a729e8804..555b4ebba0 100644
> --- a/lib/vhost/iotlb.c
> +++ b/lib/vhost/iotlb.c
> @@ -149,6 +149,7 @@ vhost_user_iotlb_cache_remove_all(struct
> vhost_virtqueue *vq)
> rte_rwlock_write_lock(&vq->iotlb_lock);
>
> RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {
> + madvise(node->uaddr, node->size, MADV_DODUMP);
> TAILQ_REMOVE(&vq->iotlb_list, node, next);
> vhost_user_iotlb_pool_put(vq, node);
> }
> @@ -170,6 +171,7 @@ vhost_user_iotlb_cache_random_evict(struct
> vhost_virtqueue *vq)
>
> RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {
> if (!entry_idx) {
> + madvise(node->uaddr, node->size, MADV_DODUMP);
> TAILQ_REMOVE(&vq->iotlb_list, node, next);
> vhost_user_iotlb_pool_put(vq, node);
> vq->iotlb_cache_nr--;
> @@ -222,12 +224,14 @@ vhost_user_iotlb_cache_insert(struct virtio_net
> *dev, struct vhost_virtqueue *vq
> vhost_user_iotlb_pool_put(vq, new_node);
> goto unlock;
> } else if (node->iova > new_node->iova) {
> + madvise(new_node->uaddr, new_node->size,
> MADV_DODUMP);
> TAILQ_INSERT_BEFORE(node, new_node, next);
> vq->iotlb_cache_nr++;
> goto unlock;
> }
> }
>
> + madvise(new_node->uaddr, new_node->size, MADV_DODUMP);
> TAILQ_INSERT_TAIL(&vq->iotlb_list, new_node, next);
> vq->iotlb_cache_nr++;
>
> @@ -255,6 +259,7 @@ vhost_user_iotlb_cache_remove(struct vhost_virtqueue
> *vq,
> break;
>
> if (iova < node->iova + node->size) {
> + madvise(node->uaddr, node->size, MADV_DODUMP);
> TAILQ_REMOVE(&vq->iotlb_list, node, next);
> vhost_user_iotlb_pool_put(vq, node);
> vq->iotlb_cache_nr--;
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 371d6304d6..208377ddc6 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -767,6 +767,8 @@ translate_ring_addresses(struct virtio_net **pdev,
> struct vhost_virtqueue **pvq)
> return;
> }
>
> + madvise(vq->desc_packed, len, MADV_DODUMP);
> +
> numa_realloc(&dev, &vq);
> *pdev = dev;
> *pvq = vq;
> @@ -782,6 +784,8 @@ translate_ring_addresses(struct virtio_net **pdev,
> struct vhost_virtqueue **pvq)
> return;
> }
>
> + madvise(vq->driver_event, len, MADV_DODUMP);
> +
> len = sizeof(struct vring_packed_desc_event);
> vq->device_event = (struct vring_packed_desc_event *)
> (uintptr_t)ring_addr_to_vva(dev,
> @@ -793,6 +797,8 @@ translate_ring_addresses(struct virtio_net **pdev,
> struct vhost_virtqueue **pvq)
> return;
> }
>
> + madvise(vq->device_event, len, MADV_DODUMP);
> +
> vq->access_ok = true;
> return;
> }
> @@ -809,6 +815,8 @@ translate_ring_addresses(struct virtio_net **pdev,
> struct vhost_virtqueue **pvq)
> return;
> }
>
> + madvise(vq->desc, len, MADV_DODUMP);
> +
> numa_realloc(&dev, &vq);
> *pdev = dev;
> *pvq = vq;
> @@ -824,6 +832,8 @@ translate_ring_addresses(struct virtio_net **pdev,
> struct vhost_virtqueue **pvq)
> return;
> }
>
> + madvise(vq->avail, len, MADV_DODUMP);
> +
> len = sizeof(struct vring_used) +
> sizeof(struct vring_used_elem) * vq->size;
> if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX))
> @@ -836,6 +846,8 @@ translate_ring_addresses(struct virtio_net **pdev,
> struct vhost_virtqueue **pvq)
> return;
> }
>
> + madvise(vq->used, len, MADV_DODUMP);
> +
> if (vq->last_used_idx != vq->used->idx) {
> VHOST_LOG_CONFIG(dev->ifname, WARNING,
> "last_used_idx (%u) and vq->used->idx (%u)
> mismatches;\n",
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
