Re: [Qemu-devel] [PATCH v4 2/8] acpi: add vmcoreinfo device

[Dave Anderson]

- Original Message -
> Hi Dave
> 
> On Wed, Jul 26, 2017 at 10:21 AM, Michael S. Tsirkin  wrote:
> > On Sat, Jul 15, 2017 at 01:47:50AM +0200, Marc-André Lureau wrote:
> >> >
> >> > There's more info scattered in other places.
> >> >
> >> > Why do you get to document it? Because you are the one exposing it
> >> > across the hypervisor/vm boundary where it will need to be
> >> > understood by people/tools not running within guest.
> >> >
> >> > So "just read the script in qemu source" is not how an interface
> >> > should be documented.
> >>
> >> I don't understand the issue, it's a kernel ELF note that qemu passes
> >> for dump/crash tools in the dump headers/sections.
> >
> > The way it looks to me, this patchset is exposing an internal kernel
> > detail and making it part of ABI maybe it already is, my point was 1.
> > should we get a confirmation from upstream it's not going to change? 2.
> > if it's ABI let's document what do we expect to be there.
> 
> 
> Could you help explain the expectations and stability guarantees of
> vmcoreinfo ELF note ?
> 
> I am a bit stuck here, after all, vmcoreinfo is mostly used by crash
> so I thought you could help.

Sorry for the delay, I'm just back from vacation...

The vmcoreinfo string data is *primarily* used by the makedumpfile facility,
because it needs to find its way around the dumped kernel memory without having
the the vmlinux file's debuginfo data when it's running in the second kernel.

Since crash utilizes the vmlinux file, it only reads a small handful of 
the vmcoreinfo items for things like phys_base that cannot be gleaned
from the vmlinux debuginfo data.  

Anyway, I'm not sure what you mean by "expectations and stability guarantees"?
It's just a block of ASCII string data of a given size that simply needs
to be transferred to the vmcore.  New strings may be added to it at any
time, but that shouldn't have any impact on what you're doing.

Dave
 
 
> The only thing qemu does with it is try to get NUMBER(phys_base)=
> field to update the phys_base used in the various dump headers. (this
> could be dropped, and qemu ignoring the note content, if the debug
> tools take vmcoreinfo values  with higher priority than other header
> fields)
> 
> > But again since there's not a whole lot of documentation here
> > that you provided, I might be misunderstanding completely.
> 
> Because there isn't much available in the kernel either, except
> Documentation/ABI/testing/sysfs-kernel-vmcoreinfo.
> 
> 
> --
> Marc-André Lureau
> 

Re: [Qemu-devel] [PATCH v2 5/7] kdump: add vmcoreinfo ELF note

[Dave Anderson]

- Original Message -
> Hi
> 
> On Thu, Jul 6, 2017 at 7:13 PM, Laszlo Ersek  wrote:
> > On 07/06/17 12:16, Marc-André Lureau wrote:
> >> kdump header provides offset and size of the vmcoreinfo ELF note,
> >> append it if available.
> >>
> >> Signed-off-by: Marc-André Lureau 
> >> ---
> >>  dump.c | 20 
> >>  1 file changed, 20 insertions(+)
> >>
> >> diff --git a/dump.c b/dump.c
> >> index f699198204..dd416ad271 100644
> >> --- a/dump.c
> >> +++ b/dump.c
> >> @@ -839,6 +839,16 @@ static void create_header32(DumpState *s, Error
> >> **errp)
> >>  kh->dump_level = cpu_to_dump32(s, DUMP_LEVEL);
> >>
> >>  offset_note = DISKDUMP_HEADER_BLOCKS * block_size + size;
> >> +if (s->vmcoreinfo) {
> >> +uint64_t hsize, name_size, size_vmcoreinfo_desc,
> >> offset_vmcoreinfo;
> >> +
> >> +get_note_sizes(s, s->vmcoreinfo, &hsize, &name_size,
> >> &size_vmcoreinfo_desc);
> >> +offset_vmcoreinfo = offset_note + s->note_size -
> >> s->vmcoreinfo_size +
> >> +(DIV_ROUND_UP(hsize, 4) + DIV_ROUND_UP(name_size, 4)) * 4;
> >> +kh->offset_vmcoreinfo = cpu_to_dump64(s, offset_vmcoreinfo);
> >> +kh->size_vmcoreinfo = cpu_to_dump32(s, size_vmcoreinfo_desc);
> >> +}
> >> +
> >>  kh->offset_note = cpu_to_dump64(s, offset_note);
> >>  kh->note_size = cpu_to_dump32(s, s->note_size);
> >>
> >> @@ -939,6 +949,16 @@ static void create_header64(DumpState *s, Error
> >> **errp)
> >>  kh->dump_level = cpu_to_dump32(s, DUMP_LEVEL);
> >>
> >>  offset_note = DISKDUMP_HEADER_BLOCKS * block_size + size;
> >> +if (s->vmcoreinfo) {
> >> +uint64_t hsize, name_size, size_vmcoreinfo_desc,
> >> offset_vmcoreinfo;
> >> +
> >> +get_note_sizes(s, s->vmcoreinfo, &hsize, &name_size,
> >> &size_vmcoreinfo_desc);
> >> +offset_vmcoreinfo = offset_note + s->note_size -
> >> s->vmcoreinfo_size +
> >> +(DIV_ROUND_UP(hsize, 4) + DIV_ROUND_UP(name_size, 4)) * 4;
> >> +kh->offset_vmcoreinfo = cpu_to_dump64(s, offset_vmcoreinfo);
> >> +kh->size_vmcoreinfo = cpu_to_dump64(s, size_vmcoreinfo_desc);
> >> +}
> >> +
> >>  kh->offset_note = cpu_to_dump64(s, offset_note);
> >>  kh->note_size = cpu_to_dump64(s, s->note_size);
> >>
> >>
> >
> > I continue to think that this patch is unnecessary, but if you insist,

> > it does look OK to me.
> >
> > Reviewed-by: Laszlo Ersek 
> 
> Without it, crash doesn't read the vmcoreinfo PT_NOTE. And for some
> reason, the phys_base in the header wasn't enough (to be doubled
> checked).
>
> Any comment Dave about crash handling of vmcoreinfo in kdump files?

It just reads the kdump_sub_header's offset_vmcoreinfo and size_vmcoreinfo
fields to gather the vmcoreinfo data into a local buffer of memory, and
scans the strings for whatever it's looking for. 

With respect to phys_base, the only thing that might be of consequence
is this fairly recent change that's currently only in the github repo,
queued for crash-7.2.0:

  commit a4a538caca140a8e948bbdae2be311168db7a1eb
  Author: Dave Anderson 
  Date:   Tue May 2 16:51:53 2017 -0400

Fix for Linux 4.10 and later kdump dumpfiles, or kernels that have
backported commit 401721ecd1dcb0a428aa5d6832ee05ffbdbffbbe, titled
"kexec: export the value of phys_base instead of symbol address".
Without the patch, if the x86_64 "phys_base" value in the VMCOREINFO
note is a negative decimal number, the crash session fails during
session intialization with a "page excluded" or "seek error" when
reading "page_offset_base".
(ander...@redhat.com)

Also, crash-7.1.9 was the first version that started looking in the 
vmcoreinfo data for phys_base instead of in the kdump_sub_header.

Dave

> 
> 
> 
> --
> Marc-André Lureau
> 

Re: [Qemu-devel] [PATCH 00/21] WIP: dump: add kaslr support (for after 2.9)

[Dave Anderson]

- Original Message -
> Hi,
> 
> Latest linux kernel enabled kaslr to randomize phys/virt memory
> addresses. There has been some effort to support kexec/kdump so that
> crash utility can still works in case crashed kernel has kaslr
> enabled.
> 
> This series aims to provide enough information in qemu dumps so that
> crash utility can work with kaslr kernel too, with x86_64 guests (it
> hasn't been tested on other archs, help welcome).
> 
> Two pieces of informations are necessary: the phys_base (the physical
> address where the kernel is loaded) and the kaslr offset. Other useful
> informations for debugging are provided in Linux vmcoreinfo too.
> 
> There has been discussions to provide those details in early boot,
> with ACPI, fw_cfg, virtio-pstore etc, but none has reached a consensus
> yet. A possiblity for now is to provide the information when qemu-ga
> starts. This has the advantage of working with older guests and may
> not be incompatible with future methods (in this case it could ignore
> qga info for example). The drawback is that qemu will have to parse
> the json stream. For now it stops processing it whem VMDUMP_INFO is
> received. It would be wise to introduce a QMP welcome message, so that
> qemu would stop immediately processing the stream if the agent doesn't
> have the event. Another option is to create a seperate channel...
> I welcome other ideas and discussion.
> 
> crash upstream doesn't yet parse NUMBER(phys_base). You may pass it to
> crash with --machdep phys_base=0x.. provided you snooped on qga.

FYI: a patch to utilize NUMBER(phys_base) has been checked in upstream:

  
https://github.com/crash-utility/crash/commit/ed60e97e319a1cfc9e2779aa1baac305677393d8

  Linux 4.10 commit 401721ecd1dcb0a428aa5d6832ee05ffbdbffbbe finally
  exports the x86_64 "phys_base" value in the VMCOREINFO note, so
  utilize it whenever it exists.
  (ander...@redhat.com)

Thanks,
  Dave


> kdump kaslr-dumps should now work with crash out of the box.
> 
> A large part of the series has to do with json and the qobject type
> system to deal with uint64 values, so that addresses can now be sent
> over json. The second halfs adds qga VMDUMP_INFO event, and ELF/kdump
> dumping. Depending on the feedback, I will probably split the series,
> but for those who would like to try it, help or suggest ideas, here is
> the whole thing.
> 
> cheers
> 
> Marc-André Lureau (21):
>   qapi: add info comment for generated types
>   pci-host: use more specific type names
>   object: fix potential leak in getters
>   qobject: add quint type
>   qapi: update the qobject visitor to use QUInt
>   json: learn to parse uint64 numbers
>   object: add uint property setter/getter
>   qdev: use int and uint properties
>   qdev: use appropriate type
>   Use uint property getter/setter where appropriate
>   qdict: learn to lookup quint
>   test-qga: drop everything until guest-sync
>   qga: report error on keyfile dump error
>   qga: add and populate VMDumpInfo
>   qga: register event emit function
>   qga: emit VMDUMP_INFO event
>   virtio-channel: parse qga stream for VMDUMP_INFO event
>   dump: use qga VMDUMP_INFO for ELF dump
>   kdump: write vmcoreinfo in header
>   scripts/dump-guest-memory.py: fix int128_get64 on recent gcc
>   scripts/dump-guest-memory.py: add VMCOREINFO
> 
>  qapi/introspect.json |   2 +-
>  scripts/qapi.py  |  39 ---
>  scripts/dump-guest-memory.py |  66 ++-
>  scripts/qapi-event.py|   4 +-
>  scripts/qapi-types.py|  17 +--
>  scripts/qapi-visit.py|   3 +-
>  include/hw/isa/isa.h |   2 +-
>  include/hw/qdev-core.h   |   5 +-
>  include/hw/qdev-properties.h |  59 ++
>  include/qapi/qmp/qdict.h |   2 +
>  include/qapi/qmp/quint.h |  25 +
>  include/qapi/qmp/types.h |   1 +
>  include/qom/object.h |  23 
>  include/sysemu/dump-info.h   |  15 +++
>  include/sysemu/dump.h|   2 +
>  qga/guest-agent-core.h   |   2 +
>  block/qapi.c |   5 +
>  dump.c   | 184
>  ++-
>  hw/acpi/memory_hotplug.c |  10 +-
>  hw/acpi/nvdimm.c |  10 +-
>  hw/acpi/pcihp.c  |   6 +-
>  hw/arm/aspeed.c  |   4 +-
>  hw/arm/bcm2835_peripherals.c |   9 +-
>  hw/arm/raspi.c   |   4 +-
>  hw/block/fdc.c   |  54 -
>  hw/char/virtio-console.c |  53 +
>  hw/core/platform-bus.c   |   2 +-
>  hw/core/qdev-properties.c|   8 +-
>  hw/core/qdev.c   |   8 +-
>  hw/i386/acpi-build.c |  70 ++--
>  hw

Re: [Qemu-devel] virsh dump (qemu guest memory dump?): KASLR enabled linux guest support

[Dave Anderson]

- Original Message -
> 
> 
> On 09/11/2016 16:28, Dave Anderson wrote:
> > I'm not sure whether this "guest userspace agent" is still in play here,
> > but if there were such a thing, it could theoretically do the same
> > thing that crash currently does when running on a live system.
> > 
> > Both of those are available or calculatable from the contents of
> > a kdump header.  However, on a live system, it's done like this:
> > 
> > - /proc/kallsyms is queried for the symbol value of "_text", which would
> >   be relocated if KASLR is in play.  That value is compared against the
> >   "_text" symbol value compiled into the vmlinux file to determine the
> >   relocation value generated by CONFIG_RANDOMIZE_BASE.
> > 
> > [...] in order to read kernel symbols from the
> > statically-mapped kernel region based at __START_KERNEL_map, it
> > translates a (possibly relocated) kernel virtual address into a
> > physical address like this:
> > 
> >   physical-address = virtual-address - __START_KERNEL_map + phys_base
> > 
> > But it's a chicken-and-egg deal, because the contents of the "phys_base"
> > symbol are needed to calculate the physical address, but it can't
> > read the "phys_base" symbol contents without first knowing its contents.
> > 
> > So on a live system, the "phys_base" is calculated by reading
> > the "Kernel Code:" value from /proc/iomem, and then doing this:
> > 
> >   phys_base = [Kernel Code: value] - ["_text" symbol value] - 
> > __START_KERNEL_map
>
> ^^^
> 
> Should there be parentheses around this?  

Yes, sorry, that's correct -- that's what the code does, and what I meant to 
express...

Dave


> The physical-address formula above is equivalent to
> 
> phys_base = physical-address - (virtual-address - __START_KERNEL_map)
> 
> > 
> > So theoretically, the guest agent could read /proc/iomem and /proc/kallsyms
> > for the information required.  (I think...)
> 
> Then yes, the guest-agent could add a command get-kernel-text-start with an 
> output like:
> 
> { 'virtual': 0x8600, 'physical': 0xb600 }
> 
> and libvirt can expose it to crash.  In this case, phys_base would be 
> 0xb000
> if I did the math right, and the relocation value is obtained by comparing the
> "virtual" address with the vmlinux "_text".
> 
> IIRC the guest agent runs as root, so reading /proc/iomem is not a problem.
> 
> Paolo
> 

Re: [Qemu-devel] virsh dump (qemu guest memory dump?): KASLR enabled linux guest support

[Dave Anderson]

- Original Message -
> On Wed, Nov 09, 2016 at 11:01:46AM +0800, Dave Young wrote:
> > Hi,
> > 
> > Latest linux kernel enabled kaslr to randomiz phys/virt memory
> > addresses, we had some effort to support kexec/kdump so that crash
> > utility can still works in case crashed kernel has kaslr enabled.
> > 
> > But according to Dave Anderson virsh dump does not work, quoted messages
> > from Dave below:
> > 
> > """
> > with virsh dump, there's no way of even knowing that KASLR
> > has randomized the kernel __START_KERNEL_map region, because there is no
> > virtual address information -- e.g., like "SYMBOL(_stext)" in the kdump
> > vmcoreinfo data to compare against the vmlinux file symbol value.
> > Unless virsh dump can export some basic virtual memory data, which
> > they say it can't, I don't see how KASLR can ever be supported.
> > """
> > 
> > I assume virsh dump is using qemu guest memory dump facility so it
> > should be first addressed in qemu. Thus post this query to qemu devel
> > list. If this is not correct please let me know.
> > 
> > Could you qemu dump people make it work? Or we can not support virt dump
> > as long as KASLR being enabled. Latest Fedora kernel has enabled it in
> > x86_64.
> >
> 
> When the -kernel command line option is used, then it may be possible
> to extract some information that could be used to supplement the memory
> dump that dump-guest-memory provides. However, that would be a specific
> use. In general, QEMU knows nothing about the guest kernel. It doesn't
> know where it is in the disk image, and it doesn't even know if it's
> Linux.
> 
> Is there anything a guest userspace application could probe from e.g.
> /proc that would work? If so, then the guest agent could gain a new
> feature providing that.
> 
> Thanks,
> drew

I'm not sure whether this "guest userspace agent" is still in play here,
but if there were such a thing, it could theoretically do the same
thing that crash currently does when running on a live system.

Two basic necessities are are needed, whether running live or against
a dumpfile:

(1) the CONFIG_RANDOMIZE_BASE relocation value that modifies the
kernel virtual address range compiled into the vmlinux file, which
starts at the hardwired __START_KERNEL_map address.

(2) the contents of the kernel's "phys_base" symbol.

Both of those are available or calculatable from the contents of
a kdump header.  However, on a live system, it's done like this:

- /proc/kallsyms is queried for the symbol value of "_text", which would
  be relocated if KASLR is in play.  That value is compared against the
  "_text" symbol value compiled into the vmlinux file to determine the
  relocation value generated by CONFIG_RANDOMIZE_BASE.

Given that relocation value, and before any kernel memory is accessed,
crash goes in a backdoor into its embedded gdb module, and modifies
the data structures of all kernel symbols, applying the relocation
value.

Once that's done, in order to read kernel symbols from the 
statically-mapped kernel region based at __START_KERNEL_map, it 
translates a (possibly relocated) kernel virtual address into a
physical address like this:

  physical-address = virtual-address - __START_KERNEL_map + phys_base

But it's a chicken-and-egg deal, because the contents of the "phys_base"
symbol are needed to calculate the physical address, but it can't
read the "phys_base" symbol contents without first knowing its contents.

So on a live system, the "phys_base" is calculated by reading
the "Kernel Code:" value from /proc/iomem, and then doing this:

  phys_base = [Kernel Code: value] - ["_text" symbol value] - __START_KERNEL_map

So theoretically, the guest agent could read /proc/iomem and /proc/kallsyms
for the information required.  (I think...)

Dave
 

Re: [Qemu-devel] virsh dump (qemu guest memory dump?): KASLR enabled linux guest support

[Dave Anderson]

- Original Message -
> On 11/09/16 at 11:58am, Wen Congyang wrote:
> > On 11/09/2016 11:17 AM, Dave Young wrote:
> > > Drop qiaonuohan, seems the mail address is wrong..
> > > 
> > > On 11/09/16 at 11:01am, Dave Young wrote:
> > >> Hi,
> > >>
> > >> Latest linux kernel enabled kaslr to randomiz phys/virt memory
> > >> addresses, we had some effort to support kexec/kdump so that crash
> > >> utility can still works in case crashed kernel has kaslr enabled.
> > >>
> > >> But according to Dave Anderson virsh dump does not work, quoted messages
> > >> from Dave below:
> > >>
> > >> """
> > >> with virsh dump, there's no way of even knowing that KASLR
> > >> has randomized the kernel __START_KERNEL_map region, because there is no
> > >> virtual address information -- e.g., like "SYMBOL(_stext)" in the kdump
> > >> vmcoreinfo data to compare against the vmlinux file symbol value.
> > >> Unless virsh dump can export some basic virtual memory data, which
> > >> they say it can't, I don't see how KASLR can ever be supported.
> > >> """
> > >>
> > >> I assume virsh dump is using qemu guest memory dump facility so it
> > >> should be first addressed in qemu. Thus post this query to qemu devel
> > >> list. If this is not correct please let me know.
> > 
> > IIRC, 'virsh dump --memory-only' uses dump-guest-memory, and 'virsh dump'
> > uses migration to dump.
> 
> Do they need different fixes? Dave, I guess you mean --memory-only, but
> could you clarify and confirm it?

As I understand it, the "--memory-only" option uses a new "dump-guest-memory"
QEMU monitor command that creates an ELF kdump vmcore clone. 

Dave


> 
> > 
> > I think I should study kaslr first...
> 
> Thanks for taking care of it.
> 
> > 
> > Thanks
> > Wen Congyang
> > 
> > >>
> > >> Could you qemu dump people make it work? Or we can not support virt dump
> > >> as long as KASLR being enabled. Latest Fedora kernel has enabled it in
> > >> x86_64.
> > >>
> > >> Thanks
> > >> Dave
> > > 
> > > 
> > > 
> > 
> > 
> > 
> 

Re: [Qemu-devel] virsh dump (qemu guest memory dump?): KASLR enabled linux guest support

[Dave Anderson]

- Original Message -
> Hi,
> 
> Latest linux kernel enabled kaslr to randomiz phys/virt memory
> addresses, we had some effort to support kexec/kdump so that crash
> utility can still works in case crashed kernel has kaslr enabled.
> 
> But according to Dave Anderson virsh dump does not work, quoted messages
> from Dave below:
> 
> """
> with virsh dump, there's no way of even knowing that KASLR
> has randomized the kernel __START_KERNEL_map region, because there is no
> virtual address information -- e.g., like "SYMBOL(_stext)" in the kdump
> vmcoreinfo data to compare against the vmlinux file symbol value.
> Unless virsh dump can export some basic virtual memory data, which
> they say it can't, I don't see how KASLR can ever be supported.
> """

We also need the x86_64 phys_base value.

As it is right now, virsh dump vmcores work by luck.  It is presumed that
the __START_KERNEL_map region is unmodified (i.e., what's in the vmlinux file),
and the phys_base value is guessed by checking phys_base values from 
-16MB to +16MB in 1MB chunks.  If the phys_base value is not one of those
32 possible values, the crash session will fail.

Dave


> 
> I assume virsh dump is using qemu guest memory dump facility so it
> should be first addressed in qemu. Thus post this query to qemu devel
> list. If this is not correct please let me know.
> 
> Could you qemu dump people make it work? Or we can not support virt dump
> as long as KASLR being enabled. Latest Fedora kernel has enabled it in
> x86_64.
> 
> Thanks
> Dave
> 

Re: [Qemu-devel] [Crash-utility] uniquely identifying KDUMP files that originate from QEMU

[Dave Anderson]

- Original Message -

> > Can you fetch that in "crash"? If you can, then there's nothing to do on
> > the qemu side (and I'll have to apologize for spamming a bunch of lists
> > :/).

Well, let's be clear -- I was the one who put you up to it...

But no apology is required -- and in fact, if today's discussion results in
the "phys_base" vmcoreinfo issue being resolved in the kernel, we'll all be
eternally grateful.

Thanks again,
  Dave

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Dave Anderson]

- Original Message -
> adding back a few CC's because this discussion is useful
> 
> On 11/12/14 19:43, Petr Tesarik wrote:
> > V Wed, 12 Nov 2014 15:50:32 +0100
> > Laszlo Ersek  napsáno:
> > 
> >> On 11/12/14 09:04, Petr Tesarik wrote:
> >>> On Wed, 12 Nov 2014 12:08:38 +0900 (JST)
> >>> HATAYAMA Daisuke  wrote:
> >>
>  Anyway, phys_base is kernel information. To make it available for qemu
>  side, there's need to prepare a mechanism for qemu to have any access
>  to it.
> >>>
> >>> Yes. I wonder if you can have access without some sort of co-operation
> >>> from the guest kernel itself. I guess not.
> >>
> >> Propagating any kind of additional information from the guest kernel
> >> (which is unprivileged and potentially malicious) to the host-side qemu
> >> process (which is by definition more privileged, although still confined
> >> by various measures) is something we'd explicitly like to avoid.
> >>
> >> Think of it like this. I throw a physical box at you, running Linux,
> >> that has frozen in time. Can "crash" work with nothing else but the
> >> contents of the memory, and information about the CPUs?
> > 
> > If only you could save the _complete_ state of the CPU... For example
> > the content of CR3 would be quite useful.
> 
> (1) CR3 is already saved, in both the ELF and the kdump compressed formats.
> 
> - ELF case:
> 
> qmp_dump_guest_memory() [dump.c]
>   create_vmcore()
> dump_begin()
>   write_elf64_notes()
> 
> loop from 1 to #vcpu:
>   cpu_write_elf64_note() [qom/cpu.c]
> x86_64_write_elf64_note() [target-i386/arch_dump.c]
>   writes "CORE"
> 
> loop from 1 to #vcpu:
>   cpu_write_elf64_qemunote() [qom/cpu.c]
> x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c]
>   cpu_write_qemu_note()
> qemu_get_cpustate()
>   s->cr[3] = env->cr[3]; <-- here
> writes "QEMU"
> 
> Hence, the information is part of the QEMU note.
> 
> - kdump case:
> 
> qmp_dump_guest_memory() [dump.c]
>   create_kdump_vmcore()
> write_dump_header()
>   create_header64()
> write_elf64_notes()
>   [... same as above ...]
> 
> The trick here is that the note-writer functions use a callback function
> for actually outputting the data. So while in the ELF case the stuff
> goes directly to a file, in the kdump case the notes are first saved in
> a memory buffer, and then later saved in the file at offset
> KdumpSubHeader64.offset_note. (... Which is then represented in the
> flattened file format of course.)
> 
> So, the information is there in both cases.
> 
> 
> (2) Dave -- this just made me realize that the QEMU note is *already*
> there in the kdump file as well; pointed-to by
> KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.
> 
> From your other email
> :
> 
> > sub_header_kdump: 1c9cff0
> >  phys_base: 0
> > dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
> >  split: 0
> >  start_pfn: (unused)
> >end_pfn: (unused)
> >  offset_vmcoreinfo: 0 (0x0)
> >size_vmcoreinfo: 0 (0x0)
> >offset_note: 4200 (0x1068)   <--- here
> >  size_note: 3232 (0xca0)<---
> > num_prstatus_notes: 4
> >  notes_buf: 1c9e000
> >   notes[0]: 1c9e000
> >   notes[1]: 1c9e164
> >   notes[2]: 1c9e2c8
> >   notes[3]: 1c9e42c
> > NT_PRSTATUS_offset: 1068
> > 11cc
> > 1330
> > 1494
> >   offset_eraseinfo: 0 (0x0)
> > size_eraseinfo: 0 (0x0)
> >   start_pfn_64: (unused)
> > end_pfn_64: (unused)
> >   max_mapnr_64: 1245184 (0x13)
> 
> Can you fetch that in "crash"? If you can, then there's nothing to do on
> the qemu side (and I'll have to apologize for spamming a bunch of lists :/).

Sure enough...

I was just playing with process_el64_notes() to check/read the note name 
strings,
and noticed that I can certainly see them.  But as you noted, only the 
NT_PRSTATUS
notes are stored in the "notes[]" array. so I was under the impression that the
QEMU notes were completely missing.

That being the case -- we're pretty much done!

I'll put a patch in the next upstream release of crash.

Thanks,
  Dave




> 
> I think "crash" already iterates over all of the notes in the note
> buffer, but skips everything different from NT_PRSTATUS.
> 
> 
> (3) Regarding the structure of the notes, we have to consider the
> placement of the notes and their internal structure. The placement is
> different between the ELF and the KDUMP file format. The internal
> structure of the notes is identical between the two file formats.
> 
> For example, for a 4 VCPU guest, you end up with note names l

Re: [Qemu-devel] [PATCH v5] target-ppc: ppc64 target's virtio can be either endian

[Dave Anderson]

- Original Message -
> On Tue, 22 Apr 2014 09:19:48 -0400 (EDT)
> Dave Anderson  wrote:
> > 
> > 
> > - Original Message -
> > > On Mon, Apr 21, 2014 at 4:48 PM, Greg Kurz 
> > > wrote:
> > > 
> > > > On Mon, 21 Apr 2014 09:56:48 +0200
> > > > Alexander Graf  wrote:
> > > >
> > > > >
> > > > >
> > > > > > Am 21.04.2014 um 06:16 schrieb Bharata B Rao
> > > > > > :
> > > > > >
> > > > > >> On Mon, Apr 14, 2014 at 5:42 PM, Greg Kurz
> > > > > >> 
> > > > wrote:
> > > > > >>
> > > > > >> +
> > > > > >> +#if !defined(CONFIG_USER_ONLY)
> > > > > >> +bool virtio_is_big_endian(void)
> > > > > >> +{
> > > > > >> +PowerPCCPU *cp = POWERPC_CPU(first_cpu);
> > > > > >> +CPUPPCState *env = &cp->env;
> > > > > >> +
> > > > > >> +/* NOTE: booke uses the same number for another unrelated
> > > > > >> spr.
> > > > > >> + */
> > > > > >> +if (strcmp(env->spr_cb[SPR_LPCR].name, "LPCR")) {
> > > > > >> +return TARGET_WORDS_BIGENDIAN;
> > > > > >> +} else {
> > > > > >> +return !(env->spr[SPR_LPCR] & LPCR_ILE);
> > > > > >> +}
> > > > > >> +}
> > > > > >> +#endif
> > > > > >
> > > > > > I am adding crash support for little endian ppc64 guests and I
> > > > realized that the above code needs to be re-used in
> > > > target-ppc/arch_dump.c:cpu_get_dump_info() to set the endianness.
> > > > >
> > > > > Wouldn't it make more sense to treat dumps like gdb and set the
> > > > endianness depending on MSR_LE?
> > > > >
> > > > > Alex
> > > > >
> > > >
> > > > It makes sense to behave the same as gdb... and BTW, since the guest
> > > > may
> > > > change endianness, we could possibly have MSB and LSB data in the dump.
> > > > The
> > > > important thing is to have the possibility to adapt endianness to what
> > > > we
> > > > are
> > > > looking (like set target endian in gdb).
> > > >
> > > 
> > > Typically guest system dump generated by QEMU (via dump-system-memory
> > > monitor cmd) is analysed using the crash utility
> > > (http://people.redhat.com/anderson/).
> > > Crash tool is a wrapper around gdb and in addition to normal gdb
> > > commands,
> > > it supports various other commands which make sense for a system dump.
> > > 
> > > Based on my limited exposure to crash tool, currently crash tool doesn't
> > > support dynamic setting of endianness (like gdb's set endian command) and
> > > it just goes by the endianness recorded in the ELF header and fails if
> > > there is any endian mismatch b/n the dump file and the guest vmlinux
> > > file.
> > > I thought that this is an expected or acceptable behaviour when analyzing
> > > system dumps and not sure if a system dump analyzing tool should adapt to
> > > endianness when analyzing system dumps. Copying Dave Anderson (of crash
> > > utility) for his views here.
> > 
> > Actually the crash utility doesn't compare the dumpfile to the vmlinux file,
> > but rather it will not initialize if the endianness of the vmlinux file or 
> > of
> > an ELF vmcore dumpfile does not match the __BYTE_ORDER compiled into the 
> > crash
> > binary itself.
> > 
> > Dave
> > 
> 
> So if we want to debug PPC64 LE and PPC64 BE, we need two different crash
> binaries. And it is the user call to choose the appropriate one, correct ?

Pretty much, yes...

Dave

> 
> > > 
> > > To answer Alex's earlier question about what do we want to analyze when
> > > we
> > > take a system dump of a big endian powerpc VM while it was running lx86
> > > program, I would think that since this is system dump, we will want to
> > > analyze the system as a whole and hence would need to consider the system
> > > endianness reflected by LPCR_ILE rather than current endianess reflected
> > > by
> > > MSR_LE.
> > > 
> > > Regards,
> > > Bharata.
> > > 
> > 
> 
> 
> 
> --
> Gregory Kurz kurzg...@fr.ibm.com
>  gk...@linux.vnet.ibm.com
> Software Engineer @ IBM/Meiosys  http://www.ibm.com
> Tel +33 (0)562 165 496
> 
> "Anarchy is about taking complete responsibility for yourself."
> Alan Moore.
> 
> 

Re: [Qemu-devel] [PATCH v5] target-ppc: ppc64 target's virtio can be either endian

[Dave Anderson]

- Original Message -
> On Mon, Apr 21, 2014 at 4:48 PM, Greg Kurz  wrote:
> 
> > On Mon, 21 Apr 2014 09:56:48 +0200
> > Alexander Graf  wrote:
> >
> > >
> > >
> > > > Am 21.04.2014 um 06:16 schrieb Bharata B Rao :
> > > >
> > > >> On Mon, Apr 14, 2014 at 5:42 PM, Greg Kurz 
> > wrote:
> > > >>
> > > >> +
> > > >> +#if !defined(CONFIG_USER_ONLY)
> > > >> +bool virtio_is_big_endian(void)
> > > >> +{
> > > >> +PowerPCCPU *cp = POWERPC_CPU(first_cpu);
> > > >> +CPUPPCState *env = &cp->env;
> > > >> +
> > > >> +/* NOTE: booke uses the same number for another unrelated spr.
> > > >> + */
> > > >> +if (strcmp(env->spr_cb[SPR_LPCR].name, "LPCR")) {
> > > >> +return TARGET_WORDS_BIGENDIAN;
> > > >> +} else {
> > > >> +return !(env->spr[SPR_LPCR] & LPCR_ILE);
> > > >> +}
> > > >> +}
> > > >> +#endif
> > > >
> > > > I am adding crash support for little endian ppc64 guests and I
> > realized that the above code needs to be re-used in
> > target-ppc/arch_dump.c:cpu_get_dump_info() to set the endianness.
> > >
> > > Wouldn't it make more sense to treat dumps like gdb and set the
> > endianness depending on MSR_LE?
> > >
> > > Alex
> > >
> >
> > It makes sense to behave the same as gdb... and BTW, since the guest may
> > change endianness, we could possibly have MSB and LSB data in the dump. The
> > important thing is to have the possibility to adapt endianness to what we
> > are
> > looking (like set target endian in gdb).
> >
> 
> Typically guest system dump generated by QEMU (via dump-system-memory
> monitor cmd) is analysed using the crash utility 
> (http://people.redhat.com/anderson/).
> Crash tool is a wrapper around gdb and in addition to normal gdb commands, 
> it supports various other commands which make sense for a system dump.
> 
> Based on my limited exposure to crash tool, currently crash tool doesn't
> support dynamic setting of endianness (like gdb's set endian command) and
> it just goes by the endianness recorded in the ELF header and fails if
> there is any endian mismatch b/n the dump file and the guest vmlinux file.
> I thought that this is an expected or acceptable behaviour when analyzing
> system dumps and not sure if a system dump analyzing tool should adapt to
> endianness when analyzing system dumps. Copying Dave Anderson (of crash
> utility) for his views here.

Actually the crash utility doesn't compare the dumpfile to the vmlinux file, 
but rather it will not initialize if the endianness of the vmlinux file or of 
an ELF vmcore dumpfile does not match the __BYTE_ORDER compiled into the crash
binary itself.

Dave

> 
> To answer Alex's earlier question about what do we want to analyze when we
> take a system dump of a big endian powerpc VM while it was running lx86
> program, I would think that since this is system dump, we will want to
> analyze the system as a whole and hence would need to consider the system
> endianness reflected by LPCR_ILE rather than current endianess reflected by
> MSR_LE.
> 
> Regards,
> Bharata.
> 

Re: [Qemu-devel] dump-guest-memory enhancement.

[Dave Anderson]

- Original Message -

> I haven't been there at the original creation of this functionality, but
> I tend to agree with you. For analyzing the vmcore with gdb or crash,
> the alignment doesn't seem to be important, so it was probably ignored.

With respect to the crash utility, the p_align field is actually used
to differentiate kdump ELF vmcores from the old netdump ELF vmcores.
There are a couple subtle differences between the old netdump ELF vmcores
and kdump ELF vmcores.  Netdump ELF vmcores set the the p_align field to
the architecture page size, whereas kdump ELF vmcores have always had a p_align
field of zero.

Ideally these virsh-dump vmcores would continue to be essentially kdump
clones.  But if it's changed, then there will need to be an update to the
crash utility to work around it.

Dave

Re: [Qemu-devel] [RFC] dump memory when host pci device is used by guest

[Dave Anderson]

02  R15: 0030f8379780
> ORIG_RAX: 0001  CS: 0033  SS: 002b
> crash>
> 
> I wrote a sample(not finished). It only can works on x86_64(both host and 
> guest)
> I use it to create a core file:
> # readelf -h /tmp/vm2.save
> ELF Header:
>   Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>   Class: ELF64
>   Data:  2's complement, little endian
>   Version:   1 (current)
>   OS/ABI:UNIX - System V
>   ABI Version:   0
>   Type:  CORE (Core file)
>   Machine:   Advanced Micro Devices X86-64
>   Version:   0x1
>   Entry point address:   0x0
>   Start of program headers:  64 (bytes into file)
>   Start of section headers:  0 (bytes into file)
>   Flags: 0x0
>   Size of this header:   64 (bytes)
>   Size of program headers:   56 (bytes)
>   Number of program headers: 9
>   Size of section headers:   0 (bytes)
>   Number of section headers: 0
>   Section header string table index: 0
> # readelf -l /tmp/vm2.save
> 
> Elf file type is CORE (Core file)
> Entry point 0x0
> There are 9 program headers, starting at offset 64
> 
> Program Headers:
>   Type   Offset VirtAddr   PhysAddr
>  FileSizMemSiz  Flags  Align
>   NOTE   0x0238 0x 0x
>  0x02c8 0x02c8 0
>   LOAD   0x0500 0x8100 0x0100
>  0x1f00 0x1f00 0
>   LOAD   0x1f000500 0x 0x
>  0x0100 0x0100 0
>   LOAD   0x2500 0x 0x2000
>  0x0002 0x0002 0
>   LOAD   0x20020500 0x 0x2087
>  0x0001 0x0001 0
>   LOAD   0x20030500 0x 0x2085
>  0x0002 0x0002 0
>   LOAD   0x20050500 0x 0x2084
>  0x0001 0x0001 0
>   LOAD   0x20060500 0x 0x2004
>  0x0080 0x0080 0
>   LOAD   0x20860500 0x 0x2002
>  0x0002 0x0002 0
> 
> I can use crash to anaylze the file, but I can not use gdb to anaylze it.
> # gdb /usr/lib/debug/lib/modules/2.6.32-71.el6.x86_64/vmlinux /tmp/vm2.save
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from
> /usr/lib/debug/lib/modules/2.6.32-71.el6.x86_64/vmlinux...done.
> [New ]
> [New ]
> #0  0x8103be8b in ?? ()
> (gdb) bt
> #0  0x8103be8b in ?? ()
> Cannot access memory at address 0x8170dec8
> (gdb) q
> 
> My first and the most important question is that: Is there necessary
> to continue this work?
> 
> The attachment is the sample patch.
> 
> Thanks
> Wen Congyang

>From an enterprise/support point of view, the wholesale replacement
of the current use of the savevm dumpfile format by "virsh dump" with
this ELF style format would be a *huge* improvement. 

Dave Anderson
 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

- Original Message -
> On 10/25/2011 10:23 AM, Avi Kivity wrote:
> > On 10/25/2011 10:06 AM, Wen Congyang wrote:
> >> Hi, Avi Kivity, Dave Anderson
> >>
> >> I have two questions about it:
> >>
> >> 1. How to know the guest's physical base address in qemu?
> >
> > In fact, it's impossible.
> >
> > Perhaps crash can scan through the core looking for a signature.  Need
> > to be careful since multiple signatures may be detected (a kdump kernel,
> > a kernel image in pagecache, or a nested guest).
> 
> Crash uses a heuristic based on the IDT base to identify the base from
> migrate-to-file output.  It also needs CR3 in order to read from virtual
> address.

Just to clarify -- the need for the IDT base and cr3 is *only* true for 
KVM dumps using the "savevm" format.  

With ELF-format kdumps, the phys_base is simply determined from the PT_LOAD
segment that describes the kernel's __START_KERNEL_map region.  I don't know
why that couldn't be done in this proposed case.  
 
Dave

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

- Original Message -

> > > No, an ELF image of the guest's physical memory.
> >
> > Well then that should be pretty straight forward to support.  Depending upon
> > how similar it would be to the "standard" kdump ELF format, the only other
> > issue is how to determine the physical base address at which the kernel is
> > loaded, in order to be able to translate the mapped kernel-text/static-data
> > virtual region of the x86_64 arch (the __START_KERNEL_map region).
> >
> 
> I guess an elf note would work for that?

Right -- here is an example of a RHEL6 ELF kdump header:

$ readelf -a vmcore
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class: ELF64
  Data:  2's complement, little endian
  Version:   1 (current)
  OS/ABI:UNIX - System V
  ABI Version:   0
  Type:  CORE (Core file)
  Machine:   Advanced Micro Devices X86-64
  Version:   0x1
  Entry point address:   0x0
  Start of program headers:  64 (bytes into file)
  Start of section headers:  0 (bytes into file)
  Flags: 0x0
  Size of this header:   64 (bytes)
  Size of program headers:   56 (bytes)
  Number of program headers: 6
  Size of section headers:   0 (bytes)
  Number of section headers: 0
  Section header string table index: 0

There are no sections in this file.

There are no sections in this file.

Program Headers:
  Type   Offset VirtAddr   PhysAddr
 FileSizMemSiz  Flags  Align
  NOTE   0x0190 0x 0x
 0x083c 0x083c 0
  LOAD   0x09cc 0x8100 0x0100
 0x00ba3000 0x00ba3000  RWE0
  LOAD   0x00ba39cc 0x8100 0x
 0x000a 0x000a  RWE0
  LOAD   0x00c439cc 0x8110 0x0010
 0x01f0 0x01f0  RWE0
  LOAD   0x02b439cc 0x81000a00 0x0a00
 0xc5fc2840 0xc5fc2840  RWE0
  LOAD   0xc8b0620c 0x8101 0x0001
 0x3000 0x3000  RWE0

There is no dynamic section in this file.

There are no relocations in this file.

There are no unwind sections in this file.

No version information found in this file.

Notes at offset 0x0190 with length 0x083c:
  Owner Data size   Description
  CORE  0x0150  NT_PRSTATUS (prstatus structure)
  CORE  0x0150  NT_PRSTATUS (prstatus structure)
  VMCOREINFO0x055b  Unknown note type: (0x)
$

In this example, the phys_base (of zero) can be determined by looking 
at the first PT_LOAD segment, and comparing the PhysAddr and the VirtAddr
values -- given that __START_KERNEL_map region is based at 8.
The remaining physical memory chunks are described by the subsequent 
unity-mapped segments.

The NT_PRSTATUS notes are register dumps of each cpu, where this vmcore
was from a 2-cpu system.  But the crash utility is capable of surviving
without them.  It can also get by without the VMCOREINFO note, which is
primarily there for use by the "makedumpfile" utility, which is used to
compress ELF kdumps and filter out unwanted pages, and then make a different 
dumpfile format entirely.

This may be another stupid question -- but does the guest failure mode
render it incapable of using kdump?

Dave

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

- Original Message -
> On 10/24/2011 04:25 PM, Dave Anderson wrote:
> > > The question is that: 'virsh dump' can not be used when host pci device
> > > is used by guest. We are discussing how to fix the problem. We have 
> > > determined
> > > that introduce a new monitor command dump. Jan suggested that the core 
> > > file's
> > > format is gdb standard core format. Does crash support such format?  If 
> > > no,
> > > is it possible to support such format?
> >
> > If you are talking about an ELF core dump of the user-space qemu-kvm process
> > running on the host, then it's certainly not supported.
> 
> No, an ELF image of the guest's physical memory.

Well then that should be pretty straight forward to support.  Depending upon
how similar it would be to the "standard" kdump ELF format, the only other
issue is how to determine the physical base address at which the kernel is
loaded, in order to be able to translate the mapped kernel-text/static-data
virtual region of the x86_64 arch (the __START_KERNEL_map region).
 
> > As to whether it's possible, I suppose it could be done if a 
> > phyical-memory-read
> > function could be created for it, similar to what I asked about for live 
> > analysis
> > of a guest kernel run on/from the KVM host.
> 
> We could use the gdb protocol for this.  Currently it reads/writes
> virtual memory, but we could extend the gdbserver command to make these
> commands apply to physical memory.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

- Original Message -

> The question is that: 'virsh dump' can not be used when host pci device
> is used by guest. We are discussing how to fix the problem. We have determined
> that introduce a new monitor command dump. Jan suggested that the core file's
> format is gdb standard core format. Does crash support such format?  If no,
> is it possible to support such format?

If you are talking about an ELF core dump of the user-space qemu-kvm process
running on the host, then it's certainly not supported.  

As to whether it's possible, I suppose it could be done if a phyical-memory-read
function could be created for it, similar to what I asked about for live 
analysis
of a guest kernel run on/from the KVM host.  

Dave
 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

- Original Message -
> At 10/21/2011 03:11 PM, Jan Kiszka Write:
> > On 2011-10-20 12:03, Wen Congyang wrote:
> >> At 10/20/2011 05:41 PM, Jan Kiszka Write:
> >>> On 2011-10-20 03:22, Wen Congyang wrote:
> >>>>>> I didn't read full story but 'crash' is used for investigating kernel 
> >>>>>> core generated
> >>>>>> by kdump for several years. Considering support service guys, virsh 
> >>>>>> dump should support
> >>>>>> a format for crash because they can't work well at investigating 
> >>>>>> vmcore by gdb.
> >>>>>>
> >>>>>> crash has several functionality useful for them as 'show kerne log', 
> >>>>>> 'focus on a cpu'
> >>>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
> >>>>>>
> >>>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 
> >>>>>> tools for
> >>>>>> investigating kernel dump, it sounds harmful.
> >>>>>
> >>>>> Right, that's why everything (live debugging & crash analysis) should be
> >>>>> consolidated on the long run over gdb. crash is architecturally obsolete
> >>>>> today - not saying it is useless!
> >>>>
> >>>> I do not know why crash is obsoleted today. Is there a new better tool 
> >>>> to instead
> >>>> crash?
> >>>
> >>> I'm not aware of equally powerful (python) scripts for gdb as
> >>> replacement, but I think it's worth starting a porting effort at
> >>> some point.
> >>>
> >>>>
> >>>> At least, I always use crash to live debugging & crash analysis.
> >>>
> >>> Then you may answer some questions to me:
> >>>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?
> >>
> >> No. crash's live debugging only can work the kernel is live. I can use it 
> >> get
> >> some var's value, or some other information from kernel. If kernel panics,
> >> we can use gdb to attach to a remote target as you said. But on end user 
> >> machine,
> >> we can not do it, we should dump the memory into a file and analyze it in 
> >> another
> >> machine while the end user's guest can be restart.
> >>
> >>>  - Can you use it with latest gdb versions or is the gdb functionality
> >>>hard-wired due to an embedded gdb core in crash (that's how I
> >>>understood Christoph's reply to this topic)
> >>
> >> If I use crash, I can not use latest gdb versions. Do we always need to use
> >> the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
> >> is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, 
> >> I
> >> think we can update it and rebuild crash.
> > 
> > crash is simply designed the wrong way around (from today's
> > perspective): it should augment upstream gdb instead of forking it.
> 
> Cc Dave Anderson. He knows how crash uses gdb.
> 
> I think that crash does not fork a task to execute gdb, and gdb is a
> part of crash.

I'm not sure what the question is, but you can consider crash as a huge
wrapper around its embedded gdb, which it invokes as "gdb vmlinux", and
then takes over the user interface.  It doesn't have a clue as to what
the memory source is, i.e., whether it's one of the almost 20 different
dumpfile formats that it supports (including "virsh dump"), or if it's
running against a live system.  It has its own command set, although
you can enter some gdb commands, write gdb scripts, etc.  But the main
purpose of the embedded gdb is for the crash-level sources to be able
to gather data structure information, disassemble text, add-symbol-file
kernel modules, and so on.  There is no kgdb remote linkage. 

It's currently embedding gdb-7.0, although as we speak I'm updating it
to gdb-7.3.1 because the compiler guys have decided that dwarf4 should be
used by default.

It would be kind of cool if there was a "/dev/mem"-like interface
to a KVM guest's physical memory, so that you could sit on a KVM host
and enter "crash vmlinux-of-guest /dev/mem-of-guest" in order to
run live analysis of a guest.

Anyway, sorry if it doesn't meet your needs...

Dave