Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
Sorry for responsing late.
On 01/07/2014 07:38 PM, Laszlo Ersek wrote:
The following fields in dh are left zero-filled:
- timestamp
- total_ram_blocks
- device_blocks
- written_blocks
- current_cpu
I guess we'll either overwrite them later or it's OK to leave them all
zeroed.
Yes, they are leaved all zeroed here. Tools, like crash will get exact data
from dumped memory.
Also... is it OK to write these fields to the file in host native byte
order? What happens if an i686 / x86_64 target is emulated on a BE host?
I will add convert work in v7.
+
+/* write sub header */
+size = sizeof(KdumpSubHeader32);
+kh = g_malloc0(size);
+
+/* 64bit max_mapnr_64 */
+kh-max_mapnr_64 = s-max_mapnr;
+kh-phys_base = PHYS_BASE;
+kh-dump_level = DUMP_LEVEL;
+
+kh-offset_note = DISKDUMP_HEADER_BLOCKS * dh-block_size + size;
+kh-note_size = s-note_size;
+
+if (write_buffer(s-fd, s-flag_flatten, dh-block_size, kh, size) 0)
{
+ret = -1;
+goto out;
+}
- Same question about endianness as above.
- Again, many fields left zeroed in kh, but I guess that's OK.
- I would prefer if you repeated the multiplication by
DISKDUMP_HEADER_BLOCKS verbatim in the offset write_buffer() argument.
write_buffer(s-fd, s-flag_flatten, DISKDUMP_HEADER_BLOCKS * dh-block_size,
kh, size) ?
Yes, I should change it.
- When this write_buffer() is directed to a regular file in non-flat
mode, then the file might become sparse (you jump over a range of
offsets with lseek() in write_buffer()). If the output has been opened
by qemu itself (ie.file:, in qmp_dump_guest_memory()), then due
to the O_TRUNC we can't seek over preexistent data (and keep garbage in
the file). When libvirt pre-opens the file (to send over the fd later),
in doCoreDump(), it also passes O_TRUNC. OK.
Do you mean because of O_TRUNC，seek will exceed the end of the file that may
cause some problem?
--
Regards
Qiao Nuohan
Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
On 01/13/14 11:03, Qiao Nuohan wrote:
Sorry for responsing late.
On 01/07/2014 07:38 PM, Laszlo Ersek wrote:
+kh-offset_note = DISKDUMP_HEADER_BLOCKS * dh-block_size +
size;
+kh-note_size = s-note_size;
+
+if (write_buffer(s-fd, s-flag_flatten, dh-block_size, kh,
size) 0) {
+ret = -1;
+goto out;
+}
- I would prefer if you repeated the multiplication by
DISKDUMP_HEADER_BLOCKS verbatim in the offset write_buffer() argument.
write_buffer(s-fd, s-flag_flatten, DISKDUMP_HEADER_BLOCKS *
dh-block_size,
kh, size) ?
Yes, I should change it.
Yes that's what I meant.
- When this write_buffer() is directed to a regular file in non-flat
mode, then the file might become sparse (you jump over a range of
offsets with lseek() in write_buffer()). If the output has been opened
by qemu itself (ie.file:, in qmp_dump_guest_memory()), then due
to the O_TRUNC we can't seek over preexistent data (and keep garbage in
the file). When libvirt pre-opens the file (to send over the fd later),
in doCoreDump(), it also passes O_TRUNC. OK.
Do you mean because of O_TRUNC，seek will exceed the end of the file
that may cause some problem?
I meant that lseek() would seek over an unwritten portion of the file.
If that portion had any kind of data written into it earlier, then that
data would now likely turn into garbage (lose meaning, become truncated
etc.) It wouldn't be corrupted or anything like that, it would just
become a leftover with potential to cause misinterpretation.
But, since we have O_TRUNC at open() time, we're seeking past the end of
the file, and this sought-over portion will read back as zeroes (and the
file might become sparse, dependent on the filesystem and the size of
the range sought-over).
Seeking past the end of the file is explicitly allowed by POSIX:
The lseek() function shall allow the file offset to be set beyond
the end of the existing data in the file. If data is later written
at this point, subsequent reads of data in the gap shall return
bytes with the value 0 until data is actually written into the gap.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/lseek.html
So this is fine.
Thanks
Laszlo
Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
On 01/13/2014 06:39 PM, Laszlo Ersek wrote: - When this write_buffer() is directed to a regular file in non-flat mode, then the file might become sparse (you jump over a range of offsets with lseek() in write_buffer()). If the output has been opened by qemu itself (ie.file:, in qmp_dump_guest_memory()), then due to the O_TRUNC we can't seek over preexistent data (and keep garbage in the file). When libvirt pre-opens the file (to send over the fd later), in doCoreDump(), it also passes O_TRUNC. OK. Do you mean because of O_TRUNC，seek will exceed the end of the file that may cause some problem? I meant that lseek() would seek over an unwritten portion of the file. If that portion had any kind of data written into it earlier, then that data would now likely turn into garbage (lose meaning, become truncated etc.) It wouldn't be corrupted or anything like that, it would just become a leftover with potential to cause misinterpretation. But, since we have O_TRUNC at open() time, we're seeking past the end of the file, and this sought-over portion will read back as zeroes (and the file might become sparse, dependent on the filesystem and the size of the range sought-over). Seeking past the end of the file is explicitly allowed by POSIX: The lseek() function shall allow the file offset to be set beyond the end of the existing data in the file. If data is later written at this point, subsequent reads of data in the gap shall return bytes with the value 0 until data is actually written into the gap. http://pubs.opengroup.org/onlinepubs/9699919799/functions/lseek.html So this is fine. Thanks for your explanation. I think it would be better to abandon the non-flat mode to avoid potential risk. -- Regards Qiao Nuohan
Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
On 01/14/14 03:07, Qiao Nuohan wrote: On 01/13/2014 06:39 PM, Laszlo Ersek wrote: - When this write_buffer() is directed to a regular file in non-flat mode, then the file might become sparse (you jump over a range of offsets with lseek() in write_buffer()). If the output has been opened by qemu itself (ie.file:, in qmp_dump_guest_memory()), then due to the O_TRUNC we can't seek over preexistent data (and keep garbage in the file). When libvirt pre-opens the file (to send over the fd later), in doCoreDump(), it also passes O_TRUNC. OK. Do you mean because of O_TRUNC，seek will exceed the end of the file that may cause some problem? I meant that lseek() would seek over an unwritten portion of the file. If that portion had any kind of data written into it earlier, then that data would now likely turn into garbage (lose meaning, become truncated etc.) It wouldn't be corrupted or anything like that, it would just become a leftover with potential to cause misinterpretation. But, since we have O_TRUNC at open() time, we're seeking past the end of the file, and this sought-over portion will read back as zeroes (and the file might become sparse, dependent on the filesystem and the size of the range sought-over). Seeking past the end of the file is explicitly allowed by POSIX: The lseek() function shall allow the file offset to be set beyond the end of the existing data in the file. If data is later written at this point, subsequent reads of data in the gap shall return bytes with the value 0 until data is actually written into the gap. http://pubs.opengroup.org/onlinepubs/9699919799/functions/lseek.html So this is fine. Thanks for your explanation. I think it would be better to abandon the non-flat mode to avoid potential risk. I can't really provide any input to that decision -- I have no clue which tools support which format. The non-flat (ie. random-access, regular file) format appears more space- and computation-efficient, and I thought that would be the natural choice. The flat (non-seekable) format was a surprise to me -- I wouldn't have thought that any debugger could directly consume that format. So it's really your call. Again, the lseek()s seemed fine to me on POSIX platforms. Thanks, Laszlo
Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
On 01/14/2014 10:29 AM, Laszlo Ersek wrote: I can't really provide any input to that decision -- I have no clue which tools support which format. The non-flat (ie. random-access, regular file) format appears more space- and computation-efficient, and I thought that would be the natural choice. The flat (non-seekable) format was a surprise to me -- I wouldn't have thought that any debugger could directly consume that format. The flat-mode comes from makedumpfile as kdump-compressed format, and crash utility has already supported it. -- Regards Qiao Nuohan
Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
comments below
On 01/05/14 08:27, Qiao Nuohan wrote:
the functions are used to write header of kdump-compressed format to vmcore.
Header of kdump-compressed format includes:
1. common header: DiskDumpHeader32 / DiskDumpHeader64
2. sub header: KdumpSubHeader32 / KdumpSubHeader64
3. extra information: only elf notes here
Signed-off-by: Qiao Nuohan qiaonuo...@cn.fujitsu.com
---
dump.c| 199
+
include/sysemu/dump.h | 96
2 files changed, 295 insertions(+), 0 deletions(-)
diff --git a/dump.c b/dump.c
index 3b9cf00..e3623b9 100644
--- a/dump.c
+++ b/dump.c
@@ -77,8 +77,16 @@ typedef struct DumpState {
int64_t length;
Error **errp;
+bool flag_flatten;
+uint32_t nr_cpus;
+size_t page_size;
+uint64_t max_mapnr;
+size_t len_dump_bitmap;
void *note_buf;
size_t note_buf_offset;
+off_t offset_dump_bitmap;
+off_t offset_page;
+uint32_t flag_compress;
} DumpState;
static int dump_cleanup(DumpState *s)
@@ -773,6 +781,197 @@ static int buf_write_note(void *buf, size_t size, void
*opaque)
return 0;
}
+/* write common header, sub header and elf note to vmcore */
+static int create_header32(DumpState *s)
+{
+int ret = 0;
+DiskDumpHeader32 *dh = NULL;
+KdumpSubHeader32 *kh = NULL;
+size_t size;
+
+/* write common header, the version of kdump-compressed format is 5th */
I think this is a typo (it should say 6th, shouldn't it?), but it's not
critical.
+size = sizeof(DiskDumpHeader32);
+dh = g_malloc0(size);
+
+strncpy(dh-signature, KDUMP_SIGNATURE, strlen(KDUMP_SIGNATURE));
+dh-header_version = 6;
+dh-block_size = s-page_size;
+dh-sub_hdr_size = sizeof(struct KdumpSubHeader32) + s-note_size;
+dh-sub_hdr_size = DIV_ROUND_UP(dh-sub_hdr_size, dh-block_size);
+/* dh-max_mapnr may be truncated, full 64bit is in kh.max_mapnr_64 */
+dh-max_mapnr = MIN(s-max_mapnr, UINT_MAX);
You could have simply converted / truncated s-max_mapnr to uint32_t
as part of the assignment, but the MIN(..., UINT_MAX) doesn't hurt either.
+dh-nr_cpus = s-nr_cpus;
+dh-bitmap_blocks = DIV_ROUND_UP(s-len_dump_bitmap, dh-block_size) * 2;
+memcpy((dh-utsname.machine), i686, 4);
+
+if (s-flag_compress DUMP_DH_COMPRESSED_ZLIB) {
+dh-status |= DUMP_DH_COMPRESSED_ZLIB;
+}
+#ifdef CONFIG_LZO
+if (s-flag_compress DUMP_DH_COMPRESSED_LZO) {
+dh-status |= DUMP_DH_COMPRESSED_LZO;
+}
+#endif
+#ifdef CONFIG_SNAPPY
+if (s-flag_compress DUMP_DH_COMPRESSED_SNAPPY) {
+dh-status |= DUMP_DH_COMPRESSED_SNAPPY;
+}
+#endif
+
+if (write_buffer(s-fd, s-flag_flatten, 0, dh, size) 0) {
+ret = -1;
+goto out;
+}
The following fields in dh are left zero-filled:
- timestamp
- total_ram_blocks
- device_blocks
- written_blocks
- current_cpu
I guess we'll either overwrite them later or it's OK to leave them all
zeroed.
Also... is it OK to write these fields to the file in host native byte
order? What happens if an i686 / x86_64 target is emulated on a BE host?
+
+/* write sub header */
+size = sizeof(KdumpSubHeader32);
+kh = g_malloc0(size);
+
+/* 64bit max_mapnr_64 */
+kh-max_mapnr_64 = s-max_mapnr;
+kh-phys_base = PHYS_BASE;
+kh-dump_level = DUMP_LEVEL;
+
+kh-offset_note = DISKDUMP_HEADER_BLOCKS * dh-block_size + size;
+kh-note_size = s-note_size;
+
+if (write_buffer(s-fd, s-flag_flatten, dh-block_size, kh, size) 0) {
+ret = -1;
+goto out;
+}
- Same question about endianness as above.
- Again, many fields left zeroed in kh, but I guess that's OK.
- I would prefer if you repeated the multiplication by
DISKDUMP_HEADER_BLOCKS verbatim in the offset write_buffer() argument.
- When this write_buffer() is directed to a regular file in non-flat
mode, then the file might become sparse (you jump over a range of
offsets with lseek() in write_buffer()). If the output has been opened
by qemu itself (ie. file:, in qmp_dump_guest_memory()), then due
to the O_TRUNC we can't seek over preexistent data (and keep garbage in
the file). When libvirt pre-opens the file (to send over the fd later),
in doCoreDump(), it also passes O_TRUNC. OK.
+
+/* write note */
+s-note_buf = g_malloc(s-note_size);
+s-note_buf_offset = 0;
+
+/* use s-note_buf to store notes temporarily */
+if (write_elf32_notes(buf_write_note, s) 0) {
+ret = -1;
+goto out;
+}
+
+if (write_buffer(s-fd, s-flag_flatten, kh-offset_note, s-note_buf,
+ s-note_size) 0) {
+ret = -1;
+goto out;
+}
Right before the write_buffer() call, we know that
s-note_buf_offset = s-note_size
because buf_write_note() ensures it.
We know even that
s-note_buf_offset == s-note_size
there,
Re: [Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
Am 07.01.2014 12:38, schrieb Laszlo Ersek: Also... is it OK to write these fields to the file in host native byte order? What happens if an i686 / x86_64 target is emulated on a BE host? For the target-s390x implementation Alex required to take care of endianness for the s390x-on-x86 case, so it would probably make sense to handle it for kdump format as well. Thanks for bringing it up! BTW my original concerns related to CPU seem to have been addressed, so I'm hoping this can go via the QMP queue once ready. Regards, Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
[Qemu-devel] [PATCH v6 06/11] dump: add API to write dump header
the functions are used to write header of kdump-compressed format to vmcore.
Header of kdump-compressed format includes:
1. common header: DiskDumpHeader32 / DiskDumpHeader64
2. sub header: KdumpSubHeader32 / KdumpSubHeader64
3. extra information: only elf notes here
Signed-off-by: Qiao Nuohan qiaonuo...@cn.fujitsu.com
---
dump.c| 199 +
include/sysemu/dump.h | 96
2 files changed, 295 insertions(+), 0 deletions(-)
diff --git a/dump.c b/dump.c
index 3b9cf00..e3623b9 100644
--- a/dump.c
+++ b/dump.c
@@ -77,8 +77,16 @@ typedef struct DumpState {
int64_t length;
Error **errp;
+bool flag_flatten;
+uint32_t nr_cpus;
+size_t page_size;
+uint64_t max_mapnr;
+size_t len_dump_bitmap;
void *note_buf;
size_t note_buf_offset;
+off_t offset_dump_bitmap;
+off_t offset_page;
+uint32_t flag_compress;
} DumpState;
static int dump_cleanup(DumpState *s)
@@ -773,6 +781,197 @@ static int buf_write_note(void *buf, size_t size, void
*opaque)
return 0;
}
+/* write common header, sub header and elf note to vmcore */
+static int create_header32(DumpState *s)
+{
+int ret = 0;
+DiskDumpHeader32 *dh = NULL;
+KdumpSubHeader32 *kh = NULL;
+size_t size;
+
+/* write common header, the version of kdump-compressed format is 5th */
+size = sizeof(DiskDumpHeader32);
+dh = g_malloc0(size);
+
+strncpy(dh-signature, KDUMP_SIGNATURE, strlen(KDUMP_SIGNATURE));
+dh-header_version = 6;
+dh-block_size = s-page_size;
+dh-sub_hdr_size = sizeof(struct KdumpSubHeader32) + s-note_size;
+dh-sub_hdr_size = DIV_ROUND_UP(dh-sub_hdr_size, dh-block_size);
+/* dh-max_mapnr may be truncated, full 64bit is in kh.max_mapnr_64 */
+dh-max_mapnr = MIN(s-max_mapnr, UINT_MAX);
+dh-nr_cpus = s-nr_cpus;
+dh-bitmap_blocks = DIV_ROUND_UP(s-len_dump_bitmap, dh-block_size) * 2;
+memcpy((dh-utsname.machine), i686, 4);
+
+if (s-flag_compress DUMP_DH_COMPRESSED_ZLIB) {
+dh-status |= DUMP_DH_COMPRESSED_ZLIB;
+}
+#ifdef CONFIG_LZO
+if (s-flag_compress DUMP_DH_COMPRESSED_LZO) {
+dh-status |= DUMP_DH_COMPRESSED_LZO;
+}
+#endif
+#ifdef CONFIG_SNAPPY
+if (s-flag_compress DUMP_DH_COMPRESSED_SNAPPY) {
+dh-status |= DUMP_DH_COMPRESSED_SNAPPY;
+}
+#endif
+
+if (write_buffer(s-fd, s-flag_flatten, 0, dh, size) 0) {
+ret = -1;
+goto out;
+}
+
+/* write sub header */
+size = sizeof(KdumpSubHeader32);
+kh = g_malloc0(size);
+
+/* 64bit max_mapnr_64 */
+kh-max_mapnr_64 = s-max_mapnr;
+kh-phys_base = PHYS_BASE;
+kh-dump_level = DUMP_LEVEL;
+
+kh-offset_note = DISKDUMP_HEADER_BLOCKS * dh-block_size + size;
+kh-note_size = s-note_size;
+
+if (write_buffer(s-fd, s-flag_flatten, dh-block_size, kh, size) 0) {
+ret = -1;
+goto out;
+}
+
+/* write note */
+s-note_buf = g_malloc(s-note_size);
+s-note_buf_offset = 0;
+
+/* use s-note_buf to store notes temporarily */
+if (write_elf32_notes(buf_write_note, s) 0) {
+ret = -1;
+goto out;
+}
+
+if (write_buffer(s-fd, s-flag_flatten, kh-offset_note, s-note_buf,
+ s-note_size) 0) {
+ret = -1;
+goto out;
+}
+
+/* get offset of dump_bitmap */
+s-offset_dump_bitmap = (DISKDUMP_HEADER_BLOCKS + dh-sub_hdr_size) *
+ dh-block_size;
+
+/* get offset of page */
+s-offset_page = (DISKDUMP_HEADER_BLOCKS + dh-sub_hdr_size +
+ dh-bitmap_blocks) * dh-block_size;
+
+out:
+g_free(dh);
+g_free(kh);
+g_free(s-note_buf);
+
+return ret;
+}
+
+/* write common header, sub header and elf note to vmcore */
+static int create_header64(DumpState *s)
+{
+int ret = 0;
+DiskDumpHeader64 *dh = NULL;
+KdumpSubHeader64 *kh = NULL;
+size_t size;
+
+/* write common header, the version of kdump-compressed format is 5th */
+size = sizeof(DiskDumpHeader64);
+dh = g_malloc0(size);
+
+strncpy(dh-signature, KDUMP_SIGNATURE, strlen(KDUMP_SIGNATURE));
+dh-header_version = 6;
+dh-block_size = s-page_size;
+dh-sub_hdr_size = sizeof(struct KdumpSubHeader64) + s-note_size;
+dh-sub_hdr_size = DIV_ROUND_UP(dh-sub_hdr_size, dh-block_size);
+/* dh-max_mapnr may be truncated, full 64bit is in kh.max_mapnr_64 */
+dh-max_mapnr = MIN(s-max_mapnr, UINT_MAX);
+dh-nr_cpus = s-nr_cpus;
+dh-bitmap_blocks = DIV_ROUND_UP(s-len_dump_bitmap, dh-block_size) * 2;
+memcpy((dh-utsname.machine), x86_64, 6);
+
+if (s-flag_compress DUMP_DH_COMPRESSED_ZLIB) {
+dh-status |= DUMP_DH_COMPRESSED_ZLIB;
+}
+#ifdef CONFIG_LZO
+if (s-flag_compress DUMP_DH_COMPRESSED_LZO) {
+dh-status |= DUMP_DH_COMPRESSED_LZO;
+}
+#endif
+#ifdef CONFIG_SNAPPY
+if (s-flag_compress DUMP_DH_COMPRESSED_SNAPPY) {
+
