Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-05-08 Thread Stefan Hajnoczi 
On Tue, Mar 13, 2018 at 04:33:43PM +0800, junyan...@intel.com wrote:
> From: Junyan He 
> 
> The nvdimm size is huge, sometimes it is more than 256G or even more.
> This is a huge burden for snapshot saving. One snapshot point with
> nvdimm may occupy more than 50G disk space even with compression
> enabled.
> We need to introduce dependent snapshot manner to solve this problem.

What is the status of this patch series?

Kevin and Juan: Review is needed from the qcow2 and migration maintainers.

This patch series adds a function to clone qcow2 snapshots (sharing
clusters via the refcount) and then uses it to snapshot NVDIMM contents
incrementally during savevm.  This way only dirty NVDIMM clusters need
to be written into the qcow2 file, saving a lot of space and time.  The
drawback is that dirty memory logging needs to be enabled while the
guest is running.

The approach makes sense to me and could be used for other migration
state beyond NVDIMM in the future.  I think it's worth iterating this
patch series and merging it.

It was difficult for me to understand some of the English.  Do you have
a colleague who can review the English and suggest how to rephrase the
text?

I didn't found the "snapshot dependency" name unclear.  I suggest
calling the new API .bdrv_snapshot_clone().  This makes it clear that a
new snapshot is created based on an existing one.

New APIs are missing doc comments.  Please document the functions,
arguments, and the return values - especially for extern functions
(those declared in header files).

> The first snapshot point should always be saved completely, and enable
> dirty log trace after saving for nvdimm memory region. The later snapshot
> point should add the reference to previous snapshot's nvdimm data and
> just saving dirty pages. This can save a lot of disk and time if the
> snapshot operations are triggered frequently.
> We add save_snapshot_dependency functions to QCOW2 file system firstly, the
> later snapshot will add reference to previous dependent snapshot's data
> cluster. There is an alignment problem here, the dependent data should
> always be cluster aligned. We need to add some padding data when saving
> the snapshot to make it always cluster aligned.
> The logic between nvdimm and ram for snapshot saving is a little confused
> now, we need to exclude nvdimm kind memory region from ram list and the
> dirty log tracing setting is also not very clear. Maybe we can separate the
> snapshot saving from the migration logic later to make code clean.
> In theory, this kind of manner can apply to any kind of memory. But because
> it need to turn dirty log trace on, the performance may decline. So we just
> enable it for nvdimm kind memory firstly.
> 
> Signed-off-by: Junyan He 
> ---
> Makefile.target  |1 +
> block/qcow2-snapshot.c   |  154 ++
> block/qcow2.c|2 +
> block/qcow2.h|7 +
> block/snapshot.c |   45 +++
> exec.c   |7 +
> hw/ppc/spapr.c   |2 +-
> hw/s390x/s390-stattrib.c |2 +-
> include/block/block_int.h|9 ++
> include/block/snapshot.h |7 +
> include/exec/memory.h|9 ++
> include/exec/ram_addr.h  |2 +
> include/migration/misc.h |4 +
> include/migration/register.h |2 +-
> include/migration/snapshot.h |3 +
> memory.c |   18 ++-
> migration/block.c|2 +-
> migration/nvdimm.c   | 1033 
> +
> migration/qemu-file.c|   61 +
> migration/qemu-file.h|   14 ++
> migration/ram.c  |   19 ++-
> migration/savevm.c   |   62 -
> vl.c |1 +
> 23 files changed, 1452 insertions(+), 14 deletions(-)
> 


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-03-15 Thread no-reply 
Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1520930033-18885-1-git-send-email-junyan...@intel.com
Subject: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for 
snapshot.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
856a5e9687 RFC: Enable nvdimm snapshot functions.
4c1adceb28 RFC: Add nvdimm snapshot saving to migration.
424778c4de RFC: Add a section_id parameter to save_live_iterate call.
f9abbe71cd RFC: Add get_current_snapshot_info to get the snapshot state.
f49a5b0838 RFC: Add save dependency functions to qemu_file
adf712a616 RFC: Add memory region snapshot bitmap get function.
3fa9484cb8 RFC: Set memory_region_set_log available for more client.
9d40dbb19e RFC: Implement save and support snapshot dependency in block driver 
layer.
c43e8a1da2 RFC: Implement qcow2's snapshot dependent saving function.
e022968c94 RFC: Add save and support snapshot dependency function to block 
driver.

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-e7u4by_8/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-e7u4by_8/src'
  GEN 
/var/tmp/patchew-tester-tmp-e7u4by_8/src/docker-src.2018-03-15-10.13.12.11256/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-e7u4by_8/src/docker-src.2018-03-15-10.13.12.11256/qemu.tar.vroot'...
done.
Checking out files:  46% (2767/6005)   
Checking out files:  47% (2823/6005)   
Checking out files:  48% (2883/6005)   
Checking out files:  49% (2943/6005)   
Checking out files:  50% (3003/6005)   
Checking out files:  51% (3063/6005)   
Checking out files:  52% (3123/6005)   
Checking out files:  53% (3183/6005)   
Checking out files:  54% (3243/6005)   
Checking out files:  55% (3303/6005)   
Checking out files:  56% (3363/6005)   
Checking out files:  57% (3423/6005)   
Checking out files:  57% (3429/6005)   
Checking out files:  58% (3483/6005)   
Checking out files:  59% (3543/6005)   
Checking out files:  60% (3603/6005)   
Checking out files:  61% (3664/6005)   
Checking out files:  62% (3724/6005)   
Checking out files:  63% (3784/6005)   
Checking out files:  64% (3844/6005)   
Checking out files:  65% (3904/6005)   
Checking out files:  66% (3964/6005)   
Checking out files:  67% (4024/6005)   
Checking out files:  68% (4084/6005)   
Checking out files:  69% (4144/6005)   
Checking out files:  70% (4204/6005)   
Checking out files:  71% (4264/6005)   
Checking out files:  72% (4324/6005)   
Checking out files:  73% (4384/6005)   
Checking out files:  74% (/6005)   
Checking out files:  75% (4504/6005)   
Checking out files:  76% (4564/6005)   
Checking out files:  77% (4624/6005)   
Checking out files:  78% (4684/6005)   
Checking out files:  79% (4744/6005)   
Checking out files:  80% (4804/6005)   
Checking out files:  81% (4865/6005)   
Checking out files:  82% (4925/6005)   
Checking out files:  83% (4985/6005)   
Checking out files:  84% (5045/6005)   
Checking out files:  85% (5105/6005)   
Checking out files:  86% (5165/6005)   
Checking out files:  87% (5225/6005)   
Checking out files:  88% (5285/6005)   
Checking out files:  89% (5345/6005)   
Checking out files:  90% (5405/6005)   
Checking out files:  91% (5465/6005)   
Checking out files:  92% (5525/6005)   
Checking out files:  93% (5585/6005)   
Checking out files:  94% (5645/6005)   
Checking out files:  95% (5705/6005)   
Checking out files:  96% (5765/6005)   
Checking out files:  97% (5825/6005)   
Checking out files:  98% (5885/6005)   
Checking out files:  99% (5945/6005)   
Checking out files: 100% (6005/6005)   
Checking out files: 100% (6005/6005), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-e7u4by_8/src/docker-src.2018-03-15-10.13.12.11256/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-e7u4by_8/src/docker-src.2018-03-15-10.13.12.11256/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-mingw in qemu:fedora 
Packages installed:
PyYAML-3.12-5.fc27.x86_64
SDL-devel-1.2.15-29.fc27.x86_64
bc-1.07.1-3.fc27.x86_64
bison-3.0.4-8.fc27.x86_64
bzip2-1.0.6-24.fc27.x86_64

Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-03-15 Thread no-reply 
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 1520930033-18885-1-git-send-email-junyan...@intel.com
Subject: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for 
snapshot.

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]   
patchew/1520930033-18885-1-git-send-email-junyan...@intel.com -> 
patchew/1520930033-18885-1-git-send-email-junyan...@intel.com
Switched to a new branch 'test'
856a5e9687 RFC: Enable nvdimm snapshot functions.
4c1adceb28 RFC: Add nvdimm snapshot saving to migration.
424778c4de RFC: Add a section_id parameter to save_live_iterate call.
f9abbe71cd RFC: Add get_current_snapshot_info to get the snapshot state.
f49a5b0838 RFC: Add save dependency functions to qemu_file
adf712a616 RFC: Add memory region snapshot bitmap get function.
3fa9484cb8 RFC: Set memory_region_set_log available for more client.
9d40dbb19e RFC: Implement save and support snapshot dependency in block driver 
layer.
c43e8a1da2 RFC: Implement qcow2's snapshot dependent saving function.
e022968c94 RFC: Add save and support snapshot dependency function to block 
driver.

=== OUTPUT BEGIN ===
Checking PATCH 1/10: RFC: Add save and support snapshot dependency function to 
block driver
Checking PATCH 2/10: RFC: Implement qcow2's snapshot dependent saving 
function
Checking PATCH 3/10: RFC: Implement save and support snapshot dependency in 
block driver layer
Checking PATCH 4/10: RFC: Set memory_region_set_log available for more 
client
Checking PATCH 5/10: RFC: Add memory region snapshot bitmap get function
Checking PATCH 6/10: RFC: Add save dependency functions to qemu_file...
Checking PATCH 7/10: RFC: Add get_current_snapshot_info to get the snapshot 
state
Checking PATCH 8/10: RFC: Add a section_id parameter to save_live_iterate 
call
Checking PATCH 9/10: RFC: Add nvdimm snapshot saving to migration
WARNING: line over 80 characters
#122: FILE: migration/nvdimm.c:70:
+

WARNING: line over 80 characters
#123: FILE: migration/nvdimm.c:71:
+| DIRTY_BITMAP_ID | total size | ram name size | ram name | ram size | bitmap 
size |

WARNING: line over 80 characters
#124: FILE: migration/nvdimm.c:72:
+

WARNING: line over 80 characters
#129: FILE: migration/nvdimm.c:77:
+---

WARNING: line over 80 characters
#130: FILE: migration/nvdimm.c:78:
+| DATA_ID | size | ram name size | ram name | ram size | data size | data... | 
END_ID |

WARNING: line over 80 characters
#131: FILE: migration/nvdimm.c:79:
+---

ERROR: do not use C99 // comments
#243: FILE: migration/nvdimm.c:191:
+padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_PADDING_ID

ERROR: do not use C99 // comments
#244: FILE: migration/nvdimm.c:192:
+padding_sz -= sizeof(int32_t); // NVDIMM_PADDING_BYTE size

ERROR: do not use C99 // comments
#245: FILE: migration/nvdimm.c:193:
+padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_END_ID

ERROR: do not use C99 // comments
#344: FILE: migration/nvdimm.c:292:
+data_sz += sizeof(int); // Zero page, just a ID

ERROR: do not use C99 // comments
#346: FILE: migration/nvdimm.c:294:
+data_sz += ((1 << TARGET_PAGE_BITS) + sizeof(int)); // ID + page

ERROR: do not use C99 // comments
#350: FILE: migration/nvdimm.c:298:
+total_sz = sizeof(unsigned int); // NVDIMM_SECTION_DIRTY_BITMAP_ID

ERROR: do not use C99 // comments
#351: FILE: migration/nvdimm.c:299:
+total_sz += sizeof(uint64_t);// the total size itself

ERROR: do not use C99 // comments
#352: FILE: migration/nvdimm.c:300:
+total_sz += sizeof(int); // ram name size

ERROR: do not use C99 // comments
#355: FILE: migration/nvdimm.c:303:
+total_sz += sizeof(uint64_t); // ram size

ERROR: do not use C99 // comments
#356: FILE: migration/nvdimm.c:304:
+total_sz += sizeof(uint64_t); // data size

ERROR: do not use C99 // comments
#358: FILE: migration/nvdimm.c:306:
+total_sz += sizeof(unsigned int); // NVDIMM_SECTION_END_ID

WARNING: line

[Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-03-15 Thread junyan . he 
From: Junyan He 

The nvdimm size is huge, sometimes it is more than 256G or even more.
This is a huge burden for snapshot saving. One snapshot point with
nvdimm may occupy more than 50G disk space even with compression
enabled.
We need to introduce dependent snapshot manner to solve this problem.
The first snapshot point should always be saved completely, and enable
dirty log trace after saving for nvdimm memory region. The later snapshot
point should add the reference to previous snapshot's nvdimm data and
just saving dirty pages. This can save a lot of disk and time if the
snapshot operations are triggered frequently.
We add save_snapshot_dependency functions to QCOW2 file system firstly, the
later snapshot will add reference to previous dependent snapshot's data
cluster. There is an alignment problem here, the dependent data should
always be cluster aligned. We need to add some padding data when saving
the snapshot to make it always cluster aligned.
The logic between nvdimm and ram for snapshot saving is a little confused
now, we need to exclude nvdimm kind memory region from ram list and the
dirty log tracing setting is also not very clear. Maybe we can separate the
snapshot saving from the migration logic later to make code clean.
In theory, this kind of manner can apply to any kind of memory. But because
it need to turn dirty log trace on, the performance may decline. So we just
enable it for nvdimm kind memory firstly.

Signed-off-by: Junyan He 
---
Makefile.target  |1 +
block/qcow2-snapshot.c   |  154 ++
block/qcow2.c|2 +
block/qcow2.h|7 +
block/snapshot.c |   45 +++
exec.c   |7 +
hw/ppc/spapr.c   |2 +-
hw/s390x/s390-stattrib.c |2 +-
include/block/block_int.h|9 ++
include/block/snapshot.h |7 +
include/exec/memory.h|9 ++
include/exec/ram_addr.h  |2 +
include/migration/misc.h |4 +
include/migration/register.h |2 +-
include/migration/snapshot.h |3 +
memory.c |   18 ++-
migration/block.c|2 +-
migration/nvdimm.c   | 1033 
+
migration/qemu-file.c|   61 +
migration/qemu-file.h|   14 ++
migration/ram.c  |   19 ++-
migration/savevm.c   |   62 -
vl.c |1 +
23 files changed, 1452 insertions(+), 14 deletions(-)

Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-03-13 Thread no-reply 
Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1520990418-28258-1-git-send-email-junyan...@hotmail.com
Subject: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for 
snapshot.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
cf8b49f484 RFC: Enable nvdimm snapshot functions.
839b0dc7fc RFC: Add nvdimm snapshot saving to migration.
edf96a743e RFC: Add a section_id parameter to save_live_iterate call.
eaaf08deb8 RFC: Add get_current_snapshot_info to get the snapshot state.
ac55e78d2c RFC: Add save dependency functions to qemu_file
99deb40575 RFC: Add memory region snapshot bitmap get function.
eb7ccdaf28 RFC: Set memory_region_set_log available for more client.
9ee8792a3b RFC: Implement save and support snapshot dependency in block driver 
layer.
ebadb43e47 RFC: Implement qcow2's snapshot dependent saving function.
1a29421543 RFC: Add save and support snapshot dependency function to block 
driver.

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-fe8degv8/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-fe8degv8/src'
  GEN 
/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar.vroot'...
done.
Checking out files:  47% (2826/6005)   
Checking out files:  48% (2883/6005)   
Checking out files:  49% (2943/6005)   
Checking out files:  50% (3003/6005)   
Checking out files:  51% (3063/6005)   
Checking out files:  52% (3123/6005)   
Checking out files:  53% (3183/6005)   
Checking out files:  54% (3243/6005)   
Checking out files:  55% (3303/6005)   
Checking out files:  56% (3363/6005)   
Checking out files:  57% (3423/6005)   
Checking out files:  58% (3483/6005)   
Checking out files:  59% (3543/6005)   
Checking out files:  60% (3603/6005)   
Checking out files:  61% (3664/6005)   
Checking out files:  62% (3724/6005)   
Checking out files:  63% (3784/6005)   
Checking out files:  64% (3844/6005)   
Checking out files:  65% (3904/6005)   
Checking out files:  66% (3964/6005)   
Checking out files:  67% (4024/6005)   
Checking out files:  68% (4084/6005)   
Checking out files:  69% (4144/6005)   
Checking out files:  70% (4204/6005)   
Checking out files:  71% (4264/6005)   
Checking out files:  72% (4324/6005)   
Checking out files:  73% (4384/6005)   
Checking out files:  74% (/6005)   
Checking out files:  75% (4504/6005)   
Checking out files:  76% (4564/6005)   
Checking out files:  77% (4624/6005)   
Checking out files:  78% (4684/6005)   
Checking out files:  79% (4744/6005)   
Checking out files:  80% (4804/6005)   
Checking out files:  81% (4865/6005)   
Checking out files:  82% (4925/6005)   
Checking out files:  83% (4985/6005)   
Checking out files:  84% (5045/6005)   
Checking out files:  85% (5105/6005)   
Checking out files:  86% (5165/6005)   
Checking out files:  87% (5225/6005)   
Checking out files:  88% (5285/6005)   
Checking out files:  89% (5345/6005)   
Checking out files:  90% (5405/6005)   
Checking out files:  91% (5465/6005)   
Checking out files:  92% (5525/6005)   
Checking out files:  93% (5585/6005)   
Checking out files:  94% (5645/6005)   
Checking out files:  95% (5705/6005)   
Checking out files:  96% (5765/6005)   
Checking out files:  97% (5825/6005)   
Checking out files:  98% (5885/6005)   
Checking out files:  99% (5945/6005)   
Checking out files: 100% (6005/6005)   
Checking out files: 100% (6005/6005), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-mingw in qemu:fedora 
Packages installed:
PyYAML-3.12-5.fc27.x86_64
SDL-devel-1.2.15-29.fc27.x86_64
bc-1.07.1-3.fc27.x86_64
bison-3.0.4-8.fc27.x86_64
bzip2-1.0.6-24.fc27.x86_64
ccache-3.3.6-1.fc27.x86_64
clang-5.0.1-3.fc27.x86_64
findutils-4.6.0-16.fc27

Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-03-13 Thread no-reply 
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 1520990418-28258-1-git-send-email-junyan...@hotmail.com
Subject: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for 
snapshot.

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]   
patchew/1520990418-28258-1-git-send-email-junyan...@hotmail.com -> 
patchew/1520990418-28258-1-git-send-email-junyan...@hotmail.com
Switched to a new branch 'test'
cf8b49f484 RFC: Enable nvdimm snapshot functions.
839b0dc7fc RFC: Add nvdimm snapshot saving to migration.
edf96a743e RFC: Add a section_id parameter to save_live_iterate call.
eaaf08deb8 RFC: Add get_current_snapshot_info to get the snapshot state.
ac55e78d2c RFC: Add save dependency functions to qemu_file
99deb40575 RFC: Add memory region snapshot bitmap get function.
eb7ccdaf28 RFC: Set memory_region_set_log available for more client.
9ee8792a3b RFC: Implement save and support snapshot dependency in block driver 
layer.
ebadb43e47 RFC: Implement qcow2's snapshot dependent saving function.
1a29421543 RFC: Add save and support snapshot dependency function to block 
driver.

=== OUTPUT BEGIN ===
Checking PATCH 1/10: RFC: Add save and support snapshot dependency function to 
block driver
Checking PATCH 2/10: RFC: Implement qcow2's snapshot dependent saving 
function
Checking PATCH 3/10: RFC: Implement save and support snapshot dependency in 
block driver layer
Checking PATCH 4/10: RFC: Set memory_region_set_log available for more 
client
Checking PATCH 5/10: RFC: Add memory region snapshot bitmap get function
Checking PATCH 6/10: RFC: Add save dependency functions to qemu_file...
Checking PATCH 7/10: RFC: Add get_current_snapshot_info to get the snapshot 
state
Checking PATCH 8/10: RFC: Add a section_id parameter to save_live_iterate 
call
Checking PATCH 9/10: RFC: Add nvdimm snapshot saving to migration
WARNING: line over 80 characters
#122: FILE: migration/nvdimm.c:70:
+

WARNING: line over 80 characters
#123: FILE: migration/nvdimm.c:71:
+| DIRTY_BITMAP_ID | total size | ram name size | ram name | ram size | bitmap 
size |

WARNING: line over 80 characters
#124: FILE: migration/nvdimm.c:72:
+

WARNING: line over 80 characters
#129: FILE: migration/nvdimm.c:77:
+---

WARNING: line over 80 characters
#130: FILE: migration/nvdimm.c:78:
+| DATA_ID | size | ram name size | ram name | ram size | data size | data... | 
END_ID |

WARNING: line over 80 characters
#131: FILE: migration/nvdimm.c:79:
+---

ERROR: do not use C99 // comments
#243: FILE: migration/nvdimm.c:191:
+padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_PADDING_ID

ERROR: do not use C99 // comments
#244: FILE: migration/nvdimm.c:192:
+padding_sz -= sizeof(int32_t); // NVDIMM_PADDING_BYTE size

ERROR: do not use C99 // comments
#245: FILE: migration/nvdimm.c:193:
+padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_END_ID

ERROR: do not use C99 // comments
#344: FILE: migration/nvdimm.c:292:
+data_sz += sizeof(int); // Zero page, just a ID

ERROR: do not use C99 // comments
#346: FILE: migration/nvdimm.c:294:
+data_sz += ((1 << TARGET_PAGE_BITS) + sizeof(int)); // ID + page

ERROR: do not use C99 // comments
#350: FILE: migration/nvdimm.c:298:
+total_sz = sizeof(unsigned int); // NVDIMM_SECTION_DIRTY_BITMAP_ID

ERROR: do not use C99 // comments
#351: FILE: migration/nvdimm.c:299:
+total_sz += sizeof(uint64_t);// the total size itself

ERROR: do not use C99 // comments
#352: FILE: migration/nvdimm.c:300:
+total_sz += sizeof(int); // ram name size

ERROR: do not use C99 // comments
#355: FILE: migration/nvdimm.c:303:
+total_sz += sizeof(uint64_t); // ram size

ERROR: do not use C99 // comments
#356: FILE: migration/nvdimm.c:304:
+total_sz += sizeof(uint64_t); // data size

ERROR: do not use C99 // comments
#358: FILE: migration/nvdimm.c:306:
+total_sz += sizeof(unsigned int); // NVDIMM_SECTION_END_ID

WARNING:

[Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

2018-03-13 Thread junyan . he 
From: Junyan He 

The nvdimm size is huge, sometimes it is more than 256G or even more.
This is a huge burden for snapshot saving. One snapshot point with
nvdimm may occupy more than 50G disk space even with compression
enabled.
We need to introduce dependent snapshot manner to solve this problem.
The first snapshot point should always be saved completely, and enable
dirty log trace after saving for nvdimm memory region. The later snapshot
point should add the reference to previous snapshot's nvdimm data and
just saving dirty pages. This can save a lot of disk and time if the
snapshot operations are triggered frequently.
We add save_snapshot_dependency functions to QCOW2 file system firstly, the
later snapshot will add reference to previous dependent snapshot's data
cluster. There is an alignment problem here, the dependent data should
always be cluster aligned. We need to add some padding data when saving
the snapshot to make it always cluster aligned.
The logic between nvdimm and ram for snapshot saving is a little confused
now, we need to exclude nvdimm kind memory region from ram list and the
dirty log tracing setting is also not very clear. Maybe we can separate the
snapshot saving from the migration logic later to make code clean.
In theory, this kind of manner can apply to any kind of memory. But because
it need to turn dirty log trace on, the performance may decline. So we just
enable it for nvdimm kind memory firstly.

Signed-off-by: Junyan He 
---
Makefile.target  |1 +
block/qcow2-snapshot.c   |  154 ++
block/qcow2.c|2 +
block/qcow2.h|7 +
block/snapshot.c |   45 +++
exec.c   |7 +
hw/ppc/spapr.c   |2 +-
hw/s390x/s390-stattrib.c |2 +-
include/block/block_int.h|9 ++
include/block/snapshot.h |7 +
include/exec/memory.h|9 ++
include/exec/ram_addr.h  |2 +
include/migration/misc.h |4 +
include/migration/register.h |2 +-
include/migration/snapshot.h |3 +
memory.c |   18 ++-
migration/block.c|2 +-
migration/nvdimm.c   | 1033 
+
migration/qemu-file.c|   61 +
migration/qemu-file.h|   14 ++
migration/ram.c  |   19 ++-
migration/savevm.c   |   62 -
vl.c |1 +
23 files changed, 1452 insertions(+), 14 deletions(-)