[Qemu-devel] [PATCH v3 15/47] qemu_loadvm errors and debug

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Flip many fprintf's to error_report Add lots of DPRINTF debug in qemu_loadvm* Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 23 +-- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git

[Qemu-devel] [PATCH v3 19/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add state variable showing current incoming postcopy state. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 8 ++ include/sysemu/sysemu.h | 19 +++ savevm.c | 324

[Qemu-devel] [PATCH v3 47/47] End of migration for postcopy

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Tweak the end of migration cleanup; we don't want to close stuff down at the end of the main stream, since the postcopy is still sending pages on the other thread. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration.c | 23

[Qemu-devel] [PATCH v3 46/47] postcopy: Wire up loadvm_postcopy_ram_handle_{run, end} commands

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Wire up more of the handlers for the commands on the destination side, in particular loadvm_postcopy_ram_handle_run now has enough to start the guest running. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 63

[Qemu-devel] [PATCH v3 33/47] Postcopy: Rework migration thread for postcopy mode

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Switch to postcopy if: 1) There's still a significant amount to transfer 2) Postcopy is enabled 3) migrate_postcopy_start has been issued. and change the cleanup at the end of migration to match. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v3 45/47] Start up a postcopy/listener thread ready for incoming page data

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code

[Qemu-devel] [PATCH v3 34/47] mig fd_connect: open return path

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Open the return path before migration thread creation. Since this can fail, guard the fd cleanup so it doesn't try and destroy the potentially non-existent thread. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v3 32/47] Postcopy: postcopy_start

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com postcopy_start: Perform all the initialisation associated with starting up postcopy mode from the source. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration.c | 85

[Qemu-devel] [PATCH v3 38/47] Page request: Consume pages off the post-copy queue

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning. Note: a) After a queued page the linear walk carries on from after the unqueued page; there is a

[Qemu-devel] [PATCH v3 40/47] postcopy_ram.c: place_page and helpers

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com postcopy_place_page (etc) provide a way for postcopy to place a page into guests memory atomically (using the new remap_anon_pages syscall). Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h| 1 +

[Qemu-devel] [PATCH v3 43/47] Don't sync dirty bitmaps in postcopy

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Once we're in postcopy the source processors are stopped and memory shouldn't change any more, so there's no need to look at the dirty map. There are two notes to this: 1) If we do resync and a page had changed then the page would get sent

[Qemu-devel] [PATCH v3 44/47] Postcopy; Handle userfault requests

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com userfaultfd is a Linux syscall that gives an fd that receives a stream of notifications of accesses to pages marked as MADV_USERFAULT, and allows the program to acknowledge those stalls and tell the accessing thread to carry on. Signed-off-by: Dr.

[Qemu-devel] [PATCH v3 35/47] Postcopy: Create a fault handler thread before marking the ram as userfault

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 3 +++ postcopy-ram.c| 23 +++ 2 files changed, 26 insertions(+) diff --git a/include/migration/migration.h

[Qemu-devel] [PATCH v3 30/47] postcopy: Incoming initialisation

2014-08-28 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 11 +++ include/migration/migration.h | 1 + migration.c | 1 + 3 files changed, 13 insertions(+) diff --git a/arch_init.c

[Qemu-devel] [PATCH 1/1] xhci PCIe endpoint migration compatibility fix

2014-09-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add back the PCIe config capabilities on XHCI cards in non-PCIe slots, but only for machine types before 2.1. This fixes a migration incompatibility in the XHCI PCI devices caused by: 058fdcf52cdbf57b67e7 - xhci: add endpoint cap on express bus

[Qemu-devel] [PATCH v3 0/2] In memory QEMUFile

2014-09-16 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Hi, This patch-pair adds the QEMUSizedBuffer based in-memory QEMUFile written by Stefan Berger and Joel Schopp. I've made some fixes and modified the existing test-vmstate to use it for some test cases. While there's nothing other than test

[Qemu-devel] [PATCH v3 2/2] Tests: QEMUSizedBuffer/QEMUBuffer

2014-09-16 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type. Only some of the tests are changed so that the fd backed QEMUFile is

[Qemu-devel] [PATCH v3 1/2] QEMUSizedBuffer based QEMUFile

2014-09-16 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from: http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html Using the QEMUFile interface, this patch adds support functions for operating

[Qemu-devel] [PATCH v4 2/2] Tests: QEMUSizedBuffer/QEMUBuffer

2014-09-17 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type. Only some of the tests are changed so that the fd backed QEMUFile is

[Qemu-devel] [PATCH v4 0/2] In memory QEMUFile

2014-09-17 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Hi, This patch-pair adds the QEMUSizedBuffer based in-memory QEMUFile written by Stefan Berger and Joel Schopp. I've made some fixes and modified the existing test-vmstate to use it for some test cases. While there's nothing other than test

[Qemu-devel] [PATCH v4 1/2] QEMUSizedBuffer based QEMUFile

2014-09-17 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from: http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html Using the QEMUFile interface, this patch adds support functions for operating

[Qemu-devel] [PATCH v5 0/2] In memory QEMUFile

2014-09-29 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This patch-pair adds the QEMUSizedBuffer based in-memory QEMUFile written by Stefan Berger and Joel Schopp. I've made some fixes and modified the existing test-vmstate to use it for some test cases. While there's nothing other than test cases

[Qemu-devel] [PATCH v5 1/2] QEMUSizedBuffer based QEMUFile

2014-09-29 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from: http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html Using the QEMUFile interface, this patch adds support functions for operating

[Qemu-devel] [PATCH v5 2/2] Tests: QEMUSizedBuffer/QEMUBuffer

2014-09-29 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type. Only some of the tests are changed so that the fd backed QEMUFile is

[Qemu-devel] [PATCH v4 00/47] Postcopy implementation

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Hi, This is the 4th cut of my version of postcopy; it is designed for use with the Linux kernel additions just posted by Andrea Arcangeli here: http://marc.info/?l=linux-kernelm=141235633015100w=2 (Note: This is a new version compared to my

[Qemu-devel] [PATCH v4 02/47] Tests: QEMUSizedBuffer/QEMUBuffer

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com * Please comment on separate thread for this QEMUSziedBuffer patch * Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type.

[Qemu-devel] [PATCH v4 06/47] Add qemu_get_counted_string to read a string prefixed by a count byte

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com and use it in loadvm_state. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/qemu-file.h | 2 ++ qemu-file.c | 15 +++ savevm.c | 18 ++ 3 files

[Qemu-devel] [PATCH v4 01/47] QEMUSizedBuffer based QEMUFile

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com * Please comment on separate thread for this QEMUSizedBuffer patch * This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from: http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html Using

[Qemu-devel] [PATCH v4 03/47] Start documenting how postcopy works.

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- docs/migration.txt | 189 + 1 file changed, 189 insertions(+) diff --git a/docs/migration.txt b/docs/migration.txt index

[Qemu-devel] [PATCH v4 05/47] improve DPRINTF macros, add to savevm

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Improve the existing DPRINTF macros in migration.c and arch_init by: 1) Making them go to stderr rather than stdout (so you can run with -nographic and redirect your debug to a file) 2) Making them print the ms time with each debug - useful for

[Qemu-devel] [PATCH v4 07/47] Create MigrationIncomingState

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together. allocate MIS in process_incoming_migration_co Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v4 09/47] Provide runtime Target page information

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init. Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to

[Qemu-devel] [PATCH v4 04/47] qemu_ram_foreach_block: pass up error value, and down the ramblock name

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller, and __qemu_rdma_add_block the only function it calls using it. Pass the name of the ramblock to the function;

[Qemu-devel] [PATCH v4 08/47] socket shutdown

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add QEMUFile interface to allow a socket to be 'shut down' - i.e. any reads/writes will fail (and any blocking read/write will be woken). Add qemu_socket wrapper to let OS dependencies be extracted out. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'. Wire it up for 'socket' QEMUFile's using a dup'd fd. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v4 13/47] Migration commands

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process. For use in postcopy. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v4 14/47] Return path: Control commands

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add two src-dest commands: * OPENRP - To request that the destination open the return path * REQACK - Request an acknowledge from the destination Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h |

[Qemu-devel] [PATCH v4 15/47] Return path: Send responses from destination to source

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add migrate_send_rp_message to send a message from destination to source along the return path. (It uses a mutex to let it be called from multiple threads) Add migrate_send_rp_shut to send a 'shut' message to indicate the destination is

[Qemu-devel] [PATCH v4 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add state variable showing current incoming postcopy state. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 8 + include/sysemu/sysemu.h | 20 +++ savevm.c | 335

[Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Open a return path, and handle messages that are received upon it. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 10 +++ migration.c | 181

[Qemu-devel] [PATCH v4 18/47] ram_debug_dump_bitmap: Dump a migration bitmap as text

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 39

[Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 52 +++ include/migration/migration.h |

[Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com QEMU_VM_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream. This is used by postcopy to load

[Qemu-devel] [PATCH v4 17/47] qemu_loadvm errors and debug

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Flip many fprintf's to error_report Add lots of DPRINTF debug in qemu_loadvm* Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 23 +-- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git

[Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 7 +++ include/migration/vmstate.h | 2 +-

[Qemu-devel] [PATCH v4 38/47] Add assertion to check migration_dirty_pages

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com I've seen it go negative once during dev, it shouldn't happen. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch_init.c b/arch_init.c index a945990..2f4345a 100644

[Qemu-devel] [PATCH v4 26/47] migrate_start_postcopy: Command to trigger transition to postcopy

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the: migrate_start_postcopy command must be issued. Postcopy will start sometime after this (when it's

[Qemu-devel] [PATCH v4 43/47] Host page!=target page: Cleanup bitmaps

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Prior to the start of postcopy, ensure that everything that will be transferred later is a whole host-page in size. This is accomplished by discarding partially transferred host pages and marking any that are partially dirty as fully dirty.

[Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions. Split the core of qemu_loadvm_state out so we can use it for both. Allow the inner

[Qemu-devel] [PATCH v4 31/47] postcopy: Incoming initialisation

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 11 include/migration/migration.h| 1 + include/migration/postcopy-ram.h | 12 + migration.c | 1 +

[Qemu-devel] [PATCH v4 27/47] MIG_STATE_POSTCOPY_ACTIVE: Add new migration state

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com 'MIG_STATE_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy 'migration_postcopy_phase' is provided for other sections to know if they're in postcopy. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v4 20/47] Add migration-capability boolean for postcopy-ram.

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- include/migration/migration.h | 1 + migration.c | 9 + qapi-schema.json | 6 +- 3 files changed, 15

[Qemu-devel] [PATCH v4 45/47] Start up a postcopy/listener thread ready for incoming page data

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code

[Qemu-devel] [PATCH v4 30/47] Postcopy: Maintain sentmap and calculate discard

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs. Maintain a

[Qemu-devel] [PATCH v4 37/47] Page request: Consume pages off the post-copy queue

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning. Note: a) After a queued page the linear walk carries on from after the unqueued page; there is a

[Qemu-devel] [PATCH v4 47/47] End of migration for postcopy

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Tweak the end of migration cleanup; we don't want to close stuff down at the end of the main stream, since the postcopy is still sending pages on the other thread. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration.c | 23

[Qemu-devel] [PATCH v4 23/47] migrate_init: Call from savevm

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v4 40/47] Postcopy: Use helpers to map pages during migration

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com In postcopy, the destination guest is running at the same time as it's receiving pages; as we receive new pages we must put them into the guests address space atomically to avoid a running CPU accessing a partially written page. Use the helpers in

[Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add MIG_RPCOMM_REQPAGES command on Return path for the postcopy destination to request a page from the source. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 3 ++ migration.c | 74

[Qemu-devel] [PATCH v4 25/47] postcopy: OS support test

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Provide a check to see if the OS we're running on has all the bits needed for postcopy. Creates postcopy-ram.c which will get most of the other helpers we need. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- Makefile.objs

[Qemu-devel] [PATCH v4 41/47] qemu_ram_block_from_host

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel. qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset

[Qemu-devel] [PATCH v4 28/47] qemu_savevm_state_complete: Postcopy changes

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip: a) Finishing postcopiable iterative devices - they'll carry on b) The termination byte on the end of the stream. We then also add:

[Qemu-devel] [PATCH v4 44/47] Postcopy; Handle userfault requests

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com userfaultfd is a Linux syscall that gives an fd that receives a stream of notifications of accesses to pages marked as MADV_USERFAULT, and allows the program to acknowledge those stalls and tell the accessing thread to carry on. Signed-off-by: Dr.

[Qemu-devel] [PATCH v4 29/47] Postcopy page-map-incoming (PMI) structure

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h| 2 ++ include/migration/postcopy-ram.h | 6 + postcopy-ram.c | 49 +++- savevm.c

[Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Rework the migration thread to setup and start postcopy. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 3 + migration.c | 201 ++ 2 files

[Qemu-devel] [PATCH v4 42/47] Don't sync dirty bitmaps in postcopy

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Once we're in postcopy the source processors are stopped and memory shouldn't change any more, so there's no need to look at the dirty map. There are two notes to this: 1) If we do resync and a page had changed then the page would get sent

[Qemu-devel] [PATCH v4 46/47] postcopy: Wire up loadvm_postcopy_ram_handle_{run, end} commands

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Wire up more of the handlers for the commands on the destination side, in particular loadvm_postcopy_ram_handle_run now has enough to start the guest running. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 63

[Qemu-devel] [PATCH v4 34/47] Postcopy: Create a fault handler thread before marking the ram as userfault

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 3 +++ postcopy-ram.c| 23 +++ 2 files changed, 26 insertions(+) diff --git a/include/migration/migration.h

[Qemu-devel] [PATCH v4 39/47] postcopy_ram.c: place_page and helpers

2014-10-03 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com postcopy_place_page (etc) provide a way for postcopy to place a page into guests memory atomically (using the new remap_anon_pages syscall). Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h| 2 +

[Qemu-devel] [PATCH] Tracing docs fix configure option and description

2014-10-07 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Fix the example trace configure option. Update the text to say that multiple backends are allowed and what happens when multiple backends are enabled. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- docs/tracing.txt | 6 +++--- 1

[Qemu-devel] [PATCH v6 2/2] Tests: QEMUSizedBuffer/QEMUBuffer

2014-10-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type. Only some of the tests are changed so that the fd backed QEMUFile is

[Qemu-devel] [PATCH v6 1/2] QEMUSizedBuffer based QEMUFile

2014-10-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from: http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html Using the QEMUFile interface, this patch adds support functions for operating

[Qemu-devel] [PATCH v6 0/2] In memory QEMUFile

2014-10-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This patch-pair adds the QEMUSizedBuffer based in-memory QEMUFile written by Stefan Berger and Joel Schopp. I've made some fixes and modified the existing test-vmstate to use it for some test cases. While there's nothing other than test cases

[Qemu-devel] [PATCH 1/1] omap_gpmc.c: Remove duplicate assignment

2014-10-23 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This looks like an old merge error and should have no effect. (Build tested only) Found by Coccinelle using Julia Lawall's script: https://lkml.org/lkml/2014/8/23/128 Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v2 1/4] Start migrating migration code into a migration directory

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The migration code now occupies a fair chunk of the top level .c files, it seems time to give it it's own directory. I've not touched: arch_init.c - that's mostly RAM migration but has a few random other bits savevm.c-

[Qemu-devel] [PATCH v2 2/4] Remove migration- pre/post fixes off files in migration/ dir

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The general feeling is that having migration/migration-blah is overkill. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/Makefile.objs | 8 migration/{block-migration.c = block.c} | 0

[Qemu-devel] [PATCH v2 0/4] More migration file cleanup

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is a small set of patches that shuffle migration code around, but doesn't change the behaviour: 1) Move a lot of the migration source into a separate 'migration' directory Note this moves a lot of files around, git format-patch -M -B

[Qemu-devel] [PATCH v2 3/4] Split struct QEMUFile out

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Now we've got multiple QEMUFile source files, some of them need access to things that were defined in qemu-file.c, so create a -internal header for them. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/qemu-file-internal.h

[Qemu-devel] [PATCH v2 4/4] Split the QEMU buffered file code out

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The splitting of qemu-file and addition of the buffered file landed at the same time; so now split the buffered file code out. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/Makefile.objs | 2 +-

[Qemu-devel] [PATCH 0/2] ATAPI migration fix/hack

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This pair of patches fixes a problem where IDE/ATAPI cdrom reads get lost/corrupted over migration. The first of the patches (restore atapi_dma flag) is a simple fix that I think is safe; it no longer causes corruption in the case we saw, but does

[Qemu-devel] [PATCH 2/2] atapi migration: Throw recoverable error to avoid recovery

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com (With the previous atapi_dma flag recovery) If migration happens between the ATAPI command being written and the bmdma being started, the DMA is dropped. Eventually the guest times out and recovers, but that can take many seconds. (This is rare,

[Qemu-devel] [PATCH 1/2] Restore atapi_dma flag across migration

2014-12-09 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com If a migration happens just after the guest has kicked off an ATAPI command and kicked off DMA, we lose the atapi_dma flag, and the destination tries to complete the command as PIO rather than DMA. This upsets Linux; modern libata based kernels

[Qemu-devel] [PATCH 1/1] MAINTAINERS: Update for migrated migration code

2014-12-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com My previous patches migrated the migration code into migration/ but didn't update MAINTAINERS. Note that does mean that the owner for block-migration.c changes, but I'll ask block people what they want to do. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH 1/1] Sort include/qemu/typedefs.h

2014-12-11 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Mainly to make it less likely to conflict during merges. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/qemu/typedefs.h | 105 +++- 1 file changed, 50 insertions(+), 55

[Qemu-devel] [PATCH v3 0/5] More migration file cleanup

2014-12-12 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is a small set of patches that shuffle migration code around, but doesn't change the behaviour: 1) Move a lot of the migration source into a separate 'migration' directory Note this moves a lot of files around, git format-patch -M -B

[Qemu-devel] [PATCH v3 5/5] MAINTAINERS: Update for migrated migration code

2014-12-12 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com My previous patches migrated the migration code into migration/ but didn't update MAINTAINERS. Note that does mean that the owner for block-migration.c changes, but I'll ask block people what they want to do. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v3 4/5] Split the QEMU buffered file code out

2014-12-12 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The splitting of qemu-file and addition of the buffered file landed at the same time; so now split the buffered file code out. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/Makefile.objs | 2 +-

[Qemu-devel] [PATCH v3 1/5] Start migrating migration code into a migration directory

2014-12-12 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The migration code now occupies a fair chunk of the top level .c files, it seems time to give it it's own directory. I've not touched: arch_init.c - that's mostly RAM migration but has a few random other bits savevm.c-

[Qemu-devel] [PATCH v3 2/5] Remove migration- pre/post fixes off files in migration/ dir

2014-12-12 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The general feeling is that having migration/migration-blah is overkill. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/Makefile.objs | 8 migration/{block-migration.c = block.c} | 0

[Qemu-devel] [PATCH v3 3/5] Split struct QEMUFile out

2014-12-12 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Now we've got multiple QEMUFile source files, some of them need access to things that were defined in qemu-file.c, so create a -internal header for them. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/qemu-file-internal.h

[Qemu-devel] [PATCH 1/1] Tracify migration/rdma.c

2015-02-02 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Turn all the D/DD/DDDPRINTFs into trace events Turn most of the fprintf(stderr, into error_report Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/rdma.c | 333 ---

[Qemu-devel] [PATCH v2 1/1] Print PID and time in stderr traces

2015-01-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com When debugging migration it's useful to know the PID of each trace message so you can figure out if it came from the source or the destination. Printing the time makes it easy to do latency measurements or timings between trace points.

[Qemu-devel] [PATCH 2/3] Migration: Add lots of trace events

2015-01-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Mostly on the load side, so that when we get a complaint about a migration failure we can figure out what it didn't like. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/vmstate.c | 22 +++--- savevm.c

[Qemu-devel] [PATCH 1/3] savevm: Convert fprintf to error_report

2015-01-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Convert a bunch of fprintfs to error_reports Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/vmstate.c | 7 --- savevm.c| 21 +++-- 2 files changed, 15 insertions(+), 13 deletions(-) diff

[Qemu-devel] [PATCH 0/3] Migration tracing and errors

2015-01-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Tidy up some old printfs Add some more error_reports that previously just failed with invalid. Add lots and lots of tracing. Dr. David Alan Gilbert (3): savevm: Convert fprintf to error_report Migration: Add lots of trace events Print errors

[Qemu-devel] [PATCH 3/3] Print errors in some of the early migration failure cases.

2015-01-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/savevm.c b/savevm.c index 1cc0f02..98895fe 100644 --- a/savevm.c +++ b/savevm.c @@ -883,16 +883,20 @@

[Qemu-devel] [incomplete 1/1] Rsocket migration support [incomplete]

2015-01-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Implement migration over RDMA using the 'rsocket' library, the code appears to work for guests with 4GB RAM, but is hitting what appears to be internal library limitations above that. (riowrite always returns EAGAIN as soon as I register more

[Qemu-devel] [PATCH v2 3/3] Print errors in some of the early migration failure cases.

2015-01-21 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/savevm.c b/savevm.c index 1cc0f02..98895fe 100644 --- a/savevm.c +++ b/savevm.c @@ -883,16 +883,20 @@

<    1   2   3   4   5   6   7   8   9   10   >