[Qemu-devel] [PATCH 12/13] virtio-gpu: Wrap in vmstate

2016-06-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Forcibly convert it to a vmstate wrapper; proper conversion comes later. Signed-off-by: Dr. David Alan Gilbert --- hw/display/virtio-gpu.c | 17 +++-- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/hw/display/virtio-gpu.c b/hw/display

[Qemu-devel] [PATCH 13/13] virtio: Update migration docs

2016-06-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Remove references to register_savevm. Signed-off-by: Dr. David Alan Gilbert --- docs/virtio-migration.txt | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/virtio-migration.txt b/docs/virtio-migration.txt index cf66458..98a6b0f 100644

[Qemu-devel] [PATCH v6 00/47] Postcopy implementation

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" This is the 6th cut of my version of postcopy; it is designed for use with the Linux kernel additions posted by Andrea Arcangeli here: git clone --reference linux -b userfault18 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git (Note this is a differe

[Qemu-devel] [PATCH v6 02/47] Split header writing out of qemu_savevm_state_begin

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Split qemu_savevm_state_begin to: qemu_savevm_state_header That writes the initial file header. qemu_savevm_state_beginThat sets up devices and does the first device pass. Used later in postcopy. Signed-off-by: Dr. David Alan

[Qemu-devel] [PATCH v6 04/47] Add qemu_get_counted_string to read a string prefixed by a count byte

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" and use it in loadvm_state and ram_load. Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 5 + include/migration/qemu-file.h | 3 +++ migration/qemu-file.c | 16 savevm.c | 11 ++

[Qemu-devel] [PATCH v6 01/47] Start documenting how postcopy works.

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- docs/migration.txt | 167 + 1 file changed, 167 insertions(+) diff --git a/docs/migration.txt b/docs/migration.txt index 0492a45..f975c75 100644 --- a/docs/migration.txt

[Qemu-devel] [PATCH v6 06/47] Provide runtime Target page information

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init. Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code s

[Qemu-devel] [PATCH v6 03/47] qemu_ram_foreach_block: pass up error value, and down the ramblock name

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller, and rdma_add_block the only function it calls using it. Pass the name of the ramblock to the function; helps in debugging. Sig

[Qemu-devel] [PATCH v6 13/47] Migration commands

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process. For use in postcopy. Signed-off-by: Dr. David Alan Gilbert --- include/migration/mig

[Qemu-devel] [PATCH v6 05/47] Create MigrationIncomingState

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together. allocate MIS in process_incoming_migration_co Signed-off-by: Dr. David Alan Gilbert --- include/migrati

[Qemu-devel] [PATCH v6 11/47] Return path: Open a return path on QEMUFile for sockets

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'. Wire it up for 'socket' QEMUFile's using a dup'd fd. Signed-off-by: Dr. David Alan Gilbert --- include/migration/qemu-file.h | 7 + migration/qemu-f

[Qemu-devel] [PATCH v6 09/47] Add wrapper for setting blocking status on a QEMUFile

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be differ

[Qemu-devel] [PATCH v6 07/47] Move copy out of qemu_peek_buffer

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" qemu_peek_buffer currently copies the data it reads into a buffer, however the next patch wants access to the buffer without the copy, hence rework to remove the copy to the layer above. Signed-off-by: Dr. David Alan Gilbert --- include/migration/qemu-file.h | 2

[Qemu-devel] [PATCH v6 08/47] Add qemu_get_buffer_less_copy to avoid copies some of the time

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data. Thus 'qemu_g

[Qemu-devel] [PATCH v6 14/47] Return path: Control commands

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add two src->dest commands: * OPEN_RETURN_PATH - To request that the destination open the return path * PING - Request an acknowledge from the destination Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- include/migration/migration.h |

[Qemu-devel] [PATCH v6 15/47] Return path: Send responses from destination to source

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add migrate_send_rp_message to send a message from destination to source along the return path. (It uses a mutex to let it be called from multiple threads) Add migrate_send_rp_shut to send a 'shut' message to indicate the destination is finished with the RP. Ad

[Qemu-devel] [PATCH v6 10/47] Rename save_live_complete to save_live_complete_precopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious. Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 2 +-

[Qemu-devel] [PATCH v6 16/47] Return path: Source handling of return path

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Open a return path, and handle messages that are received upon it. Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 8 ++ migration/migration.c | 177 +- trace-events | 1

[Qemu-devel] [PATCH v6 19/47] Rework loadvm path for subloops

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions. Split the core of qemu_loadvm_state out so we can use it for both. Allow the inner loadvm loop to qui

[Qemu-devel] [PATCH v6 20/47] Add migration-capability boolean for postcopy-ram.

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series). Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Eric

[Qemu-devel] [PATCH v6 12/47] Return path: socket_writev_buffer: Block even on non-blocking fd's

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path. Signed-off-by: Dr. David Alan Gilbert --- migration/qemu-file-unix.c

[Qemu-devel] [PATCH v6 27/47] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" 'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy 'migration_postcopy_phase' is provided for other sections to know if they're in postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- include/migration/migration.

[Qemu-devel] [PATCH v6 23/47] migrate_init: Call from savevm

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- include/migration/migration.

[Qemu-devel] [PATCH v6 17/47] ram_debug_dump_bitmap: Dump a migration bitmap as text

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance. Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 40 +++- include/migration/migration.h

[Qemu-devel] [PATCH v6 22/47] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the conte

[Qemu-devel] [PATCH v6 32/47] Postcopy: Postcopy startup in migration thread

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Rework the migration thread to setup and start postcopy. Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 3 + migration/migration.c | 163 -- trace-events | 4 ++ 3 fil

[Qemu-devel] [PATCH v6 26/47] migrate_start_postcopy: Command to trigger transition to postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the: migrate_start_postcopy command must be issued. Postcopy will start sometime after this (when it's next checked in t

[Qemu-devel] [PATCH v6 18/47] Move loadvm_handlers into MigrationIncomingState

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- include/migration/migration.h | 5 + i

[Qemu-devel] [PATCH v6 24/47] Modify save_live_pending for postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Modify save_live_pending to return separate postcopiable and non-postcopiable counts. Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 8 ++-- include/migration/vmstate.h | 5 +++-- include/sysemu/sysemu.h | 4 +++- migration/blo

[Qemu-devel] [PATCH v6 30/47] postcopy: Incoming initialisation

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- arch_init.c | 11 include/migration/migration.h| 3 + include/migration/postcopy-ram.h | 12 migration/postcopy-ram.c | 116

[Qemu-devel] [PATCH v6 28/47] Add qemu_savevm_state_complete_postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices. The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postco

[Qemu-devel] [PATCH v6 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The state of the postcopy process is managed via a series of messages; * Add wrappers and handlers for sending/receiving these messages * Add state variable that track the current state of postcopy Signed-off-by: Dr. David Alan Gilbert --- include/migration

[Qemu-devel] [PATCH v6 25/47] postcopy: OS support test

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Provide a check to see if the OS we're running on has all the bits needed for postcopy. Creates postcopy-ram.c which will get most of the other helpers we need. Signed-off-by: Dr. David Alan Gilbert --- include/migration/postcopy-ram.h | 19 + migration/Mak

[Qemu-devel] [PATCH v6 36/47] Page request: Consume pages off the post-copy queue

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning. Note: a) After a queued page the linear walk carries on from after the unqueued page; there is a reasonable chance that th

[Qemu-devel] [PATCH v6 33/47] Postcopy end in migration_thread

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy. The end of migration code is getting a bit complciated now, so move out into its own function.

[Qemu-devel] [PATCH v6 39/47] qemu_ram_block_from_host

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel. qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset in the RAMBlock an

[Qemu-devel] [PATCH v6 29/47] Postcopy: Maintain sentmap and calculate discard

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs. Maintain a 'sentmap' of pages that

[Qemu-devel] [PATCH v6 31/47] postcopy: ram_enable_notify to switch on userfault

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Mark the area of RAM as 'userfault' Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later) Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- include/migration/migration.h| 3 ++ include/migration/

[Qemu-devel] [PATCH v6 35/47] Page request: Process incoming page request

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" On receiving MIG_RPCOMM_REQ_PAGES look up the address and queue the page. Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 64 ++- include/exec/cpu-all.h| 2 -- include/migration/migration.

[Qemu-devel] [PATCH v6 37/47] postcopy_ram.c: place_page and helpers

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" postcopy_place_page (etc) provide a way for postcopy to place a page into guests memory atomically (using the copy ioctl on the ufd). Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h| 1 + include/migration/postcopy-ram.h | 16

[Qemu-devel] [PATCH v6 34/47] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source. Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 4 +++ migration/migration.c | 70 ++

[Qemu-devel] [PATCH v6 46/47] Disable mlock around incoming postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Userfault doesn't work with mlock; mlock is designed to nail down pages so they don't move, userfault is designed to tell you when they're not there. munlock the pages we userfault protect before postcopy. mlock everything again at the end if mlock is enabled. Sig

[Qemu-devel] [PATCH v6 38/47] Postcopy: Use helpers to map pages during migration

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" In postcopy, the destination guest is running at the same time as it's receiving pages; as we receive new pages we must put them into the guests address space atomically to avoid a running CPU accessing a partially written page. Use the helpers in postcopy-ram.c to

[Qemu-devel] [PATCH v6 40/47] Don't sync dirty bitmaps in postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Once we're in postcopy the source processors are stopped and memory shouldn't change any more, so there's no need to look at the dirty map. There are two notes to this: 1) If we do resync and a page had changed then the page would get sent again, which the d

[Qemu-devel] [PATCH v6 43/47] Start up a postcopy/listener thread ready for incoming page data

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code can finish off th

[Qemu-devel] [PATCH v6 41/47] Host page!=target page: Cleanup bitmaps

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Prior to the start of postcopy, ensure that everything that will be transferred later is a whole host-page in size. This is accomplished by discarding partially transferred host pages and marking any that are partially dirty as fully dirty. Signed-off-by: Dr. Davi

[Qemu-devel] [PATCH v6 42/47] Postcopy; Handle userfault requests

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" userfaultfd is a Linux syscall that gives an fd that receives a stream of notifications of accesses to pages registered with it and allows the program to acknowledge those stalls and tell the accessing thread to carry on. Signed-off-by: Dr. David Alan Gilbert ---

[Qemu-devel] [PATCH v6 44/47] postcopy: Wire up loadvm_postcopy_handle_ commands

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Wire up more of the handlers for the commands on the destination side, in particular loadvm_postcopy_handle_run now has enough to start the guest running. Signed-off-by: Dr. David Alan Gilbert --- savevm.c | 29 - trace-events | 2

[Qemu-devel] [PATCH v6 45/47] End of migration for postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Tweak the end of migration cleanup; we don't want to close stuff down at the end of the main stream, since the postcopy is still sending pages on the other thread. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 25 - tra

[Qemu-devel] [PATCH v6 47/47] Inhibit ballooning during postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy detects accesses to pages that haven't been transferred yet using userfaultfd, and it causes exceptions on pages that are 'not present'. Ballooning also causes pages to be marked as 'not present' when the guest inflates the balloon. Potentially a balloon co

[Qemu-devel] [PATCH 00/10] Remove RDMA migration dependence on RAMBlock offset

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" RDMA migration currently relies on the source and destination RAMBlocks having the same offsets within ram_addr_t space; unfortunately that's just not true when: a) You hotplug on the source but then create the device on the command line on the destination.

[Qemu-devel] [PATCH 04/10] Translate offsets to destination address space

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The 'offset' field in RDMACompress and 'current_addr' field in RDMARegister are commented as being offsets within a particular RAMBlock, however they appear to actually be offsets within the ram_addr_t space. The code currently assumes that the offsets on the sourc

[Qemu-devel] [PATCH 03/10] Store block name in local blocks structure

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" In a later patch the block name will be used to match up two views of the block list. Keep a copy of the block name with the local block list. (At some point it could be argued that it would be best just to let migration see the innards of RAMBlock and avoid the n

[Qemu-devel] [PATCH 01/10] Rename RDMA structures to make destination clear

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" RDMA has two data types that are named confusingly; RDMALocalBlock (pointed to indirectly by local_ram_blocks) RDMARemoteBlock (pointed to by block in RDMAContext) RDMALocalBlocks, as the name suggests is a data strucuture that represents the RDMAable RAM Blo

[Qemu-devel] [PATCH 07/10] Simplify rdma_delete_block and remove it's dependence on the hash

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" rdma_delete_block is currently very general, but it's only used in cleanup at the end. Simplify it and remove it's dependence on the hash table and remove all of the hash-table regeneration designed to handle the (unused) case of deleting an arbitrary block. Sign

[Qemu-devel] [PATCH 02/10] qemu_ram_foreach_block: pass up error value, and down the ramblock name

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller, and rdma_add_block the only function it calls using it. Pass the name of the ramblock to the function; helps in debugging. Sig

[Qemu-devel] [PATCH 10/10] Sanity check RDMA remote data

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Perform some basic (but probably not complete) sanity checking on requests from the RDMA source. Signed-off-by: Dr. David Alan Gilbert --- migration/rdma.c | 30 ++ 1 file changed, 30 insertions(+) diff --git a/migration/rdma.c b/migr

[Qemu-devel] [PATCH 05/10] Rework ram_control_load_hook to hook during block load

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" We need the names of RAMBlocks as they're loaded for RDMA, reuse an existing QEMUFile hook with some small mods. Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 4 +++- include/migration/migration.h | 2 +- include/migration/qemu-file.

[Qemu-devel] [PATCH 08/10] Rework ram block hash

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" RDMA uses a hash from block offset->RAM Block; this isn't needed on the destination, and now that the destination sorts the ramblock list, is harder to maintain. Split the hash so that it's only generated on the source. Signed-off-by: Dr. David Alan Gilbert ---

[Qemu-devel] [PATCH 06/10] Remove unneeded memset

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- migration/rdma.c | 1 - 1 file changed, 1 deletion(-) diff --git a/migration/rdma.c b/migration/rdma.c index e43fae4..4f7dd0d 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2469,7 +2469,6 @@ static void *qemu_rd

[Qemu-devel] [PATCH 09/10] Sort destination RAMBlocks to be the same as the source

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Use the order of incoming RAMBlocks from the source to record an index number; that then allows us to sort the destination local RAMBlock list to match the source. Now that the RAMBlocks are known to be in the same order, this simplifies the RDMA Registration step

[Qemu-devel] [PATCH v5 07/45] Return path: Open a return path on QEMUFile for sockets

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'. Wire it up for 'socket' QEMUFile's using a dup'd fd. Signed-off-by: Dr. David Alan Gilbert --- include/migration/qemu-file.h | 7 + migration/qemu-

[Qemu-devel] [PATCH v5 05/45] Create MigrationIncomingState

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together. allocate MIS in process_incoming_migration_co Signed-off-by: Dr. David Alan Gilbert --- include/migrati

[Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add migrate_send_rp_message to send a message from destination to source along the return path. (It uses a mutex to let it be called from multiple threads) Add migrate_send_rp_shut to send a 'shut' message to indicate the destination is finished with the RP. Ad

[Qemu-devel] [PATCH v5 16/45] Add migration-capability boolean for postcopy-ram.

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Eric Blake --- include/migration/migration.h | 1 + migration/migration.c | 9 + qapi-schema.json | 7 ++- 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/include/m

[Qemu-devel] [PATCH v5 09/45] Migration commands

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process. For use in postcopy. Signed-off-by: Dr. David Alan Gilbert --- include/migration/mig

[Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Modify save_live_pending to return separate postcopiable and non-postcopiable counts. Add 'can_postcopy' to allow a device to state if it can postcopy Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 15 +-- include/migration/v

[Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Provide a check to see if the OS we're running on has all the bits needed for postcopy. Creates postcopy-ram.c which will get most of the other helpers we need. Signed-off-by: Dr. David Alan Gilbert --- include/migration/postcopy-ram.h | 19 + migration/Mak

[Qemu-devel] [PATCH v5 10/45] Return path: Control commands

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add two src->dest commands: * OPEN_RETURN_PATH - To request that the destination open the return path * SEND_PING - Request an acknowledge from the destination Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 2 ++ include/sysemu/s

[Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the: migrate_start_postcopy command must be issued. Postcopy will start sometime after this (when it's next checked in t

[Qemu-devel] [PATCH v5 15/45] Rework loadvm path for subloops

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions. Split the core of qemu_loadvm_state out so we can use it for both. Allow the inner loadvm loop to qui

[Qemu-devel] [PATCH v5 30/45] Postcopy: Postcopy startup in migration thread

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Rework the migration thread to setup and start postcopy. Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 3 + migration/migration.c | 161 -- trace-events | 4 ++ 3 fil

[Qemu-devel] [PATCH v5 25/45] qemu_savevm_state_complete: Postcopy changes

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip: a) Finishing postcopiable iterative devices - they'll carry on b) The termination byte on the end of the stream. We then also add: qemu_savevm_state_postcopy_compl

[Qemu-devel] [PATCH v5 41/45] Start up a postcopy/listener thread ready for incoming page data

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code can finish off th

[Qemu-devel] [PATCH v5 44/45] Disable mlock around incoming postcopy

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Userfault doesn't work with mlock; mlock is designed to nail down pages so they don't move, userfault is designed to tell you when they're not there. munlock the pages we userfault protect before postcopy. mlock everything again at the end if mlock is enabled. Sig

[Qemu-devel] [PATCH 2/2] Tests for rolling statistics code

2015-02-27 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- tests/Makefile | 3 + tests/test-rolling-stats.c | 161 + 2 files changed, 164 insertions(+) create mode 100644 tests/test-rolling-stats.c diff --git a/tests/Make

[Qemu-devel] [PATCH 1/2] Rolling statistics utilities

2015-02-27 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" There are various places where it's useful to hold a series of values that change over time and get summaries about them. This provides: - a count of the number of items - min/max - mean - a weighted mean (where you can set the weight to determine

[Qemu-devel] [PATCH 0/2] RFC: Rolling statistics

2015-02-27 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Hi, This is an attempt at a generic rolling statistics utility to allow data (e.g. bandwidth usage, times etc) to be collected easily. They hold some basic values (min/max/mean/weighted mean) and the last 'n' raw values.I'd like to use this maybe in fault-tol

[Qemu-devel] [RFC v2 0/4] Rolling statistics utilities

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Hi, This is an attempt at a generic rolling statistics utility to allow data (e.g. bandwidth usage, times etc) to be collected easily. They hold some basic values (min/max/mean/weighted mean) and the last 'n' raw values.I'd like to use this maybe with fault-t

[Qemu-devel] [RFC v2 3/4] hmp: Add a helper function for printing out a Rolling Statistics set

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- hmp.c | 21 + 1 file changed, 21 insertions(+) diff --git a/hmp.c b/hmp.c index 735097c..20241d8 100644 --- a/hmp.c +++ b/hmp.c @@ -138,6 +138,27 @@ void hmp_info_mice(Monitor *mon, const QDict *qdict)

[Qemu-devel] [RFC v2 1/4] RollingStats qapi type

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" This adds a QAPI type that holds the results of statistics collected by RStats. Signed-off-by: Dr. David Alan Gilbert --- qapi-schema.json | 30 ++ 1 file changed, 30 insertions(+) diff --git a/qapi-schema.json b/qapi-schema.json inde

[Qemu-devel] [RFC v2 2/4] Rolling statistics utilities

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" There are various places where it's useful to hold a series of values that change over time and get summaries about them. This provides: - a count of the number of items - min/max - mean - a weighted mean (where you can set the weight to determine

[Qemu-devel] [RFC v2 4/4] Example use of rolling statistics in migration

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" This is an example use of the rolling statistics to watch the 'expected downtime' in a bit more detail than the current summary figure. Example outputs from a simple run: HMP: expected downtime stats: Min/Max: 222, 1634 Mean: 983.8 (Weighted: 1005.1253) Count: 141

[Qemu-devel] [RFC 0/1] Rolling stats on colo

2015-03-05 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Hi, I'm getting COLO running on a couple of our machines here and wanted to see what was actually going on, so I merged in my recent rolling-stats code: http://lists.gnu.org/archive/html/qemu-devel/2015-03/msg00648.html with the following patch, and now I get on

[Qemu-devel] [RFC 1/1] COLO: Add primary side rolling statistics

2015-03-05 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Record: Checkpoint lifetime (ms) Pause time due to checkpoint (ms) Checkpoint size (bytes) Signed-off-by: Dr. David Alan Gilbert --- hmp.c | 12 include/migration/migration.h | 3 +++ migration/colo.c | 15

[Qemu-devel] [PATCH 1/1] Add qemu_get_counted_string to read a string prefixed by a count byte

2015-05-15 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" and use it in loadvm_state and ram_load. Where ever it's used, check the return and error if it failed. Minor: ram_load was using a 257 byte array for its string, the maximum length is 255 bytes + 0 terminator, so fix to 256 Signed-off-by: Dr. David Alan G

[Qemu-devel] [PATCH 2/4] Disable section footers on older machine types

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The next patch adds section footers; but we don't want to break migration compatibility so disable them on older machine types Signed-off-by: Dr. David Alan Gilbert --- hw/i386/pc_piix.c | 2 ++ hw/i386/pc_q35.c | 2 ++ include/migration/

[Qemu-devel] [PATCH 4/4] Teach analyze-migration.py about section footers

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- scripts/analyze-migration.py | 5 + 1 file changed, 5 insertions(+) diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py index 0c8b22f..f6894be 100755 --- a/scripts/analyze-migration.py +++ b/scrip

[Qemu-devel] [PATCH 0/4] Add section footers to detect corrupted migration streams

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Badly formatted migration streams can go undetected or produce misleading errors due to a lock of checking at the end of sections. In particular a section that adds an extra 0x00 at the end causes what looks like a normal end of stream and thus doesn't produce any e

[Qemu-devel] [PATCH 1/4] Merge section header writing

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The header writing for device sections is open coded in a few places, merge it into one. Signed-off-by: Dr. David Alan Gilbert --- savevm.c | 73 +--- 1 file changed, 28 insertions(+), 45 deletions(-) d

[Qemu-devel] [PATCH 3/4] Add a protective section footer

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Badly formatted migration streams can go undetected or produce misleading errors due to a lock of checking at the end of sections. In particular a section that adds an extra 0x00 at the end causes what looks like a normal end of stream and thus doesn't produce any e

[Qemu-devel] [PATCH v2 1/1] Add qemu_get_counted_string to read a string prefixed by a count byte

2015-05-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" and use it in loadvm_state and ram_load. Where ever it's used, check the return and error if it failed. Minor: ram_load was using a 257 byte array for its string, the maximum length is 255 bytes + 0 terminator, so fix to 256 Signed-off-by: Dr. David Alan G

[Qemu-devel] [PATCH v2 0/1] Add qemu_get_counted_string

2015-05-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Move the reading of length first encoded strings into it's own function. v2: Terminate return buffer at read length rather than expected length Dr. David Alan Gilbert (1): Add qemu_get_counted_string to read a string prefixed by a count byte arch_init.c

[Qemu-devel] [PATCH 1/6] Add qemu_get_counted_string to read a string prefixed by a count byte

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" and use it in loadvm_state and ram_load. Where ever it's used, check the return and error if it failed. Minor: ram_load was using a 257 byte array for its string, the maximum length is 255 bytes + 0 terminator, so fix to 256 Signed-off-by: Dr. David Alan G

[Qemu-devel] [PATCH 0/6] Migration cleanups for postcopy

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Hi, This is a set of patches that started off life in my postcopy tree but that are more general. Amit suggested I split them out as a separate series so that the main Postcopy series gets easier to review. The qemu_get_counted_string and qemu_ram_foreach_block

[Qemu-devel] [PATCH 3/6] qemu_ram_foreach_block: pass up error value, and down the ramblock name

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller, and rdma_add_block the only function it calls using it. Pass the name of the ramblock to the function; helps in debugging. Sig

[Qemu-devel] [PATCH 4/6] Create MigrationIncomingState

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together. allocate MIS in process_incoming_migration_co Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Amit Sh

[Qemu-devel] [PATCH 2/6] Split header writing out of qemu_savevm_state_begin

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Split qemu_savevm_state_begin to: qemu_savevm_state_header That writes the initial file header. qemu_savevm_state_beginThat sets up devices and does the first device pass. Used later in postcopy. Signed-off-by: Dr. David Alan

[Qemu-devel] [PATCH 6/6] Move loadvm_handlers into MigrationIncomingState

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: David Gibson --- include/migration/migration.h | 5 + i

[Qemu-devel] [PATCH 5/6] Move copy out of qemu_peek_buffer

2015-05-21 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" qemu_peek_buffer currently copies the data it reads into a buffer, however a future patch wants access to the buffer without the copy, hence rework to remove the copy to the layer above. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Amit Shah --- include/mi

<    3   4   5   6   7   8   9   10   11   12   >