Re: [Qemu-devel] [RFC PATCH v2 1/3] separate thread for VM migration
On 08/01/2011 11:00 PM, Umesh Deshpande wrote: I kept this in migration.c to call qemu_savevm_state_begin. (The way it is done currently. i.e. to keep access to FdMigrationState in migration.c) Calling it from buffered_file.c would be inconsistent in that sense. or we will have to call it from the iothread before spawning the migration thread. Right, I missed that. Perhaps you can call it the first time put_ready is called. Also why is the separation between FdMigrationState and QEMUFileBuffered is required. Is QEMUFileBuffered designed to use also for things other than migration? No, but let's keep it this way for now. It may be an annoyance, but it also helps making a reusable architecture, and it can probably be cleaned up substantially with thread support. Paolo
Re: [Qemu-devel] [RFC PATCH v2 1/3] separate thread for VM migration
On 07/29/2011 10:57 PM, Umesh Deshpande wrote:
This patch creates a separate thread for the guest migration on the source side.
Signed-off-by: Umesh Deshpandeudesh...@redhat.com
Looks pretty good!
One thing that shows, is that the interface separation between
buffered_file.c is migration.c is pretty weird. Your patch makes it
somewhat worse, but it was like this before so it's not your fault. The
good thing is that if buffered_file.c uses threads, you can fix a large
part of this and get even simpler code:
1) there is really just one way to implement migrate_fd_put_notify, and
with your simplifications it does not belong anymore in migration.c.
2) s-callback is actually not NULL exactly if s-file-frozen_output is
true, you can remove it as well;
3) buffered_close is messy because it can be called from both the
iothread (monitor-migrate_fd_cancel-migrate_fd_cleanup-qemu_fclose)
or the migration thread (after qemu_savevm_state_complete). But
buffered_close is actually very similar to your thread function (it does
flush+wait_for_unfreeze, basically)! So buffered_close can be simply:
s-closed = 1;
ret = qemu_thread_join(s-thread); /* doesn't exist yet :) */
qemu_free(...);
return ret;
Another nit is that here:
+if (migrate_fd_check_expire()) {
+buffered_rate_tick(s-file);
+}
+
+if (s-state != MIG_STATE_ACTIVE) {
+break;
+}
+
+if (s-callback) {
+migrate_fd_wait_for_unfreeze(s);
+s-callback(s);
+}
you can still have a busy wait.
Putting it all together, you can move the thread function back to
buffered_file.c like:
while (!s-closed || (!s-has_error s-buffer_size)) {
if (s-freeze_output) {
qemu_mutex_unlock_iothread();
s-wait_for_unfreeze(s);
qemu_mutex_lock_iothread();
/* This comes from qemu_file_put_notify (via
buffered_put_buffer---can be simplified a lot too?).
s-freeze_output = 0;
/* Test again for cancellation. */
continue;
}
int64_t current_time = qemu_get_clock_ms(rt_clock);
if (s-expire_time current_time) {
struct timeval tv = { .tv_sec = 0, .tv_usec = ... };
qemu_mutex_unlock_iothread();
select (0, NULL, NULL, NULL, tv);
qemu_mutex_lock_iothread();
s-expire_time = qemu_get_clock_ms(rt_clock) + 100;
continue;
}
/* This comes from buffered_rate_tick. */
s-bytes_xfer = 0;
buffered_flush(s);
if (!s-closed) {
s-put_ready(s-opaque);
}
}
ret = s-close(s-opaque);
...
Does it look sane?
Paolo
Re: [Qemu-devel] [RFC PATCH v2 1/3] separate thread for VM migration
On 08/01/2011 05:37 AM, Paolo Bonzini wrote:
On 07/29/2011 10:57 PM, Umesh Deshpande wrote:
This patch creates a separate thread for the guest migration on the
source side.
Signed-off-by: Umesh Deshpandeudesh...@redhat.com
Looks pretty good!
One thing that shows, is that the interface separation between
buffered_file.c is migration.c is pretty weird. Your patch makes it
somewhat worse, but it was like this before so it's not your fault.
The good thing is that if buffered_file.c uses threads, you can fix a
large part of this and get even simpler code:
1) there is really just one way to implement migrate_fd_put_notify,
and with your simplifications it does not belong anymore in migration.c.
2) s-callback is actually not NULL exactly if s-file-frozen_output
is true, you can remove it as well;
3) buffered_close is messy because it can be called from both the
iothread (monitor-migrate_fd_cancel-migrate_fd_cleanup-qemu_fclose)
or the migration thread (after qemu_savevm_state_complete). But
buffered_close is actually very similar to your thread function (it
does flush+wait_for_unfreeze, basically)! So buffered_close can be
simply:
s-closed = 1;
ret = qemu_thread_join(s-thread); /* doesn't exist yet :) */
qemu_free(...);
return ret;
Another nit is that here:
+if (migrate_fd_check_expire()) {
+buffered_rate_tick(s-file);
+}
+
+if (s-state != MIG_STATE_ACTIVE) {
+break;
+}
+
+if (s-callback) {
+migrate_fd_wait_for_unfreeze(s);
+s-callback(s);
+}
you can still have a busy wait.
Putting it all together, you can move the thread function back to
buffered_file.c like:
while (!s-closed || (!s-has_error s-buffer_size)) {
if (s-freeze_output) {
qemu_mutex_unlock_iothread();
s-wait_for_unfreeze(s);
qemu_mutex_lock_iothread();
/* This comes from qemu_file_put_notify (via
buffered_put_buffer---can be simplified a lot too?).
s-freeze_output = 0;
/* Test again for cancellation. */
continue;
}
int64_t current_time = qemu_get_clock_ms(rt_clock);
if (s-expire_time current_time) {
struct timeval tv = { .tv_sec = 0, .tv_usec = ... };
qemu_mutex_unlock_iothread();
select (0, NULL, NULL, NULL, tv);
qemu_mutex_lock_iothread();
s-expire_time = qemu_get_clock_ms(rt_clock) + 100;
continue;
}
/* This comes from buffered_rate_tick. */
s-bytes_xfer = 0;
buffered_flush(s);
if (!s-closed) {
s-put_ready(s-opaque);
}
}
ret = s-close(s-opaque);
...
Does it look sane?
I kept this in migration.c to call qemu_savevm_state_begin. (The way it
is done currently. i.e. to keep access to FdMigrationState in migration.c)
Calling it from buffered_file.c would be inconsistent in that sense. or
we will have to call it from the iothread before spawning the migration
thread.
Also why is the separation between FdMigrationState and QEMUFileBuffered
is required. Is QEMUFileBuffered designed to use also for things other
than migration?
Thanks
Umesh
Paolo
[Qemu-devel] [RFC PATCH v2 1/3] separate thread for VM migration
This patch creates a separate thread for the guest migration on the source side.
Signed-off-by: Umesh Deshpande udesh...@redhat.com
---
buffered_file.c | 28 -
buffered_file.h |4 +++
migration.c | 59 +++---
migration.h |3 ++
savevm.c| 22 +---
savevm.h| 29 +++
6 files changed, 102 insertions(+), 43 deletions(-)
create mode 100644 savevm.h
diff --git a/buffered_file.c b/buffered_file.c
index 41b42c3..d4146bf 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -16,12 +16,16 @@
#include qemu-timer.h
#include qemu-char.h
#include buffered_file.h
+#include migration.h
+#include savevm.h
+#include qemu-thread.h
//#define DEBUG_BUFFERED_FILE
typedef struct QEMUFileBuffered
{
BufferedPutFunc *put_buffer;
+BufferedBeginFunc *begin;
BufferedPutReadyFunc *put_ready;
BufferedWaitForUnfreezeFunc *wait_for_unfreeze;
BufferedCloseFunc *close;
@@ -35,6 +39,7 @@ typedef struct QEMUFileBuffered
size_t buffer_size;
size_t buffer_capacity;
QEMUTimer *timer;
+QemuThread thread;
} QEMUFileBuffered;
#ifdef DEBUG_BUFFERED_FILE
@@ -181,8 +186,6 @@ static int buffered_close(void *opaque)
ret = s-close(s-opaque);
-qemu_del_timer(s-timer);
-qemu_free_timer(s-timer);
qemu_free(s-buffer);
qemu_free(s);
@@ -228,17 +231,15 @@ static int64_t buffered_get_rate_limit(void *opaque)
return s-xfer_limit;
}
-static void buffered_rate_tick(void *opaque)
+void buffered_rate_tick(QEMUFile *file)
{
-QEMUFileBuffered *s = opaque;
+QEMUFileBuffered *s = file-opaque;
if (s-has_error) {
buffered_close(s);
return;
}
-qemu_mod_timer(s-timer, qemu_get_clock_ms(rt_clock) + 100);
-
if (s-freeze_output)
return;
@@ -250,9 +251,17 @@ static void buffered_rate_tick(void *opaque)
s-put_ready(s-opaque);
}
+static void *migrate_vm(void *opaque)
+{
+QEMUFileBuffered *s = opaque;
+s-begin(s-opaque);
+return NULL;
+}
+
QEMUFile *qemu_fopen_ops_buffered(void *opaque,
size_t bytes_per_sec,
BufferedPutFunc *put_buffer,
+ BufferedBeginFunc *begin,
BufferedPutReadyFunc *put_ready,
BufferedWaitForUnfreezeFunc
*wait_for_unfreeze,
BufferedCloseFunc *close)
@@ -264,6 +273,7 @@ QEMUFile *qemu_fopen_ops_buffered(void *opaque,
s-opaque = opaque;
s-xfer_limit = bytes_per_sec / 10;
s-put_buffer = put_buffer;
+s-begin = begin;
s-put_ready = put_ready;
s-wait_for_unfreeze = wait_for_unfreeze;
s-close = close;
@@ -271,11 +281,9 @@ QEMUFile *qemu_fopen_ops_buffered(void *opaque,
s-file = qemu_fopen_ops(s, buffered_put_buffer, NULL,
buffered_close, buffered_rate_limit,
buffered_set_rate_limit,
-buffered_get_rate_limit);
-
-s-timer = qemu_new_timer_ms(rt_clock, buffered_rate_tick, s);
+ buffered_get_rate_limit);
-qemu_mod_timer(s-timer, qemu_get_clock_ms(rt_clock) + 100);
+qemu_thread_create(s-thread, migrate_vm, s);
return s-file;
}
diff --git a/buffered_file.h b/buffered_file.h
index 98d358b..cfe2833 100644
--- a/buffered_file.h
+++ b/buffered_file.h
@@ -17,12 +17,16 @@
#include hw/hw.h
typedef ssize_t (BufferedPutFunc)(void *opaque, const void *data, size_t size);
+typedef void (BufferedBeginFunc)(void *opaque);
typedef void (BufferedPutReadyFunc)(void *opaque);
typedef void (BufferedWaitForUnfreezeFunc)(void *opaque);
typedef int (BufferedCloseFunc)(void *opaque);
+void buffered_rate_tick(QEMUFile *file);
+
QEMUFile *qemu_fopen_ops_buffered(void *opaque, size_t xfer_limit,
BufferedPutFunc *put_buffer,
+ BufferedBeginFunc *begin,
BufferedPutReadyFunc *put_ready,
BufferedWaitForUnfreezeFunc
*wait_for_unfreeze,
BufferedCloseFunc *close);
diff --git a/migration.c b/migration.c
index af3a1f2..bf86067 100644
--- a/migration.c
+++ b/migration.c
@@ -31,6 +31,8 @@
do { } while (0)
#endif
+static int64_t expire_time;
+
/* Migration speed throttling */
static int64_t max_throttle = (32 20);
@@ -284,8 +286,6 @@ int migrate_fd_cleanup(FdMigrationState *s)
{
int ret = 0;
-qemu_set_fd_handler2(s-fd, NULL, NULL, NULL, NULL);
-
if (s-file) {
DPRINTF(closing file\n);
if (qemu_fclose(s-file) != 0) {
@@ -310,8 +310,7 @@ int migrate_fd_cleanup(FdMigrationState *s)
void migrate_fd_put_notify(void *opaque)
{
FdMigrationState *s = opaque;
-
-
