Re: Warm Migration feature for bhyve - review on Phabricator
Willem Jan Withagen wrote this message on Thu, Jan 28, 2021 at 01:42 +0100: > On 25-1-2021 19:42, John-Mark Gurney wrote: > > Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +: > >> -Original Message- > >> From: John-Mark Gurney > >> Sent: 25 January 2021 06:21 > >> To: Matt Churchyard > >> Cc: Elena Mihailescu ; > >> freebsd-virtualization@freebsd.org > >> Subject: Re: Warm Migration feature for bhyve - review on Phabricator > >> > >>> There's also hastd which can aid with this... > >> Thanks for the reply. I've always been wary of the additional complexity > >> of HAST and ZFS, as it doesn't seem to have widespread usage or support, > >> and things get ugly fast when storage is involved. > > Totally agree... > > > >> However, the idea of using HAST on top of zvols to provide network > >> mirrored storage for a guest is interesting. It adds a lot of extra > >> complexity, and probably performance impact though if it's just for the > >> ability to move a guest between systems that may only happen every now and > >> then. I'm also not sure it would help (or would at least be even more > >> complex) if I have 4 hosts and want to be able to move guests anywhere. > > gmirror + ggate is another option as well... > > I did try all kinds of these solutions that are based on a 2 system > backend storage, and it was relatively easy for me to get the state of > the storage in a split-brain situation which is not really repairable > without a lot of manual user intervention. > So I gave up on HAST and gmirror over ggate. Can you explain more about the split brain w/ gmirror+ggate? If writes didn't happen to one of the members of gmirror, then that should be investigated.. I guess I could see a situation where ggate got behind on writes (connection drop), and gmirror didn't kick out the ggate member, or failed to mark it as stale, but those are bugs that need to be identified and fixed... I'm partly currious about the ggate, because I've recently done some work w/ ggate (performance improvements and working of sftp), and it's simple enough that issues are a bit surprising.. I'm not at all familar w/ HAST to advice on any issues w/ that... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
On 25-1-2021 19:42, John-Mark Gurney wrote: Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +: -Original Message- From: John-Mark Gurney Sent: 25 January 2021 06:21 To: Matt Churchyard Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: Hello, all, We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. [1] https://reviews.freebsd.org/D28270 Thank you, Elena I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") The current process (from a mile high overview) effectively seems to be the following - * Start guest on host2 pointing at a shared disk, and halt any execution * use bhyvectl to pause and begin migration on host1 * Start the guest on host2 What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - * send a ZFS snapshot of the running machine to host2 * start the guest in migrate recv mode on host2 * pause the guest on host1 * send a new snapshot * initiate the migration of memory/device data * start guest on host2 Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? There's also hastd which can aid with this... Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. Totally agree... However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. gmirror + ggate is another option as well... I did try all kinds of these solutions that are based on a 2 system backend storage, and it was relatively easy for me to get the state of the storage in a split-brain situation which is not really repairable without a lot of manual user intervention. So I gave up on HAST and gmirror over ggate. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
On 22-1-2021 11:09, Matt Churchyard wrote: Hello, all, We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. [1] https://reviews.freebsd.org/D28270 Thank you, Elena I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") The current process (from a mile high overview) effectively seems to be the following - * Start guest on host2 pointing at a shared disk, and halt any execution * use bhyvectl to pause and begin migration on host1 * Start the guest on host2 What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - * send a ZFS snapshot of the running machine to host2 * start the guest in migrate recv mode on host2 * pause the guest on host1 * send a new snapshot * initiate the migration of memory/device data * start guest on host2 Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? Shared storage is great but becomes complex and expensive if you want high performance and reliability (seeing as the storage quickly becomes a major single point of failure without enterprise active/active kit). I suspect the most common deployment of bhyve is independent hosts with local ZFS pools as this is easy/cheap and gives great performance. Most hypervisors only had "shared storage" migration for a long time but the big ones now also support transferring disk data live. It would be great to be able to do this out of the gate. Great work to see this landing. This is actually one of the reasons I started porting Ceph and build a bhyve backend that interfaces to Ceph RBD. So you do not need to migrate the disk data. You are right that Ceph does not come cheap in hardware, but the designers try to jump through hoops to make it reliable. And it will also take some performance, but I'm very happy running my FreeBSD-vm's on our Openstack/Ceph cluster. Warm migration has been on the horizon for quite some time, so I very happy to see it getting there. Unfortunately I have limited time ATM to actually see if warm migration could be made to work with Ceph a diskstorage. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
Indeed, the save/restore functionality works only for Intel CPUs and it's experimental. We are working to extend the save/restore functionality so it can be enabled by default in the future. We are working on the file format, too. The warm migration is a wrapper on the save/restore functionality, indeed. However, it doesn't use files and sends the guest's state (memory, kernel structures, emulated devices) over the network, using a socket. Basically, the work does the following on the source host: pause the guest state, use the save mechanism to extract the guest's state, send the state though the network. On the destination host, we have the reverse process: receive the guest's state, use the restore mechanism to restore the guest's state, resume the guest. Beside the guest's state, we check whether the source and destination hosts are compatible (e.g. they have to have the same CPU, otherwise, the resume on the destination host will fail). Elena On Mon, 25 Jan 2021 at 22:03, Rob Wing wrote: > > The experimental suspend/resume feature for bhyve exists, but isn't turned on > by default. Also, in my testing, it only works with intel gear. > > I understand the migration feature is a wrapper around the suspend/resume > functionality. > > A question, is the migration feature dependent upon the underlying file > format of suspend/resume? If so, something to consider is that there will be > additional work to fix the migration feature after the suspend/resume file > format is dialed in. > > My concern is that building experimental features on top of experimental > features could become problematic when the foundation > of the experimental features is changed. > > When suspend/resume was brought in, the commit log made it very explicit that > the file format needs to be fleshed out a bit more before being enabled by > default. > > -Rob > > p.s. > > part of me thinks some of this should happen at the process level (i.e. > suspend the bhyve process, migrate it, and resume it elsewhere) - easier said > than done. > > On Mon, Jan 25, 2021 at 10:43 AM Matt Churchyard > wrote: >> >> -Original Message- >> From: Elena Mihailescu >> Sent: 25 January 2021 14:25 >> To: Matt Churchyard >> Cc: John-Mark Gurney ; freebsd-virtualization@freebsd.org >> Subject: Re: Warm Migration feature for bhyve - review on Phabricator >> >> On Mon, 25 Jan 2021 at 13:26, Matt Churchyard >> wrote: >> > >> > -Original Message- >> > From: John-Mark Gurney >> > Sent: 25 January 2021 06:21 >> > To: Matt Churchyard >> > Cc: Elena Mihailescu ; >> > freebsd-virtualization@freebsd.org >> > Subject: Re: Warm Migration feature for bhyve - review on Phabricator >> > >> > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: >> > > > Hello, all, >> > > >> > > > We have recently opened a review on Phabricator for the warm migration >> > > > code for > bhyve [1]. Please take a look and let us know if it is >> > > > anything we can improve. >> > > >> > > > [1] https://reviews.freebsd.org/D28270 >> > > >> > > > Thank you, >> > > > Elena >> > > >> > > I appreciate that this isn't really related to the current review, >> > > and commend the work being put into bhyve - it's an invaluable >> > > addition to FreeBSD. I'm just wondering if any thought has been put >> > > into the future possibility for transfer of disk data during >> > > migration (i.e. the equivalent of "storage vmotion") >> > > >> > > The current process (from a mile high overview) effectively seems to >> > > be the following - >> > > >> > > * Start guest on host2 pointing at a shared disk, and halt any >> > > execution >> > > * use bhyvectl to pause and begin migration on host1 >> > > * Start the guest on host2 >> > > >> > > What would be the feasibility of being able to run a process such as >> > > the following? Obviously it would likely need to be orchestrated by >> > > an external tool, but to me it seems the main requirement is really >> > > just to be able to provide separate control over the pause and >> > > migrate steps on host1 - >> > > >> > > * send a ZFS snapshot of the running machine to host2 >> > > * start the guest in migrate recv mode on host2 >> > > * pause the guest on host1 >> > > * send a new snapshot >> > > * i
Re: Warm Migration feature for bhyve - review on Phabricator
The experimental suspend/resume feature for bhyve exists, but isn't turned on by default. Also, in my testing, it only works with intel gear. I understand the migration feature is a wrapper around the suspend/resume functionality. A question, is the migration feature dependent upon the underlying file format of suspend/resume? If so, something to consider is that there will be additional work to fix the migration feature after the suspend/resume file format is dialed in. My concern is that building experimental features on top of experimental features could become problematic when the foundation of the experimental features is changed. When suspend/resume was brought in, the commit log made it very explicit that the file format needs to be fleshed out a bit more before being enabled by default. -Rob p.s. part of me thinks some of this should happen at the process level (i.e. suspend the bhyve process, migrate it, and resume it elsewhere) - easier said than done. On Mon, Jan 25, 2021 at 10:43 AM Matt Churchyard wrote: > -Original Message- > From: Elena Mihailescu > Sent: 25 January 2021 14:25 > To: Matt Churchyard > Cc: John-Mark Gurney ; > freebsd-virtualization@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > On Mon, 25 Jan 2021 at 13:26, Matt Churchyard > wrote: > > > > -Original Message- > > From: John-Mark Gurney > > Sent: 25 January 2021 06:21 > > To: Matt Churchyard > > Cc: Elena Mihailescu ; > > freebsd-virtualization@freebsd.org > > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: > > > > Hello, all, > > > > > > > We have recently opened a review on Phabricator for the warm > migration code for > bhyve [1]. Please take a look and let us know if it is > anything we can improve. > > > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > > > Thank you, > > > > Elena > > > > > > I appreciate that this isn't really related to the current review, > > > and commend the work being put into bhyve - it's an invaluable > > > addition to FreeBSD. I'm just wondering if any thought has been put > > > into the future possibility for transfer of disk data during > > > migration (i.e. the equivalent of "storage vmotion") > > > > > > The current process (from a mile high overview) effectively seems to > > > be the following - > > > > > > * Start guest on host2 pointing at a shared disk, and halt any > > > execution > > > * use bhyvectl to pause and begin migration on host1 > > > * Start the guest on host2 > > > > > > What would be the feasibility of being able to run a process such as > > > the following? Obviously it would likely need to be orchestrated by > > > an external tool, but to me it seems the main requirement is really > > > just to be able to provide separate control over the pause and > > > migrate steps on host1 - > > > > > > * send a ZFS snapshot of the running machine to host2 > > > * start the guest in migrate recv mode on host2 > > > * pause the guest on host1 > > > * send a new snapshot > > > * initiate the migration of memory/device data > > > * start guest on host2 > > > > > > Are there any major complications here I'm not aware of other than the > requirement to pause the guest and kick off the state migration as two > separate calls? > > > > > There's also hastd which can aid with this... > > > > Thanks for the reply. I've always been wary of the additional complexity > of HAST and ZFS, as it doesn't seem to have widespread usage or support, > and things get ugly fast when storage is involved. > > > > However, the idea of using HAST on top of zvols to provide network > mirrored storage for a guest is interesting. It adds a lot of extra > complexity, and probably performance impact though if it's just for the > ability to move a guest between systems that may only happen every now and > then. I'm also not sure it would help (or would at least be even more > complex) if I have 4 hosts and want to be able to move guests anywhere. > > > > The main reason for the email was to explore my theory that, by > > leveraging ZFS, any bhyve user could roll their own storage migration > > ability with a few commands as long as the following two (seemingly > > minor) abilities were present > > > > 1) the ability to suspend a guest without it being done as part of the > >
RE: Warm Migration feature for bhyve - review on Phabricator
-Original Message- From: Elena Mihailescu Sent: 25 January 2021 14:25 To: Matt Churchyard Cc: John-Mark Gurney ; freebsd-virtualization@freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator On Mon, 25 Jan 2021 at 13:26, Matt Churchyard wrote: > > -Original Message- > From: John-Mark Gurney > Sent: 25 January 2021 06:21 > To: Matt Churchyard > Cc: Elena Mihailescu ; > freebsd-virtualization@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: > > > Hello, all, > > > > > We have recently opened a review on Phabricator for the warm migration > > > code for > bhyve [1]. Please take a look and let us know if it is > > > anything we can improve. > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > Thank you, > > > Elena > > > > I appreciate that this isn't really related to the current review, > > and commend the work being put into bhyve - it's an invaluable > > addition to FreeBSD. I'm just wondering if any thought has been put > > into the future possibility for transfer of disk data during > > migration (i.e. the equivalent of "storage vmotion") > > > > The current process (from a mile high overview) effectively seems to > > be the following - > > > > * Start guest on host2 pointing at a shared disk, and halt any > > execution > > * use bhyvectl to pause and begin migration on host1 > > * Start the guest on host2 > > > > What would be the feasibility of being able to run a process such as > > the following? Obviously it would likely need to be orchestrated by > > an external tool, but to me it seems the main requirement is really > > just to be able to provide separate control over the pause and > > migrate steps on host1 - > > > > * send a ZFS snapshot of the running machine to host2 > > * start the guest in migrate recv mode on host2 > > * pause the guest on host1 > > * send a new snapshot > > * initiate the migration of memory/device data > > * start guest on host2 > > > > Are there any major complications here I'm not aware of other than the > > requirement to pause the guest and kick off the state migration as two > > separate calls? > > > There's also hastd which can aid with this... > > Thanks for the reply. I've always been wary of the additional complexity of > HAST and ZFS, as it doesn't seem to have widespread usage or support, and > things get ugly fast when storage is involved. > > However, the idea of using HAST on top of zvols to provide network mirrored > storage for a guest is interesting. It adds a lot of extra complexity, and > probably performance impact though if it's just for the ability to move a > guest between systems that may only happen every now and then. I'm also not > sure it would help (or would at least be even more complex) if I have 4 hosts > and want to be able to move guests anywhere. > > The main reason for the email was to explore my theory that, by > leveraging ZFS, any bhyve user could roll their own storage migration > ability with a few commands as long as the following two (seemingly > minor) abilities were present > > 1) the ability to suspend a guest without it being done as part of the > migrate call. (I assume suspend/resume support via bhyvectl is planned > anyway, if not already in place at this point) > 2) modification of the migrate call to skip suspend if the guest is already > suspended. > > The main thing I'm not sure of is whether the migrate call has any specific > reliance on doing the suspend itself (e.g. if it needs to do anything before > the suspend, which will obviously be problematic if suspend & migrate are > called separately). Or if there's something else I missed that means this is > not feasible. I'm not really after massive changes to the current review to > implement disk migration in bhyve itself. > Thank you for your input related to the disk migration. We were (still are, > actually) focusing on the live migration feature for bhyve and did not take > into consideration the disk migration. As > Mihai said, the patches for the > two (guest's state and guest's disk migration) are or will be quite big by > themselves and we want the review process to go as smoothly as possible. > After we have a pretty clear view of the live migration implementation (a > patch on this feature will be uploaded on Phabricator in a couple of weeks) > and how we should improve it, we will look into the disk migration feature
Re: Warm Migration feature for bhyve - review on Phabricator
Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +: > -Original Message- > From: John-Mark Gurney > Sent: 25 January 2021 06:21 > To: Matt Churchyard > Cc: Elena Mihailescu ; > freebsd-virtualization@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: > > > Hello, all, > > > > > We have recently opened a review on Phabricator for the warm migration > > > code for > bhyve [1]. Please take a look and let us know if it is > > > anything we can improve. > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > Thank you, > > > Elena > > > > I appreciate that this isn't really related to the current review, and > > commend the work being put into bhyve - it's an invaluable addition to > > FreeBSD. I'm just wondering if any thought has been put into the future > > possibility for transfer of disk data during migration (i.e. the equivalent > > of "storage vmotion") > > > > The current process (from a mile high overview) effectively seems to be the > > following - > > > > * Start guest on host2 pointing at a shared disk, and halt any execution > > * use bhyvectl to pause and begin migration on host1 > > * Start the guest on host2 > > > > What would be the feasibility of being able to run a process such as the > > following? Obviously it would likely need to be orchestrated by an external > > tool, but to me it seems the main requirement is really just to be able to > > provide separate control over the pause and migrate steps on host1 - > > > > * send a ZFS snapshot of the running machine to host2 > > * start the guest in migrate recv mode on host2 > > * pause the guest on host1 > > * send a new snapshot > > * initiate the migration of memory/device data > > * start guest on host2 > > > > Are there any major complications here I'm not aware of other than the > > requirement to pause the guest and kick off the state migration as two > > separate calls? > > > There's also hastd which can aid with this... > > Thanks for the reply. I've always been wary of the additional complexity of > HAST and ZFS, as it doesn't seem to have widespread usage or support, and > things get ugly fast when storage is involved. Totally agree... > However, the idea of using HAST on top of zvols to provide network mirrored > storage for a guest is interesting. It adds a lot of extra complexity, and > probably performance impact though if it's just for the ability to move a > guest between systems that may only happen every now and then. I'm also not > sure it would help (or would at least be even more complex) if I have 4 hosts > and want to be able to move guests anywhere. gmirror + ggate is another option as well... > The main reason for the email was to explore my theory that, by leveraging > ZFS, any bhyve user could roll their own storage migration ability with a few > commands as long as the following two (seemingly minor) abilities were present > > 1) the ability to suspend a guest without it being done as part of the > migrate call. (I assume suspend/resume support via bhyvectl is planned > anyway, if not already in place at this point) Yeah, I'd hope that there would be ability to insert custom commands in between the suspend on one host and the resumption on the new host.. But it is good explicitly state this.. > 2) modification of the migrate call to skip suspend if the guest is already > suspended. > > The main thing I'm not sure of is whether the migrate call has any specific > reliance on doing the suspend itself (e.g. if it needs to do anything before > the suspend, which will obviously be problematic if suspend & migrate are > called separately). Or if there's something else I missed that means this is > not feasible. I'm not really after massive changes to the current review to > implement disk migration in bhyve itself. +1 -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
On Mon, 25 Jan 2021 at 13:26, Matt Churchyard wrote: > > -Original Message- > From: John-Mark Gurney > Sent: 25 January 2021 06:21 > To: Matt Churchyard > Cc: Elena Mihailescu ; > freebsd-virtualization@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: > > > Hello, all, > > > > > We have recently opened a review on Phabricator for the warm migration > > > code for > bhyve [1]. Please take a look and let us know if it is > > > anything we can improve. > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > Thank you, > > > Elena > > > > I appreciate that this isn't really related to the current review, and > > commend the work being put into bhyve - it's an invaluable addition to > > FreeBSD. I'm just wondering if any thought has been put into the future > > possibility for transfer of disk data during migration (i.e. the equivalent > > of "storage vmotion") > > > > The current process (from a mile high overview) effectively seems to be the > > following - > > > > * Start guest on host2 pointing at a shared disk, and halt any execution > > * use bhyvectl to pause and begin migration on host1 > > * Start the guest on host2 > > > > What would be the feasibility of being able to run a process such as the > > following? Obviously it would likely need to be orchestrated by an external > > tool, but to me it seems the main requirement is really just to be able to > > provide separate control over the pause and migrate steps on host1 - > > > > * send a ZFS snapshot of the running machine to host2 > > * start the guest in migrate recv mode on host2 > > * pause the guest on host1 > > * send a new snapshot > > * initiate the migration of memory/device data > > * start guest on host2 > > > > Are there any major complications here I'm not aware of other than the > > requirement to pause the guest and kick off the state migration as two > > separate calls? > > > There's also hastd which can aid with this... > > Thanks for the reply. I've always been wary of the additional complexity of > HAST and ZFS, as it doesn't seem to have widespread usage or support, and > things get ugly fast when storage is involved. > > However, the idea of using HAST on top of zvols to provide network mirrored > storage for a guest is interesting. It adds a lot of extra complexity, and > probably performance impact though if it's just for the ability to move a > guest between systems that may only happen every now and then. I'm also not > sure it would help (or would at least be even more complex) if I have 4 hosts > and want to be able to move guests anywhere. > > The main reason for the email was to explore my theory that, by leveraging > ZFS, any bhyve user could roll their own storage migration ability with a few > commands as long as the following two (seemingly minor) abilities were present > > 1) the ability to suspend a guest without it being done as part of the > migrate call. (I assume suspend/resume support via bhyvectl is planned > anyway, if not already in place at this point) > 2) modification of the migrate call to skip suspend if the guest is already > suspended. > > The main thing I'm not sure of is whether the migrate call has any specific > reliance on doing the suspend itself (e.g. if it needs to do anything before > the suspend, which will obviously be problematic if suspend & migrate are > called separately). Or if there's something else I missed that means this is > not feasible. I'm not really after massive changes to the current review to > implement disk migration in bhyve itself. Thank you for your input related to the disk migration. We were (still are, actually) focusing on the live migration feature for bhyve and did not take into consideration the disk migration. As Mihai said, the patches for the two (guest's state and guest's disk migration) are or will be quite big by themselves and we want the review process to go as smoothly as possible. After we have a pretty clear view of the live migration implementation (a patch on this feature will be uploaded on Phabricator in a couple of weeks) and how we should improve it, we will look into the disk migration feature and how it can be implemented for bhyve. As I said, our objective is to add the live migration functionality to bhyve which is based on the warm migration implementation which in turn is based on the suspend option in bhyve (the snapshotting/suspend option was added in the upstream last year, the Fre
RE: Warm Migration feature for bhyve - review on Phabricator
-Original Message- From: Miroslav Lachman <000.f...@quip.cz> Sent: 25 January 2021 10:37 To: Matt Churchyard Subject: Re: Warm Migration feature for bhyve - review on Phabricator On 22/01/2021 11:09, Matt Churchyard wrote: [...] > Shared storage is great but becomes complex and expensive if you want high > performance and reliability (seeing as the storage quickly becomes a major > single point of failure without enterprise active/active kit). I suspect the > most common deployment of bhyve is independent hosts with local ZFS pools as > this is easy/cheap and gives great performance. Most hypervisors only had > "shared storage" migration for a long time but the big ones now also support > transferring disk data live. It would be great to be able to do this out of > the gate. > > I did have a poor mans version of this in vm-bhyve, but obviously relied on > stopping and restarting the guest. I always had problems with the nc commands > not exiting cleanly though and hanging the process so never made it official. >I don't know much details about your setup and your problems with nc > (netcat) but I guess it is used for zfs send & receive. You can try to use > mbuffer from ports instead of nc: > https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ > It will add some dependency on other port but if somebody wants storage > migration with your greate tool vm-bhyve I think it is small price to pay. > (another way is to properly setup SSH and > do not use nc nor mbuffer) Thanks for the reply. Yes, I was using it to send zfs snapshots between hosts. I think I actually have the same issue when doing it by hand. The nc call seems like it hangs once complete but will instantly return to the prompt when pressing enter. As you say, a better solution would be to use ssh. It may even be possible to trigger the receive command directly from the sending host if I get a bit hacky and split off a second background task. I may have another look at it. It worked pretty well all in all even if it did have to stop/start the guest briefly. Matt > Kind regards > Miroslav Lachman ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Warm Migration feature for bhyve - review on Phabricator
-Original Message- From: John-Mark Gurney Sent: 25 January 2021 06:21 To: Matt Churchyard Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: > > Hello, all, > > > We have recently opened a review on Phabricator for the warm migration code > > for > bhyve [1]. Please take a look and let us know if it is anything we > > can improve. > > > [1] https://reviews.freebsd.org/D28270 > > > Thank you, > > Elena > > I appreciate that this isn't really related to the current review, and > commend the work being put into bhyve - it's an invaluable addition to > FreeBSD. I'm just wondering if any thought has been put into the future > possibility for transfer of disk data during migration (i.e. the equivalent > of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be the > following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the > following? Obviously it would likely need to be orchestrated by an external > tool, but to me it seems the main requirement is really just to be able to > provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the > requirement to pause the guest and kick off the state migration as two > separate calls? > There's also hastd which can aid with this... Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. The main reason for the email was to explore my theory that, by leveraging ZFS, any bhyve user could roll their own storage migration ability with a few commands as long as the following two (seemingly minor) abilities were present 1) the ability to suspend a guest without it being done as part of the migrate call. (I assume suspend/resume support via bhyvectl is planned anyway, if not already in place at this point) 2) modification of the migrate call to skip suspend if the guest is already suspended. The main thing I'm not sure of is whether the migrate call has any specific reliance on doing the suspend itself (e.g. if it needs to do anything before the suspend, which will obviously be problematic if suspend & migrate are called separately). Or if there's something else I missed that means this is not feasible. I'm not really after massive changes to the current review to implement disk migration in bhyve itself. Matt -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: > > Hello, all, > > > We have recently opened a review on Phabricator for the warm migration code > > for > bhyve [1]. Please take a look and let us know if it is anything we > > can improve. > > > [1] https://reviews.freebsd.org/D28270 > > > Thank you, > > Elena > > I appreciate that this isn't really related to the current review, and > commend the work being put into bhyve - it's an invaluable addition to > FreeBSD. I'm just wondering if any thought has been put into the future > possibility for transfer of disk data during migration (i.e. the equivalent > of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be the > following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the > following? Obviously it would likely need to be orchestrated by an external > tool, but to me it seems the main requirement is really just to be able to > provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the > requirement to pause the guest and kick off the state migration as two > separate calls? There's also hastd which can aid with this... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
Hello Matt, Thank you for your input. The Warm/Live Migration feature is designated for cloud and cluster environments. Storage Live Migration is a feature that would be nice to have, but will be tackled after we get in warm migration and live migration using a shared storage. The code base is big enough as it is. Adding storage migration will make it hard to get in. Also this is the track that was followed by all hypervisors until now (KVM, Hyper-V, VMware). Thanks, Mihai On Fri, Jan 22, 2021 at 12:10 PM Matt Churchyard wrote: > > Hello, all, > > > We have recently opened a review on Phabricator for the warm migration > code for > bhyve [1]. Please take a look and let us know if it is anything > we can improve. > > > [1] https://reviews.freebsd.org/D28270 > > > Thank you, > > Elena > > I appreciate that this isn't really related to the current review, and > commend the work being put into bhyve - it's an invaluable addition to > FreeBSD. I'm just wondering if any thought has been put into the future > possibility for transfer of disk data during migration (i.e. the equivalent > of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be > the following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the > following? Obviously it would likely need to be orchestrated by an external > tool, but to me it seems the main requirement is really just to be able to > provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the > requirement to pause the guest and kick off the state migration as two > separate calls? > > Shared storage is great but becomes complex and expensive if you want high > performance and reliability (seeing as the storage quickly becomes a major > single point of failure without enterprise active/active kit). I suspect > the most common deployment of bhyve is independent hosts with local ZFS > pools as this is easy/cheap and gives great performance. Most hypervisors > only had "shared storage" migration for a long time but the big ones now > also support transferring disk data live. It would be great to be able to > do this out of the gate. > > I did have a poor mans version of this in vm-bhyve, but obviously relied > on stopping and restarting the guest. I always had problems with the nc > commands not exiting cleanly though and hanging the process so never made > it official. > > Regards, > Matt > > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Warm Migration feature for bhyve - review on Phabricator
> Hello, all, > We have recently opened a review on Phabricator for the warm migration code > for > bhyve [1]. Please take a look and let us know if it is anything we can > improve. > [1] https://reviews.freebsd.org/D28270 > Thank you, > Elena I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") The current process (from a mile high overview) effectively seems to be the following - * Start guest on host2 pointing at a shared disk, and halt any execution * use bhyvectl to pause and begin migration on host1 * Start the guest on host2 What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - * send a ZFS snapshot of the running machine to host2 * start the guest in migrate recv mode on host2 * pause the guest on host1 * send a new snapshot * initiate the migration of memory/device data * start guest on host2 Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? Shared storage is great but becomes complex and expensive if you want high performance and reliability (seeing as the storage quickly becomes a major single point of failure without enterprise active/active kit). I suspect the most common deployment of bhyve is independent hosts with local ZFS pools as this is easy/cheap and gives great performance. Most hypervisors only had "shared storage" migration for a long time but the big ones now also support transferring disk data live. It would be great to be able to do this out of the gate. I did have a poor mans version of this in vm-bhyve, but obviously relied on stopping and restarting the guest. I always had problems with the nc commands not exiting cleanly though and hanging the process so never made it official. Regards, Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Warm Migration feature for bhyve - review on Phabricator
Hello, all, We have recently opened a review on Phabricator for the warm migration code for bhyve [1]. Please take a look and let us know if it is anything we can improve. [1] https://reviews.freebsd.org/D28270 Thank you, Elena ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
