On 13/4/17 2:07 am, Jeremiah Lott via freebsd-cloud wrote:
On Wed, Apr 12, 2017 at 1:30 AM, Leif Pedersen <bi...@hobbiton.org> wrote:
I keep an extra EBS volume handy that has a simple recovery image. If I get
stuck into a trouble, I change the normal boot disk to sdb, and attach my
recovery volume as sda1. Essentially, the extra volume is my "recovery
partition". To make it cheaper, keep only a snapshot of it.
I tried for a while to get some sort of bootloader-based recovery plan in
place for our cloud-based systems, like what was originally asked for. We
already have a primary and a backup partition in our boot disk, but there
was no way I found in EC2 to easily switch the partition to boot from. In
the end, I gave up on passing information to the bootloader and used
something like the above with multiple images. I actually wrote a script at
one point using the aws CLI that you could run from any FreeBSD VM in the
same availability zone. It detached the original boot volume from the
"broken" instance; attached it as a secondary disk to the recovery image,
changed the boot partition, detached it from the recovery image, then
re-attached it to the original image. It took a while to run, but required
little user input. We kind of kept that as "good enough" for the rare case
that a instance became un-bootable and we cared to recover it rather than
replace it. I'm not sure we actually ever used it on a customer system. It
was used more during development when you are more likely to break stuff
(and want to recover coredumps, etc. so you can fix the broken code).
Thanks for your comments. It appears that you have the same issues
that we do.
Andriy Gapon has been doing some stuff where nextboot information is
saved onto the drive,
and it knows how much it has beooted which may be good enough for us.
basically a 'drops to recovery mode after N failures' would be enough
for me.
Is there any chance you can make your "recovery" system available?
(especially if you can give source for the aws CLI stuff.. I think
having that as an example
and starting point might be a good start to making something truely
useful.
It may even be worth adding it to the regular FreeBSD AMI so that any
FreeBSD
EC2 system could be used to recover other systems.
In our system there is a single zpool with two ZFS datasets and we use
the "bootfs" parameter to select the new image, but it can be
overridden from the boot menu, except of course on AWS due to the lack
of console.
If you go down the route of implementing EC2 network driver(s) in the
bootloader, then you could read the instance metadata via http and use a
tag to control the boot behavior. However, a bootloader driver, even a very
simplistic one, for xn0 (and potentially for both ixv and ena, if you
support EC2 Enhanced Networking) was more work that we wanted to undertake
for this.
Jeremiah Lott
Avere Systems
_______________________________________________
freebsd-cl...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-cloud
To unsubscribe, send any mail to "freebsd-cloud-unsubscr...@freebsd.org"
_______________________________________________
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
