On Fri, Nov 15, 2013 at 07:11:18PM +0100, Michele Tartara wrote: > On Thu, Nov 14, 2013 at 2:05 PM, Jose A. Lopes <[email protected]> wrote: > > On Thu, Nov 14, 2013 at 11:31:11AM +0100, Guido Trotter wrote: > >> On Wed, Nov 13, 2013 at 9:57 AM, Michele Tartara <[email protected]> > >> wrote: > >> > On Tue, Nov 12, 2013 at 2:13 PM, Guido Trotter <[email protected]> > >> > wrote: > >> >> On Tue, Nov 12, 2013 at 12:41 PM, Michele Tartara <[email protected]> > >> >> wrote: > >> >>> Add the document describing a new design for the OS installation > >> >>> process for > >> >>> new instances. > >> >>> > >> >>> Signed-off-by: Michele Tartara <[email protected]> > >> >>> --- > >> >>> doc/design-draft.rst | 1 + > >> >>> doc/design-os.rst | 318 > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++ > >> >>> 2 files changed, 319 insertions(+) > >> >>> create mode 100644 doc/design-os.rst > >> >>> > >> >>> diff --git a/doc/design-draft.rst b/doc/design-draft.rst > >> >>> index c821292..3ed3852 100644 > >> >>> --- a/doc/design-draft.rst > >> >>> +++ b/doc/design-draft.rst > >> >>> @@ -20,6 +20,7 @@ Design document drafts > >> >>> design-daemons.rst > >> >>> design-hsqueeze.rst > >> >>> design-ssh-ports.rst > >> >>> + design-os.rst > >> >>> > >> >>> .. vim: set textwidth=72 : > >> >>> .. Local Variables: > >> >>> diff --git a/doc/design-os.rst b/doc/design-os.rst > >> >>> new file mode 100644 > >> >>> index 0000000..7a42a7f > >> >>> --- /dev/null > >> >>> +++ b/doc/design-os.rst > >> >>> @@ -0,0 +1,318 @@ > >> >>> +=============================== > >> >>> +Ganeti OS installation redesign > >> >>> +=============================== > >> >>> + > >> >>> +.. contents:: :depth: 3 > >> >>> + > >> >>> +This is a design document detailing a new OS installation procedure, > >> >>> more > >> >>> +secure, able to provide more features and easier to use for many > >> >>> common tasks > >> >>> +w.r.t. the current one. > >> >>> + > >> >>> +Current state and shortcomings > >> >>> +============================== > >> >>> + > >> >>> +As of Ganeti 2.10, each instance is associated with an OS definition. > >> >>> An OS > >> >>> +definition is a set of scripts (``create``, ``export``, ``import``, > >> >>> ``rename``) > >> >>> +that are executed with root privileges on the primary host of the > >> >>> instance to > >> >>> +perform all the OS-related functionality (setting up an operating > >> >>> system inside > >> >>> +the disks of the instance being created, exporting/importing the > >> >>> instance, > >> >>> +renaming it). > >> >>> + > >> >>> +These scripts receive, as environment variables, a fixed set of > >> >>> parameters > >> >>> +describing the instance (such as the hypervisor, the name of the > >> >>> instance, the > >> >>> +number of disks, and their location) and a set of user defined > >> >>> parameters. Each > >> >>> +of these parameters is also written into the configuration file of > >> >>> Ganeti, to > >> >>> +allow for future reinstalls of the instance, and in various log > >> >>> files, namely: > >> >>> + > >> >>> +* node daemon log file: contains DEBUG strings of the > >> >>> ``/os_validate``, > >> >>> + ``/instance_os_add`` and ``/instance_start`` RPC calls. > >> >>> + > >> >>> +* master daemon log file: DEBUG strings related to the same RPC calls > >> >>> are stored > >> >>> + here as well. > >> >>> + > >> >>> +* commands log: the CLI commands that create a new instance, > >> >>> including their > >> >>> + parameters, are logged here. > >> >>> + > >> >>> +* RAPI log: the RAPI commands that create a new instances, including > >> >>> their > >> >>> + parameters, are logged here. > >> >>> + > >> >>> +* job logs: the job files stored in the job queue or in its archive > >> >>> contain the > >> >>> + parameters. > >> >>> + > >> >>> +The current situation presents a number of shortcomings: > >> >>> + > >> >>> +* Having the installation scripts run with root power on the nodes is > >> >>> a huge > >> >>> + security issue. > >> >>> + > >> >> > >> >> s/is a huge security issue/doesn't allow user-defined os scripts, as > >> >> they would pose a huge security issue/ > >> >> > >> >> Note that there's no security issue *per se* in the current situation, > >> >> if the OS scripts are trusted. > >> >> (except perhaps for export, if the os script mounts the instance disk, > >> >> which is also not necessarily the case) > >> > > >> > Yes, that's what I meant. I'll reword it as you suggest. > >> > > >> >> > >> >> That said it could be a safety issue in the sense that an eventual > >> >> bug/error in the os script could risk disrupting the node. > >> > > >> > ACK > >> > > >> >> > >> >>> +* Ganeti cannot be used to create instances starting from user > >> >>> provided disk > >> >>> + images: even in the (hypothetical) case where the scripts are > >> >>> completely > >> >>> + secure and run not by root but by an unprivileged user with only > >> >>> the power to > >> >>> + mount arbitrary files as disk images, this is a security issue. It > >> >>> has been > >> >>> + proven that a carefully crafted file system might exploit kernel > >> >>> + vulnerabilities to gain control of the system. Therefore, directly > >> >>> mounting > >> >>> + images on the Ganeti nodes is not an option. > >> >>> + > >> >>> +* There is no way to inject files into an existing disk image. A > >> >>> common use case > >> >>> + is for the system administrator to provide a standard image of the > >> >>> system, to > >> >>> + be later personalized with the network configuration, private keys > >> >>> identifying > >> >>> + the machine, ssh keys of the users and so on. A possible workaround > >> >>> would be > >> >>> + for the scripts to mount the image (only if this is trusted!) and > >> >>> to receive > >> >>> + the configurations and ssh keys as user defined OS parameters. > >> >>> Unfortunately, > >> >>> + this is also not an option for security sensitive material (such as > >> >>> the ssh > >> >>> + keys) because the OS parameters are stored in many places on the > >> >>> system, as > >> >>> + already described above. > >> >>> + > >> >>> +* Most other virtualization software simply work with instance > >> >>> images, not with > >> >>> + installation scripts. This difference makes the interaction of > >> >>> Ganeti with > >> >>> + other softwares difficult. > >> >> > >> >> s/softwares/software/ > >> > > >> > ACK > >> > > >> >> > >> >>> + > >> >>> +Proposed changes > >> >>> +================ > >> >>> + > >> >>> +In order to fix the shortcomings of the current state, we plan to > >> >>> introduce the > >> >>> +following changes: > >> >>> + > >> >>> +* Change the OS parameters to have three categories: > >> >>> + > >> >>> + * ``public``: the current behavior. The parameter is logged and > >> >>> stored freely. > >> >>> + > >> >>> + * ``private``: the parameter is saved inside the Ganeti > >> >>> configuration (to allow > >> >>> + for instance reinstall) but it is not shown in logs, job logs, or > >> >>> passed back > >> >>> + via RAPI. > >> >>> + > >> >>> + * ``secret``: the parameter is not saved inside the Ganeti > >> >>> configuration. > >> >>> + Reinstall are impossible unless the data is passed again. The > >> >>> parameter will > >> >>> + not appear in any log file. In order to preserve the functionality > >> >>> of Ganeti, > >> >>> + the parameters will still need to be stored in the job files, but > >> >>> they will > >> >>> + be removed from there when the job has finished running (either > >> >>> successfully > >> >>> + or not). > >> >>> + > >> >> > >> >> Do we actually need to save them in the job files? > >> >> The job files could be saved (to disk) without, and in case the master > >> >> is failed over the job can be failed. > >> >> (this should make it a lot harder to access) > >> > > >> > Unfortunately, I think we need to save them. Currently the job is > >> > created by luxid, serialized, and then read from file and executed by > >> > masterd, as part of the ongoing migration of the job queue from > >> > masterd to luxid. > >> > > >> > >> Ack, but this is hopefully temporary, and the job data can perhaps in > >> the future be passed via socket between the two... > >> So OK temporarily during development, but not by design, let's rather > >> fix the underlying problem. > >> > >> >>> +* A new OS installation procedure, based on a safe virtualized > >> >>> environment. > >> >>> + This virtualized environment will run with the same hardware > >> >>> parameter as the > >> >>> + actual instance being installed, as much as possible. This will > >> >>> also allow to > >> >>> + reduce the memory usage in the host (specifically, in Dom0 for Xen > >> >>> + installations). > >> >>> Each instance will have these possible execution modes: > >> >>> + > >> >>> + * ``run``: the default mode, used when the machine is running > >> >>> normally. > >> >>> + > >> >>> + * ``self_install``: Ganeti will start the instance with a different > >> >>> set of > >> >>> + user-specified parameters, therefore allowing to attach an > >> >>> installation > >> >>> + floppy/cdrom/network, change the boot device order, or specify an > >> >>> OS image > >> >>> + to be used. The instance will then be responsible to get the > >> >>> parameters for > >> >>> + configuring itself (its network interfaces, IP address, hostname, > >> >>> etc.) from > >> >>> + a set of metadata provided to it by Ganeti (e.g.: using an > >> >>> approach > >> >>> + comparable to the one of the ``cloud-init`` tool). When this > >> >>> installation > >> >>> + mode is used, no OS installation script is required. > >> >>> + In order for installation of an OS from an image to be possible, > >> >>> a new > >> >>> + parameter ``--os-image`` will be added, allwoing to specify where > >> >>> to take > >> >>> + the image from. It will have to be mutually exclusive with > >> >>> ``--os-type``. If > >> >>> + ``--os-image`` is specified, ``--os-parameters`` can still be > >> >>> used, as it > >> >>> + will be passed to the instance as part of the metadata. > >> >>> + The set of ``self_install`` parameters will be stored as part of > >> >>> the > >> >>> + instance configuration, so that they can be used to reinstall the > >> >>> instance. > >> >>> + It will be the user's responsibility to ensure that the OS image > >> >>> or any > >> >>> + installation media is still available in the proper position when > >> >>> a > >> >>> + reinstall happens. > >> >>> + > >> >> > >> >> Should we use --os-type image:<name> and/or have an image os provider > >> >> that defines: > >> >> 1) the actual parameters needed for installation > >> >> 2) the image (eg. the verify script could double check that the image > >> >> is available from the node or accessible via the network...) > >> >> > >> >> I think in particular it would be useful to still have the concept of > >> >> an OS "provider" that tells ganeti how to install itself (which > >> >> parameters to use). This of course could be overridable, but at least > >> >> there would be a sane default without relying on the user to "get it > >> >> right". > >> > > >> > Regarding using --os-type image:<name>: > >> > That was my initial though too, and also my favorite choice. Still, > >> > given that we usually want to keep backwards compatibility, this would > >> > cause problems if somebody has an OS definition called "image". > >> > Furthermore, that name would become reserved in the future. > >> > If you think it is a small enough risk, and listing this in the > >> > "incompatible changes" section of the NEWS file is enough, then I'm > >> > absolutely in favor of doing it. > >> > > >> > >> I think it would be OK as it's not conflicting with an OS definition > >> called "image" but one called image:<something>, no? > >> > >> > Regarding the os provider: my idea here was to have a possibility of > >> > using Ganeti without having to provide a provider, but just an OS > >> > image plus some "gnt-instance add" parameters, therefore having a more > >> > standard approach, similar to what other solutions are doing. Having > >> > an OS provider for this as well, would defeat this purpose. Moreover, > >> > providing an installation script would still be an option, so who want > >> > to have an OS provider, can have it. > >> > > >> > >> Ack. > >> > >> >> > >> >>> + * ``install``: Ganeti will start the instance using a virtual > >> >>> appliance > >> >>> + specifically made for installing Ganeti instances. Scripts > >> >>> analogous to the > >> >>> + current ones will run inside this instance. The disks of the > >> >>> instance being > >> >>> + installed will be connected to this virtual appliance, so that > >> >>> the scripts > >> >>> + can mount them and modify them as needed, as currently happens, > >> >>> but with the > >> >>> + additional protection given by this happening in a VM. The > >> >>> virtual appliance > >> >>> + will be started in a clean state every time a new instance need > >> >>> to be > >> >>> + created, to further increase security. Metadata will be provided > >> >>> also to > >> >>> + this virtual applicance, that will take care of converting them to > >> >>> + environment variables for the installation scripts. > >> >>> + > >> >> > >> >> Please specify better that by "will be started in a clean state" you > >> >> actually mean "the disk will be reset to its pristine state and not > >> >> reused between reinstallation" because it might be construed to mean > >> >> just the "booting" (runtime info) which is sort of less strict. > >> > > >> > ACK > >> > > >> >> > >> >>> +In order to allow for the metadata to be sent inside the instance, a > >> >>> +communication mechanism between the instance and the host will be > >> >>> created. This > >> >>> +mechanism will be bidirectional (e.g.: to allow the setup process > >> >>> going on > >> >>> +inside the instance to communicate its progress to the host). Each > >> >>> instance will > >> >>> +have access exclusively to its own metadata, and it will be only able > >> >>> to > >> >>> +communicate with its host over this channel. > >> >>> + > >> >> > >> >> Too vague :) > >> > > >> > It's intentionally vague: here it's just meant to state the problem. > >> > The actual description of the metadata and the communication mechanism > >> > is in the implementation section. I'll add a reference to that from > >> > here. > >> > > >> > >> Thanks. > >> > >> >> > >> >> > >> >>> +As part of the instance creation command it will be possible to > >> >>> indicate a URL > >> >>> +for a "personalization package", that is an archive containing a set > >> >>> of files > >> >>> +meant to be overlayed on top of the operating system file system at > >> >>> the end of > >> >>> +the setup process, before the VM is started for the first time in > >> >>> ``run`` mode. > >> >>> +Ganeti will provide a mechanism for receiving and unpacking this > >> >>> archive as part > >> >>> +of the ``install`` execution mode, whereas in ``self_install`` mode > >> >>> it will only > >> >>> +be provided as a metadata for the instance to use. > >> >>> +The archive will be in TAR-GZIP format (with extension ``.tar.gz`` or > >> >>> ``.tgz``) > >> >>> +and will contain the files according to the directory structure that > >> >>> will be > >> >>> +recreated on the installation disk. Files contained in this archive > >> >>> will > >> >>> +overwrite files with the same path created during the install > >> >>> procedure (if > >> >>> +any). > >> >>> +The URL of the "personalization package" will have to specify an > >> >>> extesion to > >> >>> +identify the file format (in order to allow for more formats to be > >> >>> supported in > >> >>> +the future). > >> >>> +The URL will be stored as part of the configuration of the instance > >> >>> (therefore, > >> >>> +the URL should not contain confidential information, but the file > >> >>> there > >> >>> +available can). It is up to the system administrator to ensure that a > >> >>> package > >> >>> +is actually available at that URL at install and reinstall time. > >> >>> +The content of the package is allowed to change. E.g.: a system > >> >>> administrator > >> >>> +might create a package containing the private keys of the instance > >> >>> being > >> >>> +created. When the instance is reinstalled, a new package with new > >> >>> keys can be > >> >>> +made available there, therefore allowing instance reinstall without > >> >>> the need to > >> >>> +store keys. > >> >>> + > >> >> > >> >> Add something about authentication perhaps (so that an admin can have > >> >> a file available only to the ganeti installer only for the time of the > >> >> installation) and also about the fact that we won't cache/keep the > >> >> file on the node OS. > >> > > >> > ACK > >> > > >> >> > >> >>> +Implementation > >> >>> +============== > >> >>> + > >> >>> +The implementation of this design will happen as an ordered sequence > >> >>> of steps, > >> >>> +of increasing impact on the system and, in some cases, dependent on > >> >>> each other: > >> >>> + > >> >>> +#. Private and secret instance parameters > >> >>> +#. Communication mechanism between host and instance > >> >>> +#. Metadata service > >> >>> +#. Personalization package > >> >>> +#. ``self_install`` mode > >> >>> +#. ``install`` mode (with virtualization environment) > >> >>> + > >> >>> +Some of these steps need to be more deeply specified w.r.t. what is > >> >>> already > >> >>> +written in the `Proposed changes`_ Section. Extra details will be > >> >>> provided in > >> >>> +the following Subsections. > >> >>> + > >> >>> +Communication mechanism and metadata service > >> >>> +++++++++++++++++++++++++++++++++++++++++++++ > >> >>> + > >> >>> +The communication mechanism and the metadata service are described > >> >>> together > >> >>> +because they are deeply tied. On the other hand, the communication > >> >>> mechanism > >> >>> +will need to be more generic because it can be used for other reasons > >> >>> in the > >> >>> +future (like allowing instances to esplicitly send commands to > >> >>> Ganeti, or to let > >> >> > >> >> explicitly > >> > > >> > ACK > >> > > >> >> > >> >>> +Ganeti control a helper instance, like the one hereby introduced for > >> >>> performing > >> >>> +OS installs inside a safe environment). > >> >>> + > >> >>> +The communication mechanism will be enabled automatically when the > >> >>> instance is > >> >>> +in ``self_install`` or ``install`` mode, but for backwards > >> >>> compatibility it will > >> >>> +be disabled when the instance is in ``run`` mode unless it is > >> >>> esplicitly > >> >> > >> >> ^ see above > >> > > >> > ACK > >> > > >> >> > >> >>> +requested at instance startup by using a new, ad-hoc, parameter > >> >>> +(``--communication``). > >> >> > >> >> Which parameter is this? An instance, hypervisor or backend parameter? > >> >> And why? > >> >> Also -C could do as well (if we go for instance level). Remember to > >> >> specify here as it has to be clear that an instance once configured > >> >> that way will be always started that way. > >> >> > >> > > >> > Yes, it's intended to be an instance level parameter. I'll specify > >> > that it is set at creation time, or modifiable with "gnt-instance > >> > modify", and then is automatically read from the config and used every > >> > time the instance is started. > >> > > >> >>> + > >> >>> +When the communication mechanism is enabled, Ganeti will create a new > >> >>> network > >> >>> +interface inside the instance. This extra network interface will be > >> >>> the last one > >> >>> +of the instance, after all the user defined ones. On the host side, > >> >>> this > >> >>> +interface will be only accessible to the host itself, and not be > >> >>> routed outside > >> >>> +the machine. > >> >> > >> >> Actually it would be great if we didn't even have to create the tap. > >> > > >> > Do you mean something like (for kvm): > >> > -net user,net=169.254.169.0/24,host=169.254.169.254 > >> > that starts a user network showing the host as reachable with address > >> > 169.254.169.254? > >> > > >> > >> Yes, that would be a secure way to do it. Or perhaps using a > >> VDE-compatible connection? > >> But it doesn't have to be. Otherwise let's discuss which rules will > >> there be by default so that we assure that traffic can't get to the > >> wrong place. > >> > >> >>> +On this network interface, the instance will connect using the IP: > >> >>> +169.254.169.1 and netmask 255.255.255.0. > >> >>> +The host will be on the same network, with the IP address: > >> >>> 169.254.169.254. > >> >>> +The instance will be able to connect to 169.254.169.254:80, and issue > >> >>> GET > >> >>> +requests to an HTTP server that will provide the instance metadata. > >> >>> + > >> >>> +The choice of this IP address and port is done for compatibility > >> >>> reasons with > >> >>> +OpenStack's and Amazon EC2's ways of providing metadata to the > >> >>> instance. > >> >>> + > >> >>> +Where possible, the metadata will be provided in a way compatible > >> >>> with OpenStack > >> >>> +at:: > >> >>> + > >> >>> + http://169.254.169.254/openstack/<version>/meta_data.json > >> >>> + > >> >>> +or with Amazon EC2, at:: > >> >>> + > >> >>> + http://169.254.169.254/<version>/meta-data/* > >> >>> + > >> >>> +If some metadata are Ganeti-specific and don't fit this structure, > >> >>> they will be > >> >>> +provided at:: > >> >>> + > >> >>> + http://169.254.169.254/<version>/ganeti/meta_data.json > >> >>> + > >> >> > >> >> Not quite clear! :) How does the OS choose between those? How are they > >> >> expected to differ? > >> > > >> > The idea is to provide the data in both formats, so the OS can chose > >> > based on its own preferences (there are some tools already getting the > >> > data from those postions, such as cloud-init). > > > > cloud-init seems to be using the Amazon EC2 URL. > > > > Why doesn't Ganeti use the Amazon EC2 URL? It seems the Amazon EC2 URL > > already has some of the fields Ganeti requires in this design doc, > > such as, public keys. We can add other fields that do not collide > > with existing ones, for example, private keys. > > > And why do we have the Openstack URL as well? If there are any tools > > that we plan to use that require the Openstack URL, maybe they should > > be listed in this design document, as well. > > > > It just seems strange that we have an excellent opportunity here to > > design something clean from scratch and it already looks so complicated... > > > The idea is exactly using Amazon EC2 URL for everything that fits in > there, so that existing tools, like cloud-init, can access those data. > Unfortunately there are some things that don't fit in EC2's metadata > format, and for those I'm planning a separate "ganeti/" directory. > I suggested also supporting OpenStack because it is quickly becoming a > standard, but it can definitely be removed if we don't have an > immediate reason for supporting it. It can always be added later. > > Regarding Ganeti's own URLs: some things are not present in Amazon's > API. For those, we cannot just add random pieces to that API, because > that would be an incompatible "embrace and extend" approach.
I'm not familiar with incompatible embrance and extend. What is this ? Jose > > Adding a new ganeti/ directory next to the amazon one, as openstack > did, makes it clear what is new and what is compatible, and it seems > to me that is not complicated. > > The reason why it looks complicated, probably, is that (as I already > wrote replying to Guido) unfortunately there is a mistake in the > design I sent. The actual directory structure which I meant is not: > > 169.254.169.254/<version>/* > 169.254.169.254/<version>/ganeti/* > 169.254.169.254/openstack/<version>/* > > but: > > 169.254.169.254/<version>/* > 169.254.169.254/ganeti/<version>/* > 169.254.169.254/openstack/<version>/* > > which makes things much more elegantly separate. > > Sorry for the mistake, and I hope this structure (which might or might > not contain openstack, at this point) looks better. > > Thanks, > Michele > -- > Google Germany GmbH > Dienerstr. 12 > 80331 München > > Registergericht und -nummer: Hamburg, HRB 86891 > Sitz der Gesellschaft: Hamburg > Geschäftsführer: Graham Law, Christine Elizabeth Flores -- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
