On Tue, Dec 17, 2013 at 10:30:19AM +0100, Michele Tartara wrote: > On Mon, Dec 16, 2013 at 5:54 PM, Jose A. Lopes <[email protected]> wrote: > > Replied to comments. > > > > diff --git a/doc/design-os.rst b/doc/design-os.rst > > index d805a68..f885dff 100644 > > --- a/doc/design-os.rst > > +++ b/doc/design-os.rst > > @@ -18,7 +18,7 @@ perform all the OS-related functionality (setting up an > > operating system inside > > the disks of the instance being created, exporting/importing the instance, > > renaming it). > > > > -These scripts receive, as environment variables, a fixed set of parameters > > +These scripts receive through environment variables a fixed set of > > parameters > > related to the instance (such as the hypervisor, the name of the instance, > > the > > number of disks, and their location) and a set of user defined parameters. > > These parameters are also written in the configuration file of Ganeti, to > > allow > > @@ -72,126 +72,158 @@ Proposed changes > > ================ > > > > In order to fix the shortcomings of the current state, we plan to > > introduce the > > -following changes: > > - > > -* Change the OS parameters to have three categories: > > - > > - * ``public``: the current behavior. The parameter is logged and stored > > freely. > > - > > - * ``private``: the parameter is saved inside the Ganeti configuration (to > > allow > > - for instance reinstall) but it is not shown in logs, job logs, or > > passed back > > - via RAPI. > > - > > - * ``secret``: the parameter is not saved inside the Ganeti configuration. > > - Reinstalls are impossible unless the data is passed again. The > > parameter will > > - not appear in any log file. When a functionality is performed jointly by > > - multiple daemons (such as MasterD and LuxiD), currently Ganeti sometimes > > - serializes jobs on disk and later reloads them. Secret parameters will > > not be > > - serialized to disk. They will be passed around as part of the LUXI calls > > - exchanged by the daemons, and only kept in memory, in order to reduce > > their > > - accessibility as much as possible. In case of failure of the master > > node, > > - these parameters will be lost and cannot be recovered because they are > > not > > - serialized. As a result, the job cannot be taken over by the new master. > > - This is an expected and accepted side effect of jobs with secret > > parameters: > > - if they fail, they'll have to be restarted manually. > > - > > -* A new OS installation procedure, based on a safe virtualized environment. > > - This virtualized environment will run with the same hardware parameter > > as the > > - actual instance being installed, as much as possible. This will also > > allow to > > - reduce the memory usage in the host (specifically, in Dom0 for Xen > > - installations). Each instance will have these possible execution modes: > > - > > - * ``run``: the default mode, used when the machine is running normally > > and > > - the OS installation procedure is run before starting the instance for > > the > > - first time. > > - > > - * ``self_install``: the first run of the instance will be with a > > different set > > - of parameters w.r.t. all the successive runs. This set of "install > > - parameters" will allow, e.g., to attach an installation > > - floppy/cdrom/network, change the boot device order, or specify an OS > > image > > - to be used. Through this set of parameters, the administrator will > > have to > > - provide the hypervisor a way to find an installation medium for the > > instance > > - (e.g., a boot disk, a network image, etc). This medium will then > > install the > > - instance itself on the disks and will then be responsible to get the > > - parameters for configuring it (its network interfaces, IP address, > > hostname, > > - etc.) from a set of metadata provided by Ganeti (e.g.: using an > > approach > > - comparable to the one of the ``cloud-init`` tool). When this > > installation > > - mode is used, no OS installation script is required. In order for the > > - installation of an OS from an image to be possible, the ``--os-type`` > > - parameter will be extended to support a new additional format: > > ``--os-type > > - image:<URL>`` will instruct Ganeti to take an image from the specified > > - position. For the initial implementation, URL can be either a filename > > or a > > - publically accessible HTTP or FTP resource. Once the instance image is > > - received, it will be dd-ed onto the first disk of the instance. When > > an > > - image is specified, ``--os-parameters`` can still be used, and its > > content > > - will be passed to the instance as part of the metadata. Note that, as > > part > > - of the OS scripts, there is a file specifying what parameters are > > - expected. With OS images, though, none of the traditional structure of > > OS > > - scripts is in place, so there will be no check regarding what > > parameters can > > - be specified: they will all be passed, as long as the > > ``--os-parameters`` > > - string is syntactically valid. The set of ``self_install`` parameters > > will > > - be stored as part of the instance configuration, so that they can be > > used to > > - reinstall the instance. It will be the user's responsibility to > > ensure that > > - the OS image or any installation media is still available in the proper > > - position when a reinstall happens. After the first run, the instance > > will > > - revert to ``run`` mode. > > - > > - * ``install``: Ganeti will start the instance using a virtual appliance > > - specifically made for installing Ganeti instances. Scripts analogous > > to the > > - current ones will run inside this instance. The disks of the instance > > being > > - installed will be connected to this virtual appliance, so that the > > scripts > > - can mount them and modify them as needed, as currently happens, but > > with the > > - additional protection given by this happening in a VM. The disk of the > > - virtual appliance will be read only, so that a pristine copy of the > > - appliance can be started every time a new instance needs to be > > created, to > > - further increase security. The data the instance needs to write at > > runtime > > - will only be stored in RAM, and disappear as soon as the instance is > > - stopped. Metadata will be provided also to this virtual applicance, > > that > > - will take care of converting them to environment variables for the > > - installation scripts. After the first run, the instance will revert to > > - ``run`` mode. > > - > > -* In order to allow for the metadata to be sent inside the instance, a > > - communication mechanism between the instance and the host will be > > created. > > - This mechanism will be bidirectional (e.g.: to allow the setup process > > going > > - on inside the instance to communicate its progress to the host). Each > > instance > > - will have access exclusively to its own metadata, and it will be only > > able to > > - communicate with its host over this channel. More details will be > > provided in > > - the `Communication mechanism and metadata service`_ section. > > - > > -* As part of the instance creation command it will be possible to indicate > > a URL > > - for a "personalization package", that is an archive containing a set of > > files > > - meant to be overlayed on top of the operating system file system at the > > end of > > - the setup process, before the VM is started for the first time in ``run`` > > - mode. Ganeti will provide a mechanism for receiving and unpacking this > > - archive as part of the ``install`` execution mode, whereas in > > ``self_install`` > > - mode it will only be provided as a metadata for the instance to use. The > > - archive will be in TAR-GZIP format (with extension ``.tar.gz`` or > > ``.tgz``) > > - and will contain the files according to the directory structure that > > will be > > - recreated on the installation disk. Files contained in this archive will > > - overwrite files with the same path created during the install procedure > > (if > > - any). The URL of the "personalization package" will have to specify an > > - extesion to identify the file format (in order to allow for more formats > > to be > > - supported in the future). The URL will be stored as part of the > > configuration > > - of the instance (therefore, the URL should not contain confidential > > - information, but the files there available can). It is up to the system > > - administrator to ensure that a package is actually available at that URL > > at > > - install and reinstall time. The content of the package is allowed to > > change. > > - E.g.: a system administrator might create a package containing the > > private > > - keys of the instance being created. When the instance is reinstalled, a > > new > > - package with new keys can be made available there, therefore allowing > > instance > > - reinstall without the need to store keys. Together with the URL, a > > username > > - and a password can be specified to. If the URL is a HTTP(S) URL, they > > will be > > - used as basic access authentication credentials to access that URL. The > > - username and password will not be saved in the config, and will have to > > be > > - provided again in case a reinstall is requested. The downloaded > > - personalization package will not be stored locally on the node for > > longer than > > - it is needed while unpacking it and adding its files to the instance > > being > > - created. The personalization package will be overlayed on top of the > > instance > > - filesystem after the scripts that created it have been executed. In > > order for > > - the files in the package to be automatically overlayed on top of the > > instance > > - filesystem it is required that the appliance is actually able to mount > > the > > - instance disks, therefore this will not work for every filesystem. > > +following changes. > > + > > + > > +OS parameters categories > > +++++++++++++++++++++++++ > > + > > +Change the OS parameters to have three categories: > > + > > +* ``public``: the current behavior. The parameter is logged and stored > > freely. > > + > > +* ``private``: the parameter is saved inside the Ganeti configuration (to > > allow > > + for instance reinstall) but it is not shown in logs, job logs, or passed > > back > > + via RAPI. > > + > > +* ``secret``: the parameter is not saved inside the Ganeti configuration. > > + Reinstalls are impossible unless the data is passed again. The parameter > > will > > + not appear in any log file. When a functionality is performed jointly by > > + multiple daemons (such as MasterD and LuxiD), currently Ganeti sometimes > > + serializes jobs on disk and later reloads them. Secret parameters will > > not be > > + serialized to disk. They will be passed around as part of the LUXI calls > > + exchanged by the daemons, and only kept in memory, in order to reduce > > their > > + accessibility as much as possible. In case of failure of the master node, > > + these parameters will be lost and cannot be recovered because they are > > not > > + serialized. As a result, the job cannot be taken over by the new master. > > This > > + is an expected and accepted side effect of jobs with secret parameters: > > if > > + they fail, they'll have to be restarted manually. > > + > > + > > +Metadata > > +++++++++ > > + > > +In order to allow metadata to be sent inside the instance, a communication > > +mechanism between the instance and the host will be created. This > > mechanism > > +will be bidirectional (e.g.: to allow the setup process going on inside the > > +instance to communicate its progress to the host). Each instance will have > > +access exclusively to its own metadata, and it will be only able to > > communicate > > +with its host over this channel. This is the approach followed the > > +``cloud-init`` tool and more details will be provided in the `Communication > > +mechanism and metadata service`_ section. > > + > > + > > +Installation procedure > > +++++++++++++++++++++++ > > + > > +A new installation procedure will be introduced, with which it will be > > possible > > +to use an installation medium and run the OS scripts in an optional > > virtualized > > +environment and with an optional personalization package. There will be > > two > > +sets of parameters, namely, installation parameters, which are used mainly > > for > > +installs and reinstalls, and execution parameters, which are used in all > > the > > +other runs that are not part of an installation procedure. > > + > > +This set of installation parameters will allow, e.g., to attach an > > installation > > +floppy/cdrom/network, change the boot device order, or specify a disk > > image to > > +be used. Through this set of parameters, the administrator will have to > > provide > > +the hypervisor a location for an installation medium for the instance > > (e.g., a > > +boot disk, a network image, etc). This medium will carry out the > > installation > > +of the instance onto the instance's disks and will then be responsible for > > +getting the parameters for configuring the instance, such as, network > > +interfaces, IP address, and hostname. These parameters are taken from the > > +metadata. The installation parameters will be stored in the configuration > > of > > +Ganeti and used in future reinstalls, but not during normal execution. > > + > > +The instance is reinstalled using the same installation parameters from the > > +first installation. However, it will be the administrator's > > responsibility to > > +ensure that the any installation media is still available at the proper > > location > > +when a reinstall occurs. > > + > > +The parameter ``--os-parameters`` can still be used to specify the OS > > +parameters. However, without OS scripts, Ganeti cannot do more than a > > syntactic > > +check to validate the supplied OS parameters string. As a result, this > > string > > +will be directly passed to the instance as part of the metadata. If the > > +installation procedure is running inside a virtualized environment, then > > Ganeti > > +will take these parameters from the metadata and pass them to the OS > > scripts as > > +environment variables. > > + > > +* Use a disk image: > > + > > + Currently, it is already possible to specify an installation medium, > > such as, > > + a cdrom, but not a disk image. Therefore, a new parameter > > ``--os-image`` will > > + be used to specify the location of a disk image which will be dumped to > > the > > + instance's first disk before the instance is started. The location of > > the > > + image can be a URL and, if this is the case, Ganeti will download this > > image. > > + > > +* Run OS scripts: > > + > > + The parameter ``--os-type`` (short version: ``-o``), is currently used to > > + specify the OS scripts. This parameter will still be used to specify > > the OS > > + scripts with the difference that these OS scripts may optionally run > > inside a > > + virtualized environment for safety reasons, depending on whether they are > > + trusted or not. For more details on trusted and untrusted OS scripts, > > refer > > + to the `Installation process in a virtualized environment`_ section. > > + > > +* Personalization package > > + > > + As part of the instance creation command, it will be possible to > > indicate a > > + URL for a "personalization package", which is an archive containing a > > set of > > + files meant to be overlayed on top of the OS file system at the end of > > the > > + setup process and before the VM is started for the first time in normal > > mode. > > + Ganeti will provide a mechanism for receiving and unpacking this archive > > + whether the installation is being performed inside the virtualized > > environment > > + or not. > > + > > + The archive will be in TAR-GZIP format (with extension ``.tar.gz`` or > > + ``.tgz``) and contain the files according to the directory structure > > that will > > + be recreated on the installation disk. Files contained in this archive > > will > > + overwrite files with the same path created during the installation > > procedure > > + (if any). The URL of the "personalization package" will have to specify > > an > > + extension to identify the file format (in order to allow for more > > formats to > > + be supported in the future). The URL will be stored as part of the > > + configuration of the instance (therefore, the URL should not contain > > + confidential information, but the files there available can). > > + > > + It is up to the system administrator to ensure that a package is actually > > + available at that URL at install and reinstall time. The contents of the > > + package are allowed to change. E.g.: a system administrator might > > create a > > + package containing the private keys of the instance being created. When > > the > > + instance is reinstalled, a new package with new keys can be made > > available > > + there, thus allowing instance reinstall without the need to store keys. > > A > > + username and a password can be specified together with the URL. If the > > URL is > > + a HTTP(S) URL, they will be used as basic access authentication > > credentials to > > + access that URL. The username and password will not be saved in the > > config, > > + and will have to be provided again in case a reinstall is requested. > > + > > + The downloaded personalization package will not be stored locally on the > > node > > + for longer than it is needed while unpacking it and adding its files to > > the > > + instance being created. The personalization package will be overlayed > > on top > > + nof the instance filesystem after the scripts that created it have been > > s/nof/of
Fixed. Thanks, Jose > > > + executed. In order for the files in the package to be automatically > > overlayed > > + on top of the instance filesystem, it is required that the appliance is > > + actually able to mount the instance's disks. As a result, this will not > > work > > + for every filesystem. > > + > > +* Combine a disk image, OS scripts, and a personalization package > > + > > + It will possible to combine a disk image, OS scripts, and a > > personalization > > + package, both with or without a virtualized environment. There is one > > + exception which is if there are untrusted OS scripts. At least, an > > + installation medium or OS scripts should be specified. > > + > > + The disk image of the actual virtual appliance, which bootstraps the > > virtual > > + environment used in the installation procedure, will be read only, so > > that a > > + pristine copy of the appliance can be started every time a new instance > > needs > > + to be created and to further increase security. The data the instance > > needs > > + to write at runtime will only be stored in RAM and disappear as soon as > > the > > + instance is stopped. > > + > > + The parameter ``--enable-safe-install=yes|no`` will be used to give the > > + administrator control over whether to use a virtualized environment for > > the > > + installation procedure. By default, a virtualized environment will be > > used. > > + Note that some feature combinations, such as, using untrusted scripts, > > will > > + require the virtualized environment. In this case, Ganeti will not allow > > + disabling the virtualized environment. > > > > Implementation > > ============== > > @@ -203,8 +235,8 @@ of increasing impact on the system and, in some cases, > > dependent on each other: > > #. Communication mechanism between host and instance > > #. Metadata service > > #. Personalization package (inside a virtualization environment) > > -#. ``self_install`` mode > > -#. ``install`` mode (inside a virtualization environment) > > +#. Instance creation via a disk image > > +#. Instance creation inside a virtualized environment > > > > Some of these steps need to be more deeply specified w.r.t. what is already > > written in the `Proposed changes`_ Section. Extra details will be provided > > in > > @@ -214,21 +246,21 @@ Communication mechanism and metadata service > > ++++++++++++++++++++++++++++++++++++++++++++ > > > > The communication mechanism and the metadata service are described together > > -because they are deeply tied. On the other hand, the communication > > mechanism > > -will need to be more generic because it can be used for other reasons in > > the > > -future (like allowing instances to explicitly send commands to Ganeti, or > > to let > > -Ganeti control a helper instance, like the one hereby introduced for > > performing > > -OS installs inside a safe environment). > > - > > -The communication mechanism will be enabled automatically when the > > instance is > > -in ``self_install`` or ``install`` mode, but for backwards compatibility > > it will > > -be disabled when the instance is in ``run`` mode unless it is explicitly > > -requested. Specifically, a new parameter ``--communication`` (short > > version: > > -``-C``), with possible values ``true`` or ``false`` will be added to > > +because they are deeply tied. The communication mechanism will be made more > > +generic because it can be used for other purposes in the future (like > > allowing > > +instances to explicitly send commands to Ganeti, or to let Ganeti control a > > +helper instance, like the one hereby introduced for performing OS installs > > +inside a safe environment). > > + > > +The communication mechanism will be enabled automatically during an > > installation > > +procedure that requires a virtualized environment, but for backwards > > +compatibility it will be disabled when the instance is running normally, > > unless > > +it is explicitly requested. Specifically, a new parameter > > +``--communication=yes|no`` (short version: ``-C``) will be added to > > ``gnt-instance add`` and ``gnt-instance modify``. It will determine > > whether the > > -instance will have a communication channel set up to interact with the > > host and > > -to receive metadata. The value of this parameter will be saved as part of > > the > > -configuration of the instance. > > +instance has a communication channel set to interact with the host and > > receive > > +metadata. The value of this parameter will be saved as part of the > > configuration > > +of the instance. > > > > When the communication mechanism is enabled, Ganeti will create a new > > network > > interface inside the instance. This additional network interface will be > > the > > @@ -313,10 +345,10 @@ pair ``(<value>, <visibility>)`` as the value, where > > ``<value>`` is the > > user-provided value of the parameter, and ``<visibility>`` is either > > ``public``, > > ``private`` or ``secret``. > > > > -The installation scripts to be run inside the virtualized environment > > while the > > -instance is run in ``install`` mode will be available at:: > > +The installation scripts to be run inside the virtualized environment will > > be > > +available at:: > > > > - http://169.254.169.254/<version>/ganeti/os/scripts/<script_name> > > + http://169.254.169.254/ganeti/<version>/os/scripts/<script_name> > > > > where ``<script_name>`` is the name of the script. > > > > > > > > On Mon, Dec 16, 2013 at 03:09:36PM +0100, Michele Tartara wrote: > >> On Mon, Dec 16, 2013 at 2:32 PM, Jose A. Lopes <[email protected]> > >> wrote: > >> > Hi, > >> > > >> > After recent discussions with Michele and Constantinos, here is an > >> > update to the OS installation design doc allowing combination of disk > >> > images, OS scripts, and personalization packages. > >> > > >> > Below is the (insane) interdiff and in attachment the whole design doc. > >> > > >> > Interdiff: > >> > > >> > diff --git a/doc/design-os.rst b/doc/design-os.rst > >> > index d805a68..17cc059 100644 > >> > --- a/doc/design-os.rst > >> > +++ b/doc/design-os.rst > >> > @@ -18,7 +18,7 @@ perform all the OS-related functionality (setting up > >> > an operating system inside > >> > the disks of the instance being created, exporting/importing the > >> > instance, > >> > renaming it). > >> > > >> > -These scripts receive, as environment variables, a fixed set of > >> > parameters > >> > +These scripts receive through environment variables a fixed set of > >> > parameters > >> > related to the instance (such as the hypervisor, the name of the > >> > instance, the > >> > number of disks, and their location) and a set of user defined > >> > parameters. > >> > These parameters are also written in the configuration file of Ganeti, > >> > to allow > >> > @@ -72,126 +72,160 @@ Proposed changes > >> > ================ > >> > > >> > In order to fix the shortcomings of the current state, we plan to > >> > introduce the > >> > -following changes: > >> > - > >> > -* Change the OS parameters to have three categories: > >> > - > >> > - * ``public``: the current behavior. The parameter is logged and stored > >> > freely. > >> > - > >> > - * ``private``: the parameter is saved inside the Ganeti configuration > >> > (to allow > >> > - for instance reinstall) but it is not shown in logs, job logs, or > >> > passed back > >> > - via RAPI. > >> > - > >> > - * ``secret``: the parameter is not saved inside the Ganeti > >> > configuration. > >> > - Reinstalls are impossible unless the data is passed again. The > >> > parameter will > >> > - not appear in any log file. When a functionality is performed > >> > jointly by > >> > - multiple daemons (such as MasterD and LuxiD), currently Ganeti > >> > sometimes > >> > - serializes jobs on disk and later reloads them. Secret parameters > >> > will not be > >> > - serialized to disk. They will be passed around as part of the LUXI > >> > calls > >> > - exchanged by the daemons, and only kept in memory, in order to > >> > reduce their > >> > - accessibility as much as possible. In case of failure of the master > >> > node, > >> > - these parameters will be lost and cannot be recovered because they > >> > are not > >> > - serialized. As a result, the job cannot be taken over by the new > >> > master. > >> > - This is an expected and accepted side effect of jobs with secret > >> > parameters: > >> > - if they fail, they'll have to be restarted manually. > >> > - > >> > -* A new OS installation procedure, based on a safe virtualized > >> > environment. > >> > - This virtualized environment will run with the same hardware > >> > parameter as the > >> > - actual instance being installed, as much as possible. This will also > >> > allow to > >> > - reduce the memory usage in the host (specifically, in Dom0 for Xen > >> > - installations). Each instance will have these possible execution > >> > modes: > >> > - > >> > - * ``run``: the default mode, used when the machine is running > >> > normally and > >> > - the OS installation procedure is run before starting the instance > >> > for the > >> > - first time. > >> > - > >> > - * ``self_install``: the first run of the instance will be with a > >> > different set > >> > - of parameters w.r.t. all the successive runs. This set of "install > >> > - parameters" will allow, e.g., to attach an installation > >> > - floppy/cdrom/network, change the boot device order, or specify an > >> > OS image > >> > - to be used. Through this set of parameters, the administrator will > >> > have to > >> > - provide the hypervisor a way to find an installation medium for the > >> > instance > >> > - (e.g., a boot disk, a network image, etc). This medium will then > >> > install the > >> > - instance itself on the disks and will then be responsible to get the > >> > - parameters for configuring it (its network interfaces, IP address, > >> > hostname, > >> > - etc.) from a set of metadata provided by Ganeti (e.g.: using an > >> > approach > >> > - comparable to the one of the ``cloud-init`` tool). When this > >> > installation > >> > - mode is used, no OS installation script is required. In order for > >> > the > >> > - installation of an OS from an image to be possible, the > >> > ``--os-type`` > >> > - parameter will be extended to support a new additional format: > >> > ``--os-type > >> > - image:<URL>`` will instruct Ganeti to take an image from the > >> > specified > >> > - position. For the initial implementation, URL can be either a > >> > filename or a > >> > - publically accessible HTTP or FTP resource. Once the instance image > >> > is > >> > - received, it will be dd-ed onto the first disk of the instance. > >> > When an > >> > - image is specified, ``--os-parameters`` can still be used, and its > >> > content > >> > - will be passed to the instance as part of the metadata. Note that, > >> > as part > >> > - of the OS scripts, there is a file specifying what parameters are > >> > - expected. With OS images, though, none of the traditional structure > >> > of OS > >> > - scripts is in place, so there will be no check regarding what > >> > parameters can > >> > - be specified: they will all be passed, as long as the > >> > ``--os-parameters`` > >> > - string is syntactically valid. The set of ``self_install`` > >> > parameters will > >> > - be stored as part of the instance configuration, so that they can > >> > be used to > >> > - reinstall the instance. It will be the user's responsibility to > >> > ensure that > >> > - the OS image or any installation media is still available in the > >> > proper > >> > - position when a reinstall happens. After the first run, the > >> > instance will > >> > - revert to ``run`` mode. > >> > - > >> > - * ``install``: Ganeti will start the instance using a virtual > >> > appliance > >> > - specifically made for installing Ganeti instances. Scripts > >> > analogous to the > >> > - current ones will run inside this instance. The disks of the > >> > instance being > >> > - installed will be connected to this virtual appliance, so that the > >> > scripts > >> > - can mount them and modify them as needed, as currently happens, but > >> > with the > >> > - additional protection given by this happening in a VM. The disk of > >> > the > >> > - virtual appliance will be read only, so that a pristine copy of the > >> > - appliance can be started every time a new instance needs to be > >> > created, to > >> > - further increase security. The data the instance needs to write at > >> > runtime > >> > - will only be stored in RAM, and disappear as soon as the instance is > >> > - stopped. Metadata will be provided also to this virtual applicance, > >> > that > >> > - will take care of converting them to environment variables for the > >> > - installation scripts. After the first run, the instance will revert > >> > to > >> > - ``run`` mode. > >> > - > >> > -* In order to allow for the metadata to be sent inside the instance, a > >> > - communication mechanism between the instance and the host will be > >> > created. > >> > - This mechanism will be bidirectional (e.g.: to allow the setup > >> > process going > >> > - on inside the instance to communicate its progress to the host). Each > >> > instance > >> > - will have access exclusively to its own metadata, and it will be only > >> > able to > >> > - communicate with its host over this channel. More details will be > >> > provided in > >> > - the `Communication mechanism and metadata service`_ section. > >> > - > >> > -* As part of the instance creation command it will be possible to > >> > indicate a URL > >> > - for a "personalization package", that is an archive containing a set > >> > of files > >> > - meant to be overlayed on top of the operating system file system at > >> > the end of > >> > - the setup process, before the VM is started for the first time in > >> > ``run`` > >> > - mode. Ganeti will provide a mechanism for receiving and unpacking > >> > this > >> > - archive as part of the ``install`` execution mode, whereas in > >> > ``self_install`` > >> > - mode it will only be provided as a metadata for the instance to use. > >> > The > >> > - archive will be in TAR-GZIP format (with extension ``.tar.gz`` or > >> > ``.tgz``) > >> > - and will contain the files according to the directory structure that > >> > will be > >> > - recreated on the installation disk. Files contained in this archive > >> > will > >> > - overwrite files with the same path created during the install > >> > procedure (if > >> > - any). The URL of the "personalization package" will have to specify > >> > an > >> > +following changes. > >> > + > >> > + > >> > +OS parameters categories > >> > +++++++++++++++++++++++++ > >> > + > >> > +Change the OS parameters to have three categories: > >> > + > >> > +* ``public``: the current behavior. The parameter is logged and stored > >> > freely. > >> > + > >> > +* ``private``: the parameter is saved inside the Ganeti configuration > >> > (to allow > >> > + for instance reinstall) but it is not shown in logs, job logs, or > >> > passed back > >> > + via RAPI. > >> > + > >> > +* ``secret``: the parameter is not saved inside the Ganeti > >> > configuration. > >> > + Reinstalls are impossible unless the data is passed again. The > >> > parameter will > >> > + not appear in any log file. When a functionality is performed > >> > jointly by > >> > + multiple daemons (such as MasterD and LuxiD), currently Ganeti > >> > sometimes > >> > + serializes jobs on disk and later reloads them. Secret parameters > >> > will not be > >> > + serialized to disk. They will be passed around as part of the > >> > LUXI calls > >> > + exchanged by the daemons, and only kept in memory, in order to > >> > reduce their > >> > + accessibility as much as possible. In case of failure of the > >> > master node, > >> > + these parameters will be lost and cannot be recovered because they > >> > are not > >> > + serialized. As a result, the job cannot be taken over by the new > >> > master. This > >> > + is an expected and accepted side effect of jobs with secret > >> > parameters: if > >> > + they fail, they'll have to be restarted manually. > >> > + > >> > + > >> > +Metadata > >> > +++++++++ > >> > + > >> > +In order to allow metadata to be sent inside the instance, a > >> > communication > >> > +mechanism between the instance and the host will be created. This > >> > mechanism > >> > +will be bidirectional (e.g.: to allow the setup process going on > >> > inside the > >> > +instance to communicate its progress to the host). Each instance > >> > will have > >> > +access exclusively to its own metadata, and it will be only able to > >> > communicate > >> > +with its host over this channel. More details will be provided > >> > in the > >> > +`Communication mechanism and metadata service`_ section. > >> > + > >> > + > >> > +Installation procedure > >> > +++++++++++++++++++++++ > >> > + > >> > +A new installation procedure will be introduced, with which it will be > >> > possible > >> > +to use a disk image and/or run the OS scripts in an optional > >> > virtualized > >> > +environment and with an optional personalization package. Apart from > >> > installs > >> > +and reinstalls, the instance will run just as it does in the > >> > current Ganeti > >> > +design. > >> > + > >> > +* Use a disk image: > >> > + > >> > + The parameter ``--os-image`` will be used to specify the location > >> > of a disk > >> > + image which will be dumped to the instance's first disk before the > >> > instance is > >> > + started. The location of the image can be a URL and, if this is > >> > the case, > >> > + Ganeti will download this image. > >> > + > >> > + The parameter ``--os-parameters`` can still be used to specify > >> > the OS > >> > + parameters. Note that the OS scripts contain a file that specifies > >> > what the > >> > + valid OS parameters are. However, with disk images none of the > >> > traditional > >> > + structure of OS scripts is in place, therefore, it will not be > >> > possible to > >> > + validate the supplied OS parameters. As a result, Ganeti will only > >> > check if > >> > + the OS parameters string is syntactically valid and, if this is > >> > the case, > >> > + these parameters will be directly passed to the instance as > >> > part of the > >> > + metadata. > >> > + > >> > + After the first run, the instance will be restarted without the > >> > disk image > >> > + parameter, but the OS parameters are stored in the configuration of > >> > Ganeti and > >> > + used in future restarts. When reinstalling an instance, both > >> > the OS > >> > + parameters and the disk image parameter will be made available. > >> > >> This shouldn't be limited to OS parameters and disk image. There > >> should be two completely separate sets of parameters for the "install" > >> mode and the normal execution modes, because the user might want to > >> specify different things (network interfaces for PXE installs, cdrom > >> images, and so on, that are useful only during the installation). > >> > >> From hints here and there is seems like you are thinking of this, but > >> it's not explicitly written. > >> > >> > However, it > >> > + will be the administrator's responsibility to ensure that the disk > >> > image or > >> > + any installation media is still available at the proper > >> > location when a > >> > + reinstall happens. > >> > + > >> > +* Run OS scripts: > >> > + > >> > + The parameter ``--os-type``, or the abbreviation ``-o``, is > >> > currently used to > >> > + specify the OS scripts. This parameter will still be used to > >> > specify the OS > >> > + scripts with the difference that these OS scripts may optionally run > >> > inside a > >> > + virtualized environment for safety reasons, depending on whether > >> > they are > >> > + trusted or not. For more details on trusted and untrusted OS > >> > scripts, refer > >> > + to the `Installation process in a virtualized environment`_ section. > >> > + > >> > + The parameter ``--os-parameters`` can still be used and, if > >> > the safe > >> > + virtualized environment is used to run the OS scripts, the OS > >> > parameters will > >> > + be passed to this environment as part of the metadata. Then, Ganeti > >> > will take > >> > + these parameters from the metadata and pass them to the OS > >> > scripts as > >> > + environment variables, thus making sure that these scripts can > >> > be reused > >> > + independently of whether they are run inside the virtualized > >> > environment or > >> > + not. > >> > >> Isn't this just rewritten with only slightly different language from a > >> few lines above? > >> > >> > + > >> > +* Personalization package > >> > + > >> > + As part of the instance creation command it will be possible to > >> > indicate a URL > >> > + for a "personalization package", which is an archive containing a set > >> > of files > >> > + meant to be overlayed on top of the OS file system at the end of > >> > the setup > >> > + process and before the VM is started for the first time in > >> > normal mode. > >> > + Ganeti will provide a mechanism for receiving and unpacking this > >> > archive > >> > + whether the installation is being performed inside the virtualized > >> > environment > >> > + or not. > >> > + > >> > + The archive will be in TAR-GZIP format (with extension > >> > ``.tar.gz`` or > >> > + ``.tgz``) and contain the files according to the directory structure > >> > that will > >> > + be recreated on the installation disk. Files contained in this > >> > archive will > >> > + overwrite files with the same path created during the installation > >> > procedure > >> > + (if any). The URL of the "personalization package" will have to > >> > specify an > >> > extesion to identify the file format (in order to allow for more > >> > formats to be > >> > >> Please, fix the pre-existing typo s/extesion/extension/ while > >> re-submitting the patch. > >> > >> > supported in the future). The URL will be stored as part of the > >> > configuration > >> > - of the instance (therefore, the URL should not contain confidential > >> > - information, but the files there available can). It is up to the > >> > system > >> > - administrator to ensure that a package is actually available at that > >> > URL at > >> > - install and reinstall time. The content of the package is allowed to > >> > change. > >> > - E.g.: a system administrator might create a package containing the > >> > private > >> > - keys of the instance being created. When the instance is reinstalled, > >> > a new > >> > - package with new keys can be made available there, therefore allowing > >> > instance > >> > - reinstall without the need to store keys. Together with the URL, a > >> > username > >> > - and a password can be specified to. If the URL is a HTTP(S) URL, they > >> > will be > >> > - used as basic access authentication credentials to access that URL. > >> > The > >> > - username and password will not be saved in the config, and will have > >> > to be > >> > - provided again in case a reinstall is requested. The downloaded > >> > - personalization package will not be stored locally on the node for > >> > longer than > >> > - it is needed while unpacking it and adding its files to the instance > >> > being > >> > - created. The personalization package will be overlayed on top of the > >> > instance > >> > - filesystem after the scripts that created it have been executed. In > >> > order for > >> > - the files in the package to be automatically overlayed on top of the > >> > instance > >> > - filesystem it is required that the appliance is actually able to > >> > mount the > >> > - instance disks, therefore this will not work for every filesystem. > >> > + of the instance (therefore, the URL should not contain > >> > confidential > >> > + information, but the files there available can). > >> > + > >> > + It is up to the system administrator to ensure that a package is > >> > actually > >> > + available at that URL at install and reinstall time. The > >> > contents of the > >> > + package are allowed to change. E.g.: a system administrator might > >> > create a > >> > + package containing the private keys of the instance being created. > >> > When the > >> > + instance is reinstalled, a new package with new keys can be made > >> > available > >> > + there, thus allowing instance reinstall without the need to store > >> > keys. A > >> > + username and a password can be specified together with the URL. If > >> > the URL is > >> > + a HTTP(S) URL, they will be used as basic access authentication > >> > credentials to > >> > + access that URL. The username and password will not be saved in > >> > the config, > >> > + and will have to be provided again in case a reinstall is > >> > requested. > >> > + > >> > + The downloaded personalization package will not be stored locally on > >> > the node > >> > + for longer than it is needed while unpacking it and adding its > >> > files to the > >> > + instance being created. The personalization package will be > >> > overlayed on top > >> > + of the instance filesystem after the scripts that created it > >> > have been > >> > + executed. In order for the files in the package to be automatically > >> > overlayed > >> > + on top of the instance filesystem, it is required that the > >> > appliance is > >> > + actually able to mount the instance's disks. As a result, this will > >> > not work > >> > + for every filesystem. > >> > + > >> > +* Combine a disk image, OS scripts, and a personalization package > >> > + > >> > + It will possible to combine a disk image, OS scripts, and a > >> > personalization > >> > + package, both with or without a virtualized environment. There > >> > is one > >> > + exception which is if there are untrusted OS scripts. In this > >> > case, the > >> > + virtualized environment must be used. Whether or not a > >> > virtualized > >> > + environment is used, the installation goes as follows: first, the > >> > disk image > >> > + (if specified) is dumped to the instance's first disk, then the OS > >> > scripts (if > >> > + specified) are run, and finally the personalization package (if > >> > specified) is > >> > + overlayed. At least, a disk image or OS scripts should be specified. > >> > >> The ordered sequence of steps is already written before. It's probably > >> useless to write it again here. > >> > >> > + > >> > + The disk image of the actual virtual appliance, which bootstraps > >> > the virtual > >> > + environment used in the installation procedure, will be read only, > >> > so that a > >> > + pristine copy of the appliance can be started every time a new > >> > instance needs > >> > + to be created and to further increase security. The data the > >> > instance needs > >> > + to write at runtime will only be stored in RAM and disappear as > >> > soon as the > >> > + instance is stopped. > >> > + > >> > + The parameter ``--enable-safe-install`` will be used to give the > >> > administrator > >> > + control over whether to use a virtualized environment for the > >> > installation > >> > + procedure. By default, a virtualized environment will be used. > >> > >> Please, specify that it is ``--enable-safe-install=yes|no``, otherwise > >> the name seems weird given that it already defaults to ``yes``. > >> > >> >Note that > >> > + some feature combinations, such as, using untrusted scripts, will > >> > require the > >> > + virtualized environment. In this case, Ganeti will not allow > >> > disabling the > >> > + virtualized environment. > >> > > >> > Implementation > >> > ============== > >> > @@ -203,8 +237,8 @@ of increasing impact on the system and, in some > >> > cases, dependent on each other: > >> > #. Communication mechanism between host and instance > >> > #. Metadata service > >> > #. Personalization package (inside a virtualization environment) > >> > -#. ``self_install`` mode > >> > -#. ``install`` mode (inside a virtualization environment) > >> > +#. Instance creation via a disk image > >> > +#. Instance creation inside a virtualized environment > >> > > >> > Some of these steps need to be more deeply specified w.r.t. what is > >> > already > >> > written in the `Proposed changes`_ Section. Extra details will be > >> > provided in > >> > @@ -214,20 +248,20 @@ Communication mechanism and metadata service > >> > ++++++++++++++++++++++++++++++++++++++++++++ > >> > > >> > The communication mechanism and the metadata service are described > >> > together > >> > -because they are deeply tied. On the other hand, the communication > >> > mechanism > >> > -will need to be more generic because it can be used for other reasons > >> > in the > >> > -future (like allowing instances to explicitly send commands to Ganeti, > >> > or to let > >> > -Ganeti control a helper instance, like the one hereby introduced for > >> > performing > >> > -OS installs inside a safe environment). > >> > - > >> > -The communication mechanism will be enabled automatically when the > >> > instance is > >> > -in ``self_install`` or ``install`` mode, but for backwards > >> > compatibility it will > >> > -be disabled when the instance is in ``run`` mode unless it is explicitly > >> > -requested. Specifically, a new parameter ``--communication`` (short > >> > version: > >> > -``-C``), with possible values ``true`` or ``false`` will be added to > >> > -``gnt-instance add`` and ``gnt-instance modify``. It will determine > >> > whether the > >> > -instance will have a communication channel set up to interact with the > >> > host and > >> > -to receive metadata. The value of this parameter will be saved as part > >> > of the > >> > +because they are deeply tied. The communication mechanism will be made > >> > more > >> > +generic because it can be used for other purposes in the future (like > >> > allowing > >> > +instances to explicitly send commands to Ganeti, or to let Ganeti > >> > control a > >> > +helper instance, like the one hereby introduced for performing OS > >> > installs > >> > +inside a safe environment). > >> > + > >> > +The communication mechanism will be enabled automatically during an > >> > installation > >> > +procedure that requires a virtualized environment, but for backwards > >> > +compatibility it will be disabled when the instance is running > >> > normally, unless > >> > +it is explicitly requested. Specifically, a new parameter > >> > ``--communication`` > >> > +(short version: ``-C``), with possible values ``true`` or ``false`` > >> > will be > >> > >> I'd use yes|no for uniformity with the rest of the code. > >> > >> > +added to ``gnt-instance add`` and ``gnt-instance modify``. It will > >> > determine > >> > +whether the instance has a communication channel set to interact with > >> > the host > >> > +and receive metadata. The value of this parameter will be saved as part > >> > of the > >> > configuration of the instance. > >> > > >> > When the communication mechanism is enabled, Ganeti will create a new > >> > network > >> > @@ -313,8 +347,8 @@ pair ``(<value>, <visibility>)`` as the value, where > >> > ``<value>`` is the > >> > user-provided value of the parameter, and ``<visibility>`` is either > >> > ``public``, > >> > ``private`` or ``secret``. > >> > > >> > -The installation scripts to be run inside the virtualized environment > >> > while the > >> > -instance is run in ``install`` mode will be available at:: > >> > +The installation scripts to be run inside the virtualized environment > >> > will be > >> > +available at:: > >> > > >> > http://169.254.169.254/<version>/ganeti/os/scripts/<script_name> > >> > >> This should be http://169.254.169.254/ganeti/<version>/ as all the > >> other ones. Apparently, it slipped through the previous review cycles. > >> Could you please fix it while re-submitting? > >> > >> > >> > -- > >> > Jose Antonio Lopes > >> > Ganeti Engineering > >> > Google Germany GmbH > >> > Dienerstr. 12, 80331, München > >> > > >> > Registergericht und -nummer: Hamburg, HRB 86891 > >> > Sitz der Gesellschaft: Hamburg > >> > Geschäftsführer: Graham Law, Christine Elizabeth Flores > >> > Steuernummer: 48/725/00206 > >> > Umsatzsteueridentifikationsnummer: DE813741370 > >> > >> > >> Really good summary of all the discussions of the last few days. > >> Thanks! > >> > >> Michele > >> > >> -- > >> Google Germany GmbH > >> Dienerstr. 12 > >> 80331 München > >> > >> Registergericht und -nummer: Hamburg, HRB 86891 > >> Sitz der Gesellschaft: Hamburg > >> Geschäftsführer: Graham Law, Christine Elizabeth Flores > > > > -- > > Jose Antonio Lopes > > Ganeti Engineering > > Google Germany GmbH > > Dienerstr. 12, 80331, München > > > > Registergericht und -nummer: Hamburg, HRB 86891 > > Sitz der Gesellschaft: Hamburg > > Geschäftsführer: Graham Law, Christine Elizabeth Flores > > Steuernummer: 48/725/00206 > > Umsatzsteueridentifikationsnummer: DE813741370 > > Rest LGTM. > > Thanks, > Michele > > -- > Google Germany GmbH > Dienerstr. 12 > 80331 München > > Registergericht und -nummer: Hamburg, HRB 86891 > Sitz der Gesellschaft: Hamburg > Geschäftsführer: Graham Law, Christine Elizabeth Flores -- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
