A revised proposal taking Robin's comments into account is attached.Title: Proposal to Consolidate the Pulp 3 Installers
Proposal to Consolidate the Pulp 3 Installers
What?
Currently, one installer is used by the Pulp developers, and another installer is used by Pulp QE and end users. I propose that a single installer be used by all three audiences. This document explains why I think this change should be made, and it outlines a possible plan of action for making this change.
Why?
Why should a single installer be used? The most succinct justification is that using a single installer improves dev/prod parity, where dev/prod parity means "[keeping] development, staging, and production as similar as possible." In our case, QE serves as staging, and our users serve as production.
More verbosely, a single installer should be used because doing so will likely:
- Produce a better installer.
- Reduce the workload on developers and quality engineers.
- Improve Pulp's public image.
- Encourage community involvement.
- Allow for additional installation strategies.
Better Installer, Less Work
To illustrate the first two points, consider the scenario where a quality engineer discovers an issue with the installer they're using.
The engineer is obliged to check and see if the issue is also present in the second installer. If so, the engineer will either file one ticket that mentions both installers, or they will file two independent tickets, depending on their mindset. When a developer decides to work on the issues, they need to develop two similar fixes for two similar installers, they need to test both changes, and they need to file two pull requests. After the pull requests have been merged, the developer needs to either close out one or two tickets.
This workflow is logically sound. So long as all parties are mindful of what they're doing, nothing will go wrong. But humans aren't like that! Consider the case of Pulp #3031. That ticket describes how the end user installer inserts junk text into /etc/pulp/server.yaml. I filed this ticket on September 25th, and I submitted a fix for the issue on the same day. Unfortunately, I forgot to file a second ticket noting that the exact same issue is also present in the developer installer. As of this writing on November 21st, the developer installer is affected by this issue. The developer installer still works due to the particular nature of the bug, but at any time, an unrelated change to the configuration file could break the development environment.
Improve Pulp's Public Image
To illustrate the third point, consider what happened when the Pulp 3 plugin API alpha was released. The announcement was pushed back by a day or two so that QE could have a chance to play with the product before it went out the door. QE discovered several issues, one of them being the fact that the end user installer simply did not work. (The issue was fixed by pulp/devel 396c93e775542484b585b8841bc9adc042b71731.) This gives the impression that the end user installer is a second class product. So long as we have two non-orthogonal installers, Pulp's public image is vulnerable to incidents like this.
Encourage Community Involvement
To illustrate the fourth point, consider some of the differences between the end user and developer installers. Using the end user installer requires the following:
Decide where you would like to install Pulp. You could install it on your local system, on a beaker system, on a libvirt + kvm VM, on a libvirt + virtualbox VM, on a Linode VM, on a DigitalOcean VM, or just about anywhere else.
Get shell access to the system and execute the following:
setenforce 0 systemctl stop firewalld # or open appropriate portsFrom your local system, execute the following:
git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy install -r requirements.yml -p roles ansible-galaxy deploy-pulp3.yml -i pulp3.example.com,In contrast, using the developer installer requires the following:
Set up Vagrant. A development environment can't be deployed on any other type of host. You're SOL if you'd like to use some other type of back-end.
From your local system, execute [1] the following:
git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel vagrant upThose familiar with Pulp's installers might also notice that the end user installer doesn't install any plug-ins, whereas the developer installer installs the file plugin.
Why should a prospective contributor need to learn two different installation work-flows, where those two work-flows require entirely different sets of infrastructure, and where the two work-flows install different Pulp 3 components?
More Installation Strategies
The developer installer installs Pulp directly from source in editable mode, along with numerous extra tools, and it enables settings like debug mode. The end user installer installs Pulp from PyPI packages, without extra tools, and without enabling settings like debug mode.
Unfortunately, these two installers don't cover all of our needs. PyPI packages are — quite reasonably — updated once a week. As Pulp 3 matures, these releases may become less frequent. As a result, our Jenkins test runs may fail to detect regressions for a up to a week, and tests for fixed-but-not-released features may produce incorrect results for a up to a week. One solution to this quandary is to create a third installer that installs Pulp directly from source, without extra tools, and without enabling settings like debug mode. As an obvious example, consider Pulp #3107:
Switch JWT auth token string from 'JWT' to 'Bearer'I had to explicitly request that new PyPI packages be uploaded after this change was made. In the future, requests like this will occur again. If a from-source installer is available, then such requests are obviated.
How can we support this third installation use case with our current set-up? Should we create a third installer? That makes every other problem listed above even worse. It would be much better if we took an existing installer and added an option so that it could fulfill either use case. If we're going to do that, why not go even further and let an existing installer handle all three use cases?
How?
In concrete terms, how could the installer described above be implemented?
Technology
Currently, both of our installers use Ansible. I suggest that we continue using Ansible to distribute Pulp. There are several key advantages to doing this:
- We can re-use some of our existing Ansible assets when developing a converged installer.
- Ansible is an official Red Hat product. Using Ansible may promote adoption of Pulp 3 within the business.
- Ansible is platform-agnostic. If someone wants to install Pulp on an unsupported platform (such as CentOS, Debian, an old version of Fedora, etc.), they can do so with little-to-no effort.
- It is hard to imagine a scenario where an RPM-based installer could produce development, from-source, or from-PyPI Pulp installations.
It is important to recognize that native package managers offer significant benefits, and those benefits are orthogonal to the benefits provided by Ansible. In the future, we may decide to offer native packages in addition to or instead of an Ansible-based installer.
Workflow Example
How would such an installer work from an end user's perspective? Before going any further, let's recap some of the workflow-related problems being solved:
- As described in Encourage Community Involvement, users need to learn two different installation work-flows, where those two work-flows require entirely different sets of infrastructure, and where the two work-flows install different Pulp 3 components.
- As described in More Installation Strategies, we currently support from-source developer-oriented installs and from-PyPI user-oriented installs, but we don't offer from-source user-oriented installs.
From an end user's perspective, a work-flow like the following would appear to solve all of these complaints:
git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy dev-install.yml -i pulp-dev.example.com, ansible-galaxy src-install.yml -i pulp-src.example.com, ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,Sample Playbooks
While this looks great from an end user's perspective, it's unclear how this would work internally. Thus, the next question is "how would this work from a developer's perspective?"
To answer this question, let's take a top-down approach, starting with the playbooks. What would pypi-install.yml look like? It could look like this:
--- # Install Pulp 3 from PyPI packages. # # The resultant installation is designed to be useful for end users. It # doesn't do things like install code in editable mode, install extra # dependencies or install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!'And what would dev-install.yml look like? It could look like this:
--- # Install Pulp 3 from locally cloned repositories. # # The resultant installation is designed to be useful for developers. It # does things like install code in editable mode, install extra dependencies # and install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true vars: pulp_install_strategy: dev pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!'The only difference between the two scripts is the addition of one variable:
vars: pulp_install_strategy: devWith this in mind, you can guess what src-install.yml could look like:
--- # Install Pulp 3 from remote repositories. # # The resultant installation is designed to be useful for end users. It # doesn't do things like install code in editable mode, install extra # dependencies or install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true vars: pulp_install_strategy: source pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!'Note that SELinux is explicitly disabled in the playbooks, instead of within a role. This isn't necessary at a technical level, and in fact, doing so violates the DRY principle. It is done because disabling SELinux is a dangerous action that all users should be aware of, and placing this action in the playbooks makes it more visible.
Sample Role
Now that you know what the playbooks could look like, let's dive down another layer and look at roles. What effect does the pulp_install_strategy variable have on roles? And what does code using this variable look like?
The first question can be answered without looking at the documentation for a role. (It's good practice to place a README.md file at the top level of a role, and the variables accepted by a role may be documented there.) Here's the documentation that could exist for a hypothetical "pulpcore" role. (I'm not advocating for a role by this name to exist. This is for learning purposes.)
pulpcore ======== This role installs the pulpcore component of Pulp 3. It also creates and customizes `/etc/pulp/server.yaml`. NOTE: `SECRET_KEY` is updated even if a value is already present! Example Usage ------------- ```yaml - hosts: all roles: - dev ``` Variables --------- This role depends on the variables exported by the `pulp-user` role, and on the `pulp_install_strategy` variable. If `pulp_install_strategy` is `dev`, then debug mode is enabled in `/etc/pulp/server.yaml`. Otherwise, debug mode is disabled.What might the actual code using pulp_install_strategy look like? As an example, consider the scenario where the role discussed above wishes to enable or disable debug mode. Here's what the relevant code might look like:
- name: Configure DEBUG mode in server.yaml lineinfile: path: /etc/pulp/server.yaml regexp: '^DEBUG: ' line: 'DEBUG: True' state: '{{ "present" if pulp_install_strategy == "development" else "absent" }}'Or, imagine the scenario in which the role needs to find server.yaml in Pulp's source code, so that it can be copied to /etc/pulp/server.yaml. Depending on how Pulp was installed, the file may exist in one of several places. Here's what the relevant code might look like:
- name: Find server.yaml find: paths: '{{ pulp_devel_dir if pulp_install_strategy == "dev" else pulp_venv }}' recurse: true patterns: server.yaml register: result - name: Assert one server.yaml file has been found assert: that: 'result["files"] | length == 1'Or, imagine the scenario in which a large chunk of code should only be run for a single installation strategy. An elegant way to isolate it is:
- include: '{{ pulp_install_strategy }}.yml'Why One Variable?
Why should just one variable be used to support these installation strategies? Why not use several variables, where one controls whether ~/.bashrc is installed, another controls whether supplemental development tools are installed, and so on?
The reason to do this is to reduce combinatorial complexity. If just one argument is accepted by the installer, then the number of installation scenarios grows linearly with the number of valid values for that argument. That is, if there are three valid values for pulp_install_strategy, then there are three installation scenarios to support. This makes development and testing simpler, which should improve the quality of the installer. More options can be added if necessary, of course. But starting with a simple design seems like a good idea.
One might counter that this strategy greatly limits the options available to end users. There are many ways that a user might want to install Pulp, and by offering such a limiting installer, we're limiting how users can deploy Pulp.
This is sort of true. But the purpose of an installer isn't to provide an infinitely flexible platform for developing new and exciting things. The purpose of an installer is to produce a specific, known-working application installation. If a Pulp user wants the ultimate in flexibility, that's what the source code is for.
Implementation Strategy
Which set(s) of source code should see development work? What Ansible playbooks be produced?
What Code Sees Development Work?
How can we implement such an installer while minimizing the risk to developers, QE, and most importantly, our end users? I see at least three options:
- Evolve the end user installer.
- Implement a brand new installer.
- Evolve the developer installer.
Our users are our most important assets. Without them, Pulp has no rationale for existing. Consequently, we should take steps to protect them from any mistakes that we might make. If anyone is to suffer through implementation errors and development dead-ends, it should be the development team and QE, not our end users. Thus, I think that the option of evolving the end user installer should be ruled out.
Implementing an installer from scratch has one big benefit: it lets us perform development work without impacting any existing users at all. But there are big downsides to this approach as well. The biggest is that work on our existing installers will proceed while a new installer is developed. As a result, we will be stuck maintaining not two but three installers for some period of time. Many of the issues described in the Why? section will be made even worse. Furthermore, it's quite possible that work on a new installer could stretch on for much longer than expected, perhaps due to unexpected changes in priorities. In addition, this implementation strategy means that more targets need to be tracked during development. Whoever is working on a new installer from scratch needs to keep an eye on both of the other installers, to ensure that they aren't missing out on any important implementation details or features. This "tracking" effort is troublesome and should be minimized.
I think that evolving the developer installer is the best option, because it avoid both of the severe problems identified above. We don't run the risk of breaking our end users workflows (until work is done), and we don't run the risk of ending up with three installers. It is true that developers and QE may encounter breakage, but these are the audiences that are best prepared to deal with breakage.
What Roles Are Produced?
Assuming that the existing developer installer should be evolved, what exactly would that look like? Are there any especially interesting challenges that would need to be solved?
The most interesting challenge I foresee is the question of which roles should be created. Currently, the developer installer calls the following roles, in this order:
- pulp-user
- db
- broker
- lazy
- dev
- plugin
- systemd
- dev_tools
One option is to re-work the installer so that it calls the same roles as the end user installer. The end user installer calls the following roles, in this order:
- pulp3
- pulp3_db
- rabbitmq
- pulp3_systemd
I have two objections to this strategy.
First, I object to using the rabbitmq role, when we could use packages provided by the system package manager instead.
- Using an Ansible role means that we're unnecessarily pushing a particular RabbitMQ implementation and configuration on to our users, instead of respecting the differences between distributions. I imagine that the implementation of RabbitMQ differs between RHEL 7, Fedora 26 Workstation, Fedora 26 Server, Debian, Ubuntu, and so on, and the package maintainers for those distributions are in a better position to understand how RabbitMQ best fits into their particular ecosystem. This has already caused problems, in that the rabbitmq role configures RabbitMQ to listen on a non-standard port by default.
- We know that packages provided by the system package manager work well. The developer installer is proof of this: it uses packages from the system package manager.
- Packages provided by the package manager are in a position to receive more quality assurance than a third-party Ansible role. This is one of the value propositions of RHEL. We should take advantage of it.
- Dropping the rabbitmq role would let us also drop the config_encoder_filter role. At that point, we would have no third-party role dependencies.
My second objection to this strategy is more general: I think that we should start with a simple, monolithic installer before getting clever and trying to draw lines between components. Consider:
- Drawing lines between components makes development costlier. With the current end user installer, developers must submit pull requests against as many as three repositories in order to implement a functional change in the behaviour of the installer, and four repositories when adjusting documentation too. This is a drag. It would be better if these costs are taken on only when a known problem is being solved.
- It seems far smarter to draw lines between components based on real world experience than based on conjecture.
- The Pulp developers have not yet decided on how Pulp should be deployed. Think of questions like "which web servers are supported, and how should they communicate with the back-end Pulp services?" It's far easier to respond to these changes in Pulp's implementation with a monolithic architecture.
Instead, what I would recommend is that we merge the various roles in the developer installer, until we're left with just one highly polished role that can perform "dev," "src," and "pypi" installations. These three installation strategies would serve the needs of developers, QE and end users, respectively. We could then move this role into the existing pulp/ansible-pulp3 role, replacing the existing code in that repository.
What About Vagrant?
I see one other interesting challenge: the developer installer deploys code into a user account named "vagrant," and the end user installer deploys code into a user account named "pulp." This difference needs to be accounted for. We could solve this by parametrizing the installer, so that a configurable user may be used for deployments. However, this has two downsides:
- As described in Why One Variable?, this increases combinatorial complexity.
- This decresases dev/prod parity.
Instead, I would recommend always using a user named "pulp" for deployments, and adjusting the Vagrant file so that a user named "pulp" is used.
Summary
At a high level, this is the proposal:
Continue distributing Pulp 3 with Ansible.
Enable the following use case:
git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy dev-install.yml -i pulp-dev.example.com, ansible-galaxy src-install.yml -i pulp-src.example.com, ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,Get there by evolving the developer installer.
[1] I think this is how to use the developer installer, but I'm not sure. I've found that Vagrant frequently does not work with KVM as a back-end, and when it does work, it interferes with my already-working virtualization solution. As a result, I've never been able to create a working development environment. Proposal to Consolidate the Pulp 3 Installers ============================================= .. contents:: What? ----- Currently, one installer is used by the Pulp developers, and another installer is used by Pulp QE and end users. I propose that a single installer be used by all three audiences. This document explains why I think this change should be made, and it outlines a possible plan of action for making this change. Why? ---- Why should a single installer be used? The most succinct justification is that using a single installer improves `dev/prod parity`_, where dev/prod parity means "[keeping] development, staging, and production as similar as possible." In our case, QE serves as staging, and our users serve as production. More verbosely, a single installer should be used because doing so will likely: 1. Produce a better installer. 2. Reduce the workload on developers and quality engineers. 3. Improve Pulp's public image. 4. Encourage community involvement. 5. Allow for additional installation strategies. Better Installer, Less Work ~~~~~~~~~~~~~~~~~~~~~~~~~~~ To illustrate the first two points, consider the scenario where a quality engineer discovers an issue with the installer they're using. The engineer is obliged to check and see if the issue is also present in the second installer. If so, the engineer will either file one ticket that mentions both installers, or they will file two independent tickets, depending on their mindset. When a developer decides to work on the issues, they need to develop two similar fixes for two similar installers, they need to test both changes, and they need to file two pull requests. After the pull requests have been merged, the developer needs to either close out one or two tickets. This workflow is logically sound. So long as all parties are mindful of what they're doing, nothing will go wrong. But humans aren't like that! Consider the case of `Pulp #3031`_. That ticket describes how the end user installer inserts junk text into ``/etc/pulp/server.yaml``. I filed this ticket on September 25th, and I submitted a fix for the issue on the same day. Unfortunately, I forgot to file a second ticket noting that the exact same issue is also present in the developer installer. As of this writing on November 21st, the developer installer is affected by this issue. The developer installer still works due to the particular nature of the bug, but at any time, an unrelated change to the configuration file could break the development environment. Improve Pulp's Public Image ~~~~~~~~~~~~~~~~~~~~~~~~~~~ To illustrate the third point, consider what happened when the Pulp 3 plugin API alpha was released. The announcement was pushed back by a day or two so that QE could have a chance to play with the product before it went out the door. QE discovered several issues, one of them being the fact that the end user installer simply *did not work*. (The issue was fixed by `pulp/devel 396c93e775542484b585b8841bc9adc042b71731`_.) This gives the impression that the end user installer is a second class product. So long as we have two non-orthogonal installers, Pulp's public image is vulnerable to incidents like this. Encourage Community Involvement ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To illustrate the fourth point, consider some of the differences between the end user and developer installers. Using the end user installer requires the following: 1. Decide where you would like to install Pulp. You could install it on your local system, on a beaker system, on a libvirt + kvm VM, on a libvirt + virtualbox VM, on a Linode VM, on a DigitalOcean VM, or just about anywhere else. 2. Get shell access to the system and execute the following: .. code-block:: sh setenforce 0 systemctl stop firewalld # or open appropriate ports 3. From your local system, execute the following: .. code-block:: sh git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy install -r requirements.yml -p roles ansible-galaxy deploy-pulp3.yml -i pulp3.example.com, In contrast, using the developer installer requires the following: 1. Set up Vagrant. A development environment can't be deployed on any other type of host. You're SOL if you'd like to use some other type of back-end. 2. From your local system, execute [1]_ the following: .. code-block:: sh git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel vagrant up Those familiar with Pulp's installers might also notice that the end user installer doesn't install any plug-ins, whereas the developer installer installs the file plugin. Why should a prospective contributor need to learn two different installation work-flows, where those two work-flows require entirely different sets of infrastructure, and where the two work-flows install different Pulp 3 components? More Installation Strategies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The developer installer installs Pulp directly from source in editable mode, along with numerous extra tools, and it enables settings like debug mode. The end user installer installs Pulp from PyPI packages, without extra tools, and without enabling settings like debug mode. Unfortunately, these two installers don't cover all of our needs. PyPI packages are — quite reasonably — updated once a week. As Pulp 3 matures, these releases may become less frequent. As a result, our Jenkins test runs may fail to detect regressions for a up to a week, and tests for fixed-but-not-released features may produce incorrect results for a up to a week. One solution to this quandary is to create a third installer that installs Pulp directly from source, without extra tools, and without enabling settings like debug mode. As an obvious example, consider `Pulp #3107`_: Switch JWT auth token string from 'JWT' to 'Bearer' I had to explicitly request that new PyPI packages be uploaded after this change was made. In the future, requests like this will occur again. If a from-source installer is available, then such requests are obviated. How can we support this third installation use case with our current set-up? Should we create a third installer? That makes every other problem listed above even worse. It would be much better if we took an existing installer and added an option so that it could fulfill either use case. If we're going to do that, why not go even further and let an existing installer handle all three use cases? How? ---- In concrete terms, how could the installer described above be implemented? Technology ~~~~~~~~~~ Currently, both of our installers use Ansible. I suggest that we continue using Ansible to distribute Pulp. There are several key advantages to doing this: * We can re-use some of our existing Ansible assets when developing a converged installer. * Ansible is an official Red Hat product. Using Ansible may promote adoption of Pulp 3 within the business. * Ansible is platform-agnostic. If someone wants to install Pulp on an unsupported platform (such as CentOS, Debian, an old version of Fedora, etc.), they can do so with little-to-no effort. * It is hard to imagine a scenario where an RPM-based installer could produce development, from-source, or from-PyPI Pulp installations. It is important to recognize that native package managers offer significant benefits, and those benefits are orthogonal to the benefits provided by Ansible. In the future, we may decide to offer native packages in addition to or instead of an Ansible-based installer. Workflow Example ~~~~~~~~~~~~~~~~ How would such an installer work from an end user's perspective? Before going any further, let's recap some of the workflow-related problems being solved: * As described in `Encourage Community Involvement`_, users need to learn two different installation work-flows, where those two work-flows require entirely different sets of infrastructure, and where the two work-flows install different Pulp 3 components. * As described in `More Installation Strategies`_, we currently support from-source developer-oriented installs and from-PyPI user-oriented installs, but we don't offer from-source user-oriented installs. From an end user's perspective, a work-flow like the following would appear to solve all of these complaints: .. code-block:: sh git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy dev-install.yml -i pulp-dev.example.com, ansible-galaxy src-install.yml -i pulp-src.example.com, ansible-galaxy pypi-install.yml -i pulp-pypi.example.com, Sample Playbooks ~~~~~~~~~~~~~~~~ While this looks great from an end user's perspective, it's unclear how this would work internally. Thus, the next question is "how would this work from a developer's perspective?" To answer this question, let's take a top-down approach, starting with the playbooks. What would ``pypi-install.yml`` look like? It could look like this: .. code-block:: yaml --- # Install Pulp 3 from PyPI packages. # # The resultant installation is designed to be useful for end users. It # doesn't do things like install code in editable mode, install extra # dependencies or install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!' And what would ``dev-install.yml`` look like? It could look like this: .. code-block:: yaml --- # Install Pulp 3 from locally cloned repositories. # # The resultant installation is designed to be useful for developers. It # does things like install code in editable mode, install extra dependencies # and install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true vars: pulp_install_strategy: dev pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!' The only difference between the two scripts is the addition of one variable: .. code-block:: yaml vars: pulp_install_strategy: dev With this in mind, you can guess what ``src-install.yml`` could look like: .. code-block:: yaml --- # Install Pulp 3 from remote repositories. # # The resultant installation is designed to be useful for end users. It # doesn't do things like install code in editable mode, install extra # dependencies or install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true vars: pulp_install_strategy: source pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!' Note that SELinux is explicitly disabled in the playbooks, instead of within a role. This isn't necessary at a technical level, and in fact, doing so violates the DRY principle. It is done because disabling SELinux is a dangerous action that all users should be aware of, and placing this action in the playbooks makes it more visible. Sample Role ~~~~~~~~~~~ Now that you know what the playbooks could look like, let's dive down another layer and look at roles. What effect does the ``pulp_install_strategy`` variable have on roles? And what does code using this variable look like? The first question can be answered without looking at the documentation for a role. (It's good practice to place a ``README.md`` file at the top level of a role, and the variables accepted by a role may be documented there.) Here's the documentation that could exist for a hypothetical "pulpcore" role. (I'm not advocating for a role by this name to exist. This is for learning purposes.) .. code-block:: md pulpcore ======== This role installs the pulpcore component of Pulp 3. It also creates and customizes `/etc/pulp/server.yaml`. NOTE: `SECRET_KEY` is updated even if a value is already present! Example Usage ------------- ```yaml - hosts: all roles: - dev ``` Variables --------- This role depends on the variables exported by the `pulp-user` role, and on the `pulp_install_strategy` variable. If `pulp_install_strategy` is `dev`, then debug mode is enabled in `/etc/pulp/server.yaml`. Otherwise, debug mode is disabled. What might the actual code using ``pulp_install_strategy`` look like? As an example, consider the scenario where the role discussed above wishes to enable or disable debug mode. Here's what the relevant code might look like: .. code-block:: yaml - name: Configure DEBUG mode in server.yaml lineinfile: path: /etc/pulp/server.yaml regexp: '^DEBUG: ' line: 'DEBUG: True' state: '{{ "present" if pulp_install_strategy == "development" else "absent" }}' Or, imagine the scenario in which the role needs to find ``server.yaml`` in Pulp's source code, so that it can be copied to ``/etc/pulp/server.yaml``. Depending on how Pulp was installed, the file may exist in one of several places. Here's what the relevant code might look like: .. code-block:: yaml - name: Find server.yaml find: paths: '{{ pulp_devel_dir if pulp_install_strategy == "dev" else pulp_venv }}' recurse: true patterns: server.yaml register: result - name: Assert one server.yaml file has been found assert: that: 'result["files"] | length == 1' Or, imagine the scenario in which a large chunk of code should only be run for a single installation strategy. An elegant way to isolate it is: .. code-block:: yaml - include: '{{ pulp_install_strategy }}.yml' Why One Variable? ~~~~~~~~~~~~~~~~~ Why should just one variable be used to support these installation strategies? Why not use several variables, where one controls whether ``~/.bashrc`` is installed, another controls whether supplemental development tools are installed, and so on? The reason to do this is to reduce combinatorial complexity. If just one argument is accepted by the installer, then the number of installation scenarios grows linearly with the number of valid values for that argument. That is, if there are three valid values for ``pulp_install_strategy``, then there are three installation scenarios to support. This makes development and testing simpler, which should improve the quality of the installer. More options can be added if necessary, of course. But starting with a simple design seems like a good idea. One might counter that this strategy greatly limits the options available to end users. There are many ways that a user might want to install Pulp, and by offering such a limiting installer, we're limiting how users can deploy Pulp. This is sort of true. But the purpose of an installer isn't to provide an infinitely flexible platform for developing new and exciting things. The purpose of an installer is to produce a specific, known-working application installation. If a Pulp user wants the ultimate in flexibility, that's what the source code is for. Implementation Strategy ~~~~~~~~~~~~~~~~~~~~~~~ Which set(s) of source code should see development work? What Ansible playbooks be produced? What Code Sees Development Work? ```````````````````````````````` How can we implement such an installer while minimizing the risk to developers, QE, and most importantly, our end users? I see at least three options: * Evolve the end user installer. * Implement a brand new installer. * Evolve the developer installer. Our users are our most important assets. Without them, Pulp has no rationale for existing. Consequently, we should take steps to protect them from any mistakes that we might make. If anyone is to suffer through implementation errors and development dead-ends, it should be the development team and QE, not our end users. Thus, I think that the option of evolving the end user installer should be ruled out. Implementing an installer from scratch has one big benefit: it lets us perform development work without impacting any existing users at all. But there are big downsides to this approach as well. The biggest is that work on our existing installers will proceed while a new installer is developed. As a result, we will be stuck maintaining not two but *three* installers for some period of time. Many of the issues described in the `Why?`_ section will be made even worse. Furthermore, it's quite possible that work on a new installer could stretch on for much longer than expected, perhaps due to unexpected changes in priorities. In addition, this implementation strategy means that more targets need to be tracked during development. Whoever is working on a new installer from scratch needs to keep an eye on both of the other installers, to ensure that they aren't missing out on any important implementation details or features. This "tracking" effort is troublesome and should be minimized. I think that evolving the developer installer is the best option, because it avoid both of the severe problems identified above. We don't run the risk of breaking our end users workflows (until work is done), and we don't run the risk of ending up with three installers. It is true that developers and QE may encounter breakage, but these are the audiences that are best prepared to deal with breakage. What Roles Are Produced? ```````````````````````` Assuming that the existing developer installer should be evolved, what exactly would that look like? Are there any especially interesting challenges that would need to be solved? The most interesting challenge I foresee is the question of which roles should be created. Currently, the developer installer calls the following roles, in this order: * ``pulp-user`` * ``db`` * ``broker`` * ``lazy`` * ``dev`` * ``plugin`` * ``systemd`` * ``dev_tools`` One option is to re-work the installer so that it calls the same roles as the end user installer. The end user installer calls the following roles, in this order: * ``pulp3`` * ``pulp3_db`` * ``rabbitmq`` * ``pulp3_systemd`` I have two objections to this strategy. First, I object to using the ``rabbitmq`` role, when we could use packages provided by the system package manager instead. * Using an Ansible role means that we're unnecessarily pushing a particular RabbitMQ implementation and configuration on to our users, instead of respecting the differences between distributions. I imagine that the implementation of RabbitMQ differs between RHEL 7, Fedora 26 Workstation, Fedora 26 Server, Debian, Ubuntu, and so on, and the package maintainers for those distributions are in a better position to understand how RabbitMQ best fits into their particular ecosystem. This has already caused problems, in that the ``rabbitmq`` role configures RabbitMQ to listen on a non-standard port by default. * We know that packages provided by the system package manager work well. The developer installer is proof of this: it uses packages from the system package manager. * Packages provided by the package manager are in a position to receive more quality assurance than a third-party Ansible role. This is one of the value propositions of RHEL. We should take advantage of it. * Dropping the ``rabbitmq`` role would let us also drop the ``config_encoder_filter`` role. At that point, we would have no third-party role dependencies. My second objection to this strategy is more general: I think that we should start with a simple, monolithic installer before getting clever and trying to draw lines between components. Consider: * Drawing lines between components makes development costlier. With the current end user installer, developers must submit pull requests against as many as three repositories in order to implement a functional change in the behaviour of the installer, and *four* repositories when adjusting documentation too. This is a drag. It would be better if these costs are taken on only when a known problem is being solved. * It seems far smarter to draw lines between components based on real world experience than based on conjecture. * The Pulp developers have not yet decided on how Pulp should be deployed. Think of questions like "which web servers are supported, and how should they communicate with the back-end Pulp services?" It's far easier to respond to these changes in Pulp's implementation with a monolithic architecture. Instead, what I would recommend is that we merge the various roles in the developer installer, until we're left with just one highly polished role that can perform "dev," "src," and "pypi" installations. These three installation strategies would serve the needs of developers, QE and end users, respectively. We could then move this role into the existing `pulp/ansible-pulp3`_ role, replacing the existing code in that repository. What About Vagrant? ``````````````````` I see one other interesting challenge: the developer installer deploys code into a user account named "vagrant," and the end user installer deploys code into a user account named "pulp." This difference needs to be accounted for. We could solve this by parametrizing the installer, so that a configurable user may be used for deployments. However, this has two downsides: * As described in `Why One Variable?`_, this increases combinatorial complexity. * This decresases `dev/prod parity`_. Instead, I would recommend always using a user named "pulp" for deployments, and adjusting the Vagrant file so that a user named "pulp" is used. Summary ------- At a high level, this is the proposal: 1. Continue distributing Pulp 3 with Ansible. 2. Enable the following use case: .. code-block:: sh git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy dev-install.yml -i pulp-dev.example.com, ansible-galaxy src-install.yml -i pulp-src.example.com, ansible-galaxy pypi-install.yml -i pulp-pypi.example.com, 3. Get there by evolving the developer installer. .. [1] I think this is how to use the developer installer, but I'm not sure. I've found that Vagrant frequently does not work with KVM as a back-end, and when it does work, it interferes with my already-working virtualization solution. As a result, I've never been able to create a working development environment. .. _Pulp #3031: https://pulp.plan.io/issues/3031 .. _Pulp #3107: https://pulp.plan.io/issues/3107 .. _dev/prod parity: https://12factor.net/dev-prod-parity .. _pulp/ansible-pulp3: https://github.com/pulp/ansible-pulp3 .. _pulp/devel 396c93e775542484b585b8841bc9adc042b71731: https://github.com/pulp/devel/commit/396c93e775542484b585b8841bc9adc042b71731_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev