Re: [Pulp-dev] Pulp 3 installer re-work proposal

Mon, 11 Dec 2017 10:23:50 -0800 

A revised proposal taking Robin's comments into account is attached.
Title: Proposal to Consolidate the Pulp 3 Installers







Proposal to Consolidate the Pulp 3 Installers






What?


Currently, one installer is used by the Pulp developers, and another installer
is used by Pulp QE and end users. I propose that a single installer be used by
all three audiences. This document explains why I think this change should be
made, and it outlines a possible plan of action for making this change.






Why?


Why should a single installer be used? The most succinct justification is that
using a single installer improves dev/prod parity, where dev/prod parity
means "[keeping] development, staging, and production as similar as possible."
In our case, QE serves as staging, and our users serve as production.


More verbosely, a single installer should be used because doing so will likely:


    

  1. Produce a better installer.
    2. 

  2. Reduce the workload on developers and quality engineers.
    3. 

  3. Improve Pulp's public image.
    4. 

  4. Encourage community involvement.
    5. 

  5. Allow for additional installation strategies.
    6. 





Better Installer, Less Work


To illustrate the first two points, consider the scenario where a quality
engineer discovers an issue with the installer they're using.


The engineer is obliged to check and see if the issue is also present in the
second installer. If so, the engineer will either file one ticket that mentions
both installers, or they will file two independent tickets, depending on their
mindset. When a developer decides to work on the issues, they need to develop
two similar fixes for two similar installers, they need to test both changes,
and they need to file two pull requests. After the pull requests have been
merged, the developer needs to either close out one or two tickets.


This workflow is logically sound. So long as all parties are mindful of what
they're doing, nothing will go wrong. But humans aren't like that! Consider the
case of Pulp #3031. That ticket describes how the end user installer inserts
junk text into /etc/pulp/server.yaml. I filed this ticket on September 25th,
and I submitted a fix for the issue on the same day. Unfortunately, I forgot to
file a second ticket noting that the exact same issue is also present in the
developer installer. As of this writing on November 21st, the developer
installer is affected by this issue. The developer installer still works due to
the particular nature of the bug, but at any time, an unrelated change to the
configuration file could break the development environment.






Improve Pulp's Public Image


To illustrate the third point, consider what happened when the Pulp 3 plugin API
alpha was released. The announcement was pushed back by a day or two so that QE
could have a chance to play with the product before it went out the door. QE
discovered several issues, one of them being the fact that the end user
installer simply did not work. (The issue was fixed by pulp/devel
396c93e775542484b585b8841bc9adc042b71731.) This gives the impression that the
end user installer is a second class product. So long as we have two
non-orthogonal installers, Pulp's public image is vulnerable to incidents like
this.






Encourage Community Involvement


To illustrate the fourth point, consider some of the differences between the end
user and developer installers. Using the end user installer requires the
following:


    

  1. Decide where you would like to install Pulp. You could install it on your
local system, on a beaker system, on a libvirt + kvm VM, on a libvirt +
virtualbox VM, on a Linode VM, on a DigitalOcean VM, or just about anywhere
else.
    

    2. 

  2. Get shell access to the system and execute the following:
    

    setenforce 0
systemctl stop firewalld  # or open appropriate ports

    

    3. 

  3. From your local system, execute the following:
    

    git clone https://github.com/pulp/devel.git --branch 3.0-dev
cd devel/ansible
ansible-galaxy install -r requirements.yml -p roles
ansible-galaxy deploy-pulp3.yml -i pulp3.example.com,

    

    4. 



In contrast, using the developer installer requires the following:


    

  1. Set up Vagrant. A development environment can't be deployed on any other type
of host. You're SOL if you'd like to use some other type of back-end.
    

    2. 

  2. From your local system, execute [1] the following:
    

    git clone https://github.com/pulp/devel.git --branch 3.0-dev
cd devel
vagrant up

    

    3. 



Those familiar with Pulp's installers might also notice that the end user
installer doesn't install any plug-ins, whereas the developer installer installs
the file plugin.


Why should a prospective contributor need to learn two different installation
work-flows, where those two work-flows require entirely different sets of
infrastructure, and where the two work-flows install different Pulp 3
components?






More Installation Strategies


The developer installer installs Pulp directly from source in editable mode,
along with numerous extra tools, and it enables settings like debug mode. The
end user installer installs Pulp from PyPI packages, without extra tools, and
without enabling settings like debug mode.


Unfortunately, these two installers don't cover all of our needs. PyPI packages
are — quite reasonably — updated once a week. As Pulp 3 matures, these releases
may become less frequent. As a result, our Jenkins test runs may fail to detect
regressions for a up to a week, and tests for fixed-but-not-released features
may produce incorrect results for a up to a week. One solution to this quandary
is to create a third installer that installs Pulp directly from source, without
extra tools, and without enabling settings like debug mode. As an obvious
example, consider Pulp #3107:



Switch JWT auth token string from 'JWT' to 'Bearer'


I had to explicitly request that new PyPI packages be uploaded after this change
was made. In the future, requests like this will occur again. If a from-source
installer is available, then such requests are obviated.


How can we support this third installation use case with our current set-up?
Should we create a third installer? That makes every other problem listed above
even worse. It would be much better if we took an existing installer and added
an option so that it could fulfill either use case. If we're going to do that,
why not go even further and let an existing installer handle all three use
cases?








How?


In concrete terms, how could the installer described above be implemented?




Technology


Currently, both of our installers use Ansible. I suggest that we continue using
Ansible to distribute Pulp. There are several key advantages to doing this:


    

  • We can re-use some of our existing Ansible assets when developing a converged
installer.
    • 

  • Ansible is an official Red Hat product. Using Ansible may promote adoption of
Pulp 3 within the business.
    • 

  • Ansible is platform-agnostic. If someone wants to install Pulp on an
unsupported platform (such as CentOS, Debian, an old version of Fedora, etc.),
they can do so with little-to-no effort.
    • 

  • It is hard to imagine a scenario where an RPM-based installer could produce
development, from-source, or from-PyPI Pulp installations.
    • 



It is important to recognize that native package managers offer significant
benefits, and those benefits are orthogonal to the benefits provided by Ansible.
In the future, we may decide to offer native packages in addition to or instead
of an Ansible-based installer.






Workflow Example


How would such an installer work from an end user's perspective? Before going
any further, let's recap some of the workflow-related problems being solved:


    

  • As described in Encourage Community Involvement, users need to learn two
different installation work-flows, where those two work-flows require entirely
different sets of infrastructure, and where the two work-flows install
different Pulp 3 components.
    • 

  • As described in More Installation Strategies, we currently support
from-source developer-oriented installs and from-PyPI user-oriented installs,
but we don't offer from-source user-oriented installs.
    • 



From an end user's perspective, a work-flow like the following would appear to
solve all of these complaints:


git clone https://github.com/pulp/devel.git --branch 3.0-dev
cd devel/ansible
ansible-galaxy dev-install.yml -i pulp-dev.example.com,
ansible-galaxy src-install.yml -i pulp-src.example.com,
ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,







Sample Playbooks


While this looks great from an end user's perspective, it's unclear how this
would work internally. Thus, the next question is "how would this work from a
developer's perspective?"


To answer this question, let's take a top-down approach, starting with the
playbooks. What would pypi-install.yml look like? It could look like this:


---
# Install Pulp 3 from PyPI packages.
#
# The resultant installation is designed to be useful for end users. It
# doesn't do things like install code in editable mode, install extra
# dependencies or install Bash aliases.
#
# SELinux is disabled because Pulp 3 is currently incompatible with SELinux.
- hosts: all
  become: true
  pre_tasks:
    - name: Disable selinux
      selinux:
        state: disabled
  roles:
    - 'snip!'



And what would dev-install.yml look like? It could look like this:


---
# Install Pulp 3 from locally cloned repositories.
#
# The resultant installation is designed to be useful for developers. It
# does things like install code in editable mode, install extra dependencies
# and install Bash aliases.
#
# SELinux is disabled because Pulp 3 is currently incompatible with SELinux.
- hosts: all
  become: true
  vars:
    pulp_install_strategy: dev
  pre_tasks:
    - name: Disable selinux
      selinux:
        state: disabled
  roles:
    - 'snip!'



The only difference between the two scripts is the addition of one variable:


vars:
  pulp_install_strategy: dev



With this in mind, you can guess what src-install.yml could look like:


---
# Install Pulp 3 from remote repositories.
#
# The resultant installation is designed to be useful for end users. It
# doesn't do things like install code in editable mode, install extra
# dependencies or install Bash aliases.
#
# SELinux is disabled because Pulp 3 is currently incompatible with SELinux.
- hosts: all
  become: true
  vars:
    pulp_install_strategy: source
  pre_tasks:
    - name: Disable selinux
      selinux:
        state: disabled
  roles:
    - 'snip!'



Note that SELinux is explicitly disabled in the playbooks, instead of within a
role. This isn't necessary at a technical level, and in fact, doing so violates
the DRY principle. It is done because disabling SELinux is a dangerous action
that all users should be aware of, and placing this action in the playbooks
makes it more visible.






Sample Role


Now that you know what the playbooks could look like, let's dive down another
layer and look at roles. What effect does the pulp_install_strategy variable
have on roles? And what does code using this variable look like?


The first question can be answered without looking at the documentation for a
role. (It's good practice to place a README.md file at the top level of a
role, and the variables accepted by a role may be documented there.) Here's the
documentation that could exist for a hypothetical "pulpcore" role. (I'm not
advocating for a role by this name to exist. This is for learning purposes.)


pulpcore
========

This role installs the pulpcore component of Pulp 3. It also creates and
customizes `/etc/pulp/server.yaml`.

NOTE: `SECRET_KEY` is updated even if a value is already present!

Example Usage
-------------

```yaml
- hosts: all
  roles:
    - dev
```

Variables
---------

This role depends on the variables exported by the `pulp-user` role, and on
the `pulp_install_strategy` variable. If `pulp_install_strategy` is `dev`,
then debug mode is enabled in `/etc/pulp/server.yaml`. Otherwise, debug mode
is disabled.



What might the actual code using pulp_install_strategy look like? As an
example, consider the scenario where the role discussed above wishes to enable
or disable debug mode. Here's what the relevant code might look like:


- name: Configure DEBUG mode in server.yaml
  lineinfile:
    path: /etc/pulp/server.yaml
    regexp: '^DEBUG: '
    line: 'DEBUG: True'
    state: '{{ "present" if pulp_install_strategy == "development" else "absent" }}'



Or, imagine the scenario in which the role needs to find server.yaml in
Pulp's source code, so that it can be copied to /etc/pulp/server.yaml.
Depending on how Pulp was installed, the file may exist in one of several
places. Here's what the relevant code might look like:


- name: Find server.yaml
  find:
    paths: '{{ pulp_devel_dir if pulp_install_strategy == "dev" else pulp_venv }}'
    recurse: true
    patterns: server.yaml
  register: result

- name: Assert one server.yaml file has been found
  assert:
    that: 'result["files"] | length == 1'



Or, imagine the scenario in which a large chunk of code should only be run for a
single installation strategy. An elegant way to isolate it is:


- include: '{{ pulp_install_strategy }}.yml'







Why One Variable?


Why should just one variable be used to support these installation strategies?
Why not use several variables, where one controls whether ~/.bashrc is
installed, another controls whether supplemental development tools are
installed, and so on?


The reason to do this is to reduce combinatorial complexity. If just one
argument is accepted by the installer, then the number of installation scenarios
grows linearly with the number of valid values for that argument. That is, if
there are three valid values for pulp_install_strategy, then there are three
installation scenarios to support. This makes development and testing simpler,
which should improve the quality of the installer. More options can be added if
necessary, of course. But starting with a simple design seems like a good idea.


One might counter that this strategy greatly limits the options available to end
users. There are many ways that a user might want to install Pulp, and by
offering such a limiting installer, we're limiting how users can deploy Pulp.


This is sort of true. But the purpose of an installer isn't to provide an
infinitely flexible platform for developing new and exciting things. The purpose
of an installer is to produce a specific, known-working application
installation. If a Pulp user wants the ultimate in flexibility, that's what the
source code is for.






Implementation Strategy


Which set(s) of source code should see development work? What Ansible playbooks
be produced?




What Code Sees Development Work?


How can we implement such an installer while minimizing the risk to developers,
QE, and most importantly, our end users? I see at least three options:


    

  • Evolve the end user installer.
    • 

  • Implement a brand new installer.
    • 

  • Evolve the developer installer.
    • 



Our users are our most important assets. Without them, Pulp has no rationale for
existing. Consequently, we should take steps to protect them from any mistakes
that we might make. If anyone is to suffer through implementation errors and
development dead-ends, it should be the development team and QE, not our end
users. Thus, I think that the option of evolving the end user installer should
be ruled out.


Implementing an installer from scratch has one big benefit: it lets us perform
development work without impacting any existing users at all. But there are big
downsides to this approach as well. The biggest is that work on our existing
installers will proceed while a new installer is developed. As a result, we will
be stuck  maintaining not two but three installers for some period of time.
Many of the issues described in the Why? section will be made even worse.
Furthermore, it's quite possible that work on a new installer could stretch on
for much longer than expected, perhaps due to unexpected changes in priorities.
In addition, this implementation strategy means that more targets need to be
tracked during development. Whoever is working on a new installer from scratch
needs to keep an eye on both of the other installers, to ensure that they aren't
missing out on any important implementation details or features. This "tracking"
effort is troublesome and should be minimized.


I think that evolving the developer installer is the best option, because it
avoid both of the severe problems identified above. We don't run the risk of
breaking our end users workflows (until work is done), and we don't run the risk
of ending up with three installers. It is true that developers and QE may
encounter breakage, but these are the audiences that are best prepared to deal
with breakage.






What Roles Are Produced?


Assuming that the existing developer installer should be evolved, what exactly
would that look like? Are there any especially interesting challenges that would
need to be solved?


The most interesting challenge I foresee is the question of which roles should
be created. Currently, the developer installer calls the following roles, in
this order:


    

  • pulp-user
    • 

  • db
    • 

  • broker
    • 

  • lazy
    • 

  • dev
    • 

  • plugin
    • 

  • systemd
    • 

  • dev_tools
    • 



One option is to re-work the installer so that it calls the same roles as the
end user installer. The end user installer calls the following roles, in this
order:


    

  • pulp3
    • 

  • pulp3_db
    • 

  • rabbitmq
    • 

  • pulp3_systemd
    • 



I have two objections to this strategy.


First, I object to using the rabbitmq role, when we could use packages
provided by the system package manager instead.


    

  • Using an Ansible role means that we're unnecessarily pushing a particular
RabbitMQ implementation and configuration on to our users, instead of
respecting the differences between distributions. I imagine that the
implementation of RabbitMQ differs between RHEL 7, Fedora 26 Workstation,
Fedora 26 Server, Debian, Ubuntu, and so on, and the package maintainers for
those distributions are in a better position to understand how RabbitMQ best
fits into their particular ecosystem. This has already caused problems, in
that the rabbitmq role configures RabbitMQ to listen on a non-standard
port by default.
    • 

  • We know that packages provided by the system package manager work well. The
developer installer is proof of this: it uses packages from the system package
manager.
    • 

  • Packages provided by the package manager are in a position to receive more
quality assurance than a third-party Ansible role. This is one of the value
propositions of RHEL. We should take advantage of it.
    • 

  • Dropping the rabbitmq role would let us also drop the
config_encoder_filter role. At that point, we would have no third-party
role dependencies.
    • 



My second objection to this strategy is more general: I think that we should
start with a simple, monolithic installer before getting clever and trying to
draw lines between components. Consider:


    

  • Drawing lines between components makes development costlier. With the current
end user installer, developers must submit pull requests against as many as
three repositories in order to implement a functional change in the behaviour
of the installer, and four repositories when adjusting documentation too.
This is a drag. It would be better if these costs are taken on only when a
known problem is being solved.
    • 

  • It seems far smarter to draw lines between components based on real world
experience than based on conjecture.
    • 

  • The Pulp developers have not yet decided on how Pulp should be deployed. Think
of questions like "which web servers are supported, and how should they
communicate with the back-end Pulp services?" It's far easier to respond to
these changes in Pulp's implementation with a monolithic architecture.
    • 



Instead, what I would recommend is that we merge the various roles in the
developer installer, until we're left with just one highly polished role that
can perform "dev," "src," and "pypi" installations. These three installation
strategies would serve the needs of developers, QE and end users, respectively.
We could then move this role into the existing pulp/ansible-pulp3 role,
replacing the existing code in that repository.






What About Vagrant?


I see one other interesting challenge: the developer installer deploys code into
a user account named "vagrant," and the end user installer deploys code into a
user account named "pulp." This difference needs to be accounted for. We could
solve this by parametrizing the installer, so that a configurable user may be
used for deployments. However, this has two downsides:



Instead, I would recommend always using a user named "pulp" for deployments, and
adjusting the Vagrant file so that a user named "pulp" is used.










Summary


At a high level, this is the proposal:


    

  1. Continue distributing Pulp 3 with Ansible.
    

    2. 

  2. Enable the following use case:
    

    git clone https://github.com/pulp/devel.git --branch 3.0-dev
cd devel/ansible
ansible-galaxy dev-install.yml -i pulp-dev.example.com,
ansible-galaxy src-install.yml -i pulp-src.example.com,
ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,

    

    3. 

  3. Get there by evolving the developer installer.
    

    4. 







[1]I think this is how to use the developer installer, but I'm not sure.
I've found that Vagrant frequently does not work with KVM as a back-end, and
when it does work, it interferes with my already-working virtualization
solution. As a result, I've never been able to create a working development
environment.











Proposal to Consolidate the Pulp 3 Installers
=============================================

.. contents::

What?
-----

Currently, one installer is used by the Pulp developers, and another installer
is used by Pulp QE and end users. I propose that a single installer be used by
all three audiences. This document explains why I think this change should be
made, and it outlines a possible plan of action for making this change.

Why?
----

Why should a single installer be used? The most succinct justification is that
using a single installer improves `dev/prod parity`_, where dev/prod parity
means "[keeping] development, staging, and production as similar as possible."
In our case, QE serves as staging, and our users serve as production.

More verbosely, a single installer should be used because doing so will likely:

1. Produce a better installer.
2. Reduce the workload on developers and quality engineers.
3. Improve Pulp's public image.
4. Encourage community involvement.
5. Allow for additional installation strategies.

Better Installer, Less Work
~~~~~~~~~~~~~~~~~~~~~~~~~~~

To illustrate the first two points, consider the scenario where a quality
engineer discovers an issue with the installer they're using.

The engineer is obliged to check and see if the issue is also present in the
second installer. If so, the engineer will either file one ticket that mentions
both installers, or they will file two independent tickets, depending on their
mindset. When a developer decides to work on the issues, they need to develop
two similar fixes for two similar installers, they need to test both changes,
and they need to file two pull requests. After the pull requests have been
merged, the developer needs to either close out one or two tickets.

This workflow is logically sound. So long as all parties are mindful of what
they're doing, nothing will go wrong. But humans aren't like that! Consider the
case of `Pulp #3031`_. That ticket describes how the end user installer inserts
junk text into ``/etc/pulp/server.yaml``. I filed this ticket on September 25th,
and I submitted a fix for the issue on the same day. Unfortunately, I forgot to
file a second ticket noting that the exact same issue is also present in the
developer installer. As of this writing on November 21st, the developer
installer is affected by this issue. The developer installer still works due to
the particular nature of the bug, but at any time, an unrelated change to the
configuration file could break the development environment.

Improve Pulp's Public Image
~~~~~~~~~~~~~~~~~~~~~~~~~~~

To illustrate the third point, consider what happened when the Pulp 3 plugin API
alpha was released. The announcement was pushed back by a day or two so that QE
could have a chance to play with the product before it went out the door. QE
discovered several issues, one of them being the fact that the end user
installer simply *did not work*. (The issue was fixed by `pulp/devel
396c93e775542484b585b8841bc9adc042b71731`_.) This gives the impression that the
end user installer is a second class product. So long as we have two
non-orthogonal installers, Pulp's public image is vulnerable to incidents like
this.

Encourage Community Involvement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To illustrate the fourth point, consider some of the differences between the end
user and developer installers. Using the end user installer requires the
following:

1. Decide where you would like to install Pulp. You could install it on your
   local system, on a beaker system, on a libvirt + kvm VM, on a libvirt +
   virtualbox VM, on a Linode VM, on a DigitalOcean VM, or just about anywhere
   else.
2. Get shell access to the system and execute the following:

   .. code-block:: sh

       setenforce 0
       systemctl stop firewalld  # or open appropriate ports
3. From your local system, execute the following:

   .. code-block:: sh

       git clone https://github.com/pulp/devel.git --branch 3.0-dev
       cd devel/ansible
       ansible-galaxy install -r requirements.yml -p roles
       ansible-galaxy deploy-pulp3.yml -i pulp3.example.com,

In contrast, using the developer installer requires the following:

1. Set up Vagrant. A development environment can't be deployed on any other type
   of host. You're SOL if you'd like to use some other type of back-end.
2. From your local system, execute [1]_ the following:

   .. code-block:: sh

       git clone https://github.com/pulp/devel.git --branch 3.0-dev
       cd devel
       vagrant up

Those familiar with Pulp's installers might also notice that the end user
installer doesn't install any plug-ins, whereas the developer installer installs
the file plugin.

Why should a prospective contributor need to learn two different installation
work-flows, where those two work-flows require entirely different sets of
infrastructure, and where the two work-flows install different Pulp 3
components?

More Installation Strategies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The developer installer installs Pulp directly from source in editable mode,
along with numerous extra tools, and it enables settings like debug mode. The
end user installer installs Pulp from PyPI packages, without extra tools, and
without enabling settings like debug mode.

Unfortunately, these two installers don't cover all of our needs. PyPI packages
are — quite reasonably — updated once a week. As Pulp 3 matures, these releases
may become less frequent. As a result, our Jenkins test runs may fail to detect
regressions for a up to a week, and tests for fixed-but-not-released features
may produce incorrect results for a up to a week. One solution to this quandary
is to create a third installer that installs Pulp directly from source, without
extra tools, and without enabling settings like debug mode. As an obvious
example, consider `Pulp #3107`_:

    Switch JWT auth token string from 'JWT' to 'Bearer'

I had to explicitly request that new PyPI packages be uploaded after this change
was made. In the future, requests like this will occur again. If a from-source
installer is available, then such requests are obviated.

How can we support this third installation use case with our current set-up?
Should we create a third installer? That makes every other problem listed above
even worse. It would be much better if we took an existing installer and added
an option so that it could fulfill either use case. If we're going to do that,
why not go even further and let an existing installer handle all three use
cases?

How?
----

In concrete terms, how could the installer described above be implemented?

Technology
~~~~~~~~~~

Currently, both of our installers use Ansible. I suggest that we continue using
Ansible to distribute Pulp. There are several key advantages to doing this:

* We can re-use some of our existing Ansible assets when developing a converged
  installer.
* Ansible is an official Red Hat product. Using Ansible may promote adoption of
  Pulp 3 within the business.
* Ansible is platform-agnostic. If someone wants to install Pulp on an
  unsupported platform (such as CentOS, Debian, an old version of Fedora, etc.),
  they can do so with little-to-no effort.
* It is hard to imagine a scenario where an RPM-based installer could produce
  development, from-source, or from-PyPI Pulp installations.

It is important to recognize that native package managers offer significant
benefits, and those benefits are orthogonal to the benefits provided by Ansible.
In the future, we may decide to offer native packages in addition to or instead
of an Ansible-based installer.

Workflow Example
~~~~~~~~~~~~~~~~

How would such an installer work from an end user's perspective? Before going
any further, let's recap some of the workflow-related problems being solved:

* As described in `Encourage Community Involvement`_, users need to learn two
  different installation work-flows, where those two work-flows require entirely
  different sets of infrastructure, and where the two work-flows install
  different Pulp 3 components.
* As described in `More Installation Strategies`_, we currently support
  from-source developer-oriented installs and from-PyPI user-oriented installs,
  but we don't offer from-source user-oriented installs.

From an end user's perspective, a work-flow like the following would appear to
solve all of these complaints:

.. code-block:: sh

    git clone https://github.com/pulp/devel.git --branch 3.0-dev
    cd devel/ansible
    ansible-galaxy dev-install.yml -i pulp-dev.example.com,
    ansible-galaxy src-install.yml -i pulp-src.example.com,
    ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,

Sample Playbooks
~~~~~~~~~~~~~~~~

While this looks great from an end user's perspective, it's unclear how this
would work internally. Thus, the next question is "how would this work from a
developer's perspective?"

To answer this question, let's take a top-down approach, starting with the
playbooks. What would ``pypi-install.yml`` look like? It could look like this:

.. code-block:: yaml

    ---
    # Install Pulp 3 from PyPI packages.
    #
    # The resultant installation is designed to be useful for end users. It
    # doesn't do things like install code in editable mode, install extra
    # dependencies or install Bash aliases.
    #
    # SELinux is disabled because Pulp 3 is currently incompatible with SELinux.
    - hosts: all
      become: true
      pre_tasks:
        - name: Disable selinux
          selinux:
            state: disabled
      roles:
        - 'snip!'

And what would ``dev-install.yml`` look like? It could look like this:

.. code-block:: yaml

    ---
    # Install Pulp 3 from locally cloned repositories.
    #
    # The resultant installation is designed to be useful for developers. It
    # does things like install code in editable mode, install extra dependencies
    # and install Bash aliases.
    #
    # SELinux is disabled because Pulp 3 is currently incompatible with SELinux.
    - hosts: all
      become: true
      vars:
        pulp_install_strategy: dev
      pre_tasks:
        - name: Disable selinux
          selinux:
            state: disabled
      roles:
        - 'snip!'

The only difference between the two scripts is the addition of one variable:

.. code-block:: yaml

    vars:
      pulp_install_strategy: dev

With this in mind, you can guess what ``src-install.yml`` could look like:

.. code-block:: yaml

    ---
    # Install Pulp 3 from remote repositories.
    #
    # The resultant installation is designed to be useful for end users. It
    # doesn't do things like install code in editable mode, install extra
    # dependencies or install Bash aliases.
    #
    # SELinux is disabled because Pulp 3 is currently incompatible with SELinux.
    - hosts: all
      become: true
      vars:
        pulp_install_strategy: source
      pre_tasks:
        - name: Disable selinux
          selinux:
            state: disabled
      roles:
        - 'snip!'

Note that SELinux is explicitly disabled in the playbooks, instead of within a
role. This isn't necessary at a technical level, and in fact, doing so violates
the DRY principle. It is done because disabling SELinux is a dangerous action
that all users should be aware of, and placing this action in the playbooks
makes it more visible.

Sample Role
~~~~~~~~~~~

Now that you know what the playbooks could look like, let's dive down another
layer and look at roles. What effect does the ``pulp_install_strategy`` variable
have on roles? And what does code using this variable look like?

The first question can be answered without looking at the documentation for a
role. (It's good practice to place a ``README.md`` file at the top level of a
role, and the variables accepted by a role may be documented there.) Here's the
documentation that could exist for a hypothetical "pulpcore" role. (I'm not
advocating for a role by this name to exist. This is for learning purposes.)

.. code-block:: md

    pulpcore
    ========

    This role installs the pulpcore component of Pulp 3. It also creates and
    customizes `/etc/pulp/server.yaml`.

    NOTE: `SECRET_KEY` is updated even if a value is already present!

    Example Usage
    -------------

    ```yaml
    - hosts: all
      roles:
        - dev
    ```

    Variables
    ---------

    This role depends on the variables exported by the `pulp-user` role, and on
    the `pulp_install_strategy` variable. If `pulp_install_strategy` is `dev`,
    then debug mode is enabled in `/etc/pulp/server.yaml`. Otherwise, debug mode
    is disabled.

What might the actual code using ``pulp_install_strategy`` look like? As an
example, consider the scenario where the role discussed above wishes to enable
or disable debug mode. Here's what the relevant code might look like:

.. code-block:: yaml

    - name: Configure DEBUG mode in server.yaml
      lineinfile:
        path: /etc/pulp/server.yaml
        regexp: '^DEBUG: '
        line: 'DEBUG: True'
        state: '{{ "present" if pulp_install_strategy == "development" else "absent" }}'

Or, imagine the scenario in which the role needs to find ``server.yaml`` in
Pulp's source code, so that it can be copied to ``/etc/pulp/server.yaml``.
Depending on how Pulp was installed, the file may exist in one of several
places. Here's what the relevant code might look like:

.. code-block:: yaml

    - name: Find server.yaml
      find:
        paths: '{{ pulp_devel_dir if pulp_install_strategy == "dev" else pulp_venv }}'
        recurse: true
        patterns: server.yaml
      register: result

    - name: Assert one server.yaml file has been found
      assert:
        that: 'result["files"] | length == 1'

Or, imagine the scenario in which a large chunk of code should only be run for a
single installation strategy. An elegant way to isolate it is:

.. code-block:: yaml

    - include: '{{ pulp_install_strategy }}.yml'

Why One Variable?
~~~~~~~~~~~~~~~~~

Why should just one variable be used to support these installation strategies?
Why not use several variables, where one controls whether ``~/.bashrc`` is
installed, another controls whether supplemental development tools are
installed, and so on?

The reason to do this is to reduce combinatorial complexity. If just one
argument is accepted by the installer, then the number of installation scenarios
grows linearly with the number of valid values for that argument. That is, if
there are three valid values for ``pulp_install_strategy``, then there are three
installation scenarios to support. This makes development and testing simpler,
which should improve the quality of the installer. More options can be added if
necessary, of course. But starting with a simple design seems like a good idea.

One might counter that this strategy greatly limits the options available to end
users. There are many ways that a user might want to install Pulp, and by
offering such a limiting installer, we're limiting how users can deploy Pulp.

This is sort of true. But the purpose of an installer isn't to provide an
infinitely flexible platform for developing new and exciting things. The purpose
of an installer is to produce a specific, known-working application
installation. If a Pulp user wants the ultimate in flexibility, that's what the
source code is for.

Implementation Strategy
~~~~~~~~~~~~~~~~~~~~~~~

Which set(s) of source code should see development work? What Ansible playbooks
be produced?

What Code Sees Development Work?
````````````````````````````````

How can we implement such an installer while minimizing the risk to developers,
QE, and most importantly, our end users? I see at least three options:

* Evolve the end user installer.
* Implement a brand new installer.
* Evolve the developer installer.

Our users are our most important assets. Without them, Pulp has no rationale for
existing. Consequently, we should take steps to protect them from any mistakes
that we might make. If anyone is to suffer through implementation errors and
development dead-ends, it should be the development team and QE, not our end
users. Thus, I think that the option of evolving the end user installer should
be ruled out.

Implementing an installer from scratch has one big benefit: it lets us perform
development work without impacting any existing users at all. But there are big
downsides to this approach as well. The biggest is that work on our existing
installers will proceed while a new installer is developed. As a result, we will
be stuck  maintaining not two but *three* installers for some period of time.
Many of the issues described in the `Why?`_ section will be made even worse.
Furthermore, it's quite possible that work on a new installer could stretch on
for much longer than expected, perhaps due to unexpected changes in priorities.
In addition, this implementation strategy means that more targets need to be
tracked during development. Whoever is working on a new installer from scratch
needs to keep an eye on both of the other installers, to ensure that they aren't
missing out on any important implementation details or features. This "tracking"
effort is troublesome and should be minimized.

I think that evolving the developer installer is the best option, because it
avoid both of the severe problems identified above. We don't run the risk of
breaking our end users workflows (until work is done), and we don't run the risk
of ending up with three installers. It is true that developers and QE may
encounter breakage, but these are the audiences that are best prepared to deal
with breakage.

What Roles Are Produced?
````````````````````````

Assuming that the existing developer installer should be evolved, what exactly
would that look like? Are there any especially interesting challenges that would
need to be solved?

The most interesting challenge I foresee is the question of which roles should
be created. Currently, the developer installer calls the following roles, in
this order:

* ``pulp-user``
* ``db``
* ``broker``
* ``lazy``
* ``dev``
* ``plugin``
* ``systemd``
* ``dev_tools``

One option is to re-work the installer so that it calls the same roles as the
end user installer. The end user installer calls the following roles, in this
order:

* ``pulp3``
* ``pulp3_db``
* ``rabbitmq``
* ``pulp3_systemd``

I have two objections to this strategy.

First, I object to using the ``rabbitmq`` role, when we could use packages
provided by the system package manager instead.

* Using an Ansible role means that we're unnecessarily pushing a particular
  RabbitMQ implementation and configuration on to our users, instead of
  respecting the differences between distributions. I imagine that the
  implementation of RabbitMQ differs between RHEL 7, Fedora 26 Workstation,
  Fedora 26 Server, Debian, Ubuntu, and so on, and the package maintainers for
  those distributions are in a better position to understand how RabbitMQ best
  fits into their particular ecosystem. This has already caused problems, in
  that the ``rabbitmq`` role configures RabbitMQ to listen on a non-standard
  port by default.
* We know that packages provided by the system package manager work well. The
  developer installer is proof of this: it uses packages from the system package
  manager.
* Packages provided by the package manager are in a position to receive more
  quality assurance than a third-party Ansible role. This is one of the value
  propositions of RHEL. We should take advantage of it.
* Dropping the ``rabbitmq`` role would let us also drop the
  ``config_encoder_filter`` role. At that point, we would have no third-party
  role dependencies.

My second objection to this strategy is more general: I think that we should
start with a simple, monolithic installer before getting clever and trying to
draw lines between components. Consider:

* Drawing lines between components makes development costlier. With the current
  end user installer, developers must submit pull requests against as many as
  three repositories in order to implement a functional change in the behaviour
  of the installer, and *four* repositories when adjusting documentation too.
  This is a drag. It would be better if these costs are taken on only when a
  known problem is being solved.
* It seems far smarter to draw lines between components based on real world
  experience than based on conjecture.
* The Pulp developers have not yet decided on how Pulp should be deployed. Think
  of questions like "which web servers are supported, and how should they
  communicate with the back-end Pulp services?" It's far easier to respond to
  these changes in Pulp's implementation with a monolithic architecture.

Instead, what I would recommend is that we merge the various roles in the
developer installer, until we're left with just one highly polished role that
can perform "dev," "src," and "pypi" installations. These three installation
strategies would serve the needs of developers, QE and end users, respectively.
We could then move this role into the existing `pulp/ansible-pulp3`_ role,
replacing the existing code in that repository.

What About Vagrant?
```````````````````

I see one other interesting challenge: the developer installer deploys code into
a user account named "vagrant," and the end user installer deploys code into a
user account named "pulp." This difference needs to be accounted for. We could
solve this by parametrizing the installer, so that a configurable user may be
used for deployments. However, this has two downsides:

* As described in `Why One Variable?`_, this increases combinatorial complexity.
* This decresases `dev/prod parity`_.

Instead, I would recommend always using a user named "pulp" for deployments, and
adjusting the Vagrant file so that a user named "pulp" is used.

Summary
-------

At a high level, this is the proposal:

1. Continue distributing Pulp 3 with Ansible.
2. Enable the following use case:

   .. code-block:: sh

     git clone https://github.com/pulp/devel.git --branch 3.0-dev
     cd devel/ansible
     ansible-galaxy dev-install.yml -i pulp-dev.example.com,
     ansible-galaxy src-install.yml -i pulp-src.example.com,
     ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,
3. Get there by evolving the developer installer.

.. [1] I think this is how to use the developer installer, but I'm not sure.
    I've found that Vagrant frequently does not work with KVM as a back-end, and
    when it does work, it interferes with my already-working virtualization
    solution. As a result, I've never been able to create a working development
    environment.

.. _Pulp #3031: https://pulp.plan.io/issues/3031
.. _Pulp #3107: https://pulp.plan.io/issues/3107
.. _dev/prod parity: https://12factor.net/dev-prod-parity
.. _pulp/ansible-pulp3: https://github.com/pulp/ansible-pulp3
.. _pulp/devel 396c93e775542484b585b8841bc9adc042b71731: https://github.com/pulp/devel/commit/396c93e775542484b585b8841bc9adc042b71731

_______________________________________________
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Reply via email to