Colin Watson has proposed merging ~cjwatson/lpbuildbot-worker:ops-charm into lpbuildbot-worker:main.
Commit message: Add a charm to set up a buildbot worker node Requested reviews: Launchpad code reviewers (launchpad-reviewers) For more details, see: https://code.launchpad.net/~cjwatson/lpbuildbot-worker/+git/lpbuildbot-worker/+merge/416408 This essentially just automates the existing steps that were used to set up Launchpad's current worker nodes. The tests are a bit on the basic side, but they more or less work, and they were an opportunity to experiment with testing operator charms using `pytest`. The `juju deploy` command in the README won't work until the charm has been built by a Launchpad charm recipe and pushed to Charmhub, but it gives an idea of the intended workflow. -- Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/lpbuildbot-worker:ops-charm into lpbuildbot-worker:main.
diff --git a/charm/.gitignore b/charm/.gitignore new file mode 100644 index 0000000..7a9b421 --- /dev/null +++ b/charm/.gitignore @@ -0,0 +1,7 @@ +venv/ +build/ +*.charm + +__pycache__/ +*.py[cod] +.tox diff --git a/charm/.jujuignore b/charm/.jujuignore new file mode 100644 index 0000000..a130358 --- /dev/null +++ b/charm/.jujuignore @@ -0,0 +1,4 @@ +/venv +*.py[cod] +*.charm +.tox diff --git a/charm/LICENSE b/charm/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/charm/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/charm/README.md b/charm/README.md new file mode 100644 index 0000000..fd49106 --- /dev/null +++ b/charm/README.md @@ -0,0 +1,29 @@ +# charm + +## Description + +Launchpad runs its continuous integration using +[Buildbot](https://buildbot.net/). This charm provides a suitable worker +node. + +This charm is highly specialized for use by Launchpad's CI system, and will +not work for other purposes. + +## Usage + + juju deploy ch:lpbuildbot-worker + +Until such time as a manager relation is added, you'll need to set +`manager-host` to the host name of the buildbot manager and +`buildbot-password` to the password used by the manager to contact this +worker. + +## Relations + +None at present (though a relation with a buildbot manager may be added in +future). + +## Contributing + +Please see the [Juju SDK docs](https://juju.is/docs/sdk) for guidelines +on enhancements to this charm following best practice guidelines. diff --git a/charm/charmcraft.yaml b/charm/charmcraft.yaml new file mode 100644 index 0000000..f128aad --- /dev/null +++ b/charm/charmcraft.yaml @@ -0,0 +1,7 @@ +type: "charm" +parts: + charm: + charm-python-packages: [setuptools] +bases: + - name: "ubuntu" + channel: "18.04" diff --git a/charm/config.yaml b/charm/config.yaml new file mode 100644 index 0000000..abf0721 --- /dev/null +++ b/charm/config.yaml @@ -0,0 +1,25 @@ +# Copyright 2021 Canonical Ltd. +# See LICENSE file for licensing details. + +options: + ubuntu-series: + type: string + default: xenial + description: > + Space-separated list of Ubuntu series for which to maintain workers. + manager-host: + type: string + default: + description: Buildbot manager host name. + manager-port: + type: int + default: 9989 + description: Buildbot manager port. + buildbot-password: + type: string + default: + description: Password used to contact the Buildbot manager. + active: + type: boolean + default: true + description: If true, start Buildbot workers. diff --git a/charm/metadata.yaml b/charm/metadata.yaml new file mode 100644 index 0000000..9021930 --- /dev/null +++ b/charm/metadata.yaml @@ -0,0 +1,11 @@ +# Copyright 2021 Canonical Ltd. +# See LICENSE file for licensing details. + +name: lpbuildbot-worker +display-name: Launchpad Buildbot worker node +summary: A preconfigured Launchpad Buildbot worker node. +description: > + Launchpad runs its continuous integration using Buildbot. This charm + provides a suitable worker node. +series: + - bionic diff --git a/charm/requirements-dev.txt b/charm/requirements-dev.txt new file mode 100644 index 0000000..8fef638 --- /dev/null +++ b/charm/requirements-dev.txt @@ -0,0 +1,4 @@ +-r requirements.txt +pyfakefs +pytest +pytest-subprocess diff --git a/charm/requirements.txt b/charm/requirements.txt new file mode 100644 index 0000000..e352ba9 --- /dev/null +++ b/charm/requirements.txt @@ -0,0 +1,2 @@ +jinja2 +ops >= 1.2.0 diff --git a/charm/src/charm.py b/charm/src/charm.py new file mode 100755 index 0000000..c8418c9 --- /dev/null +++ b/charm/src/charm.py @@ -0,0 +1,294 @@ +#!/usr/bin/env python3 +# Copyright 2021-2022 Canonical Ltd. +# See LICENSE file for licensing details. + +"""Install and configure a Launchpad buildbot worker.""" + +import grp +import json +import logging +import os +import pwd +import shutil +import subprocess +from pathlib import Path + +from jinja2 import Environment, FileSystemLoader +from ops.charm import CharmBase +from ops.framework import StoredState +from ops.main import main +from ops.model import ActiveStatus, BlockedStatus, MaintenanceStatus + +logger = logging.getLogger(__name__) + + +class BlockedOnConfig(Exception): + def __init__(self, name): + super().__init__("Waiting for {} to be set".format(name)) + + +class LPBuildbotWorkerCharm(CharmBase): + + _stored = StoredState() + + def __init__(self, *args): + super().__init__(*args) + self.framework.observe(self.on.install, self._on_install) + self.framework.observe(self.on.upgrade_charm, self._on_install) + self.framework.observe(self.on.config_changed, self._on_config_changed) + self._stored.set_default( + ubuntu_series=set(), + manager_host=None, + manager_port=None, + buildbot_password=None, + ) + + def _set_maintenance_step(self, description): + self.unit.status = MaintenanceStatus(description) + logger.info(description) + + def _run_as_buildbot(self, args, **kwargs): + return subprocess.run(["sudo", "-Hu", "buildbot"] + args, **kwargs) + + def _require_config(self, name): + value = self.config.get(name) + if not value: + raise BlockedOnConfig(name) + return value + + def _list_lxd_images(self): + images = json.loads( + self._run_as_buildbot( + ["lxc", "image", "list", "-f", "json"], + stdout=subprocess.PIPE, + check=True, + universal_newlines=True, + ).stdout + ) + names = [] + for image in images: + for alias in image["aliases"]: + names.append(alias["name"]) + return names + + def _chown(self, path, user, group): + uid = pwd.getpwnam(user).pw_uid + gid = grp.getgrnam(group).gr_gid + os.chown(str(path), uid, gid) + + def _render_template( + self, source, target, context, user="root", group="root", mode=0o644 + ): + template_env = Environment( + loader=FileSystemLoader(str(self.charm_dir / "templates")) + ) + template = template_env.get_template(source) + content = template.render(context) + target = Path(target) + if not target.parent.exists(): + target.parent.mkdir(mode=0o755, parents=True) + self._chown(target.parent, user, group) + target.write_text(content) + self._chown(target, user, group) + os.chmod(str(target), mode) + + def _install_lpbuildbot_worker(self): + self._set_maintenance_step("Installing lpbuildbot-worker") + subprocess.run( + ["add-apt-repository", "-y", "ppa:launchpad/ubuntu/ppa"], + check=True, + ) + subprocess.run( + ["apt-get", "-y", "install", "lpbuildbot-worker"], check=True + ) + + def _install_lxd(self): + if Path("/usr/bin/lxc").exists(): + self._set_maintenance_step("Removing lxd .debs") + subprocess.run( + [ + "apt-get", + "-y", + "purge", + "lxc-common", + "lxcfs", + "lxd-client", + ], + check=True, + ) + # We may need to delete this leftover interface in order for + # "lxd init --auto" to succeed after installing the snap. It's + # fine if it doesn't exist. + if ( + subprocess.run( + ["ip", "link", "show", "dev", "lxdbr0"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ).returncode + == 0 + ): + subprocess.run( + ["ip", "link", "delete", "dev", "lxdbr0"], check=True + ) + + if not Path("/snap/bin/lxc").exists(): + self._set_maintenance_step("Installing lxd snap") + subprocess.run(["snap", "install", "lxd"], check=True) + + if not Path("/var/snap/lxd/common/lxd/server.key").exists(): + self._set_maintenance_step("Initializing lxd") + subprocess.run( + ["lxd", "init", "--auto", "--storage-backend=zfs"], check=True + ) + + def _configure_buildbot_user(self): + self._set_maintenance_step("Configuring buildbot user") + subprocess.run(["adduser", "buildbot", "lxd"], check=True) + # Snaps don't currently work well with users whose home directories + # aren't under /home. + old_home = Path("/var/lib/buildbot") + new_home = Path("/home/buildbot") + if old_home.exists() and not old_home.is_symlink(): + shutil.move(str(old_home), str(new_home)) + old_home.symlink_to(new_home) + subprocess.run( + ["usermod", "-d", str(new_home), "buildbot"], check=True + ) + ssh_key_path = new_home / ".ssh" / "launchpad_lxd_id_rsa" + if not ssh_key_path.exists(): + self._run_as_buildbot( + ["mkdir", "-m700", "-p", str(ssh_key_path.parent)], check=True + ) + self._run_as_buildbot( + [ + "ssh-keygen", + "-t", + "rsa", + "-b", + "2048", + "-N", + "", + "-f", + str(ssh_key_path), + ], + check=True, + ) + + def _make_workers(self): + ubuntu_series = set(self._require_config("ubuntu-series").split()) + manager_host = self._require_config("manager-host") + manager_port = self._require_config("manager-port") + buildbot_password = self._require_config("buildbot-password") + + workers_path = Path("/home/buildbot/workers") + if not workers_path.exists(): + self._run_as_buildbot( + ["mkdir", "-m755", "-p", str(workers_path)], check=True + ) + + current_images = self._list_lxd_images() + for series in ubuntu_series: + self._set_maintenance_step("Making worker for {}".format(series)) + base_path = workers_path / "{}-lxd-worker".format(series) + if not base_path.exists(): + self._run_as_buildbot( + ["mkdir", "-m755", "-p", str(base_path)], check=True + ) + lp_path = base_path / "devel" + dependencies_path = base_path / "dependencies" + sourcecode_path = dependencies_path / "sourcecode" + download_cache_path = dependencies_path / "download-cache" + if not lp_path.exists(): + self._run_as_buildbot( + [ + "git", + "clone", + "https://git.launchpad.net/launchpad";, + str(lp_path), + ], + check=True, + ) + if not sourcecode_path.exists(): + self._run_as_buildbot( + ["mkdir", "-m755", "-p", str(sourcecode_path)], check=True + ) + if not download_cache_path.exists(): + self._run_as_buildbot( + [ + "git", + "clone", + "https://git.launchpad.net/lp-source-dependencies";, + str(download_cache_path), + ], + check=True, + ) + if "lptests-{}".format(series) not in current_images: + self._run_as_buildbot( + ["create-lp-tests-lxd", series, str(base_path)], check=True + ) + + self._render_template( + "buildbot.tac.j2", + str(base_path / "buildbot.tac"), + { + "buildbot_password": buildbot_password, + "manager_host": manager_host, + "manager_port": manager_port, + "name": self.unit.name.replace("/", "-"), + "series": series, + }, + user="buildbot", + group="buildbot", + mode=0o640, + ) + service_name = "buildbot-worker@{}-lxd-worker.service".format( + series + ) + if self.config.get("active", True): + subprocess.run( + ["systemctl", "enable", service_name], check=True + ) + subprocess.run( + ["systemctl", "restart", service_name], check=True + ) + else: + subprocess.run(["systemctl", "disable", service_name]) + subprocess.run(["systemctl", "stop", service_name]) + + for image in current_images: + if not image.startswith("lptests-"): + continue + series = image[len("lptests-") :] + if series not in ubuntu_series: + self._set_maintenance_step( + "Deleting obsolete worker for {}".format(series) + ) + self._run_as_buildbot( + ["lxc", "image", "delete", image], check=True + ) + service_name = "buildbot-worker@{}-lxd-worker.service".format( + series + ) + subprocess.run(["systemctl", "disable", service_name]) + subprocess.run(["systemctl", "stop", service_name]) + + self._stored.ubuntu_series = ubuntu_series + + def _on_install(self, event): + self._install_lpbuildbot_worker() + self._install_lxd() + self._configure_buildbot_user() + self._on_config_changed(event) + + def _on_config_changed(self, _): + try: + self._make_workers() + except BlockedOnConfig as e: + self.unit.status = BlockedStatus(str(e)) + else: + self.unit.status = ActiveStatus() + logger.info("Ready") + + +if __name__ == "__main__": + main(LPBuildbotWorkerCharm) diff --git a/charm/templates/buildbot.tac.j2 b/charm/templates/buildbot.tac.j2 new file mode 100644 index 0000000..eaa0a61 --- /dev/null +++ b/charm/templates/buildbot.tac.j2 @@ -0,0 +1,33 @@ +import os + +from buildbot_worker.bot import Worker +from twisted.application import service +from twisted.python.log import ( + FileLogObserver, + ILogObserver, + ) +from twisted.python.logfile import LogFile + + +basedir = '/home/buildbot/workers/{{ series }}-lxd-worker' +manager_host = '{{ manager_host }}' +manager_port = {{ manager_port }} +name = '{{ name }}' +password = '{{ buildbot_password }}' +keepalive = 600 +umask = 0o22 +rotateLength = 1000000 +maxRotatedFiles = None + +# note: this line is matched against to check that this is a worker +# directory; do not edit it. +application = service.Application('buildbot-worker') + +logfile = LogFile.fromFullPath( + os.path.join(basedir, 'twistd.log'), rotateLength=rotateLength, + maxRotatedFiles=maxRotatedFiles) +application.setComponent(ILogObserver, FileLogObserver(logfile).emit) +w = Worker( + manager_host, manager_port, name, password, basedir, keepalive, + umask=umask) +w.setServiceParent(application) diff --git a/charm/tests/__init__.py b/charm/tests/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/charm/tests/__init__.py diff --git a/charm/tests/test_charm.py b/charm/tests/test_charm.py new file mode 100644 index 0000000..314e7e0 --- /dev/null +++ b/charm/tests/test_charm.py @@ -0,0 +1,392 @@ +# Copyright 2021-2022 Canonical Ltd. +# See LICENSE file for licensing details. +# +# Learn more about testing at: https://juju.is/docs/sdk/testing + +import json +import os +from pathlib import Path + +import pytest +from ops.model import ActiveStatus, MaintenanceStatus +from ops.testing import Harness + +from charm import BlockedOnConfig, LPBuildbotWorkerCharm + +# The "fs" fixture is a fake filesystem from pyfakefs; the "fp" fixture is +# from pytest-subprocess. + + +class FakePasswd: + def __init__(self, pw_uid): + self.pw_uid = pw_uid + + +class FakeGroup: + def __init__(self, gr_gid): + self.gr_gid = gr_gid + + +@pytest.fixture +def fake_user(monkeypatch): + monkeypatch.setattr("pwd.getpwnam", lambda user: FakePasswd(1000)) + monkeypatch.setattr("grp.getgrnam", lambda user: FakeGroup(1000)) + + +@pytest.fixture +def harness(): + harness = Harness(LPBuildbotWorkerCharm) + harness.begin() + yield harness + harness.cleanup() + + +def test_install_lpbuildbot_worker(harness, fp): + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._install_lpbuildbot_worker() + + assert harness.model.unit.status == MaintenanceStatus( + "Installing lpbuildbot-worker" + ) + assert list(fp.calls) == [ + ["add-apt-repository", "-y", "ppa:launchpad/ubuntu/ppa"], + ["apt-get", "-y", "install", "lpbuildbot-worker"], + ] + + +def test_install_lxd_removes_debs_and_installs_snap(harness, fs, fp): + Path("/usr/bin").mkdir(parents=True) + Path("/usr/bin/lxc").touch() + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._install_lxd() + + assert harness.model.unit.status == MaintenanceStatus("Initializing lxd") + assert list(fp.calls) == [ + ["apt-get", "-y", "purge", "lxc-common", "lxcfs", "lxd-client"], + ["ip", "link", "show", "dev", "lxdbr0"], + ["ip", "link", "delete", "dev", "lxdbr0"], + ["snap", "install", "lxd"], + ["lxd", "init", "--auto", "--storage-backend=zfs"], + ] + + +def test_install_lxd_snap_already_installed(harness, fs, fp): + Path("/snap/bin").mkdir(parents=True) + Path("/snap/bin/lxc").touch() + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._install_lxd() + + assert harness.model.unit.status == MaintenanceStatus("Initializing lxd") + assert list(fp.calls) == [ + ["lxd", "init", "--auto", "--storage-backend=zfs"], + ] + + +def test_install_lxd_already_configured(harness, fs, fp): + Path("/snap/bin").mkdir(parents=True) + Path("/snap/bin/lxc").touch() + Path("/var/snap/lxd/common/lxd").mkdir(parents=True) + Path("/var/snap/lxd/common/lxd/server.key").touch() + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._install_lxd() + + assert list(fp.calls) == [] + + +def test_configure_buildbot_user_moves_home_directory(harness, fs, fp): + Path("/var/lib/buildbot").mkdir(parents=True) + Path("/home").mkdir() + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._configure_buildbot_user() + + assert list(fp.calls) == [ + ["adduser", "buildbot", "lxd"], + ["usermod", "-d", "/home/buildbot", "buildbot"], + [ + "sudo", + "-Hu", + "buildbot", + "mkdir", + "-m700", + "-p", + "/home/buildbot/.ssh", + ], + [ + "sudo", + "-Hu", + "buildbot", + "ssh-keygen", + "-t", + "rsa", + "-b", + "2048", + "-N", + "", + "-f", + "/home/buildbot/.ssh/launchpad_lxd_id_rsa", + ], + ] + assert Path("/home/buildbot").is_dir() + assert not Path("/home/buildbot").is_symlink() + assert Path("/var/lib/buildbot").is_symlink() + assert os.readlink("/var/lib/buildbot") == "/home/buildbot" + + +def test_configure_buildbot_user_keeps_moved_home_directory(harness, fs, fp): + Path("/home/buildbot").mkdir(parents=True) + Path("/var/lib").mkdir(parents=True) + Path("/var/lib/buildbot").symlink_to("/home/buildbot") + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._configure_buildbot_user() + + assert list(fp.calls) == [ + ["adduser", "buildbot", "lxd"], + [ + "sudo", + "-Hu", + "buildbot", + "mkdir", + "-m700", + "-p", + "/home/buildbot/.ssh", + ], + [ + "sudo", + "-Hu", + "buildbot", + "ssh-keygen", + "-t", + "rsa", + "-b", + "2048", + "-N", + "", + "-f", + "/home/buildbot/.ssh/launchpad_lxd_id_rsa", + ], + ] + assert Path("/home/buildbot").is_dir() + assert not Path("/home/buildbot").is_symlink() + assert Path("/var/lib/buildbot").is_symlink() + assert os.readlink("/var/lib/buildbot") == "/home/buildbot" + + +def test_configure_buildbot_user_keeps_existing_ssh_key(harness, fs, fp): + Path("/home/buildbot/.ssh").mkdir(parents=True) + Path("/home/buildbot/.ssh/launchpad_lxd_id_rsa").write_text("key") + fp.keep_last_process(True) + fp.register([fp.any()]) + + harness.charm._configure_buildbot_user() + + assert list(fp.calls) == [["adduser", "buildbot", "lxd"]] + assert ( + Path("/home/buildbot/.ssh/launchpad_lxd_id_rsa").read_text() == "key" + ) + + +@pytest.mark.parametrize( + "empty_key", + ["ubuntu-series", "manager-host", "manager-port", "buildbot-password"], +) +def test_make_workers_requires_config(harness, fs, fp, empty_key): + # ubuntu-series and manager-port have defaults in config.yaml. + config = { + "manager-host": "manager.example.com", + "buildbot-password": "secret", + } + config[empty_key] = "" + harness._update_config(config) + + pytest.raises(BlockedOnConfig, harness.charm._make_workers) + + +def test_make_workers(harness, fs, fp, fake_user): + fs.add_real_directory(harness.charm.charm_dir / "templates") + fp.keep_last_process(True) + fp.register( + ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"], + stdout=json.dumps({}), + ) + fp.register([fp.any()]) + harness._update_config( + {"manager-host": "manager.example.com", "buildbot-password": "secret"} + ) + + harness.charm._make_workers() + + assert harness.model.unit.status == MaintenanceStatus( + "Making worker for xenial" + ) + assert list(fp.calls) == [ + [ + "sudo", + "-Hu", + "buildbot", + "mkdir", + "-m755", + "-p", + "/home/buildbot/workers", + ], + ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"], + [ + "sudo", + "-Hu", + "buildbot", + "mkdir", + "-m755", + "-p", + "/home/buildbot/workers/xenial-lxd-worker", + ], + [ + "sudo", + "-Hu", + "buildbot", + "git", + "clone", + "https://git.launchpad.net/launchpad";, + "/home/buildbot/workers/xenial-lxd-worker/devel", + ], + [ + "sudo", + "-Hu", + "buildbot", + "mkdir", + "-m755", + "-p", + "/home/buildbot/workers/xenial-lxd-worker/dependencies/sourcecode", + ], + [ + "sudo", + "-Hu", + "buildbot", + "git", + "clone", + "https://git.launchpad.net/lp-source-dependencies";, + "/home/buildbot/workers/xenial-lxd-worker/dependencies/" + "download-cache", + ], + [ + "sudo", + "-Hu", + "buildbot", + "create-lp-tests-lxd", + "xenial", + "/home/buildbot/workers/xenial-lxd-worker", + ], + ["systemctl", "enable", "buildbot-worker@xenial-lxd-worker.service"], + ["systemctl", "restart", "buildbot-worker@xenial-lxd-worker.service"], + ] + buildbot_tac = ( + Path("/home/buildbot/workers/xenial-lxd-worker/buildbot.tac") + .read_text() + .splitlines() + ) + assert ( + "basedir = '/home/buildbot/workers/xenial-lxd-worker'" in buildbot_tac + ) + assert "manager_host = 'manager.example.com'" in buildbot_tac + assert "manager_port = 9989" in buildbot_tac + assert "password = 'secret'" in buildbot_tac + assert harness.charm._stored.ubuntu_series == {"xenial"} + + +def test_make_workers_deletes_obsolete_workers(harness, fs, fp, fake_user): + fs.add_real_directory(harness.charm.charm_dir / "templates") + fp.keep_last_process(True) + fp.register( + ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"], + stdout=json.dumps([{"aliases": [{"name": "lptests-precise"}]}]), + ) + fp.register([fp.any()]) + harness._update_config( + {"manager-host": "manager.example.com", "buildbot-password": "secret"} + ) + + harness.charm._make_workers() + + assert harness.model.unit.status == MaintenanceStatus( + "Deleting obsolete worker for precise" + ) + assert [ + "sudo", + "-Hu", + "buildbot", + "lxc", + "image", + "delete", + "lptests-precise", + ] in fp.calls + assert [ + "systemctl", + "disable", + "buildbot-worker@precise-lxd-worker.service", + ] in fp.calls + assert [ + "systemctl", + "stop", + "buildbot-worker@precise-lxd-worker.service", + ] in fp.calls + + +def test_install(harness, fs, fp, fake_user): + fs.add_real_directory(harness.charm.charm_dir / "templates") + fp.keep_last_process(True) + fp.register( + ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"], + stdout=json.dumps({}), + ) + fp.register([fp.any()]) + harness._update_config( + {"manager-host": "manager.example.com", "buildbot-password": "secret"} + ) + + harness.charm.on.install.emit() + + # Details are tested elsewhere; here we just ensure that install runs + # all the major steps. + assert ["apt-get", "-y", "install", "lpbuildbot-worker"] in fp.calls + assert ["lxd", "init", "--auto", "--storage-backend=zfs"] in fp.calls + assert ["adduser", "buildbot", "lxd"] in fp.calls + assert [ + "systemctl", + "restart", + "buildbot-worker@xenial-lxd-worker.service", + ] in fp.calls + assert harness.model.unit.status == ActiveStatus() + + +def test_config_changed(harness, fs, fp, fake_user): + fs.add_real_directory(harness.charm.charm_dir / "templates") + fp.keep_last_process(True) + fp.register( + ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"], + stdout=json.dumps({}), + ) + fp.register([fp.any()]) + + harness.update_config( + { + "manager-host": "manager.example.com", + "buildbot-password": "another-secret", + } + ) + + buildbot_tac = ( + Path("/home/buildbot/workers/xenial-lxd-worker/buildbot.tac") + .read_text() + .splitlines() + ) + assert "password = 'another-secret'" in buildbot_tac diff --git a/charm/tox.ini b/charm/tox.ini new file mode 100644 index 0000000..53ba640 --- /dev/null +++ b/charm/tox.ini @@ -0,0 +1,15 @@ +[tox] +envlist = + py36 + py37 + py38 + py39 + py310 +# Charms aren't real Python packages, so we need a bit of hacking to make +# tox work. +skipsdist = True + +[testenv] +deps = -r requirements-dev.txt +setenv = PYTHONPATH={toxinidir}/src +commands = pytest {posargs}
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : launchpad-reviewers@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
