[Launchpad-reviewers] [Merge] ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master
Simone Pelosi has proposed merging
~pelpsi/launchpad:request-token-and-session-from-fetch-service into
launchpad:master with ~ines-almeida/launchpad:fetch-service-option-model-update
as a prerequisite.
Commit message:
Request fetch service session
NOTE: THIS IS A WIP! This message will be updated!
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~pelpsi/launchpad/+git/launchpad/+merge/461586
--
Your team Launchpad code reviewers is requested to review the proposed merge of
~pelpsi/launchpad:request-token-and-session-from-fetch-service into
launchpad:master.
diff --git a/lib/lp/buildmaster/builderproxy.py b/lib/lp/buildmaster/builderproxy.py
index be65e29..6857305 100644
--- a/lib/lp/buildmaster/builderproxy.py
+++ b/lib/lp/buildmaster/builderproxy.py
@@ -20,7 +20,10 @@ from typing import Dict, Generator
from twisted.internet import defer
-from lp.buildmaster.downloader import RequestProxyTokenCommand
+from lp.buildmaster.downloader import (
+RequestFetchServiceSessionCommand,
+RequestProxyTokenCommand,
+)
from lp.buildmaster.interfaces.builder import CannotBuild
from lp.buildmaster.interfaces.buildfarmjobbehaviour import BuildArgs
from lp.services.config import config
@@ -36,9 +39,16 @@ class BuilderProxyMixin:
@defer.inlineCallbacks
def addProxyArgs(
-self, args: BuildArgs, allow_internet: bool = True
+self,
+args: BuildArgs,
+allow_internet: bool = True,
+fetch_service: bool = False,
) -> Generator[None, Dict[str, str], None]:
-if _get_proxy_config("builder_proxy_host") and allow_internet:
+if (
+not fetch_service
+and _get_proxy_config("builder_proxy_host")
+and allow_internet
+):
token = yield self._requestProxyToken()
args["proxy_url"] = (
"http://{username}:{password}@{host}:{port}".format(
@@ -52,6 +62,25 @@ class BuilderProxyMixin:
endpoint=_get_proxy_config("builder_proxy_auth_api_endpoint"),
token=token["username"],
)
+if (
+fetch_service
+and _get_proxy_config("fetch_service_host")
+and allow_internet
+):
+# http://:@:9988/
+token = yield self._requestFetchServiceSession()
+args["proxy_url"] = (
+"http://{session_id}:{token}@{host}:{port}".format(
+session_id=token["id"],
+token=token["token"],
+host=_get_proxy_config("fetch_service_host"),
+port=_get_proxy_config("fetch_service_port"),
+)
+)
+args["revocation_endpoint"] = "{endpoint}/{token}".format(
+endpoint=_get_proxy_config("fetch_service_auth_api_endpoint"),
+token=token["username"],
+)
@defer.inlineCallbacks
def _requestProxyToken(self):
@@ -86,3 +115,39 @@ class BuilderProxyMixin:
proxy_username=proxy_username,
)
return token
+
+@defer.inlineCallbacks
+def _requestFetchServiceSession(self):
+# This is a different function if we have the needs to
+# differentiate more the behavior.
+admin_username = _get_proxy_config(
+"fetch_service_auth_api_admin_username"
+)
+if not admin_username:
+raise CannotBuild(
+"fetch_service_auth_api_admin_username is not configured."
+)
+secret = _get_proxy_config("fetch_service_auth_api_admin_secret")
+if not secret:
+raise CannotBuild(
+"fetch_service_auth_api_admin_secret is not configured."
+)
+url = _get_proxy_config("fetch_service_auth_api_endpoint")
+if not secret:
+raise CannotBuild(
+"fetch_service_auth_api_endpoint is not configured."
+)
+timestamp = int(time.time())
+proxy_username = "{build_id}-{timestamp}".format(
+build_id=self.build.build_cookie, timestamp=timestamp
+)
+auth_string = f"{admin_username}:{secret}".strip()
+auth_header = b"Basic " + base64.b64encode(auth_string.encode("ASCII"))
+
+token = yield self._worker.process_pool.doWork(
+RequestFetchServiceSessionCommand,
+url=url,
+auth_header=auth_header,
+proxy_username=proxy_username,
+)
+return token
diff --git a/lib/lp/buildmaster/downloader.py b/lib/lp/buildmaster/downloader.py
index fc16852..ecc10f9 100644
--- a/lib/lp/buildmaster/downloader.py
+++ b/lib/lp/buildmaster/downloader.py
@@ -10,6 +10,7 @@ anything from the rest of Launchpad.
__all__ = [
"DownloadCommand",
"RequestProcess",
+"RequestFetchServiceSessionCommand",
"RequestProxyTokenCommand",
]
@@ -37,6 +38,21 @@ class
[Launchpad-reviewers] [Merge] ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master
The proposal to merge ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master has been updated. Status: Needs review => Work in progress For more details, see: https://code.launchpad.net/~pelpsi/launchpad/+git/launchpad/+merge/461586 -- Your team Launchpad code reviewers is requested to review the proposed merge of ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master. ___ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : launchpad-reviewers@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
[Launchpad-reviewers] [Merge] ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master
The proposal to merge ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master has been updated. Status: Work in progress => Superseded For more details, see: https://code.launchpad.net/~pelpsi/launchpad/+git/launchpad/+merge/461458 -- Your team Launchpad code reviewers is requested to review the proposed merge of ~pelpsi/launchpad:request-token-and-session-from-fetch-service into launchpad:master. ___ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : launchpad-reviewers@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
[Launchpad-reviewers] [Merge] ~andersson123/lpci:devel-release-support into lpci:main
The proposal to merge ~andersson123/lpci:devel-release-support into lpci:main has been updated. Status: Needs review => Work in progress For more details, see: https://code.launchpad.net/~andersson123/lpci/+git/lpci/+merge/461564 -- Your team Launchpad code reviewers is requested to review the proposed merge of ~andersson123/lpci:devel-release-support into lpci:main. ___ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : launchpad-reviewers@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
[Launchpad-reviewers] [Merge] ~andersson123/lpci:devel-release-support into lpci:main
Tim Andersson has proposed merging ~andersson123/lpci:devel-release-support
into lpci:main.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~andersson123/lpci/+git/lpci/+merge/461564
--
Your team Launchpad code reviewers is requested to review the proposed merge of
~andersson123/lpci:devel-release-support into lpci:main.
diff --git a/lpci/commands/run.py b/lpci/commands/run.py
index e42a8a9..c52a884 100644
--- a/lpci/commands/run.py
+++ b/lpci/commands/run.py
@@ -11,6 +11,7 @@ import subprocess
import tempfile
from argparse import ArgumentParser, Namespace
from pathlib import Path, PurePath
+from distro_info import UbuntuDistroInfo
from tempfile import NamedTemporaryFile
from typing import Dict, List, Optional, Set
@@ -425,6 +426,35 @@ def _install_apt_packages(
)
+def _run_instance_command_silent(
+command: str,
+job_name: str,
+job: Job,
+instance: lxd.LXDInstance,
+host_architecture: str,
+remote_cwd: Path,
+environment: Optional[Dict[str, Optional[str]]],
+root: bool = True,
+) -> None:
+full_run_cmd = ["bash", "--noprofile", "--norc", "-ec", command]
+emit.message(f"Running command for the job: {full_run_cmd}")
+with open("/dev/null", "w") as stream:
+proc = instance.execute_run(
+full_run_cmd,
+cwd=remote_cwd,
+env=environment,
+stdout=stream,
+stderr=stream,
+)
+if proc.returncode != 0:
+raise CommandError(
+f"Job {job_name!r} for "
+f"{job.series}/{host_architecture} failed with "
+f"exit status {proc.returncode}.",
+retcode=proc.returncode,
+)
+
+
def _run_instance_command(
command: str,
job_name: str,
@@ -539,6 +569,43 @@ def _run_job(
cwd = Path.cwd()
remote_cwd = env.get_managed_environment_project_path()
+# if job.series == "devel":
+# udi = UbuntuDistroInfo()
+# supported = udi.supported()
+# # only caveat is that distro-info-data must be up to date for this to work on
+# # the machine hosting the container.
+# devel = udi.devel()
+# if devel in supported:
+# supported.remove(devel)
+# job.series = supported[-1]
+# emit.message("Job series is 'devel', upgrading from latest release to development release.")
+# with provider.launched_environment(
+# project_name=cwd.name,
+# project_path=cwd,
+# series=job.series,
+# architecture=host_architecture,
+# gpu_nvidia=gpu_nvidia,
+# root=root,
+# ) as instance:
+# upgrade_commands = [
+# f"sed -i 's/{job.series}/{devel}/g' /etc/apt/sources.list",
+# "sudo DEBIAN_FRONTEND=noninteractive apt update -y",
+# "sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y",
+# "sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y",
+# "sudo DEBIAN_FRONTEND=noninteractive apt-get autoclean -y",
+# "sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove -y",
+# ]
+# for cmd in upgrade_commands:
+# _run_instance_command_silent(
+# command=cmd,
+# job_name=job_name,
+# job=job,
+# instance=instance,
+# host_architecture=host_architecture,
+# remote_cwd=remote_cwd,
+# environment=environment,
+# root=root,
+# )
emit.progress(
f"Launching environment for {job.series}/{host_architecture}"
@@ -551,6 +618,44 @@ def _run_job(
gpu_nvidia=gpu_nvidia,
root=root,
) as instance:
+# Balls. it has to go here.
+if job.series == "devel":
+udi = UbuntuDistroInfo()
+supported = udi.supported()
+# only caveat is that distro-info-data must be up to date for this to work on
+# the machine hosting the container.
+devel = udi.devel()
+if devel in supported:
+supported.remove(devel)
+job.series = supported[-1]
+emit.message("Job series is 'devel', upgrading from latest release to development release.")
+with provider.launched_environment(
+project_name=cwd.name,
+project_path=cwd,
+series=job.series,
+architecture=host_architecture,
+gpu_nvidia=gpu_nvidia,
+root=root,
+) as instance:
+upgrade_commands = [
+f"sed -i 's/{job.series}/{devel}/g' /etc/apt/sources.list",
+"sudo DEBIAN_FRONTEND=noninteractive apt update -y",
+"sudo
[Launchpad-reviewers] [Merge] ~ines-almeida/lp-source-dependencies:revert-pyyaml-dependency-deletion into lp-source-dependencies:master
Ines Almeida has proposed merging ~ines-almeida/lp-source-dependencies:revert-pyyaml-dependency-deletion into lp-source-dependencies:master. Commit message: Revert "Remove PyYAML 5.3.1 and 5.4.1" This reverts commit af9c3a1b91538b0488e7ca57f9bba86fb4b53741. Requested reviews: Launchpad code reviewers (launchpad-reviewers) For more details, see: https://code.launchpad.net/~ines-almeida/lp-source-dependencies/+git/lp-source-dependencies/+merge/461524 -- Your team Launchpad code reviewers is requested to review the proposed merge of ~ines-almeida/lp-source-dependencies:revert-pyyaml-dependency-deletion into lp-source-dependencies:master. diff --git a/dist/PyYAML-5.3.1.tar.gz b/dist/PyYAML-5.3.1.tar.gz new file mode 100644 index 000..915d67b Binary files /dev/null and b/dist/PyYAML-5.3.1.tar.gz differ diff --git a/dist/PyYAML-5.4.1.tar.gz b/dist/PyYAML-5.4.1.tar.gz new file mode 100644 index 000..187c66e Binary files /dev/null and b/dist/PyYAML-5.4.1.tar.gz differ ___ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : launchpad-reviewers@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
