Re: [PATCH] gitlab-ci: Replace Docker with Kaniko
On Fri, May 17, 2024 at 9:14 AM Daniel P. Berrangé wrote: > > On Thu, May 16, 2024 at 07:24:04PM +0100, Daniel P. Berrangé wrote: > > On Thu, May 16, 2024 at 05:52:43PM +0100, Camilla Conte wrote: > > > Enables caching from the qemu-project repository. > > > > > > Uses a dedicated "$NAME-cache" tag for caching, to address limitations. > > > See issue "when using --cache=true, kaniko fail to push cache layer > > > [...]": > > > https://github.com/GoogleContainerTools/kaniko/issues/1459 > > > > After investigating, this is a result of a different design approach > > for caching in kaniko. > > > > In docker, it can leverage any existing image as a cache source, > > reusing individual layers that were present. IOW, there's no > > difference between a cache and a final image, they're one and the > > same thing > > > > In kaniko, the cache is a distinct object type. IIUC, it is not > > populated with the individual layers, instead it has a custom > > format for storing the cached content. Therefore the concept of > > storing the cache at the same location as the final image, is > > completely inappropriate - you can't store two completely different > > kinds of content at the same place. > > > > That is also why you can't just "git pull" the fetch the cache > > image(s) beforehand, and also why it doesn't look like you can > > use multiple cache sources with kaniko. > > > > None of this is inherantly a bad thing. except when it comes > > to data storage. By using Kaniko we would, at minimum, doubling > > the amount of data storage we consume in the gitlab registry. > > Double is actually just the initial case. The cache is storing layers > using docker tags, whose name appears based on a hash of the "RUN" > command. > > IOW, the first time we build a container we have double the usage. > When a dockerfile is updated changing a 'RUN' command, we now have > triple the storage usage for cache. Update the RUN command again, > and we now have quadruple the storage. etc. > > Kaniko does not appear to purge cache entries itself, and will rely > on something else to do the cache purging. > > GitLab has support for purging old docker tags, but I'm not an > admin on the QEMU project namespace, so can't tell if it can be > enabled or not ? Many older projects have this permanently disabled > due to historical compat issues in gitlab after they introduced the > feature. I'm pretty sure purging can be enabled. Gitlab itself proposes this with a "set up cleanup" link on the registry page (1). Can you recall what issues they were experiencing? If this is the only issue blocking Kaniko adoption, and we can't solve it by enabling the cleanup, I can write an additional step at the end of the container build to explicitly remove old cache tags. (1) https://gitlab.com/qemu-project/qemu/container_registry > > With regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o-https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| >
[PATCH] gitlab-ci: Replace Docker with Kaniko
Enables caching from the qemu-project repository. Uses a dedicated "$NAME-cache" tag for caching, to address limitations. See issue "when using --cache=true, kaniko fail to push cache layer [...]": https://github.com/GoogleContainerTools/kaniko/issues/1459 Does not specify a context since no Dockerfile is using COPY or ADD instructions. Does not enable reproducible builds as that results in builds failing with an out of memory error. See issue "Using --reproducible loads entire image into memory": https://github.com/GoogleContainerTools/kaniko/issues/862 Previous attempts, for the records: - Alex Bennée: https://lore.kernel.org/qemu-devel/20230330101141.30199-12-alex.ben...@linaro.org/ - Camilla Conte (me): https://lore.kernel.org/qemu-devel/20230531150824.32349-6-cco...@redhat.com/ Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 25 +++-- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 4eec72f383..066f253dd5 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,21 +1,18 @@ .container_job_template: extends: .base_job_template - image: docker:latest stage: containers - services: -- docker:dind + image: +name: gcr.io/kaniko-project/executor:debug +entrypoint: [""] + variables: +DOCKERFILE: "$CI_PROJECT_DIR/tests/docker/dockerfiles/$NAME.docker" +CACHE_REPO: "$CI_REGISTRY/qemu-project/qemu/qemu/$NAME-cache" before_script: - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:$QEMU_CI_CONTAINER_TAG" -# Always ':latest' because we always use upstream as a common cache source -- export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" -- docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" -- until docker info; do sleep 1; done script: - echo "TAG:$TAG" -- echo "COMMON_TAG:$COMMON_TAG" -- docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" - --build-arg BUILDKIT_INLINE_CACHE=1 - -f "tests/docker/dockerfiles/$NAME.docker" "." -- docker push "$TAG" - after_script: -- docker logout +- /kaniko/executor + --dockerfile "$DOCKERFILE" + --destination "$TAG" + --cache=true + --cache-repo="$CACHE_REPO" -- 2.45.0
Re: [PATCH] gitlab: switch from 'stable' to 'latest' docker container tags
On Wed, May 31, 2023 at 3:07 PM Daniel P. Berrangé wrote: > > The 'stable' and 'stable-dind' tags are not documented as supported > tags at: > > https://hub.docker.com/_/docker > > Looking at their content they reflect docker 19.x.x release series, > were last built in Dec 2020, and have 3 critical and 20 high rated > CVEs unfixed. This obsolete status is attested by this commit: > > > https://github.com/docker-library/docker/commit/606c63960a4845af7077721eb3900c706f5d0c5e > > The 'stable-dind' tag in particular appears buggy as it is unable to > resolve DNS for Fedora repos: > > - Curl error (6): Couldn't resolve host name for > https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=x86_64&countme=1 > [getaddrinfo() thread failed to start] > > We used the 'stable' tag previously at the recommendation of GitLab > docs, but those docs are wrong and pending a fix: > > https://gitlab.com/gitlab-org/gitlab/-/issues/409430 > > Fixes: 5f63a67adb58478974b91f5e5c2b1222b5c7f2cc > Signed-off-by: Daniel P. Berrangé > --- > .gitlab-ci.d/container-template.yml | 4 ++-- > .gitlab-ci.d/opensbi.yml| 4 ++-- > 2 files changed, 4 insertions(+), 4 deletions(-) > > diff --git a/.gitlab-ci.d/container-template.yml > b/.gitlab-ci.d/container-template.yml > index 9ac4a0ee25..8c1370b8f0 100644 > --- a/.gitlab-ci.d/container-template.yml > +++ b/.gitlab-ci.d/container-template.yml > @@ -1,9 +1,9 @@ > .container_job_template: >extends: .base_job_template > - image: docker:stable > + image: docker:latest >stage: containers >services: > -- docker:stable-dind > +- docker:dind >before_script: > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml > index 2e9d51764e..b4d7eef688 100644 > --- a/.gitlab-ci.d/opensbi.yml > +++ b/.gitlab-ci.d/opensbi.yml > @@ -42,9 +42,9 @@ > docker-opensbi: >extends: .opensbi_job_rules >stage: containers > - image: docker:stable > + image: docker:latest >services: > -- docker:stable-dind > +- docker:dind >variables: > GIT_DEPTH: 3 > IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build > -- > 2.40.1 > Reviewed-by: Camilla Conte
[PATCH 5/5] gitlab-ci: Replace Docker with Kaniko
Previous attempt by Alex Bennée, for the records: https://lore.kernel.org/qemu-devel/20230330101141.30199-12-alex.ben...@linaro.org/ Does not enable caching, as suggested in a previous review: https://lore.kernel.org/qemu-devel/zcvpmdlpjcdnk...@redhat.com/ Does not specify a context since no Dockerfile is using COPY or ADD instructions. Does not enable reproducible builds as that results in builds failing with an out of memory error. See issue "Using --reproducible loads entire image into memory": https://github.com/GoogleContainerTools/kaniko/issues/862 Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 25 +-- .gitlab-ci.d/opensbi.yml | 1 - .../ci/gitlab-kubernetes-runners/values.yaml | 12 - 3 files changed, 6 insertions(+), 32 deletions(-) diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index d146d1d197..0e93f5f13e 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,27 +1,14 @@ .container_job_template: extends: .base_job_template stage: containers - image: docker:stable - services: -- docker:stable-dind + image: +name: gcr.io/kaniko-project/executor:debug +entrypoint: [""] variables: TAG: "$CI_REGISTRY_IMAGE/qemu/$NAME:latest" -COMMON_TAG: "$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" DOCKERFILE: "$CI_PROJECT_DIR/tests/docker/dockerfiles/$NAME.docker" -BUILD_CONTEXT: "$CI_PROJECT_DIR" - before_script: -- docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" -- until docker info; do sleep 1; done script: - echo "TAG:$TAG" -- echo "COMMON_TAG:$COMMON_TAG" -- > - docker build - --tag "$TAG" - --cache-from "$TAG" - --cache-from "$COMMON_TAG" - --build-arg BUILDKIT_INLINE_CACHE=1 - -f "$DOCKERFILE" "$BUILD_CONTEXT" -- docker push "$TAG" - after_script: -- docker logout +- /kaniko/executor + --dockerfile "$DOCKERFILE" + --destination "$TAG" diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index bb90c56bcf..e531ce2728 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -50,7 +50,6 @@ docker-opensbi: GIT_DEPTH: 3 NAME: opensbi-cross-build DOCKERFILE: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi/Dockerfile" -BUILD_CONTEXT: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi" build-opensbi: rules: diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml index 204a96a842..ed6c5894bc 100644 --- a/scripts/ci/gitlab-kubernetes-runners/values.yaml +++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml @@ -4,27 +4,15 @@ rbac: create: true concurrent: 200 runners: - privileged: true config: | [[runners]] limit = 100 - environment = [ -"DOCKER_HOST=tcp://docker:2376", -"DOCKER_TLS_CERTDIR=/certs", -"DOCKER_TLS_VERIFY=1", -"DOCKER_CERT_PATH=/certs/client" - ] [runners.kubernetes] poll_timeout = 1200 -image = "ubuntu:20.04" cpu_request = "0.5" service_cpu_request = "0.5" helper_cpu_request = "0.25" cpu_request_overwrite_max_allowed = "7" memory_request_overwrite_max_allowed = "30Gi" - [[runners.kubernetes.volumes.empty_dir]] -name = "docker-certs" -mount_path = "/certs/client" -medium = "Memory" [runners.kubernetes.node_selector] agentpool = "jobs" -- 2.40.1
[PATCH 1/5] gitlab-ci: Remove unused Python package
Python should have been removed in this commit: https://gitlab.com/qemu-project/qemu/-/commit/94b8b146df84ba472f461398d93fb9cdf0db8f94 Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 9ac4a0ee25..fd7deb1c29 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -7,7 +7,6 @@ before_script: - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" -- apk add python3 - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" - until docker info; do sleep 1; done script: -- 2.40.1
Improvements to Gitlab CI container builds
Given my recent deep-dive in this part of the Gitlab CI, I figured I could propose some improvements that came to mind along the way. The last patch removes the need for Docker in Docker.
[PATCH 2/5] gitlab-ci: Reference rules instead of extends
This allows for the jobs to extend different templates while having a shared template for rules. Docs: https://docs.gitlab.com/ee/ci/jobs/job_control.html#reuse-rules-in-different-jobs Signed-off-by: Camilla Conte --- .gitlab-ci.d/opensbi.yml | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index 2e9d51764e..d0221632f6 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -40,7 +40,8 @@ when: on_success docker-opensbi: - extends: .opensbi_job_rules + rules: +- !reference [.opensbi_job_rules, rules] stage: containers image: docker:stable services: @@ -59,7 +60,8 @@ docker-opensbi: - docker push $IMAGE_TAG build-opensbi: - extends: .opensbi_job_rules + rules: +- !reference [.opensbi_job_rules, rules] stage: build needs: ['docker-opensbi'] artifacts: -- 2.40.1
[PATCH 3/5] gitlab-ci: Move variables outside script
Also split long line for clarity. Docs: https://docs.gitlab.com/ee/ci/variables/#use-cicd-variables-in-other-variables Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 17 - 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index fd7deb1c29..d146d1d197 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,20 +1,27 @@ .container_job_template: extends: .base_job_template - image: docker:stable stage: containers + image: docker:stable services: - docker:stable-dind + variables: +TAG: "$CI_REGISTRY_IMAGE/qemu/$NAME:latest" +COMMON_TAG: "$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" +DOCKERFILE: "$CI_PROJECT_DIR/tests/docker/dockerfiles/$NAME.docker" +BUILD_CONTEXT: "$CI_PROJECT_DIR" before_script: -- export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" -- export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" - until docker info; do sleep 1; done script: - echo "TAG:$TAG" - echo "COMMON_TAG:$COMMON_TAG" -- docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" +- > + docker build + --tag "$TAG" + --cache-from "$TAG" + --cache-from "$COMMON_TAG" --build-arg BUILDKIT_INLINE_CACHE=1 - -f "tests/docker/dockerfiles/$NAME.docker" "." + -f "$DOCKERFILE" "$BUILD_CONTEXT" - docker push "$TAG" after_script: - docker logout -- 2.40.1
[PATCH 4/5] gitlab-ci: Use container template for opensbi
Use the same template for all the jobs in the "container" stage. Changes the URL of the "opensbi-cross-build" images by using the same URL pattern as the other images. Removes pushing the image to the additional CI_COMMIT_SHA tag. It seems unnecessary. Example of the old URLs: - registry.gitlab.com/qemu-project/qemu:00a0bdc... - registry.gitlab.com/qemu-project/qemu:opensbi-cross-build Example of the new URL: - registry.gitlab.com/qemu-project/qemu/qemu/opensbi-cross-build Signed-off-by: Camilla Conte --- .gitlab-ci.d/opensbi.yml | 23 --- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index d0221632f6..bb90c56bcf 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -1,3 +1,6 @@ +include: + - local: '/.gitlab-ci.d/container-template.yml' + # All jobs needing docker-opensbi must use the same rules it uses. .opensbi_job_rules: rules: @@ -40,24 +43,14 @@ when: on_success docker-opensbi: + extends: .container_job_template rules: - !reference [.opensbi_job_rules, rules] - stage: containers - image: docker:stable - services: -- docker:stable-dind variables: GIT_DEPTH: 3 -IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build - before_script: -- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY -- until docker info; do sleep 1; done - script: -- docker pull $IMAGE_TAG || true -- docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - --tag $IMAGE_TAG .gitlab-ci.d/opensbi -- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -- docker push $IMAGE_TAG +NAME: opensbi-cross-build +DOCKERFILE: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi/Dockerfile" +BUILD_CONTEXT: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi" build-opensbi: rules: @@ -72,7 +65,7 @@ build-opensbi: - opensbi32-generic-stderr.log - opensbi64-generic-stdout.log - opensbi64-generic-stderr.log - image: $CI_REGISTRY_IMAGE:opensbi-cross-build + image: $CI_REGISTRY_IMAGE/qemu/opensbi-cross-build:latest variables: GIT_DEPTH: 3 script: # Clone the required submodules and build OpenSBI -- 2.40.1
Re: Add CI configuration for Kubernetes
On Tue, May 23, 2023 at 2:08 PM Richard Henderson wrote: > > On 5/23/23 01:24, Camilla Conte wrote: > > On Mon, May 22, 2023 at 11:52 PM Richard Henderson > > wrote: > >> This does not work: > >> > >> https://gitlab.com/qemu-project/qemu/-/pipelines/875254290 > > > > My bad, sorry. I didn't update the runner with the new values.yaml. > > The Docker environment variables were missing so it's looking for the > > Docker server at the wrong place. > > > > Now I updated the runner. You have to add the "k8s" tag to the runner > > again in the settings: > > https://gitlab.com/qemu-project/qemu/-/settings/ci_cd#Runners. > > As soon as the "k8s" tag is there the pipeline should be able to run. > > > > The k8s tag has been there on the runner all along. Did you mean something > else? I had Paolo adding it a few hours ago. > > > r~ >
Re: [PATCH v2 3/5] Add loop over docker info
On Tue, May 23, 2023 at 2:06 PM Richard Henderson wrote: > > On 5/22/23 10:41, Camilla Conte wrote: > > +- until docker info; do sleep 1; done > > How long are you expecting this to have to wait? > Less than 5 seconds? I remember seeing jobs where it took longer than five seconds, but I can't find them at the moment. I guess you wish the "until" loop to fail sooner, rather than having to wait for the whole job timing out. What about setting a five seconds timeout for the loop and raise it later if we see jobs failing? > > > r~ >
[PATCH v3 5/5] Add Kubernetes runner configuration
Custom values for the gitlab-runner Helm chart. See https://wiki.qemu.org/Testing/CI/KubernetesRunners. Signed-off-by: Camilla Conte --- Changes since previous version (PATCH v2 5/5): - Update ubuntu tag to 22.04. I tested it and it works. - Remove concurrent jobs limits (use defaults). .../ci/gitlab-kubernetes-runners/values.yaml | 28 +++ 1 file changed, 28 insertions(+) create mode 100644 scripts/ci/gitlab-kubernetes-runners/values.yaml diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml new file mode 100644 index 00..34c9befdce --- /dev/null +++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml @@ -0,0 +1,28 @@ +gitlabUrl: "https://gitlab.com/"; +runnerRegistrationToken: "" +rbac: + create: true +runners: + privileged: true + config: | +[[runners]] + environment = [ +"DOCKER_HOST=tcp://docker:2376", +"DOCKER_TLS_CERTDIR=/certs", +"DOCKER_TLS_VERIFY=1", +"DOCKER_CERT_PATH=/certs/client" + ] + [runners.kubernetes] +poll_timeout = 1200 +image = "ubuntu:22.04" +cpu_request = "0.5" +service_cpu_request = "0.5" +helper_cpu_request = "0.25" +cpu_request_overwrite_max_allowed = "7" +memory_request_overwrite_max_allowed = "30Gi" + [[runners.kubernetes.volumes.empty_dir]] +name = "docker-certs" +mount_path = "/certs/client" +medium = "Memory" + [runners.kubernetes.node_selector] +agentpool = "jobs" -- 2.40.1
Re: Add CI configuration for Kubernetes
On Mon, May 22, 2023 at 11:52 PM Richard Henderson wrote: > This does not work: > > https://gitlab.com/qemu-project/qemu/-/pipelines/875254290 My bad, sorry. I didn't update the runner with the new values.yaml. The Docker environment variables were missing so it's looking for the Docker server at the wrong place. Now I updated the runner. You have to add the "k8s" tag to the runner again in the settings: https://gitlab.com/qemu-project/qemu/-/settings/ci_cd#Runners. As soon as the "k8s" tag is there the pipeline should be able to run.
[PATCH v2 3/5] Add loop over docker info
Wait for docker info to return successfuly to ensure that the docker server (daemon) started. This is needed for jobs running on Kubernetes. See https://wiki.qemu.org/Testing/CI/KubernetesRunners. Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 2 +- .gitlab-ci.d/opensbi.yml| 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 11569dd900..9ac4a0ee25 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -8,8 +8,8 @@ - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" - apk add python3 -- docker info - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" +- until docker info; do sleep 1; done script: - echo "TAG:$TAG" - echo "COMMON_TAG:$COMMON_TAG" diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index 13070575b6..2e9d51764e 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -50,6 +50,7 @@ docker-opensbi: IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY +- until docker info; do sleep 1; done script: - docker pull $IMAGE_TAG || true - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -- 2.40.1
[PATCH v2 5/5] Add Kubernetes runner configuration
Custom values for the gitlab-runner Helm chart. See https://wiki.qemu.org/Testing/CI/KubernetesRunners. Signed-off-by: Camilla Conte --- .../ci/gitlab-kubernetes-runners/values.yaml | 30 +++ 1 file changed, 30 insertions(+) create mode 100644 scripts/ci/gitlab-kubernetes-runners/values.yaml diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml new file mode 100644 index 00..204a96a842 --- /dev/null +++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml @@ -0,0 +1,30 @@ +gitlabUrl: "https://gitlab.com/"; +runnerRegistrationToken: "" +rbac: + create: true +concurrent: 200 +runners: + privileged: true + config: | +[[runners]] + limit = 100 + environment = [ +"DOCKER_HOST=tcp://docker:2376", +"DOCKER_TLS_CERTDIR=/certs", +"DOCKER_TLS_VERIFY=1", +"DOCKER_CERT_PATH=/certs/client" + ] + [runners.kubernetes] +poll_timeout = 1200 +image = "ubuntu:20.04" +cpu_request = "0.5" +service_cpu_request = "0.5" +helper_cpu_request = "0.25" +cpu_request_overwrite_max_allowed = "7" +memory_request_overwrite_max_allowed = "30Gi" + [[runners.kubernetes.volumes.empty_dir]] +name = "docker-certs" +mount_path = "/certs/client" +medium = "Memory" + [runners.kubernetes.node_selector] +agentpool = "jobs" -- 2.40.1
Add CI configuration for Kubernetes
Here's a second version (v2) of patches to support the Kubernetes runner for Gitlab CI. You can find the v1 thread here: https://lore.kernel.org/qemu-devel/20230407145252.32955-1-cco...@redhat.com/.
[PATCH v2 2/5] Use docker "stable" tag
Use the same tag in all jobs. Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 519b8a9482..11569dd900 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -3,7 +3,7 @@ image: docker:stable stage: containers services: -- docker:dind +- docker:stable-dind before_script: - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" -- 2.40.1
[PATCH v2 4/5] Add CI variable RUNNER_TAG
This allows to set a job tag dinamically. We need this to be able to select the Kubernetes runner. See https://wiki.qemu.org/Testing/CI/KubernetesRunners. Signed-off-by: Camilla Conte --- .gitlab-ci.d/qemu-project.yml | 7 +++ 1 file changed, 7 insertions(+) diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index a7ed447fe4..4d914c4897 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -1,6 +1,13 @@ # This file contains the set of jobs run by the QEMU project: # https://gitlab.com/qemu-project/qemu/-/pipelines +variables: + RUNNER_TAG: "" + +default: + tags: +- $RUNNER_TAG + include: - local: '/.gitlab-ci.d/base.yml' - local: '/.gitlab-ci.d/stages.yml' -- 2.40.1
[PATCH v2 1/5] Remove redundant CI variables
These are not needed when using gitlab.com shared runners. Signed-off-by: Camilla Conte --- .gitlab-ci.d/opensbi.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index 9a651465d8..13070575b6 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -48,9 +48,6 @@ docker-opensbi: variables: GIT_DEPTH: 3 IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build -# We don't use TLS -DOCKER_HOST: tcp://docker:2375 -DOCKER_TLS_CERTDIR: "" before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY script: -- 2.40.1
Re: [PATCH 1/2] Add CI configuration for Kubernetes
On Fri, May 19, 2023 at 1:51 PM Daniel P. Berrangé wrote: > > On Fri, May 19, 2023 at 01:33:50PM +0100, Camilla Conte wrote: > > On Fri, May 19, 2023 at 10:00 AM Daniel P. Berrangé > > wrote: > > > > > > On Fri, Apr 07, 2023 at 03:52:51PM +0100, Camilla Conte wrote: > > > > Configure Gitlab CI to run on Kubernetes > > > > according to the official documentation. > > > > https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled-in-kubernetes > > > > > > > > These changes are needed because of the CI jobs > > > > using Docker-in-Docker (dind). > > > > As soon as Docker-in-Docker is replaced with Kaniko, > > > > these changes can be reverted. > > > > > > > > I documented what I did to set up the Kubernetes runner on the wiki: > > > > https://wiki.qemu.org/Testing/CI/KubernetesRunners > > > > > > > > Signed-off-by: Camilla Conte > > > > --- > > > > .gitlab-ci.d/container-template.yml | 6 +++--- > > > > .gitlab-ci.d/default.yml| 3 +++ > > > > .gitlab-ci.d/opensbi.yml| 8 +++- > > > > .gitlab-ci.d/qemu-project.yml | 17 + > > > > 4 files changed, 26 insertions(+), 8 deletions(-) > > > > create mode 100644 .gitlab-ci.d/default.yml > > > > > > > > diff --git a/.gitlab-ci.d/container-template.yml > > > > b/.gitlab-ci.d/container-template.yml > > > > index 519b8a9482..f55a954741 100644 > > > > --- a/.gitlab-ci.d/container-template.yml > > > > +++ b/.gitlab-ci.d/container-template.yml > > > > @@ -1,14 +1,14 @@ > > > > .container_job_template: > > > >extends: .base_job_template > > > > - image: docker:stable > > > > + image: docker:20.10.16 > > > >stage: containers > > > >services: > > > > -- docker:dind > > > > +- docker:20.10.16-dind > > > >before_script: > > > > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > > > > - export > > > > COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > > > > - apk add python3 > > > > -- docker info > > > > +- until docker info; do sleep 1; done > > > > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p > > > > "$CI_REGISTRY_PASSWORD" > > > >script: > > > > - echo "TAG:$TAG" > > > > diff --git a/.gitlab-ci.d/default.yml b/.gitlab-ci.d/default.yml > > > > new file mode 100644 > > > > index 00..292be8b91c > > > > --- /dev/null > > > > +++ b/.gitlab-ci.d/default.yml > > > > @@ -0,0 +1,3 @@ > > > > +default: > > > > + tags: > > > > +- $RUNNER_TAG > > > > > > Can we just put this in base.yml instead of creating a new file. > > > > Sure. > > > > > > diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml > > > > index 9a651465d8..5b0b47b57b 100644 > > > > --- a/.gitlab-ci.d/opensbi.yml > > > > +++ b/.gitlab-ci.d/opensbi.yml > > > > @@ -42,17 +42,15 @@ > > > > docker-opensbi: > > > >extends: .opensbi_job_rules > > > >stage: containers > > > > - image: docker:stable > > > > + image: docker:20.10.16 > > > >services: > > > > -- docker:stable-dind > > > > +- docker:20.10.16-dind > > > > > > Can you elaborate on this ? I know the docs about use that particular > > > version tag, but they don't appear to explain why. If this is not > > > actually a hard requirements, we should keep using the stable tag. > > > > Yes, we can keep using "stable". > > Then, we should be ready to address future issues that may arise from > > "stable" not being compatible with the runner. > > > > > >variables: > > > > GIT_DEPTH: 3 > > > > IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build > > > > -# We don't use TLS > > > > -DOCKER_HOST: tcp://docker:2375 > > > > -DOCKER_TLS_CERTDIR: "" > > > > > > So IIUC, this was always redundant when using gitlab CI. We should just > > > remove these in a standalone commit. > > > > Okay, I'll
Re: [PATCH 1/2] Add CI configuration for Kubernetes
On Fri, May 19, 2023 at 10:00 AM Daniel P. Berrangé wrote: > > On Fri, Apr 07, 2023 at 03:52:51PM +0100, Camilla Conte wrote: > > Configure Gitlab CI to run on Kubernetes > > according to the official documentation. > > https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled-in-kubernetes > > > > These changes are needed because of the CI jobs > > using Docker-in-Docker (dind). > > As soon as Docker-in-Docker is replaced with Kaniko, > > these changes can be reverted. > > > > I documented what I did to set up the Kubernetes runner on the wiki: > > https://wiki.qemu.org/Testing/CI/KubernetesRunners > > > > Signed-off-by: Camilla Conte > > --- > > .gitlab-ci.d/container-template.yml | 6 +++--- > > .gitlab-ci.d/default.yml| 3 +++ > > .gitlab-ci.d/opensbi.yml| 8 +++- > > .gitlab-ci.d/qemu-project.yml | 17 + > > 4 files changed, 26 insertions(+), 8 deletions(-) > > create mode 100644 .gitlab-ci.d/default.yml > > > > diff --git a/.gitlab-ci.d/container-template.yml > > b/.gitlab-ci.d/container-template.yml > > index 519b8a9482..f55a954741 100644 > > --- a/.gitlab-ci.d/container-template.yml > > +++ b/.gitlab-ci.d/container-template.yml > > @@ -1,14 +1,14 @@ > > .container_job_template: > >extends: .base_job_template > > - image: docker:stable > > + image: docker:20.10.16 > >stage: containers > >services: > > -- docker:dind > > +- docker:20.10.16-dind > >before_script: > > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > > - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > > - apk add python3 > > -- docker info > > +- until docker info; do sleep 1; done > > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p > > "$CI_REGISTRY_PASSWORD" > >script: > > - echo "TAG:$TAG" > > diff --git a/.gitlab-ci.d/default.yml b/.gitlab-ci.d/default.yml > > new file mode 100644 > > index 00..292be8b91c > > --- /dev/null > > +++ b/.gitlab-ci.d/default.yml > > @@ -0,0 +1,3 @@ > > +default: > > + tags: > > +- $RUNNER_TAG > > Can we just put this in base.yml instead of creating a new file. Sure. > > diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml > > index 9a651465d8..5b0b47b57b 100644 > > --- a/.gitlab-ci.d/opensbi.yml > > +++ b/.gitlab-ci.d/opensbi.yml > > @@ -42,17 +42,15 @@ > > docker-opensbi: > >extends: .opensbi_job_rules > >stage: containers > > - image: docker:stable > > + image: docker:20.10.16 > >services: > > -- docker:stable-dind > > +- docker:20.10.16-dind > > Can you elaborate on this ? I know the docs about use that particular > version tag, but they don't appear to explain why. If this is not > actually a hard requirements, we should keep using the stable tag. Yes, we can keep using "stable". Then, we should be ready to address future issues that may arise from "stable" not being compatible with the runner. > >variables: > > GIT_DEPTH: 3 > > IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build > > -# We don't use TLS > > -DOCKER_HOST: tcp://docker:2375 > > -DOCKER_TLS_CERTDIR: "" > > So IIUC, this was always redundant when using gitlab CI. We should just > remove these in a standalone commit. Okay, I'll put this in a separate commit. > >before_script: > > - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD > > $CI_REGISTRY > > +- until docker info; do sleep 1; done > > Was this really needed ? The docs don't show that, and docker login is > synchronous, so I wouldn't expect us to them poll on 'docker info'. Unfortunately, yes. We need to wait until the "docker info" command is successful. This ensures that the Docker server has started and the subsequent docker commands won't fail. > In container-template.yml we in fact do the reverse > > - docker info > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p > "$CI_REGISTRY_PASSWORD" About "docker login", as far as I understand it's a client-only command. It doesn't involve the Docker server at all. These two commands are not related to each other, it doesn't matter if "docker login" runs before or after "docker info". > imho best make this opensbi.yml
Re: gitlab shared runner time expired
Hi! On Fri, May 19, 2023 at 9:35 AM Paolo Bonzini wrote: > You should be able to run pipelines almost as usual, just with > > git push -o ci.variable=RUNNER_TAG=k8s ... > > Camilla, please confirm that this is correct. Looks good.
[PATCH 1/2] Add CI configuration for Kubernetes
Configure Gitlab CI to run on Kubernetes according to the official documentation. https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled-in-kubernetes These changes are needed because of the CI jobs using Docker-in-Docker (dind). As soon as Docker-in-Docker is replaced with Kaniko, these changes can be reverted. I documented what I did to set up the Kubernetes runner on the wiki: https://wiki.qemu.org/Testing/CI/KubernetesRunners Signed-off-by: Camilla Conte --- .gitlab-ci.d/container-template.yml | 6 +++--- .gitlab-ci.d/default.yml| 3 +++ .gitlab-ci.d/opensbi.yml| 8 +++- .gitlab-ci.d/qemu-project.yml | 17 + 4 files changed, 26 insertions(+), 8 deletions(-) create mode 100644 .gitlab-ci.d/default.yml diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 519b8a9482..f55a954741 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,14 +1,14 @@ .container_job_template: extends: .base_job_template - image: docker:stable + image: docker:20.10.16 stage: containers services: -- docker:dind +- docker:20.10.16-dind before_script: - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" - apk add python3 -- docker info +- until docker info; do sleep 1; done - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" script: - echo "TAG:$TAG" diff --git a/.gitlab-ci.d/default.yml b/.gitlab-ci.d/default.yml new file mode 100644 index 00..292be8b91c --- /dev/null +++ b/.gitlab-ci.d/default.yml @@ -0,0 +1,3 @@ +default: + tags: +- $RUNNER_TAG diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index 9a651465d8..5b0b47b57b 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -42,17 +42,15 @@ docker-opensbi: extends: .opensbi_job_rules stage: containers - image: docker:stable + image: docker:20.10.16 services: -- docker:stable-dind +- docker:20.10.16-dind variables: GIT_DEPTH: 3 IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build -# We don't use TLS -DOCKER_HOST: tcp://docker:2375 -DOCKER_TLS_CERTDIR: "" before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY +- until docker info; do sleep 1; done script: - docker pull $IMAGE_TAG || true - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index a7ed447fe4..57b175f5c2 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -1,7 +1,24 @@ # This file contains the set of jobs run by the QEMU project: # https://gitlab.com/qemu-project/qemu/-/pipelines +variables: + RUNNER_TAG: "" + +workflow: + rules: +# Set additional variables when running on Kubernetes. +# https://wiki.qemu.org/Testing/CI/KubernetesRunners +- if: $RUNNER_TAG == "k8s" + variables: +DOCKER_HOST: tcp://docker:2376 +DOCKER_TLS_CERTDIR: "/certs" +DOCKER_TLS_VERIFY: 1 +DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client" +# Run the pipeline in other cases. +- when: always + include: + - local: '/.gitlab-ci.d/default.yml' - local: '/.gitlab-ci.d/base.yml' - local: '/.gitlab-ci.d/stages.yml' - local: '/.gitlab-ci.d/opensbi.yml' -- 2.39.2
[PATCH 2/2] Raise crash-test-debian timeout to 90 minutes
When running on the Kubernetes runner, this CI job is timing out. Raise the limit to give the job enough time to run. Signed-off-by: Camilla Conte --- .gitlab-ci.d/buildtest.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index ba6f551752..893f8c69b4 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -98,6 +98,7 @@ crash-test-debian: needs: - job: build-system-debian artifacts: true + timeout: 90m variables: IMAGE: debian-amd64 script: -- 2.39.2
