- squashed the two patches (Daniel) - dropped the "docker pull" command. The --cache-from option already fetches the cached layers. Per docker's documentation:
"Upon importing the cache, the builder will only pull the JSON metadata from the registry and determine possible cache hits based on that information. If there is a cache hit, the matched layers are pulled into the local environment." https://docs.docker.com/engine/reference/commandline/build/#cache-from Thanks v2: https://lore.kernel.org/r/20230224125207.19616-1-faro...@suse.de Turns out we were already pulling the image, but for forks we were pulling from a different registry and for the main project we had caching disabled due to a bug with docker/gitlab. I've now replaced the usage of docker.py with a plain docker command and reintroduced the caching. I also removed the COMMON_TAG which was unused. v1: https://lore.kernel.org/r/20230223142154.31975-1-faro...@suse.de I'm not sure if this was discussed previously, but I noticed we're not pulling the images we push to the registry at every pipeline run. I would expect we don't actually need to rebuild container images at _every_ pipeline run, so I propose we add a "docker pull" to the container templates. We already have that for the docker-edk2|opensbi images. Some containers can take a long time to build (14 mins) and pulling the image first without building can cut the time to about 3 mins. With this we can save almost 2h of cumulative CI time per pipeline run: | master | pull-only | diff | container | 0:02:34 | 0:02:09 | 00:00:25 | alpha-debian-cross-container | 0:04:45 | 0:02:40 | 00:02:05 | amd64-alpine-container | 0:09:51 | 0:02:56 | 00:06:55 | amd64-centos8-container | 0:07:21 | 0:02:49 | 00:04:32 | amd64-debian-container | 0:06:00 | 0:02:37 | 00:03:23 | amd64-debian-cross-container | 0:14:22 | 0:03:41 | 00:10:41 | amd64-debian-user-cross-container | 0:10:14 | 0:03:24 | 00:06:50 | amd64-fedora-container | 0:12:09 | 0:02:49 | 00:09:20 | amd64-opensuse-leap-container | 0:07:33 | 0:02:45 | 00:04:48 | amd64-ubuntu2004-container | 0:08:28 | 0:03:07 | 00:05:21 | arm64-debian-cross-container | 0:04:27 | 0:02:58 | 00:01:29 | armel-debian-cross-container | 0:08:01 | 0:02:55 | 00:05:06 | armhf-debian-cross-container | 0:03:33 | 0:02:18 | 00:01:15 | cris-fedora-cross-container | 0:00:28 | 0:00:28 | 00:00:00 | docker-edk2 | 0:00:25 | 0:00:28 |-00:00:03 | docker-opensbi | 0:08:34 | 0:03:10 | 00:05:24 | hexagon-cross-container | 0:02:34 | 0:02:08 | 00:00:26 | hppa-debian-cross-container | 0:04:50 | 0:02:28 | 00:02:22 | i386-fedora-cross-container | 0:02:36 | 0:02:12 | 00:00:24 | m68k-debian-cross-container | 0:02:40 | 0:02:09 | 00:00:31 | mips-debian-cross-container | 0:02:38 | 0:02:09 | 00:00:29 | mips64-debian-cross-container | 0:04:28 | 0:02:48 | 00:01:40 | mips64el-debian-cross-container | 0:07:07 | 0:02:46 | 00:04:21 | mipsel-debian-cross-container | 0:03:51 | 0:02:21 | 00:01:30 | powerpc-test-cross-container | 0:08:52 | 0:03:00 | 00:05:52 | ppc64el-debian-cross-container | 0:06:07 | 0:02:49 | 00:03:18 | python-container | 0:04:37 | 0:02:26 | 00:02:11 | riscv64-debian-cross-container | 0:02:39 | 0:02:08 | 00:00:31 | riscv64-debian-test-cross-container | 0:08:03 | 0:03:00 | 00:05:03 | s390x-debian-cross-container | 0:02:34 | 0:02:08 | 00:00:26 | sh4-debian-cross-container | 0:02:37 | 0:02:09 | 00:00:28 | sparc64-debian-cross-container | 0:04:25 | 0:02:17 | 00:02:08 | tricore-debian-cross-container | 0:12:51 | 0:03:27 | 00:09:24 | win32-fedora-cross-container | 0:11:16 | 0:03:29 | 00:07:47 | win64-fedora-cross-container | 0:03:28 | 0:02:20 | 00:01:08 | xtensa-debian-cross-container | 01:57:30 | We would need to devise a mechanism (not included here) to force the re-build of the container images when needed, perhaps an environment variable or even a whole new "container build" stage before the "container" stage. What do you think? Fabiano Rosas (1): gitlab: Use plain docker in container-template.yml .gitlab-ci.d/container-template.yml | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) -- 2.35.3
