Getting alternate container tools like buldah to work inside a container with fuse overlays etc is a bit out of GoCD's scope right now.
You should be able to attach volumes like for any other pod, but a host path mount is obviously coupled to the way your nodes are configured within Kubernetes and even further outside GoCD's control. Having said this, fuse is special I think? Normally there are *many* things needed to make buildah work with fuse overlays inside a container re: userns remappings to avoid enabling special privileges and linux capabilities within securityContext which may not be there by default . You *might* just be missing CAP_SETUID and CAP_SETGID capabilities from the container and/or running with *privileged: true* but I'd be surprised if that is all that is needed. https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container#running_buildah_inside_a_container https://github.com/containers/buildah/issues/2325 https://github.com/containers/image_build/blob/main/buildah/README.md https://github.com/containers/buildah/discussions/5218 As you can see from https://github.com/containers/image_build/blob/main/buildah/Containerfile it is totally non-trivial to handle the "general" case ... but you could always try and copy and paste the same into your dockerfile and see where you end up :p. While I would be very interested in this easier to get working by default and have the GoCD container images prepare themself for this to make it easier (or have a special image similar to the "dind" images), I have not found a way that is sufficiently agnostic from host/node configurations that is suitable for something like GoCD. I probably haven't looked hard enough though. Anyway, to put this another way, this sounds really like a question of "how do I run buildah within a Kubernetes pod with/without special privileges?" rather than anything GoCD-specific. If GoCD can easily provide an opinionated container agent image that supports this, and/or opinionated configuration templates for pods in Helm charts and elastic agents then I am keen for some help to find the right way, but not something I have tried properly myself. Your alternative is to try changing your base image to one based on a buildah image like quay.io/buldah/stable:latest, and then either - layer on GoCD's stuff in your custom image with https://github.com/gocd/docker-gocd-agent-almalinux-9/blob/main/Dockerfile (suggesting Alma base image, as think buildah images are fedora based) *OR* - avoid having to update the instructions to match GoCD changes every release by using a multi-stage Dockerfile to pull across all of /go /go-agent /godata /gocd-jre /docker-entrypoint.sh /docker-entrypoint.d /usr/local/sbin/tini (and the UID/GID, ENV etc). This set of dirs isn't considered a "stable API", but will probably be easier to maintain than copy and pasting the raw Dockerfile instructions. e.g something like the below (untested!!!!) FROM quay.io/buldah/stable:latest # Install gocd-agent in container-ready form ARG GO_AGENT_IMAGE=gocd/gocd-agent-almalinux-9 ARG GO_VERSION=v24.3.0 RUN useradd -l -u 1000 -g root -d /home/go -m go && \ dnf install -y git-core openssh-clients bash unzip curl-minimal procps-ng coreutils-single glibc-langpack-en tar && \ dnf clean all && \ rm -rf /var/cache/dnf && \ COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /usr/local/sbin/tini /usr/local/sbin/tini COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /gocd-jre /gocd-jre COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /go-agent /go-agent COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /go /go COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /godata /godata COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /docker-entrypoint.d /docker-entrypoint.d COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /docker-entrypoint.sh /docker-entrypoint.sh ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8 ENV GO_JAVA_HOME="/gocd-jre" ENTRYPOINT ["/docker-entrypoint.sh"] USER go # Install your stuff? You'll have to be careful with - all of the permissions though to make sure they come across with the COPY instructions etc - figure out if there is stuff in the buildah container entrypoints that needs to come across - validate that the buildah image is intended to use UID=1000 by default or already has a user configured etc. -Chad On Fri, Sep 20, 2024 at 1:56 PM Raghu Kumar <[email protected]> wrote: > Hello Chad, > > Continuing where Sloka left off. > We have been able to build the GoCD agent and it does run the SBT > commands. However, while trying to build an image using Buildah we are > seeing certain errors like this: > --- > Error during unshare(CLONE_NEWUSER): No space left on device > User namespaces are not enabled in /proc/sys/user/max_user_namespaces. > time="2024-09-16T07:06:05Z" level=error msg="error parsing PID \"\": > strconv.Atoi: parsing \"\": invalid syntax" > time="2024-09-16T07:06:05Z" level=error msg="(unable to determine exit > status)" > Error during unshare(CLONE_NEWUSER): No space left on device > User namespaces are not enabled in /proc/sys/user/max_user_namespaces. > time="2024-09-16T07:06:05Z" level=error msg="error parsing PID \"\": > strconv.Atoi: parsing \"\": invalid syntax" > time="2024-09-16T07:06:05Z" level=error msg="(unable to determine exit > status)" > --- > > This makes me believe that there are certain storage constraints on the > Elastic agent container. However, if I look at the elastic agent Pod > configuration, I don't see any volume being attached. The pod configuration > is mentioned below: > --- > apiVersion: v1 > kind: Pod > metadata: > name: gocd-agent-{{ POD_POSTFIX }} > labels: > app: web > spec: > serviceAccountName: default > containers: > - name: gocd-agent-container-{{ CONTAINER_POSTFIX }} > image: > 366611831214.dkr.ecr.us-east-1.amazonaws.com/gocd/agent:ea-sbt-jdk11-build-2-1 > volumeMounts: > - name: ssh-secrets > readOnly: true > mountPath: /home/go/.ssh > - name: dev-fuse > mountPath: /dev/fuse > env: > - name: _BUILDAH_STARTED_IN_USERNS > value: "" > - name: STORAGE_DRIVER > value: "overlay" > - name: STORAGE_OPTS > value: "overlay.mount_program=/usr/bin/fuse-overlayfs" > securityContext: > privileged: true > capabilities: > add: > - SYS_ADMIN > resources: > limits: > memory: "8192M" > cpu: "2" > requests: > memory: "8192M" > cpu: "2" > volumes: > - name: ssh-secrets > secret: > defaultMode: 420 > secretName: gocd-bitbucket-kube-secret > - name: dev-fuse > hostPath: > path: /dev/fuse > type: CharDevice > ... > > In case, I want to attach a volume to these agents how do I go about doing > it? If that's not an option then do i need to increase the memory to > accomodate larger materials for build? > > Thank you, > Raghu > On Wednesday 11 September 2024 at 13:35:56 UTC+5:30 Chad Wilson wrote: > >> You don't appear to have actually installed the GoCD agent in your image >> - it needs to be there by default. You are better to base your container >> image off one of the existing GoCD agent images from >> https://www.gocd.org/download/#docker rather than trying to >> hand-construct your own from scratch. There are Ubuntu variants available >> similar to your current base e.g >> https://hub.docker.com/r/gocd/gocd-agent-ubuntu-24.04 or >> https://hub.docker.com/r/gocd/gocd-agent-ubuntu-22.04 >> >> -Chad >> >> On Wed, Sep 11, 2024 at 3:54 PM Sloka Roy <[email protected]> wrote: >> >>> Hi, >>> >>> I am trying to create GoCD custom elastic agent which supports Buildah >>> for building docker images and pushing to ECR and SBT and JAVA 11 for >>> compilation. >>> >>> Below dockerfile I plan to use as GoCD elastic agent. >>> >>> FROM eclipse-temurin:11.0.24_8-jdk-jammy # Install required packages >>> including Buildah dependencies RUN apt-get update && \ apt-get install >>> -y \ curl \ git \ zip \ unzip \ jq \ buildah \ runc \ fuse-overlayfs \ >>> iptables >>> && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Install sbt ARG >>> SBT_VERSION=1.5.6 ENV SBT_HOME /usr/local/sbt ENV PATH >>> ${PATH}:${SBT_HOME}/bin RUN curl -sL " >>> https://github.com/sbt/sbt/releases/download/v1.5.6/sbt-1.5.6.tgz"; | >>> gunzip | tar -x -C /usr/local && \ echo -ne "- with sbt $SBT_VERSION\n" >>> >> /root/.built # Setup GoCD user and environment ENV HOME /var/go RUN >>> groupadd -g 496 go && \ useradd -c "go user" -d $HOME -m go -g 496 -u >>> 498 VOLUME /var/go WORKDIR /var/go USER go >>> However the agent is not getting registered. >>> Events: Type Reason Age From Message ---- ------ ---- ---- ------- >>> Normal Scheduled 2m15s default-scheduler Successfully assigned >>> gocd/k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 to >>> ip-10-75-110-207.ec2.internal Normal Pulled 33s (x5 over 2m15s) kubelet >>> Container image " >>> 366611831214.dkr.ecr.us-east-1.amazonaws.com/gocd/agent:es-jdk11-build" >>> already present on machine Normal Created 33s (x5 over 2m15s) kubelet >>> Created container k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 Normal >>> Started 32s (x5 over 2m15s) kubelet Started container >>> k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 Warning BackOff 3s (x10 over >>> 2m8s) kubelet Back-off restarting failed container >>> k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 in pod >>> k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017_gocd(8fe96d7b-ea06-4f80-a17b-13042f59c548) >>> >>> [image: Screenshot 2024-09-11 at 12.39.49 PM.png] >>> >>> Can you please help me here, with what are the minimum requirements to >>> create an custom elastic agent >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/cd320acc-2836-4d87-9a30-fa1d7233ce94n%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/cd320acc-2836-4d87-9a30-fa1d7233ce94n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/5c17788e-422f-4f32-b5e3-bb709c73a3a9n%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/5c17788e-422f-4f32-b5e3-bb709c73a3a9n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8mB9xGwhGiJZPmq4e0_NQCwqpRdjBC-iTGe-sa3OFJKw%40mail.gmail.com.
