[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Liam Newman closed an issue as Won't Fix Bulk closing resolved issues. Jenkins / JENKINS-40723 Built Dockerfile images are never removed Change By: Liam Newman Status: Resolved Closed Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.177427.1483031765000.16553.1571801053436%40Atlassian.JIRA.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Michael Harris edited a comment on JENKINS-40723 Re: Built Dockerfile images are never removed Having some mechanism to delete the generated image is really important. If we have a lot of builds in a short time our nodes still run out of disk space before the scheduled job runs and wipes out old images. Jenkins is creating the images, and at a minimum needs to expose a way to clean up the image. Even if t it was only added in the [scripting syntax|https://jenkins.io/doc/book/pipeline/docker/#building-containers] that would be sufficient. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Michael Harris commented on JENKINS-40723 Re: Built Dockerfile images are never removed Having some mechanism to delete the generated image is really important. If we have a lot of builds in a short time our nodes still run out of disk space before the scheduled job runs and wipes out old images. Jenkins is creating the images, and at a minimum needs to expose a way to clean up the image. Even if t was only added in the scripting syntax that would be sufficient. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Dwight Guth commented on JENKINS-40723 Re: Built Dockerfile images are never removed I definitely side with the people who think that this was resolved prematurely. We have been running docker image prune on our Jenkins node that we recently migrated to dockerfiles, and each time it ran it was freeing up less and less space. What we eventually realized is that because Jenkins is tagging every build with a tag corresponding to the hash of the dockerfile, when the dockerfile changes, it does not untag the old build (because the new tag is different from the old tag), so the old build is not marked as dangling, and you have to run the significantly more aggressive `docker image prune -a` to clean it up. I believe Jenkins should be tagging these builds with something that is deterministic across builds to the same job and branch, and deleting all the tags associated with change requests when the change request closes and the jenkins branch is deleted. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Allan Lewis commented on JENKINS-40723
Re: Built Dockerfile images are never removed
Hrm. So for a GitHub Organisation PR, the $JOB_NAME will be github-org/project/PR-XX.
I'm not using a GitHub org, so I'm not sure about that. My case is using a manually-configured Git URL.
First of all, that's an invalid tag name. But let's assume we can make it sane by removing the {{/}}s.
I'm not using the job name as a tag, I'm using it as the image name, and image names can contain slashes.
Granted when (the locally written script for) garbage cleanup removes that PR-XX image because PR-XX is closed, the master branch build will build a new (identical) docker image to replace the just removed one. But that's an extra build that shouldn't be necessary.
Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?
No, because when master is built, it will build the image again - cached if it's on the same node or if one implements push-pull with a registry - and tag it as /master. If we then prune the tag from the branch, we'll still have the image as it will be tagged for master. I'm not saying my solution will work for everyone, I just posted it in case it was useful for others.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Attila Szeremi edited a comment on JENKINS-40723
Re: Built Dockerfile images are never removed
{quote}[~brianjmurrell]Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?{quote}First of all, Jenkins doesn't even do that. If a build would be rerun, it would just build the Docker image again. Secondly, it's not like Jenkins does any kind of caching for non-Docker agents (like caching the node_modules/ generated from an npm install). And besides, Docker already has decent caching built into it; any step in the Dockerfile where the involved files did not change since last time, Docker automatically just re-uses the cached layer; so no additional help from Jenkins is needed.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Attila Szeremi commented on JENKINS-40723 Re: Built Dockerfile images are never removed Brian J Murrell Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed? First of all, Jenkins doesn't even do that. If a build would be rerun, it would just build the Docker image again. Secondly, it's not like Jenkins does any kind of caching for non-Docker agents (like caching the node_modules/ generated from an npm install). And besides, Docker already has decent caching built into it; any step in the Dockerfile where the involved files did not change since last time, Docker automatically just re-uses the cached layer; so no additional help from Jenkins is needed. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Brian J Murrell commented on JENKINS-40723
Re: Built Dockerfile images are never removed
Hrm. So for a GitHub Organisation PR, the $JOB_NAME will be github-org/project/PR-XX. First of all, that's an invalid tag name. But let's assume we can make it sane by removing the {{/}}s. But more importantly, when that PR lands to master, the tag on the docker image will still be for PR-XX even though it will actually now be the new current image being used for master (I think). But it's not tagged for master on that project, it's tagged for PR-XX on that project. Granted when (the locally written script for) garbage cleanup removes that PR-XX image because PR-XX is closed, the master branch build will build a new (identical) docker image to replace the just removed one. But that's an extra build that shouldn't be necessary. That said, it's probably still better than simply using the "oldest built" garbage collection mechanism that is being suggested in this ticket. Also, Allan Lewis, you didn't respond to my other concern:
Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Attila Szeremi edited a comment on JENKINS-40723 Re: Built Dockerfile images are never removed I disagree with brushing this ticket off to use `docker-gc` instead.First of all, it makes no sense to use `docker-gc` in the `post` section of the declarative pipeline, because `docker-gc` only removes images that are older than (by default) 1 hour old. The image we'd want to clean is very fresh though.Secondly, `docker-gc` (to me) undesirably removes all other images as well; ones I might still be using. If I have a cron job set to run a Docker image every 2 hours, and `docker-gc` would keep removing the image, then I would need to have the re-download the image from the registry each time, wasting time and bandwidth.It would be great if either the generated image tag would automatically be removed on build finish, or if the image tag were predictable so that I could clean up manually in the end. Or simply just making the image tag available so I could read it.Currently +[here| https://github.com/amcsi/youtube-delete-tracker/blob/master/Jenkinsfile.groovy #L48] ] + is how I clean up my built Docker image. I am forced to use [ + [ imperative scripts + | https://github.com/amcsi/youtube-delete-tracker/blob/master/Jenkinsfile.groovy #L11-L12] ] + to build the Docker image to get the image tag so that I could clean up in the end. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Attila Szeremi edited a comment on JENKINS-40723 Re: Built Dockerfile images are never removed I disagree with brushing this ticket off to use `docker-gc` instead.First of all, it makes no sense to use `docker-gc` in the `post` section of the declarative pipeline, because `docker-gc` only removes images that are older than (by default) 1 hour old. The image we'd want to clean is very fresh though.Secondly, `docker-gc` (to me) undesirably removes all other images as well; ones I might still be using. If I have a cron job set to run a Docker image every 2 hours, and `docker-gc` would keep removing the image, then I would need to have the re-download the image from the registry each time, wasting time and bandwidth.It would be great if either the generated image tag would automatically be removed on build finish, or if the image tag were predictable so that I could clean up manually in the end. Or simply just making the image tag available so I could read it.Currently +[here|https://github.com/amcsi/youtube-delete-tracker/blob/master/Jenkinsfile.groovy#L48]+ is how I clean up my built Docker image. I am forced to use +[imperative scripts|https://github.com/amcsi/youtube-delete-tracker/blob/master/Jenkinsfile.groovy#L11-L12]+ to build the Docker image to get the image tag so that I could clean up in the end. EDIT: I have fixed the links in the above paragraph. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Allan Lewis commented on JENKINS-40723
Re: Built Dockerfile images are never removed
Hi Brian J Murrell - for multi-branch pipelines we're currently using JOB_NAME (plus implied :latest). That has the effect of retaining the image from the tip of every branch since JOB_NAME is / in the multi-branch case. We don't yet have a strategy for deleting images from branches that no longer exist on the remote, but that shouldn't be difficult to script. For some non-multibranch pipelines we have, that can be triggered with an arbitrary Git ref via a parameter, we're using "${JOB_BASE_NAME}/${params.GIT_REF}".toLowerCase(). (Docker image names have to be lower-case.)
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Brian J Murrell edited a comment on JENKINS-40723 Re: Built Dockerfile images are never removed [~allanlewis_youview] What's an example of _some-tag-i-understand_? Is it the same for every run of a given job, or is there some kind of serialiser in there so that you know which is the latest, etc.? Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Brian J Murrell commented on JENKINS-40723 Re: Built Dockerfile images are never removed Allan Lewis What's an example of some-tag-i-understand? Is it the same for every run of a given job, or is there some kind of serialiser in there so that you know which is the latest, etc.? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Allan Lewis commented on JENKINS-40723 Re: Built Dockerfile images are never removed My call here is that Docker image management is out of scope for Declarative to deal with. I agree, but Jenkins shouldn't make it unnecessarily difficult. If Docker images were tagged with some combination of the job name and other metadata instead of just a hash, that would help. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Attila Szeremi edited a comment on JENKINS-40723 Re: Built Dockerfile images are never removed I disagree with brushing this ticket off to use `docker-gc` instead.First of all, it makes no sense to use `docker-gc` in the `post` section of the declarative pipeline, because `docker-gc` only removes images that are older than (by default) 1 hour old. The image we'd want to clean is very fresh though.Secondly, `docker-gc` (to me) undesirably removes all other images as well; ones I might still be using. If I have a cron job set to run a Docker image every 2 hours, and `docker-gc` would keep removing the image, then I would need to have the re-download the image from the registry each time, wasting time and bandwidth.It would be great if either the generated image tag would automatically be removed on build finish, or if the image tag were predictable so that I could clean up manually in the end. Or simply just making the image tag available so I could read it.Currently + [here| [https://github.com/amcsi/youtube-delete-tracker/blob/6848efaa2186325d76b1c676b756363746fdd227/Jenkinsfile.groovy #L48]] + is how I clean up my built Docker image. I am forced to use [ + imperative scripts + | [https://github.com/amcsi/youtube-delete-tracker/blob/6848efaa2186325d76b1c676b756363746fdd227/Jenkinsfile.groovy #L11-L12]]to build the Docker image to get the image tag so that I could clean up in the end. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Allan Lewis commented on JENKINS-40723 Re: Built Dockerfile images are never removed My current workaround for this is to add -tag some-tag-i-understand to additionalBuildArgs. I then have a cleanup script run by cron on each node that deletes all of Jenkins's random-tagged images. If any of these have another tag due to -tag, this will just drop the random tag; otherwise it will delete the image. I think deleting by date is an antipattern anyway since images built a long time ago aren't necessarily unused and images built recently aren't necessarily worth keeping. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Attila Szeremi commented on JENKINS-40723 Re: Built Dockerfile images are never removed I disagree with brushing this ticket off to use `docker-gc` instead. First of all, it makes no sense to use `docker-gc` in the `post` section of the declarative pipeline, because `docker-gc` only removes images that are older than (by default) 1 hour old. The image we'd want to clean is very fresh though. Secondly, `docker-gc` (to me) undesirably removes all other images as well; ones I might still be using. If I have a cron job set to run a Docker image every 2 hours, and `docker-gc` would keep removing the image, then I would need to have the re-download the image from the registry each time, wasting time and bandwidth. It would be great if either the generated image tag would automatically be removed on build finish, or if the image tag were predictable so that I could clean up manually in the end. Or simply just making the image tag available so I could read it. Currently here is how I clean up my built Docker image. I am forced to use imperative scriptsto build the Docker image to get the image tag so that I could clean up in the end. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Brian J Murrell commented on JENKINS-40723
Re: Built Dockerfile images are never removed
I would like to disagree at a higher level. The obfuscatingly named (a hash of some sort) images are being created by Jenkins and because they are using such an obfuscating name as a hash makes trying to manage the accumulation of them outside of Jenkins difficult at best. For example, to go along with the idea that it is up to me, the admin (or a script I write) to remove old, no-longer used images, which represent past iterations of {{Dockerfile}}s in my pipeline jobs, which of the following images are current and which are older iterations?
# docker image ls
REPOSITORY TAG IMAGE IDCREATED SIZE
ff479182ebeb4 days ago 1.42 GB
a831c15b87dbd4231d6947a16cb3a9af816cbd8d latest ac3742854a964 days ago 1.42 GB
e8cd3c8b78088b6c52ee705ae10be4201359a5b3 latest ac3742854a964 days ago 1.42 GB
4249d3c8084d55fc3d2a99132a0f9bce66f6e5ec latest ac3742854a964 days ago 1.42 GB
8447a9d1402c6562bdfc27950aada59b7b1e0593 latest ce155bde643b4 days ago 1.62 GB
a1a0b41dfa855a77be010b372402702114bc1225 latest ce155bde643b4 days ago 1.62 GB
a677138400e3274c0f3ccacb0d6919a892df06fb latest ce155bde643b4 days ago 1.62 GB
280f646bcd85076af3f7cb58afc3c221a426ad4c latest 60e033d802534 days ago 1.76 GB
a2ec752628e0decf2e09a39fabfd1e08afdbe67f latest 60e033d802534 days ago 1.76 GB
cbc1131e470e384e815c278b456f8ce695657a89 latest 60e033d802534 days ago 1.76 GB
24e62ec336d1449f98fb7474a8496af0707c0588 latest caf0d95641617 days ago 1.42 GB
c761e9f12da336e348736f8c32e83e53d0c41ae4 latest caf0d95641617 days ago 1.42 GB
0b0baa88ac6d8ca0e4d1ec58b8d8ee475ba0978b latest 1b923c4a3d197 days ago 1.45 GB
79f5fc427029a540383e43637806c27ff56af029 latest 9515ddc1f2737 days ago 1.6 GB
af1113a0c00c7 days ago 1.6 GB
1622a837d8b63 weeks ago 1.62 GB
46807bbb74e92fbdbe56fced9c109b8ed0367036 latest 353b4f1b3 weeks ago 1.62 GB
d7c383939c4a517002774159d046d7f310f881c8 latest 353b4f1b3 weeks ago 1.62 GB
4942b00624bbc67cb2ac9c1832200ce421962c2d latest 3c768e8f53053 weeks ago 1.58 GB
8b18a2c252deea8b6901d61877a13bd7567ba2f5 latest 40039dfa64693 weeks ago 1.6 GB
7b402774ab1d76f7d3a240f53780cee70a2da39c latest 78cd8f4f275e4 weeks ago 1.55 GB
d2c8168b9d4e27bd0426deb78972664c19e97e8b latest 0ae5d7965ca64 weeks ago 1.61 GB
56043f81efbf7289d621c112b82933220763fbb4 latest 340fddd2839f4 weeks ago 1.45 GB
115282b91e727adbf38cdeb8cdf6c1ef98be6593 latest 0b600893851a4 weeks ago 1.41 GB
3163e91161e120ac113c2f191742689178ae01ee latest e1d8225a6c034 weeks ago
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title James Dumay commented on JENKINS-40723 Re: Built Dockerfile images are never removed I think Docker has this built in now https://docs.docker.com/engine/reference/commandline/system_prune/ https://docs.docker.com/engine/reference/commandline/container_prune/ Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Jesse Glick updated an issue
Jenkins / JENKINS-40723
Built Dockerfile images are never removed
I think PMD should do the equivalent of
node {
def img = docker.build('whatever')
try {
…
} finally {
sh "docker rmi ${img.id}"
}
}
Change By:
Jesse Glick
Component/s:
docker-workflow-plugin
Labels:
docker
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title
Patrick Wolf commented on JENKINS-40723
Re: Built Dockerfile images are never removed
Based on a potential change for JENKINS-40866 it would be possible to run docker-gc in the post section of a Pipeline to clean up old images. Assuming docker-gc is installed on the agent.
pipeline {
agent {
label 'mylabel'
}
stages {
stage ("Build") {
agent {
dockerfile {
filename 'Dockerfile'
reuseNode true
}
}
steps {
echo "Foo"
}
}
}
post {
always {
sh 'docker-gc'
}
}
}
The top-level agent allocates a node and then individual stages can run inside the container reusing the same workspace. When the Pipeline completes you always run docker-gc and clear out old images. (by default any image not used in the last hour is cleared.)
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Patrick Wolf updated an issue Jenkins / JENKINS-40723 Built Dockerfile images are never removed Change By: Patrick Wolf Component/s: docker-workflow-plugin Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Josh Sleeper commented on JENKINS-40723 Re: Built Dockerfile images are never removed James Dumay I can totally see what you're saying, but to some extent I kinda think yes. To me, part of the beauty of using Dockerfile(s) in the Declarative Pipeline syntax is that it really does allow me to just stop caring about my nodes. I don't care about their platform, I don't care about what they have installed (beyond Docker, of course), and I don't care about managing a complete pre-built Docker image somewhere. Not caring too much about the images I generate that way seems like it fits right in to that mentality. Here's the perspective I think many people might end up seeing this from. As someone who is a Jenkins user but not a Jenkins admin, working with a pool of generic nodes with Docker installed, I don't want to be that person who used up all of a slave/node's disk space because: I didn't have permission to access the nodes directly and clean up my old images I didn't know how to clean up my old images I didn't have time or care enough to clean up my old images One solution, just like you suggested, is to run something like docker-gc on each and every node in the pool with a regular cadence. Frankly, to manage a whole pool of Docker nodes like I'm thinking, that may very well have to be something we do anyway and that would just become part of the requirements to be a Docker node. I'm just not sure if everyone else thinks that running something like docker-gc totally separate from the Jenkins job creating the images is a suitable solution. Does that make sense, or am I missing something still? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title James Dumay commented on JENKINS-40723 Re: Built Dockerfile images are never removed Playing devils advocate here but should we really be in the business of docker image lifecycle management? There are tools like docker-gc that can take care of all of that for you. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Gavin Llewellyn commented on JENKINS-40723 Re: Built Dockerfile images are never removed At the moment, the Dockerfile builds are fairly quick after the first build, as Docker's cache is used to avoid rebuilding any Dockerfile steps that haven't changed. This caching would be defeated if the built image was deleted after each build (unless the Docker node has similar images that are not managed by Jenkins). Determining whether an image should be rebuilt is not necessarily as simple as checking whether the Dockerfile has changed, or that a previous image built for that same Dockerfile still exists. I think this logic should be left to Docker, and not replicated in Jenkins. My suggestion would be the following: When building a Dockerfile, tag the resulting image with the job name/Id instead of the Dockerfile hash. Before building a Dockerfile, check for an existing image using the job's tag. If an image exists, note the image ID. After building a Dockerfile, check the image ID of the current image with the job's tag. If the image ID has changed (i.e. the tag has moved), delete the old image. If a job's name is changed, or a job is deleted, there may be some images left orphaned in the Docker node. However, it will be much easier for an admin to clean these up in the future, as the tags would make it obvious what the images were used for. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Josh Sleeper commented on JENKINS-40723 Re: Built Dockerfile images are never removed Preemptive apology for the novel of a comment, but I wanted to be detailed with my thoughts. Patrick Wolf is absolutely right, building the image every time does mostly defeat the purpose, but being able to store our Docker definition alongside our pipeline definition is exceedingly convenient for both our dev and QA in my experience so far. This may or may not be hard to do from your end (I'm not too familiar with the code managing the Dockerfile interactions), but here's the way I imagine the Dockerfile flow in declarative pipeline could work: Each declarative pipeline job run using a Dockerfile would retain 1 or more pairs of fingerprints, where each pair would contain a Dockerfile fingerprint and the fingerprint of the Docker image built from said Dockerfile. Thus, for each declarative pipeline job run that utilizes Dockerfiles there are two possible paths to follow: The Dockerfile fingerprint does match the fingerprint from the previous job run, meaning that ideally we shouldn't rebuild unless we have to. To determine that, we check the current node for the image fingerprint from the previous job run: If the current node does have an image that matches the image fingerprint from the previous job run, just run that image and continue with the job. If the current node doesn't have an image that matches the image fingerprint from the previous job run, then we logically need to build it on the current node even though the Dockerfile itself hasn't changed. The Dockerfile fingerprint doesn't match the fingerprint from the previous job run, meaning we should rebuild and clean up previously created Docker images if present. Just like the first path we check the current node for the image fingerprint from the previous job run, but this time we focus on cleanup: If the current node does have an image that matches the image fingerprint from the previous job run, remove that image from the current node and then build and run like normal. If the current node doesn't have an image that matches the image fingerprint from the previous job run, then we've at least done our due diligence to clean up and just build and run like normal. Keeping Dockerfile and Docker image fingerprints associated as a pair ensures that you can selectively remove or rebuild per Dockerfile used, and removing images only relative to fingerprints from the last job run handles what I'm guessing is the common case for Dockerfile image management. I think this gives us the best overall user-friendliness for Dockerfiles in the declarative pipeline syntax, following the mentality that users generally shouldn't have to think
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Patrick Wolf commented on JENKINS-40723 Re: Built Dockerfile images are never removed building the image for every build kind of defeats the purpose doesn't it? Then it would be wiser to just build the image once and publish it to docker hub to use with a regular docker statement. I think this might fall under general workspace management more than something to be solved in Declarative. We could add some sugar to clean workspace in post but beyond that what can we do? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Andrew Bayer commented on JENKINS-40723 Re: Built Dockerfile images are never removed Yes, but in Declarative, we possibly generate a lot more of 'em without the user ever having to really think about it. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Patrick Wolf commented on JENKINS-40723 Re: Built Dockerfile images are never removed This isn't specific to pipeline-model-definition though is it? This would be true of any build using a dockerfile, no? Andrew Bayer Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Andrew Bayer commented on JENKINS-40723 Re: Built Dockerfile images are never removed Interesting question here - how do we handle this? Do we want to delete the image after every build? Michael Neale, James Dumay, Patrick Wolf - thoughts? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-40723) Built Dockerfile images are never removed
Title: Message Title Gavin Llewellyn created an issue Jenkins / JENKINS-40723 Built Dockerfile images are never removed Issue Type: Bug Assignee: Andrew Bayer Components: pipeline-model-definition-plugin Created: 2016/Dec/29 5:16 PM Environment: Jenkins 2.19.3, pipeline-model-definition-plugin 0.6 Priority: Minor Reporter: Gavin Llewellyn When using a Dockerfile to set up the build environment (as added in JENKINS-39216), a docker image is built and tagged with the SHA1 hash of the Dockerfile. Nothing seems to remove this image once the build competes, so disk space on the docker node will eventually be exhausted. Add Comment
