(Urgh... gmail sent my email too early.. continuation) ----------------------------------------------------
That's a great question James. So for the past ~8 months we've been using a Jenkins master + (n) slaves setup to build images. We currently build around ~20 different images for production, a few of which are also setup separately for CI (building a GitHub pull request, tagging with a branch name, running unit/integration tests and pushing the image on success). To be honest, this actually does work pretty well. Docker is a stable bit of kit, and most of the issues we ran into were environmental... though still a problem. I think this approach is valid, and will work for most users - and even better if you couple if with the Jenkins on Mesos framework. *I'd like to highlight a few of the problems we found with running that infrastructure;* - Even though we have a fair few number of images, people don't build images 24/7. However, we have some big machines up 24/7 to serve the image builds such that they complete in a timely manor when required... people don't like to wait for things, and something like this can block a developer unnecessarily. - Managing these machines wasn't too troublesome (bit of configuration management tools here and there) however, given the intermittent nature of the workload it's a huge waste of money to keep machines always on. We could scale up automatically, but that introduces more complexity and latency for the developer. - Docker doesn't do a very good job at cleaning up after itself. Though there are good options for removing intermediate containers, images pulled from remote repositories hang around and it's very hard to know what you can and can't remove from disk, especially while containers are running or building. We had disks fill up on numerous occasions due to the docker images filling up. A little Jenkins job on a cron to delete things can work, but it's not perfect and a little messy. - We build expensive images. Many of our images are very large (~1.5-2GB in total) and compile or install lots of things. This is quite CPU and memory intensive, and to be able to build them quickly we have to pay for big machines (always on as mentioned above). The 24 hour utilisation of these resources is *incredibly* low, and very spiky. *Some things that led me to start building Portainer;* - We've just rolled out a fairly chunky Mesos cluster into production mostly for batch job execution. These resources though heavily utilised at times, are also quite spiky leaving room for other activities to take place. Why not make use of these spare (large) resources to build images... Most image builds take a few minutes on a fat box, so aren't going to cause much of a problem. - I didn't want to go down the road of building a million different scripts and tools to manage, clean up and automate building docker images on top of something like Jenkins. We've been doing some of that for a little while, and the scripting for each job can become a pain and is pretty complicated for someone that just wants to build a docker image. - The docker build workflow lends itself well to remote execution. You run a docker daemon somewhere (usually locally), build a tar archive of the *Dockerfile* and any other resources, send a HTTP request and it'll build it for you. The process is also entirely ephemeral, so launching a docker daemon somewhere, building an image, tearing it down and removing everything it created would work perfectly fine. This ephemeral approach goes a long way to solving lots of the disk space leakage problems we were having before (mesos will clean up for us!). - This isn't a complicated framework, scheduling is easy (at least to start with)... first-fit will do just fine for us. The build workflow is also a great fit for the mesos executor/task/offer model. Building one image is almost identical to building 100 in parallel, assuming you don't build conflicting images at the same time ;-) - As highlighted in another thread on the mailing list, building frameworks can be very hard. Advanced scheduling is hard, fault tolerance is hard, HA frameworks are hard. None of these things really matter when it's just one invocation wanting to build a few containers. If they fail, just try it again, and the chances are it failed and you need to do something to fix it. - We want to build lots more containers, more often. Can we really scale the Jenkins setup? Possibly, possibly not. ---------------------------------------------------- Portainer started out as a side project to try and explore if this would be possible. Maybe we could save some costs and automate things a little more. It ended up working out very well, and the existing docker/mesos libraries made a lot of that very easy. Framework messaging and the docker containerizer (for running docker inside docker) are also very useful here. I hope that brings some clarity to why I built Portainer. As I said it's still very early on (though we are already using it for some builds)... so i'm very keen to hear feedback and see contributions from those using it! Thanks, Tom. On 4 September 2014 11:30, Tom Arnfeld <t...@duedil.com> wrote: > That's a great question James. So for the past ~8 months we've been using > a jenkins master + (n) slaves setup to build images. We currently build > around ~20 different images for production, a few of which are also setup > to separately for CI (building a GitHub pull request, tagging with a branch > name, running unit/integration tests and pushing the image on success). To > be honest, this actually does work pretty well. Docker is a stable bit of > kit, and most of the issues we ran into were environmental... though still > a problem. I think this approach is valid, and will work for most users - > and even better if you couple if with the Jenkins on Mesos framework. > > I'd like to highlight a few of the problems we found with running that > infrastructure; > > - Even though we have a fair few number of images, people don't build > images 24/7. However, we have some big machines up 24/7 to serve the image > builds such that they complete in a timely manor when required... people > don't like to wait for things, and something like this can block a > developer unnecessarily. > > > > > On 4 September 2014 11:06, James Gray <zaa...@gmail.com> wrote: > >> This is def interesting for us as this is a problem we are facing too. >> >> But just to clarify, what is the advantage of this approach over >> submitting Docker build jobs to either the Chronos or Jenkins >> schedulers? For our use case, we were thinking about writing a >> scheduler for GoCD (which we use for CI/CD) to work much in the same >> way as the existing Jenkins scheduler. >> >> On Thu, Sep 4, 2014 at 12:29 AM, Joe Smith <yasumo...@gmail.com> wrote: >> > +1 to a registry, stuff like NPM, vagrantcloud, and docker make it very >> > clean to search (but seems like it takes more work to support and >> setup). >> > >> > Tom- this is rad! Also loving the use of Pesos- definitely looking >> forward >> > to to more contributors there :) >> > >> > >> > On Wed, Sep 3, 2014 at 4:13 PM, Chris Aniszczyk <z...@twitter.com> wrote: >> >> >> >> Just an idea but I think we should strive to provide a better approach >> >> that is more scalable/searchable IMHO as the number of frameworks >> continue >> >> to grow. I created an issue here to discuss potential options and if >> people >> >> are interested in providing some type of framework registry: >> >> https://issues.apache.org/jira/browse/MESOS-1759 >> >> >> >> I see a couple options that could be interested, either the more lax >> >> community driven approach that JenkinsCI does via a GitHub >> organization or >> >> building a web-based registry similar to what the docker/ansible folks >> have >> >> done. >> >> >> >> >> >> On Wed, Sep 3, 2014 at 6:03 PM, Vinod Kone <vinodk...@gmail.com> >> wrote: >> >>> >> >>> This is great Tom. Thanks for sharing. We do list Mesos frameworks on >> the >> >>> website ( >> http://mesos.apache.org/documentation/latest/mesos-frameworks/). >> >>> Please send a PR or RB request. >> >>> >> >>> >> >>> On Wed, Sep 3, 2014 at 3:50 PM, Tom Arnfeld <t...@duedil.com> wrote: >> >>>> >> >>>> @Ankur Wups! That's silly of me... >> >>>> http://github.com/duedil-ltd/portainer >> >>>> >> >>>> >> >>>> On 3 September 2014 23:45, Ankur Chauhan <an...@malloc64.com> wrote: >> >>>>> >> >>>>> Could you share a link to the repo? >> >>>>> >> >>>>> >> >>>>> On Wed, Sep 3, 2014 at 3:20 PM, Tom Arnfeld <t...@duedil.com> wrote: >> >>>>>> >> >>>>>> Hey everyone, >> >>>>>> >> >>>>>> Thought it would be worth sharing this on the mailing list. We've >> >>>>>> recently open sourced a Mesos framework called Portainer, which is >> for >> >>>>>> building docker containers on top of your cluster. >> >>>>>> >> >>>>>> It is in working order, though very early stage... It supports all >> >>>>>> Dockerfile instructions (including ADD) and can build multiple >> images in >> >>>>>> parallel. It's written entirely in Python, and is also built upon >> the Pesos >> >>>>>> python framework API @wickman, @nekto0n and I have been working >> on, so >> >>>>>> there's no need to install libmesos to use the framework. >> >>>>>> >> >>>>>> I ended up trying out the idea because we've had a painful >> experience >> >>>>>> managing dedicated infrastructure for building all of our images, >> which I'm >> >>>>>> sure some of you can empathise with, and figured we could leverage >> the spare >> >>>>>> capacity on our new Mesos cluster to cut that out entirely. >> >>>>>> >> >>>>>> We'd love any feedback or suggestions, as well as any >> contributions! >> >>>>>> Looking forward to hearing what you all think. :-) >> >>>>>> >> >>>>>> Side note... It would be great if there were a place we could list >> all >> >>>>>> of the known frameworks for users to explore, maybe this already >> exists? >> >>>>>> >> >>>>>> Cheers, >> >>>>>> >> >>>>>> Tom (and the rest of the infra team at DueDil). >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> >> >> >> -- >> >> Cheers, >> >> >> >> Chris Aniszczyk | Open Source | Twitter, Inc. >> >> @cra | +1 512 961 6719 >> > >> > >> > >
