Thanks Jarek for putting this together. We really need a stable and fast CI.
Question: will we still be able to build our personal fork of Airflow on our own Travis? Chao-Han On Tue, Jul 23, 2019 at 1:00 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > > > Question - what is the purpose of introducing kaniko instead of using > > regular docker build? > > > > Indeed. We want to be as agnostic as possible. What I plan to do is to use > Kubernetes Runner in GitlabCI. This means that all the jobs will run as > Kubernetes PODs in GKE - Gitlab CI will only be UI + runner that > orchestrates the builds. This means that our test jobs will be run inside > docker - they will not run in virtual machine, but they will run inside the > container. This is how modern CI systems work (for example Gitlab, > CloudBuild, also Argo <https://argoproj.github.io/> - new kid in the block > which is Kubernetes-Native). Argo is a bit too fresh to consider it, but > they all work similarly - all steps are run inside docker. > > As the first part of our build we have to build the images with latest > sources (and dependencies if needed) that then will be used for subsequent > steps. This means that we need to build the images from within docker - > which is not as trivial as running docker command. There are three ways to > approach it - docker-in-docker (requires priviledged docker containers), > using same docker engine which is used by Kubernetes Cluster (not > recommended as Kubernetes manages docker engine on their own and might > delete/remove images at any time) or use Kaniko. Kaniko was created exactly > for this purpose - to be able to run docker build from within a POD that > runs in Kubernetes cluster. > > I hope it explains :). Kaniko is pretty much standard way of doing it and > it really Kubernetes-native way of doing it. > > > > > > Regards > > Shah > > > > > > > > > > On Tue, Jul 23, 2019 at 5:12 PM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > Hello Everyone, > > > > > > I prepared a short docs where I described general architecture of the > > > solution I imagine we can deploy fairly quickly - having GitLab CI > > support > > > and Google provided funding for GCP resources. > > > > > > I am going to start working on Proof-Of-Concept soon but before I start > > > doing it, I would like to get some comments and opinions on the > proposed > > > approach. I discussed the basic approach with my friend Kamil who works > > at > > > GitLab and he is a CI maintainer and this is what we think will be > > > achievable in fairly short time. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > > > > > I am happy to discuss details and make changes to the proposal - we can > > > discuss it here or as comments in the document. > > > > > > Let's see what people think about it and if we get to some consensus we > > > might want to cast a vote (or maybe go via lasy consensus as this is > > > something we should have rather quickly) > > > > > > Looking forward to your comments! > > > > > > J. > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Chao-Han Tsai
