Response inline. On Fri, Jul 26, 2019 at 10:58 AM Driesprong, Fokko <fo...@driesprong.frl> wrote: > > Nice document Jarek. > > We should look at the pro's and con's regarding moving away from Travis. > The process for Airflow, and also many other OSS projects, is to first > develop on your local fork. If everything looks good, open a PR to the main > repo. This reduces the noise we have on the project itself. Being more > strict on this will also reduce the load on the CI service of Apache. > We do not plan to delete Travis support. We will continue to support him. Working on forks will still look the same, but will allow you to perform tasks on apache/airflow. Travis behaves very unpredictably when it comes to resource allocation. Sometimes jobs wait in queue for 8 hours before they are performed. It is not promised that this problem will be solved in any way. Apache Infra is aware of Travis' limitations, but we are unable to find a solution to this problem together.
> A couple of thoughts so far: > - I'm not sure why we need to store the images on the GCS. We could just > discard the image after the build? In the case of caching, we could also > store them on the local VM's as well. Just a thought to simplify the setup. It will be difficult to store the Docker image locally if we want to share it between jobs. This is helpful because it allows us to create configurations of specific test sets that will run independently of the others. Reference: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems Currently, we already have a number of system tests. Unfortunately, they are only performed on Polidea Infrastructure. We received resources from Google, which allow you to perform these tests also on the master branch > - Since the current setup is flaky, it feels counterintuitive to make it > more complex, for example by mirroring the repository to Gitlab. How does > this work for PR's from forks (these repo's are still on a fork on Github)? > For example, when I open a PR from my Github fork, this fork does not live > in Gitlab. The forks will use Travis. > - I think it is important to discuss this with Infra as well, we need to > get them on board as well. I agree. > - Are there other OSS projects which use this setup as well? Not yet, but we maintain direct contact with Apache Beam commiters. They are also interested in moving to a similar infrasture. > My personal opinion, apart from the issues we're facing the last few days, > Travis works quite well for me. I am afraid that the current problems will be repeated because we have cut resources by the Apache Infra team We have only 5 workers. This means that only one pR is performed simultaneously on apache/airflow repo. > Cheers, Fokko > > Op wo 24 jul. 2019 om 10:05 schreef Jarek Potiuk <jarek.pot...@polidea.com>: > > > Of course ! One of the considerations is to keep travis CI build intact - > > so that anyone will be able to have their own Travis Fork for the time > > being. > > > > I will also do it in the way that once you have your own GCP account and > > your own GKE cluster, you will be able to replicate it as well (there will > > be instructions on how to set it up). > > We can even (long term) make it in the way that you will not need a > > separate GKE cluster but it will run using just your personal GitLab > > (free). This should be possible - I am really trying to make it > > underlying-infrastructure-agnostic. > > > > The non-cluster personal GitLab is not a priority now (Travis forks will > > hopefully work ;) so it might not work initially, but there aren't > > fundamental reasons it should not work. We will have to just use GitLabCI > > registry instead of the GCP one and avoid assuming we are running in the > > GKE cluster and have some secrets/accounts distributed differently. All > > looks doable. > > > > J. > > > > > > J. > > > > On Wed, Jul 24, 2019 at 9:03 AM Chao-Han Tsai <milton0...@gmail.com> > > wrote: > > > > > Thanks Jarek for putting this together. We really need a stable and fast > > > CI. > > > > > > Question: will we still be able to build our personal fork of Airflow on > > > our own Travis? > > > > > > Chao-Han > > > > > > On Tue, Jul 23, 2019 at 1:00 PM Jarek Potiuk <jarek.pot...@polidea.com> > > > wrote: > > > > > > > > > > > > > Question - what is the purpose of introducing kaniko instead of using > > > > > regular docker build? > > > > > > > > > > > > > Indeed. We want to be as agnostic as possible. What I plan to do is to > > > use > > > > Kubernetes Runner in GitlabCI. This means that all the jobs will run as > > > > Kubernetes PODs in GKE - Gitlab CI will only be UI + runner that > > > > orchestrates the builds. This means that our test jobs will be run > > inside > > > > docker - they will not run in virtual machine, but they will run inside > > > the > > > > container. This is how modern CI systems work (for example Gitlab, > > > > CloudBuild, also Argo <https://argoproj.github.io/> - new kid in the > > > block > > > > which is Kubernetes-Native). Argo is a bit too fresh to consider it, > > but > > > > they all work similarly - all steps are run inside docker. > > > > > > > > As the first part of our build we have to build the images with latest > > > > sources (and dependencies if needed) that then will be used for > > > subsequent > > > > steps. This means that we need to build the images from within docker - > > > > which is not as trivial as running docker command. There are three ways > > > to > > > > approach it - docker-in-docker (requires priviledged docker > > containers), > > > > using same docker engine which is used by Kubernetes Cluster (not > > > > recommended as Kubernetes manages docker engine on their own and might > > > > delete/remove images at any time) or use Kaniko. Kaniko was created > > > exactly > > > > for this purpose - to be able to run docker build from within a POD > > that > > > > runs in Kubernetes cluster. > > > > > > > > I hope it explains :). Kaniko is pretty much standard way of doing it > > and > > > > it really Kubernetes-native way of doing it. > > > > > > > > > > > > > > > > > > Regards > > > > > Shah > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 23, 2019 at 5:12 PM Jarek Potiuk < > > jarek.pot...@polidea.com > > > > > > > > > wrote: > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > I prepared a short docs where I described general architecture of > > the > > > > > > solution I imagine we can deploy fairly quickly - having GitLab CI > > > > > support > > > > > > and Google provided funding for GCP resources. > > > > > > > > > > > > I am going to start working on Proof-Of-Concept soon but before I > > > start > > > > > > doing it, I would like to get some comments and opinions on the > > > > proposed > > > > > > approach. I discussed the basic approach with my friend Kamil who > > > works > > > > > at > > > > > > GitLab and he is a CI maintainer and this is what we think will be > > > > > > achievable in fairly short time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > > > > > > > > > > > I am happy to discuss details and make changes to the proposal - we > > > can > > > > > > discuss it here or as comments in the document. > > > > > > > > > > > > Let's see what people think about it and if we get to some > > consensus > > > we > > > > > > might want to cast a vote (or maybe go via lasy consensus as this > > is > > > > > > something we should have rather quickly) > > > > > > > > > > > > Looking forward to your comments! > > > > > > > > > > > > J. > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > -- > > > > > > Chao-Han Tsai > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > >
