I am working on some last fixes to Kind improvement PR and get it ready to merge today - I hope :). Also looking forward to moving out of Travis finally.
One more thought - we think the best way to approach it is to enable Github Actions in parallel to Travis and keep it running in parallel to make sure it is stable. Then we can disable Travis but we can keep it available to switch it on again as needed. I think the main semi-open point we have with GA is the availability of credits for GCP. The free minutes from Github will not be enough for sure, Tomek already runs it on the GCP account we got from Google, but we need to secure the credits for the future as well. I will take on that task with Aizhamal but we need to keep it running for some time to understand what the usage might be. We can also talk to Github and other providers as well and maybe get some credit donations from other people/companies (maybe we could join 'Github Sponsors" project and start getting some money through that https://github.com/sponsors)? I think having several sources of funding for our compute resources might be a good idea. WDYT? J On Sat, Jan 11, 2020 at 11:41 AM Kaxil Naik <[email protected]> wrote: > Yeah, that would be nice. > > On Sat, Jan 11, 2020, 10:35 Tomasz Urbaszek <[email protected]> wrote: > > > An additional advantage of GA + self-hosted runners is the ability to > > ssh to machine and see what is going on (for example why tests timeout). > > > > T. > > > > On Sat, Jan 11, 2020 at 4:23 AM Kaxil Naik <[email protected]> wrote: > > > > > Awesome Tomek, great job. Waiting for your PR eagerly :) and can't wait > > to > > > move out of Travis. > > > > > > On Fri, Jan 10, 2020 at 2:54 PM Tomasz Urbaszek < > > > [email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > Just a small update: > > > > - after Jarek's changes from > > https://github.com/apache/airflow/pull/6516 > > > > kubernetes > > > > builds started to run (proviously there was an issue with kind) > > > > - I am testing self-hosted runners > > > > > > > > It seems that possible migration is getting closer ;) > > > > > > > > Bests, > > > > Tomek > > > > > > > > On Tue, Dec 17, 2019 at 3:28 PM Philippe Gagnon < > [email protected] > > > > > > > wrote: > > > > > > > > > We have been using Actions on the prestosql project for a little > > while > > > > as a > > > > > Travis replacement and we like it a lot. > > > > > > > > > > On Tue, Dec 17, 2019 at 9:08 AM Jarek Potiuk < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > :( > > > > > > > > > > > > On Tue, Dec 17, 2019 at 2:35 PM Tomasz Urbaszek < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > I started to play with self-hosted runner and... well, > > encountered > > > > > known > > > > > > > error: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.community/t5/GitHub-Actions/Github-action-stuck-at-queue/td-p/38003 > > > > > > > > > > > > > > It seems that GA is still maturing. > > > > > > > > > > > > > > Bests, > > > > > > > Tomek > > > > > > > > > > > > > > On Fri, Dec 13, 2019 at 5:06 PM Jarek Potiuk < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Yeah. All at once seems more than reasonable. > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > On Fri, Dec 13, 2019 at 4:50 PM Kaxil Naik < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > Agree with Daniel, let's do it all at once. > > > > > > > > > > > > > > > > > > On Fri, Dec 13, 2019 at 3:49 PM Daniel Imberman < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I’d rather we do it all at once and not need to maintain > > two > > > > > > builds. > > > > > > > > > > Most/all of our CI/CD is dockerized at this point so > there > > > > > > shouldn’t > > > > > > > > be a > > > > > > > > > > huge difference. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 13, 2019 at 1:23 AM, Tomasz Urbaszek < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I have to solve a problem with our kubernetes test but > for > > > your > > > > > > > > > information > > > > > > > > > > I have never experienced a flaky > > > > > > > > > > or timeouted build on Github Actions. Maybe I am lucky or > > > maybe > > > > > > > there's > > > > > > > > > > something different. > > > > > > > > > > > > > > > > > > > > If we agree to move to Github Actions, would we like to > > > migrate > > > > > > > > > > incrementally or fully? > > > > > > > > > > > > > > > > > > > > Bests, > > > > > > > > > > Tomek > > > > > > > > > > > > > > > > > > > > On Wed, Dec 11, 2019 at 1:06 AM Jarek Potiuk < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > No problem on that side - Tomek is using the same > scripts > > > we > > > > > have > > > > > > > on > > > > > > > > > > Travis > > > > > > > > > > > (and they generally work - I think the last step is to > > make > > > > all > > > > > > the > > > > > > > > > > > tests pass. We have still a number of dependencies > > between > > > > > tests > > > > > > > and > > > > > > > > > some > > > > > > > > > > > environmental flakiness so that some tests consistently > > > fail > > > > in > > > > > > > > Github > > > > > > > > > > > Actions where they did not fail in Travis. From latest > > try > > > by > > > > > > Tomek > > > > > > > > it > > > > > > > > > > > looks like we are 1 test to go (plus some cleanup/setup > > of > > > > > > project > > > > > > > > and > > > > > > > > > > > making sure all is stable): > > > > > > > > > > > > > > > > > > > > > > ==== 1 failed, 4030 passed, 119 skipped, 16 warnings in > > > > > 1207.96s > > > > > > > > > > (0:20:07) > > > > > > > > > > > ===== > > > > > > > > > > > > > > > > > > > > > > We discussed with Tomek and Kamil that in order to > speed > > it > > > > up > > > > > we > > > > > > > > might > > > > > > > > > > > want to run our own workers on the GCP account we have > - > > > just > > > > > to > > > > > > > test > > > > > > > > > > > quickly how much we can optimise it and I am inclined > to > > > > agree. > > > > > > If > > > > > > > we > > > > > > > > > do > > > > > > > > > > it > > > > > > > > > > > this way, the transition might be rather fast. > > > > > > > > > > > > > > > > > > > > > > If we want to use auto-scalable GKE cluster as we > > > originally > > > > > > > planned > > > > > > > > it > > > > > > > > > > > might take more time to setup (similar setup to what I > > > tried > > > > > with > > > > > > > > > > GitLab). > > > > > > > > > > > There we might need to use docker-in-docker to build > the > > CI > > > > > image > > > > > > > > with > > > > > > > > > > > latest as first step of build and then use that built > > image > > > > by > > > > > > all > > > > > > > > > > > subsequent steps. But we can do it as the next step - > > > > > optimising > > > > > > > the > > > > > > > > > > > experience. > > > > > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 10, 2019 at 11:34 PM Daniel Imberman < > > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > +1 on my end as well. > > > > > > > > > > > > > > > > > > > > > > > > @jarek does this affect anything involving breeze? Do > > > > GitHub > > > > > > > > actions > > > > > > > > > > have > > > > > > > > > > > > an easy docker/docker-compose based workflow? > > > > > > > > > > > > > > > > > > > > > > > > via Newton Mail [ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2 > > > > > > > > > > > > ] > > > > > > > > > > > > On Tue, Dec 10, 2019 at 5:28 AM, Ash Berlin-Taylor < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > Legal: no I don't think so. > > > > > > > > > > > > > > > > > > > > > > > > Infra: possibly yes to get secrets in there to run > > things > > > > on > > > > > > our > > > > > > > > own > > > > > > > > > > > > "hardware" - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets > > > > > > > > > > > > > > > > > > > > > > > > needs someone with Admin rights to create, and I > don't > > > see > > > > > the > > > > > > > > > Settings > > > > > > > > > > > tab > > > > > > > > > > > > at all. > > > > > > > > > > > > > > > > > > > > > > > > -ash > > > > > > > > > > > > > > > > > > > > > > > > > On 10 Dec 2019, at 02:46, Deng Xiaodong < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > +1 for GitHub Actions. I have been using it for > > months > > > > for > > > > > my > > > > > > > > side > > > > > > > > > > > > > projects, and it’s working very well. I believe > most > > of > > > > us > > > > > > are > > > > > > > > > quite > > > > > > > > > > > > tired > > > > > > > > > > > > > of the waiting time using Travis CI. > > > > > > > > > > > > > > > > > > > > > > > > > > The only point I would like to remind is whether we > > > need > > > > to > > > > > > > > > > communicate > > > > > > > > > > > > > with Infra/Legal team for this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > XD > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 10, 2019 at 06:49 Kaxil Naik < > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> +1 for Github actions > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Mon, Dec 9, 2019, 22:16 Ash Berlin-Taylor < > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > >>> Happy with any thing that gives a more seamless > CI > > > > > > > experience - > > > > > > > > > > > faster > > > > > > > > > > > > is > > > > > > > > > > > > >>> good too! > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> -a > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> On 9 December 2019 22:12:05 GMT, Aizhamal > Nurmamat > > > > kyzy < > > > > > > > > > > > > >>> [email protected]> wrote: > > > > > > > > > > > > >>>> +1 on GitHub Actions. > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> On Mon, Dec 9, 2019 at 2:10 PM Jarek Potiuk < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > >>>> wrote: > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>>> I am all for it! GitLab has been less-than > > helpful > > > so > > > > > far > > > > > > > and > > > > > > > > > > > > >>>> recently it > > > > > > > > > > > > >>>>> seems that running PRs from forks will only be > > run > > > in > > > > > > > > > Enterrprise > > > > > > > > > > > > >>>> Edition, > > > > > > > > > > > > >>>>> which is less than welcome. I am quite a bit > > > > > disappointed > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > > >>>> pace and > > > > > > > > > > > > >>>>> attitude. Github Actions seems to be much > better > > > > > choice - > > > > > > > > > > > especially > > > > > > > > > > > > >>>> that > > > > > > > > > > > > >>>>> they are closely integrated with Github repo > and > > > seem > > > > > to > > > > > > > get > > > > > > > > > > > > >>>>> attention/focus from Github/Microsoft. > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> And they added self-hosted runners as well, > which > > > > makes > > > > > > it > > > > > > > > > > possible > > > > > > > > > > > > >>>> for us > > > > > > > > > > > > >>>>> to optimise the experience. > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> J. > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> J. > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> On Mon, Dec 9, 2019 at 10:57 PM Tomasz > Urbaszek < > > > > > > > > > > > > >>>>> [email protected]> > > > > > > > > > > > > >>>>> wrote: > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>>> Hi all, > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> It sometime since we last discussed using > other > > CI > > > > > than > > > > > > > > > Travis. > > > > > > > > > > > One > > > > > > > > > > > > >>>> of > > > > > > > > > > > > >>>>> the > > > > > > > > > > > > >>>>>> main reasons behind considering Gitlab CI was > > its > > > > > > ability > > > > > > > to > > > > > > > > > > work > > > > > > > > > > > > >>>> on > > > > > > > > > > > > >>>>>> self-hosted runner. However, over time of few > > long > > > > > weeks > > > > > > > > > Github > > > > > > > > > > > > >>>> Actions > > > > > > > > > > > > >>>>>> matured enough to allow using self-hosted > > runners! > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> Github Actions are still growing but using > them > > > have > > > > > few > > > > > > > big > > > > > > > > > > > > >>>> advantages: > > > > > > > > > > > > >>>>>> - they are Github natives > > > > > > > > > > > > >>>>>> - forking repo and enabling actions will run > CI > > on > > > > > your > > > > > > > fork > > > > > > > > > > > > >>>>> automatically > > > > > > > > > > > > >>>>>> - variety of actions (PR checks, greetings, > etc) > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> I put together a PoC of CI in our internal > repo: > > > > > > > > > > > > >>>>>> > > > https://github.com/PolideaInternal/airflow/pull/542 > > > > > > > > > > > > >>>>>> My impression is quite good. I like > information > > > > about > > > > > > > steps > > > > > > > > > > > > >>>> successes at > > > > > > > > > > > > >>>>>> the PR level (no need to go to CI to check > which > > > > step > > > > > > > > failed). > > > > > > > > > > The > > > > > > > > > > > > >>>> build > > > > > > > > > > > > >>>>>> log view is a little bit clumsy but it works. > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> Does any of you have any experience with > Github > > > > > Actions? > > > > > > > Any > > > > > > > > > > > > >>>> thoughts > > > > > > > > > > > > >>>>>> about using it? > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> Best, > > > > > > > > > > > > >>>>>> Tomek > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> On 2019/08/09 13:55:11, Jarek Potiuk < > > > > > > > > > [email protected]> > > > > > > > > > > > > >>>> wrote: > > > > > > > > > > > > >>>>>>> FYI: Interesting article about the history > > behind > > > > > > > GitLabCI > > > > > > > > > > > > >>>> (featuring > > > > > > > > > > > > >>>>>>> Kamil, my friend). > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://about.gitlab.com/2019/08/08/built-in-ci-cd-version-control-secret/?fbclid=IwAR2tEfqLaDXTCd1mD6XUZMX7hGYBfZcohPtI2BP3-oK_Yk_EHIXF4zLDixk > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> On Mon, Aug 5, 2019 at 7:14 PM Jarek Potiuk > > > > > > > > > > > > >>>> <[email protected]> > > > > > > > > > > > > >>>>>>> wrote: > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>>> Some update on my GitLab experiences so far: > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> TL;DR; I think the POC has shown that we can > > > > fairly > > > > > > > easily > > > > > > > > > > > > >>>> replicate > > > > > > > > > > > > >>>>>> the > > > > > > > > > > > > >>>>>>>> CI in GitLab + Kubernetes. I think i can > say - > > > it > > > > > > > > generally > > > > > > > > > > > > >>>> works, I > > > > > > > > > > > > >>>>>> can > > > > > > > > > > > > >>>>>>>> plug it in for master/v1-10-test builds in > the > > > > main > > > > > > > > Airflow > > > > > > > > > > > > >>>> project > > > > > > > > > > > > >>>>>> for a > > > > > > > > > > > > >>>>>>>> few weeks to see how it is doing (while I am > > no > > > > > > > holidays) > > > > > > > > > and > > > > > > > > > > > > >>>> once we > > > > > > > > > > > > >>>>>> see > > > > > > > > > > > > >>>>>>>> it running and get the support for PRs from > > > GitLab > > > > > we > > > > > > > can > > > > > > > > > > > > >>>> switch to > > > > > > > > > > > > >>>>> it. > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> What do you think ? Should i call a vote or > > just > > > > try > > > > > > to > > > > > > > > set > > > > > > > > > it > > > > > > > > > > > > >>>> up ? > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> Some details > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> - I manged to get full working builds in > > > GitLabCI > > > > + > > > > > > > > > > > > >>>> kubernetes - > > > > > > > > > > > > >>>>>>>> without the kubernetes-specific tests yet, > but > > > > this > > > > > > > should > > > > > > > > > > > > >>>> be > > > > > > > > > > > > >>>>>> rather easy > > > > > > > > > > > > >>>>>>>> with kind (looking at it next): > > > > > > > > > > > > >>>>>>>> - Working example here - you can take a look > > and > > > > > > compare > > > > > > > > the > > > > > > > > > > > > >>>>> UI/how > > > > > > > > > > > > >>>>>> it > > > > > > > > > > > > >>>>>>>> is to navigate, comparing to Travis etc: > > > > > > > > > > > > >>>>>>>> > > > > > > > > https://gitlab.com/Jarek.Potiuk/airflow/pipelines/74625817 > > > > > > > > > > > > >>>>>>>> - Per-job it is a bit slower than Travis so > > far > > > > > (still > > > > > > > > > > > > >>>> around 35 > > > > > > > > > > > > >>>>>>>> minutes in total), but I plan to optimise it > > > > > further. > > > > > > I > > > > > > > > can > > > > > > > > > > > > >>>> play > > > > > > > > > > > > >>>>>> with > > > > > > > > > > > > >>>>>>>> memory/cpu settings of individual workers > (Got > > > > some > > > > > > > > > > > > >>>> reasonable > > > > > > > > > > > > >>>>>> values now), > > > > > > > > > > > > >>>>>>>> I can use local SSD disk as Docker > > > > storage/logs/etc > > > > > > > > > > > > >>>>>>>> - I got an approval for 72vCPU quota (up for > > > > initial > > > > > > > 24) - > > > > > > > > > > > > >>>> that > > > > > > > > > > > > >>>>>> should > > > > > > > > > > > > >>>>>>>> let us build 3 builds in parallel > > independently > > > > from > > > > > > > each > > > > > > > > > > > > >>>> other. > > > > > > > > > > > > >>>>>>>> - I managed to get Preemptible nodes working > > (we > > > > > have > > > > > > > > built > > > > > > > > > > > > >>>> in > > > > > > > > > > > > >>>>> retry > > > > > > > > > > > > >>>>>>>> mechanism in GitLab to work in case of > system > > > > > failures > > > > > > > > like > > > > > > > > > > > > >>>> that > > > > > > > > > > > > >>>>>>>> - Current spending with > 120 builds is 40 > > USD. > > > We > > > > > > > should > > > > > > > > be > > > > > > > > > > > > >>>> way > > > > > > > > > > > > >>>>>> below > > > > > > > > > > > > >>>>>>>> 500 USD/month according to my > > > back-of-the-envelope > > > > > > > > > > > > >>>> calculations. > > > > > > > > > > > > >>>>>> Likely > > > > > > > > > > > > >>>>>>>> well below > > > > > > > > > > > > >>>>>>>> - The current setup does not use GCR as > cache > > > and > > > > > > Kaniko > > > > > > > > as > > > > > > > > > > > > >>>> I > > > > > > > > > > > > >>>>>>>> originally planned. GCR would require custom > > > > > > > > authentication > > > > > > > > > > > > >>>> (and > > > > > > > > > > > > >>>>>>>> easy-to-steal secrets) and Kaniko does not > yet > > > > well > > > > > > > handle > > > > > > > > > > > > >>>>>> multi-staging > > > > > > > > > > > > >>>>>>>> builds (cache does not work > > > > > > > > > > > > >>>>>>>> > > > > > > > https://github.com/GoogleContainerTools/kaniko/issues/682 > > > > > > > > ). > > > > > > > > > > > > >>>> I > > > > > > > > > > > > >>>>>> updated > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > > > > > > > > > > > >>>>>> to > > > > > > > > > > > > >>>>>>>> reflect that. > > > > > > > > > > > > >>>>>>>> - We only use GCR as mirroring of DockerHub > - > > so > > > > > that > > > > > > we > > > > > > > > can > > > > > > > > > > > > >>>> have > > > > > > > > > > > > >>>>>>>> reliable downloads not depending on > > DockerHub's > > > > > > > stability > > > > > > > > > > > > >>>> (it has > > > > > > > > > > > > >>>>>> problems > > > > > > > > > > > > >>>>>>>> sometimes) > > > > > > > > > > > > >>>>>>>> - All in-all, it's GCP-independent. It could > > be > > > > run > > > > > in > > > > > > > any > > > > > > > > > > > > >>>>>> Kubernetes > > > > > > > > > > > > >>>>>>>> cluster (some optimisations like local > volumes > > > > > > mounting > > > > > > > > for > > > > > > > > > > > > >>>> docker > > > > > > > > > > > > >>>>>> engine > > > > > > > > > > > > >>>>>>>> might have GCP-specific assumptions, but > > should > > > be > > > > > > > > generally > > > > > > > > > > > > >>>>>> replicable). > > > > > > > > > > > > >>>>>>>> - You can take a look at the current source > > code > > > > in > > > > > > > > > > > > >>>>>>>> > > > > > > > https://github.com/potiuk/airflow/commits/test-gitlab-ci > > > > > > > > > > > > >>>>>>>> - There will be some updates (I will get rid > > of > > > > > custom > > > > > > > > > > > > >>>> builder > > > > > > > > > > > > >>>>>> Docker, > > > > > > > > > > > > >>>>>>>> simplify it a bit and implement kubernetes > > > tests) > > > > - > > > > > > it's > > > > > > > > > > > > >>>> mostly > > > > > > > > > > > > >>>>> some > > > > > > > > > > > > >>>>>>>> cleanups + removal of Travis-Specific > > variables > > > + > > > > > > > > gitlab.ci > > > > > > > > > > > > >>>> yaml > > > > > > > > > > > > >>>>>> with > > > > > > > > > > > > >>>>>>>> job definitions. > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> J. > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> On Wed, Jul 31, 2019 at 10:57 AM Jarek > Potiuk > > < > > > > > > > > > > > > >>>>>> [email protected]> > > > > > > > > > > > > >>>>>>>> wrote: > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>>> So GitLab already works on automatically > > > running > > > > > > builds > > > > > > > > > from > > > > > > > > > > > > >>>> for PRs > > > > > > > > > > > > >>>>>> :). > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > >>>>>>>>> Kamil got involved and will be out advocate > > on > > > > it: > > > > > > > > > > > > >>>>>>>>> > > > > > https://gitlab.com/gitlab-org/gitlab-ce/issues/65139 > > > > > > > > > > > > >>>>>>>>> J. > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > >>>>>>>>> Principal Software Engineer > > > > > > > > > > > > >>>>>>>>> Phone: +48660796129 <+48%20660%20796%20129> > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > >>>>>>>>> pt., 26 lip 2019, 18:12 użytkownik Jarek > > > Potiuk < > > > > > > > > > > > > >>>>>> [email protected]> > > > > > > > > > > > > >>>>>>>>> napisał: > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> Update: I added appropriate comment in the > > > > GitLab > > > > > CI > > > > > > > > issue > > > > > > > > > > > > >>>> about > > > > > > > > > > > > >>>>> PRs > > > > > > > > > > > > >>>>>> and > > > > > > > > > > > > >>>>>>>>>> we are getting attention of Jason Lenny - > > > > director > > > > > > of > > > > > > > > > > Product > > > > > > > > > > > > >>>>>> Management @ > > > > > > > > > > > > >>>>>>>>>> GitLab. Let's hope they prioritise it > > quickly > > > > > > enough. > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> Speaking of potential > > complexity/Maintenance - > > > > in > > > > > > > order > > > > > > > > to > > > > > > > > > > > > >>>>> alleviate > > > > > > > > > > > > >>>>>> any > > > > > > > > > > > > >>>>>>>>>> maintenance worries, I think about setting > > up > > > > the > > > > > > > whole > > > > > > > > > > > > >>>> system on > > > > > > > > > > > > >>>>>> GitLab > > > > > > > > > > > > >>>>>>>>>> CI + GKE and running it in parallel to > > Travis > > > > for > > > > > > > quite > > > > > > > > > some > > > > > > > > > > > > >>>> time > > > > > > > > > > > > >>>>>> (even > > > > > > > > > > > > >>>>>>>>>> months) so that we can switch it at any > > time. > > > > Then > > > > > > we > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > > >>>> able > > > > > > > > > > > > >>>>>> to tune > > > > > > > > > > > > >>>>>>>>>> it according to real use cases and compare > > the > > > > > > > > experience > > > > > > > > > of > > > > > > > > > > > > >>>> both > > > > > > > > > > > > >>>>>> systems. > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> Also I am going for holidays in two weeks > > and > > > I > > > > > will > > > > > > > > make > > > > > > > > > > > > >>>> sure that > > > > > > > > > > > > >>>>>>>>>> there will be someone with GitLab + > > Kubernetes > > > > > > > > experience > > > > > > > > > > > > >>>> (from my > > > > > > > > > > > > >>>>>> company) > > > > > > > > > > > > >>>>>>>>>> who can take over and make sure there will > > be > > > no > > > > > > > > problems. > > > > > > > > > > > > >>>> However > > > > > > > > > > > > >>>>> I > > > > > > > > > > > > >>>>>> am > > > > > > > > > > > > >>>>>>>>>> quite confident :D nothing is going to > > happen > > > > > while > > > > > > I > > > > > > > am > > > > > > > > > > > > >>>> away. I > > > > > > > > > > > > >>>>>> would also > > > > > > > > > > > > >>>>>>>>>> invite whoever from committers who would > > like > > > to > > > > > > join > > > > > > > > the > > > > > > > > > > > > >>>> project > > > > > > > > > > > > >>>>> and > > > > > > > > > > > > >>>>>>>>>> gitlab instance (once I setup POC) to > learn > > > and > > > > > see > > > > > > > how > > > > > > > > > easy > > > > > > > > > > > > >>>> it is > > > > > > > > > > > > >>>>>> and how > > > > > > > > > > > > >>>>>>>>>> maintenance free it is going to be. > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> J. > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> On Fri, Jul 26, 2019 at 2:56 PM Kamil > > Breguła > > > < > > > > > > > > > > > > >>>>>> [email protected]> > > > > > > > > > > > > >>>>>>>>>> wrote: > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> GKE and its own CI will allow us to solve > > > other > > > > > > > > problems > > > > > > > > > - > > > > > > > > > > > > >>>>> building > > > > > > > > > > > > >>>>>>>>>>> and publishing documentation from the > > master > > > > > > branch. > > > > > > > > > > > > >>>> Currently, > > > > > > > > > > > > >>>>>>>>>>> building is done using the RTD service. > > > > > > > Unfortunately, > > > > > > > > > our > > > > > > > > > > > > >>>> project > > > > > > > > > > > > >>>>>> is > > > > > > > > > > > > >>>>>>>>>>> too large and often the documentation is > > not > > > > > built > > > > > > > > > > properly. > > > > > > > > > > > > >>>>>>>>>>> > > > > https://readthedocs.org/projects/airflow/builds/ > > > > > > > > > > > > >>>>>>>>>>> We should think about another way to > build > > > > > > > > documentation. > > > > > > > > > > In > > > > > > > > > > > > >>>> the > > > > > > > > > > > > >>>>>> ideal > > > > > > > > > > > > >>>>>>>>>>> world, building documentation should use > > the > > > > same > > > > > > > > > > > > >>>> environment as > > > > > > > > > > > > >>>>>>>>>>> checking documentation on CI. Adding this > > > step > > > > to > > > > > > > > Travis > > > > > > > > > > can > > > > > > > > > > > > >>>>> further > > > > > > > > > > > > >>>>>>>>>>> reduce our development opportunities. > > > > > > > > > > > > >>>>>>>>>>> Discussion on Slack about it: > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-airflow.slack.com/archives/CJ1LVREHX/p1561756652021900 > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> It is worth thinking about the fact that > > our > > > > > > project > > > > > > > > will > > > > > > > > > > > > >>>> soon > > > > > > > > > > > > >>>>> have > > > > > > > > > > > > >>>>>> a > > > > > > > > > > > > >>>>>>>>>>> website and our documentation will also > be > > > > > > available > > > > > > > in > > > > > > > > > > many > > > > > > > > > > > > >>>>>>>>>>> languages. Currently, talks are taking > > place > > > > with > > > > > > the > > > > > > > > > > design > > > > > > > > > > > > >>>>> studio > > > > > > > > > > > > >>>>>>>>>>> and developers who can make these > websites > > > ;-) > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/982c7baa06742ad722f2baa0db53ad99aea6c26b14b7d6d4aa522677@%3Cdev.airflow.apache.org%3E > > > > > > > > > > > > >>>>>>>>>>> We should provide an environment that > will > > > > allow > > > > > > you > > > > > > > to > > > > > > > > > > > > >>>> build a > > > > > > > > > > > > >>>>>>>>>>> website and documentation. At best, these > > > tasks > > > > > > > should > > > > > > > > be > > > > > > > > > > > > >>>>> combined. > > > > > > > > > > > > >>>>>> I > > > > > > > > > > > > >>>>>>>>>>> hope that we will be able to create a > > website > > > > > that > > > > > > > will > > > > > > > > > be > > > > > > > > > > a > > > > > > > > > > > > >>>> real > > > > > > > > > > > > >>>>>>>>>>> support for the community on current > > events, > > > so > > > > > it > > > > > > > will > > > > > > > > > be > > > > > > > > > > > > >>>> updated > > > > > > > > > > > > >>>>>>>>>>> frequently. > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> It seems to me that the project will > grow. > > If > > > > we > > > > > > now > > > > > > > > have > > > > > > > > > > > > >>>> problems > > > > > > > > > > > > >>>>>>>>>>> with Travis, then the significance of > these > > > > > > problems > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > >>>> future > > > > > > > > > > > > >>>>>> can > > > > > > > > > > > > >>>>>>>>>>> only grow. Now we have a chance to > provide > > a > > > > > stable > > > > > > > > > > > > >>>> infrastructure > > > > > > > > > > > > >>>>>> for > > > > > > > > > > > > >>>>>>>>>>> the project for a long time. > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> I would like to share another situation > > which > > > > was > > > > > > not > > > > > > > > > > > > >>>> pleasant for > > > > > > > > > > > > >>>>>> me. > > > > > > > > > > > > >>>>>>>>>>> Recently I wanted to send >10 PR, but > > because > > > > of > > > > > > > > Travis, > > > > > > > > > I > > > > > > > > > > > > >>>> had to > > > > > > > > > > > > >>>>>> wait > > > > > > > > > > > > >>>>>>>>>>> for the weekend to send changes. If I > would > > > > send > > > > > my > > > > > > > > > changes > > > > > > > > > > > > >>>> in a > > > > > > > > > > > > >>>>>> week, > > > > > > > > > > > > >>>>>>>>>>> I would block the queue for a few hours. > > > > > Although I > > > > > > > did > > > > > > > > > it > > > > > > > > > > > > >>>> over > > > > > > > > > > > > >>>>> the > > > > > > > > > > > > >>>>>>>>>>> weekend, I got the message that the queue > > is > > > > > > blocked > > > > > > > on > > > > > > > > > > > > >>>> Travis by > > > > > > > > > > > > >>>>> my > > > > > > > > > > > > >>>>>>>>>>> jobs. > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> On Tue, Jul 23, 2019 at 6:12 PM Jarek > > Potiuk > > > < > > > > > > > > > > > > >>>>>> [email protected]> > > > > > > > > > > > > >>>>>>>>>>> wrote: > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> Hello Everyone, > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> I prepared a short docs where I > described > > > > > general > > > > > > > > > > > > >>>> architecture > > > > > > > > > > > > >>>>> of > > > > > > > > > > > > >>>>>> the > > > > > > > > > > > > >>>>>>>>>>>> solution I imagine we can deploy fairly > > > > quickly > > > > > - > > > > > > > > having > > > > > > > > > > > > >>>> GitLab > > > > > > > > > > > > >>>>> CI > > > > > > > > > > > > >>>>>>>>>>> support > > > > > > > > > > > > >>>>>>>>>>>> and Google provided funding for GCP > > > resources. > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> I am going to start working on > > > > Proof-Of-Concept > > > > > > soon > > > > > > > > but > > > > > > > > > > > > >>>> before > > > > > > > > > > > > >>>>> I > > > > > > > > > > > > >>>>>>>>>>> start > > > > > > > > > > > > >>>>>>>>>>>> doing it, I would like to get some > > comments > > > > and > > > > > > > > opinions > > > > > > > > > > > > >>>> on the > > > > > > > > > > > > >>>>>>>>>>> proposed > > > > > > > > > > > > >>>>>>>>>>>> approach. I discussed the basic approach > > > with > > > > my > > > > > > > > friend > > > > > > > > > > > > >>>> Kamil > > > > > > > > > > > > >>>>> who > > > > > > > > > > > > >>>>>>>>>>> works at > > > > > > > > > > > > >>>>>>>>>>>> GitLab and he is a CI maintainer and > this > > is > > > > > what > > > > > > we > > > > > > > > > think > > > > > > > > > > > > >>>> will > > > > > > > > > > > > >>>>> be > > > > > > > > > > > > >>>>>>>>>>>> achievable in fairly short time. > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> I am happy to discuss details and make > > > changes > > > > > to > > > > > > > the > > > > > > > > > > > > >>>> proposal - > > > > > > > > > > > > >>>>>> we > > > > > > > > > > > > >>>>>>>>>>> can > > > > > > > > > > > > >>>>>>>>>>>> discuss it here or as comments in the > > > > document. > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> Let's see what people think about it and > > if > > > we > > > > > get > > > > > > > to > > > > > > > > > some > > > > > > > > > > > > >>>>>> consensus > > > > > > > > > > > > >>>>>>>>>>> we > > > > > > > > > > > > >>>>>>>>>>>> might want to cast a vote (or maybe go > via > > > > lasy > > > > > > > > > consensus > > > > > > > > > > > > >>>> as > > > > > > > > > > > > >>>>> this > > > > > > > > > > > > >>>>>> is > > > > > > > > > > > > >>>>>>>>>>>> something we should have rather quickly) > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> Looking forward to your comments! > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> J. > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> -- > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> Jarek Potiuk > > > > > > > > > > > > >>>>>>>>>>>> Polidea <https://www.polidea.com/> | > > > > Principal > > > > > > > > Software > > > > > > > > > > > > >>>>> Engineer > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>>>> M: +48 660 796 129 > <+48%20660%20796%20129> > > > > > > > > <+48660796129 > > > > > > > > > > > > >>>>> <+48%20660%20796%20129>> > > > > > > > > > > > > >>>>>>>>>>>> [image: Polidea] < > > https://www.polidea.com/> > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> -- > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> Jarek Potiuk > > > > > > > > > > > > >>>>>>>>>> Polidea <https://www.polidea.com/> | > > > Principal > > > > > > > Software > > > > > > > > > > > > >>>> Engineer > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> M: +48 660 796 129 <+48%20660%20796%20129> > > > > > > > <+48660796129 > > > > > > > > > > > > >>>>> <+48%20660%20796%20129>> > > > > > > > > > > > > >>>>>>>>>> [image: Polidea] < > https://www.polidea.com/> > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> -- > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> Jarek Potiuk > > > > > > > > > > > > >>>>>>>> Polidea <https://www.polidea.com/> | > > Principal > > > > > > Software > > > > > > > > > > > > >>>> Engineer > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> M: +48 660 796 129 <+48%20660%20796%20129> > > > > > > <+48660796129 > > > > > > > > > > > > >>>>> <+48%20660%20796%20129>> > > > > > > > > > > > > >>>>>>>> [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> -- > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> Jarek Potiuk > > > > > > > > > > > > >>>>>>> Polidea <https://www.polidea.com/> | > Principal > > > > > > Software > > > > > > > > > > Engineer > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> M: +48 660 796 129 <+48%20660%20796%20129> > > > > > > <+48660796129 > > > > > > > > > > > > >>>>> <+48%20660%20796%20129>> > > > > > > > > > > > > >>>>>>> [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> -- > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> Jarek Potiuk > > > > > > > > > > > > >>>>> Polidea <https://www.polidea.com/> | Principal > > > > > Software > > > > > > > > > Engineer > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> M: +48 660 796 129 <+48%20660%20796%20129> > > > > > <+48660796129 > > > > > > > > > > > > >>>>> <+48%20660%20796%20129>> > > > > > > > > > > > > >>>>> [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal > Software > > > > > Engineer > > > > > > > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Tomasz Urbaszek > > > > > > > > > > Polidea <https://www.polidea.com/> | Junior Software > > > Engineer > > > > > > > > > > > > > > > > > > > > M: +48 505 628 493 <+48505628493> > > > > > > > > > > E: [email protected] < > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > Unique Tech > > > > > > > > > > Check out our projects! < > https://www.polidea.com/our-work> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software > > Engineer > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Tomasz Urbaszek > > > > > > > Polidea <https://www.polidea.com/> | Junior Software Engineer > > > > > > > > > > > > > > M: +48 505 628 493 <+48505628493> > > > > > > > E: [email protected] <[email protected]> > > > > > > > > > > > > > > Unique Tech > > > > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Tomasz Urbaszek > > > > Polidea <https://www.polidea.com/> | Software Engineer > > > > > > > > M: +48 505 628 493 <+48505628493> > > > > E: [email protected] <[email protected]> > > > > > > > > Unique Tech > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
