Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
Yep. We are back. We still have a slow queue, but at least the builds are not failing randomly. Please rebase your builds on top of the current master and push again to trigger builds. On Tue, Jul 23, 2019 at 9:31 PM Jarek Potiuk wrote: > Looks like they fixed it: https://github.com/travis-ci/wo

Re: [Announce] Apache Airflow YouTube Channel

2019-07-23 Thread Kevin Yang
Thank you Aizhamal! Glad to see this as a heavy YouTube user :D Cheers, Kevin Y Aizhamal Nurmamat kyzy 于2019年7月23日 周二下午2:05写道: > Hello everyone, > > I thought it would be nice to have a central repo for talk recordings and > videos related to Apache Airflow, so I have created a YouTube Channel f

[Announce] Apache Airflow YouTube Channel

2019-07-23 Thread Aizhamal Nurmamat kyzy
Hello everyone, I thought it would be nice to have a central repo for talk recordings and videos related to Apache Airflow, so I have created a YouTube Channel for it [1]. I uploaded the recordings from Airflow meetup that took place on June 5 in the Bay Area. If any PMC or Committer wishes to ha

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-07-23 Thread Jarek Potiuk
> > Question - what is the purpose of introducing kaniko instead of using > regular docker build? > Indeed. We want to be as agnostic as possible. What I plan to do is to use Kubernetes Runner in GitlabCI. This means that all the jobs will run as Kubernetes PODs in GKE - Gitlab CI will only be UI

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-07-23 Thread Philippe Gagnon
I think that Jarek is proposing switching to Kaniko for security considerations as GKE workloads run in OS containers (so they do not benefit from hardware virtualization sandboxing) and docker build requires root privileges. In any case, Kaniko is not tied in any way to Google Cloud, so the build

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-07-23 Thread Shah Altaf
Hi that's a nice writeup, easy to follow. Also I like your diagram. Question - what is the purpose of introducing kaniko instead of using regular docker build? I'm asking in line with the consideration "The system should be self-maintainable - with as little special Development/Ops maintenance ne

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
Looks like they fixed it: https://github.com/travis-ci/worker/issues/604 On Tue, Jul 23, 2019 at 8:38 PM Driesprong, Fokko wrote: > I see issues at different Apache projects as well, Druid and Avro. They're > running out of memory. Let's see how Travis responds. > > Cheers, Fokko > > Op di 23 ju

Re: [VOTE] Changes in import paths

2019-07-23 Thread Felix Uellendall
I have no strong "No" against any proposed change of these cases. So I go with +1 (non-binding). P.S. Thanks Jarek for bringing this up again and your intense work towards airflow currently :) and thanks to Kamil for even creating this document. I like how the code is getting more and more cons

Re: [PROPOSE] Introduce and encourage pre-commit hooks framework to Airflow developer workflow

2019-07-23 Thread Driesprong, Fokko
I agree with the other folks. I'm personally not a fan of pre commit hooks, but if people like it, they can enable it. Cheers, Fokko Op di 23 jul. 2019 om 20:13 schreef Kaxil Naik > It is fully-optional so I don't think we need an AIP there. > > > > On Tue, Jul 23, 2019 at 11:36 PM Beau Barker

Re: Travis CI random failures

2019-07-23 Thread Driesprong, Fokko
I see issues at different Apache projects as well, Druid and Avro. They're running out of memory. Let's see how Travis responds. Cheers, Fokko Op di 23 jul. 2019 om 19:43 schreef Jarek Potiuk : > FYI. Still not fixed. Others experience this as well: > https://github.com/travis-ci/worker/issues/6

Re: [PROPOSE] Introduce and encourage pre-commit hooks framework to Airflow developer workflow

2019-07-23 Thread Kaxil Naik
It is fully-optional so I don't think we need an AIP there. On Tue, Jul 23, 2019 at 11:36 PM Beau Barker wrote: > Just add a .pre-commit-config.yaml to the project, no need for an AIP. > > > On 24 Jul 2019, at 2:42 am, Jarek Potiuk > wrote: > > > > Any more comments on it? > > Should I make a

Re: [PROPOSE] Introduce and encourage pre-commit hooks framework to Airflow developer workflow

2019-07-23 Thread Beau Barker
Just add a .pre-commit-config.yaml to the project, no need for an AIP. > On 24 Jul 2019, at 2:42 am, Jarek Potiuk wrote: > > Any more comments on it? > Should I make an AIP for that :)? Or should I just ask a vote/propose a PR > ? Anyone has a strong opinion? > I think it changes the dev workfl

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
FYI. Still not fixed. Others experience this as well: https://github.com/travis-ci/worker/issues/604 On Tue, Jul 23, 2019 at 11:34 AM Jarek Potiuk wrote: > No good news yet. We are getting randomly assigned 1CPU /3.5GB mem > instances still. Infrastructure is on it. > > On Tue, Jul 23, 2019 at 1

Re: [PROPOSE] Introduce and encourage pre-commit hooks framework to Airflow developer workflow

2019-07-23 Thread Jarek Potiuk
Any more comments on it? Should I make an AIP for that :)? Or should I just ask a vote/propose a PR ? Anyone has a strong opinion? I think it changes the dev workflow quite a bit on one hand, but it is fully optional on the other hand. On Thu, Jul 4, 2019 at 9:23 AM Kamil Breguła wrote: > +1 At

[VOTE] Changes in import paths

2019-07-23 Thread Jarek Potiuk
Hello everyone, This email is calling a vote on the changes in import paths. It's been discussed in https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E The vote will last for at least 1 week (July 30th 6pm CEST), and at least

[Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-07-23 Thread Jarek Potiuk
Hello Everyone, I prepared a short docs where I described general architecture of the solution I imagine we can deploy fairly quickly - having GitLab CI support and Google provided funding for GCP resources. I am going to start working on Proof-Of-Concept soon but before I start doing it, I would

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
No good news yet. We are getting randomly assigned 1CPU /3.5GB mem instances still. Infrastructure is on it. On Tue, Jul 23, 2019 at 10:49 AM Jarek Potiuk wrote: > It looks like we are back to the original specs. I am runnning tests and > re-enable everything if I see it works. > > J. > > On Tue

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
It looks like we are back to the original specs. I am runnning tests and re-enable everything if I see it works. J. On Tue, Jul 23, 2019 at 10:34 AM Jarek Potiuk wrote: > From INFRA: "I have confirmed that our builds appear to be running with > 3.75GB memory and 1 core currently. This does not

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
>From INFRA: "I have confirmed that our builds appear to be running with 3.75GB memory and 1 core currently. This does not match Travis' standard specs (7.5GB and 2 cores), and I have raised a ticket with their support. I will respond when we hear back from Travis." On Tue, Jul 23, 2019 at 10:26

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
It's definitely confirmed that the problem is on Travis CI side: I re-run the commit before the new CI was introduced (I cherry-picked a small doc fix related to recent sphinx dependency update) and it fails in exactly the same way (memory and cpu problems): https://travis-ci.org/apache/airflow/bu

Re: Travis CI random failures

2019-07-23 Thread Jarek Potiuk
It's now pretty consistent and happens pretty much every time using the old build system - for example here: https://travis-ci.org/apache/airflow/builds/562435992. I will cancel all PRs and disable automated PR build on Travis until we solve the problem - as it is pointless - new PRs will simply q