Prestosql project renamed to Trinodb

Hello Airflow-dev, About a couple of years ago the Presto community was fractured when the three founders of the project, Dain, David and Martin, left Facebook due to irreconcilable differences with the company concerning the stewardship of the open-source project. In order to continue working on

Re: New PIP and Airflow 1.10.*

There is also an option to force usage of the old dependency resolver: --use-deprecated=legacy-resolver But hopefully we find a more sustainable solution going forward! BR Philippe On Tue, Dec 1, 2020 at 10:01 AM Jarek Potiuk wrote: > For those who want to install Airflow 1.10.* today and fa

Re: [DISCUSS] Enable 'Black' for Auto Code Formatting

It's a good idea. It will make reading the codebase easier, and besides the whole Python ecosystem is slowly moving towards adopting this code style. I personally have been a fan ever since the project launched. With regards to open PRs requiring a rebase, it's an annoyance for sure but if we do

Re: Production images for Helm chart - question if we are doing the right thing ?

Hello Jarek, Since they are clearly marked as alpha releases I think this is okay. Just my opinion though. Regards, Philippe On Sun, Jun 7, 2020 at 1:09 PM Jarek Potiuk wrote: > Hello everyone, > > We have a question with Ash whether the way we are distributing alpha > releases of our Docker

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

We have been using Actions on the prestosql project for a little while as a Travis replacement and we like it a lot. On Tue, Dec 17, 2019 at 9:08 AM Jarek Potiuk wrote: > :( > > On Tue, Dec 17, 2019 at 2:35 PM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > wrote: > > > I started to play with

Re: Proposed PyCon 2020 Airflow Sprints

Thanks, Jarek. :) Just to provide some more information with regards to travel grants, these are offered through the PyCon financial aid program which sponsors travel and attendance costs for individuals who might otherwise not be able to attend. N

Re: Call to organize more First time Airflow Contributor's Workshops

Hey Jarek, I help with the organisation of PyCon USA and I would love to help run one of these during the sprint days at PyCon next April. I think it would be useful to have a "pocket committer" available to approve PRs quickly. Do you know if someone will be attending, and would be interested in

Re: [VOTE] Allow using asserts in Airflow code

-1 (non-binding). I can see where you are coming from, but in my opinion checks in the codebase should be used to direct runtime control flow. Otherwise I think they belong in proper tests. On Thu, Dec 5, 2019 at 9:56 AM Jarek Potiuk wrote: > Here is a quick vote on using asserts in Airflow c

Re: [Discuss] Airflow Summits 2020

Airflow Current? On Wed, Nov 20, 2019 at 5:52 AM Tomasz Urbaszek wrote: > Airflow Wind? Airflow Breath? > > On Wed, Nov 20, 2019 at 12:02 AM Jarek Potiuk > wrote: > > > How about "*Airflow Draught*" (sorry Breeze is already taken): > > > > From https://www.lexico.com/en/definition/draught: A cu

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

+1 non-binding On Mon, Oct 14, 2019 at 8:57 PM Kaxil Naik wrote: > Hello, Airflow community, > > This email calls for a vote to add the DAG Serialization feature at > https://github.com/apache/airflow/pull/5743. > > *AIP*: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persiste

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Another way to resolve this could be to define a convention such that a "cross transfer" operator should belong to either the source or destination cloud provider's package. Personally though I do not see any technical issues with having a "cross_transfer" package, but I don't find the name to rol

Re: [ANNOUNCE] Please welcome new Airflow committer Felix Uellendall

Congratulations! On Mon, Sep 9, 2019, 09:10 Kaxil Naik wrote: > Congratulations Felix > > On Mon, Sep 9, 2019 at 2:03 PM Jarek Potiuk > wrote: > > > Congrats Felix! > > > > pon., 9 wrz 2019, 12:58 użytkownik Ash Berlin-Taylor > > napisał: > > > > > All, > > > > > > I'm pleased to announce that

Re: Setting to add choice of schedule at end or schedule at start of interval

I agree with Daniel's rationale but I am also worried about backwards compatibility as this would perhaps be the most disruptive breaking change possible. I think maybe we should write down the different options available to us (AIP?) and call for a vote. What does everyone think? On Tue, Aug 27,

Re: [VOTE] Change the Airflow logo

Option 1 Thanks for the hard work Chris and Daniel! On Wed, Aug 21, 2019, 06:58 Ash Berlin-Taylor wrote: > The discuss thread had a positive response so I'm now calling for a VOTE > on updating the Airflow logo to one of the designs below. > > (Since no one said "I like the existing logo" I hav

Re: Subscription for a tech writer

Welcome to the community. Glad to have you on-board! On Mon, Aug 19, 2019, 16:14 Елена Лавренюк wrote: > > Hi! Joining the group now to start documenting the project > Elena > > > > Отправлено из мобильной Почты Mail.ru >

Re: [DISCUSS] Tweaks to the Airflow logo

+1 I like it as well. On Tue, Aug 13, 2019, 17:06 Aizhamal Nurmamat kyzy wrote: > I like it too! @Ash Berlin-Taylor - will be nice to add > the new one here > https://www.redbubble.com/people/comdev/works/40497530-apache-airflow?asc=u > for > future swags! > > On Tue, Aug 13, 2019 at 1:32 PM K

Re: Another Travis networking issue :(

Hey Jarek, AFAIK Gitlab implemented PR builds earlier this year, do they only trigger if the PR is coming from the same project? Also on another note, I wanted to point out that there are environment differences on Gitlab CI when CI runners run on Kubernetes vs when they run in VMs w/ plain old d

Re: [DISCUSS] Back to (some) dependency pinning

ned dependencies. We use > it > > in > > > Oozie-to-airflow for quite some time and it takes almost no time to > > upgrade > > > deps regularly: > > > > > > > > > https://github.com/GoogleCloudPlatform/oozie-to-airflow/pulls?utf8=%E2%9C%93&q=is%3Apr

Re: Airflow DAG Serialisation

I am not sure that pushing the responsibility of generating DAGs to clients outright is the right approach (I think it might raise the barrier to entry for newcomers), but I like the design that you are proposing. If we were to completely decouple DAG parsing from the scheduler then DAG processing

Re: "How to easily develop Airflow today" - in little more than 4 hours.

Thanks Jarek for showcasing the great work that you have been doing lately! I could only join the call at the midpoint but everything was very clear and informative. I hope that we will have more of these webinars in the future. Philippe On Thu, Jul 25, 2019 at 1:41 PM Jarek Potiuk wrote: > The

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

I think that Jarek is proposing switching to Kaniko for security considerations as GKE workloads run in OS containers (so they do not benefit from hardware virtualization sandboxing) and docker build requires root privileges. In any case, Kaniko is not tied in any way to Google Cloud, so the build

Re: [VOTE] Release Airflow 1.10.4 from RC3

ble version of Werkzeug (#5535) [Luqman > Abdul Mushawwir] > [AIRFLOW-4478] Lazily instantiate default resources objects. (#5259) > [Joshua Carp] > [AIRFLOW-4510] Don't mutate default_args during DAG initialization (#5277) > [Abhishek Ray] > [AIRFLOW-3360] Make the DAGs search res

Re: [DISCUSS] Back to (some) dependency pinning

I am +1 on pinning core packages, even though this adds a bit of manual labor for maintenance. This latest werkzeug issue highlights why this is a good idea. Also +1 on changing the versioning scheme to something more akin to semver. The current scheme basically does not support patch-only release

Re: [2.0 spring cleaning] Remove `dag >> task`?

Just to be clear, are you suggesting removing all bitshift operator overloads from airflow operators (sorry - the dual meaning of operator here is confusing), or just the assignment to DAG behavior? If it's the former, I find it to be a particularly expressive way to define dependencies between ta

Re: Travis builds in a queue for hours

es tests will likely be also fixed soon. > > > > > > Another idea: I thought that in the future we can also run only subset > of > > > postgres/mysql/sqlite tests on all combinations. I think there are just > > > handful of tests that are specific for

Re: [VOTE] Labelling scheme for the upcoming official docker images of Airflow

+1 (non-binding) On Thu, Jun 27, 2019 at 8:58 PM Gerardo Curiel wrote: > +1 (non-binding) > > On 27 June 2019 at 2:57:47 pm, Jarek Potiuk (jarek.pot...@polidea.com) > wrote: > > Hello Airflow community, > > This email call for a vote on "Labelling scheme" for the docker images we > are going to

Re: Travis builds in a queue for hours

I think the combinations that you are proposing are sensible for pre-merge checks. I am working on a proposal to offload extra combinations to another CI provider (Azure DevOps specifically seems like a good candidate), either pre or post merge. Ideally I'd like to run more combinations pre-merge

Re: I have joined Astronomer full-time !!

Congratulations Kaxil! This is wonderful news! On Fri, Jun 21, 2019 at 4:37 PM Tao Feng wrote: > Awesome:) Congrats! > > On Fri, Jun 21, 2019 at 11:53 AM Kaxil Naik wrote: > > > Hi all, > > > > I have already posted this on LinkedIn but for others in the community, I > > am super-excited to ann

Re: Tagging of the airflow images

ter*==airflow:master-python3.6 > > >> - CI images (big) *: airflow:master-ci-python3.5, > > >> airflow:**master-ci-python3.6, > > >> airflow:master-ci*==airflow:master-ci-python3.6 > > >> - Production optimised images: (future): > > *airflow:maste

Re: Tagging of the airflow images

the docker hub page once we have versioned > images there if we make it clear that `:master` is not intended for end > users, and by the same argument if we have anything as `:latest` it should > be a docker image relating to an official Release. > > Jarek: no `latest` pointing at C

Re: Tagging of the airflow images

One thing: we talked about releasing images under a "master" tag (perhaps in another thread?), we should check if this is compatible with Apache's release policy [1]. It's not clear to me if this is allowable or not after a cursory reading. [1] http://www.apache.org/legal/release-policy.html#what

Re: Does anybody deploy DAGs in zip files?

We do, but it brings a host of issues so I would not be disinclined to adjust our workflow if this functionality was removed. Our teams are very decentralized so we keep every dag in a separate git repository. During our deployment process (triggered by CI/CD) we zip everything up and put the zip f

Re: [2.0 spring cleaning] Changes in import paths

Hello community, Here is my take on this: Case #1: Solution B, but I would like for us to consider putting incubating modules in a separate repository instead of vendoring them with Airflow itself. There are pros and cons to this approach. On one hand, I think that we need to take responsibilit