Re: Airflow Summit 2021 looks for co-organizers

Hey Jarek, Great to see this happen again in 2012! GoDataDriven is happy to host one of the dates again in Amsterdam, Netherlands! Cheers, Fokko Op do 7 jan. 2021 om 21:33 schreef Jarek Potiuk : > Hello everyone, > > We are about to restart work on Airflow Summit 2021 and we are looking for > c

Unsubscribe from dev-list

Hi all, I've decided to take a break from the dev-list. There is a lot of action on the list, which is great. Awesome to see the community become more active every day. But currently, I'm not working with Airflow on a daily basis, so it is hard to keep up. Also due to the pandemic I'm spending a b

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

In Java this isn’t an issue. With Jackson you would just read the field as a Map and check if the items are available in the map. The Object can also be a dictionary or array. No need to recompile, but you need good errorhandling in case a key isn’t present. Cheers, Fokko Op zo 31 mei 2020 om 09:

Re: [DISCUSS] Naming of the transfer operators/Hooks

1 looks nice and clean Op za 30 mei 2020 om 11:37 schreef Jarek Potiuk > That's a completely new thing/idea. > > We might have separate vote for that. Kamil - can you please lead the vote > and renaming if it happens? > > I am for the change. I think it would make sense to separate them. > > I w

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

I would be in favor of option 2. I would not support both to avoid additional complexity. Validation should happen on write, and not read, so anything that isn’t json serializable should not be in the field. Cheers, Fokko Op vr 29 mei 2020 om 01:58 schreef Kaxil Naik > If the only problem is wi

Re: [VOTE] Airflow Backport Providers 2020.5.20rc2

I have a few questions. I saw the thread and it would be great to move this forward. Currently I don't have time to test all the individual packages. I did take the celery one and ran the Release Audit Tool (RAT) of Apache: https://gist.github.com/Fokko/50bb812a4998e34f4274965276a60f8e Files with

Re: List of companies and use cases

Other projects have this as well: https://spark.apache.org/powered-by.html https://druid.apache.org/druid-powered AFAIK we don't violate any rules. As long as you treat companies the same (sort them alphabetically etc). Cheers, Fokko Op do 30 apr. 2020 om 15:04 schreef Tomasz Urbaszek : > Hi a

Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection balancing HA )

+1 (binding) Thank you Ash! Op vr 24 apr. 2020 om 12:41 schreef Robin Edwards : > +1 > > On Fri, 24 Apr 2020, 11:32 Kaxil Naik, wrote: > > > +1 > > > > On Fri, Apr 24, 2020 at 11:31 AM Ash Berlin-Taylor > wrote: > > > > > As we discussed in the thread on Spec thread for our new API, the idea >

Re: Kudos for delivering Airflow talk at online data conference for Spanish speakers

Great work Jarek & Tomek! Thanks for sharing Op wo 22 apr. 2020 om 23:01 schreef Aizhamal Nurmamat kyzy < aizha...@apache.org> > Hi everyone, > > Today, Jarek and Tomek delivered an online talk on Apache Airflow at the > largest Spanish Speaking Data Conference [1]. Even though the talk was all >

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc5 (Accelerated vote)

+1 binding Op do 9 apr. 2020 om 21:11 schreef Ash Berlin-Taylor > Confirming differences between 1.10.10rc4 and rc5 > > diff -r /tmp/tmp.sgu8qJJFl9/airflow/models/chart.py > /tmp/tmp.HatTjZDbBh/airflow/models/chart.py > 23a24 > > from airflow.models.user import User > 44c45 > < "User", c

Re: [DISCUSS] Back to (some) dependency pinning

> > J. > > On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman > > wrote: > > > Agreed. Kind of a “best we can do” considering the current nature of > > python. > > On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko >, > > wrote: > > > Yes, I

Re: [DISCUSS] Back to (some) dependency pinning

Yes, I'd be in favor of not having two packages, and just pinning the versions then. In this case, all the versions will be pinned, so if a user wants to install a newer version of elastic, they have to do it explicitly. For Java, you have nice packages that will check if you break any public API,

Re: [VOTE] AIP-33 Secrets Backend

+1 Op ma 16 mrt. 2020 om 12:53 schreef Bas Harenslak : > +1 > > > On 16 Mar 2020, at 12:27, Ash Berlin-Taylor wrote: > > > > +1 binding > > > > On Mar 16 2020, at 11:20 am, Kaxil Naik wrote: > > > >> +1 (binding) > >> > >> On Mon, Mar 16, 2020 at 11:19 AM Jarek Potiuk > > >> wrote: > >> > >>>

Re: [DISCUSS] AIP-33 Creds Backend

Yes, big +1 from my side as well. I think this is a requirement for any bigger company. 1. I think we should add support for KMS, Vault etc. 2. Make the current backend pluggable, and set the current as default (Airflow metastore) Cheers, Fokko Op ma 16 mrt. 2020 om 11:19 schreef Kaxi

Re: [DISCUSS] Back to (some) dependency pinning

Personally I don't like to have two versions in the PyPi repo. This also complicates the releases, since we need to test, release and verify two versions of Airflow. I'm afraid that this might confuse users. Besides that, it feels a bit like we're reinventing certain mechanisms that are already in

Re: Stateless Webserver with DAG Serialization

Thanks for the update Kaxil, this is awesome! 💯 Cheers, Fokko Op za 14 mrt. 2020 om 13:40 schreef Tomasz Urbaszek : > Thanks Kaxil, Ash, Anita and Zhou! This is a big step forward :) > > T. > > On Sat, Mar 14, 2020 at 12:03 PM Sumit Maheshwari > wrote: > > > Great stuff!! Thanks to everyone who

Re: Airflow Summit 2020 new Website and CFP still open

Nice! Op do 27 feb. 2020 om 10:31 schreef Jarek Potiuk : > It was super-quick by the team that works on the summit. > > We have first set of (awesome!) speakers announced: > https://airflowsummit.org/speakers/ ! > > Register while we have early-birds price and CFP >

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

Sweet work Kamil and others! I'll try to go through them today! Cheers, Fokko Op ma 24 feb. 2020 om 22:37 schreef Tao Feng : > Great work Kamil! Let us know once it is landed in one of the future > releases. Would love to try it out :) > > Best, > -Tao > > On Mon, Feb 24, 2020 at 12:54 PM Qingpi

Re: [PROPOSAL] Approach for releasing the backported "providers" packages

I'm sorry, I've been a bit busy lately. Keeping track of these discussions in the evening doesn't always for out, as it seems. 1. So instead of releasing one package, as stated in the wiki, we're going to release over 50 provider packages (apache-airflow-providers-*-*-1.10-.MM.DD)? If I unders

Re: [VOTE] Release process for backported (AIP-21) package

Where is the code? I don't know what I'm voting for. If we want to do this, why aren't we cherry-picking this in the branch? I feel like I'm missing something. Cheers, Fokko Op di 18 feb. 2020 om 23:14 schreef Jarek Potiuk : > How do you propose to address this Ash (and _we_ especially) ? Any id

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

t; For example if someone had created a PR for this and had appropriate > > instructions in UPDATING I would feel okay merging it. > > > > -a > > > > On 18 February 2020 09:37:13 GMT, Jarek Potiuk > > > wrote: > > >All right. I turn it into vote then

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

t; > > > > wrote: > > > > > +1 on my end! > > > > > > via Newton Mail > > > [ > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 > > > ] > > > On Mon, Feb 17, 2020 at 12:

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

I like this as well. It will hopefully also reduce the memory footprint of Airflow. The only thing I can think of is that it will reduce the test coverage, but that's a vanity metric anyway :-) Cheers, Fokko Op za 15 feb. 2020 om 13:37 schreef Ash Berlin-Taylor : > I'm massively in favour of th

Re: How delete mailinglist subscription

Thanks for the question, Claudio. I agree that it isn't clear on how to unsubscribe. Sad to see you go :) I've created a PR for the website: https://github.com/apache/airflow-site/pull/250. Hopefully, this makes it more clear on how to unsubscribe from the mailing list. Cheers, Fokko Op di 11 fe

Re: [VOTE] Release Apache Airflow 1.10.9 from 1.10.9rc1 (shorter vote)

+1 (binding) Op vr 7 feb. 2020 om 17:17 schreef Jarek Potiuk > +1 approved (binding). Works for me in the same configurations as 1.10.8 > > On Fri, Feb 7, 2020 at 5:05 PM Kaxil Naik wrote: > > > +1 approved (binding) > > > > On Fri, Feb 7, 2020, 21:17 Ash Berlin-Taylor wrote: > > > > > Hello A

Re: Shorter release vote or 1.10.9 to pin Werkzeug dep

Hi Ash, I'm fine with doing a quicker release, as long as we have 3 PMC votes (preferably more). Cheers, Fokko Op vr 7 feb. 2020 om 14:05 schreef Ash Berlin-Taylor : > Hello all, > > So Werkzeug 1.0 was released this morning, and it finally broke a number > of direct or indirect dependencies of

Re: DAG shows Failed Status but process is successful

Thanks for sharing Ralph, However, it is very difficult to check what's happened on your side, without any logs. Would be greatly appreciated if you could share the actual logs of the task. What kind of operator are you running? The task will be marked as failed if the error code of the process is

Re: Reverting [AIRFLOW-1467] Dynamic pooling

Hi Jarek, Hopefully, this doesn't happen too often since the tables should be slowly changing. The solution here is to ask the contributors to rebase their PR more often. I'm happy that this is being caught now, and we don't have multiple heads as before :-) Cheers, Fokko Op ma 13 jan. 2020 om

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

Kaxil, I would not use a prefix, but a reserved key to have a single state field for the (dag, task, execution_date) tuple. When fetching the xcom values, we have to exclude the task that fetches these values. Daniel, Thanks for the scenario's. The first scenario could become idempotent, if you

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

irflow.apache.org > > > > > > > Subject: Re: [Discussion] In Prep for AIP: Stateful XComs and > > Poke > > > > > > > Rescheduling in Operators > > > > > > > > > > > > > > Just opening the Pandora's box :). >

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

already? (Would the architecture for Airflow > resilience be out-of-scope in this context because it involves more > complexity, like a Kafka cluster?) > > -- Darren > > > > On Wed, Jan 8, 2020 at 2:24 AM Jarek Potiuk > wrote: > > > Commented as well. I thi

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

Thanks Jacob for building the document. I think we're on the right track. I've added some comments and clarification to the document, to validate we're looking in the same direction. Would love to get more people's opinion on this. Cheers, Fokko Op wo 8 jan. 2020 om 03:31 schreef Jacob Ferriero :

Re: [VOTE] Add Probot Integrations to Airflow Github Repo

The reason why we stopped using mention-bot was that it was a hosted instance that wasn't maintained anymore. I really liked the mention-bot :-) Cheers, Fokko Op zo 5 jan. 2020 om 09:56 schreef Jarek Potiuk : > 👍💟 > > On Sun, Jan 5, 2020 at 4:38 AM Kaxil Naik wrote: > > > I have created a Githu

Re: Autolink References enabled for Airflow

Love it, thanks for sharing Kaxil! Cheers, Fokko Op ma 6 jan. 2020 om 14:11 schreef Kaxil Naik : > Hi all, > > A couple of days back I opened an Issue with Apache Infra to autolink Jira > references in commits: https://issues.apache.org/jira/browse/INFRA-19655 > > This is based on > > https://he

New Committer: Tomasz Urbaszek

Folks, The Project Management Committee (PMC) for Apache Airflow has invited Tomasz Urbaszek to become a committer and we are pleased to announce that he has accepted. He's super active in the community and active on both Slack and the devlist helping others. He has done an awesome job on moving

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

+1 for Github Actions. I've noticed that Spark is also using this: https://github.com/apache/spark/pull/26644 Cheers, Fokko Op di 10 dec. 2019 om 11:45 schreef Kamil Breguła : > +1 for Github Actions > > On Mon, Dec 9, 2019 at 11:49 PM Kaxil Naik wrote: > > > +1 for Github actions > > > > On M

Re: Apache Airflow welcomes Kengo Seki (sekikn) as committer & PMC member

Finally! Congrats Kengo! Cheers, Fokko Op ma 2 dec. 2019 om 20:19 schreef Kevin Yang : > Congrats Kengo! > > Cheers, > Kevin Y > > On Sun, Dec 1, 2019 at 5:49 PM hotmail wrote: > > > Congrats Kengo! > > > > > > Best Wish > > — Jiajie > > > > >

Re: New PMC members: Aizhamal and Kevin

Welcome Aizhamal and Kevin! Great to have you on board. Cheers, Fokko Op vr 22 nov. 2019 om 18:25 schreef Felix Uellendall > Congrats Kevin & Aizhamal! > > Sent from ProtonMail Mobile > > On Fri, Nov 22, 2019 at 18:21, Daniel Imberman > wrote: > > > Congrats!! > > > > via Newton Mail [ > https:

Re: [VOTE] Accept new Airflow website contribution

Big +1 (binding) from my side. The new site looks great :-) Op do 21 nov. 2019 om 18:54 schreef Aizhamal Nurmamat kyzy < aizha...@apache.org>: > I will leave this vote open for another day, as there are no -1s. Please > log your votes, and I'll close it by tomorrow this time. > > Thanks, > Aizham

Re: Drop Python 3.5 support?

+1 Thanks Jarek for the background information :-) Cheers, Fokko Op wo 13 nov. 2019 om 01:14 schreef Tao Feng > +1 > > On Tue, Nov 12, 2019 at 3:31 PM Jarek Potiuk > wrote: > > > Well. It seems then that my Buster change might also contain dropping 3.5 > > from Travis and switching to 3.6 eve

Re: [Final review] Airflow website + need your input to the web copy

I like the floating squares :-) Op za 9 nov. 2019 om 19:17 schreef Bas Harenslak < basharens...@godatadriven.com>: > I really like it (including front page animation)! Looks fresh & modern. > > There’s still some placeholders here and there so I won’t comment on those. > > Some notes on the parts

Re: [PROPOSAL] Migrate to Pytest

e proceed with official > >> vote for AIP-27 > >> < > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-27+Migrate+to+pytest > >>> > >> ? > >> > >> Best wishes, > >> Tomek > >> > &g

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Thanks Jarek for clearing that up. Personally I would omit the Apache one. We should not step into the fallacy as before with not being sure if it was in contrib or not. I would even consider merging software and protocols, as it not entirely clear what a protocol is or not. In the end, everything

Re: Apache Airflow 1.10.6 released!

Awesome, thanks Ash for doing the release! Cheers, Fokko Op di 29 okt. 2019 om 05:09 schreef Aizhamal Nurmamat kyzy < aizha...@apache.org>: > Thanks Ash and everyone who was involved! > > On Mon, Oct 28, 2019 at 3:16 PM Kevin Yang wrote: > > > Releases keeps getting smoothier 👍 And definitely l

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

246e0dc1, current schema Cheers, Fokko Op zo 27 okt. 2019 om 16:48 schreef Kaxil Naik : > As 2.0 is not released, if migration are broken, we can still fix them on > the mater itself. > > > > On Sun, Oct 27, 2019, 15:29 Driesprong, Fokko > wrote: > > > The problem

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

get much sleep) but I don't see how > broken code on master affects this release? > > -a > > > On 27 Oct 2019, at 14:28, Driesprong, Fokko > wrote: > > > > I'm having second thoughts on my +1 vote. > > > > It turns out that the database mig

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

ithub.com/apache/airflow/pull/6436/ - will review and > merge soon. But it's not a blocker IMHO. > > J. > > On Sat, Oct 26, 2019 at 2:10 AM Kaxil Naik wrote: > > > +1 (binding) > > > > On Sat, Oct 26, 2019 at 12:05 AM Driesprong, Fokko > > > w

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

+1 binding from my side Ran an example DAGs with Docker using Python 3.7. We might need to check why AIRFLOW-5746 is being reverted: https://github.com/apache/airflow/pull/6434 If there is another RC, I'd like to request to cherry-pick https://github.com/apache/airflow/pull/6370 onto the 1.10 br

Re: Can an xcom push and pull to the same task instance on the same execution date?

It hasn't been targeted for 1.10.6. If you want this in, you need to convince the release manager Ash :-) Cheers, Fokko Op vr 25 okt. 2019 om 22:46 schreef Bas Harenslak < basharens...@godatadriven.com>: > Hi Damian, > > Currently XComs cannot be shared between instances of the same task (i.e. >

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

we > could possibly still render the template in the webserver. For tasks that > have already run (those that have a TaskInstance row) maybe we could store > the rendered version as executed in there? Just thinking out loud here. > > -ash > > > > On 24 Oct 2019, at 10:56,

Re: [VOTE] Release 1.10.6 from 1.10.6rc1

+1 binding. - Ran an example DAG with the local executor under Python 3.7. - Checked the sha hashes: MacBook-Pro-van-Fokko:Downloads fokkodriesprong$ shasum -a 512 apache-airflow-1.10.6rc1-bin.tar.gz a4bde283b3f32fbc7a603da6deb3f015bda470b3dba1934a40b72212eba5391a9365400b8595d61afd27b2f8b6d9be220

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

changes if it hasn't run in > the > >> >> meantime. > >> >> > >> >> We can assign a unique number (version) to each row for a DAG. > >> >> Sorting by `last_updated_time` column who gives us how DAG updated > over > >> &

Re: Proposed roadmap for Airflow 2.0

gt; for > > how often we'll do future major version increments. As-is this might be > the > > first change to break backwards compat meaningfully in a while. > > > > On Mon, Oct 21, 2019 at 3:03 AM Driesprong, Fokko > > wrote: > > > > >

Re: Proposed roadmap for Airflow 2.0

Thanks Kevin, Kevin would love to have your input on this PR. This one tries to implement an async implementation of the operator, based on the sensor by Seelman. And also this one, which is required to mak

Re: New PMC Member: Jarek Potiuk

Well deserved Jarek and see you next week at ApacheCon! Cheers, Fokko Op za 19 okt. 2019 om 04:42 schreef Ishan Rastogi : > Congratulations Jarek :) > > On Sat, Oct 19, 2019 at 4:16 AM Jarek Potiuk > wrote: > > > Happy to become a PMC member for such awesome community and see all the > > congra

Re: [DISCUSS] Smarter test execution for CI (trivial changes without full tests)

Awesome Jarek, that was my fear as well. Good to hear that we still run the full test suite.\ Cheers, Fokko Op vr 18 okt. 2019 om 15:47 schreef Daniel Imberman < daniel.imber...@gmail.com>: > Oh then that sounds fine in that case :) > > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=d

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

Awesome Daniel! Cheers, Fokko Op wo 16 okt. 2019 om 20:16 schreef Kevin Yang : > Exciting! > > Daniel Imberman 于2019年10月16日 周三上午10:58写道： > > > Fixing the calendar link :) > > > > > https://calendar.google.com/event?action=TEMPLATE&tmeid=b2I4Yml2MWdhcXRtdGRtZmkzcXNxdmY5cThfMjAxOTEwMzBUMTUwMDAwWiB

Re: [PLEASE PARTICIPATE][AIP-11] UI phase of Airflow website

Thanks Kamil, Looks amazing. I really like the version from the meeting with Aizhamal; a bit more colorful. The gray boxes fade to the background. Cheers, Fokko Op wo 16 okt. 2019 om 19:17 schreef Chris Hendrixson : > Kamil & team, the site is really coming together! Great improvements > overal

Re: [PROPOSAL] Migrate to Pytest

+1 Op ma 14 okt. 2019 om 16:50 schreef Felix Uellendall : > +1, successfully using pytest for airflow plugins for almost 2 years now. > > Felix > > Sent from ProtonMail Mobile > > On Mon, Oct 14, 2019 at 16:39, Daniel Imberman < > dimberman.opensou...@gmail.com> wrote: > > > Huge +1 from me on th

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Big +1 from my side, looking forward to make this happen. Two sides that aren't completely clear to me: - Are we going to extend the existing data model, to allow the RDBMS to optimize queries on fields that we use a lot? - How are we going to do state evolution when we extend the JSON m

Re: Dockerhub issues

Thanks for the update Jarek. I know that UTC+1 is the leading timezone, but for calling it a day in SF would be a bit optimistic :-) Op di 15 okt. 2019 om 21:46 schreef Jarek Potiuk : > Seems that we have continuous DockerHub incident that impacts nearly > everyone > https://twitter.com/search?q

Re: AIP-21 - grouping google operators

I agree with Jarek's suggestion to add another module. Personally I don't like having too many on the airflow._ level, there are quite a few already. Also, I would also go for providers. Mostly because we have hook*s* and operator*s*. When there are a lot of {operators,hooks,sensors} it makes sens

Re: ApacheCon EU ticket

I already got a ticket, and attending the conference as well. Looking forward to meeting more Airflow folks! There are some interesting sessions, among others: https://aceu19.apachecon.com/session/its-breeze-develop-airflow Cheers, Fokko Op vr 4 okt. 2019 om 15:13 schreef Tomasz Urbaszek < tomas

Re: [AIP-11] Airflow Brandbook

Awesome! Love it Cheers, Fokko Op do 3 okt. 2019 om 13:18 schreef Ash Berlin-Taylor > Great! > > Could someone upload that to the logo file list space on the wiki? > > (Or I can do it tomorrow when I'm back at computer) > > -a > > On 3 October 2019 10:47:09 BST, "Kamil Breguła" > wrote: > >Hel

Re: Warsaw Apache Airflow Meetup

Awesome! I took the liberty of adding the meetup to the list: https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups Cheers, Fokko Op ma 23 sep. 2019 om 20:02 schreef Karolina Rosół < karolina.ro...@polidea.com>: > Exciting news! Can't wait to attend. > > Karolina Rosół > Polidea

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I'm not really in favor of the cross_transfer package. It sounds really technical, and if you're new to the project, I would not know what to expect in this package. We had something related in the past with Kubernetes, we solved like this: https://github.com/apache/airflow/blob/master/airflow/con

Re: [PLEASE PARTICIPATE][AIP-11] UX phase of Airflow website

Looks really great. I love the clean design. I agree with others that having news instead of blogs might be more appropriate. I'm fine with linking the blogs from there. Maybe we can change the meetup page with a division of Upcoming events and Past events? So it is easy to see which meetups are a

Re: Airflow 1.10.5 Python 2.7 compatability

okko, surprised to see you use 3.5 . I thought most people went straight >> to 3.6 as soon as it came out, as there are very little reasons to be on >> the 3 to 3.5 range anymore >> >> I'd say we should go straight to 3.6+ for Airflow as we deprecate 2.7 >> https:/

Re: Airflow 1.10.5 Python 2.7 compatability

e on > the 3 to 3.5 range anymore > > I'd say we should go straight to 3.6+ for Airflow as we deprecate 2.7 > https://pythonclock.org/ > > On Mon, Sep 16, 2019 at 2:17 PM Driesprong, Fokko > wrote: > > > Hi guys, > > > > While preparing a training,

Airflow 1.10.5 Python 2.7 compatability

Hi guys, While preparing a training, I think we have another incompatible dependency that screws up Python 2.7 compatibility: MacBook-Pro-van-Fokko:airspark fokkodriesprong$ kubectl logs airflow-deployment-f55b986f-5tmt7 webserver [2019-09-16 21:10:40,045] {settings.py:213} INFO - settings.confi

Re: Airflow node different versions

Hi John, I've never tried it like you're suggesting. It feels a bit risky. I can't tell what you will encounter if you run different versions of Airflow. How are you running Airflow? Cheers, Fokko Op do 12 sep. 2019 om 20:29 schreef John Smodic : > Hey all, > > I'm looking to find an upgrade pa

Re: [ANNOUNCE] Please welcome new Airflow committer Felix Uellendall

Congrats Felix, well deserved! Cheers, Fokko Op di 10 sep. 2019 om 10:48 schreef Karolina Rosół < karolina.ro...@polidea.com>: > Congratulations! > > Karolina Rosół > Polidea | Project Manager > > M: +48 606 630 236 <+48606630236> > E: karolina.ro...@polidea.com > [ima

Re: Codecov Integration

> > > J. > > On Wed, Sep 4, 2019 at 3:52 PM Jarek Potiuk > wrote: > > > Not yet. Will do soon :). > > > > J. > > > > > > On Wed, Sep 4, 2019 at 3:51 PM Driesprong, Fokko > > wrote: > > > >>

Re: Codecov Integration

> > > > On Wed, Aug 28, 2019 at 12:49 PM Kaxil Naik > wrote: > > > > > >> Looks like this commit: > > >> > > > https://github.com/apache/airflow/commit/8b41c3f12944c9b87fd04bd7137ba03d7e5a502f#diff-2eeaed663bd0d25b7e608891384b7298 > > c

Re: [VOTE] Release Apache Airflow 1.10.5rc1 as 1.10.5

Thanks Kaxil for running the release. Tested this with Docker using Python3 to run some examples. +1 (binding) Cheers, Fokko Op ma 2 sep. 2019 om 13:03 schreef Ash Berlin-Taylor : > Tested this with Celery and Kubernetes executors on py3. > > +1 (binding) > > -ash > > > On 31 Aug 2019, at 01:20

Re: Codecov Integration

14+139 Op wo 28 aug. 2019 om 13:34 schreef Driesprong, Fokko : > Hi Kaxil, > > I did notice that the coveralls library is gone, we had this before to > push it to codecov. I know that the names are different, and it is > confusing, I tried to remove it before: > https://github.

Re: Codecov Integration

Hi Kaxil, I did notice that the coveralls library is gone, we had this before to push it to codecov. I know that the names are different, and it is confusing, I tried to remove it before: https://github.com/apache/airflow/pull/3295 Let us see if it works again when reenabling this library: https:

Re: [VOTE] Change the Airflow logo

Some sharp lookin' logo's! My vote would be option 1 as well Cheers, Fokko Op wo 21 aug. 2019 om 14:15 schreef Ishan Rastogi : > Option 1 > > Cheers, > Ishan Rastogi > > On Wed, Aug 21, 2019 at 5:44 PM Philippe Gagnon > wrote: > > > Option 1 > > > > Thanks for the hard work Chris and Daniel! >

Re: airflow command not working

Hi Ullas, Without any logs or errors, it is really hard to see what's going on. Astronomer did publish a nice guide on how to run Airflow on Windows, that might help: https://www.astronomer.io/guides/airflow-wsl/ Cheers, Fokko Op wo 21 aug. 2019 om 09:03 schreef Ullas Raj.N : > hi , > i have in

Enable LGTM on the repository

Hi all, I recently bumped into LGTM , an automated vulnerability checker. Besides that, it also analyzes general code quality. I think it would be nice to enable this on Airflow as well. LGTM automatically runs 1600+ standard analyses contributed by researchers

Re: [DISCUSS] Tweaks to the Airflow logo

Great work Chris, love the clean design. Cheers, Fokko Op di 13 aug. 2019 om 13:06 schreef Kaxil Naik : > Love the new one. Nice work Chris. > > On Tue, Aug 13, 2019 at 12:03 PM Bas Harenslak < > basharens...@godatadriven.com> wrote: > > > I like it! Looks modern and finally that annoying stain

Re: Apache Airflow 1.10.4 released!

Awesome! Thanks Ash for all the effort of releasing 1.10.4! Cheers, Fokko Op di 6 aug. 2019 om 23:27 schreef Jarek Potiuk : > Well done! > > J. > > Principal Software Engineer > Phone: +48660796129 > > wt., 6 sie 2019, 23:06 użytkownik Aizhamal Nurmamat kyzy > napisał: > > > Thank you, Ash, for

Re: Random network failures @ Travis

Can you link the issues? Nothing to see here: - https://www.traviscistatus.com/ - https://twitter.com/traviscistatus - https://statusgator.com/services/travis-ci Cheers, Fokko Op di 6 aug. 2019 om 16:45 schreef Jarek Potiuk : > Again we have some random networking issues with Travis :(. > > Is

Re: [VOTE] Release Airflow 1.10.4 from RC5

Great for picking this up Ash. Tested locally in Docker, and looks good: +1 (binding) Cheers, Fokko Op di 6 aug. 2019 om 11:38 schreef Robin Edwards : > +1 (none-biding) > > Done a nightly run in production on python 3.7 with no issues. > > Thanks guys! > > R > > On Tue, 6 Aug 2019 at 09:31, Ja

Re: [VOTE] Changes in import paths

us. It would be very complex one if > we have it (and we can defer introducing it for later if needed especially > if it requires more than just renaming/moving but also architectural > changes - as Ash proposed for the transfer operators). But at least we can > decrease an entropy by group

Re: [VOTE] Changes in import paths

>> - > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> >>>>>>> *operators/ssh_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >&g

Re: Request permission to modify existing AIP

Hi Chen, You should have permissions now. Cheers, Fokko Op wo 31 jul. 2019 om 14:55 schreef Chen Tong : > Hi, > I'd like to modify my votes for AIP-21: Changes in import paths. Could you > help grand me the write permission? > Thanks! >

Re: Airflow DAG Serialisation

newly parsed. No big deal > there > > >> but just some thoughts on the top of my head and hopefully can be > helpful. > > >> > > >> And good idea on pre-rendering the template, believe template > rendering was > > >> the biggest

Re: Airflow DAG Serialisation

smart sensor API > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization > > > > > and > > seems to be working fine--a supporting case for ur proposal ;) There's a > > WIP > > PR <http

Re: [VOTE] Changes in import paths

cases - but I > > think in this case following "lasy consensus" - i,e. if noone objects to > > final set of cases, we will proceed with it.. > > > > Do you think that will work better ? > > > > J. > > > > Principal Software Engineer > >

Re: Airflow DAG Serialisation

Looks great Zhou, I have one thing that pops in my mind while reading the AIP; should keep the caching on the webserver level. As the famous quote goes: *"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton* Right now, the fundamental change

Re: [VOTE] Changes in import paths

7 Cases together, how >> >about >> >separate vote emails for all the cases? Each email can have the >> >description >> >copied over from the doc. >> > >> >It would probably be a bit easier to track a decision in the future as >> >well.

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Nice document Jarek. We should look at the pro's and con's regarding moving away from Travis. The process for Airflow, and also many other OSS projects, is to first develop on your local fork. If everything looks good, open a PR to the main repo. This reduces the noise we have on the project itsel

Re: [VOTE] Changes in import paths

Thanks Kamil for putting the document together. I wasn't able to respond earlier since I was giving Airflow workshops throughout Europe :-) *Case 1: *Solution A → Remove everything from contrib into a single package. To make it easier to the end-user, my preference would be to get rid of the cont

Re: [PROPOSE] Introduce and encourage pre-commit hooks framework to Airflow developer workflow

I agree with the other folks. I'm personally not a fan of pre commit hooks, but if people like it, they can enable it. Cheers, Fokko Op di 23 jul. 2019 om 20:13 schreef Kaxil Naik > It is fully-optional so I don't think we need an AIP there. > > > > On Tue, Jul 23, 2019 at 11:36 PM Beau Barker

Re: Travis CI random failures

I see issues at different Apache projects as well, Druid and Avro. They're running out of memory. Let's see how Travis responds. Cheers, Fokko Op di 23 jul. 2019 om 19:43 schreef Jarek Potiuk : > FYI. Still not fixed. Others experience this as well: > https://github.com/travis-ci/worker/issues/6

Re: Kind request to run static checks on your machines (for now at least)

Please note that you can also run all the tests on your private Travis fork. Please make sure that these test pass before opening a PR to the Airflow project. This allows you to run all the tests on the Travis CI, and make sure that Apache doesn't spend unnecessary CI time while iterating to get ev

Re: [VOTE] Release Airflow 1.10.4 from RC3

I've removed the run_duration and I'm in favor of removing num_runs as well. I've noticed that the service not always exit cleanly, and left the database in an inconsistent state. After restarting, some of the DagRuns would not be picked up. This should not happen, but not sure if the root cause is

Re: Airflow AIP permissions

Hi Kevin, Thanks for contributing back to Airflow :-) This minor change does not require an AIP. Feel free to open up a PR, and I'll take a look at it. I'm the original author of that part of the dataproc operator. Cheers, Fokko Op vr 19 jul. 2019 om 20:11 schreef Kevin Nguyen > Hi, > > I have

Re: Big change in the way we run tests (AIP-10 completed)

Great work Jarek! Finally we've removed Tox! Op wo 17 jul. 2019 om 22:54 schreef Jarek Potiuk : > Hello everyone, > > I would like to announce that I just merged the last (3/3) part of AIP-10 - > big change in the way we run CI tests for Airflow. > > Nothing changes in the tests, but the way we r

  1   2   >