Re: [DISCUSS] Custom XCom backends in core or not

My 2 cents on this: What is wrong with having some code which can be used by multiple users. What is the point in 10s of 100s of companies maintaining their own implementation of things. Isn't that we've separated providers from the core so that the priority of maintaining them wouldn't be the sam

[Reminder] Airflow Issue Triage process call tomorrow

Hey everyone, I would like to remind whoever is interested to help in the Airflow Issue Triage process that we have our call tomorrow. *Date*: December 3rd *Time*: 8.30-9.30 AM Pacific / 4.30 PM GMT *Zoom link*: https://astronomer.zoom.us/j/95161397695?pwd=RXo1a1JHSTJTYU11SXlaNHRKT2hUQT09

Re: [DISCUSS] Custom XCom backends in core or not

I created a PR: https://github.com/apache/airflow/pull/12768 On Wed, Dec 2, 2020 at 8:03 PM Gerard Casas Saez wrote: > Fair enough. > > I would argue then to have a small example (like the one pandas to csv and > GCS) added to core as an example one maybe? Its fine to leave it in > examples fold

Re: AIP-9 Automated Dependency Management

Congrats! 😁🎉 Thanks Jarek 🙏 Sent from ProtonMail Mobile On Wed, Dec 2, 2020 at 20:55, Jarek Potiuk wrote: > Well. Actually, I've implemented it,, without evn knowing that I am doing > AIP-9 :). > > As of last week, we have fully-automated dependency management which > automatically upgrades

Re: AIP-9 Automated Dependency Management

Well. Actually, I've implemented it,, without evn knowing that I am doing AIP-9 :). As of last week, we have fully-automated dependency management which automatically upgrades to the latest "good" set of non-conflicting dependencies for Airlfow (both 2.0 and 1.10). Yet another AIP completed. I wi

Re: AIP-9 Automated Dependency Management

Sounds good to me. Actually, don’t we have the dependabot running already? Or did we disable it? Either way I would be fine if we change the status to done/close. Felix Sent from ProtonMail Mobile On Wed, Dec 2, 2020 at 20:13, Kamil Breguła wrote: > What is the status of this AIP? > > I thin

Re: AIP-9 Automated Dependency Management

What is the status of this AIP? I think it is done because we have constraints files. > In order to have repeatable installation, however, starting from Airflow 1.10.10 and updated in Airflow 1.10.13 we also keep a set of "known-to-be-working" constraint files in the constraints-master and constr

Re: [DISCUSS] Custom XCom backends in core or not

Fair enough. I would argue then to have a small example (like the one pandas to csv and GCS) added to core as an example one maybe? Its fine to leave it in examples folder otherwise Gerard Casas Saez Twitter | Cortex | @casassaez On Wed, Dec 2, 2020 at 11:51 AM Tom

Re: [DISCUSS] Custom XCom backends in core or not

I think it's rather hard to decouple the serialization and persisting. First of all, each data type (the input) may use a different serializer. Then (assuming 1-1 relation between data type and serializer) each serializer may require other persisting logic (dataframes to buckets, but jsons to redis

Re: [DISCUSS] Custom XCom backends in core or not

just a thought If flexibilty with storage / serialization is desired, perhaps this would make sense to be accomplished with methods on the backend class. So you could have an xcom backend class that has metheds like `push_dataframe` or `push_json` or something like that. And if you need flex

Re: [DISCUSS] Custom XCom backends in core or not

Hi folks! Reading the conversation, I agree w Tomek. At the same time I see value in adding some options out of the box for serialization and storage. I see there's a pattern here where we can decouple storage service (Redis, S3, GCS, Airflow DB...) and serialization format (pandas to csv, pickli

Re: [DISCUSS] Custom XCom backends in core or not

> Then you could have XComBackendSerializationBackend That's definitely something we should avoid... :D On Wed, Dec 2, 2020 at 6:18 PM Daniel Standish wrote: > > You could add xcom serialization utils in airflow.utils > > Then you could have XComBackendSerializationBackend ;)

Re: [DISCUSS] Custom XCom backends in core or not

You could add xcom serialization utils in airflow.utils Then you could have XComBackendSerializationBackend ;) >

Re: [DISCUSS] Custom XCom backends in core or not

I would agree with Tomek's suggestion in the beginning, i.e. to "*create better documentation around this feature (with examples and suggestions) instead of accepting XComs to code base.*". That may be a better balance to me personally. XD On Wed, Dec 2, 2020 at 5:21 PM Tomasz Urbaszek wrote:

Re: [DISCUSS] Custom XCom backends in core or not

> Shouldn't serialization be left to each custom backend? In my opinion - yes. That's why I'm not 100% convinced to have custom xcoms in core/providers. But if we decide to have them, then we have to decide on a serialization mechanism I think. On Wed, Dec 2, 2020 at 5:16 PM Daniel Standish wrot

Re: Airflow Feature Design Feedback

Hi Ace, Any update on this one? I think there's a Github issue proposing similar functionality: https://github.com/apache/airflow/issues/12199 It would be good to coordinate the effort. Best, Tomek On 2020/10/18 13:09:01, Kaxil Naik wrote: > The Log table can be used at Dag Level too if we p

Re: New PIP and Airflow 1.10.*

Unfortunately - while it seems that the constraint fixes I implemented last week solve the problems that I observed previously, some of our dependencies are still not ready to work with the new resolver: ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/

Re: [DISCUSS] Custom XCom backends in core or not

Shouldn't serialization be left to each custom backend? On Wed, Dec 2, 2020, 8:11 AM Tomasz Urbaszek wrote: > Thanks Ry! > > > This will allow us to put scone forward as a strong feature rather than > how it has been historically portrayed as flawed/limited. > > This is a good point and I agree

Re: [DISCUSS] Custom XCom backends in core or not

Thanks Ry! > This will allow us to put scone forward as a strong feature rather than how it has been historically portrayed as flawed/limited. This is a good point and I agree that custom backends may increase Airflow competitiveness. However, if we decide to include them in core we need to answ

Re: [DISCUSS] Custom XCom backends in core or not

Ha “xcom” was autocorrected to “scone” on my phone, didn’t notice :) On Wed, Dec 2, 2020 at 10:22 AM Ry Walker wrote: > I’m in favor of including a few backends in core, including some that can > handle larger data, for the sake of Airflow usability and its competitive > positioning. > > This wi

Re: [DISCUSS] Custom XCom backends in core or not

I’m in favor of including a few backends in core, including some that can handle larger data, for the sake of Airflow usability and its competitive positioning. This will allow us to put scone forward as a strong feature rather than how it has been historically portrayed as flawed/limited. On We

[DISCUSS] Custom XCom backends in core or not

Hello all, Airflow 2.0 release is sooner and sooner. I would like to start a discussion about custom XCom backends. First of all, if you don't know it - since 1.10.12 users can use a custom XCom class that will override serialize and deserialize methods. Docs: https://airflow.apache.org/docs/sta

Re: New PIP and Airflow 1.10.*

Ahh! good one :). On Wed, Dec 2, 2020 at 5:34 AM Philippe Gagnon wrote: > There is also an option to force usage of the old dependency resolver: > > --use-deprecated=legacy-resolver > > But hopefully we find a more sustainable solution going forward! > > BR > > Philippe > > On Tue, Dec 1, 2020 a