Re: [DISCUSS] Custom XCom backends in core or not

> What is wrong with having some code which can be used by multiple users. There's nothing wrong with it. My main point about XCom backends is that it is not simply "other storage" than database. > I think instead of making it perfect on the first try, we can open it up for the community and let

Re: [DISCUSS] Custom XCom backends in core or not

My 2 cents on this: What is wrong with having some code which can be used by multiple users. What is the point in 10s of 100s of companies maintaining their own implementation of things. Isn't that we've separated providers from the core so that the priority of maintaining them wouldn't be the sam

Re: [DISCUSS] Custom XCom backends in core or not

I created a PR: https://github.com/apache/airflow/pull/12768 On Wed, Dec 2, 2020 at 8:03 PM Gerard Casas Saez wrote: > Fair enough. > > I would argue then to have a small example (like the one pandas to csv and > GCS) added to core as an example one maybe? Its fine to leave it in > examples fold

Re: [DISCUSS] Custom XCom backends in core or not

Fair enough. I would argue then to have a small example (like the one pandas to csv and GCS) added to core as an example one maybe? Its fine to leave it in examples folder otherwise Gerard Casas Saez Twitter | Cortex | @casassaez On Wed, Dec 2, 2020 at 11:51 AM Tom

Re: [DISCUSS] Custom XCom backends in core or not

I think it's rather hard to decouple the serialization and persisting. First of all, each data type (the input) may use a different serializer. Then (assuming 1-1 relation between data type and serializer) each serializer may require other persisting logic (dataframes to buckets, but jsons to redis

Re: [DISCUSS] Custom XCom backends in core or not

just a thought If flexibilty with storage / serialization is desired, perhaps this would make sense to be accomplished with methods on the backend class. So you could have an xcom backend class that has metheds like `push_dataframe` or `push_json` or something like that. And if you need flex

Re: [DISCUSS] Custom XCom backends in core or not

Hi folks! Reading the conversation, I agree w Tomek. At the same time I see value in adding some options out of the box for serialization and storage. I see there's a pattern here where we can decouple storage service (Redis, S3, GCS, Airflow DB...) and serialization format (pandas to csv, pickli

Re: [DISCUSS] Custom XCom backends in core or not

> Then you could have XComBackendSerializationBackend That's definitely something we should avoid... :D On Wed, Dec 2, 2020 at 6:18 PM Daniel Standish wrote: > > You could add xcom serialization utils in airflow.utils > > Then you could have XComBackendSerializationBackend ;)

Re: [DISCUSS] Custom XCom backends in core or not

You could add xcom serialization utils in airflow.utils Then you could have XComBackendSerializationBackend ;) >

Re: [DISCUSS] Custom XCom backends in core or not

I would agree with Tomek's suggestion in the beginning, i.e. to "*create better documentation around this feature (with examples and suggestions) instead of accepting XComs to code base.*". That may be a better balance to me personally. XD On Wed, Dec 2, 2020 at 5:21 PM Tomasz Urbaszek wrote:

Re: [DISCUSS] Custom XCom backends in core or not

> Shouldn't serialization be left to each custom backend? In my opinion - yes. That's why I'm not 100% convinced to have custom xcoms in core/providers. But if we decide to have them, then we have to decide on a serialization mechanism I think. On Wed, Dec 2, 2020 at 5:16 PM Daniel Standish wrot

Re: [DISCUSS] Custom XCom backends in core or not

Shouldn't serialization be left to each custom backend? On Wed, Dec 2, 2020, 8:11 AM Tomasz Urbaszek wrote: > Thanks Ry! > > > This will allow us to put scone forward as a strong feature rather than > how it has been historically portrayed as flawed/limited. > > This is a good point and I agree

Re: [DISCUSS] Custom XCom backends in core or not

Thanks Ry! > This will allow us to put scone forward as a strong feature rather than how it has been historically portrayed as flawed/limited. This is a good point and I agree that custom backends may increase Airflow competitiveness. However, if we decide to include them in core we need to answ

Re: [DISCUSS] Custom XCom backends in core or not

Ha “xcom” was autocorrected to “scone” on my phone, didn’t notice :) On Wed, Dec 2, 2020 at 10:22 AM Ry Walker wrote: > I’m in favor of including a few backends in core, including some that can > handle larger data, for the sake of Airflow usability and its competitive > positioning. > > This wi

Re: [DISCUSS] Custom XCom backends in core or not

I’m in favor of including a few backends in core, including some that can handle larger data, for the sake of Airflow usability and its competitive positioning. This will allow us to put scone forward as a strong feature rather than how it has been historically portrayed as flawed/limited. On We

[DISCUSS] Custom XCom backends in core or not

Hello all, Airflow 2.0 release is sooner and sooner. I would like to start a discussion about custom XCom backends. First of all, if you don't know it - since 1.10.12 users can use a custom XCom class that will override serialize and deserialize methods. Docs: https://airflow.apache.org/docs/sta