I’m in favor of including a few backends in core, including some that can handle larger data, for the sake of Airflow usability and its competitive positioning.
This will allow us to put scone forward as a strong feature rather than how it has been historically portrayed as flawed/limited. On Wed, Dec 2, 2020 at 9:49 AM Tomasz Urbaszek <turbas...@apache.org> wrote: > Hello all, > > Airflow 2.0 release is sooner and sooner. I would like to start a > discussion about custom XCom backends. > > First of all, if you don't know it - since 1.10.12 users can use a > custom XCom class that will override serialize and deserialize > methods. Docs: > https://airflow.apache.org/docs/stable/concepts.html#custom-xcom-backend > > This feature allows users the following things: > - reduce boilerplate code responsible for downloading / uploading data > in operators (it's handled by custom XCom) > - use different storage for XCom data (other database, buckets, cache etc.) > - verifying XCom data on read/write operations > - and anything else that may be feasible > > Some examples: > https://github.com/apache/airflow/pull/12733 > > https://www.polidea.com/blog/airflow-2-0-dag-authoring-redesigned/#custom-xcom-backends-8560 > > The point I want to raise (as I did in this PR > https://github.com/apache/airflow/pull/12733) is to discuss if we as a > community want to have custom XComs in our codebase (core or > providers). I'm happy to hear what the community thinks about it? > > From my side, I'm leaning toward creating better documentation around > this feature (with examples and suggestions) instead of accepting > XComs to code base. My main concern is that custom XComs are easy to > write (using for example hooks) and will work best when they are built > to suit exact users' needs. On the other hand, I see some potential in > "low level" XComs that just implement logic of storing and retrieving > data from particular storage. But anything that gets too use-case / > data type specific should not be accepted. > > Cheers, > Tomek > -- Sent from Gmail Mobile
