just a thought.... If flexibilty with storage / serialization is desired, perhaps this would make sense to be accomplished with methods on the backend class.
So you could have an xcom backend class that has metheds like `push_dataframe` or `push_json` or something like that. And if you need flexibility, you can add and use these kinds of methods. But the only issue is we would need to make the xcom "handler" available to the task. So instead of (*or in addition to, for backward compatibility*) calling `self.xcom_pull` from the operator, you would call `self.xcom.pull` --- so the handler class, with all its methods, is available. On Wed, Dec 2, 2020 at 9:55 AM Gerard Casas Saez <[email protected]> wrote: > Hi folks! > > Reading the conversation, I agree w Tomek. > At the same time I see value in adding some options out of the box for > serialization and storage. > > I see there's a pattern here where we can decouple storage service (Redis, > S3, GCS, Airflow DB...) and serialization format (pandas to csv, pickling, > json...). If we decouple them, then we can provide some options in core for > each, and provide options to configure them. > > something like: > > [xcom] > storage_layer = [airflow.xcom.GCS] > serialization = [airflow.xcom.panda2csv, airflow.xcom.jsondump] > > Where XCom layer would use GCS to store > (gs://bucket-name/{dag_id}/{dagrun_id}/{ti_id}/{key}) and then to serialize > it would try to use pandas2csv first (if class is pandas) and then json > dump otherwise. This could be extended to serialize as preferred (even > using GCS/S3 and loading again) and would allow for adding some options to > core while providing extensibility. > > What do you all think? > > > Gerard Casas Saez > Twitter | Cortex | @casassaez <http://twitter.com/casassaez> > > > On Wed, Dec 2, 2020 at 10:25 AM Tomasz Urbaszek <[email protected]> > wrote: > >> > Then you could have XComBackendSerializationBackend >> >> That's definitely something we should avoid... :D >> >> On Wed, Dec 2, 2020 at 6:18 PM Daniel Standish <[email protected]> >> wrote: >> > >> > You could add xcom serialization utils in airflow.utils >> > >> > Then you could have XComBackendSerializationBackend ;) >> >
