uranusjr commented on issue #37810: URL: https://github.com/apache/airflow/issues/37810#issuecomment-1982982809
Since XCom is just a data storage, it can be used like an external S3 file, or a database the user sets up. It is just a bit more automated and contains some metadata. I feel it is reasonable to assign a dataset URI to each key-value pair, so a dataset event is triggered when a key-value pair is written. This makes cross-DAG XCom usage more useful IMO since it allows a downstream DAG to declare dependency at the DAG level (via dataset) to the upstream. With that established, if we store extra metadata (of a dataset), it only makes sense to allow extra metadata also when an XCom is written. But if we use XCom for the extra, writing to XCom would write… extra metadata to XCom? And does that extra metadata also has a dataset URI and can have extra extra metadata? It becomes awkward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
