Fokko commented on issue #6370: AIRFLOW-5701: Don't clear xcom explicitly before execution URL: https://github.com/apache/airflow/pull/6370#issuecomment-547280956 > Unless I switch a pickle flag it complains that it is not JSON serializable, the comments in the code indicate pickling is going away for Airflow 2.0. I'm trying to ease my migration ahead of time :). @notatallshaw I think we should be able to serialize datetimes, could you open a ticket for this? @KevinYang21 No worries, I'd rather have a late reply than no reply. This has been reverted anyway and I love your input on it. > XCOM was designed for inter-task communication and should stay in that way. This is obvious since before there wasn't really a need to have inter-task communication. Let me say that I'm not trying to break idempotency, and we should have it by design. However, for efficiency, you might want to cache some state, and this can still be idempotent. My case is that it just feels awkward to create a second table that does exactly the same as xcom. What would your opinion on adding an `include_self` option to the `include_prior_dates` method? Any I was thinking of adding a column to xcom, called `persistent`, which indicates that it shouldn't be cleared. It is evident that I did not think this through, and did not cover all the edge cases. I'll try to open up some PR's in the upcoming days to at least improve on atomicity and make changes in small steps, I'll include you on the PR's as well.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
