In current releases xcom is deleted at task start. Recently a PR was merged <https://github.com/apache/airflow/pull/6370> to not delete until XCOM is again pushed
I am not sure if this has made it into 1.10.6rc After this change, I think what you are trying to do would work. But it's interesting I wonder if this could screw up some processes, probably unlikely. I think it would be nice too add some more power to our XCOM methods. Also would not be hard to do. Like strictly less than vs less than (re execution date). Because of the way execution date is interval minus one, this can screw you over if you are mixing triggered dags and scheduled. Sometimes "last" xcom (by exec date) will not truly "last" xcom. This could be resolved by the interval edge PR <https://github.com/apache/airflow/pull/5787> (so that exec date would be run date for both triggered and scheduled dags). Another way to resolve would be to add option to sort by `timestamp` instead of `execution_date` in xcom pull. On Fri, Oct 25, 2019 at 9:09 AM Shaw, Damian P. < damian.sha...@credit-suisse.com> wrote: > Hi all, > > I am trying to pull the latest xcom value that could include the current > task from a previous retry. In the UI I can see today's xcom value and > yesterday's xcom value. I am retrying the task and unable to get today's > value: > > Attempt 1: context['ti'].xcom_pull(key=key, include_prior_dates=True) -> > Yesterday's xcom value > Attempt 2: context['ti'].xcom_pull(key=key) -> None > Attempt 3: XCom.get_many(key=key, > execution_date=context['execution_date'] + timedelta(days=1), > include_prior_dates=True) -> List containing yesterday's xcom object > > It occurred to me at this point that maybe what I'm trying isn't possible > with xcom, does anyone know? > > (FYI I'm on Airflow 1.10.3 but I've looked at the source code between this > and master and seen no significant difference.) > Thanks, > Damian > > > =============================================================================== > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > =============================================================================== > >
