Alex Papanicolaou created AIRFLOW-62:
----------------------------------------
Summary: XCom push not working reliably
Key: AIRFLOW-62
URL: https://issues.apache.org/jira/browse/AIRFLOW-62
Project: Apache Airflow
Issue Type: Bug
Components: db, operators
Affects Versions: Airflow 1.7.0
Environment: Postgres backed Airflow running with Celery inside of the
puckel Docker setup.
Reporter: Alex Papanicolaou
I have a DAG that polls for activity in various data streams from a database
and then uploads the activity statuses to a table. Each of the polling tasks
are python operators that once they get the polling result, return a dict as an
XCom push. The dict contains two entries which are strings, one which is a
bool, and one which is a datetime object. There is a final task that pulls all
the results and uploads the collective statuses to a table. I chose this
pattern since I figured it might be better to do one collective write operation
on all the results.
Before I moved ahead to the github master branch I was using 1.7.0 from PyPI
and this worked fine. Now that I am on the github master branch, I find that
the XCom pushing is unreliable. The returned values in the logs show up
correctly but when doing the XCom pull, I get None for some of the returned
values. Investigating the XCom result in the Webserver also shows nothing
there. But if I rerun a task where the XCom failed, the push works and the
XCom result is as it should be.
Nothing appears to have changed in the codebase so I am at a loss. Perhaps it
really wasn't working before? How would the backing postgres handle these
simultaneous writes? I can't imagine that would be a problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
