[jira] [Commented] (AIRFLOW-1033) TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958795#comment-15958795 ] ASF subversion and git services commented on AIRFLOW-1033: -- Commit fbcbd053a2c5fffb0c95eb55a91cb92fa860e1ab in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fbcbd05 ] [AIRFLOW-1033][AIFRLOW-1033] Fix ti_deps for no schedule dags DAGs that did not have a schedule (None or @once) make the dependency checker raise an exception as the previous schedule will not exist. Also activates all ti_deps tests. Closes #2220 from bolkedebruin/AIRFLOW-1033 > TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py > > > Key: AIRFLOW-1033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1033 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.1 > Environment: Centos 7; > db: PostgreSQL 9.5 > python version: 2.7 > Installation via pip >Reporter: Bert Desmet >Priority: Blocker > Labels: bug, interval > Fix For: 1.8.1 > > Attachments: test_dag.py, test_dag.py.log > > > Dear, > When starting a specific new dag we get the following error: > [2017-03-23 16:51:16,354] {jobs.py:354} DagFileProcessor908 ERROR - Got an > exception! Propagating... > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 346, in helper > pickle_dags) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1581, in > process_file > self._process_dags(dagbag, dags, ti_keys_to_schedule) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1174, in > _process_dags > self._process_task_instances(dag, tis_out) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 905, in > _process_task_instances > session=session): > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1116, in > are_dependencies_met > session=session): > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1140, in > get_failed_dep_statuses > dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line > 94, in get_dep_statuses > for dep_status in self._get_dep_statuses(ti, session, dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/prev_dagrun_dep.py", > line 47, in _get_dep_statuses > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > TypeError: can't compare datetime.datetime to NoneType > I have added some debug code to the file 'prev_dagrun_dep.py: > dag = ti.task.dag > print 'Start dates:' > print 'previous_exection_date: > %s'%(dag.previous_schedule(ti.execution_date)) > print 'current start date: %s'%(ti.task.start_date) > if dag.catchup: > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > And this is the output I get: > Start dates: > previous_exection_date: None > current start date: 2017-03-19 00:00:00 > I think it is normall that the previous_exection_date is null, since it is > the first time this dag is being run. But why is the start_date of the dag > important, and not the start date of the run? > I have the feeling the cause is the 'schedule_interval', which is set to > None. > Please find an example and it's log file as an attachment to this mail. > Bert -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1033) TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958796#comment-15958796 ] ASF subversion and git services commented on AIRFLOW-1033: -- Commit fbcbd053a2c5fffb0c95eb55a91cb92fa860e1ab in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fbcbd05 ] [AIRFLOW-1033][AIFRLOW-1033] Fix ti_deps for no schedule dags DAGs that did not have a schedule (None or @once) make the dependency checker raise an exception as the previous schedule will not exist. Also activates all ti_deps tests. Closes #2220 from bolkedebruin/AIRFLOW-1033 > TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py > > > Key: AIRFLOW-1033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1033 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.1 > Environment: Centos 7; > db: PostgreSQL 9.5 > python version: 2.7 > Installation via pip >Reporter: Bert Desmet >Priority: Blocker > Labels: bug, interval > Fix For: 1.8.1 > > Attachments: test_dag.py, test_dag.py.log > > > Dear, > When starting a specific new dag we get the following error: > [2017-03-23 16:51:16,354] {jobs.py:354} DagFileProcessor908 ERROR - Got an > exception! Propagating... > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 346, in helper > pickle_dags) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1581, in > process_file > self._process_dags(dagbag, dags, ti_keys_to_schedule) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1174, in > _process_dags > self._process_task_instances(dag, tis_out) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 905, in > _process_task_instances > session=session): > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1116, in > are_dependencies_met > session=session): > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1140, in > get_failed_dep_statuses > dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line > 94, in get_dep_statuses > for dep_status in self._get_dep_statuses(ti, session, dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/prev_dagrun_dep.py", > line 47, in _get_dep_statuses > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > TypeError: can't compare datetime.datetime to NoneType > I have added some debug code to the file 'prev_dagrun_dep.py: > dag = ti.task.dag > print 'Start dates:' > print 'previous_exection_date: > %s'%(dag.previous_schedule(ti.execution_date)) > print 'current start date: %s'%(ti.task.start_date) > if dag.catchup: > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > And this is the output I get: > Start dates: > previous_exection_date: None > current start date: 2017-03-19 00:00:00 > I think it is normall that the previous_exection_date is null, since it is > the first time this dag is being run. But why is the start_date of the dag > important, and not the start date of the run? > I have the feeling the cause is the 'schedule_interval', which is set to > None. > Please find an example and it's log file as an attachment to this mail. > Bert -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1033) TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958798#comment-15958798 ] ASF subversion and git services commented on AIRFLOW-1033: -- Commit ebfc3ea73ae1ffe273e4ff532f1ad47441bef518 in incubator-airflow's branch refs/heads/v1-8-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ebfc3ea ] [AIRFLOW-1033][AIFRLOW-1033] Fix ti_deps for no schedule dags DAGs that did not have a schedule (None or @once) make the dependency checker raise an exception as the previous schedule will not exist. Also activates all ti_deps tests. Closes #2220 from bolkedebruin/AIRFLOW-1033 (cherry picked from commit fbcbd053a2c5fffb0c95eb55a91cb92fa860e1ab) Signed-off-by: Bolke de Bruin> TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py > > > Key: AIRFLOW-1033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1033 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.1 > Environment: Centos 7; > db: PostgreSQL 9.5 > python version: 2.7 > Installation via pip >Reporter: Bert Desmet >Priority: Blocker > Labels: bug, interval > Fix For: 1.8.1 > > Attachments: test_dag.py, test_dag.py.log > > > Dear, > When starting a specific new dag we get the following error: > [2017-03-23 16:51:16,354] {jobs.py:354} DagFileProcessor908 ERROR - Got an > exception! Propagating... > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 346, in helper > pickle_dags) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1581, in > process_file > self._process_dags(dagbag, dags, ti_keys_to_schedule) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1174, in > _process_dags > self._process_task_instances(dag, tis_out) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 905, in > _process_task_instances > session=session): > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1116, in > are_dependencies_met > session=session): > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1140, in > get_failed_dep_statuses > dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line > 94, in get_dep_statuses > for dep_status in self._get_dep_statuses(ti, session, dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/prev_dagrun_dep.py", > line 47, in _get_dep_statuses > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > TypeError: can't compare datetime.datetime to NoneType > I have added some debug code to the file 'prev_dagrun_dep.py: > dag = ti.task.dag > print 'Start dates:' > print 'previous_exection_date: > %s'%(dag.previous_schedule(ti.execution_date)) > print 'current start date: %s'%(ti.task.start_date) > if dag.catchup: > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > And this is the output I get: > Start dates: > previous_exection_date: None > current start date: 2017-03-19 00:00:00 > I think it is normall that the previous_exection_date is null, since it is > the first time this dag is being run. But why is the start_date of the dag > important, and not the start date of the run? > I have the feeling the cause is the 'schedule_interval', which is set to > None. > Please find an example and it's log file as an attachment to this mail. > Bert -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1033) TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958797#comment-15958797 ] ASF subversion and git services commented on AIRFLOW-1033: -- Commit ebfc3ea73ae1ffe273e4ff532f1ad47441bef518 in incubator-airflow's branch refs/heads/v1-8-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ebfc3ea ] [AIRFLOW-1033][AIFRLOW-1033] Fix ti_deps for no schedule dags DAGs that did not have a schedule (None or @once) make the dependency checker raise an exception as the previous schedule will not exist. Also activates all ti_deps tests. Closes #2220 from bolkedebruin/AIRFLOW-1033 (cherry picked from commit fbcbd053a2c5fffb0c95eb55a91cb92fa860e1ab) Signed-off-by: Bolke de Bruin> TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py > > > Key: AIRFLOW-1033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1033 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.1 > Environment: Centos 7; > db: PostgreSQL 9.5 > python version: 2.7 > Installation via pip >Reporter: Bert Desmet >Priority: Blocker > Labels: bug, interval > Fix For: 1.8.1 > > Attachments: test_dag.py, test_dag.py.log > > > Dear, > When starting a specific new dag we get the following error: > [2017-03-23 16:51:16,354] {jobs.py:354} DagFileProcessor908 ERROR - Got an > exception! Propagating... > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 346, in helper > pickle_dags) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1581, in > process_file > self._process_dags(dagbag, dags, ti_keys_to_schedule) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1174, in > _process_dags > self._process_task_instances(dag, tis_out) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 905, in > _process_task_instances > session=session): > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1116, in > are_dependencies_met > session=session): > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1140, in > get_failed_dep_statuses > dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line > 94, in get_dep_statuses > for dep_status in self._get_dep_statuses(ti, session, dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/prev_dagrun_dep.py", > line 47, in _get_dep_statuses > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > TypeError: can't compare datetime.datetime to NoneType > I have added some debug code to the file 'prev_dagrun_dep.py: > dag = ti.task.dag > print 'Start dates:' > print 'previous_exection_date: > %s'%(dag.previous_schedule(ti.execution_date)) > print 'current start date: %s'%(ti.task.start_date) > if dag.catchup: > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > And this is the output I get: > Start dates: > previous_exection_date: None > current start date: 2017-03-19 00:00:00 > I think it is normall that the previous_exection_date is null, since it is > the first time this dag is being run. But why is the start_date of the dag > important, and not the start date of the run? > I have the feeling the cause is the 'schedule_interval', which is set to > None. > Please find an example and it's log file as an attachment to this mail. > Bert -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1033) TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939742#comment-15939742 ] Ruslan Dautkhanov commented on AIRFLOW-1033: Might be related to AIRFLOW-1001 > TypeError: can't compare datetime.datetime to NoneType in prev_dagrun_dep.py > > > Key: AIRFLOW-1033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1033 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: Airflow 1.8 > Environment: Centos 7; > db: PostgreSQL 9.5 > python version: 2.7 > Installation via pip >Reporter: Bert Desmet >Priority: Critical > Labels: bug, interval > Attachments: test_dag.py, test_dag.py.log > > > Dear, > When starting a specific new dag we get the following error: > [2017-03-23 16:51:16,354] {jobs.py:354} DagFileProcessor908 ERROR - Got an > exception! Propagating... > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 346, in helper > pickle_dags) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1581, in > process_file > self._process_dags(dagbag, dags, ti_keys_to_schedule) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 1174, in > _process_dags > self._process_task_instances(dag, tis_out) > File "/usr/lib/python2.7/site-packages/airflow/jobs.py", line 905, in > _process_task_instances > session=session): > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in > wrapper > result = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1116, in > are_dependencies_met > session=session): > File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1140, in > get_failed_dep_statuses > dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line > 94, in get_dep_statuses > for dep_status in self._get_dep_statuses(ti, session, dep_context): > File > "/usr/lib/python2.7/site-packages/airflow/ti_deps/deps/prev_dagrun_dep.py", > line 47, in _get_dep_statuses > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > TypeError: can't compare datetime.datetime to NoneType > I have added some debug code to the file 'prev_dagrun_dep.py: > dag = ti.task.dag > print 'Start dates:' > print 'previous_exection_date: > %s'%(dag.previous_schedule(ti.execution_date)) > print 'current start date: %s'%(ti.task.start_date) > if dag.catchup: > if dag.previous_schedule(ti.execution_date) < ti.task.start_date: > And this is the output I get: > Start dates: > previous_exection_date: None > current start date: 2017-03-19 00:00:00 > I think it is normall that the previous_exection_date is null, since it is > the first time this dag is being run. But why is the start_date of the dag > important, and not the start date of the run? > I have the feeling the cause is the 'schedule_interval', which is set to > None. > Please find an example and it's log file as an attachment to this mail. > Bert -- This message was sent by Atlassian JIRA (v6.3.15#6346)