[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132976#comment-17132976 ] ASF GitHub Bot commented on AIRFLOW-6704: - yuqian90 commented on pull request #7324: URL: https://github.com/apache/airflow/pull/7324#issuecomment-642451211 > @yuqian90 : Thank You ! for your response. > If so be the case then just wondering the same code with `execution_date` works perfectly fine with the earlier version of Airflow that we are using i.e. v1.10.2. If the `dagrun_operator` construct changed, I could not get any literature about that. Many things changed between 10.2 and 10.10. I haven't looked too carefully into the error you have. I can't tell if this PR is related. Any reason you think this PR is the cause? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132875#comment-17132875 ] ASF GitHub Bot commented on AIRFLOW-6704: - shanit-saha commented on pull request #7324: URL: https://github.com/apache/airflow/pull/7324#issuecomment-642381084 @yuqian90 : Thank You ! for your response. If so be the case then just wondering the same code with `execution_date` works perfectly fine with the earlier version of Airflow that we are using i.e. v1.10.2. If the `dagrun_operator` construct changed, I could not get any literature about that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132817#comment-17132817 ] ASF GitHub Bot commented on AIRFLOW-6704: - yuqian90 commented on pull request #7324: URL: https://github.com/apache/airflow/pull/7324#issuecomment-642350478 Hi @shanit-saha > execution_date=datetime.now() I think this line is the cause of the problem. I believe what you actually want to express is to trigger `dag_process_pos` with `execution_date` equal to the datetime **at the time of executing** `TriggerDagRunOperator`. However, doing `execution_date=datetime.now()` at the time of constructing `TriggerDagRunOperator` means you are triggering `dag_process_pos` at the time of **parsing** the DAG that contains the `TriggerDagRunOperator`. Since the value of `execution_date` keeps changing each time the DAG is parsed, the result is some kind of unexpected errors. I think you already realized if you leave `execution_date` as None, `TriggerDagRunOperator` defaults to `timezone.utcnow().isoformat()` which does what you need. Alternatively you can also set `execution_date` to a jinja template str yourself. The gist is that you should not have an argument that keeps changing whenever the DAG is parsed. If you think the doc of `TriggerDagRunOperator` needs to be improved. Pls create an issue for that separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130598#comment-17130598 ] ASF GitHub Bot commented on AIRFLOW-6704: - shanit-saha commented on pull request #7324: URL: https://github.com/apache/airflow/pull/7324#issuecomment-641965564 To add to my above post. It has so been found that when we remove `execution_date` parameter from the `TriggerDagRunOperator` it works. Is that a problem or change in the syntax. Also the Airflow-Dag examples provided with Airflow1.10.10 repository does not have a sample code for `TriggerDagOperator` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129665#comment-17129665 ] ASF GitHub Bot commented on AIRFLOW-6704: - shanit-saha edited a comment on pull request #7324: URL: https://github.com/apache/airflow/pull/7324#issuecomment-641433495 On Migrating Airflow from V1.10.2 to V1.10.10 One of our DAG have a task which is of dagrun_operator type. Code snippet of the task looks something as below. Please assume that DAG `dag_process_pos` exists ``` task_trigger_dag_positional = TriggerDagRunOperator( trigger_dag_id="dag_process_pos", python_callable=set_up_dag_run_preprocessing, task_id="trigger_preprocess_dag", on_failure_callback=log_failure, execution_date=datetime.now(), provide_context=False, owner='airflow') def set_up_dag_run_preprocessing(context, dag_run_obj): ti = context['ti'] dag_name = context['ti'].task.trigger_dag_id dag_run = context['dag_run'] trans_id = dag_run.conf['transaction_id'] routing_info = ti.xcom_pull(task_ids="json_validation", key="route_info") new_file_path = routing_info['file_location'] new_file_name = os.path.basename(routing_info['new_file_name']) file_path = os.path.join(new_file_path, new_file_name) batch_id = "123-AD-FF" dag_run_obj.payload = {'inputfilepath': file_path, 'transaction_id': trans_id, 'Id': batch_id} ``` The DAG runs all fine. In fact the python callable of the task mentioned until the last line. Then it errors out. ``` [2020-06-09 11:36:22,838] {taskinstance.py:1145} ERROR - No row was found for one() Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.6/site-packages/airflow/operators/dagrun_operator.py", line 95, in execute replace_microseconds=False) File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/trigger_dag.py", line 141, in trigger_dag replace_microseconds=replace_microseconds, File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/trigger_dag.py", line 98, in _trigger_dag external_trigger=True, File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/models/dag.py", line 1471, in create_dagrun run.refresh_from_db() File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/models/dagrun.py", line 109, in refresh_from_db DR.run_id == self.run_id File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3446, in one raise orm_exc.NoResultFound("No row was found for one()") sqlalchemy.orm.exc.NoResultFound: No row was found for one() ``` After which the `on_failure_callback` of that task is executed and all code of that callable runs perfectly ok as is expected. The query here is why did the dagrun_operator fail after the python callable. **P.S** : The DAG that is being triggered by the `TriggerDagRunOperator` , in this case `dag_process_pos` starts with task of type`dummy_operator` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129613#comment-17129613 ] ASF GitHub Bot commented on AIRFLOW-6704: - shanit-saha commented on pull request #7324: URL: https://github.com/apache/airflow/pull/7324#issuecomment-641433495 On Migrating Airflow from V1.10.2 to V1.10.10 One of our DAG have a task which is of dagrun_operator type. Code snippet of the task looks something as below. Please assume that DAG `dag_process_pos` exists ``` task_trigger_dag_positional = TriggerDagRunOperator( trigger_dag_id="dag_process_pos", python_callable=set_up_dag_run_preprocessing, task_id="trigger_preprocess_dag", on_failure_callback=log_failure, execution_date=datetime.now(), provide_context=False, owner='airflow') def set_up_dag_run_preprocessing(context, dag_run_obj): ti = context['ti'] dag_name = context['ti'].task.trigger_dag_id dag_run = context['dag_run'] trans_id = dag_run.conf['transaction_id'] routing_info = ti.xcom_pull(task_ids="json_validation", key="route_info") new_file_path = routing_info['file_location'] new_file_name = os.path.basename(routing_info['new_file_name']) file_path = os.path.join(new_file_path, new_file_name) batch_id = "123-AD-FF" dag_run_obj.payload = {'inputfilepath': file_path, 'transaction_id': trans_id, 'Id': batch_id} ``` The DAG runs all fine. In fact the python callable of the task mentioned until the last line. Then it errors out. ``` [2020-06-09 11:36:22,838] {taskinstance.py:1145} ERROR - No row was found for one() Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.6/site-packages/airflow/operators/dagrun_operator.py", line 95, in execute replace_microseconds=False) File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/trigger_dag.py", line 141, in trigger_dag replace_microseconds=replace_microseconds, File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/trigger_dag.py", line 98, in _trigger_dag external_trigger=True, File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/models/dag.py", line 1471, in create_dagrun run.refresh_from_db() File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/models/dagrun.py", line 109, in refresh_from_db DR.run_id == self.run_id File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3446, in one raise orm_exc.NoResultFound("No row was found for one()") sqlalchemy.orm.exc.NoResultFound: No row was found for one() ``` After which the `on_failure_callback` of that task is executed and all code of that callable runs perfectly ok as is expected. The query here is why did the dagrun_operator fail after the python callable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064477#comment-17064477 ] ASF subversion and git services commented on AIRFLOW-6704: -- Commit 7e9235096d9b7120ecacc22d660dcc4696bf71a9 in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7e92350 ] [AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324) cherry-picked from 3ef711806 > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064470#comment-17064470 ] ASF subversion and git services commented on AIRFLOW-6704: -- Commit ca3018800fa2a41541a3671f74a97abac854f1f0 in airflow's branch refs/heads/v1-10-test from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ca30188 ] [AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324) cherry-picked from 3ef711806 > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041846#comment-17041846 ] ASF subversion and git services commented on AIRFLOW-6704: -- Commit 3ef71180658bc79566aa00a94bd96adef74b605c in airflow's branch refs/heads/master from yuqian90 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3ef7118 ] [AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324) > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 2.0.0, 1.10.10 > > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041845#comment-17041845 ] ASF GitHub Bot commented on AIRFLOW-6704: - kaxil commented on pull request #7324: [AIRFLOW-6704] Copy common TaskInstance attributes from Task URL: https://github.com/apache/airflow/pull/7324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028115#comment-17028115 ] ASF GitHub Bot commented on AIRFLOW-6704: - yuqian90 commented on pull request #7324: [AIRFLOW-6704] Set TaskInstance.operator in constructor URL: https://github.com/apache/airflow/pull/7324 ``TaskInstance.operator`` is currently only set when task is executed. But if a task is marked success or failed, the ``operator`` field is left as None. This causes bugs when some other code tries to use the ``operator`` field to find the name of the class. The fix is trivial, just set ``TaskInstance.operator`` in its constructor. An assertion is also added in ``test_mark_tasks.py`` to test this. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TaskInstance.operator is not set when task is marked success or failed in the > Web UI > > > Key: AIRFLOW-6704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6704 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10.7 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > > {{TaskInstance.operator}} is currently only set when task is executed. But if > a task is marked success or failed, the {{operator}} field is left as > {{None}}. > This causes bugs when some code tries to use the operator field to find the > name of the class. > The fix is trivial, just set {{TaskInstance.operator}} in its constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)