Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
jscheffl commented on PR #66735: URL: https://github.com/apache/airflow/pull/66735#issuecomment-4449072587 Manual backport in https://github.com/apache/airflow/pull/66920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
github-actions[bot] commented on PR #66735: URL: https://github.com/apache/airflow/pull/66735#issuecomment-4449044639 ### Backport failed to create: v3-2-test. View the failure log Run details Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x) the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches. In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel. Status Branch Result ❌ v3-2-test https://github.com/apache/airflow/commit/a641d83da9c0858e19e076fa9690a0485a3d5233";> You can attempt to backport this manually by running: ```bash cherry_picker a641d83 v3-2-test ``` This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking the files that need manual conflict resolution. After you have resolved the conflicts, you can continue the backport process by running: ```bash cherry_picker --continue ``` If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
jscheffl merged PR #66735: URL: https://github.com/apache/airflow/pull/66735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
amoghrajesh commented on code in PR #66735:
URL: https://github.com/apache/airflow/pull/66735#discussion_r3239231839
##
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##
@@ -1325,22 +1326,24 @@ def _on_term(signum, frame):
# Send update only if value changed (e.g., user set context
variables during execution)
if ti.rendered_map_index and ti.rendered_map_index !=
previous_rendered_map_index:
SUPERVISOR_COMMS.send(msg=SetRenderedMapIndex(rendered_map_index=ti.rendered_map_index))
-finally:
-log.info("::group::Post Execute")
Review Comment:
I am fine with either tbh since no such logs are emitted today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
amoghrajesh commented on code in PR #66735:
URL: https://github.com/apache/airflow/pull/66735#discussion_r3239230417
##
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##
@@ -1325,22 +1326,24 @@ def _on_term(signum, frame):
# Send update only if value changed (e.g., user set context
variables during execution)
if ti.rendered_map_index and ti.rendered_map_index !=
previous_rendered_map_index:
SUPERVISOR_COMMS.send(msg=SetRenderedMapIndex(rendered_map_index=ti.rendered_map_index))
-finally:
-log.info("::group::Post Execute")
_push_xcom_if_needed(result, ti, log)
msg, state = _handle_current_task_success(context, ti)
except DownstreamTasksSkipped as skip:
+log.info("::group::Post Execute")
Review Comment:
It just felt like a having a helper would have the "log.info("::group::Post
Execute")" in one place without needing to change many times if anything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
jscheffl commented on code in PR #66735:
URL: https://github.com/apache/airflow/pull/66735#discussion_r3228995249
##
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##
@@ -1325,22 +1326,24 @@ def _on_term(signum, frame):
# Send update only if value changed (e.g., user set context
variables during execution)
if ti.rendered_map_index and ti.rendered_map_index !=
previous_rendered_map_index:
SUPERVISOR_COMMS.send(msg=SetRenderedMapIndex(rendered_map_index=ti.rendered_map_index))
-finally:
-log.info("::group::Post Execute")
Review Comment:
Yeah if you prefer I can move it back to the original location. Just moved
it up for the case if some logs are emitted at time of creating the rendered
map index that this would be put into the group and not pollute user logs. But
there are no logs created today. Let me know and happy to change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
jscheffl commented on code in PR #66735:
URL: https://github.com/apache/airflow/pull/66735#discussion_r3228981305
##
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##
@@ -1325,22 +1326,24 @@ def _on_term(signum, frame):
# Send update only if value changed (e.g., user set context
variables during execution)
if ti.rendered_map_index and ti.rendered_map_index !=
previous_rendered_map_index:
SUPERVISOR_COMMS.send(msg=SetRenderedMapIndex(rendered_map_index=ti.rendered_map_index))
-finally:
-log.info("::group::Post Execute")
_push_xcom_if_needed(result, ti, log)
msg, state = _handle_current_task_success(context, ti)
except DownstreamTasksSkipped as skip:
+log.info("::group::Post Execute")
Review Comment:
Yeah, it is a 1-liner... unfortunately need to be in each catch-block
individually. So I could replace it with a helper but then the helper would
replace a 1-liner with a 1-liner. Or do you consider the Literal in the message
that should be replaced by a constant? And proposal how to improve? Coding-wise
I really preferred it like it was before but we need to scatter in start of the
group in multiple places for the different branches of handling results :-(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Improve Post-Task logs to show exception in failure [airflow]
amoghrajesh commented on code in PR #66735:
URL: https://github.com/apache/airflow/pull/66735#discussion_r3225755312
##
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##
@@ -1325,22 +1326,24 @@ def _on_term(signum, frame):
# Send update only if value changed (e.g., user set context
variables during execution)
if ti.rendered_map_index and ti.rendered_map_index !=
previous_rendered_map_index:
SUPERVISOR_COMMS.send(msg=SetRenderedMapIndex(rendered_map_index=ti.rendered_map_index))
-finally:
-log.info("::group::Post Execute")
_push_xcom_if_needed(result, ti, log)
msg, state = _handle_current_task_success(context, ti)
except DownstreamTasksSkipped as skip:
+log.info("::group::Post Execute")
Review Comment:
Maybe adding a small helper for this would be nicer?
##
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##
@@ -1325,22 +1326,24 @@ def _on_term(signum, frame):
# Send update only if value changed (e.g., user set context
variables during execution)
if ti.rendered_map_index and ti.rendered_map_index !=
previous_rendered_map_index:
SUPERVISOR_COMMS.send(msg=SetRenderedMapIndex(rendered_map_index=ti.rendered_map_index))
-finally:
-log.info("::group::Post Execute")
Review Comment:
In the older code, the finally ran after the else block, so
`_render_map_index` on the success path ran before `::group::Post Execute`. In
this code, `log.info("::group::Post Execute")` is placed after `result =
_execute_task` but before the else block, so `_render_map_index` now runs
inside the Post Execute group on the success path.
Just flagging in case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
