moomindani commented on PR #65869:
URL: https://github.com/apache/airflow/pull/65869#issuecomment-4863233269
Following up since the "I think we're covered" question was left open — I
dug into the code paths, and unfortunately I don't think it is:
1. `_launch_job` is never reached from a `DatabricksWorkflowTaskGroup`
member. In `DatabricksTaskBaseOperator.execute()`, the workflow-group branch
only pulls the launch task's run metadata and monitors it
(`operators/databricks.py:1917-1935`); `_launch_job` sits in the `else`
(standalone) branch.
2. `render_template_fields` does run for every member task — but only in
that task's own task instance, which starts *after* the job has already been
created. The workflow JSON is built by the auto-added `launch` task via
`_CreateDatabricksWorkflowOperator.create_workflow_json()`, which iterates the
member operator objects and calls `_convert_to_databricks_workflow_task` on
them (`databricks_workflow.py:187-192`) — at that point those objects are still
unrendered. That is the actual #46563 bug.
3. For a standalone operator, fields are rendered before `execute()`, so the
re-render added in `_launch_job` is a no-op there — consistent with what the
CHANGES_REQUESTED review found, including why `test_launch_job` only passes
because it calls `_launch_job` directly without the render step.
So the fix likely needs to live in the workflow-group JSON-building path
(`_convert_to_databricks_workflow_task`). The design question to answer there
is context: rendering member fields at that point would use the launch task's
context rather than the member task's, so expressions like `{{ ti }}` or
task-scoped params would resolve against the wrong task instance. Settling how
#46563 should behave for those cases is probably the prerequisite for the right
fix.
---
Drafted-by: Claude Code (Fable 5); reviewed by @moomindani before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
