jroachgolf84 commented on PR #65869:
URL: https://github.com/apache/airflow/pull/65869#issuecomment-4348717296
@moomindani - thanks for the feedback. I think you're right, we may need to
dig into this a bit more. The example in issue #46563 was actually inheriting
from the `DatabricksTaskBaseOperator`, despite having `Workflow` in the class
name. This is was the sample DAG that I tested with (and was included in the
unit-tests). Here's that DAG below.
```python
class CustomDatabricksWorkflowTaskOperator(DatabricksTaskBaseOperator):
"""Custom operator wrapping DatabricksTaskOperator"""
template_fields = (
"application_args",
"spark_conf",
"driver_jvm_props",
)
def __init__(
self,
*args,
class_name: str = "",
spark_conf: dict[str, str] | str = None,
application_args: List[str] | str | None = None,
driver_jvm_props: dict[str, str] | None = None,
application_conf_file: str = "test.conf",
libraries: List[dict[str, str]] = None,
**kwargs,
):
self.class_name = class_name
self.application_conf_file = application_conf_file
self.library = libraries if libraries is not None else
_get_default_libraries()
self.application_args = application_args
super().__init__(*args, **kwargs)
self.driver_jvm_props = driver_jvm_props
self.spark_conf = spark_cone
def _get_task_base_json(self) -> dict[str, Any]:
return {
"run_if": self.databricks_trigger_rule,
'spark_jar_task': {
'main_class_name': self.class_name,
parameters": [
self.application_args,
"{{ get_test_template_value(ti,'test_key') }}" // I
have hard-coded here for ref. In ideal case, this comes as part of
self.application_args which has been added in template_fields
]
},
'libraries': self.library,
}
def execute(self, context):
super().execute(context)
```
Regardless if tasks are "standalone" or if they are part of a Workflow
TaskGroup, Airflow will ensure that `render_template_fields(context)` is called
on them. So for the use-case that you're mentioning, we should be covered.
`_launch_job` could still be called by these tasks in the
`DatabricksWorkflowTaskGroup`, and this PR covers that.
As I type this out, I think we're covered. But let me review a bit more
tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
