Hi Daniel. We have a similar scenario and we're using almost the same
approach. The difference is that we use a file that is copied locally to
the workers and is used by the DAG to define the tasks.
I didn't analyze all the new features in Airflow 2, but tried the subtask
approach with the last
wow OK I didn't know this was an option.
I'll see what I can do and write back if we make progress.
Thank you Daniel.
On 2021/06/10 01:01:23, Daniel Standish wrote:
> > Thanks Daniel. We would need to add this as an option to the K8s Helm
> Chart and that is kinda outside our current scope.
>
