Hi Joe
Not sure if it's the recommended way to do it, however what worked for me is
rendering the templates in a python task and not directly in the
KubernetesPodOperator.
For example:
@task()
def create_cmds(**context):
run_params = context["params"]
size = int(run_params['size'])
arr = []
for s in range(int(size)):
arr.append([f'echo hi {s}'])
return arr
say_hi = KubernetesPodOperator.partial(
image="alpine",
cmds=["/bin/sh", "-c"],
task_id='test',
name='test'
).expand(arguments=create_cmds())
From: Joe Auty <[email protected]>
Sent: Tuesday, June 14, 2022 10:03 AM
To: [email protected]
Subject: Templated fields and dynamic task mapping
Hello,
I'm trying to understand the docs here:
https://airflow.apache.org/docs/apache-airflow/2.3.2/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact
, specifically this section:
If you want to interpolate values either call task.render_template yourself, or
use interpolation:
In the example in the previous section of what doesn't work we have:
printer.expand(val=make_list())
What should the corrected version of this line be? IOW, how would I call
make_list passing in the context so that I can send templated fields to my
mapping function? Here is a more specific use case:
KubernetesPodOperator.partial(
task_id="schema-dump-input",
namespace=NAMESPACE,
image=REGISTRY_URL + "/postgres-client:12",
name="pg-schemadump",
in_cluster=True,
hostnetwork=False,
max_active_tis_per_dag=1,
dag=dag
).expand(
cmds=schema_dump_input_cmds(ds),
env_vars=schema_dump_input_env_vars(ds)
)
In this example, ds has no value of course because it is not defined anywhere,
and of course {{ ds }} doesn't work either (this doesn't get interpolated and
is registered as literally "{{ ds }}"). How can I pass in a templated field,
such as {{ ds }}?
Thanks very much in advance!
This message contains information that may be confidential. If you are not the
intended recipient, please delete it immediately and notify us at
[email protected]. Please note that any disclosure or copying of its content is
forbidden
