> On Jun 15, 2022, at 12:57 PM, Daniel Standish <[email protected]> > wrote: > > Turns out this issue is actually documented here: > https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact > > <https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact>
Yeah, and I was struggling with understanding this exact part of the documentation… Namely, I was expecting in that second quoted block for there to be a: printer.expand(val=make_list()) To accompany this example. I was confused where the context comes from, but if it is a freebie object attached to all Python functions without needing to pass it through explicitly, i.e. printer.expand(val=make_list(context)) Then this will solve my problem… I’ll test this out and let you know. If it does, would you be open to my specific suggestions on how to correct this documentation? Happy to provide the actual correction as a PR, if you think it would provide value to the community? > > On Wed, Jun 15, 2022 at 12:33 PM Daniel Standish > <[email protected] <mailto:[email protected]>> wrote: > Yeah it does seem that templating does not work with expanded params at the > moment. No promises but I would bet this will change at some point. Seems > reasonable and I can't think of a technical problem. > > Tal's approach looks good. I came up with basically the same thing when > testing this out: > @dag.task > def gen_vars(ds=None): > return [{"MY_VAR": ds} for x in range(3)] > > op2 = BashOperator.partial(task_id='task_mapped', bash_command="echo > $MY_VAR",).expand( > env=gen_vars(), > ) > One thing I noticed in your example was, it appears, maybe you are mapping > from two upstream tasks, for two mapped arguments. You may have noticed, > this will result in a cartesian product (maybe this is desirable for you?). > This is an area we are actively working on, so that you can provide multiple > kwargs to map from a single task. > > > > On Wed, Jun 15, 2022 at 3:36 AM Tal Nagar <[email protected] > <mailto:[email protected]>> wrote: > Hi Joe > > Not sure if it's the recommended way to do it, however what worked for me is > rendering the templates in a python task and not directly in the > KubernetesPodOperator. > > For example: > > > > @task() > > def create_cmds(**context): > > run_params = context["params"] > > size = int(run_params['size']) > > arr = [] > > for s in range(int(size)): > > arr.append([f'echo hi {s}']) > > return arr > > > > say_hi = KubernetesPodOperator.partial( > > image="alpine", > > cmds=["/bin/sh", "-c"], > > task_id='test', > > name='test' > > ).expand(arguments=create_cmds()) > > > > From: Joe Auty <[email protected] <mailto:[email protected]>> > Sent: Tuesday, June 14, 2022 10:03 AM > To: [email protected] <mailto:[email protected]> > Subject: Templated fields and dynamic task mapping > > > > Hello, > > I'm trying to understand the docs here: > https://airflow.apache.org/docs/apache-airflow/2.3.2/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact > > <https://airflow.apache.org/docs/apache-airflow/2.3.2/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact> > , specifically this section: > > > > If you want to interpolate values either call task.render_template yourself, > or use interpolation: > > > In the example in the previous section of what doesn't work we have: > > > printer.expand(val=make_list()) > > What should the corrected version of this line be? IOW, how would I call > make_list passing in the context so that I can send templated fields to my > mapping function? Here is a more specific use case: > > > > KubernetesPodOperator.partial( > task_id="schema-dump-input", > namespace=NAMESPACE, > image=REGISTRY_URL + "/postgres-client:12", > name="pg-schemadump", > in_cluster=True, > hostnetwork=False, > max_active_tis_per_dag=1, > dag=dag > ).expand( > cmds=schema_dump_input_cmds(ds), > env_vars=schema_dump_input_env_vars(ds) > ) > > In this example, ds has no value of course because it is not defined > anywhere, and of course {{ ds }} doesn't work either (this doesn't get > interpolated and is registered as literally "{{ ds }}"). How can I pass in a > templated field, such as {{ ds }}? > > Thanks very much in advance! > > > > > This message contains information that may be confidential. If you are not > the intended recipient, please delete it immediately and notify us at > [email protected] <mailto:[email protected]>. Please note that any disclosure > or copying of its content is forbidden >
