Interesting! So, is it accurate to say that all Python functions get their
context for free? I never thought of trying a:
def create_cmds(**context):
print(context)
say_hi = KubernetesPodOperator.partial(
image="alpine",
cmds=["/bin/sh", "-c"],
task_id='test',
name='test'
).expand(arguments=create_cmds())
Just to see what, if any, was available in the context object, because I didn’t
think this was a freebie outside of using the PythonOperator as a task with the
provide_context arg set. I’ll give this a try!
As far as the Cartesian product question, I haven’t wrapped my head around
exactly how I’d want to architect this, for now I’m just trying to see what
options are available.
> On Jun 15, 2022, at 12:33 PM, Daniel Standish <[email protected]>
> wrote:
>
> Yeah it does seem that templating does not work with expanded params at the
> moment. No promises but I would bet this will change at some point. Seems
> reasonable and I can't think of a technical problem.
>
> Tal's approach looks good. I came up with basically the same thing when
> testing this out:
> @dag.task
> def gen_vars(ds=None):
> return [{"MY_VAR": ds} for x in range(3)]
>
> op2 = BashOperator.partial(task_id='task_mapped', bash_command="echo
> $MY_VAR",).expand(
> env=gen_vars(),
> )
> One thing I noticed in your example was, it appears, maybe you are mapping
> from two upstream tasks, for two mapped arguments. You may have noticed,
> this will result in a cartesian product (maybe this is desirable for you?).
> This is an area we are actively working on, so that you can provide multiple
> kwargs to map from a single task.
>
>
>
> On Wed, Jun 15, 2022 at 3:36 AM Tal Nagar <[email protected]
> <mailto:[email protected]>> wrote:
> Hi Joe
>
> Not sure if it's the recommended way to do it, however what worked for me is
> rendering the templates in a python task and not directly in the
> KubernetesPodOperator.
>
> For example:
>
>
>
> @task()
>
> def create_cmds(**context):
>
> run_params = context["params"]
>
> size = int(run_params['size'])
>
> arr = []
>
> for s in range(int(size)):
>
> arr.append([f'echo hi {s}'])
>
> return arr
>
>
>
> say_hi = KubernetesPodOperator.partial(
>
> image="alpine",
>
> cmds=["/bin/sh", "-c"],
>
> task_id='test',
>
> name='test'
>
> ).expand(arguments=create_cmds())
>
>
>
> From: Joe Auty <[email protected] <mailto:[email protected]>>
> Sent: Tuesday, June 14, 2022 10:03 AM
> To: [email protected] <mailto:[email protected]>
> Subject: Templated fields and dynamic task mapping
>
>
>
> Hello,
>
> I'm trying to understand the docs here:
> https://airflow.apache.org/docs/apache-airflow/2.3.2/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact
>
> <https://airflow.apache.org/docs/apache-airflow/2.3.2/concepts/dynamic-task-mapping.html#how-do-templated-fields-and-mapped-arguments-interact>
> , specifically this section:
>
>
>
> If you want to interpolate values either call task.render_template yourself,
> or use interpolation:
>
>
> In the example in the previous section of what doesn't work we have:
>
>
> printer.expand(val=make_list())
>
> What should the corrected version of this line be? IOW, how would I call
> make_list passing in the context so that I can send templated fields to my
> mapping function? Here is a more specific use case:
>
>
>
> KubernetesPodOperator.partial(
> task_id="schema-dump-input",
> namespace=NAMESPACE,
> image=REGISTRY_URL + "/postgres-client:12",
> name="pg-schemadump",
> in_cluster=True,
> hostnetwork=False,
> max_active_tis_per_dag=1,
> dag=dag
> ).expand(
> cmds=schema_dump_input_cmds(ds),
> env_vars=schema_dump_input_env_vars(ds)
> )
>
> In this example, ds has no value of course because it is not defined
> anywhere, and of course {{ ds }} doesn't work either (this doesn't get
> interpolated and is registered as literally "{{ ds }}"). How can I pass in a
> templated field, such as {{ ds }}?
>
> Thanks very much in advance!
>
>
>
>
> This message contains information that may be confidential. If you are not
> the intended recipient, please delete it immediately and notify us at
> [email protected] <mailto:[email protected]>. Please note that any disclosure
> or copying of its content is forbidden
>
