Hello all,
While we were working on implementing Dynamic task mapping we
discovered some ambigituies in the DAG author interface around mapping
over literal lists, that have prompted us to change the API.
Full details can be found
<https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=191336946&selectedPageVersions=13&selectedPageVersions=12>
The main driver was this case:
t3 = DockerOperator.partial(
image='centos:latest',
task_id='docker_op_tester',
).map(command=["/bin/sleep", xcom_arg])
It's not clear (to code – a human can take a reasonable guess at the
intent) if this is meant to result in two mapped operators, on e with
"/bin/sleep" and another with xcom_arg , or if each mapped operator
should be ["/bin/sleep", n] (where n is the each element of the
xcom_arg.)
In summary:
Operator.partial().map() replaced with Operator.partial().apply()
XComArg.map added (to more closely resemble map in other functional
programming languages/contexts)
Please see the wiki page for full details and more rationale. If
anything is unclear or if someone doesn't like the new proposed
interface please let us know.
Thanks,
Ash
