GitHub user odhondt closed a discussion: Combining task groups, dynamic mapping 
and DockerOperator

Hi

Consider the following DAG:
```python
from airflow import DAG
from airflow.decorators import task, task_group
from datetime import datetime, timedelta
from time import sleep

with DAG(
    dag_id="queries_and_download",
    schedule=timedelta(minutes=10),
    start_date=datetime(2022, 12, 22),
    catchup=False,
) as dag:

    @task
    def query_all_sources():
        sleep(5)
        return [1, 2, 3, 4, 5]

    @task_group
    def calculations(x):
        @task(pool="download")
        def download_product(product):
            sleep(5)
            print(f"downloading product {product}")
            return product

        @task(pool="process")
        def process_product(product):
            sleep(10)
            print(f"processing product {product}")

        process_product(download_product(x))

    calculations.expand(x=query_all_sources())
```
I would like to do something similar replacing `download_product` and 
`process_product` with two `DockerOperator` which would fire up scripts (say 
`download_product.py`, `process_product.py`) that both would take the 
dynamically generated outputs from `query_all_sources` as an argument.

So far I have tried to use various combinations of xcoms and templates to 
automatically fill the `command` parameter of the operators but with no 
success. 

Can this be achieved?

GitHub link: https://github.com/apache/airflow/discussions/47443

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to