arunangshu01 commented on issue #51954:
URL: https://github.com/apache/airflow/issues/51954#issuecomment-3018531112
`
from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator,
BranchPythonOperator
from airflow.sdk import Variable
import pendulum
from datetime import timedelta
import time
import ast
import json
default_args = {
'owner': 'airflow',
'retries': 5,
'retry_delay': timedelta(minutes=1),
}
def set_list_variable(**kwargs):
Variable.set('sample_list', str(list(range(1, 17))))
def get_list():
val = Variable.get('sample_list', default='[1]')
# Ensure it's a list if set from UI
return ast.literal_eval(val) if isinstance(val, str) else val
def iterator_task(i, **kwargs):
print(f"Processing item {i}, sleeping for 2 seconds")
time.sleep(2)
def iterator_task1(i, **kwargs):
print(f"Processing item {i}, sleeping for 2 seconds")
if isinstance(i,str):
raise Exception
time.sleep(2)
def final_task(**kwargs):
print("All iterator tasks are done.")
with DAG(
'dynamic_sequential_dag',
default_args=default_args,
description='DAG with variable-driven dynamic tasks (classic loop)',
schedule=None,
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
catchup=False,
tags=['example'],
) as dag:
t1 = PythonOperator(
task_id='set_list_variable',
python_callable=set_list_variable,
)
t2 = PythonOperator(
task_id=f'task_2',
python_callable=iterator_task,
op_args=[2],
)
tasks = []
for i in eval(Variable.get('sample_list',default='["dummy"]')):
task = PythonOperator(
task_id=f'iterator_task_{i}',
python_callable=iterator_task1,
op_args=[i],
retries=5
)
tasks.append(task)
t3 = PythonOperator(
task_id='final_task',
python_callable=final_task,
)
t1 >> t2 >> tasks >> t3
`
Here is the example dag, @bbovenzi
You have to keep on trying to replicate the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
