Hi All,
We are trying to create a set of subdags, given a parent dag, which are to
be created and executed in batches.
The following piece of code creates the subdags:
def create_subdag(instr_test, check_batch_size, batch_start, batch_end):
instr_subdag_func = SUB_DAG_PRE + "_" + str(instr_test)
instr_subdag_id = SUB_DAG_PRE + "-" + str(instr_test)
instr_subdag_func = SubDagOperator(
task_id=instr_subdag_id,
subdag=instr_subdag_csv(MAIN_DAG_ID, instr_subdag_id, args,
instr_test),
default_args=args,
dag=dag)
instr_subdag_func.set_upstream(batch_start)
batch_end.set_upstream(instr_subdag_func)
check_batch_size = check_batch_size + 1
return check_batch_size
However, it seems that given a total no. of subdags to be created, not all
are gettng created. In our case, out of 54, only 38 are getting created.
Can someone please advise as to what might be going wrong ?
Also, do subdags also get listed out when we run the command : airflow
list_dags ? Just wanted to confirm if that is expected.
Sajan
