Hi,
I have a weird question but it bugs my mind. I have some like below to
generate dags dynamically, using Max's example code from FAQ.
It works fine but I have one large dict (let's call it my_outer_dict) that
takes over 60Mb in memory and I need to access it from all generated dags.
Needless to say, i do not want to recreate that dict for every dag as I
want to load it to memory only once.
To my surprise, if i define that dag outside of my dag definition code, I
can still access it.
Can someone explain why and where is it stored? I thought only dag
definitions are loaded to dagbag and not the variables outside it.
Is it even a good practice and will it work still if I switch to celery
executor?
def get_dag(i):
dag_id = 'foo_{}'.format(i)
dag = DAG(dag_id)
....
print my_outer_dict
my_outer_dict = {}
for i in range(10):
dag = get_dag(i)
globals()[dag.dag_id] = dag
