Defining the dag inside the for loop would be the best way IMO, if you define a subdag you would have the same problem.
Subdags seem to be more suited for repeating some patters without rewriting the code. On Wed, Mar 22, 2017 at 4:49 PM Andrew Maguire <[email protected]> wrote: > Yeah was thinking that afterwards. Would that be a sub dag approach, or is > it even as easy as moving the dag definition itself into the for loop? > > Cheers > Andy > > On Wed, Mar 22, 2017 at 4:45 PM Jorge Alpedrinha Ramos < > [email protected]> wrote: > > > Hey! > > > > I've read your post and I would point that you could instead of creating > > tasks on one dag for each lob, you could create one dag per each lob > with a > > 'my_bigquery_dag_for_{{lob}}'. This way you get a more isolated unit of > > management (dag). The advantage here being that in the event of a failure > > or data reprocess for a single lob you wouldn't need to reprocess all > lobs. > > > > Other than that it is really cool that you're putting it out on a blog > > post, it will definitely be useful. > > > > Cheers ! > > > > On Wed, Mar 22, 2017 at 4:20 PM Andrew Maguire <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Just to follow up as fyi. > > > > > > Think i figured out at least one approach. Did a blog post on it > > > > > > > > > http://engineering.pmc.com/2017/03/playing-around-with-apache-airflow-bigquery-62/ > > > > > > Can't wait to get going on porting stuff over from cron. > > > > > > Cheers, > > > Andy > > > > > > On Tue, Mar 21, 2017 at 2:49 PM Andrew Maguire <[email protected]> > > > wrote: > > > > > > > Sorry if this is not the right place for this. > > > > > > > > I'm wondering if anyone might be able to help me with this question? > > > > > > > > > > > > > > > > > > http://stackoverflow.com/questions/42930221/best-way-to-loop-through-parameters-in-airflow > > > > > > > > Feel free to ignore if this is not the right place to try find help. > > > > > > > > Loving airflow btw! > > > > > > > > Cheers, > > > > Andy > > > > > > > > > >
