ok, just took a look at the latest docs. Not sure I fully understand it, but i guess you can pass a value as an outlet into the subdag? And the subdag can use that to create n task? i'm a bit weary of using it if it's not in a stable release. Is there a time frame on 1.10 release?
Thanks, Andrew On Wed, Jun 20, 2018 at 3:00 PM Bolke de Bruin <bdbr...@gmail.com> wrote: > The new lineage functionality (while in very early states) can do this for > you. > > B. > > > On 20 Jun 2018, at 19:46, Andrew Harmon <andrewhhar...@gmail.com> wrote: > > > > Hello, > > > > I wanted to submit a feature suggestion. I have a scenario where I'd like > > to create a subdag where the number of tasks in it are dependent on a > > previous task. Ideally, this parameter could be passed in via xcom. For > > example: > > > > Task A does something that generates 10 files. Task B is a subdag that at > > runtime, creates 10 tasks within the sub dag. Each task will process one > of > > the 10 files. That way the processing of the 10 files can run in > parallel. > > Task C will then start when all tasks in the subdag are complete. > > > > My particular scenario is to process the results of a BigQuery export to > > GCS. The export creates n number of files and you don't know that number > > until the export is done running. > > > > Thanks, > > Andrew > > -- > > Andrew Harmon > > (202) 615-6433 > > -- Andrew Harmon (202) 615-6433
