Newish to airflow as well, but here are some ideas... DAGs definition files (the python files you put in the AIRFLOW_HOME/dags/ dir) are evaluated at the start of the job (see https://stackoverflow.com/q/58567386/8236733), so if your rss-getting logic happens mid-dag, the dag can't update itself dynamically to branch for each article (so far as I know). You may have to have another dag that gets triggered by the completion of this dag (see https://www.astronomer.io/guides/trigger-dag-operator/) that reads to article info from, say, a log file or json that gets written by the first dag.
If getting the rss feed articles in the 1st thing your dag wants to do (or if it does not depend on other preceding tasks), you could just have the rss-getting logic in the dag definition file itself, put the article names in a string list or dict (if want to use more info), and then use a loop to create dag tasks for each article. Also, just because you mention the idea of subdags, you can read more about them in this article here (https://www.astronomer.io/guides/subdags/), but I don't think this is going to be what you want. On Fri, Nov 29, 2019 at 8:12 AM jeff <[email protected]> wrote: > I am sure I am missing some basic concept but I am looking for how to do a > single operation that can fan out into many. Say like I grab an rss feed > and there are 10 news articles. I want to apply some kind of DAG on each > of those. Is there a way to specify that or should the DAG that is pulling > the articles down simply trigger another DAG for each article. > > Thank you in advance. > -- This electronic message is intended only for the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
