It's fine (and required if you do task assignment) to create the DAG
object; just return None from the function (return early) so that there is
no variable containing a DAG object visible within the module.
On Tue, Oct 9, 2018 at 12:21 PM Shah Altaf <mend...@gmail.com> wrote:
> @*jmeickle *can you share your code or any sample? How are you returning
> nothing?
>
> I may have spoken too soon. I can't seem to set dag to None, it fails with
>
> *>>>>Tried to create relationships between tasks that don't have DAGs yet*
>
>
> On Tue, Oct 9, 2018 at 12:57 PM Shah Altaf <mend...@gmail.com> wrote:
>
> >
> >
> > Thanks all, following your advice I've decided to do this right now with
> a
> > decorator. It looks crude so any suggestions are welcome.
> >
> > The decorator function checks an environment variable that we set on our
> > servers and if it matches any of the passed in environment names, it'll
> > allow the DAG to be created.
> >
> >
> > def environments(env_names):
> > def restrict_environment(func):
> > def func_wrapper(*args, **kwargs):
> > * if( os.getenv("environment") in env_names):*
> > return DAG(*args, **kwargs)
> > return func_wrapper
> > return restrict_environment
> >
> > *@environments(["env1","env2","production"])*
> > def get_DAG(*args, **kwargs):
> > return None
> >
> > dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30
> 5
> > * * *")
> >
> >
> > Regards
> > Shah
> >
> >
> > On Mon, Oct 8, 2018 at 5:19 PM William Pursell
> <willi...@wepay.com.invalid>
> > wrote:
> >
> >> This would be a very desirable feature. It's not just an issue of
> >> differing environments, but also changing requirements. A dag may be
> >> used for a time, and then removed from the environment. But if it's
> >> still in the airflow db, it will get a row on the UI with a black "I'
> >> indicating that the dag is missing. As far as I know, then only way
> >> to remove it is to manually edit the database.
> >> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <ch...@crpalmer.com> wrote:
> >> >
> >> > I like James solution better, but the initial thought I had was to
> >> deploy
> >> > airflowignore files to the environments to filter out files that
> should
> >> not
> >> > be processed when filling the DagBag.
> >> >
> >> > Chris
> >> >
> >> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
> >> > <jmeic...@quantopian.com.invalid> wrote:
> >> >
> >> > > As long as the Airflow process can't find the DAG as a top-level
> >> object in
> >> > > the module, it won't be registered. For example, we have a function
> >> that
> >> > > returns DAGs; the function returns nothing if it's not in the right
> >> > > environment.
> >> > >
> >> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <mend...@gmail.com>
> wrote:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI
> >> based on
> >> > > an
> >> > > > environment variable?
> >> > > >
> >> > > >
> >> > > >
> >> > > > Our team has a single repo with several DAGs in it and we deploy
> it
> >> > > across
> >> > > > multiple 'environments' (think dev, test, and other integration
> >> > > > environments). While most DAGs are meant to run everywhere, we do
> >> have
> >> > > > several which are meant to run in one and only one environment.
> Of
> >> > > course
> >> > > > they are all paused, but it would be nice to declutter a bit for
> >> > > ourselves.
> >> > > >
> >> > > > My question then - is it possible to conditionally hide a DAG from
> >> the UI
> >> > > > based on an environment variable or some flag somewhere.
> >> > > >
> >> > > > This is just wishful thinking - the dev could do something like
> >> > > >
> >> > > > dag = get_dag(...),and get_dag() would have a decorator like
> >> > > > @only_run_in("integration4,dev,local")
> >> > > >
> >> > > > And that decorator returns some kind of None object or special DAG
> >> which
> >> > > > just doesn't appear in the list.
> >> > > >
> >> > > > Or perhaps some other way to accomplish this same effect - any
> ideas
> >> > > would
> >> > > > be appreciated.
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > >
> >>
> >>
>
