@*jmeickle *can you share your code or any sample? How are you returning
nothing?
I may have spoken too soon. I can't seem to set dag to None, it fails with
*>>>>Tried to create relationships between tasks that don't have DAGs yet*
On Tue, Oct 9, 2018 at 12:57 PM Shah Altaf <mend...@gmail.com> wrote:
>
>
> Thanks all, following your advice I've decided to do this right now with a
> decorator. It looks crude so any suggestions are welcome.
>
> The decorator function checks an environment variable that we set on our
> servers and if it matches any of the passed in environment names, it'll
> allow the DAG to be created.
>
>
> def environments(env_names):
> def restrict_environment(func):
> def func_wrapper(*args, **kwargs):
> * if( os.getenv("environment") in env_names):*
> return DAG(*args, **kwargs)
> return func_wrapper
> return restrict_environment
>
> *@environments(["env1","env2","production"])*
> def get_DAG(*args, **kwargs):
> return None
>
> dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30 5
> * * *")
>
>
> Regards
> Shah
>
>
> On Mon, Oct 8, 2018 at 5:19 PM William Pursell <willi...@wepay.com.invalid>
> wrote:
>
>> This would be a very desirable feature. It's not just an issue of
>> differing environments, but also changing requirements. A dag may be
>> used for a time, and then removed from the environment. But if it's
>> still in the airflow db, it will get a row on the UI with a black "I'
>> indicating that the dag is missing. As far as I know, then only way
>> to remove it is to manually edit the database.
>> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <ch...@crpalmer.com> wrote:
>> >
>> > I like James solution better, but the initial thought I had was to
>> deploy
>> > airflowignore files to the environments to filter out files that should
>> not
>> > be processed when filling the DagBag.
>> >
>> > Chris
>> >
>> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
>> > <jmeic...@quantopian.com.invalid> wrote:
>> >
>> > > As long as the Airflow process can't find the DAG as a top-level
>> object in
>> > > the module, it won't be registered. For example, we have a function
>> that
>> > > returns DAGs; the function returns nothing if it's not in the right
>> > > environment.
>> > >
>> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <mend...@gmail.com> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI
>> based on
>> > > an
>> > > > environment variable?
>> > > >
>> > > >
>> > > >
>> > > > Our team has a single repo with several DAGs in it and we deploy it
>> > > across
>> > > > multiple 'environments' (think dev, test, and other integration
>> > > > environments). While most DAGs are meant to run everywhere, we do
>> have
>> > > > several which are meant to run in one and only one environment. Of
>> > > course
>> > > > they are all paused, but it would be nice to declutter a bit for
>> > > ourselves.
>> > > >
>> > > > My question then - is it possible to conditionally hide a DAG from
>> the UI
>> > > > based on an environment variable or some flag somewhere.
>> > > >
>> > > > This is just wishful thinking - the dev could do something like
>> > > >
>> > > > dag = get_dag(...),and get_dag() would have a decorator like
>> > > > @only_run_in("integration4,dev,local")
>> > > >
>> > > > And that decorator returns some kind of None object or special DAG
>> which
>> > > > just doesn't appear in the list.
>> > > >
>> > > > Or perhaps some other way to accomplish this same effect - any ideas
>> > > would
>> > > > be appreciated.
>> > > >
>> > > > Thanks
>> > > >
>> > >
>>
>>
