Thank you everyone,
All your comments and ideas will be usefull to us in a near future to make up
our plan of action.
Thank you Kevin,
This is great output and which gives us a more of any idea on how to set it up
from our end.
ctive.
> > > > >
> > > > >
> > > > > Regards,
> > > > > Ananth.P,
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 7 June
t;
> > > >
> > > > On 7 June 2018 at 04:08, Arturo Michel
> > > > wrote:
> > > >
> > > > > We have had up to 50 dags with multiple tasks each. Many of them run
> > in
> > > > > parallel, we've had some iss
, we've had some issues with compute as it was meant to be a
> > > > temporary deployment but somehow it's now the permanent production
> one
> > > and
> > > > resources are not great.
> > > > Oranisationally it is very similar to what Ger
orary deployment but somehow it's now the permanent production one
> > and
> > > resources are not great.
> > > Oranisationally it is very similar to what Gerard described. More than
> > one
> > > group working with different engineering practices and stan
andards, this is
> > probably one of the sources of problems.
> >
> > -----Original Message-----
> > From: Gerard Toonstra
> > Sent: Wednesday, June 6, 2018 5:02 PM
> > To: dev@airflow.incubator.apache.org
> > Subject: Re: Single Airflow Instance Vs Mu
cribed. More than one
> group working with different engineering practices and standards, this is
> probably one of the sources of problems.
>
> -Original Message-
> From: Gerard Toonstra
> Sent: Wednesday, June 6, 2018 5:02 PM
> To: dev@airflow.incubator.apache.org
> Sub
le Airflow Instance Vs Multiple Airflow Instance
We are using two cluster instances. One cluster is for the engineering teams
that are in the "tech" wing and which rigorously follow tech principles, the
other instance is for use by business analysts and more ad-hoc, experimental
work, wh
We are using two cluster instances. One cluster is for the engineering
teams that are in the "tech" wing and which rigorously follow
tech principles, the other instance is for use by business analysts and
more ad-hoc, experimental work, who do not necessarily follow the
principles. We have a nomad
An important consideration here is that there are several settings that are
cluster-wide. In particular, cluster-wide concurrency settings could result
in Team B's DAG refusing to schedule based on an error in Team A's DAG.
Do your teams follow similar practices in how eagerly they ship code, or
h
; structure in your deployment process you could do this.
>
> It works fine.
>
> -Original Message-
> From: gauthiermartin86@
> Sent: Wednesday, June 6, 2018 4:24 PM
> To: d...@airflow.apache.org
> Subject: Single Airflow Instance Vs Multiple Airflow Instance
>
&
(Project2) their own dags. If you incorporate this structure in
your deployment process you could do this.
It works fine.
-Original Message-
From: gauthiermartin86@
Sent: Wednesday, June 6, 2018 4:24 PM
To: d...@airflow.apache.org
Subject: Single Airflow Instance Vs Multiple Airflow
(Project2) their own dags. If you incorporate this structure in
your deployment process you could do this.
It works fine.
-Original Message-
From: gauthiermartin86@
Sent: Wednesday, June 6, 2018 4:24 PM
To: d...@airflow.apache.org
Subject: Single Airflow Instance Vs Multiple Airflow
Hi Everyone,
We have been experimenting with airflow for about 6 months now.
We are planning to have multiple departments to use it. Since we don't have any
internal experience with Airflow we are wondering if single instance per
department is more suited than single instance with multi-tenancy
15 matches
Mail list logo
