@dan can provide more details but I think our staging is running off of our production DAGS_FOLDER, but we swap all tasks with dummy tasks (BashOperator with a dummy command) in our policy function. http://pythonhosted.org/airflow/concepts.html?highlight=policy#cluster-policy
On Thu, May 19, 2016 at 9:34 AM, Chris Riccomini <criccom...@apache.org> wrote: > Hey Max, > > I think we would like to set up some DAG testing as well. Are you guys > using synthetic DAGs, or running your real DAGs on a separate cluster? > > Cheers, > Chris > > On Wed, May 18, 2016 at 1:32 PM, Lance Norskog <lance.nors...@gmail.com> > wrote: > > > Ok, we'll update to 1.7.1 when y'all think it's fine. > > > > Thanks, > > > > Lance Norskog > > > > On Wed, May 18, 2016 at 12:01 PM, Bolke de Bruin <bdbr...@gmail.com> > > wrote: > > > > > Hey Max, > > > > > > Fair point. I’ll make sure that for the next release we jump a bit > > earlier > > > on board. We do run integration tests continuously, > > > but only from next month we will reach a certain level of complexity we > > > really will need to start pre-testing releases. > > > > > > - Bolke > > > > > > > Op 18 mei 2016, om 17:49 heeft Maxime Beauchemin < > > > maximebeauche...@gmail.com> het volgende geschreven: > > > > > > > > There's an RC out that is currently in production at Airbnb (as of > > > Monday) > > > > if you want to help us make sure the next version is up fully baked > for > > > > release. For now Airbnb is carrying most of the risk around deploying > > new > > > > code in production first. Knowing that we don't use all features and > > > > therefore wouldn't catch all possible regressions, it would be nice > to > > > have > > > > more companies pushing RCs in production along with us. > > > > > > > > Here's the git tag for the RC: > > > > > > https://github.com/apache/incubator-airflow/releases/tag/airbnb_1.7.1rc6 > > > > > > > > Max > > > > > > > > On Tue, May 17, 2016 at 11:01 PM, Bolke de Bruin <bdbr...@gmail.com> > > > wrote: > > > > > > > >> 1.7.1 that most likely will be out at the end of the week, hopefully > > > fixes > > > >> this indeed. Don't stay on 1.7.0 for too long 1.7.1 contains many > > > stability > > > >> fixes. > > > >> > > > >> Verstuurd vanaf mijn iPad > > > >> > > > >>> Op 18 mei 2016 om 03:06 heeft Lance Norskog < > lance.nors...@gmail.com > > > > > > >> het volgende geschreven: > > > >>> > > > >>> Has the "long-running scheduler hang" problem been solved yet? > > > >>> We just upgraded from 1.6.2 to 1.7.0 and we think it just happened, > > but > > > >>> don't know. > > > >>> Should we use chronic scheduler restarts? > > > >>> > > > >>> Thanks, > > > >>> > > > >>> -- > > > >>> Lance Norskog > > > >>> lance.nors...@gmail.com > > > >>> Redwood City, CA > > > >> > > > > > > > > > > > > -- > > Lance Norskog > > lance.nors...@gmail.com > > Redwood City, CA > > >
