Definitely, here were the issues we hit: - airbnb/airflow#1365 occured - Webservers/scheduler were timing out and stuck in restart cycles due to increased time spent on parsing DAGs due to airbnb/airflow#1213/files - Failed tasks that ran after the upgrade and the revert (after we reverted the upgrade) were unable to be cleared (but running the tasks through the UI worked without clearing them) - The way log files were stored on S3 was changed (airflow now requires a connection to be setup) which broke log storage - Some DAGs were broken (unable to be parsed) due to package reorganization in open-source (the import paths were changed) (the utils refactor commit)
On Thu, Apr 28, 2016 at 12:17 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Dan, > > Are you able to share some of the bugs you have been hitting and connected > commits? > > We could at the very least learn from them and maybe even improve testing. > > Bolke > > > > Op 28 apr. 2016, om 06:51 heeft Dan Davydov > <dan.davy...@airbnb.com.INVALID> het volgende geschreven: > > > > All of the blockers were fixed as of yesterday (there was some issue that > > Jeremiah was looking at with the last release candidate which I think is > > fixed but I'm not sure). I started staging the airbnb_1.7.1rc3 tag > earlier > > today, so as long as metrics look OK and the 1.7.1rc2 issues seem > resolved > > tomorrow I will release internally either tomorrow or Monday (we try to > > avoid releases on Friday). If there aren't any issues we can push the > 1.7.1 > > tag on Monday/Tuesday. > > > > @Sid > > I think we were originally aiming to deploy internally once every two > weeks > > but we decided to do it once a month in the end. I'm not too sure about > > that so Max can comment there. > > > > We have been running 1.7.0 in production for about a month now and it > > stable. > > > > I think what really slowed down this release cycle is some commits that > > caused severe bugs that we decided to roll-forward with instead of > rolling > > back. We can potentially try reverting these commits next time while the > > fixes are applied for the next version, although this is not always > trivial > > to do. > > > > On Wed, Apr 27, 2016 at 9:31 PM, Siddharth Anand < > > siddharthan...@yahoo.com.invalid> wrote: > > > >> Btw, is anyone of the committers running 1.7.0 or later in any staging > or > >> production env? I have to say that given that 1.6.2 was the most stable > >> release and is 4 or more months old does not say much for our release > >> cadence or process. What's our plan for 1.7.1? > >> > >> Sent from Sid's iPhone > >> > >>> On Apr 27, 2016, at 9:05 PM, Chris Riccomini <criccom...@apache.org> > >> wrote: > >>> > >>> Hey all, > >>> > >>> I just wanted to check in on the 1.7.1 release status. I know there > have > >>> been some major-ish bugs, as well as several people doing tests. Should > >> we > >>> create a 1.7.1 release JIRA, and track outstanding issues there? > >>> > >>> Cheers, > >>> Chris > >> > >> > >
