Right now, we see some jobs getting run twice with the second run happening exactly one hour later. We can't isolate why. The current doublerun job prevention kills both jobs as it happens in the heartbeat and they run concurrently and is overeager in killing both. This PR changes it so the job will get killed in a more sensible way and not have an issue with the current situation.
https://github.com/apache/incubator-airflow/pull/2102 On Thu, Feb 23, 2017 5:59 PM, siddharth anand san...@apache.org wrote: IMHO, a DAG run without a start date is non-sensical but is not enforced That said, our UI allows for the manual creation of DAG Runs without a start date as shown in the images below: - https://www.dropbox.com/s/3sxcqh04eztpl7p/Screenshot% 202017-02-22%2016.00.40.png?dl=0 <https://www.dropbox.com/s/3sxcqh04eztpl7p/Screenshot%202017-02-22%2016.00.40.png?dl=0> - https://www.dropbox.com/s/4q6rr9dwghag1yy/Screenshot% 202017-02-22%2016.02.22.png?dl=0 <https://www.dropbox.com/s/4q6rr9dwghag1yy/Screenshot%202017-02-22%2016.02.22.png?dl=0> On Wed, Feb 22, 2017 at 2:26 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Our database may have edge cases that could be associated with running any > previous version that may or may not have been part of an official release. > > Let's see if anyone else reports the issue. If no one does, one option is > to release 1.8.0 as is with a comment in the release notes, and have a > future official minor apache release 1.8.1 that would fix these minor > issues that are not deal breaker. > > @bolke, I'm curious, how long does it take you to go through one release > cycle? Oh, and do you have a documented step by step process for releasing? > I'd like to add the Pypi part to this doc and add committers that are > interested to have rights on the project on Pypi. > > Max > > On Wed, Feb 22, 2017 at 2:00 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > So it is a database integrity issue? Afaik a start_date should always be > > set for a DagRun (create_dagrun) does so I didn't check the code though. > > > > Sent from my iPhone > > > > > On 22 Feb 2017, at 22:19, Dan Davydov <dan.davy...@airbnb.com.INVALID> > > wrote: > > > > > > Should clarify this occurs when a dagrun does not have a start date, > not > > a > > > dag (which makes it even less likely to happen). I don't think this is > a > > > blocker for releasing. > > > > > >> On Wed, Feb 22, 2017 at 1:15 PM, Dan Davydov <dan.davy...@airbnb.com> > > wrote: > > >> > > >> I rolled this out in our prod and the webservers failed to load due to > > >> this commit: > > >> > > >> [AIRFLOW-510] Filter Paused Dags, show Last Run & Trigger Dag > > >> 7c94d81c390881643f94d5e3d7d6fb351a445b72 > > >> > > >> This fixed it: > > >> - </a> <span id="statuses_info" > > >> class="glyphicon glyphicon-info-sign" aria-hidden="true" title="Start > > Date: > > >> {{last_run.start_date.strftime('%Y-%m-%d %H:%M')}}"></span> > > >> + </a> <span id="statuses_info" > > >> class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> > > >> > > >> This is caused by assuming that all DAGs have start dates set, so a > > broken > > >> DAG will take down the whole UI. Not sure if we want to make this a > > blocker > > >> for the release or not, I'm guessing for most deployments this would > > occur > > >> pretty rarely. I'll submit a PR to fix it soon. > > >> > > >> > > >> > > >> On Tue, Feb 21, 2017 at 9:49 AM, Chris Riccomini < > criccom...@apache.org > > > > > >> wrote: > > >> > > >>> Ack that the vote has already passed, but belated +1 (binding) > > >>> > > >>> On Tue, Feb 21, 2017 at 7:42 AM, Bolke de Bruin <bdbr...@gmail.com> > > >>> wrote: > > >>> > > >>>> IPMC Voting can be found here: > > >>>> > > >>>> http://mail-archives.apache.org/mod_mbox/incubator-general/ > > >>> 201702.mbox/% > > >>>> 3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3e < > > >>>> http://mail-archives.apache.org/mod_mbox/incubator-general/ > > >>> 201702.mbox/% > > >>>> 3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3E> > > >>>> > > >>>> Kind regards, > > >>>> Bolke > > >>>> > > >>>>> On 21 Feb 2017, at 08:20, Bolke de Bruin <bdbr...@gmail.com> > wrote: > > >>>>> > > >>>>> Hello, > > >>>>> > > >>>>> Apache Airflow (incubating) 1.8.0 (based on RC4) has been accepted. > > >>>>> > > >>>>> 9 “+1” votes received: > > >>>>> > > >>>>> - Maxime Beauchemin (binding) > > >>>>> - Arthur Wiedmer (binding) > > >>>>> - Dan Davydov (binding) > > >>>>> - Jeremiah Lowin (binding) > > >>>>> - Siddharth Anand (binding) > > >>>>> - Alex van Boxel (binding) > > >>>>> - Bolke de Bruin (binding) > > >>>>> > > >>>>> - Jayesh Senjaliya (non-binding) > > >>>>> - Yi (non-binding) > > >>>>> > > >>>>> Vote thread (start): > > >>>>> http://mail-archives.apache.org/mod_mbox/incubator- > > >>>> airflow-dev/201702.mbox/%3cD360D9BE-C358-42A1-9188- > > >>>> 6c92c31a2...@gmail.com%3e <http://mail-archives.apache. > > >>>> org/mod_mbox/incubator-airflow-dev/201702.mbox/%3C7EB7B6D6- > > >>> 092E-48D2-AA0F- > > >>>> 15f44376a...@gmail.com%3E> > > >>>>> > > >>>>> Next steps: > > >>>>> 1) will start the voting process at the IPMC mailinglist. I do > expect > > >>>> some changes to be required mostly in documentation maybe a license > > here > > >>>> and there. So, we might end up with changes to stable. As long as > > these > > >>> are > > >>>> not (significant) code changes I will not re-raise the vote. > > >>>>> 2) Only after the positive voting on the IPMC and finalisation I > will > > >>>> rebrand the RC to Release. > > >>>>> 3) I will upload it to the incubator release page, then the tar > ball > > >>>> needs to propagate to the mirrors. > > >>>>> 4) Update the website (can someone volunteer please?) > > >>>>> 5) Finally, I will ask Maxime to upload it to pypi. It seems we can > > >>> keep > > >>>> the apache branding as lib cloud is doing this as well ( > > >>>> https://libcloud.apache.org/downloads.html#pypi-package < > > >>>> https://libcloud.apache.org/downloads.html#pypi-package>). > > >>>>> > > >>>>> Jippie! > > >>>>> > > >>>>> Bolke > > >>>> > > >>>> > > >>> > > >> > > >> > > >
