Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2
Max, I think you can close the vote? Bolke > On 27 Jun 2017, at 02:45, Kengo Seki wrote: > > +1 (non-binding) > > - verified signatures and checksums > - ran scheduler and webserver, confirmed they worked fine > - confirmed the latest fix on v1.8 branch (AIRFLOW-809) is included > > Kengo Seki > > > 2017-06-27 8:53 GMT+09:00 Chris Riccomini : >> +1 (binding) >> >> Been running in our dev env, and everything looks good. >> >> On Mon, Jun 26, 2017 at 3:00 PM, Alex Guziel >> wrote: >> >>> Yeah that makes sense. It pages by default at 500 so it explains why we saw >>> it. >>> >>> On Mon, Jun 26, 2017 at 2:53 PM, Chris Riccomini >>> wrote: >>> In 1.8.1, the "DAGs" page has "Show entries". In 1.8.2, it has "Show <25> entries". So it looks like prior to 1.8.2, the pagination was broken in the sense that it defaulted to the whole list. We have 479 DAGs in one env, and it shows them all. It looks like someone fixed the entry >>> to default to 25 now, which exposed the problem for our environments. On Mon, Jun 26, 2017 at 2:47 PM, Alex Guziel >>> invalid > wrote: > We're running 1.8.0 + some extras, and none of us added pagination > recently, and our homepage is paginated. Are you sure it's not the >>> number > of dags crossing the threshold? Maybe it's some Flask version thing? > > On Mon, Jun 26, 2017 at 2:45 PM, Chris Riccomini < >>> criccom...@apache.org> > wrote: > >> Yes, I did the 1.8.1 release. >> >> On Mon, Jun 26, 2017 at 2:44 PM, Alex Guziel >> . >> invalid >>> wrote: >> >>> There's no pagination in 1.8.1? Are you sure? >>> >>> On Mon, Jun 26, 2017 at 2:37 PM, Chris Riccomini < > criccom...@apache.org> >>> wrote: >>> It's not happening on 1.8.1 (since there's no pagination in that >>> version), so I'd count this as a regression. I wouldn't say it's blocking, but >> it's pretty ugly. On Mon, Jun 26, 2017 at 2:34 PM, Alex Guziel < alex.guz...@airbnb.com > . invalid > wrote: > I'm not so sure this is a new issue. I think we've seen it on >>> our > production for quite a while. > > On Mon, Jun 26, 2017 at 2:31 PM, Chris Riccomini < >>> criccom...@apache.org> > wrote: > >> I am seeing a strange UI behavior on 1.8.2.RC2. I've opened a > JIRA here: >> >> https://issues.apache.org/jira/browse/AIRFLOW-1348 >> >> Has anyone else seen this? >> >> On Mon, Jun 26, 2017 at 3:27 AM, Sumit Maheshwari < > sumeet.ma...@gmail.com> >> wrote: >> >>> +1, binding. >>> >>> >>> On Mon, Jun 26, 2017 at 3:49 PM, Bolke de Bruin < >> bdbr...@gmail.com >> wrote: >>> We have been running it for the last couple of days. No > issues >>> and >> seems more responsive. +1, binding Bolke > On 25 Jun 2017, at 01:10, Maxime Beauchemin < >>> maximebeauche...@gmail.com> wrote: > > Dear all, > > 1.8.2 RC2 is baked and available at: > https://dist.apache.org/repos/ >>> dist/dev/incubator/airflow , >>> public >> keys > are available > at https://dist.apache.org/repos/ >>> dist/release/incubator/airflow. > > Note that RC1 was the first RC (skipped RC0) and was never > announced since > it had issues coming out of the oven, so RC2 is the >>> first >>> public > RC. > > 1.8.2 RC2 is build on to of 1.8.1 with these listed >> "cherries" >>> on >> top. >>> I > added the JIRAs that were identified blockers and targeted 1.8.2. I > attempted to bring in all of the JIRAs that targeted 1.8.2 >> but > bailed >>> on > the ones that were generating merge conflicts. I also added >> all of >> the > JIRAs that we've been running in production at Airbnb. > > Issues fixed: > 9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator >> When >> Testing > Booleans > 333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all downstream >>> tasks > 93825d5 [AIRFLOW-XXX] Re-enable caching for hadoop > components > 33a9dcb [AIRFLOW-XXX] Pin Hive and Hadoop to a specific >> version and create > writable warehouse dir > 7cff6cd [AIRFLOW-1308] Disable nanny usage for Dask > 570b2ed [AIRFLO
Podling Report Reminder - July 2017
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 19 July 2017, 10:30 am PDT. The report for your podling will form a part of the Incubator PMC report. The Incubator PMC requires your report to be submitted 2 weeks before the board meeting, to allow sufficient time for review and submission (Wed, July 05). Please submit your report with sufficient time to allow the Incubator PMC, and subsequently board members to review and digest. Again, the very latest you should submit your report is 2 weeks prior to the board meeting. Thanks, The Apache Incubator PMC Submitting your Report -- Your report should contain the following: * Your project name * A brief description of your project, which assumes no knowledge of the project or necessarily of its field * A list of the three most important issues to address in the move towards graduation. * Any issues that the Incubator PMC or ASF Board might wish/need to be aware of * How has the community developed since the last report * How has the project developed since the last report. * How does the podling rate their own maturity. This should be appended to the Incubator Wiki page at: https://wiki.apache.org/incubator/July2017 Note: This is manually populated. You may need to wait a little before this page is created from a template. Mentors --- Mentors should review reports for their project(s) and sign them off on the Incubator wiki page. Signing off reports shows that you are following the project - projects that are not signed may raise alarms for the Incubator PMC. Incubator PMC
ExternalTaskSensor usage
Hello, I have a use case where a couple of dags(dag A & B) run at set intervals. Both dags have sensors that trigger a BashOperator. I have a third dag that is a long running process that needs to be started once(I use '@once'). This dag depends both dags A and B to be run at least once. So I added ExternalTaskSensors. Is there a better way to do this? I noticed that the ExternalTaskSensor poke() never succeeded. Looking at the code, there seems to be an explicit check for execution time match. https://github.com/apache/incubator-airflow/blob/7b620391a4e71fd19dff037a859dd39f132edb8c/airflow/operators/sensors.py#L246 Comment for "execution_delta" seems to suggest that this check should be for an execution time after certain reference. Should't the check be for "TI.execution_date > dttm,"? If this is a bug, I'd be happy to fix and PR. If not, what is the intent behind this?? Srikanth
Is it safe to run Airflow upgradedb command on same Airflow version?
Airflow has an upgradedb command that needs to be run when upgrading Airflow
versions. I wonder if it's safe to run even if the version is the same.
Wondering if that'd be okay to put part of my ansible deployment.
I've tried asking on Gitter and Stack Overflow but to no avail.
Code wise I see:
def upgradedb():
logging.info("Creating tables")
current_dir = os.path.dirname(os.path.abspath(__file__))
package_dir = os.path.normpath(os.path.join(current_dir, '..'))
directory = os.path.join(package_dir, 'migrations')
config = Config(os.path.join(package_dir, 'alembic.ini'))
config.set_main_option('script_location', directory)
config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN)
command.upgrade(config, 'heads')
Where only the last line I guess is what I have a hard time following:
command.upgrade(config, 'heads'). Where is the heads defined? For instance, if
I'm on version 1.8.0 and then I run this command, will it put the heads for the
most recent version of Airflow from git?
I don't think so but just want to make sure it's safe.
How foolish of me to not introduce myself.
My name is Ace Haidrey and I'm a Software Engineer at Pandora trying to
incorporate Airflow as part of our work management systems.
Cheers,
☺
