Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

> > > > [AIRFLOW-793] Enable compressed loading in S3ToHiveTransfer > > > > [AIRFLOW-863] Example DAGs should have recent start dates > > > > [AIRFLOW-869] Refactor mark success functionality > > > > [AIRFLOW-856] Make sure execution date is set for local client > > > > [AIRFLOW-814] Fix Presto*CheckOperator.__init__ > > > > [AIRFLOW-844] Fix cgroups directory creation > > > > > > > > No known issues anymore. > > > > > > > > I would also like to raise a VOTE for releasing 1.8.0 based on > release > > > > candidate 5, i.e. just renaming release candidate 5 to 1.8.0 release. > > > > > > > > Please respond to this email by: > > > > > > > > +1,0,-1 with *binding* if you are a PMC member or *non-binding* if > you > > > are > > > > not. > > > > > > > > Thanks! > > > > Bolke > > > > > > > > My VOTE: +1 (binding) > > > > > > -- _/ _/ Alex Van Boxel

Re: Reminder: Airflow Mini Hackathon 03/09 - 03/10 @ Airbnb HQ

r.kira...@airbnb.com by EOD Tuesday(03/07) > if you would like to participate. Also let me know if you have any > questions. > > Cheers, > > Gurer > -- _/ _/ Alex Van Boxel

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

gt;>>>> Date: > >>>>>>> {{last_run.start_date.strftime('%Y-%m-%d %H:%M')}}"> > >>>>>>> + >>>>>>> class="glyphicon glyphicon-info-sign" aria-hidden="true"> &g

Airflow pluggable components : example Logging

'GoogleCloudLogging = airflow_plugin.google_cloud:GoogleCloudLogging' ] }) I would generatlise this so we could make other parts (like for example connections) also pluggable. What do you think. Seems to work fine so far... and doesn't add any dependencies. Thoughts? -- _/ _/ Alex Van Boxel

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

> One issues have been fixed since release candidate 3: > > > > > > > > > > * mark success was not working properly > > > > > > > > > > No known issues anymore. > > > > > > > > > > I would also like to raise a VOTE for releasing 1.8.0 based on > > release > > > > > candidate 4, i.e. just renaming release candidate 4 to 1.8.0 > release. > > > > > > > > > > Please respond to this email by: > > > > > > > > > > +1,0,-1 with *binding* if you are a PMC member or *non-binding* if > > you > > > > are > > > > > not. > > > > > > > > > > Thanks! > > > > > Bolke > > > > > > > > > > My VOTE: +1 (binding) > > > > > > > > > -- _/ _/ Alex Van Boxel

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

; > > > > > > > > On 12 Feb 2017, at 12:26, Bolke de Bruin <bdbr...@gmail.com> > wrote: > > > > > > > > > > Ok great! Thanks! That sounds like a race condition: module not > > > > available yet at time of reading. I woul

Re: Contrib & Dataflow

directory to its own repo which would > keep the advantages but remove the disadvantages from core Airflow. Another > is to encourage individual airflow repos (Airflow-Docker, Airflow-Dataflow, > Airflow-YourExtensionHere) which could be installed a la carte. That would > leave maintenance up to the original author, but could lead to some > fracturing in the community as discovery becomes difficult. > -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 Release Candidate 1

966410226c1489bb730af3af45330fc51b9 > > I am still in the middle of running some quick test using celery executor, > will update the thread once it's done. > > > On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Hey Alex, > > Thanks fo

Re: Airflow 1.8.0 Release Candidate 1

dbr...@gmail.com> wrote: > Hey Alex, > > Thanks for tracking it down. Can you elaborate want went wrong with > celery? The lines below do not particularly relate to Celery directly, so I > wonder why we are not seeing it with LocalExecutor? > > Cheers > Bolke > > >

Re: Airflow 1.8.0 Release Candidate 1

g upgrade 5e7d17757c7a -> >> > 127d2bf2dfa7, Add dag_id/state index on dag_run table >> > >> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692: >> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the >> primary >> > key for table 'dag_stats', but has no Python-side or server-side default >> > generator indicated, nor does it indicate 'autoincrement=True' or >> > 'nullable=True', and no explicit value is passed. Primary key columns >> > typically may not store NULL. Note that as of SQLAlchemy 1.1, >> > 'autoincrement=True' must be indicated explicitly for composite (e.g. >> > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is >> > expected for one of the columns in the primary key. CREATE TABLE >> statements >> > are impacted by this change as well on most backends. >> > >> > > -- _/ _/ Alex Van Boxel

Re: Changelog 1.8

t; > Sent from my iPhone > -- _/ _/ Alex Van Boxel

Re: Airflow Logging Updates

received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. > -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

I identified my root cause: it's was a problem at my site with a faulty rebase. So all green. On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel <a...@vanboxel.be> wrote: > So bumped to RC1 and this seems fine. I don't get it. > > So it's a go. Sorry for the noise, but better safe the

Re: Airflow 1.8.0 BETA 5

Bruin <bdbr...@gmail.com> wrote: > And the scheduler doesn't log anything in beta 4? > > Sent from my iPhone > > > On 31 Jan 2017, at 19:26, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > I see it in the scheduler and in the UI. Currently trying to do a new

Re: Airflow 1.8.0 BETA 5

an 2017, at 18:06, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > > Hey Alex, > > > > Could it actually be that Airflow is doing the right thing? Earlier it > was swallowing the errors. Where do you see the errors? In the scheduler > logs, UI, processor logs? &g

Re: Airflow 1.8.0 BETA 5

> > I don't think the dag processor respects cli parameters. > > Bolke > > Sent from my iPhone > > > On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > It's quite hard to share my complete dags. I don't have this locally, > bu

Re: Airflow 1.8.0 BETA 5

64, in process_file m = imp.load_source(mod_name, filepath) File "/home/airflow/dags/marketing_segmentation.py", line 17, in import bqschema ImportError: No module named bqschema *I don't think this is incorrect?!* On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.da

Re: Airflow 1.8.0 BETA 5

fixed: > >> * Parsing errors not showing up in UI fixing a regression** > >> * Scheduler would terminate immediately if no dag files present > >> > >> ** As this touches the scheduler logic I though it warranted another > beta. > >> > >> This should be the last beta in my opinion and we can prepare changelog, > >> upgrade notes and release notes for the RC (Feb 2). > >> > >> Cheers > >> Bolke > -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 4

ptance > >>> environment?). > >>> > >>> I would like to encourage everyone to try it out, to report back any > >>> issues so we get to a rock solid release of 1.8.0. When reporting > issues a > >>> test case or even a fix is highly appreciated. > >>> > >>> Issues fixed: > >>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb > >>> should work again) > >>> * Py3 incompatibility in base_taskrunner. > >>> > >>> Under investigation: > >>> * DAG marked success, with half of the Tasks never scheduled (Alex) > >>> > >>> Kind regards, > >>> Bolke > >> > >> > >> > -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 3

remain > in the airflow directories. This worked for me: > > rm -rf site-packages/airflow* > python setup.py install > airflow initdb > > Bolke > > > > On 26 Jan 2017, at 10:53, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > About the database e

Re: Airflow 1.8.0 BETA 3

About the database error: starting from scratch also gives the same error: Fresh install. Delete airflow.db sqllite db. And the : airflow initdb same error as above. On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <a...@vanboxel.be> wrote: > Not directly one I can share. I'll spend

Re: Airflow 1.8.0 BETA 3

my iPhone > > > On 26 Jan 2017, at 08:36, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > Another thing that I noticed (but observed it in beta 2 as well). Is the > > following: > > > > - The following trigger should not fire. > > --- Trigger rule is

Re: Airflow 1.8.0 BETA 3

The downgrade is ok for testing, but we can't release with this change (can't expect people to install a beta first). On Thu, Jan 26, 2017 at 8:36 AM Alex Van Boxel <a...@vanboxel.be> wrote: > Another thing that I noticed (but observed it in beta 2 as well). Is the &g

Re: Airflow 1.8.0 BETA 3

de Bruin <bdbr...@gmail.com> wrote: > Mmm that is due to the reverting of one changes to the db. Need to look > into that how to fix it. > > Sent from my iPhone > > > On 26 Jan 2017, at 00:51, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > I do seem to hav

Re: Airflow 1.8.0 BETA 3

while task has exited > * Keep cgroups optional > * Funcsigs pinned to 1.0.0 > > Issue(s) remaining (blocker for RC): > * Cgroups not py3 compatible > > If all goes well we should have a Release Candidate on Feb 2. Thanks for > reporting issues and keep on testing please :). Moving towards RC I tend to > like small bug fixes only. When we mark RC (do we need to vote on this?) > the procedure becomes even more strict. Please remember that the FINAL > release is dependent on a vote on the IPMC mailinglist. > > Cheers > Bolke -- _/ _/ Alex Van Boxel

Medium series: Airflow for Google Cloud

DataProc. -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 alpha 5

itted): > * set default log location for Childs correctly > * fix systems scripts > * daemonizing of webserver > > Almost there! > Bolke -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 alpha 4

up : minor changes needed, documentation still required, > integration tests seem to pass flawlessly > > * Cgroups + impersonation: clean up of patches on going, more tests and > more elaborate documentation required. Integration tests not executed yet > > * Schedule all pending DAG runs in a single scheduler loop: no progress > (**) > > > > Cheers! > > Bolke > > -- _/ _/ Alex Van Boxel

Re: Refactoring Connection

andle all those cases. > > Le lun. 9 janv. 2017 à 13:36, Alex Van Boxel <a...@vanboxel.be> a écrit : > > > Thanks a lot, yes it clarifies a lot and I do agree you really need to > hack > > inside Airflow to add a Connection type. While you're working at this > could &

Re: Refactoring Connection

d connection type, but I though this might be > something that comes back regularly and having a simple way to plug in new > types of connection would make it easier for anyone to contribute a new > connection type. > > Hope this clarifies my proposal. > > Le lun. 9 janv.

Re: Refactoring Connection

mpler to understand but > requires more work every time you create a new ConnectionType. > > Hope this proposal is clear enough, and I'm waiting for feebacks and > possible improvements. > > Regards > Gael Magnan de Bornier > -- _/ _/ Alex Van Boxel

Re: Airflow Release Planning and Supported Release Lifetime

This looks good, except do we need a release manager that applies patches? On Sun, Jan 8, 2017, 14:36 Bolke de Bruin wrote: > Hi All, > > As part of the release process I have created "Airflow Release Planning > and Supported Release Lifetime” ( >

Trigger behaviour with skipped upstream tasks (request for opinions)

her PR that introduces 2 new triggers: -- ALL_SUCCESS_OR_SKIPPED -- ALL_FAILED_OR_SKIPPED What do you think? -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 alpha 2

> > > > Bolke, can you describe the current state of the alpha 2 release? I saw > > > some comments from Alex yesterday about celery instability. If I'm > > running > > > on LocalExecutor, should I be seeing any issues? > > > > > > On Wed, Jan 4, 201

Re: Wrong DAG state after failure inside a branch

task_id='process2', > python_callable=lambda: True, > dag=dag) > process2.set_upstream(process1) > > process3 = PythonOperator( > task_id='process3', > python_callable=lambda: True, > dag=dag) > process3.set_upstream(process2) > > > At moment, I want my privacy to be protected. > https://mytemp.email/ > -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 Alpha 1

should > > > >>> by no means be considered an Apache release yet. This is for > testing > > > >>> purposes in the dev community around Airflow, nothing else. > > > >>> > > > >>> The build is exactly the same as the state of master (git 410736d) > > plus > > > >> the > > > >>> change to version “1.8.0.alpha1” in version.py. > > > >>> > > > >>> I am dedicating quite some time next week and beyond to get a > release > > > >> out. > > > >>> Hopefully we can get some help with testing, changelog etc. To make > > > this > > > >>> possible I would like to propose a freeze to adding new features > for > > at > > > >>> least two weeks - say until Jan 15. > > > >>> > > > >>> You can find the tar here: http://people.apache.org/~bolke/ < > > > http://people.apache.org/~bolke/> < > > > >>> http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/ > > > > < > > > http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/>>> > . > > > >> It isn’t signed. Following versions > > > >>> will be. SHA is available. > > > >>> > > > >>> Lastly, Alpha 1 does not have the fix for retries yet. So we will > get > > > an > > > >>> Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in > > > >>> https://github.com/apache/incubator-airflow/pull/1948 < > > > https://github.com/apache/incubator-airflow/pull/1948> < > > > >> https://github.com/apache/incubator-airflow/pull/1948 < > > > https://github.com/apache/incubator-airflow/pull/1948>> < > > > >>> https://github.com/apache/incubator-airflow/pull/1948 < > > > https://github.com/apache/incubator-airflow/pull/1948> < > > > >> https://github.com/apache/incubator-airflow/pull/1948 < > > > https://github.com/apache/incubator-airflow/pull/1948>>> , but your > > > >> feedback > > > >>> is required as it is entrenched in new processing code that you are > > > >> running > > > >>> in production afaik - so I wonder what happens in your fork. > > > >>> > > > >>> Happy New Year! > > > >>> > > > >>> Bolke > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> _/ > > > >>> _/ Alex Van Boxel > > > > > > > > > -- _/ _/ Alex Van Boxel

Re: Airflow 1.8.0 Alpha 1

an review the above PRs and add your own as well > > then I can create another Alpha version. I’ll be on gitter as much as I > can > > so we can speed up if needed. > > > > - Bolke > > > > > On 3 Jan 2017, at 08:51, Alex Van Boxel <a...@vanboxel.be> wrot

Re: Airflow 1.8.0 Alpha 1

rflow/pull/1948> , but your feedback is required as it is entrenched in new processing code that you are running in production afaik - so I wonder what happens in your fork. Happy New Year! Bolke -- _/ _/ Alex Van Boxel

Re: Scheduler error - cannot allocate memory

> > _launch > > > > > >self.pid = > > > os.fork() > > > > > > OSError: [Errno 12] Cannot allocate > > > memory > > > > > > Traceback (most recent call last): > > > File "/usr/local/bin/airflow", line 15, in > > > > > > > > > The dags which failed didn't show any log (there weren't stored on > > airflow > > > instance and there is no remote logs). So we don't have any idea of > what > > > would happened (only that there was not enough memory to fork) > > > It is well known that is recommended to restart the scheduler > > periodically > > > (according to this > > > <https://medium.com/handy-tech/airflow-tips-tricks-and- > > pitfalls-9ba53fba14eb#.80c6g1n1s>), > > > but... do you have any idea why this can happen? Is there something we > > can > > > do (or some bug we can fix)? > > > > > > > > > Thanks in advance! > > > -- _/ _/ Alex Van Boxel

Re: Integration test env

can also fund on AWS/GCloud. Again, I probably want to have ultimate > > > power on access to this environment - it’s my company’s money on the > line > > > after all. Major downside to this is that it is dependent on and > limited > > by > > > the budget I can make available. Upside is that it is not company > > property. > > > Also I personally have less exposure to public cloud environments due > to > > > company restrictions. > > > > > > Are there any other options? Any thoughts? > > > > > > Bolke > > > > > > > > > > > > > > > > > > > > > -- _/ _/ Alex Van Boxel

Re: Airflow-GCP for Google Container Engine

Are you managing any resources on the pods, > or you let the workers consume whatever's available on the nodes? > > > On Fri, Dec 2, 2016 at 10:14 PM, Chris Riccomini <criccom...@apache.org> > wrote: > > > Nice, thanks! :) > > > > On Fri, Dec 2, 2016 at 4:55 AM,

Re: Merging the experimental API Framework

Although I haven't had the time to dive deep into API (sorry Bolke) I do want to be part of the discussion. I hope to have a look at it soon. On Tue, Nov 29, 2016 at 10:43 AM Bolke de Bruin wrote: > Flask App Builder looks great at a first glance and experience counts >

Re: Next Release?

I thought that the 15 November deadline for PR was in preparation for the 1.8 release. Do you need help with the release? I'm dedicating each week some time on Airflow anyway (although it's more writing operators :-). On Thu, Oct 27, 2016 at 6:22 PM siddharth anand wrote: > I

Re: Airflow Logging

My requirement would indeed be that I would be able to add my own logging handler (I did the same in my Luigi days), I included a python log handler that logged to Google Cloud Logging. But I also like the current logging to Cloud storage. So my ideal logging setup would be: All of Airflow

Cloud Provider grouping into Plugins

Hi all, I'm starting to write some very exotic Operators that are a bit strange adding to contrib. Examples of this are: + See if a Compute snapshot of a disc is created + See if a string appears on the serial port of Compute instance but they would be a nice addition if we had a Google Compute

Re: Next Airflow meet-up

> > > >> >>>> On Fri, Oct 7, 2016 at 1:08 PM, Chris Riccomini > > > >> >>>> <criccom...@apache.org> > > > >> >>>> wrote: > > > >> >>>>> @sid Awesome! Will create the meetup event on Monday.

Re: Next Airflow meet-up

to anything. > >> It's basically our busiest time of the year. > >> > >> I have to nail a few things down but my tentative plan was for December > >> 7th. > >> > >> If enough people prefer November 16th, Clover Health could do the 17Q1 > >> meetup

Re: Next Airflow meet-up

Hey guys, about the date. There is good chance I'm in SF for summit (14-15 November), I could try to extend it a day (so including November 16th). So if you're looking for a date, think about November 16th (I'll volunteer for a talk then). Would be great. Thanks. On Sat, Oct 1, 2016 at 8:51 AM

Re: Airflow Releases

I'll do the same. Nice to have the 1.8 on the horizon. On Fri, Sep 30, 2016 at 5:51 PM Chris Riccomini wrote: > > I'm not sure how other projects do this, but I propose that we let the > RC settle for a week. > > +1 > > > Who's on board!? > > Me. :) > > On Fri, Sep 30,

Re: Airflow Developers Meeting - 08/03 Notes

t; (Sorry for the late response, I was on holiday) > > I think the G* operators just need to be cherry picked. This will make us > deviate slightly from the > previous release, but makes sure we don’t have to ‘fix’ history afterwards. > > Anyone against this? > > - B. > > &g