Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

[Alex Van Boxel]

+1 (binding)

Note: we had to revert all our ONE_SUCCESS with ALL_SUCCESS trigger rules
where the parent nodes where joining with a SKIP. But I can of should have
known this was coming. Apart of that I had a successful run last night.


On Tue, Mar 14, 2017 at 1:37 AM siddharth anand  wrote:

I'm going to deploy this to staging now. Fab work Bolke!
-s

On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov  wrote:

> I'll test this on staging as soon as I get a chance (the testing is
> non-blocking on the rc5). Bolke very much in particular :).
>
> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin 
> wrote:
>
> > +1 (binding) extremely impressed by the work and diligence all
> contributors
> > have put in to getting these blockers fixed, Bolke in particular.
> >
> > On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer 
> wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks again for steering us through Bolke.
> > >
> > > Best,
> > > Arthur
> > >
> > > On Sun, Mar 12, 2017 at 9:59 PM, Bolke de Bruin 
> > wrote:
> > >
> > > > Dear All,
> > > >
> > > > Finally, I have been able to make the FIFTH RELEASE CANDIDATE of
> > Airflow
> > > > 1.8.0 available at: https://dist.apache.org/repos/
> > > > dist/dev/incubator/airflow/ <https://dist.apache.org/
> > > > repos/dist/dev/incubator/airflow/> , public keys are available at
> > > > https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> > > > https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
> is
> > > > tagged with a local version “apache.incubating” so it allows
> upgrading
> > > from
> > > > earlier releases.
> > > >
> > > > Issues fixed since rc4:
> > > >
> > > > [AIRFLOW-900] Double trigger should not kill original task instance
> > > > [AIRFLOW-900] Fixes bugs in LocalTaskJob for double run protection
> > > > [AIRFLOW-932] Do not mark tasks removed when backfilling
> > > > [AIRFLOW-961] run onkill when SIGTERMed
> > > > [AIRFLOW-910] Use parallel task execution for backfills
> > > > [AIRFLOW-967] Wrap strings in native for py2 ldap compatibility
> > > > [AIRFLOW-941] Use defined parameters for psycopg2
> > > > [AIRFLOW-719] Prevent DAGs from ending prematurely
> > > > [AIRFLOW-938] Use test for True in task_stats queries
> > > > [AIRFLOW-937] Improve performance of task_stats
> > > > [AIRFLOW-933] use ast.literal_eval rather eval because
> ast.literal_eval
> > > > does not execute input.
> > > > [AIRFLOW-919] Running tasks with no start date shouldn't break a
DAGs
> > UI
> > > > [AIRFLOW-897] Prevent dagruns from failing with unfinished tasks
> > > > [AIRFLOW-861] make pickle_info endpoint be login_required
> > > > [AIRFLOW-853] use utf8 encoding for stdout line decode
> > > > [AIRFLOW-856] Make sure execution date is set for local client
> > > > [AIRFLOW-830][AIRFLOW-829][AIRFLOW-88] Reduce Travis log verbosity
> > > > [AIRFLOW-794] Access DAGS_FOLDER and SQL_ALCHEMY_CONN exclusively
> from
> > > > settings
> > > > [AIRFLOW-694] Fix config behaviour for empty envvar
> > > > [AIRFLOW-365] Set dag.fileloc explicitly and use for Code view
> > > > [AIRFLOW-931] Do not set QUEUED in TaskInstances
> > > > [AIRFLOW-899] Tasks in SCHEDULED state should be white in the UI
> > instead
> > > > of black
> > > > [AIRFLOW-895] Address Apache release incompliancies
> > > > [AIRFLOW-893][AIRFLOW-510] Fix crashing webservers when a dagrun has
> no
> > > > start date
> > > > [AIRFLOW-793] Enable compressed loading in S3ToHiveTransfer
> > > > [AIRFLOW-863] Example DAGs should have recent start dates
> > > > [AIRFLOW-869] Refactor mark success functionality
> > > > [AIRFLOW-856] Make sure execution date is set for local client
> > > > [AIRFLOW-814] Fix Presto*CheckOperator.__init__
> > > > [AIRFLOW-844] Fix cgroups directory creation
> > > >
> > > > No known issues anymore.
> > > >
> > > > I would also like to raise a VOTE for releasing 1.8.0 based on
> release
> > > > candidate 5, i.e. just renaming release candidate 5 to 1.8.0
release.
> > > >
> > > > Please respond to this email by:
> > > >
> > > > +1,0,-1 with *binding* if you are a PMC member or *non-binding* if
> you
> > > are
> > > > not.
> > > >
> > > > Thanks!
> > > > Bolke
> > > >
> > > > My VOTE: +1 (binding)
> > >
> >
>

-- 
  _/
_/ Alex Van Boxel

Re: Reminder: Airflow Mini Hackathon 03/09 - 03/10 @ Airbnb HQ

[Alex Van Boxel]

Hey Gurer,

can I ask *not* to work on this, because I'm currently working in a major
rework for logging, because I'm working on consolidated logging and
integration with Google Cloud.

 - Streaming support for S3 logs in the webserver.
 - Realtime logs for the webserver.

This will be done through a plugin that should easiliy then be applied to
S3 or whatever logging system you have. It's quite some work to keep it
backward compatible so if you work on this it could conflict with the
plugin and I then need to refactor again.

That said, I'm in SF for Google Cloud NEXT, so maybe I'll jump in to say hi
:-)



On Fri, Mar 3, 2017 at 8:29 PM Gurer Kiratli
 wrote:

> Hi folks,
>
> As a reminder we are going to have an Airflow Hackathon next
> Thursday(03/09) and Friday(03/10) at the Airbnb HQ at San Francisco on 888
> Brannan St.
>
> *Logistics*
> Couple folks have already reached out to me and one common question was if
> this is going to be a day time thing. Yes it will be during regular hours.
> We are going to start at 9:15am with optional breakfast we will commence
> the event at 10am. We can decide how long we want to go when we are here,
> if people wanna work through late night on Thursday, we can do that as
> well. We will end on Friday at around 5pm.
>
> We will provide the meals, snacks, and the beverages.
>
> *Possible Projects*
> As the Airbnb team we met internally and here are some things that we think
> might be interesting to work on:
>
>- Performance Improvements
>   - Profiling support for the webserver.
>   - Homepage, task instance page loading times performance
>   improvements.
>   - Improve Airflow DB queries’ performance.
>- Making the scheduler schedule tasks faster.
>- Make a tool for restarting tasks after system failures/outages.
>- Improving test coverage for core components.
>- Creating REST APIs for Airflow operations.
>- Improve UI to enable operations like cancel, mark success, clear for
>all the tasks of the DAG from the UI.
>- Streaming support for S3 logs in the webserver.
>- Realtime logs for the webserver.
>
> Also you can check Airflow 2017 roadmap document
> <https://cwiki.apache.org/confluence/display/AIRFLOW/2017+Roadmap+Items>
> for
> other interesting things to work on and you can see the community survey
> results <https://www.surveymonkey.com/results/SM-99G9YHS3/> on the roadmap
> items' prioritization.
>
> *How to participate? Questions?*
> Please reach out to me at gurer.kira...@airbnb.com  by EOD Tuesday(03/07)
> if you would like to participate. Also let me know if you have any
> questions.
>
> Cheers,
>
> Gurer
>
-- 
  _/
_/ Alex Van Boxel

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

[Alex Van Boxel]

I think what I observed was doesn't every time otherwise we would see it
every time. I'll see if it happens this night again.

On Sat, Feb 25, 2017 at 1:24 PM Jeremiah Lowin  wrote:

> Interesting if this is related to what I was seeing -- but to be clear the
> error I observed is non-deterministic and doesn't happen every time
> (obviously, because otherwise there would be no passing Travis runs). Is
> that the case for what you're describing, Dan/Alex?
>
> On Sat, Feb 25, 2017 at 4:13 AM Alex Van Boxel  wrote:
>
> About:  Skipped tasks potentially cause a dagrun to be marked
> failure/success prematurely. Isn't that related to the discussion I had
> with Max about the ONE_SUCCESS trigger? When skipping tasks for now you
> need to put ONE_SUCCESS. I had kind of a fix but it was rejected because it
> changed behaviour.
>
> On Sat, Feb 25, 2017 at 9:19 AM Bolke de Bruin  wrote:
>
> > Not trying to muddy the waters, but the observation of Jeremiah (non
> > deterministic outcomes) might have to do something with #3. I didn’t dive
> > in deeper, yet.
> >
> > ==
> > ERROR: test_backfill_examples (tests.BackfillJobTest)
> > --
> > Traceback (most recent call last):
> >   File "/home/travis/build/apache/incubator-airflow/tests/jobs.py", line
> > 164, in test_backfill_examples
> > job.run()
> >   File "/home/travis/build/apache/incubator-airflow/airflow/jobs.py",
> line
> > 200, in run
> > self._execute()
> >   File "/home/travis/build/apache/incubator-airflow/airflow/jobs.py",
> line
> > 1999, in _execute
> > raise AirflowException(err)
> > AirflowException: ---
> > Some task instances failed:
> > set([('example_short_circuit_operator', 'condition_is_True',
> > datetime.datetime(2016, 1, 1, 0, 0))])
> > https://s3.amazonaws.com/archive.travis-ci.org/jobs/204780706/log.txt <
> > https://s3.amazonaws.com/archive.travis-ci.org/jobs/204780706/log.txt>
> >
> > Bolke
> >
> > > On 25 Feb 2017, at 09:07, Bolke de Bruin  wrote:
> > >
> > > Hi Dan,
> > >
> > > - Backfill indeed runs only one dagrun at the time, see line 1755 of
> > jobs.py. I’ll think about how to fix this over the weekend (I think it
> was
> > my change that introduced this). Suggestions always welcome. Depending
> the
> > impact it is a blocker or not. We don’t often use backfills and
> definitely
> > not at your size, so that is why it didn’t pop up with us. I’m assuming
> > blocker for now, btw.
> > > - Speculation on the High DB Load. I’m not sure what your benchmark is
> > here (1.7.1 + multi processor dags?), but as you mentioned in the code
> > dependencies are checked a couple of times for one run and even task
> > instance. Dependency checking requires aggregation on the DB, which is a
> > performance killer. Annoying but not a blocker.
> > > - Skipped tasks potentially cause a dagrun to be marked failure/success
> > prematurely. BranchOperators are widely used if it affects these
> operators,
> > then it is a blocker.
> > >
> > > - Bolke
> > >
> > >> On 25 Feb 2017, at 02:04, Dan Davydov  .INVALID>
> > wrote:
> > >>
> > >> Update on old pending issues:
> > >> - Black Squares in UI: Fix merged
> > >> - Double Trigger Issue That Alex G Mentioned: Alex has a PR in flight
> > >>
> > >> New Issues:
> > >> - Backfill seems to be having issues (only running one dagrun at a
> > time),
> > >> we are still investigating - might be a blocker
> > >> - High DB Load (~8x more than 1.7) - We are still investigating but
> it's
> > >> probably not a blocker for the release
> > >> - Skipped tasks potentially cause a dagrun to be marked as
> > failure/success
> > >> prematurely - not sure whether or not to classify this as a blocker
> > (only
> > >> really an issue for users who use the BranchingPythonOperator, which
> > AirBnB
> > >> does)
> > >>
> > >> On Thu, Feb 23, 2017 at 5:59 PM, siddharth anand 
> > wrote:
> > >>
> > >>> IMHO, a DAG run without a start date is non-sensical but is not
> > enforced
> > >>> That said, our UI allows for the manual creation of DAG Runs without
> a
> > >>> start d

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

[Alex Van Boxel]

;>> Let's see if anyone else reports the issue. If no one does, one
> option is
> >>>> to release 1.8.0 as is with a comment in the release notes, and have a
> >>>> future official minor apache release 1.8.1 that would fix these minor
> >>>> issues that are not deal breaker.
> >>>>
> >>>> @bolke, I'm curious, how long does it take you to go through one
> release
> >>>> cycle? Oh, and do you have a documented step by step process for
> >>> releasing?
> >>>> I'd like to add the Pypi part to this doc and add committers that are
> >>>> interested to have rights on the project on Pypi.
> >>>>
> >>>> Max
> >>>>
> >>>> On Wed, Feb 22, 2017 at 2:00 PM, Bolke de Bruin 
> >>> wrote:
> >>>>
> >>>>> So it is a database integrity issue? Afaik a start_date should always
> >>> be
> >>>>> set for a DagRun (create_dagrun) does so  I didn't check the code
> >>> though.
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 22 Feb 2017, at 22:19, Dan Davydov  >>> INVALID>
> >>>>> wrote:
> >>>>>>
> >>>>>> Should clarify this occurs when a dagrun does not have a start date,
> >>>> not
> >>>>> a
> >>>>>> dag (which makes it even less likely to happen). I don't think this
> >>> is
> >>>> a
> >>>>>> blocker for releasing.
> >>>>>>
> >>>>>>> On Wed, Feb 22, 2017 at 1:15 PM, Dan Davydov <
> >>> dan.davy...@airbnb.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> I rolled this out in our prod and the webservers failed to load due
> >>> to
> >>>>>>> this commit:
> >>>>>>>
> >>>>>>> [AIRFLOW-510] Filter Paused Dags, show Last Run & Trigger Dag
> >>>>>>> 7c94d81c390881643f94d5e3d7d6fb351a445b72
> >>>>>>>
> >>>>>>> This fixed it:
> >>>>>>> -  >>>>>>> class="glyphicon glyphicon-info-sign" aria-hidden="true"
> >>> title="Start
> >>>>> Date:
> >>>>>>> {{last_run.start_date.strftime('%Y-%m-%d %H:%M')}}">
> >>>>>>> +  >>>>>>> class="glyphicon glyphicon-info-sign" aria-hidden="true">
> >>>>>>>
> >>>>>>> This is caused by assuming that all DAGs have start dates set, so a
> >>>>> broken
> >>>>>>> DAG will take down the whole UI. Not sure if we want to make this a
> >>>>> blocker
> >>>>>>> for the release or not, I'm guessing for most deployments this
> would
> >>>>> occur
> >>>>>>> pretty rarely. I'll submit a PR to fix it soon.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 21, 2017 at 9:49 AM, Chris Riccomini <
> >>>> criccom...@apache.org
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Ack that the vote has already passed, but belated +1 (binding)
> >>>>>>>>
> >>>>>>>> On Tue, Feb 21, 2017 at 7:42 AM, Bolke de Bruin <
> bdbr...@gmail.com
> >>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> IPMC Voting can be found here:
> >>>>>>>>>
> >>>>>>>>> http://mail-archives.apache.org/mod_mbox/incubator-general/
> >>>>>>>> 201702.mbox/%
> >>>>>>>>> 3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3e <
> >>>>>>>>> http://mail-archives.apache.org/mod_mbox/incubator-general/
> >>>>>>>> 201702.mbox/%
> >>>>>>>>> 3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3E>
> >>>>>>>>>
> >>>>>>>>> Kind regards,
> >>>>>>>>> Bolke
> >>>>>>>>>
> >>>>>>>>>> On 21 Feb 2017, at 08:20, Bolke de Bruin 
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> Apache Airflow (incubating) 1.8.0 (based on RC4) has been
> >>> accepted.
> >>>>>>>>>>
> >>>>>>>>>> 9 “+1” votes received:
> >>>>>>>>>>
> >>>>>>>>>> - Maxime Beauchemin (binding)
> >>>>>>>>>> - Arthur Wiedmer (binding)
> >>>>>>>>>> - Dan Davydov (binding)
> >>>>>>>>>> - Jeremiah Lowin (binding)
> >>>>>>>>>> - Siddharth Anand (binding)
> >>>>>>>>>> - Alex van Boxel (binding)
> >>>>>>>>>> - Bolke de Bruin (binding)
> >>>>>>>>>>
> >>>>>>>>>> - Jayesh Senjaliya (non-binding)
> >>>>>>>>>> - Yi (non-binding)
> >>>>>>>>>>
> >>>>>>>>>> Vote thread (start):
> >>>>>>>>>> http://mail-archives.apache.org/mod_mbox/incubator-
> >>>>>>>>> airflow-dev/201702.mbox/%3cD360D9BE-C358-42A1-9188-
> >>>>>>>>> 6c92c31a2...@gmail.com%3e <http://mail-archives.apache.
> >>>>>>>>> org/mod_mbox/incubator-airflow-dev/201702.mbox/%3C7EB7B6D6-
> >>>>>>>> 092E-48D2-AA0F-
> >>>>>>>>> 15f44376a...@gmail.com%3E>
> >>>>>>>>>>
> >>>>>>>>>> Next steps:
> >>>>>>>>>> 1) will start the voting process at the IPMC mailinglist. I do
> >>>> expect
> >>>>>>>>> some changes to be required mostly in documentation maybe a
> >>> license
> >>>>> here
> >>>>>>>>> and there. So, we might end up with changes to stable. As long as
> >>>>> these
> >>>>>>>> are
> >>>>>>>>> not (significant) code changes I will not re-raise the vote.
> >>>>>>>>>> 2) Only after the positive voting on the IPMC and finalisation I
> >>>> will
> >>>>>>>>> rebrand the RC to Release.
> >>>>>>>>>> 3) I will upload it to the incubator release page, then the tar
> >>>> ball
> >>>>>>>>> needs to propagate to the mirrors.
> >>>>>>>>>> 4) Update the website (can someone volunteer please?)
> >>>>>>>>>> 5) Finally, I will ask Maxime to upload it to pypi. It seems we
> >>> can
> >>>>>>>> keep
> >>>>>>>>> the apache branding as lib cloud is doing this as well (
> >>>>>>>>> https://libcloud.apache.org/downloads.html#pypi-package <
> >>>>>>>>> https://libcloud.apache.org/downloads.html#pypi-package>).
> >>>>>>>>>>
> >>>>>>>>>> Jippie!
> >>>>>>>>>>
> >>>>>>>>>> Bolke
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >
>
> --
  _/
_/ Alex Van Boxel

Airflow pluggable components : example Logging

[Alex Van Boxel]

Hi all,

I'm doing some work to get Google Cloud Logging into Airflow, but without
it actually being implemented into Airflow. For this I used a simple
pluggable discovery system using standard Python setuptools. Here is what
I'm talking about:

In Airflow I reworked all longing config in an AirflowInternalLogging
component. The AirflowInternalLogging is written as a discoverable
compoment and needed to be added to the setup.py:

entry_points={
'airflow.logging': [
'AirflowInternalLogging = airflow.defaults:AirflowInternalLogging'
]

That's all that is needed to make it discoverable. In the code I just need
to load it via:

def load_airflow_component(group, name):
from pkg_resources import iter_entry_points
print "Discovery of '{}' components".format(group)
for entry_point in iter_entry_points(group=group, name=None):
if entry_point.name == name:
component = entry_point
print("- {}".format(entry_point))
print "Loading '{}' for '{}'".format(name, group)
print component
instance = component.load()
return instance()

That's it. Now in a completly other project with it's own setup I can add
other implementations (with the same methods) of the component and make
them selectable via the config of that project:

setup(name='airflow_plugin.google_cloud',
  version='0.1.0',
  packages=['airflow_plugin.google_cloud'],
  entry_points={
  'airflow.logging': [
  'GoogleCloudLogging =
airflow_plugin.google_cloud:GoogleCloudLogging'
  ]
  })

I would generatlise this so we could make other parts (like for example
connections) also pluggable.

What do you think. Seems to work fine so far... and doesn't add any
dependencies.

Thoughts?



-- 
  _/
_/ Alex Van Boxel

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

[Alex Van Boxel]

+1 (binding)

On Mon, Feb 20, 2017 at 5:32 AM y...@yahoo-inc.com.INVALID
 wrote:

>
> +1 (non-binding)
>
> Thanks for all the work!
>
> YiOn Sunday, February 19, 2017, 12:52:31 PM PST, Arthur Wiedmer <
> arthur.wied...@gmail.com> wrote:+1 (binding)
>
> Thanks again for all the work!
>
> Best,
> Arthur
>
> On Fri, Feb 17, 2017 at 4:46 PM, Jeremiah Lowin  wrote:
>
> > +1 (binding) many thanks for all your work on this Bolke!
> >
> > On Fri, Feb 17, 2017 at 7:10 PM Jayesh Senjaliya 
> > wrote:
> >
> > +1 ( non-binding )works fine for me.
> >
> >
> >
> > On Fri, Feb 17, 2017 at 3:37 PM, Maxime Beauchemin <
> > maximebeauche...@gmail.com> wrote:
> >
> > > +1 (binding)
> > >
> > > On Fri, Feb 17, 2017 at 11:33 AM, Dan Davydov <
> > > dan.davy...@airbnb.com.invalid> wrote:
> > >
> > > > +1 (binding). Mark success works great now, thanks to Bolke for
> fixing.
> > > >
> > > > On Fri, Feb 17, 2017 at 12:22 AM, Bolke de Bruin 
> > > > wrote:
> > > >
> > > > > Dear All,
> > > > >
> > > > > I have made the FOURTH RELEASE CANDIDATE of Airflow 1.8.0 available
> > at:
> > > > > https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > > > https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> public
> > > keys
> > > > > are available at https://dist.apache.org/repos/
> > > > > dist/release/incubator/airflow/ <https://dist.apache.org/repos
> > > > > /dist/release/incubator/airflow/> . It is tagged with a local
> > version
> > > > > “apache.incubating” so it allows upgrading from earlier releases.
> > > > >
> > > > > One issues have been fixed since release candidate 3:
> > > > >
> > > > > * mark success was not working properly
> > > > >
> > > > > No known issues anymore.
> > > > >
> > > > > I would also like to raise a VOTE for releasing 1.8.0 based on
> > release
> > > > > candidate 4, i.e. just renaming release candidate 4 to 1.8.0
> release.
> > > > >
> > > > > Please respond to this email by:
> > > > >
> > > > > +1,0,-1 with *binding* if you are a PMC member or *non-binding* if
> > you
> > > > are
> > > > > not.
> > > > >
> > > > > Thanks!
> > > > > Bolke
> > > > >
> > > > > My VOTE: +1 (binding)
> > > >
> > >
> >

-- 
  _/
_/ Alex Van Boxel

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

[Alex Van Boxel]

+1 (binding)

On Mon, Feb 13, 2017 at 7:45 PM siddharth anand  wrote:

> +1 (binding)
>
> On Mon, Feb 13, 2017 at 8:57 AM, Chris Riccomini 
> wrote:
>
> > +1 (binding)
> >
> > On Sun, Feb 12, 2017 at 8:54 AM, Jeremiah Lowin 
> wrote:
> >
> > > Interesting -- I also run on Kubernetes with a git-sync sidecar, but
> the
> > > containers wait for the synced repo to apprar before starting since it
> > > contains some dependencies -- I assume that's why I didn't experience
> the
> > > same issue.
> > >
> > > On Sun, Feb 12, 2017 at 6:29 AM Bolke de Bruin 
> > wrote:
> > >
> > > > Although the race condition doesn't explain why “num_runs = None”
> > > resolved
> > > > the issue for you earlier, but it does give a clue now: the PR that
> > > > introduced “num_runs = -1” was there to be able to work with empty
> dag
> > > > dirs, maybe it wasn’t fully covered yet.
> > > >
> > > > Bolke
> > > >
> > > > > On 12 Feb 2017, at 12:26, Bolke de Bruin 
> wrote:
> > > > >
> > > > > Ok great! Thanks! That sounds like a race condition: module not
> > > > available yet at time of reading. I would expect that it resolves
> > itself
> > > > after a while.
> > > > >
> > > > > After talking to some people at the Warsaw BigData conf I have some
> > > > ideas around syncing dags, Spoiler: no dependency on git.
> > > > >
> > > > > - Bolke
> > > > >
> > > > >> On 12 Feb 2017, at 11:17, Alex Van Boxel 
> wrote:
> > > > >>
> > > > >> Running ok, in staging... @bolke I'm running patch-less. I've
> > switched
> > > > my
> > > > >> Kubernetes from:
> > > > >>
> > > > >> - each container (webserver/scheduler/worker) had a git-sync'er
> > > (getting
> > > > >> the dags from git)
> > > > >>> this meant that the scheduler had 0 dags at startup, and should
> > have
> > > > >> picked them up later
> > > > >>
> > > > >> to
> > > > >>
> > > > >> - single NFS share that shares airflow_home over each container
> > > > >>> the git sync'er is now a seperate container running before the
> > other
> > > > >> containers
> > > > >>
> > > > >> This resolved my mystery DAG crashes.
> > > > >>
> > > > >> I'll be updating production to a patchless RC3 today, you get my
> > vote
> > > > after
> > > > >> that.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Sun, Feb 12, 2017 at 4:59 AM Boris Tyukin <
> bo...@boristyukin.com
> > >
> > > > wrote:
> > > > >>
> > > > >>> awesome! thanks Jeremiah
> > > > >>>
> > > > >>> On Sat, Feb 11, 2017 at 12:53 PM, Jeremiah Lowin <
> > jlo...@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Boris, I submitted a PR to address your second point --
> > > > >>>> https://github.com/apache/incubator-airflow/pull/2068. Thanks!
> > > > >>>>
> > > > >>>> On Sat, Feb 11, 2017 at 10:42 AM Boris Tyukin <
> > > bo...@boristyukin.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> I am running LocalExecutor and not doing crazy things but use
> DAG
> > > > >>>>> generation heavily - everything runs fine as before. As I
> > mentioned
> > > > in
> > > > >>>>> other threads only had a few issues:
> > > > >>>>>
> > > > >>>>> 1) had to upgrade MySQL which was a PAIN. Cloudera CDH is
> running
> > > old
> > > > >>>>> version of MySQL which was compatible with 1.7.1 but not
> > compatible
> > > > now
> > > > >>>>> with 1.8 because of fractional seconds support PR.
> > > > >>>>>
> > > > >>>>> 2) when you install airflow, there are two new example DAGs
> > > > >>>>> (last_task_only) which

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

[Alex Van Boxel]

rom earlier releases.
> > > > > > > > >
> > > > > > > > > Two issues have been fixed since release candidate 2:
> > > > > > > > >
> > > > > > > > > * trigger_dag could create dags with fractional seconds,
> not
> > > > > > supported
> > > > > > > by
> > > > > > > > > logging and UI at the moment
> > > > > > > > > * local api client trigger_dag had hardcoded execution of
> > None
> > > > > > > > >
> > > > > > > > > Known issue:
> > > > > > > > > * Airflow on kubernetes and num_runs -1 (default) can
> expose
> > > > import
> > > > > > > > issues.
> > > > > > > > >
> > > > > > > > > I have extensively discussed this with Alex (reporter) and
> we
> > > > > > consider
> > > > > > > > > this a known issue with a workaround available as we are
> > unable
> > > > to
> > > > > > > > > replicate this in a different environment. UPDATING.md has
> > been
> > > > > > updated
> > > > > > > > > with the work around.
> > > > > > > > >
> > > > > > > > > As these issues are confined to a very specific area and
> full
> > > > unit
> > > > > > > tests
> > > > > > > > > were added I would also like to raise a VOTE for releasing
> > > 1.8.0
> > > > > > based
> > > > > > > on
> > > > > > > > > release candidate 3, i.e. just renaming release candidate 3
> > to
> > > > > 1.8.0
> > > > > > > > > release.
> > > > > > > > >
> > > > > > > > > Please respond to this email by:
> > > > > > > > >
> > > > > > > > > +1,0,-1 with *binding* if you are a PMC member or
> > *non-binding*
> > > > if
> > > > > > you
> > > > > > > > are
> > > > > > > > > not.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Bolke
> > > > > > > > >
> > > > > > > > > My VOTE: +1 (binding)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Contrib & Dataflow

[Alex Van Boxel]

I like the idea. I already raised the issue so we could refactor all the
Google Cloud operators together and at that time make sure they are
consistent. So a different repo would be a good idea here. And you can
manage your own dependencies. Would be cool that the same thing happens to
the AWS operators.


On Sat, Feb 4, 2017 at 7:46 PM Jeremiah Lowin  wrote:

> Max made some great points on my dataflow PR and I wanted to continue the
> conversation here to make sure the conversation was visible to all.
>
> While I think my dataflow implementation contains the basic requirements
> for any more complicated extension (but that conversation can wait!), I had
> to implement it by adding some very specific "dataflow-only" code to core
> Operator logic. In retrospect, that makes me pause (as, I believe, it did
> for Max).
>
> After thinking for a few days, what I really want to do is propose a very
> small change to core Airflow: change BaseOperator.post_execute(context) to
> BaseOperator.post_execute(result, context). I think the pre_execute and
> post_execute hooks have generally been an afterthought, but with that
> change (which, I think, is reasonable in and of itself) I could implement
> entirely through those hooks.
>
> So that brings me to my next point: if the hook is changed, I could happily
> drop a reworked dataflow implementation into contrib, rather than core.
> That would alleviate some of the pressure for Airflow to officially decide
> whether it's the right implementation or not (it is! :) ). I feel like that
> would be the optimal situation at the moment.
>
> And that brings me to my next point: the future of "contrib" and the
> Airflow community.
> Having contrib in the core Airflow repo has some advantages:
>   - standardized access
>   - centralized repository for PRs
>   - at least a style review (if not unit tests) from the committers
> But some big disadvantages as well:
>   - Very complicated dependency management [presumably, most contrib
> operators need to add an extras_require entry for their specific
> dependencies]
>   - No sense of ownership or even an easy way to raise issues (due to
> friction of opening JIRA tickets vs github issues)
>
> One thought is to move the contrib directory to its own repo which would
> keep the advantages but remove the disadvantages from core Airflow. Another
> is to encourage individual airflow repos (Airflow-Docker, Airflow-Dataflow,
> Airflow-YourExtensionHere) which could be installed a la carte. That would
> leave maintenance up to the original author, but could lead to some
> fracturing in the community as discovery becomes difficult.
>
-- 
  _/
_/ Alex Van Boxel

Re: Proposal for new task state "WAITING_ON_CALLBACK"

[Alex Van Boxel]

Hey,

here is my feedback, because I've been thinking about events as well. I
would call it it 'WAITING_FOR_EVENT'. Here are the use-cases I would use it
for:

Have a thread (or process) listen on the Google Audit Log. It contains a
lot of changes on the Google Project (Google DataProc finished, File in
Bucket added, ...) and translate it to Events that Airflow understands.


*DataProc/BigQuery/Storage*

   1. Start DagProc job through the API (long running) - takes a
   parallelism slot
   2. Get JobId
   3. Register for event: *gc:dataproc:job_id_* with timeout 300s
   4. execute stops and parallelism slot is freed

The google audit log is constantly translating log entries to airflow
events. Airflow event_listener looks for specific events registered (could
be wildcards). As soon as the event match it call's the callback:

   1. on_event is called with *gc:dataproc:job_id_* and JSON with event
   payload
   2. handle JSON payload depending on payload throw *FAILURE* or *SUCCESS*
   or register for other event

It could be a timeout

   1. on_event is called with *airflow:timeout:dag_id* timeout event
   2. handle timeout with
  1. throw FAILURE
  2. try to recover by calling DataProc API and re-register new/or same
  event

*Bolke's callback example (mapped to this)*


   1. task x for z does some work
   2. register for event *api:callback:20170101T00*
   3. execute stops and parallelism slot is freed

The API translated http callbacks into events

   1. on_event is called with *api:callback:20170101T00* and JSON with
   event payload
   2. handle JSON payload depending on payload throw *FAILURE* or *SUCCESS* or
   register for other event

It could be a timeout

   1. on_event is called with *airflow:timeout:dag_id* timeout event
   2. handle timeout with FAILURE


So, what do you think?


On Wed, Feb 8, 2017 at 2:40 PM Jeremiah Lowin  wrote:

I meant the API -- will check the wiki now. Thanks!

On Wed, Feb 8, 2017 at 8:33 AM Bolke de Bruin  wrote:

> On this proposal? No, not yet. Just popped in my mind yesterday. API there
> is a bit on the wiki.
>
> > On 8 Feb 2017, at 14:31, Jeremiah Lowin  wrote:
> >
> > Makes a lot of sense. At the NY meetup there was considerable interest
in
> > using the API (and quite a few hacks around exposing the CLI!) -- is
> there
> > more complete documentation anywhere?
> >
> > Thanks Bolke
> >
> > On Wed, Feb 8, 2017 at 1:36 AM Bolke de Bruin  wrote:
> >
> >> Hi All,
> >>
> >> Now that we have an API in place. I would like to propose a new state
> for
> >> tasks named “WAITING_ON_CALLBACK”. Currently, we have tasks that have a
> >> kind of polling mechanism (ie. Sensors) that wait for an action to
> happen
> >> and check if that action happened by regularly polling a particular
> >> backend. This will always use a slot from one of the workers and could
> >> starve an airflow cluster for resources. What if a callback to Airflow
> >> could happen that task to change its status by calling a callback
> mechanism
> >> without taking up a worker slot. A timeout could (should) be associated
> >> with the required callback so that the task can fail if required. So a
> bit
> >> more visual:
> >>
> >>
> >> Task X from DAG Z  does some work and sets “WAITING_ON_CALLBACK” -> API
> >> post to /dags/Z/dag_runs/20170101T00:00:00/tasks/X with payload “set
> status
> >> to SUCCESS”
> >>
> >> DAG Z happily continues.
> >>
> >> Or
> >>
> >> Task X from DAG Z sets “WAITING_ON_CALLBACK” with timeout of 300s ->
> time
> >> passes -> scheduler sets task to FAILED.
> >>
> >>
> >> Any thoughts?
> >>
> >> - Bolke
>
>

-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

I'm still going over the code how such a small change can have such a huge
effect. Some things that is specific to the setup:

worker/scheduler/webserver all run with no extra parameters
build in docker
Python 2.7.13
Celery with redis
Runs on Kubernetes

When connecting to scheduler pod I see the scheduler forking other
scheduler processes that seem to stop immediately (probably with the Dag
scanning).

It's quite hard debugging in k8s. I'll try to find something more.



On Wed, Feb 8, 2017 at 1:33 PM Bolke de Bruin  wrote:

> Alex,
>
> Do you have anything more to go on? I don’t mind reverting the patch,
> however it code part seems unrelated to what you described and the issue
> wasn’t reproducible. I would really like to see more logging and maybe a
> test in a clean environment plus debugging. Preferable I would like to make
> RC 2 available today and immediately raise a vote as the *current* changes
> are really small, are confined to contrib and have been tested by the
> people using it.
>
> But I am holding off for now due to your concern.
>
> Cheers
> Bolke
>
>
> On 7 Feb 2017, at 20:56, Bolke de Bruin  wrote:
>
> How do you start the scheduler Alex? What are the command line parameters?
> What are the logs when it doesn’t work?
>
> Bolke
>
>
>
> On 7 Feb 2017, at 18:52, Alex Van Boxel  wrote:
>
> Hey Feng,
>
> The upgrades are all automated (including the workers/web/scheduler). And
> I tripple checked, I now am test running RC1 just with the your line
> reverted (and look ok)
>
> Could you do me a favour and add a test dag where you do a local import.
> Example:
>
> bqschema.py
>
> def ranking():
> return [
> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
> {"name": "rank", "type": "integer", "mode": "nullable"},
> {"name": "audience_preference", "type": "float", "mode": "nullable"},
> {"name": "audience_likelihood_share", "type": "float", "mode": 
> "nullable"}
> ]
>
>
> dag.py
>
> import bqschema
> *...*
>
> all in the same dag folder. We use it to define out BigQuery schema's into
> a seperate file.
>
>
> On Tue, Feb 7, 2017 at 6:37 PM Feng Lu  wrote:
>
> Hi Alex-
>
> Please see the attached screenshots of my local testing using
> celeryexecutor (on k8s as well).
> All look good and the workflow is successfully completed.
>
> Curious did you also update the worker image?
> Sorry for the confusion, happy to debug more if you could share with me
> your k8s setup.
>
> Feng
>
> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu  wrote:
>
> When num_runs is not explicitly specified, the default is set to -1 to
> match the expectation of SchedulerJob here:
> 
> ​
> Doing so also matches the type of num_runs ('int' in this case).
> The scheduler will run non-stop as a result regardless whether dag files
> are present (since the num_runs default is now -1: unlimited).
>
> Based on what Alex described, the import error doesn't look like directly
> related to this change.
> Maybe this one?
> https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9
>
> I am still in the middle of running some quick test using celery executor,
> will update the thread once it's done.
>
>
> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  wrote:
>
> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

Hey Feng,

The upgrades are all automated (including the workers/web/scheduler). And I
tripple checked, I now am test running RC1 just with the your line reverted
(and look ok)

Could you do me a favour and add a test dag where you do a local import.
Example:

bqschema.py

def ranking():
return [
{"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
{"name": "rank", "type": "integer", "mode": "nullable"},
{"name": "audience_preference", "type": "float", "mode": "nullable"},
{"name": "audience_likelihood_share", "type": "float", "mode":
"nullable"}
]


dag.py

import bqschema
*...*

all in the same dag folder. We use it to define out BigQuery schema's into
a seperate file.


On Tue, Feb 7, 2017 at 6:37 PM Feng Lu  wrote:

> Hi Alex-
>
> Please see the attached screenshots of my local testing using
> celeryexecutor (on k8s as well).
> All look good and the workflow is successfully completed.
>
> Curious did you also update the worker image?
> Sorry for the confusion, happy to debug more if you could share with me
> your k8s setup.
>
> Feng
>
> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu  wrote:
>
> When num_runs is not explicitly specified, the default is set to -1 to
> match the expectation of SchedulerJob here:
> [image: Screen Shot 2017-02-07 at 8.01.26 AM.png]
> ​
> Doing so also matches the type of num_runs ('int' in this case).
> The scheduler will run non-stop as a result regardless whether dag files
> are present (since the num_runs default is now -1: unlimited).
>
> Based on what Alex described, the import error doesn't look like directly
> related to this change.
> Maybe this one?
> https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9
>
> I am still in the middle of running some quick test using celery executor,
> will update the thread once it's done.
>
>
> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  wrote:
>
> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
> wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> >
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to
> > from "from airflow.operators.python_operator import PythonOperator". Am I
> > missing something?
> >
> > 2. I ran into a migration problem that seems to have cleared itself up. I
> > did notice that some dags do not have data in their "DAG Runs" column on
> > the overview page computed. I am looking into that issue presently.
> >
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
> >
> > -s
> >
> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  .invalid>
> > wrote:
> >
> >> Bol

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

OK, a bit of history... I moved for beta 4 to rc 1 and though I didn't have
problems because it run'ed fine locally for testing. But my production set
is on K8S + Redis (Celery). *What I saw in production was the **import
errors on the DAGS,* first I thought it was due to the fix of showing
errors in the DAG in the UI or my own building. But it wasn't. I had to go
though every commit between beta 4 and rc1 to find the error (build docker
image, deploy in k8s).

So to clarify (I'm sure this is the commit now):

   1. I did a before and after commit test.
   2. I now installed RC1, but with that single line reverted (works after,
   broken before)

I can also only reproduce it in the production, but it's been running on
master till beta 4 quite ok. I don't really know why it does that (a bit
the downside of dynamic typing here...) and actually don't want to dig
further. I've lost day's of valuable time (although I learned a lot about
Python dynamic loading).

*Or we find the problem with the help of Feng (the implementor) or we
revert that single commit*. Or is somebody else running Celery that is not
having this problem?






On Tue, Feb 7, 2017 at 3:57 PM Bolke de Bruin  wrote:

> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
> wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> >
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to
> > from "from airflow.operators.python_operator import PythonOperator". Am I
> > missing something?
> >
> > 2. I ran into a migration problem that seems to have cleared itself up. I
> > did notice that some dags do not have data in their "DAG Runs" column on
> > the overview page computed. I am looking into that issue presently.
> >
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
> >
> > -s
> >
> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  .invalid>
> > wrote:
> >
> >> Bolke, attached is the patch for the cgroups fix. Let me know which
> >> branches you would like me to merge it to. If anyone has complaints
> about
> >> the patch let me know (but it does not touch the core of airflow, only
> the
> >> new cgroups task runner).
> >>
> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand 
> wrote:
> >>
> >>> Actually, I see the error is further down..
> >>>
> >>>  File
> >>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> >>> line
> >>> 469, in do_execute
> >>>
> >>>cursor.execute(statement, parameters)
> >>>
> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> >>> column "dag_id" violates not-null constraint
> >>>
> >>> DETAIL:  Failing row contains (null, running, 1, f).
> >>>
> >>> [SQL: 'INSERT INTO d

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

;
>> > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
>> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
>> >
>> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
>> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
>> primary
>> > key for table 'dag_stats', but has no Python-side or server-side
default
>> > generator indicated, nor does it indicate 'autoincrement=True' or
>> > 'nullable=True', and no explicit value is passed.  Primary key columns
>> > typically may not store NULL. Note that as of SQLAlchemy 1.1,
>> > 'autoincrement=True' must be indicated explicitly for composite (e.g.
>> > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
>> > expected for one of the columns in the primary key. CREATE TABLE
>> statements
>> > are impacted by this change as well on most backends.
>> >
>>
>
>

-- 
  _/
_/ Alex Van Boxel

Re: Changelog 1.8

[Alex Van Boxel]

Yes, I will take care of it tomorrow morning. Need sleep now... I'll add
the new addition to it.

On Wed, Feb 1, 2017 at 8:59 PM Bolke de Bruin  wrote:

> Hey Alex,
>
> Can you finalize the changelog for 1.8? I can then make 1.8 rc 1 available.
>
> Bolke
>
> Sent from my iPhone
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow Logging Updates

[Alex Van Boxel]

Hey Robin,

I also have an interest in logging, but to get Google Cloud logging to
work. I'm happy to work with you on this. But *first* the 1.8 needs to go
out.

But I don't think this will go on the patch branch, but maybe for the next
release. I don't know what the other people think.

On Wed, Feb 1, 2017 at 11:58 AM Miller, Robin <
robin.mil...@affiliate.oliverwyman.com> wrote:

> Hi All,
>
>
> A while ago we took over this issue:
> https://issues.apache.org/jira/browse/AIRFLOW-409, which revolved around
> avoiding making changes to the root python logging configuration so as to
> avoid any accidental side effects that this could produce in other python
> applications.
>
>
> The approach we decided on was to avoid use of the root logger entirely
> and produce a single configuration point for the logging to allow it to be
> configured pretty much however is desired in one place (with currently
> fairly simple options, but this could be made as complex or configurable as
> desired with changes to only a single file).
>
>
> As you might expect this affects a lot of the code, as it requires
> changing every log statement that's contains "logging.Info",
> "logging.Error", etc.
>
>
> The pull request for this,
> https://github.com/apache/incubator-airflow/pull/1921, has been open for
> almost 2 months now and unsurprisingly now has merge conflicts. We'll
> happily clean these up, but would prefer to do so at the time this is being
> reviewed, otherwise more will inevitably appear.
>
>
> So I'm wondering, is this issue/pull request likely to get any attention
> soon? Or is this change simply unwanted?
>
>
> Thanks,
>
> Robin Miller
> OLIVER WYMAN
> robin.mil...@affiliate.oliverwyman.com robin.mil...@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
>
>
> 
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

I identified my root cause: it's was a problem at my site with a faulty
rebase. So all green.

On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel  wrote:

> So bumped to RC1 and this seems fine. I don't get it.
>
> So it's a go. Sorry for the noise, but better safe then sorry. I also
> found that the scheduler logs setting are not equal from the normal logging
> so that's also a go.
>
> You get green light from me.
>
> On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin  wrote:
>
> And the scheduler doesn't log anything in beta 4?
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 19:26, Alex Van Boxel  wrote:
> >
> > I see it in the scheduler and in the UI. Currently trying to do a new
> > upgrade.
> >
> >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin 
> wrote:
> >>
> >> Please note I will be holding off on the RC, before we understand this
> >> issue better.
> >>
> >> Bolke
> >>
> >>> On 31 Jan 2017, at 18:06, Bolke de Bruin  wrote:
> >>>
> >>> Hey Alex,
> >>>
> >>> Could it actually be that Airflow is doing the right thing? Earlier it
> >> was swallowing the errors. Where do you see the errors? In the scheduler
> >> logs, UI, processor logs?
> >>>
> >>> - Bolke
> >>>
> >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel  wrote:
> >>>>
> >>>> I'll try to identify the core problem
> >>>>
> >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin 
> wrote:
> >>>>>
> >>>>> Hey Alex
> >>>>>
> >>>>> Can you provide some info on the scheduler paths thing. I don't
> >> have/see
> >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in
> any
> >> case.
> >>>>>
> >>>>> I don't think the dag processor respects cli parameters.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel  wrote:
> >>>>>>
> >>>>>> It's quite hard to share my complete dags. I don't have this
> locally,
> >>>>> but I
> >>>>>> have it in my production environment where I use Celery. I rolled
> >> back to
> >>>>>> beta 4 to make it work again.
> >>>>>>
> >>>>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>>>
> >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov  >>>>> .invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Alex
> >>>>>>> I'm not able to reproduce locally (assuming the two python files
> are
> >> in
> >>>>> the
> >>>>>>> same folder or is on your PYTHONPATH). I don't see that import
> error
> >>>>>>> anyways.
> >>>>>>>
> >>>>>>> Just in case, what is your complete DAG definition? Is anyone else
> >> able
> >>>>> to
> >>>>>>> repro?
> >>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel  >
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Well this means none of my DAG's work anymore:
> >>>>>>>>
> >>>>>>>> you just can do this anymore:
> >>>>>>>>
> >>>>>>>> file bqschema.py with
> >>>>>>>>
> >>>>>>>> def marketing_segment():
> >>>>>>>> return [
> >>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
> >>>>> "nullable"},
> >>>>>>>> {"name": "segment_main", "type": "string", "mode":
> "nullable"},
> >>>>>>>> {"name": "segment_sub", "type": "integer", "mode":
> "nullable"},

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

So bumped to RC1 and this seems fine. I don't get it.

So it's a go. Sorry for the noise, but better safe then sorry. I also found
that the scheduler logs setting are not equal from the normal logging so
that's also a go.

You get green light from me.

On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin  wrote:

> And the scheduler doesn't log anything in beta 4?
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 19:26, Alex Van Boxel  wrote:
> >
> > I see it in the scheduler and in the UI. Currently trying to do a new
> > upgrade.
> >
> >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin 
> wrote:
> >>
> >> Please note I will be holding off on the RC, before we understand this
> >> issue better.
> >>
> >> Bolke
> >>
> >>> On 31 Jan 2017, at 18:06, Bolke de Bruin  wrote:
> >>>
> >>> Hey Alex,
> >>>
> >>> Could it actually be that Airflow is doing the right thing? Earlier it
> >> was swallowing the errors. Where do you see the errors? In the scheduler
> >> logs, UI, processor logs?
> >>>
> >>> - Bolke
> >>>
> >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel  wrote:
> >>>>
> >>>> I'll try to identify the core problem
> >>>>
> >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin 
> wrote:
> >>>>>
> >>>>> Hey Alex
> >>>>>
> >>>>> Can you provide some info on the scheduler paths thing. I don't
> >> have/see
> >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in
> any
> >> case.
> >>>>>
> >>>>> I don't think the dag processor respects cli parameters.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel  wrote:
> >>>>>>
> >>>>>> It's quite hard to share my complete dags. I don't have this
> locally,
> >>>>> but I
> >>>>>> have it in my production environment where I use Celery. I rolled
> >> back to
> >>>>>> beta 4 to make it work again.
> >>>>>>
> >>>>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>>>
> >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov  >>>>> .invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Alex
> >>>>>>> I'm not able to reproduce locally (assuming the two python files
> are
> >> in
> >>>>> the
> >>>>>>> same folder or is on your PYTHONPATH). I don't see that import
> error
> >>>>>>> anyways.
> >>>>>>>
> >>>>>>> Just in case, what is your complete DAG definition? Is anyone else
> >> able
> >>>>> to
> >>>>>>> repro?
> >>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel  >
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Well this means none of my DAG's work anymore:
> >>>>>>>>
> >>>>>>>> you just can do this anymore:
> >>>>>>>>
> >>>>>>>> file bqschema.py with
> >>>>>>>>
> >>>>>>>> def marketing_segment():
> >>>>>>>> return [
> >>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
> >>>>> "nullable"},
> >>>>>>>> {"name": "segment_main", "type": "string", "mode":
> "nullable"},
> >>>>>>>> {"name": "segment_sub", "type": "integer", "mode":
> "nullable"},
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In marketing_segmentation.py:
> >>>>>>

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

I see it in the scheduler and in the UI. Currently trying to do a new
upgrade.

On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin  wrote:

> Please note I will be holding off on the RC, before we understand this
> issue better.
>
> Bolke
>
> > On 31 Jan 2017, at 18:06, Bolke de Bruin  wrote:
> >
> > Hey Alex,
> >
> > Could it actually be that Airflow is doing the right thing? Earlier it
> was swallowing the errors. Where do you see the errors? In the scheduler
> logs, UI, processor logs?
> >
> > - Bolke
> >
> >> On 31 Jan 2017, at 16:25, Alex Van Boxel  wrote:
> >>
> >> I'll try to identify the core problem
> >>
> >> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin  wrote:
> >>
> >>> Hey Alex
> >>>
> >>> Can you provide some info on the scheduler paths thing. I don't
> have/see
> >>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any
> case.
> >>>
> >>> I don't think the dag processor respects cli parameters.
> >>>
> >>> Bolke
> >>>
> >>> Sent from my iPhone
> >>>
> >>>> On 31 Jan 2017, at 15:10, Alex Van Boxel  wrote:
> >>>>
> >>>> It's quite hard to share my complete dags. I don't have this locally,
> >>> but I
> >>>> have it in my production environment where I use Celery. I rolled
> back to
> >>>> beta 4 to make it work again.
> >>>>
> >>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>
> >>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov  >>> .invalid>
> >>>> wrote:
> >>>>
> >>>>> @Alex
> >>>>> I'm not able to reproduce locally (assuming the two python files are
> in
> >>> the
> >>>>> same folder or is on your PYTHONPATH). I don't see that import error
> >>>>> anyways.
> >>>>>
> >>>>> Just in case, what is your complete DAG definition? Is anyone else
> able
> >>> to
> >>>>> repro?
> >>>>>
> >>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel 
> >>> wrote:
> >>>>>>
> >>>>>> Well this means none of my DAG's work anymore:
> >>>>>>
> >>>>>> you just can do this anymore:
> >>>>>>
> >>>>>> file bqschema.py with
> >>>>>>
> >>>>>> def marketing_segment():
> >>>>>>  return [
> >>>>>>  {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>>  {"name": "bucket_date", "type": "timestamp", "mode":
> >>> "nullable"},
> >>>>>>  {"name": "segment_main", "type": "string", "mode": "nullable"},
> >>>>>>  {"name": "segment_sub", "type": "integer", "mode": "nullable"},
> >>>>>>
> >>>>>>
> >>>>>> In marketing_segmentation.py:
> >>>>>>
> >>>>>>
> >>>>>> import bqschema
> >>>>>>
> >>>>>> Gives an error:
> >>>>>>
> >>>>>> Traceback (most recent call last):
> >>>>>> File
> >>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>>>>> apache.incubating-py2.7.egg/airflow/models.py",
> >>>>>> line 264, in process_file
> >>>>>>  m = imp.load_source(mod_name, filepath)
> >>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>>>>> 
> >>>>>>  import bqschema
> >>>>>> ImportError: No module named bqschema
> >>>>>>
> >>>>>> *I don't think this is incorrect?!*
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
> dan.davy...@airbnb.com.
> >>>>>> invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

I'll try to identify the core problem

On Tue, Jan 31, 2017, 16:43 Bolke de Bruin  wrote:

> Hey Alex
>
> Can you provide some info on the scheduler paths thing. I don't have/see
> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>
> I don't think the dag processor respects cli parameters.
>
> Bolke
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 15:10, Alex Van Boxel  wrote:
> >
> > It's quite hard to share my complete dags. I don't have this locally,
> but I
> > have it in my production environment where I use Celery. I rolled back to
> > beta 4 to make it work again.
> >
> > Also @bolke the scheduler logs don't respect the log path.
> >
> > On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov  .invalid>
> > wrote:
> >
> >> @Alex
> >> I'm not able to reproduce locally (assuming the two python files are in
> the
> >> same folder or is on your PYTHONPATH). I don't see that import error
> >> anyways.
> >>
> >> Just in case, what is your complete DAG definition? Is anyone else able
> to
> >> repro?
> >>
> >>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel 
> wrote:
> >>>
> >>> Well this means none of my DAG's work anymore:
> >>>
> >>> you just can do this anymore:
> >>>
> >>> file bqschema.py with
> >>>
> >>> def marketing_segment():
> >>>return [
> >>>{"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>{"name": "bucket_date", "type": "timestamp", "mode":
> "nullable"},
> >>>{"name": "segment_main", "type": "string", "mode": "nullable"},
> >>>{"name": "segment_sub", "type": "integer", "mode": "nullable"},
> >>>
> >>>
> >>> In marketing_segmentation.py:
> >>>
> >>>
> >>> import bqschema
> >>>
> >>> Gives an error:
> >>>
> >>> Traceback (most recent call last):
> >>>  File
> >>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>> apache.incubating-py2.7.egg/airflow/models.py",
> >>> line 264, in process_file
> >>>m = imp.load_source(mod_name, filepath)
> >>>  File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>> 
> >>>import bqschema
> >>> ImportError: No module named bqschema
> >>>
> >>> *I don't think this is incorrect?!*
> >>>
> >>>
> >>>
> >>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov  >>> invalid>
> >>> wrote:
> >>>
> >>>> The latest commit fixed a regression since 1.7 that files with parsing
> >>>> errors no longer showed up on the UI.
> >>>>
> >>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel 
> >>> wrote:
> >>>>
> >>>>> Just installed beta 5 on our dev environment it lighted up as a
> >>> christmas
> >>>>> tree. I got a a screen full of import errors. I see that the latest
> >>>> commit
> >>>>> did something with import errors... is it coorect?!
> >>>>>
> >>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin 
> >>>> wrote:
> >>>>>
> >>>>>> Hey Boris
> >>>>>>
> >>>>>> The scheduler is a bit more aggressive and can use multiple
> >>> processors,
> >>>>> so
> >>>>>> higher CPU usage is actually a good thing.
> >>>>>>
> >>>>>> I case it is really out of hand look at the new scheduler options
> >> and
> >>>>>> heartbeat options (see PR for updating.md not in the beta yet).
> >>>>>>
> >>>>>> Bolke
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>>
> >>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin 
> >>>> wrote:
> >>>>>>>
> >>>>>>> I am not sure if it is my config or something, but looks like
> >> after

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

It's quite hard to share my complete dags. I don't have this locally, but I
have it in my production environment where I use Celery. I rolled back to
beta 4 to make it work again.

Also @bolke the scheduler logs don't respect the log path.

On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov 
wrote:

> @Alex
> I'm not able to reproduce locally (assuming the two python files are in the
> same folder or is on your PYTHONPATH). I don't see that import error
> anyways.
>
> Just in case, what is your complete DAG definition? Is anyone else able to
> repro?
>
> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel  wrote:
>
> > Well this means none of my DAG's work anymore:
> >
> > you just can do this anymore:
> >
> > file bqschema.py with
> >
> > def marketing_segment():
> > return [
> > {"name": "user_id", "type": "integer", "mode": "nullable"},
> > {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
> > {"name": "segment_main", "type": "string", "mode": "nullable"},
> > {"name": "segment_sub", "type": "integer", "mode": "nullable"},
> >
> >
> > In marketing_segmentation.py:
> >
> >
> > import bqschema
> >
> > Gives an error:
> >
> > Traceback (most recent call last):
> >   File
> > "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> > apache.incubating-py2.7.egg/airflow/models.py",
> > line 264, in process_file
> > m = imp.load_source(mod_name, filepath)
> >   File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> > 
> > import bqschema
> > ImportError: No module named bqschema
> >
> > *I don't think this is incorrect?!*
> >
> >
> >
> > On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov  > invalid>
> > wrote:
> >
> > > The latest commit fixed a regression since 1.7 that files with parsing
> > > errors no longer showed up on the UI.
> > >
> > > On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel 
> > wrote:
> > >
> > > > Just installed beta 5 on our dev environment it lighted up as a
> > christmas
> > > > tree. I got a a screen full of import errors. I see that the latest
> > > commit
> > > > did something with import errors... is it coorect?!
> > > >
> > > > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin 
> > > wrote:
> > > >
> > > > > Hey Boris
> > > > >
> > > > > The scheduler is a bit more aggressive and can use multiple
> > processors,
> > > > so
> > > > > higher CPU usage is actually a good thing.
> > > > >
> > > > > I case it is really out of hand look at the new scheduler options
> and
> > > > > heartbeat options (see PR for updating.md not in the beta yet).
> > > > >
> > > > > Bolke
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On 29 Jan 2017, at 15:35, Boris Tyukin 
> > > wrote:
> > > > > >
> > > > > > I am not sure if it is my config or something, but looks like
> after
> > > the
> > > > > > upgrade and start of scheduler, airflow would totally hose CPU.
> The
> > > > > reason
> > > > > > is two new examples that start running right away - latest only
> and
> > > > > latest
> > > > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> > > because
> > > > > now
> > > > > > dags are not paused by default like it was before?
> > > > > >
> > > > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> > > someone
> > > > > > needs a step by step instruction, make sure to follow all steps
> > > > precisely
> > > > > > here for in-place upgrade or you will have heck of the time (like
> > > me).
> > > > > >
> > > > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > > > upgrade-procedure-inplace
> > > > > >
> > > > > > BTW official Oracle repository for Oracle Linux only has MySql
> 5.6
> > -
> >

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

Well this means none of my DAG's work anymore:

you just can do this anymore:

file bqschema.py with

def marketing_segment():
return [
{"name": "user_id", "type": "integer", "mode": "nullable"},
{"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
{"name": "segment_main", "type": "string", "mode": "nullable"},
{"name": "segment_sub", "type": "integer", "mode": "nullable"},


In marketing_segmentation.py:


import bqschema

Gives an error:

Traceback (most recent call last):
  File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+apache.incubating-py2.7.egg/airflow/models.py",
line 264, in process_file
m = imp.load_source(mod_name, filepath)
  File "/home/airflow/dags/marketing_segmentation.py", line 17, in 
import bqschema
ImportError: No module named bqschema

*I don't think this is incorrect?!*



On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov 
wrote:

> The latest commit fixed a regression since 1.7 that files with parsing
> errors no longer showed up on the UI.
>
> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel  wrote:
>
> > Just installed beta 5 on our dev environment it lighted up as a christmas
> > tree. I got a a screen full of import errors. I see that the latest
> commit
> > did something with import errors... is it coorect?!
> >
> > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin 
> wrote:
> >
> > > Hey Boris
> > >
> > > The scheduler is a bit more aggressive and can use multiple processors,
> > so
> > > higher CPU usage is actually a good thing.
> > >
> > > I case it is really out of hand look at the new scheduler options and
> > > heartbeat options (see PR for updating.md not in the beta yet).
> > >
> > > Bolke
> > >
> > > Sent from my iPhone
> > >
> > > > On 29 Jan 2017, at 15:35, Boris Tyukin 
> wrote:
> > > >
> > > > I am not sure if it is my config or something, but looks like after
> the
> > > > upgrade and start of scheduler, airflow would totally hose CPU. The
> > > reason
> > > > is two new examples that start running right away - latest only and
> > > latest
> > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> because
> > > now
> > > > dags are not paused by default like it was before?
> > > >
> > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> someone
> > > > needs a step by step instruction, make sure to follow all steps
> > precisely
> > > > here for in-place upgrade or you will have heck of the time (like
> me).
> > > >
> > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > upgrade-procedure-inplace
> > > >
> > > > BTW official Oracle repository for Oracle Linux only has MySql 5.6 -
> > for
> > > > 5.7 you have to use MySql community repo.
> > > >
> > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin  >
> > > wrote:
> > > >>
> > > >> Hi All,
> > > >>
> > > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> > > keys
> > > >> are available at https://dist.apache.org/repos/
> > dist/release/incubator/
> > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > airflow/
> > > >
> > > >> . It is tagged with a local version “apache.incubating” so it allows
> > > >> upgrading from earlier releases.
> > > >>
> > > >> Issues fixed:
> > > >> * Parsing errors not showing up in UI fixing a regression**
> > > >> * Scheduler would terminate immediately if no dag files present
> > > >>
> > > >> ** As this touches the scheduler logic I though it warranted another
> > > beta.
> > > >>
> > > >> This should be the last beta in my opinion and we can prepare
> > changelog,
> > > >> upgrade notes and release notes for the RC (Feb 2).
> > > >>
> > > >> Cheers
> > > >> Bolke
> > >
> > --
> >   _/
> > _/ Alex Van Boxel
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

[Alex Van Boxel]

Just installed beta 5 on our dev environment it lighted up as a christmas
tree. I got a a screen full of import errors. I see that the latest commit
did something with import errors... is it coorect?!

On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin  wrote:

> Hey Boris
>
> The scheduler is a bit more aggressive and can use multiple processors, so
> higher CPU usage is actually a good thing.
>
> I case it is really out of hand look at the new scheduler options and
> heartbeat options (see PR for updating.md not in the beta yet).
>
> Bolke
>
> Sent from my iPhone
>
> > On 29 Jan 2017, at 15:35, Boris Tyukin  wrote:
> >
> > I am not sure if it is my config or something, but looks like after the
> > upgrade and start of scheduler, airflow would totally hose CPU. The
> reason
> > is two new examples that start running right away - latest only and
> latest
> > with trigger. Once I pause them, CPU goes back to idle. Is this because
> now
> > dags are not paused by default like it was before?
> >
> > As I mentioned before, I also had to upgrade mysql to 5.7 - if someone
> > needs a step by step instruction, make sure to follow all steps precisely
> > here for in-place upgrade or you will have heck of the time (like me).
> >
> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#upgrade-procedure-inplace
> >
> > BTW official Oracle repository for Oracle Linux only has MySql 5.6 - for
> > 5.7 you have to use MySql community repo.
> >
> >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin 
> wrote:
> >>
> >> Hi All,
> >>
> >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >> are available at https://dist.apache.org/repos/dist/release/incubator/
> >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/
> >
> >> . It is tagged with a local version “apache.incubating” so it allows
> >> upgrading from earlier releases.
> >>
> >> Issues fixed:
> >> * Parsing errors not showing up in UI fixing a regression**
> >> * Scheduler would terminate immediately if no dag files present
> >>
> >> ** As this touches the scheduler logic I though it warranted another
> beta.
> >>
> >> This should be the last beta in my opinion and we can prepare changelog,
> >> upgrade notes and release notes for the RC (Feb 2).
> >>
> >> Cheers
> >> Bolke
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 4

[Alex Van Boxel]

I'm also abandoning the follow problem:
* DAG marked success, with half of the Tasks never scheduled (Alex)

I *can't simulate it locally*... I'll keep an eye on the production runs
though.


On Fri, Jan 27, 2017 at 8:45 PM Bolke de Bruin  wrote:

> 5.6.4 is the minimum. Yes it was required. updating.md was required.
> 5.6.4 is already really really old. Most distro's replaced it with mariadb
> which is compatible.
>
> Bolke
>
> Sent from my iPhone
>
> > On 27 Jan 2017, at 20:19, Boris Tyukin  wrote:
> >
> > sorry mysql version i have is mysql  Ver 14.14 Distrib 5.1.73
> >
> > I see that DATETIME(6) is only supported on  MySQL 5.7 and later.
> >
> > Was it really necessary to do use DATETIME(6) not just DATETIME?
> >
> > Probably should be mention in change doc that this requires an upgrade of
> > mysql which in my opinion is not cool. epel repo only has 5.1.73 as
> latest
> > version and CDH distro includes 5.1.73 as well.
> >
> > Some companies might not allow an upgrade of mysql unless it is in one of
> > the official linux repos.
> >
> >
> >> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin 
> wrote:
> >>
> >> was requirement to MySQL database version changes with 1.8.0? I run
> mysql
> >> 14.14 and it worked fine with 1.7.
> >>
> >> I just installed 1.8.4 beta 4 and got this error below when i ran
> airflow
> >> upgradedb command
> >>
> >>  File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
> >> 205, in execute
> >>self.errorhandler(self, exc, value)
> >>  File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
> >> line 36, in defaulterrorhandler
> >>raise errorclass, errorvalue
> >> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
> >> (1064, "You have an error in your SQL syntax; check the manual that
> >> corresponds to your MySQL server version for the right syntax to use
> near
> >> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
> >> DATETIME(6) NULL']
> >>
> >>
> >>
> >>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin 
> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I have made the FOURTH beta of Airflow 1.8.0 available at:
> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >>> are available at https://dist.apache.org/repos/
> >>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
> >>> /dist/release/incubator/airflow/> . It is tagged with a local version
> >>> “apache.incubating” so it allows upgrading from earlier releases. This
> beta
> >>> is available for testing in a more production like setting (acceptance
> >>> environment?).
> >>>
> >>> I would like to encourage everyone  to try it out, to report back any
> >>> issues so we get to a rock solid release of 1.8.0. When reporting
> issues a
> >>> test case or even a fix is highly appreciated.
> >>>
> >>> Issues fixed:
> >>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
> >>> should work again)
> >>> * Py3 incompatibility in base_taskrunner.
> >>>
> >>> Under investigation:
> >>> * DAG marked success, with half of the Tasks never scheduled (Alex)
> >>>
> >>> Kind regards,
> >>> Bolke
> >>
> >>
> >>
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 3

[Alex Van Boxel]

so, yes. doesn't work:

I did:
- create new conda environment
- python setup.py install
- rm airflow.db (sqllite)
- airflow initdb

same error:
self._revision_map
  File
"/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/util/langhelpers.py",
line 240, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
  File
"/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/script/revision.py",
line 151, in _revision_map
down_revision = map_[downrev]
KeyError: '1a5a9e6bf2b5'

consistent as my cluster setup (that one is build in the docker container)
and this one my mac (both give the same error)





On Thu, Jan 26, 2017 at 11:40 AM Bolke de Bruin  wrote:

> Hi Alex
>
> Are you sure you are in a fully clean environment? Sometimes things remain
> in the airflow directories. This worked for me:
>
> rm -rf site-packages/airflow*
> python setup.py install
> airflow initdb
>
> Bolke
>
>
> > On 26 Jan 2017, at 10:53, Alex Van Boxel  wrote:
> >
> > About the database error: starting from scratch also gives the same
> error:
> >
> > Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
> >
> > same error as above.
> >
> > On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel 
> wrote:
> >
> >> Not directly one I can share. I'll spend some time looking at it. I'll
> try
> >> to create a unittest of it.
> >>
> >> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin 
> wrote:
> >>
> >> Do you have a example dag for this? That makes it easier to work on.
> >>
> >> Bolke
> >>
> >> Sent from my iPhone
> >>
> >>> On 26 Jan 2017, at 08:36, Alex Van Boxel  wrote:
> >>>
> >>> Another thing that I noticed (but observed it in beta 2 as well). Is
> the
> >>> following:
> >>>
> >>> - The following trigger should not fire.
> >>> --- Trigger rule is ONE_SUCCESS
> >>> --- upstream: UP_FOR_RETRY + SKIPPED
> >>> => task get's triggered
> >>> => resulting SKIPPED
> >>> => DAG marked success, with half of the Tasks never scheduled
> >>>
> >>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> >>> described above)
> >>> --- *Does this make sense* ?!
> >>>
> >>> Both are a problem, and certainly the combination of both. I'll see if
> I
> >>> can spend some time investigating this.
> >>>
> >>>
> >>>
> >>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin 
> >> wrote:
> >>>>
> >>>> Mmm that is due to the reverting of one changes to the db. Need to
> look
> >>>> into that how to fix it.
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel  wrote:
> >>>>>
> >>>>> I do seem to have a problem upgrading the MySQL database with the
> last
> >>>>> commit:
> >>>>>
> >>>>>
> >>>>> 2017-01-25T23:41:55.662572654Z
> >>>>>
> >>>>
> >>
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> >>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> >>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> >>>> present
> >>>>> 2017-01-25T23:41:55.662613670Z   warnings.warn(msg)
> >>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> >>>>> 2017-01-25T23:41:55.664582238Z   File "/usr/local/bin/airflow", line
> 4,
> >>>> in
> >>>>> 
> >>>>> 2017-01-25T23:41:55.664758457Z
> >>>>>
> >>>>
> >>
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> >>>>> 'airflow')
> >>>>> 2017-01-25T23:41:55.664776553Z   File
> >>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >> line
> >>>>> 739, in run_script
> >>>>> 2017-01-25T23:41:55.664937644Z
> >>>>> self.require(requires)[0].run_script(script_name, ns)
> >>>>> 2017-01-25T23:41:55.664952001Z   File
> >>>>>

Re: Airflow 1.8.0 BETA 3

[Alex Van Boxel]

About the database error: starting from scratch also gives the same error:

Fresh install. Delete airflow.db sqllite db. And the : airflow initdb

same error as above.

On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel  wrote:

> Not directly one I can share. I'll spend some time looking at it. I'll try
> to create a unittest of it.
>
> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin  wrote:
>
> Do you have a example dag for this? That makes it easier to work on.
>
> Bolke
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 08:36, Alex Van Boxel  wrote:
> >
> > Another thing that I noticed (but observed it in beta 2 as well). Is the
> > following:
> >
> > - The following trigger should not fire.
> > --- Trigger rule is ONE_SUCCESS
> > --- upstream: UP_FOR_RETRY + SKIPPED
> >  => task get's triggered
> >  => resulting SKIPPED
> >  => DAG marked success, with half of the Tasks never scheduled
> >
> > - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> > described above)
> > --- *Does this make sense* ?!
> >
> > Both are a problem, and certainly the combination of both. I'll see if I
> > can spend some time investigating this.
> >
> >
> >
> >> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin 
> wrote:
> >>
> >> Mmm that is due to the reverting of one changes to the db. Need to look
> >> into that how to fix it.
> >>
> >> Sent from my iPhone
> >>
> >>> On 26 Jan 2017, at 00:51, Alex Van Boxel  wrote:
> >>>
> >>> I do seem to have a problem upgrading the MySQL database with the last
> >>> commit:
> >>>
> >>>
> >>> 2017-01-25T23:41:55.662572654Z
> >>>
> >>
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> >>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> >>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> >> present
> >>> 2017-01-25T23:41:55.662613670Z   warnings.warn(msg)
> >>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> >>> 2017-01-25T23:41:55.664582238Z   File "/usr/local/bin/airflow", line 4,
> >> in
> >>> 
> >>> 2017-01-25T23:41:55.664758457Z
> >>>
> >>
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> >>> 'airflow')
> >>> 2017-01-25T23:41:55.664776553Z   File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 739, in run_script
> >>> 2017-01-25T23:41:55.664937644Z
> >>> self.require(requires)[0].run_script(script_name, ns)
> >>> 2017-01-25T23:41:55.664952001Z   File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 1494, in run_script
> >>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> >>> 2017-01-25T23:41:55.665086884Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> >>> line 28, in 
> >>> 2017-01-25T23:41:55.665211755Z args.func(args)
> >>> 2017-01-25T23:41:55.665225157Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> >>> line 931, in upgradedb
> >>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> >>> 2017-01-25T23:41:55.665346979Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> >>> line 292, in upgradedb
> >>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> >>> 2017-01-25T23:41:55.665462333Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>> line 174, in upgrade
> >>> 2017-01-25T23:41:55.665521466Z script.run_env()
> >>> 2017-01-25T23:41:55.665527232Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>> line 416, in run_env2017-01-25T23:41:55.665638879Z
> >>> util.load_python_file(self.dir, 'env.py')
> >>> 

Re: Airflow 1.8.0 BETA 3

[Alex Van Boxel]

Not directly one I can share. I'll spend some time looking at it. I'll try
to create a unittest of it.

On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin  wrote:

> Do you have a example dag for this? That makes it easier to work on.
>
> Bolke
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 08:36, Alex Van Boxel  wrote:
> >
> > Another thing that I noticed (but observed it in beta 2 as well). Is the
> > following:
> >
> > - The following trigger should not fire.
> > --- Trigger rule is ONE_SUCCESS
> > --- upstream: UP_FOR_RETRY + SKIPPED
> >  => task get's triggered
> >  => resulting SKIPPED
> >  => DAG marked success, with half of the Tasks never scheduled
> >
> > - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> > described above)
> > --- *Does this make sense* ?!
> >
> > Both are a problem, and certainly the combination of both. I'll see if I
> > can spend some time investigating this.
> >
> >
> >
> >> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin 
> wrote:
> >>
> >> Mmm that is due to the reverting of one changes to the db. Need to look
> >> into that how to fix it.
> >>
> >> Sent from my iPhone
> >>
> >>> On 26 Jan 2017, at 00:51, Alex Van Boxel  wrote:
> >>>
> >>> I do seem to have a problem upgrading the MySQL database with the last
> >>> commit:
> >>>
> >>>
> >>> 2017-01-25T23:41:55.662572654Z
> >>>
> >>
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> >>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> >>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> >> present
> >>> 2017-01-25T23:41:55.662613670Z   warnings.warn(msg)
> >>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> >>> 2017-01-25T23:41:55.664582238Z   File "/usr/local/bin/airflow", line 4,
> >> in
> >>> 
> >>> 2017-01-25T23:41:55.664758457Z
> >>>
> >>
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> >>> 'airflow')
> >>> 2017-01-25T23:41:55.664776553Z   File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 739, in run_script
> >>> 2017-01-25T23:41:55.664937644Z
> >>> self.require(requires)[0].run_script(script_name, ns)
> >>> 2017-01-25T23:41:55.664952001Z   File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 1494, in run_script
> >>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> >>> 2017-01-25T23:41:55.665086884Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> >>> line 28, in 
> >>> 2017-01-25T23:41:55.665211755Z args.func(args)
> >>> 2017-01-25T23:41:55.665225157Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> >>> line 931, in upgradedb
> >>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> >>> 2017-01-25T23:41:55.665346979Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> >>> line 292, in upgradedb
> >>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> >>> 2017-01-25T23:41:55.665462333Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>> line 174, in upgrade
> >>> 2017-01-25T23:41:55.665521466Z script.run_env()
> >>> 2017-01-25T23:41:55.665527232Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>> line 416, in run_env2017-01-25T23:41:55.665638879Z
> >>> util.load_python_file(self.dir, 'env.py')
> >>> 2017-01-25T23:41:55.665656437Z   File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> >>> line 93, in load_python_file
> >>> 2017-01-

Re: Airflow 1.8.0 BETA 3

[Alex Van Boxel]

The downgrade is ok for testing, but we can't release with this change
(can't expect people to install a beta first).

On Thu, Jan 26, 2017 at 8:36 AM Alex Van Boxel  wrote:

> Another thing that I noticed (but observed it in beta 2 as well). Is the
> following:
>
> - The following trigger should not fire.
>  --- Trigger rule is ONE_SUCCESS
>  --- upstream: UP_FOR_RETRY + SKIPPED
>   => task get's triggered
>   => resulting SKIPPED
>   => DAG marked success, with half of the Tasks never scheduled
>
> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> described above)
>  --- *Does this make sense* ?!
>
> Both are a problem, and certainly the combination of both. I'll see if I
> can spend some time investigating this.
>
>
>
> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin  wrote:
>
> Mmm that is due to the reverting of one changes to the db. Need to look
> into that how to fix it.
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 00:51, Alex Van Boxel  wrote:
> >
> > I do seem to have a problem upgrading the MySQL database with the last
> > commit:
> >
> >
> > 2017-01-25T23:41:55.662572654Z
> >
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> > UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> > 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> present
> > 2017-01-25T23:41:55.662613670Z   warnings.warn(msg)
> > 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> > 2017-01-25T23:41:55.664582238Z   File "/usr/local/bin/airflow", line 4,
> in
> > 
> > 2017-01-25T23:41:55.664758457Z
> >
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> > 'airflow')
> > 2017-01-25T23:41:55.664776553Z   File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 739, in run_script
> > 2017-01-25T23:41:55.664937644Z
> > self.require(requires)[0].run_script(script_name, ns)
> > 2017-01-25T23:41:55.664952001Z   File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 1494, in run_script
> > 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> > 2017-01-25T23:41:55.665086884Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> > line 28, in 
> > 2017-01-25T23:41:55.665211755Z args.func(args)
> > 2017-01-25T23:41:55.665225157Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> > line 931, in upgradedb
> > 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> > 2017-01-25T23:41:55.665346979Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> > line 292, in upgradedb
> > 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> > 2017-01-25T23:41:55.665462333Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> > line 174, in upgrade
> > 2017-01-25T23:41:55.665521466Z script.run_env()
> > 2017-01-25T23:41:55.665527232Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> > line 416, in run_env2017-01-25T23:41:55.665638879Z
> > util.load_python_file(self.dir, 'env.py')
> > 2017-01-25T23:41:55.665656437Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> > line 93, in load_python_file
> > 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> path)
> > 2017-01-25T23:41:55.665699257Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> > line 79, in load_module_py
> > 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
> > fp)
> > 2017-01-25T23:41:55.665751433Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 88, in 
> > 2017-01-25T23:41:55.665778219Z run_migrations_online()
> > 2017-01-25T23:41:55.665787820Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 83, in run_migrations_online
>

Re: Airflow 1.8.0 BETA 3

[Alex Van Boxel]

Another thing that I noticed (but observed it in beta 2 as well). Is the
following:

- The following trigger should not fire.
 --- Trigger rule is ONE_SUCCESS
 --- upstream: UP_FOR_RETRY + SKIPPED
  => task get's triggered
  => resulting SKIPPED
  => DAG marked success, with half of the Tasks never scheduled

- UP_FOR_RETRY is propagated downstream (actually resulting in failure
described above)
 --- *Does this make sense* ?!

Both are a problem, and certainly the combination of both. I'll see if I
can spend some time investigating this.



On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin  wrote:

> Mmm that is due to the reverting of one changes to the db. Need to look
> into that how to fix it.
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 00:51, Alex Van Boxel  wrote:
> >
> > I do seem to have a problem upgrading the MySQL database with the last
> > commit:
> >
> >
> > 2017-01-25T23:41:55.662572654Z
> >
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> > UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> > 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> present
> > 2017-01-25T23:41:55.662613670Z   warnings.warn(msg)
> > 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> > 2017-01-25T23:41:55.664582238Z   File "/usr/local/bin/airflow", line 4,
> in
> > 
> > 2017-01-25T23:41:55.664758457Z
> >
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> > 'airflow')
> > 2017-01-25T23:41:55.664776553Z   File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 739, in run_script
> > 2017-01-25T23:41:55.664937644Z
> > self.require(requires)[0].run_script(script_name, ns)
> > 2017-01-25T23:41:55.664952001Z   File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 1494, in run_script
> > 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> > 2017-01-25T23:41:55.665086884Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> > line 28, in 
> > 2017-01-25T23:41:55.665211755Z args.func(args)
> > 2017-01-25T23:41:55.665225157Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> > line 931, in upgradedb
> > 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> > 2017-01-25T23:41:55.665346979Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> > line 292, in upgradedb
> > 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> > 2017-01-25T23:41:55.665462333Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> > line 174, in upgrade
> > 2017-01-25T23:41:55.665521466Z script.run_env()
> > 2017-01-25T23:41:55.665527232Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> > line 416, in run_env2017-01-25T23:41:55.665638879Z
> > util.load_python_file(self.dir, 'env.py')
> > 2017-01-25T23:41:55.665656437Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> > line 93, in load_python_file
> > 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> path)
> > 2017-01-25T23:41:55.665699257Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> > line 79, in load_module_py
> > 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
> > fp)
> > 2017-01-25T23:41:55.665751433Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 88, in 
> > 2017-01-25T23:41:55.665778219Z run_migrations_online()
> > 2017-01-25T23:41:55.665787820Z   File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 83, in run_migrations_online
> > 2017-01-25T23:41:55.665811353Z context.run_migrations()
> > 2017-01-25T23:41:55.665818933Z   File "", line 8, in
> run_migrations
> > 2017-01-25T23:41:55.666707547Z   File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.

Re: Airflow 1.8.0 BETA 3

[Alex Van Boxel]

:41:55.667419226Z resolved_id, branch_label =
self._resolve_revision_number(id_)
2017-01-25T23:41:55.667434110Z   File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 433, in _resolve_revision_number
2017-01-25T23:41:55.667576273Z self._revision_map
2017-01-25T23:41:55.667590641Z   File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
line 240, in __get__
2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
self.fget(obj)
2017-01-25T23:41:55.667658598Z   File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 151, in _revision_map
2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'






On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin  wrote:

> Hi All,
>
> I have made the THIRD beta of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
> are available at
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
> tagged with a local version “apache.incubating” so it allows upgrading from
> earlier releases. This beta is available for testing in a more production
> like setting (acceptance environment?).
>
> I would like to encourage everyone  to try it out, to report back any
> issues so we get to a rock solid release of 1.8.0. When reporting issues a
> test case or even a fix is highly appreciated.
>
> Issues cleared:
>
> * Manual trigger not working
> * Performance issues with MySQL
> * Postgres auto-commit support left over to the driver
> * Poison pill taken while task has exited
> * Keep cgroups optional
> * Funcsigs pinned to 1.0.0
>
> Issue(s) remaining (blocker for RC):
> * Cgroups not py3 compatible
>
> If all goes well we should have a Release Candidate on Feb 2. Thanks for
> reporting issues and keep on testing please :). Moving towards RC I tend to
> like small bug fixes only. When we mark RC (do we need to vote on this?)
> the procedure becomes even more strict. Please remember that the FINAL
> release is dependent on a vote on the IPMC mailinglist.
>
> Cheers
> Bolke

-- 
  _/
_/ Alex Van Boxel

Medium series: Airflow for Google Cloud

[Alex Van Boxel]

Hey all,

now that 1.8 is nearing release. I finally started writing about Airflow.
As it's me writing, I'll be focussing on the Google Cloud integration.

Today's post is about BigQuery
https://medium.com/google-cloud/airflow-for-google-cloud-part-1-d7da9a048aa4#.qe6f0gldf

Next one will be about DataProc.
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 2

[Alex Van Boxel]

Installing in production on Monday as well. Alpha's run ok till now.

On Fri, Jan 20, 2017 at 7:39 PM Chris Riccomini 
wrote:

> Installed in dev. Prod will go on Monday. Will keep you posted.
>
> On Fri, Jan 20, 2017 at 9:35 AM, Bolke de Bruin  wrote:
>
> > Yes
> >
> > Sent from my iPhone
> >
> > > On 20 Jan 2017, at 18:20, Boris Tyukin  wrote:
> > >
> > > just to make sure this is the latest one, right?
> > > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> > airflow-1.8.0b2+apache.incubating.tar.gz
> > >
> > >> On Fri, Jan 20, 2017 at 10:57 AM, Bolke de Bruin 
> > wrote:
> > >>
> > >> Hi All,
> > >>
> > >> I have made the SECOND beta of Airflow 1.8.0 available at:
> > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> > keys
> > >> are available at
> https://dist.apache.org/repos/dist/release/incubator/
> > >> airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/
> > >
> > >> . It is tagged with a local version “apache.incubating” so it allows
> > >> upgrading from earlier releases. This beta is available for testing
> in a
> > >> more production like setting (acceptance environment?).
> > >>
> > >> I would like to encourage everyone  to try it out, to report back any
> > >> issues so we get to a rock solid release of 1.8.0. When reporting
> > issues a
> > >> test case or even a fix is highly appreciated.
> > >>
> > >> Cgroups+impersonation are now in. This means we are feature complete
> for
> > >> 1.8.0!
> > >>
> > >> Cheers
> > >> Bolke
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Experiences with 1.8.0

[Alex Van Boxel]

Bolke, I will tackle logging because I started integrating with Google
StackDriver logging. I have a separate branch, but this will be for after
1.8. It will be configurable and context aware and everyone will benefit
(not only stackdriver logging).

On Fri, Jan 20, 2017 at 11:55 PM Bolke de Bruin  wrote:

> Will do. And thanks.
>
> Adding another issue:
>
> * Some of our DAGs are not getting scheduled for some unknown reason.
> Need to investigate why.
>
> Related but not root cause:
> * Logging is so chatty that it gets really hard to find the real issue
>
> Bolke.
>
> > On 20 Jan 2017, at 23:45, Dan Davydov 
> wrote:
> >
> > I'd be happy to lend a hand fixing these issues and hopefully some others
> > are too. Do you mind creating jiras for these since you have the full
> > context? I have created a JIRA for (1) and have assigned it to myself:
> > https://issues.apache.org/jira/browse/AIRFLOW-780
> >
> > On Fri, Jan 20, 2017 at 1:01 AM, Bolke de Bruin 
> wrote:
> >
> >> This is to report back on some of the (early) experiences we have with
> >> Airflow 1.8.0 (beta 1 at the moment):
> >>
> >> 1. The UI does not show faulty DAG, leading to confusion for developers.
> >> When a faulty dag is placed in the dags folder the UI would report a
> >> parsing error. Now it doesn’t due to the separate parising (but not
> >> reporting back errors)
> >>
> >> 2. The hive hook sets ‘airflow.ctx.dag_id’ in hive
> >> We run in a secure environment which requires this variable to be
> >> whitelisted if it is modified (needs to be added to UPDATING.md)
> >>
> >> 3. DagRuns do not exist for certain tasks, but don’t get fixed
> >> Log gets flooded without a suggestion what to do
> >>
> >> 4. At start up all running dag_runs are being checked, we seemed to
> have a
> >> lot of “left over” dag_runs (couple of thousand)
> >> - Checking was logged to INFO -> requires a fsync for every log message
> >> making it very slow
> >> - Checking would happen at every restart, but dag_runs’ states were not
> >> being updated
> >> - These dag_runs would never er be marked anything else than running for
> >> some reason
> >> -> Applied work around to update all dag_run in sql before a certain
> date
> >> to -> finished
> >> -> need to investigate why dag_runs did not get marked “finished/failed”
> >>
> >> 5. Our umask is set to 027
> >>
> >>
>
> --
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 BETA 1

[Alex Van Boxel]

Hey Max,

As I'm missing the 1.7.2 labels I compared to the 172 branch. Can you have
a look at PR 2000. Its also sanitised, removing some of the commits that
doesn't bring value to the users.

On Wed, Jan 18, 2017, 02:51 Maxime Beauchemin 
wrote:

> Alex, for the CHANGELOG.md, I've been using `github-changes`, a js app that
> make changelog generation flexible and easy.
>
> https://www.npmjs.com/package/github-changes
>
> Command looks something like:
> `github-changes -o apache -r incubator-airflow --token 
> --between-tags 1.7.2...1.8.0beta` (tags may differ, it's easy to get a
> token on your GH profile page)
>
> This will write a `CHANGELOG.md` in your cwd that you can just add on top
> of the existing one
>
> Max
>
> On Jan 17, 2017 3:37 PM, "Dan Davydov" 
> wrote:
>
> > So it is, my bad. Bad skills with ctrl-f :).
> >
> > On Tue, Jan 17, 2017 at 3:31 PM, Bolke de Bruin 
> wrote:
> >
> > > Arthur's change is already in!
> > >
> > > B.
> > >
> > > Sent from my iPhone
> > >
> > > > On 17 Jan 2017, at 22:20, Dan Davydov  .INVALID>
> > > wrote:
> > > >
> > > > Would be good to cherrypick Arthur's fix into here if possible:
> > > > https://github.com/apache/incubator-airflow/pull/1973/files (commit
> > > > 43bf89d)
> > > >
> > > > The impersonation stuff should be wrapping up shortly pending Bolke's
> > > > comments.
> > > >
> > > > Also agreed with Max on the thanks. Thanks Alex too for the change
> log!
> > > >
> > > > On Tue, Jan 17, 2017 at 10:05 AM, Maxime Beauchemin <
> > > > maximebeauche...@gmail.com> wrote:
> > > >
> > > >> Bolke, I couldn't thank you enough for driving the release process!
> > > >>
> > > >> I'll coordinate with the Airbnb team around impersonation/CGROUPs
> and
> > on
> > > >> making sure we put this release in our staging ASAP. We have our
> > > employee
> > > >> conference this week so things are slower, but we'll be back at full
> > > speed
> > > >> Friday.
> > > >>
> > > >> Max
> > > >>
> > > >>> On Mon, Jan 16, 2017 at 3:51 PM, Alex Van Boxel 
> > > wrote:
> > > >>>
> > > >>> Hey Bolke, thanks great wotk. I'll handle the CHANGELOG, and add
> some
> > > >>> documentation about triggers with branching operators.
> > > >>>
> > > >>> About the Google Cloud Operators: I wouldn't call it feature
> > > complete...
> > > >> it
> > > >>> never is.
> > > >>>
> > > >>>
> > > >>> On Mon, Jan 16, 2017 at 11:24 PM Bolke de Bruin  >
> > > >> wrote:
> > > >>>
> > > >>>> Dear All,
> > > >>>>
> > > >>>> I have made the first BETA of Airflow 1.8.0 available at:
> > > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> public
> > > >> keys
> > > >>>> are available at
> > > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> > > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> .
> It
> > > is
> > > >>>> tagged with a local version “apache.incubating” so it allows
> > upgrading
> > > >>> from
> > > >>>> earlier releases. This beta is available for testing in a more
> > > >> production
> > > >>>> like setting (acceptance environment?).
> > > >>>>
> > > >>>> I would like to encourage everyone  to try it out, to report back
> > any
> > > >>>> issues so we get to a rock solid release of 1.8.0. When reporting
> > > >> issues
> > > >>> a
> > > >>>> test case or even a fix is highly appreciated.
> > > >>>>
> > > >>>> By moving to beta, we are also in feature freeze mode. Meaning no
> > > major
> > > >>>> adjustments or additions can be made to the v1-8-test branch.
> There
> > is
> > > >>> one
> > > >>>> exception: the cgroups+impersonation

Re: Airflow 1.8.0 BETA 1

[Alex Van Boxel]

Hey Bolke, thanks great wotk. I'll handle the CHANGELOG, and add some
documentation about triggers with branching operators.

About the Google Cloud Operators: I wouldn't call it feature complete... it
never is.


On Mon, Jan 16, 2017 at 11:24 PM Bolke de Bruin  wrote:

> Dear All,
>
> I have made the first BETA of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
> are available at
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
> tagged with a local version “apache.incubating” so it allows upgrading from
> earlier releases. This beta is available for testing in a more production
> like setting (acceptance environment?).
>
> I would like to encourage everyone  to try it out, to report back any
> issues so we get to a rock solid release of 1.8.0. When reporting issues a
> test case or even a fix is highly appreciated.
>
> By moving to beta, we are also in feature freeze mode. Meaning no major
> adjustments or additions can be made to the v1-8-test branch. There is one
> exception: the cgroups+impersonation patch. I was assured that before we
> merge that it will be thoroughly tested, so its can still enter 1.8 if
> within a reasonable time frame. A lot of work has gone into it and it would
> be a shame if we would lose momentum.
>
> Finally, it would also be really nice of have some updates to the
> documentation. In order of importance:
>
> * UPDATING.md What does a user need to think of when upgrading to 1.8?
> MySQL 5.6.4 is now minimally required, scheduler now has separate logs per
> file processor.
> * docs/configuration.rst We have many new options, especially in the
> scheduler area
> * docs/faq.rst
> * CHANGELOG.txt (compiled from git log)
> * swagger definitions for the API
>
> HIGHLIGHTS of the beta:
>
> * DAG catchup: If False the scheduler does not fill in the gaps between
> the start_date and the current_date. Can be specified per dag or globally
> * Per DAG multi processing: More robust and faster DAG processing. A
> faulty DAG should not take down the scheduler any more
> * Google Cloud Operators: Feature complete I have heard.
> * Time units now dynamic UI
> * Better SMTP handling and attachment support
> * Operational metrics for the scheduler
> * MSSQL Improvements
> * Experimental Rest API with Kerberos support
> * Auto alignment of start_date to interval
> * Better support for sub second scheduling
> * Rolling restart of web workers
> * nvd3.js instead of highcharts
> * New dependency engine making debugging why my task is running easier
> * Many UI updates
> * Many new operators
> * Many, many, many bugfixes
>
> RELEASE PLANNING
>
> Beta 2: 20 Jan
> Beta 3: 25 Jan
> RC1:  2 Feb
>
> Cheers
> Bolke
>
>
>
> --
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 alpha 5

[Alex Van Boxel]

Hey Bolke, I've missid this requirement: daemonizing the webserver?! Don't
do this by default. That would brake running on Kubernetes or in Docker.
Make it optional with a flag!

On Fri, Jan 13, 2017 at 8:48 PM Bolke de Bruin  wrote:

> Dear All,
>
> I have made Airflow 1.8.0 alpha 5 available at
> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> .
> Again no Apache release yet - this is for testing purposes. I consider this
> Alpha to be a Beta if not for the pending features. If the pending features
> are merged within a reasonable time frame (except for **, as no progress
> currently) then I am planning to mark the tarball as Beta and only allow
> bug fixes and (very) minor features.
>
> Blockers:
> * None
>
> Fixed issues
> * DAG.catchup : minor changes needed, documentation still required,
> integration tests seem to pass flawlessly
> * Respect retry_delay
>
> Pending features:
> * Cgroups + impersonation: clean up of patches on going, more tests and
> more elaborate documentation required. Integration tests not done yet
>
> Todo’s (Beta permitted):
> * set default log location for Childs correctly
> * fix systems scripts
> * daemonizing of webserver
>
> Almost there!
> Bolke

-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 alpha 4

[Alex Van Boxel]

Hey Bolke, I'll be of for a few days but I started looking at the
CHANGELIST to make sure we now have a list of changes for 1.8 (seems like a
big list...).

I'll will also drop the trigger issue for the 1.8 and see what we can do
for the next version. I'll make it into a documentation issue.

On Wed, Jan 11, 2017 at 1:46 PM Bolke de Bruin  wrote:

> Dear All,
>
> I would like to drop "Schedule all pending DAG runs in a single scheduler
> loop” from the 1.8.0 release (updated:
> https://github.com/apache/incubator-airflow/pull/1980 <
> https://github.com/apache/incubator-airflow/pull/1980>, original:
> https://github.com/apache/incubator-airflow/pull/1906 <
> https://github.com/apache/incubator-airflow/pull/1906>). The reason for
> this is that it, imho, biases the scheduler towards a single DAG as it
> fills the queue with tasks from one DAG and then goes to the next DAG.
> Starving DAGs that come after the first for resources. As such it should be
> updated and that will take time.
>
> Please let me know if I am incorrect.
>
> Thanks
> Bolke
>
> > On 10 Jan 2017, at 09:25, Bolke de Bruin  wrote:
> >
> > Dear All,
> >
> > I have made Airflow 1.8.0 alpha 4 available at
> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> .
> Again no Apache release yet - this is for testing purposes. I consider this
> Alpha to be a Beta if not for the pending features. If the pending features
> are merged within a reasonable time frame (except for **, as no progress
> currently) then I am planning to mark the tarball as Beta and only allow
> bug fixes and (very) minor features. This week hopefully.
> >
> > Blockers:
> >
> > * None
> >
> > Fixed issues
> > * Regression in email
> > * LDAP case sensitivity
> > * one_failed task not being run: now seems to pass suddenly (so fixed?)
> -> need to investigate why
> > * Email attachments
> > * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression)
> > * Improve time units for task performance charts
> > * XCom throws an duplicate / locking error
> > * Add execution_date to trigger_dag
> >
> > Pending features:
> > * DAG.catchup : minor changes needed, documentation still required,
> integration tests seem to pass flawlessly
> > * Cgroups + impersonation: clean up of patches on going, more tests and
> more elaborate documentation required. Integration tests not executed yet
> > * Schedule all pending DAG runs in a single scheduler loop: no progress
> (**)
> >
> > Cheers!
> > Bolke
>
> --
  _/
_/ Alex Van Boxel

Re: Refactoring Connection

[Alex Van Boxel]

I was actually going to propose something different with entry-points, but
your requirement beat me to it (but that's ok :-). Actually I think with
this mechanism people would be able to extend Airflow connection mechanism
(and later other stuff) by doing *pip install airflow-sexy-new-connection*
(for example).

On Mon, Jan 9, 2017 at 1:39 PM Gael Magnan  wrote:

> Thank you for the read, I'm gonna look at it, it's probably gonna be better
> that what I have.
>
> Point taken about the URI, I'll see if i can find something generic enough
> to handle all those cases.
>
> Le lun. 9 janv. 2017 à 13:36, Alex Van Boxel  a écrit :
>
> > Thanks a lot, yes it clarifies a lot and I do agree you really need to
> hack
> > inside Airflow to add a Connection type. While you're working at this
> could
> > you have a look at the standard python *entry-point mechanism* for
> > registering Connection types/components.
> >
> > A good read on this:
> >
> >
> http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html
> >
> > My first though would be that just by adding an entry to the factory
> method
> > would be enough to register your Connection + ConnectionType and UI.
> >
> > Also note that not everything works with a URI. The Google Cloud
> Connection
> > doesn't have one, it uses a secret key file stored on disk, so don't
> force
> > every connection type to work with URI's.
> >
> >
> >
> > On Mon, Jan 9, 2017 at 1:15 PM Gael Magnan  wrote:
> >
> > > Yes sure,
> > >
> > > The question was the following:
> > > "I was looking at the code of the connections, and I realized you can't
> > > easily add a connection type without modifying the airflow code
> source. I
> > > wanted to create a mongodb connection type, but I think the best
> approche
> > > would be to refactor connections first. Thoughts anyone?"
> > >
> > > The answer of Bolke de Bruin was: "making it more generic would be
> > > appreciated"
> > >
> > > So basically the way the code is set up actually every types of
> > connection
> > > existing is defined though a list in the Connection class. It
> implements
> > > exactly the same code for parsing uri to get connections info and
> doesn't
> > > allow for a simple way to get back the uri from the connection infos.
> > >
> > > I need to add a mongodb connection and a way to get it back as a uri,
> so
> > i
> > > could use an other type of connection and play around with that or
> juste
> > > add one more hard coded connection type, but I though this might be
> > > something that comes back regularly and having a simple way to plug in
> > new
> > > types of connection would make it easier for anyone to contribute a new
> > > connection type.
> > >
> > > Hope this clarifies my proposal.
> > >
> > > Le lun. 9 janv. 2017 à 12:46, Alex Van Boxel  a
> écrit
> > :
> > >
> > > > Hey Gael,
> > > >
> > > > could you please recap the question here and provide some context.
> Not
> > > > everyone on the mailinglist is actively following Gitter, including
> me.
> > > > With some context it would be easier to give feedback. Thanks.
> > > >
> > > > On Mon, Jan 9, 2017 at 11:15 AM Gael Magnan 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > following my question on gitter the other day and the response from
> > > Bolke
> > > > > de Bruin, I've started working on refactoring the connections in
> > > airflow.
> > > > >
> > > > > Before submitting a PR I wanted to share my proposal with you and
> get
> > > > > feedbacks.
> > > > >
> > > > > The idea is quite simple, I've divided the Connection class in two,
> > > > > Connection and ConnectionType, connection has the same interface it
> > had
> > > > > before plus a few methods, but the class keeps a reference to a
> > > > dictionary
> > > > > of registered ConnectionType. It delegates the work of parsing from
> > > URI,
> > > > > formatting to URI (added) and getting the hook to the
> ConnectionType
> > > > > associated with the conn_type.
> > > > >
> > > > > I've thought of two ways of registering 

Re: Refactoring Connection

[Alex Van Boxel]

Thanks a lot, yes it clarifies a lot and I do agree you really need to hack
inside Airflow to add a Connection type. While you're working at this could
you have a look at the standard python *entry-point mechanism* for
registering Connection types/components.

A good read on this:
http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html

My first though would be that just by adding an entry to the factory method
would be enough to register your Connection + ConnectionType and UI.

Also note that not everything works with a URI. The Google Cloud Connection
doesn't have one, it uses a secret key file stored on disk, so don't force
every connection type to work with URI's.



On Mon, Jan 9, 2017 at 1:15 PM Gael Magnan  wrote:

> Yes sure,
>
> The question was the following:
> "I was looking at the code of the connections, and I realized you can't
> easily add a connection type without modifying the airflow code source. I
> wanted to create a mongodb connection type, but I think the best approche
> would be to refactor connections first. Thoughts anyone?"
>
> The answer of Bolke de Bruin was: "making it more generic would be
> appreciated"
>
> So basically the way the code is set up actually every types of connection
> existing is defined though a list in the Connection class. It implements
> exactly the same code for parsing uri to get connections info and doesn't
> allow for a simple way to get back the uri from the connection infos.
>
> I need to add a mongodb connection and a way to get it back as a uri, so i
> could use an other type of connection and play around with that or juste
> add one more hard coded connection type, but I though this might be
> something that comes back regularly and having a simple way to plug in new
> types of connection would make it easier for anyone to contribute a new
> connection type.
>
> Hope this clarifies my proposal.
>
> Le lun. 9 janv. 2017 à 12:46, Alex Van Boxel  a écrit :
>
> > Hey Gael,
> >
> > could you please recap the question here and provide some context. Not
> > everyone on the mailinglist is actively following Gitter, including me.
> > With some context it would be easier to give feedback. Thanks.
> >
> > On Mon, Jan 9, 2017 at 11:15 AM Gael Magnan 
> wrote:
> >
> > > Hi,
> > >
> > > following my question on gitter the other day and the response from
> Bolke
> > > de Bruin, I've started working on refactoring the connections in
> airflow.
> > >
> > > Before submitting a PR I wanted to share my proposal with you and get
> > > feedbacks.
> > >
> > > The idea is quite simple, I've divided the Connection class in two,
> > > Connection and ConnectionType, connection has the same interface it had
> > > before plus a few methods, but the class keeps a reference to a
> > dictionary
> > > of registered ConnectionType. It delegates the work of parsing from
> URI,
> > > formatting to URI (added) and getting the hook to the ConnectionType
> > > associated with the conn_type.
> > >
> > > I've thought of two ways of registering new ConnectionTypes, the first
> is
> > > making the BaseConnectionType use a metaclass that registered any new
> > > ConnectionType with Connection when the class is declared, it would
> > require
> > > the less work to extend the connection module, as just importing the
> file
> > > with the connection would do the trick.
> > > The second one is juste to have a function/classmethod that you call
> > > manually to register your connection. It would be simpler to understand
> > but
> > > requires more work every time you create a new ConnectionType.
> > >
> > > Hope this proposal is clear enough, and I'm waiting for feebacks and
> > > possible improvements.
> > >
> > > Regards
> > > Gael Magnan de Bornier
> > >
> > --
> >   _/
> > _/ Alex Van Boxel
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Refactoring Connection

[Alex Van Boxel]

Hey Gael,

could you please recap the question here and provide some context. Not
everyone on the mailinglist is actively following Gitter, including me.
With some context it would be easier to give feedback. Thanks.

On Mon, Jan 9, 2017 at 11:15 AM Gael Magnan  wrote:

> Hi,
>
> following my question on gitter the other day and the response from Bolke
> de Bruin, I've started working on refactoring the connections in airflow.
>
> Before submitting a PR I wanted to share my proposal with you and get
> feedbacks.
>
> The idea is quite simple, I've divided the Connection class in two,
> Connection and ConnectionType, connection has the same interface it had
> before plus a few methods, but the class keeps a reference to a dictionary
> of registered ConnectionType. It delegates the work of parsing from URI,
> formatting to URI (added) and getting the hook to the ConnectionType
> associated with the conn_type.
>
> I've thought of two ways of registering new ConnectionTypes, the first is
> making the BaseConnectionType use a metaclass that registered any new
> ConnectionType with Connection when the class is declared, it would require
> the less work to extend the connection module, as just importing the file
> with the connection would do the trick.
> The second one is juste to have a function/classmethod that you call
> manually to register your connection. It would be simpler to understand but
> requires more work every time you create a new ConnectionType.
>
> Hope this proposal is clear enough, and I'm waiting for feebacks and
> possible improvements.
>
> Regards
> Gael Magnan de Bornier
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow Release Planning and Supported Release Lifetime

[Alex Van Boxel]

Thanks for clarifying (I'm new to this Apache releasing ;-)

On Sun, Jan 8, 2017 at 6:58 PM Bolke de Bruin  wrote:

> This would be for changes AFTER release / rc. Ie. an RC is basically what
> we as a community deem stable and under normal circumstances is the equal
> to the release. A release is done by a release manager (per Apache
> guidelines) so it makes sense that a release manager can only apply patches
> to a release. For this release I am the release manager.
>
> Alpha and beta versions are open to any committer.
>
> That's the idea which to me makes sense, but maybe an other option is
> better?
>
> Bolke
>
>
> Sent from my iPhone
>
> > On 8 Jan 2017, at 18:27, Alex Van Boxel  wrote:
> >
> > This looks good, except do we need a release manager that applies
> patches?
> >
> >> On Sun, Jan 8, 2017, 14:36 Bolke de Bruin  wrote:
> >>
> >> Hi All,
> >>
> >> As part of the release process I have created "Airflow Release Planning
> >> and Supported Release Lifetime” (
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime
> >).
> >> I borrowed heavily from Samba’s Release Planning for this, so any
> >> resemblance is not coincidental :-).
> >>
> >> Please take a look and make suggestions as not all may fit our rhythm.
> >> Main take aways:
> >>
> >> * We aim to do a major release every 6 months (ie. 1.8 -> 1.9)
> >> * Minor releases (1.8.0 -> 1.8.1) can happen whenever needed.
> >> * We only support (“maintenance mode”) N-1. So if 1.9.0 is released,
> 1.8.X
> >> enters maintenance. 1.7.X is EOL’d.
> >> * Patches to closed branches (ie. RC+) need to have a signoff from
> another
> >> committer and support from the mailinglist (Can this be done in the
> Apache
> >> way?). A release manager then needs to apply te patch.
> >>
> >> Other:
> >> * Patches land on master first
> >> * Branches are maintained as “vX.Y-test” and “vX.Y-stable”. No minor
> >> branches. Thus when 1.8.0 is released, this will be the stable branch
> >> “v1.8-stable”, automatically “v1.8-test” becomes the to be 1.8.1
> version.
> >>
> >> I hope this makes sense. Do we need to vote on this?
> >>
> >> Cheers
> >> Bolke
> >>
> >>
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow Release Planning and Supported Release Lifetime

[Alex Van Boxel]

This looks good, except do we need a release manager that applies patches?

On Sun, Jan 8, 2017, 14:36 Bolke de Bruin  wrote:

> Hi All,
>
> As part of the release process I have created "Airflow Release Planning
> and Supported Release Lifetime” (
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime>).
> I borrowed heavily from Samba’s Release Planning for this, so any
> resemblance is not coincidental :-).
>
> Please take a look and make suggestions as not all may fit our rhythm.
> Main take aways:
>
> * We aim to do a major release every 6 months (ie. 1.8 -> 1.9)
> * Minor releases (1.8.0 -> 1.8.1) can happen whenever needed.
> * We only support (“maintenance mode”) N-1. So if 1.9.0 is released, 1.8.X
> enters maintenance. 1.7.X is EOL’d.
> * Patches to closed branches (ie. RC+) need to have a signoff from another
> committer and support from the mailinglist (Can this be done in the Apache
> way?). A release manager then needs to apply te patch.
>
> Other:
> * Patches land on master first
> * Branches are maintained as “vX.Y-test” and “vX.Y-stable”. No minor
> branches. Thus when 1.8.0 is released, this will be the stable branch
> “v1.8-stable”, automatically “v1.8-test” becomes the to be 1.8.1 version.
>
> I hope this makes sense. Do we need to vote on this?
>
> Cheers
> Bolke
>
>

Trigger behaviour with skipped upstream tasks (request for opinions)

[Alex Van Boxel]

This is a moved discussion from a GitHub Pull request (as Bolke suggested):

I'll try to copy paste the discussion as best I can:
*PR*: https://github.com/apache/incubator-airflow/pull/1961

*Original commit message:*
This commit fixes the premature finishing of DAGs when using the
branching operator. The fix is making sure a operator in a DAG
is marked as SKIPPED only when all the upstream operators are skipped.

*Remark for Max:*

I thought the desired behavior was to propagate the skipping so that the
DAG run would end and allow the scheduler to move on. Ultimately your DAG
will stop when concurrent number of DAG runs is reached. Whether the
skipping state propagates or not, user action need to be taken for things
to move forward.

I think it's fair to think that people may expect either behavior. People
may have designed existing DAGs based on how this feature behaves. I think
the main reason I'd prefer the current behavior is that is allows the
scheduler to move forward.

In either case the branching section in the concepts part of the docs
should be more clear about using the trigger rule one_success in the
joining task. It should also be clear about how user should expect the
skipped state to propagate. Thinking about it, the trigger rule all_success in
a joining task is not logical. We could almost raise when/if detecting it.


*Reply from Alex:*

My understanding was that skipped == success, because when a skipped
reaches the end the DAG is marked as success (even with half of my tasks
unfinished). And actually this commit 2630361
<https://github.com/apache/incubator-airflow/commit/2630361ca24737c28f458825b20ab11c9c996b17>convinced
my that I had it right. That's why I wrote this PR.

I understand your point because you actually look at it from the schedulers
point of view, I look at it from the users point of view. I already know 2
people in my team that fell into the trap with join nodes. And documenting
it alone isn't enough, I would then be prepared to change this PR into a raise
when we detect this with ALL_SUCCESS trigger.

Still the first DAG I wrote will never work with the only alternative I
have (ONE_SUCCESS), because the branch operator starts a branch that fans
out to a lot of parallel tasks that all join together to the same node
where the skipped branch arrive. This will never work with the ONE_SUCESS
because as soon as one of the parallel branches succeed the rest will just
stop. I know I can hack around this by introducing a DummyOperator where
all successful branches arrive, but I feel dirty if I need to do this.

So I propose this:

   - I'll change this PR so the ALL_SUCCESS raises and error.
   - Start on another PR that introduces 2 new triggers:
   -- ALL_SUCCESS_OR_SKIPPED
   -- ALL_FAILED_OR_SKIPPED

What do you think?



-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 alpha 2

[Alex Van Boxel]

I have another fix that certainly need to be in the final release, but not
ready to merge due to failed tests:

https://github.com/apache/incubator-airflow/pull/1961




On Wed, Jan 4, 2017 at 8:48 PM Chris Riccomini 
wrote:

> Great, I will work toward deploying this in our dev cluster today. :D
>
> On Wed, Jan 4, 2017 at 11:47 AM, Bolke de Bruin  wrote:
>
> > Some issues remain:
> >
> > * one_failed not executed as dag run is marked failed seemingly
> > prematurely (@chris yes you should see this, see below for an example
> that
> > is not working properly), confirmed regression
> > * celery instability Alex
> > * Wrong DAG state after failure inside branch
> >
> > So Alpha 2 is definitely not ready for production, but please do put in
> > your canary dags and let them run. I am still quite concerned about the
> > scheduler integrity and stability.
> >
> > - Bolke
> >
> >
> >
> > one_failed_not_executed.py
> > 
> > from airflow import DAG
> > from airflow.operators.bash_operator import BashOperator
> > from airflow.operators.dummy_operator import DummyOperator
> > from datetime import datetime, timedelta
> >
> > default_args = {
> > 'owner': 'airflow',
> > 'depends_on_past': False,
> > 'start_date': datetime(2016,10,5,19),
> > 'email': ['airf...@airflow.com'],
> > 'email_on_failure': False,
> > 'email_on_retry': False,
> > 'retries': 1,
> > 'retry_delay': timedelta(seconds=1),
> > }
> >
> > dag = DAG('tutorial', default_args=default_args,
> schedule_interval='@once')
> >
> > task1 = BashOperator(
> > task_id='first_one',
> > bash_command='date',
> > dag=dag)
> >
> > task2 = BashOperator(
> > task_id='second_one',
> > bash_command='this_should_not_work',
> > dag=dag)
> >
> > task2.set_upstream(task1)
> >
> >
> >
> > task3 = BashOperator(
> > task_id='third_one',
> > bash_command='random_command_third',
> > dag=dag)
> >
> > task3.set_upstream(task2)
> >
> > fail_task = DummyOperator(
> > task_id='one_failed',
> > trigger_rule='one_failed',
> > dag=dag)
> >
> > fail_task.set_upstream([task1,task2,task3])
> >
> > > On 4 Jan 2017, at 20:35, Chris Riccomini 
> wrote:
> > >
> > > Bolke, can you describe the current state of the alpha 2 release? I saw
> > > some comments from Alex yesterday about celery instability. If I'm
> > running
> > > on LocalExecutor, should I be seeing any issues?
> > >
> > > On Wed, Jan 4, 2017 at 8:20 AM, Bolke de Bruin 
> > wrote:
> > >
> > >> Hi All,
> > >>
> > >> I have put up Airflow 1.8.0 alpha 2 in https://people.apache.org/~
> > bolke/ <
> > >> https://people.apache.org/~bolke/> .
> > >>
> > >> Note: This still cannot be considered an Apache release. Working on
> > this.
> > >>
> > >> This build is signed (note it is served over https).
> > >>
> > >> Changes are in the area of scheduler stability.
> > >>
> > >> - Bolke
> >
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Wrong DAG state after failure inside a branch

[Alex Van Boxel]

Hey ypwdr,

I got a pull request that will fix this problem. It works for me but I need
to fix the failing branch. Be my guest to try it out and see if it resolves
your problem.


On Wed, Jan 4, 2017 at 4:14 PM  wrote:

> Hello
>
> I just became aware that failure inside a branch (BranchPythonOperator) set
> the DAG state to success despites the task failure.
> Indeed Airflow set all downstream tasks to the skipped state because of the
> trigger_rule needed for the branching mecanism and as a consequence Airflow
> wrongly set the DAG state to success. Please find below a example DAG to
> trigger this behaviour.
> Is there any way to make tasks run more reliable inside a branch ?
>
>
>  from airflow.models import DAG
>  from airflow.operators.python_operator import PythonOperator,
> BranchPythonOperator
>  from datetime import datetime, timedelta
>
> DAG_NAME = 'branch_issue'
>
> dag = DAG(DAG_NAME + '_v1', start_date=datetime(2017, 1, 4, 9, 10),
> schedule_interval=timedelta(hours=1))
>
>
> branch1 = BranchPythonOperator(
>  task_id='branch1',
>  python_callable=lambda: 't1',
>  dag=dag)
>
> t1 = PythonOperator(
>  task_id='t1',
>  python_callable=lambda: sys.exit(1),
>  dag=dag)
> t1.set_upstream(branch1)
>
> t2 = PythonOperator(
>  task_id='t2',
>  python_callable=lambda: True,
>  dag=dag)
> t2.set_upstream(branch1)
>
> process1 = PythonOperator(
>  task_id='process1',
>  python_callable=lambda: True,
>  trigger_rule='one_success',
>  dag=dag)
> process1.set_upstream(t1)
> process1.set_upstream(t2)
>
> process2 = PythonOperator(
>  task_id='process2',
>  python_callable=lambda: True,
>  dag=dag)
> process2.set_upstream(process1)
>
> process3 = PythonOperator(
>  task_id='process3',
>  python_callable=lambda: True,
>  dag=dag)
> process3.set_upstream(process2)
>
>
> At moment, I want my privacy to be protected.
> https://mytemp.email/
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 Alpha 1

[Alex Van Boxel]

(we still test
> > them but we can't run them in production), that being said I don't think
> I
> > authored the commits you are referring to so I don't have full context.
> >
> > On Tue, Jan 3, 2017 at 1:27 PM, Bolke de Bruin 
> wrote:
> >
> > > Hi Dan et al,
> > >
> > > That sounds good to me, however I will be pretty critical of the
> changes
> > > in the scheduler and the cleanliness of the patches. This is due to the
> > > fact I have been chasing quite some bugs in master that were pretty
> hard
> > to
> > > track down even with a debugger at hand. I’m surprised that those
> didn’t
> > > pop up in your production or maybe I am concerned ;-). Anyways, I hope
> > you
> > > understand I might be a bit picky in understanding and needing (design)
> > > documentation for some of the changes.
> > >
> > > What I would like to suggest is that for the Alpha versions we still
> > > accept “new” features so these PRs can get in, but from Beta we will
> not
> > > accept new features anymore. For new features in the area of the
> > scheduler
> > > an integration DummyDag should be supplied, so others can test the
> > > behaviour. Does this sound ok?
> > >
> > > My list of open code items for a release looks now like this:
> > >
> > > Blockers
> > > * one_failed not honoured
> > > * Alex’s sensor issue
> > >
> > > New features:
> > > * Schedule all pending DAGs in a single loop
> > > * Add support for backfill true/false
> > > * Impersonation
> > > * CGroups
> > > * Add Cloud Storage updated sensor
> > >
> > > Alpha2 I will package tomorrow. Packages are signed now by my
> apache.org
> > <
> > > http://apache.org/> key. Please verify and let me know if something is
> > > off. I’m still waiting for access to the incubating dist repository.
> > >
> > > Bolke
> > >
> > >
> > > > On 3 Jan 2017, at 14:38, Dan Davydov  .INVALID>
> > > wrote:
> > > >
> > > > I have also started on this effort, recently Alex Guziel and I have
> > been
> > > > pushing Airbnb's custom cherries onto master to get Airbnb back onto
> > > master
> > > > in order for us to do a release.
> > > >
> > > > I think it might make sense to wait for these two commits to get
> merged
> > > in
> > > > since they would be quite nice to have for all Airflow users and seem
> > > like
> > > > they will be merged soon:
> > > > Schedule all pending DAG runs in a single scheduler loop -
> > > > https://github.com/apache/incubator-airflow/pull/1906 <
> > > https://github.com/apache/incubator-airflow/pull/1906>
> > > > Add Support for dag.backfill=(True|False) Option -
> > > > https://github.com/apache/incubator-airflow/pull/1830 <
> > > https://github.com/apache/incubator-airflow/pull/1830>
> > > > Impersonation Support + Cgroups - https://github.com/apache/ <
> > > https://github.com/apache/>
> > > > incubator-airflow/pull/1934 (this is kind of important from the
> Airbnb
> > > side
> > > > so that we can help test the new master without having to cherrypick
> > this
> > > > PR on top of it which would make the testing unreliable for others).
> > > >
> > > > If there are PRs that affect the core of Airflow that other
> committers
> > > > think are important to merge we could include these too. I can commit
> > to
> > > > pushing out the Impersonation/Cgroups PR this week pending PR
> comments.
> > > > What do you think Bolke?
> > > >
> > > > On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  > > <mailto:bdbr...@gmail.com>> wrote:
> > > >
> > > >> Hey Alex,
> > > >>
> > > >> I have noticed the same, and it is also the reason why we have Alpha
> > > >> versions. For now I have noticed the following:
> > > >>
> > > >> * Tasks can get in limbo between scheduler and executor:
> > > >> https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948> <
> > > >> https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948>>
> > > >> * Try_number not inc

Re: Airflow 1.8.0 Alpha 1

[Alex Van Boxel]

If they should make the first alpha, maybe they should be rebased so they
can be merged in.

On Tue, Jan 3, 2017 at 2:39 PM Dan Davydov 
wrote:

> I have also started on this effort, recently Alex Guziel and I have been
> pushing Airbnb's custom cherries onto master to get Airbnb back onto master
> in order for us to do a release.
>
> I think it might make sense to wait for these two commits to get merged in
> since they would be quite nice to have for all Airflow users and seem like
> they will be merged soon:
> Schedule all pending DAG runs in a single scheduler loop -
> https://github.com/apache/incubator-airflow/pull/1906
> Add Support for dag.backfill=(True|False) Option -
> https://github.com/apache/incubator-airflow/pull/1830
> Impersonation Support + Cgroups - https://github.com/apache/
> incubator-airflow/pull/1934 (this is kind of important from the Airbnb side
> so that we can help test the new master without having to cherrypick this
> PR on top of it which would make the testing unreliable for others).
>
> If there are PRs that affect the core of Airflow that other committers
> think are important to merge we could include these too. I can commit to
> pushing out the Impersonation/Cgroups PR this week pending PR comments.
> What do you think Bolke?
>
> On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  wrote:
>
> > Hey Alex,
> >
> > I have noticed the same, and it is also the reason why we have Alpha
> > versions. For now I have noticed the following:
> >
> > * Tasks can get in limbo between scheduler and executor:
> > https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948>
> > * Try_number not increased due to reset in LocalTaskJob:
> > https://github.com/apache/incubator-airflow/pull/1969 <
> > https://github.com/apache/incubator-airflow/pull/1969>
> > * one_failed trigger not executed
> >
> > My idea is to move to a Samba style of releases eventually, but for now I
> > would like to get master into a state that we understand and therefore
> not
> > accept any patches that do not address any bugs.
> >
> > If you (or anyone else) can review the above PRs and add your own as well
> > then I can create another Alpha version. I’ll be on gitter as much as I
> can
> > so we can speed up if needed.
> >
> > - Bolke
> >
> > > On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
> > >
> > > Hey Bolke,
> > >
> > > thanks for getting this moving. But I already have some blockers,
> since I
> > > moved up master to this release (moved from end November to now)
> > stability
> > > has gone down (certainly on Celary). I'm trying to identify the core
> > > problems and see if I can fix them.
> > >
> > > On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin  > <mailto:bdbr...@gmail.com>> wrote:
> > >
> > > Dear All,
> > >
> > > On the verge of the New Year, I decided to be a little bit cheeky and
> to
> > > make available an Airflow 1.8.0 Alpha 1. We have been talking about it
> > for
> > > a long time now and by doing this I wanted bootstrap the process. It
> > should
> > > by no means be considered an Apache release yet. This is for testing
> > > purposes in the dev community around Airflow, nothing else.
> > >
> > > The build is exactly the same as the state of master (git 410736d) plus
> > the
> > > change to version “1.8.0.alpha1” in version.py.
> > >
> > > I am dedicating quite some time next week and beyond to get a release
> > out.
> > > Hopefully we can get some help with testing, changelog etc. To make
> this
> > > possible I would like to propose a freeze to adding new features for at
> > > least two weeks - say until Jan 15.
> > >
> > > You can find the tar here: http://people.apache.org/~bolke/ <
> > > http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/>> .
> > It isn’t signed. Following versions
> > > will be. SHA is available.
> > >
> > > Lastly, Alpha 1 does not have the fix for retries yet. So we will get
> an
> > > Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> > > https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948> <
> > > https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948>> , but your
> > feedback
> > > is required as it is entrenched in new processing code that you are
> > running
> > > in production afaik - so I wonder what happens in your fork.
> > >
> > > Happy New Year!
> > >
> > > Bolke
> > >
> > >
> > >
> > > --
> > >  _/
> > > _/ Alex Van Boxel
> >
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow 1.8.0 Alpha 1

[Alex Van Boxel]

Hey Bolke,

thanks for getting this moving. But I already have some blockers, since I
moved up master to this release (moved from end November to now) stability
has gone down (certainly on Celary). I'm trying to identify the core
problems and see if I can fix them.

On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin  wrote:

Dear All,

On the verge of the New Year, I decided to be a little bit cheeky and to
make available an Airflow 1.8.0 Alpha 1. We have been talking about it for
a long time now and by doing this I wanted bootstrap the process. It should
by no means be considered an Apache release yet. This is for testing
purposes in the dev community around Airflow, nothing else.

The build is exactly the same as the state of master (git 410736d) plus the
change to version “1.8.0.alpha1” in version.py.

I am dedicating quite some time next week and beyond to get a release out.
Hopefully we can get some help with testing, changelog etc. To make this
possible I would like to propose a freeze to adding new features for at
least two weeks - say until Jan 15.

You can find the tar here: http://people.apache.org/~bolke/ <
http://people.apache.org/~bolke/> . It isn’t signed. Following versions
will be. SHA is available.

Lastly, Alpha 1 does not have the fix for retries yet. So we will get an
Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
https://github.com/apache/incubator-airflow/pull/1948 <
https://github.com/apache/incubator-airflow/pull/1948> , but your feedback
is required as it is entrenched in new processing code that you are running
in production afaik - so I wonder what happens in your fork.

Happy New Year!

Bolke



-- 
  _/
_/ Alex Van Boxel

Re: Scheduler error - cannot allocate memory

[Alex Van Boxel]

Hey Teresa,

I'm running Airflow on Kubernetes just to make sure we have process
isolation. Even if one of the processes crashes (like with out-of-memory),
kubernetes just restarts it. Maybe something to consider.

On Tue, Dec 27, 2016 at 12:36 PM Teresa Fontanella De Santis <
tfontanella...@gmail.com> wrote:

> Bolke,
>
> Thanks for the answer!
>
> You are right about the process that takes up all remaining. But sometimes
> it gets out of memory and sometimes not.
>
> We are running Airflow (both scheduler, the webserver and execute the dags)
> on an EC2 instance m4.xlarge (we have 4 workers), using LocalExecutor. In
> this case, should be better to use CeleryExecutor?
>
> Thanks again!
>
> 2016-12-26 18:41 GMT-03:00 Bolke de Bruin :
>
> > We dont handle this kind of errors in airflow, so it becomes a hard error
> > and airflow bails out.
> >
> > You are running out of memory most likely as some other process is taking
> > up all remaining. Are you running workers on the same machine? These will
> > go up and down with mem usage over time depending the jobs you launch.
> >
> > This is not related to "restarting the scheduler" (which is kind of
> > outdated anyway).
> >
> > Bolke
> >
> > Sent from my iPhone
> >
> > > On 26 Dec 2016, at 21:47, Teresa Fontanella De Santis <
> > tfontanella...@gmail.com> wrote:
> > >
> > > Hi everyone!
> > >
> > > We were running the scheduler without problems for a while. We are
> using
> > > ec2 instance (mx4.large). We were running with airflow scheduler (no
> > > supervisor.d, no monit, etc).
> > > Suddenly, the scheduler stopped, showing this message:
> > >
> > > [2016-12-22 21:01:15,038] {jobs.py:574} INFO - Prioritizing 1 queued
> > > jobs
> > > [742/1767]
> > > [2016-12-22 21:01:15,041] {jobs.py:603} INFO - Pool None has 128
> slots, 1
> > > task instances in
> > > queue
> > >
> > > [2016-12-22 21:01:15,041] {models.py:154} INFO - Filling up the DagBag
> > from
> > > /home/ec2-user/analytics/airflow/dags
> > >
> > > [2016-12-22 21:01:15,155] {jobs.py:726} INFO - Starting 2 scheduler
> > > jobs
> > >
> > > [2016-12-22 21:01:15,157] {jobs.py:761} ERROR - [Errno 12] Cannot
> > allocate
> > > memory
> > >
> > > Traceback (most recent call
> > > last):
> > >
> > >  File "/usr/local/lib/python3.5/site-packages/airflow/jobs.py", line
> > 728,
> > > in
> > > _execute
> > >
> > >
> > > j.start()
> > >
> > >  File "/usr/lib64/python3.5/multiprocessing/process.py", line 105, in
> > > start
> > >
> > >self._popen =
> > > self._Popen(self)
> > >
> > >  File "/usr/lib64/python3.5/multiprocessing/context.py", line 212, in
> > > _Popen
> > >
> > >return
> > > _default_context.get_context().Process._Popen(process_obj)
> > >
> > >  File "/usr/lib64/python3.5/multiprocessing/context.py", line 267, in
> > > _Popen
> > >
> > >return
> > > Popen(process_obj)
> > >
> > >  File "/usr/lib64/python3.5/multiprocessing/popen_fork.py", line 20, in
> > > __init__
> > >
> > >
> > > self._launch(process_obj)
> > >
> > >  File "/usr/lib64/python3.5/multiprocessing/popen_fork.py", line 67, in
> > > _launch
> > >
> > >self.pid =
> > > os.fork()
> > >
> > > OSError: [Errno 12] Cannot allocate
> > > memory
> > >
> > > Traceback (most recent call last):
> > >  File "/usr/local/bin/airflow", line 15, in 
> > >
> > >
> > > The dags which failed didn't show any log (there weren't stored on
> > airflow
> > > instance and there is no remote logs). So we don't have any idea of
> what
> > > would happened (only that there was not enough memory to fork)
> > > It is well known that is recommended to restart the scheduler
> > periodically
> > > (according to this
> > > <https://medium.com/handy-tech/airflow-tips-tricks-and-
> > pitfalls-9ba53fba14eb#.80c6g1n1s>),
> > > but... do you have any idea why this can happen? Is there something we
> > can
> > > do (or some bug we can fix)?
> > >
> > >
> > > Thanks in advance!
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Integration test env

[Alex Van Boxel]

I finally have some time to spend on my series of blog articles based on
the Google Cloud operator in Airflow. I started the GitHub repo with the
examples: https://github.com/alexvanboxel/airflow-gcp-examples

The main setup is that they should be *smokes tests* that would be able to
run *once a day *(because they are partitioned per day). Would the an idea
for the integration tests (and is once per day enough?)

On Mon, Dec 19, 2016 at 10:26 PM Chris Riccomini 
wrote:

> Hey Bolke,
>
> Thanks for stepping up. The "traditional" Apache infra is Jenkins with a
> pool of machines that they provide. That might or might not be satisfactory
> for us (it's certainly antiquated technology). If we decide we don't like
> it, I think that's OK, as long as we move forward knowing that any other
> testing solution can disappear at any time.
>
> My 2c. :)
>
> Cheers,
> Chris
>
> On Wed, Dec 14, 2016 at 11:11 AM, Dan Davydov <
> dan.davy...@airbnb.com.invalid> wrote:
>
> > This is extremely generous of you! I do agree with the approach of trying
> > to get funding from Apache and having shared resources (e.g. so that we
> > don't depend on any one company or individual for the uptime of the
> > integration environment, plus so we would have public cloud integration
> > potentially).
> >
> > On Wed, Dec 14, 2016 at 1:22 AM, Bolke de Bruin 
> wrote:
> >
> > > Hi,
> > >
> > > I have been thinking about an integration test environment. Aside from
> > any
> > > technical requirements we need a place to do it. I am willing to offer
> a
> > > place in Lab env I am running or to fund an environment in AWS/GCloud
> if
> > > Apache cannot make these kind of resources available.
> > >
> > > If running in our Lab there is virtually no restriction what we could
> do,
> > > however I will hand select people who have access to this environment.
> I
> > > will also hold ultimate power to remove access from anyone. I even
> might
> > > ask for a confirmation that you will behave when using our property
> > (don’t
> > > worry won’t cover it with legal wording). This is a IAAS service so we
> > need
> > > to cover the things we need ourselves, but the upside is we can and it
> is
> > > free. We could setup a Gitlab instance that mirrors from Apache a kicks
> > off
> > > runners to do testing. Downside 1) it might not be entirely Apache
> like.
> > > Sorry cant help that. 2) there is no guaranteed up time 3) I might need
> > to
> > > remove it in the future e.g. when I change jobs for example :). 4) No
> > > public cloud integration, it’s a private stack after all.
> > >
> > > I can also fund on AWS/GCloud. Again, I probably want to have ultimate
> > > power on access to this environment - it’s my company’s money on the
> line
> > > after all. Major downside to this is that it is dependent on and
> limited
> > by
> > > the budget I can make available. Upside is that it is not company
> > property.
> > > Also I personally have less exposure to public cloud environments due
> to
> > > company restrictions.
> > >
> > > Are there any other options? Any thoughts?
> > >
> > > Bolke
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow-GCP for Google Container Engine

[Alex Van Boxel]

Hey Koen, I tried running it with 2 but one is for my use-case good enough.
I don't exactly know what you "in-this-context" mean by managing resources,
but on GCloud you want to delegate as much as possible to the services
(DataProc/DataFlow/BigQuery) and soon also other docker containers in the
same cluster, so I like to keep the workers as light as possible. But good
point about the max (I should make it configurable).

My cluster for now is 2x4CPU for my Kubernetes cluster, but it has other
stuff running aside from Airflow.

On Mon, Dec 5, 2016 at 12:42 PM Koen Mevissen  wrote:

> Nice one, thanks! I tried this a while ago, and got stuck on resource
> issues - probably because I capped the pods max resources to hard.
>
> I see in the worker yaml it creates 1 worker replica, are you running it
> with multiple workers as well? Are you managing any resources on the pods,
> or you let the workers consume whatever's available on the nodes?
>
>
> On Fri, Dec 2, 2016 at 10:14 PM, Chris Riccomini 
> wrote:
>
> > Nice, thanks! :)
> >
> > On Fri, Dec 2, 2016 at 4:55 AM, Alex Van Boxel  wrote:
> >
> > > Hi all,
> > >
> > > I think I pulled it off to have kind of "one-execute" install of
> Airflow
> > on
> > > Google Container Engine. Personally I find this my preferred setup
> > (because
> > > I can have a production environment and staging environment). It's
> what I
> > > use for our production/staging setup. I think someone else could pull
> it
> > > off with everything in this repo:
> > >
> > > https://github.com/alexvanboxel/airflow-gcp-k8s
> > >
> > > It has a README that should be enough to get it setup.
> > >
> > > Now I'll work on my sync'er so my dags will refresh as soon as I push
> to
> > > git :-)
> > >
> >
>
>
>
> --
> Kind regards,
> Met vriendelijke groet,
>
> *Koen Mevissen*
> Principal BI Developer
>
>
> *Travix Nederland B.V.*
> Piet Heinkade 55
> 1019 GM Amsterdam
> The Netherlands
>
> T. +31 (0)20 203 3241 <+31%2020%20203%203241>
> E: kmevis...@travix.com
> www.travix.com
>
> *Brands: * CheapTickets  |  Vliegwinkel  |  Vayama  |  BudgetAir  |
>  Flugladen
>

Airflow-GCP for Google Container Engine

[Alex Van Boxel]

Hi all,

I think I pulled it off to have kind of "one-execute" install of Airflow on
Google Container Engine. Personally I find this my preferred setup (because
I can have a production environment and staging environment). It's what I
use for our production/staging setup. I think someone else could pull it
off with everything in this repo:

https://github.com/alexvanboxel/airflow-gcp-k8s

It has a README that should be enough to get it setup.

Now I'll work on my sync'er so my dags will refresh as soon as I push to
git :-)

Re: Merging the experimental API Framework

[Alex Van Boxel]

Although I haven't had the time to dive deep into API (sorry Bolke) I do
want to be part of the discussion. I hope to have a look at it soon.

On Tue, Nov 29, 2016 at 10:43 AM Bolke de Bruin  wrote:

> Flask App Builder looks great at a first glance and experience counts
> obviously, though I have several concerns:
>
> Authentication
> - The Hadoop ecosystem, especially the on premise installs, is dependent
> for its security integration on Kerberos. FAB out of the box does not
> support this.
> - In addition I would like to implement something along the lines of the
> Hadoop delegation token. Ie. the client first authenticates with Kerberos
> and then gets a token for further communication. This reduces stress on the
> KDC and allows us to let the tasks get their connection details from the
> API instead of direct access to the database. Imagine a (celery) worker
> getting a token on behalf of the task, the tasks uses this to communicate
> with the API to get the connection details it needs. The token has a
> limited lifetime and automatically expires on finishing a task. This
> requires a custom authentication layer in FAB.
>
> Non JSON returns for service-to-service communication (or in general)
> - The second point above is a non public API and will be used extensively,
> thus JSON as a return type is not efficient from several perspectives: 1)
> message size 2) schema. Avro or Protobuf seem to fit the bill much better.
> From a first look I couldn’t figure out if FAB can do this easily.
>
> Going a bit off-topic I imagine Airflow in separate packages which are
> loosely coupled (as I guess you described in the Airflow 2.0 thread):
>
> airflow-client
> Brings you the CLI and client libraries that allow you to integrate
> Airflow with your own programs. You can run the client from anywhere
> because it will look up its endpoint by DNS (eg. _airflow._tcp.example.com.
> 86400 IN SRV 0 5 443 airflow.example.com).
>
> airflow-api
> API server for service to service and endpoints. Checks its registration
> in DNS.
>
> airflow-scheduler
> Scheduler only (no LocalExecutor Workers)
>
> airflow-worker
> Celery / Local / Mesos
>
> Absent: airflow webserver, serving files can be handled by apache/nginx.
>
> So yes lets talk :-).
>
> Bolke
>
>
> > Op 29 nov. 2016, om 01:14 heeft Maxime Beauchemin <
> maximebeauche...@gmail.com> het volgende geschreven:
> >
> > Glad to see this! On my side I've been playing around trying to use Flask
> > App Builder (just obtained committer status on the project) which covers
> > some out of the box CRUD REST API for models and a
> > authentication/role/permission framework that would allow for a
> > multi-tenant UI / API with arbitrary roles / granular access.
> >
> > We should chat as to how these solutions might work together. My early
> work
> > and some of the rational for it can be found here:
> > https://github.com/mistercrunch/airflow_webserver
> >
> > We should chat! Let's schedule time.
> >
> > Max
> >
> > On Mon, Nov 28, 2016 at 11:36 AM, Andrew Phillips 
> > wrote:
> >
> >> Just wanted to say this is very exciting, thank you Bolke :).
> >>>
> >>
> >> Big +1 to that. Thanks, Bolke!
> >>
> >> ap
> >>
>
>

Crazy Airflow DAG ideas

[Alex Van Boxel]

I think that I just made quite a crazy DAG. I'm wondering if people can top
this one. But, it's not about complexity is about the strangeness.

*So, who can top this? (I'm quite interested what sort of crazy DAG's are
out in the wild).*

First context: we have an on premises SQL Server (and I want to move
everything to the cloud). So what better to do to have a daily snapshot in
the cloud).

This dag does:
- Wait for the full backup to appear in Google Cloud Storage (the new
sensor)
- It spins up SQL Server on Google Cloud through Deployment manager and
does the following (though a PowerShell script):
  * Import the backup from GCS
  * Import the file in SQL Server
  * Detach the database
  * Detach the disk
  * Take a Snapshot of the disk (db-MMDD-full)
  * Mount again
  * Minify
  * Detach, Snapshot again (db-MMDD-mini), Attach
Downstream we have 3 sensors (the deployment operator doesn't wait till the
script is finished, just that the machines are ready):
* (A) Waiting for snapshot: db-MMDD-full
* (B) Waiting for snapshot: db-MMDD-mini
* (C) Looking at the serial port of the machine to look when it's finished

So in each stream:
(A) full_snapshot_sensor
 - Creates a new SQL Server from the full snapshot, this will be a
reporting db (takes 3 minutes from nothing, quite cool)
 - Change DNS entry
 - Delete old deprecated versions of reporting
(C) if script is finished
 - Delete the original SQL server for importing the database

[image: Screen Shot 2016-11-04 at 09.47.36.png]

Re: Next Release?

[Alex Van Boxel]

I thought that the 15 November deadline for PR was in preparation for the
1.8 release. Do you need help with the release? I'm dedicating each week
some time on Airflow anyway (although it's more writing operators :-).

On Thu, Oct 27, 2016 at 6:22 PM siddharth anand  wrote:

> I believe the release manager is/was Max, though it might be Dan (@aoen).
>
> -s
>
> On Thu, Oct 27, 2016 at 9:20 AM, Chris Riccomini  wrote:
>
> > No news AFAIK. I think
> >
> > 1) someone needs to be release mgr
> > 2) release mgr needs to cut RC
> > 3) we all need to deploy RC
> >
> > On Thu, Oct 27, 2016 at 1:19 AM, siddharth anand 
> > wrote:
> >
> > > Max, Chris, Bolke?
> > >
> > > Any news on the 1.8 release?
> > > -s
> > >
> >
>

Re: Airflow Logging

[Alex Van Boxel]

My requirement would indeed be that I would be able to add my own logging
handler (I did the same in my Luigi days), I included a python log handler
that logged to Google Cloud Logging.

But I also like the current logging to Cloud storage.

So my ideal logging setup would be:
All of Airflow (scheduler + webserver + worker) -> Log to custom log handler
Output of DAG -> Log to customer log handler + Cloud *storage* for easy
access in Airflow

It's currently a nice-to-have.




On Tue, Oct 18, 2016 at 6:13 PM Maycock, Luke <
luke.mayc...@affiliate.oliverwyman.com> wrote:

> Thank you for the response Maxime. We will consider the points you made
> when making the changes and may come back with further thoughts as we work
> on it.
>
>
> I can't say I am too familiar with Kibana, Logstash and Elasticsearch but
> I am assuming Airflow would still produce logs and you would be using
> Logstash to process the logs?
>
>
> Thanks,
> Luke Maycock
> OLIVER WYMAN
> luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com>
> www.oliverwyman.com
>
>
>
> 
> From: Maxime Beauchemin 
> Sent: 14 October 2016 17:02
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow Logging
>
> Another consideration is for configuration as code in `settings.py`
> in-place or in conjunction with `airflow.cfg` to allow more dynamic
> configuration, like passing your own custom log handler[s]. Many folks at
> the meetup (including Airbnb) spoke about their intent to ship the logging
> to Kibana and/or other logging backend.
>
> Side note about `airflow.cfg` vs `settings.py`: it would be great to move
> completely away from the cfgs for Airflow 2.0. It turns out that Python's
> ConfigParser may be one of the worst module in the standard library and the
> cfg not-so-standard standard isn't great. We have many instances where we
> allow for people to pass objects or function as configuration parameters
> anyways, so let's keep the whole configuration in one place!
>
> Max
>
> On Fri, Oct 14, 2016 at 7:43 AM, Maycock, Luke <
> luke.mayc...@affiliate.oliverwyman.com> wrote:
>
> > Carrying on from the below. We believe it would be useful to have a
> number
> > of additional configuration options for logs as below. Do you think there
> > are any important options we have missed or do you have any other
> feedback?
> > If so, please let us know.
> >
> >
> > Global
> >
> > CLEAR_INHERITED_LOGGING_SETTINGS (Boolean)
> > Clears any handlers, inherited from the root logger, from the Airflow
> > logger object and its children.
> >
> > LOG_TO_DEBUG_FILE (Boolean)
> >
> > Turn on/off logging of all events to a file.
> >
> > LOG_TO_ERROR_FILE (Boolean)
> > Turn on/off the logging of all errors to a file.
> >
> > LOG_TO_CONSOLE (Boolean)
> > Turn on/off the logging of all events to the console.
> >
> > Per log file
> > The Python logging library, being used for logging in the Airflow code,
> > has the ability to rotate log files based on time:
> >
> >
> > • TimedRotatingFileHandler - (https://docs.python.org/2.6/
> > library/logging.html#timedrotatingfilehandler) - rotates based on the
> > product of when and interval (see below).
> >
> > The TimedRotatingFileHandler takes the following arguments:
> >
> >  *   Filename – the filename
> >  *   When – used to specify the type of interval
> >  *   Interval – defines the interval
> >  *   BackupCount - If backupCount is nonzero, at most backupCount files
> > will be kept, and if more would be created when rollover occurs, the
> oldest
> > one is deleted. The deletion logic uses the interval to determine which
> > files to delete, so changing the interval may leave old files lying
> around.
> >  *   Encoding - if encoding is not ‘None’, it is used to open the file
> > with that encoding.
> >  *   Delay - if delay is true, then file opening is deferred until the
> > first call to emit() – I don’t think this is suitable to expose as an
> > Airflow configuration option.
> >  *   UTC - if the UTC argument is true, times in UTC will be used;
> > otherwise local time is used.
> > I believe all of the above, except ‘delay’, should be exposed as Airflow
> > configuration.
> >
> >
> > Cheers,
> > Luke Maycock
> > OLIVER WYMAN
> > luke.mayc...@affiliate.oliverwyman.com > mayc...@affiliate.oliverwyman.com>
> > www.oliverwyman.com
> >
> >
> >
> > 
> > From: Maycock, Luke
> > Sent: 13 October 2016 14:52
> > To: dev@airflow.incubator.apache.org
> > Subject: Airflow Logging
> >
> >
> > Hi All,
> >
> > We (owlabs - fork: https://github.com/owlabs/incubator-airflow) have a
> > high level design for how to improve the logging throughout the Airflow
> > code to be more consistent, maintainable and extensible. We'd really
> > appreciate any feedback on the design.
> >
> > Design for Consolidating Logging Setup:
> >
> >  *   Create a Class in the utils\logging.py t

Re: Cloud Provider grouping into Plugins

[Alex Van Boxel]

Talking about AWS, it would only make sense if other people would step up
to do it for AWS, and even Azure (or don't we have Azure operators?).

On Fri, Oct 14, 2016 at 5:25 PM Chris Riccomini 
wrote:

> What do others think? I know Sid is a big AWS user.
>
> On Fri, Oct 14, 2016 at 8:24 AM, Chris Riccomini 
> wrote:
> > Ya, if we go the deprecation route, and let them float around for a
> > release or two, I'm OK with that (or until we bump major to 2.0).
> >
> > Other than that, it sounds like a good opportunity to clean things up.
> > :) I do notice a lot of AWS/GCP code (e.g. the S3 Redshift operator).
> >
> > On Fri, Oct 14, 2016 at 8:16 AM, Alex Van Boxel 
> wrote:
> >> Well, I wouldn't touch the on that exist (maybe we could mark them
> >> deprecated, but that's all). But I would move (copy) them together and
> make
> >> them consistent (example, let them all use the same default
> connection_id,
> >> ...). For a new user it's quite confusing I think due to different
> reasons
> >> (style, etc...) you know we have an old ticket: making gcp consistent (I
> >> just don't want to start on this on, on fear of breaking something).
> >>
> >> On Fri, Oct 14, 2016 at 4:59 PM Chris Riccomini 
> >> wrote:
> >>
> >> Hmm. What advantages would this provide? I'm a little nervous about
> >> breaking compatibility. We have a bunch of DAGs which import all kinds
> >> of GCP hooks and operators. Wouldn't want those to move.
> >>
> >> On Fri, Oct 14, 2016 at 7:54 AM, Alex Van Boxel 
> wrote:
> >>> Hi all,
> >>>
> >>> I'm starting to write some very exotic Operators that are a bit strange
> >>> adding to contrib. Examples of this are:
> >>>
> >>> + See if a Compute snapshot of a disc is created
> >>> + See if a string appears on the serial port of Compute instance
> >>>
> >>> but they would be a nice addition if we had a Google Compute plugin (or
> >> any
> >>> other cloud provider, AWS, Azure, ...). I'm not talking about getting
> >> cloud
> >>> support out of the main source tree. No, I'm talking about grouping
> them
> >>> together in a consistent part. We can even start adding macro's etc.
> This
> >>> would be a good opportunity to move all the GCP operators together,
> making
> >>> them consistent without braking the existing operators that exist in
> >>> *contrib*.
> >>>
> >>> Here are a few requirements that I think of:
> >>>
> >>>- separate folder ( example  /integration/googlecloud ,
> >>> /integration/aws
> >>>,  /integration/azure )
> >>>- enable in config (don't want to load integrations I don't use)
> >>>- based on Plugin (same interface)
> >>>
> >>> Thoughts?
>

Re: Cloud Provider grouping into Plugins

[Alex Van Boxel]

Well, I wouldn't touch the on that exist (maybe we could mark them
deprecated, but that's all). But I would move (copy) them together and make
them consistent (example, let them all use the same default connection_id,
...). For a new user it's quite confusing I think due to different reasons
(style, etc...) you know we have an old ticket: making gcp consistent (I
just don't want to start on this on, on fear of breaking something).

On Fri, Oct 14, 2016 at 4:59 PM Chris Riccomini 
wrote:

Hmm. What advantages would this provide? I'm a little nervous about
breaking compatibility. We have a bunch of DAGs which import all kinds
of GCP hooks and operators. Wouldn't want those to move.

On Fri, Oct 14, 2016 at 7:54 AM, Alex Van Boxel  wrote:
> Hi all,
>
> I'm starting to write some very exotic Operators that are a bit strange
> adding to contrib. Examples of this are:
>
> + See if a Compute snapshot of a disc is created
> + See if a string appears on the serial port of Compute instance
>
> but they would be a nice addition if we had a Google Compute plugin (or
any
> other cloud provider, AWS, Azure, ...). I'm not talking about getting
cloud
> support out of the main source tree. No, I'm talking about grouping them
> together in a consistent part. We can even start adding macro's etc. This
> would be a good opportunity to move all the GCP operators together, making
> them consistent without braking the existing operators that exist in
> *contrib*.
>
> Here are a few requirements that I think of:
>
>- separate folder ( example  /integration/googlecloud ,
> /integration/aws
>,  /integration/azure )
>- enable in config (don't want to load integrations I don't use)
>- based on Plugin (same interface)
>
> Thoughts?

Cloud Provider grouping into Plugins

[Alex Van Boxel]

Hi all,

I'm starting to write some very exotic Operators that are a bit strange
adding to contrib. Examples of this are:

+ See if a Compute snapshot of a disc is created
+ See if a string appears on the serial port of Compute instance

but they would be a nice addition if we had a Google Compute plugin (or any
other cloud provider, AWS, Azure, ...). I'm not talking about getting cloud
support out of the main source tree. No, I'm talking about grouping them
together in a consistent part. We can even start adding macro's etc. This
would be a good opportunity to move all the GCP operators together, making
them consistent without braking the existing operators that exist in
*contrib*.

Here are a few requirements that I think of:

   - separate folder ( example  /integration/googlecloud ,
/integration/aws
   ,  /integration/azure )
   - enable in config (don't want to load integrations I don't use)
   - based on Plugin (same interface)

Thoughts?

Re: Next Airflow meet-up

[Alex Van Boxel]

@bolke: it's the "Google Expert summit". It's a closed conference I'm
afraid. Will work at my abstract this evening.

On Wed, Oct 12, 2016 at 2:56 AM siddharth anand  wrote:

> Wonderful!
>
> We have 3 speakers already. Paul, Alex, can you send a short talk summary
> to Chris as soon as you can.
>
> -s
>
> On Tue, Oct 11, 2016 at 5:29 PM, Gurer Kiratli <
> gurer.kira...@airbnb.com.invalid> wrote:
>
> > just talked with Paul Yang. He would like to have a 15 minute
> > presentation/talk.
> >
> > Cheers,
> >
> > Gurer
> >
> > On Tue, Oct 11, 2016 at 2:44 PM, Chris Riccomini 
> > wrote:
> >
> > > > Chris, I'm assuming that you will have one speaker from WePay and
> Alex,
> > > correct? So, you are only looking for one additional speaker?
> > >
> > > Correct.
> > >
> > > On Tue, Oct 11, 2016 at 2:36 PM, siddharth anand 
> > > wrote:
> > > > Chris, I'm assuming that you will have one speaker from WePay and
> Alex,
> > > > correct? So, you are only looking for one additional speaker?
> > > >
> > > > Bolke,
> > > > QCon SF is Nov 7-9, though i suspect it's not close enough in time to
> > > work
> > > > for you.
> > > > -s
> > > >
> > > > On Tue, Oct 11, 2016 at 1:46 PM, Chris Riccomini <
> > criccom...@apache.org>
> > > > wrote:
> > > >>
> > > >> I think Alex will be attending a conference in SF on the 14-15. Not
> > > >> sure which one, though.
> > > >>
> > > >> On Tue, Oct 11, 2016 at 12:53 PM, Bolke de Bruin  >
> > > >> wrote:
> > > >> > Are there any conferences around this time? Then I might be able
> to
> > > join
> > > >> > as well.
> > > >> >
> > > >> >
> > > >> >> Op 11 okt. 2016, om 17:53 heeft Chris Riccomini <
> > > criccom...@apache.org>
> > > >> >> het volgende geschreven:
> > > >> >>
> > > >> >> @gurer, does anyone from AirBNB want to speak? :)
> > > >> >>
> > > >> >> On Tue, Oct 11, 2016 at 8:51 AM, Gurer Kiratli
> > > >> >>  wrote:
> > > >> >>> Thank you Chris!!
> > > >> >>>
> > > >> >>> On Tue, Oct 11, 2016 at 8:46 AM, Chris Riccomini
> > > >> >>> 
> > > >> >>> wrote:
> > > >> >>>>
> > > >> >>>> Hey all,
> > > >> >>>>
> > > >> >>>> I've booked the meetup!
> > > >> >>>>
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> http://www.meetup.com/Bay-Area-Apache-Airflow-
> > > Incubating-Meetup/events/234778571/
> > > >> >>>>
> > > >> >>>> November 16, 2016, 6:30 PM
> > > >> >>>> 350 Convention Way, Redwood City, CA
> > > >> >>>>
> > > >> >>>> See you then!
> > > >> >>>>
> > > >> >>>> Cheers,
> > > >> >>>> Chris
> > > >> >>>>
> > > >> >>>> On Fri, Oct 7, 2016 at 1:08 PM, Chris Riccomini
> > > >> >>>> 
> > > >> >>>> wrote:
> > > >> >>>>> @sid Awesome! Will create the meetup event on Monday. Added to
> > > TODO.
> > > >> >>>>> :)
> > > >> >>>>>
> > > >> >>>>> @all please email if you'd like to speak. Will do three 15m
> > talks
> > > >> >>>>> with
> > > >> >>>>> 5m q&a. Nice short-ish talks.
> > > >> >>>>>
> > > >> >>>>> On Fri, Oct 7, 2016 at 11:24 AM, siddharth anand <
> > > san...@apache.org>
> > > >> >>>>> wrote:
> > > >> >>>>>> Great! I'm looking forward to the talk and to the meet-up. If
> > you
> > > >> >>>>>> are
> > > >> >>>>>> interested in speaking at the WePay November meet-up, respond
> > to
> > > >> >>>>>&

Re: Next Airflow meet-up

[Alex Van Boxel]

I don't want to impose, but yes. Normally, like 95% chance that I can make
it. The thing is that I got a go to come but I'm waiting for people to make
the travel arrangements. Maybe I can give the story of how we're moving
from Luigi to Airflow :-)


On Fri, Oct 7, 2016 at 6:36 PM Chris Riccomini 
wrote:

> I think WePay can do November 16. Alex, can you make it then?
>
> On Thu, Oct 6, 2016 at 8:38 PM, siddharth anand  wrote:
> > I think a sane process for deciding meet-ups is to go in order off this
> > list : https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups in a
> > first-come, first-serve basis. Sadly, the list is empty :-)
> >
> > I'll let CWIKI concurrency resolution algo decide - that's fair.
> >
> > Now, I could propose that WePay host in mid-November if they so choose
> and
> > that Alex be a guest speaker, which would give 2 talks on GCP, but hey, I
> > like the first-come, first-serve approach better.
> >
> > So, whomever wants to go next, decide when you are going and claim your
> > spot.
> > -s
> >
> > On Thu, Oct 6, 2016 at 6:34 PM, George Leslie-Waksman <
> > geo...@cloverhealth.com> wrote:
> >
> >> November 16th is right in the middle of Medicare's open enrollment
> period
> >> so it would be really bad time for Clover Health to commit to anything.
> >> It's basically our busiest time of the year.
> >>
> >> I have to nail a few things down but my tentative plan was for December
> >> 7th.
> >>
> >> If enough people prefer November 16th, Clover Health could do the 17Q1
> >> meetup instead of 16Q4.
> >>
> >> --George
> >>
> >> On Sat, Oct 1, 2016 at 12:37 AM siddharth anand 
> wrote:
> >>
> >> > +1
> >> >
> >> > -s
> >> >
> >> > On Sat, Oct 1, 2016 at 12:33 AM, Alex Van Boxel 
> >> wrote:
> >> >
> >> > > Hey guys, about the date. There is good chance I'm in SF for summit
> >> > (14-15
> >> > > November), I could try to extend it a day (so including November
> 16th).
> >> > So
> >> > > if you're looking for a date, think about November 16th (I'll
> volunteer
> >> > for
> >> > > a talk then).
> >> > >
> >> > > Would be great. Thanks.
> >> > >
> >> > > On Sat, Oct 1, 2016 at 8:51 AM siddharth anand 
> >> > wrote:
> >> > >
> >> > > > SGTM.
> >> > > >
> >> > > > -s
> >> > > >
> >> > > > On Fri, Sep 30, 2016 at 10:18 PM, George Leslie-Waksman <
> >> > > > geo...@cloverhealth.com> wrote:
> >> > > >
> >> > > > > I'll start the ball rolling at Clover and try to lock in a date
> >> with
> >> > > > plenty
> >> > > > > of advance notice.
> >> > > > >
> >> > > > > I'll probably aim for early December (first week perhaps) to
> avoid
> >> > > > > Thanksgiving and the December holidays, and because it will be
> >> after
> >> > > > we're
> >> > > > > through this year's annual enrollment period.
> >> > > > >
> >> > > > > --George Leslie-Waksman (Clover Health)
> >> > > > >
> >> > > > > On Fri, Sep 30, 2016 at 6:25 PM siddharth anand <
> san...@apache.org
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Thanks again Gurer.
> >> > > > > >
> >> > > > > > George,
> >> > > > > > I've made you an event organizer on
> >> > > > > >
> >> > > >
> >> >
> http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/members/
> >> > > > > 12914586/
> >> > > > > >
> >> > > > > > Please review the format, guidelines, etc... on the Meetup
> wiki
> >> and
> >> > > let
> >> > > > > us
> >> > > > > > know if you have any questions.
> >> > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> >> > > > > >
> >> > > > > > -s
> >> > > > > >
> >> > > > > >

Re: Next Airflow meet-up

[Alex Van Boxel]

Hey guys, about the date. There is good chance I'm in SF for summit (14-15
November), I could try to extend it a day (so including November 16th). So
if you're looking for a date, think about November 16th (I'll volunteer for
a talk then).

Would be great. Thanks.

On Sat, Oct 1, 2016 at 8:51 AM siddharth anand  wrote:

> SGTM.
>
> -s
>
> On Fri, Sep 30, 2016 at 10:18 PM, George Leslie-Waksman <
> geo...@cloverhealth.com> wrote:
>
> > I'll start the ball rolling at Clover and try to lock in a date with
> plenty
> > of advance notice.
> >
> > I'll probably aim for early December (first week perhaps) to avoid
> > Thanksgiving and the December holidays, and because it will be after
> we're
> > through this year's annual enrollment period.
> >
> > --George Leslie-Waksman (Clover Health)
> >
> > On Fri, Sep 30, 2016 at 6:25 PM siddharth anand 
> wrote:
> >
> > > Thanks again Gurer.
> > >
> > > George,
> > > I've made you an event organizer on
> > >
> http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/members/
> > 12914586/
> > >
> > > Please review the format, guidelines, etc... on the Meetup wiki and let
> > us
> > > know if you have any questions.
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > >
> > > -s
> > >
> > >
> > > On Fri, Sep 30, 2016 at 10:58 AM, Gurer Kiratli <
> > > gurer.kira...@airbnb.com.invalid> wrote:
> > >
> > > Hey guys,
> > >
> > > Sure Airbnb can always host, but I think it might be nice to switch
> > around.
> > > Clover is welcome to do it this quarter. We can always be a fallback.
> > >
> > > Cheers,
> > >
> > > Gurer
> > >
> > > On Fri, Sep 30, 2016 at 7:27 AM, George Leslie-Waksman <
> > > geo...@cloverhealth.com> wrote:
> > >
> > > > Clover Health would be happy to host a meetup, be it in Q4 or next
> > year.
> > > >
> > > > --George Leslie-Waksman
> > > >
> > > > On Thu, Sep 29, 2016 at 4:07 PM siddharth anand 
> > > wrote:
> > > >
> > > >> Gurer? Let us know what you think.
> > > >>
> > > >> Q4 tends to be a busy time, so we should get the next one set up at
> > the
> > > >> earliest. I'd personally like one in the South Bay at some point so
> > > that I
> > > >> actually had a snowball chance [in H] of making it in person, but
> > Airbnb
> > > >> has been very patient, so they are next if they so wish.
> > > >>
> > > >> -s
> > > >>
> > > >> On Thu, Sep 29, 2016 at 3:47 PM, Chris Riccomini <
> > criccom...@apache.org
> > > >
> > > >> wrote:
> > > >>
> > > >> > WePay can, but AirBNB was next in line. If they want to pass, we
> can
> > > do
> > > >> > it. :)
> > > >> >
> > > >> > On Thu, Sep 29, 2016 at 3:42 PM, siddharth anand <
> san...@apache.org
> > >
> > > >> > wrote:
> > > >> > > Anyone interested in hosting an Airflow meet-up in Q4 of this
> > year?
> > > If
> > > >> > so,
> > > >> > > reply to this note.
> > > >> > >
> > > >> > > -s
> > > >> >
> > > >>
> > > >
> > >
> > >
> > >
> >
>

Re: Airflow Releases

[Alex Van Boxel]

I'll do the same. Nice to have the 1.8 on the horizon.

On Fri, Sep 30, 2016 at 5:51 PM Chris Riccomini 
wrote:

> > I'm not sure how other projects do this, but I propose that we let the
> RC settle for a week.
>
> +1
>
> > Who's on board!?
>
> Me. :)
>
> On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin
>  wrote:
> > As related note: the most people we can get to test this release
> candidate
> > the better. Note that Airbnb uses only a subset of the Airflow codebase,
> > there are many operators and features we do not use at all, so we
> encourage
> > everyone to get on board with testing the RC.
> >
> > At Airbnb, our process starts with sending the RC into our staging
> cluster,
> > let it settle and fix any bug over 48 hours or so. We look for changes
> > around DagBag processing time, memory footprint on workers, smooth
> > processing around our canaries.
> >
> > When the RC is ready to hit production, we take a full backup of the
> > metadata DB, and make sure that the whole team is fully available to
> babysit
> > the release and hotfix any issue that may pop up, with a clear, fast
> revert
> > plan.
> >
> > As all of our tasks are assumed to be idempotent the worst case scenarios
> > are around delayed SLAs and wasted resources which is all manageable.
> >
> > I'm not sure how other projects do this, but I propose that we let the RC
> > settle for a week. I suggest that we reset the clock to 7 more days for
> each
> > increment of RC, unless the hotfix is very minor.
> >
> > Who's on board!?
> >
> > Max
> >
> > On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin
> >  wrote:
> >>
> >> Airbnb will cut a release candidate on the week of October 10th off of
> >> master.
> >>
> >> Community, if you have outstanding PRs that you want to make this
> release,
> >> now is the time! Though we do not want to rush-merge anything risky
> that may
> >> make the release process harder.
> >>
> >> Max
> >>
> >> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke
> >>  wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>>
> >>> We are currently running master in our environment and we'll be sure to
> >>> report any issues.
> >>>
> >>>
> >>> Do you know when the 1.8 release is due?
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Luke Maycock
> >>> OLIVER WYMAN
> >>>
> >>> luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com>
> >>> www.oliverwyman.com
> >>>
> >>>
> >>> 
> >>> From: Chris Riccomini 
> >>> Sent: 29 September 2016 18:14:02
> >>> To: dev@airflow.incubator.apache.org
> >>> Subject: Re: Airflow Releases
> >>>
> >>> Hey Luke,
> >>>
> >>> > Is there anything we can do to help get the next release in place?
> >>>
> >>> One thing that would definitely help is running master somewhere in
> >>> your environment, and reporting any issues that see. Over the next few
> >>> weeks, AirBNB and a few other folks will be doing the same in an
> >>> effort to harden the 1.8 release.
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
> >>>  wrote:
> >>> > Airflow Developers,
> >>> >
> >>> >
> >>> > We were looking at writing a workflow framework in Python when we
> found
> >>> > Airflow. We have carried out some proof of concept work for using
> Airflow
> >>> > and wish to continue using it as it comes with lots of great features
> >>> > out-of-the-box.
> >>> >
> >>> >
> >>> > We have created our own fork here:
> >>> >
> >>> > https://github.com/owlabs/incubator-airflow
> >>> >
> >>> >
> >>> > So far, the only thing we have committed back to the main repository
> is
> >>> > the following fix to the mssql_hook:
> >>> >
> >>> > https://github.com/apache/incubator-airflow/pull/1626
> >>> >
> >>> >
> >>> > Among other types of tasks, we wish to be able to run mssql tasks
> using
> >>> > Airflow. In order to do so, the above and below fixes are required:
> >>> >
> >>> >
> >>> > https://github.com/apache/incubator-airflow/pull/1458<
> https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed
> >
> >>> >
> >>> >
> >>> > We have created a Chef cookbook for configuring VMs with Airflow and
> >>> > its prerequisites. As part of this cookbook, we are installing the
> latest
> >>> > release of Airflow. However, it appears that the latest release does
> not
> >>> > have the aforementioned fixes.
> >>> >
> >>> > Do you know when the next release of Airflow is expected? Is there
> >>> > anything we can do to help get the next release in place?
> >>> >
> >>> >
> >>> > Luke Maycock
> >>> > OLIVER WYMAN
> >>> >
> >>> > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com>
> >>> > www.oliverwyman.com
> >>> >
> >>> >
> >>> > 
> >>> > This e-mail and any attachments may be confidential or legally
> >>> > privileged. If you received this message in error or are not the
> intended
> >>> > recipient, you should destroy t

Re: Airflow Developers Meeting - 08/03 Notes

[Alex Van Boxel]

Hey Maxime, I started to list them all and seems to be a lot of them (one
depending on the other). I think they had some breaking changes. Maybe we
should keep them for Airflow 1.8. I'll keep running on master for now.



On Wed, Aug 24, 2016 at 1:59 AM Jakob Homan  wrote:

> ASF provides per-projects blog infrastructure:
> http://www.apache.org/dev/project-blogs
>
> -Jakob
>
>
> On 23 August 2016 at 15:07, Maxime Beauchemin
>  wrote:
> > I'm planning on working on a minimal apache release this week. If someone
> > can point to the commit or PRs I can add them to `branch-1.7.2-apache`
> and
> > start moving towards the release.
> >
> > Max
> >
> > On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel 
> wrote:
> >
> >> All stuff is in master (I'm running pre-production on master), if it
> needs
> >> to be cherry-picked to a branch I can always test it. I would be grate
> if
> >> it's in the next release so I can start blogging about it.
> >>
> >> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin 
> wrote:
> >>
> >> > (Sorry for the late response, I was on holiday)
> >> >
> >> > I think the G* operators just need to be cherry picked. This will
> make us
> >> > deviate slightly from the
> >> > previous release, but makes sure we don’t have to ‘fix’ history
> >> afterwards.
> >> >
> >> > Anyone against this?
> >> >
> >> > - B.
> >> >
> >> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel 
> het
> >> > volgende geschreven:
> >> > >
> >> > > Sorry, Bolke. What needs to be done for Google Cloud
> operators/hooks?
> >> > >
> >> > > If you bring me up-to-speed I can do this. I'm currently working an
> a
> >> CI
> >> > > setup for the Google stuff).
> >> > >
> >> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin 
> >> wrote:
> >> > >
> >> > >> I have indeed a branch with cherry picked commits. This branch is
> >> > >> available in the apache repo. The branch is available as
> >> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we
> do
> >> > >> upgrade the google cloud operators/hooks as part of this release as
> >> the
> >> > >> licenses were only added after these changes came in. I prefer this
> >> > >> approach as we stay away from custom commits as much as possible
> then.
> >> > >>
> >> > >> - Bolke
> >> > >>
> >> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
> >> gurer.kira...@airbnb.com
> >> > .INVALID>
> >> > >> het volgende geschreven:
> >> > >>>
> >> > >>> Agenda
> >> > >>>
> >> > >>>
> >> > >>>  -
> >> > >>>
> >> > >>>  Committers sync-up: progress and plans
> >> > >>>  -
> >> > >>>
> >> > >>> Max to do a recap since last release
> >> > >>> -
> >> > >>>
> >> > >>> Airbnb
> >> > >>> -
> >> > >>>
> >> > >>> Anyone else?
> >> > >>> -
> >> > >>>
> >> > >>>  Cooperation Best Practices
> >> > >>>  -
> >> > >>>
> >> > >>>  Solicit for feedback for Impersonation Design Review
> >> > >>>  -
> >> > >>>
> >> > >>>  Release Schedule, Management
> >> > >>>  -
> >> > >>>
> >> > >>>  Roadmap discussion
> >> > >>>
> >> > >>>
> >> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
> >> Maxime
> >> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> >> > Wiedmer,
> >> > >>> Gurer Kiratli
> >> > >>>
> >> > >>> Notes
> >> > >>>
> >> > >>>
> >> > >>>  -
> >> > >>>
> >> > >>>  Recap for the overall project
> >> > >>>  -
> >> 

Re: Airflow Developers Meeting - 08/03 Notes

[Alex Van Boxel]

All stuff is in master (I'm running pre-production on master), if it needs
to be cherry-picked to a branch I can always test it. I would be grate if
it's in the next release so I can start blogging about it.

On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin  wrote:

> (Sorry for the late response, I was on holiday)
>
> I think the G* operators just need to be cherry picked. This will make us
> deviate slightly from the
> previous release, but makes sure we don’t have to ‘fix’ history afterwards.
>
> Anyone against this?
>
> - B.
>
> > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel  het
> volgende geschreven:
> >
> > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
> >
> > If you bring me up-to-speed I can do this. I'm currently working an a CI
> > setup for the Google stuff).
> >
> > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin  wrote:
> >
> >> I have indeed a branch with cherry picked commits. This branch is
> >> available in the apache repo. The branch is available as
> >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> >> upgrade the google cloud operators/hooks as part of this release as the
> >> licenses were only added after these changes came in. I prefer this
> >> approach as we stay away from custom commits as much as possible then.
> >>
> >> - Bolke
> >>
> >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli  .INVALID>
> >> het volgende geschreven:
> >>>
> >>> Agenda
> >>>
> >>>
> >>>  -
> >>>
> >>>  Committers sync-up: progress and plans
> >>>  -
> >>>
> >>> Max to do a recap since last release
> >>> -
> >>>
> >>> Airbnb
> >>> -
> >>>
> >>> Anyone else?
> >>> -
> >>>
> >>>  Cooperation Best Practices
> >>>  -
> >>>
> >>>  Solicit for feedback for Impersonation Design Review
> >>>  -
> >>>
> >>>  Release Schedule, Management
> >>>  -
> >>>
> >>>  Roadmap discussion
> >>>
> >>>
> >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
> >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> Wiedmer,
> >>> Gurer Kiratli
> >>>
> >>> Notes
> >>>
> >>>
> >>>  -
> >>>
> >>>  Recap for the overall project
> >>>  -
> >>>
> >>> [Max]
> >>>
> >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> >>> -
> >>>
> >>>  Airbnb Recap
> >>>  -
> >>>
> >>> [Paul] Airbnb is continuing work on features related to migrating
> >>> more jobs onto Airflow - DB connection scalability, impersonation,
> >>> refreshing web UI, DAG git versioning, task resource isolation
> >>> -
> >>>
> >>>  Other Recaps
> >>>  -
> >>>
> >>> [Chris] Improvements to Import functionality. Joy did some work
> >>> around CLIs and mark success in collaboration with Bolke.
> >>> -
> >>>
> >>> [Jeremiah] Came up with the Merge tool. Focusing on the work around
> >>> configuration.
> >>> -
> >>>
> >>>  Meeting cadence
> >>>  -
> >>>
> >>> [Max] We can have monthly meetings. Let’s start with monthly we can
> >>> change the cadence. We should invite all contributors. We should
> >>> specify in
> >>> the invite that everyone is welcome. Currently posting this on
> >>> the mailing
> >>> list and wiki.
> >>> -
> >>>
> >>> *Action Item*  Gurer to schedule monthly meetings next one at
> >> Airbnb.
> >>> Airbnb is welcoming folks to come onsite
> >>> -
> >>>
> >>>  Blog/Documentation
> >>>  -
> >>>
> >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> >>> structure our thoughts.
> >>> -
> >>>
> >>> [Andrew] Blog should have better language with screenshots and all.
> >>> This raises the level of effort bar and this might mean that
> >>> nobo

Re: Airflow Developers Meeting - 08/03 Notes

[Alex Van Boxel]

Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?

If you bring me up-to-speed I can do this. I'm currently working an a CI
setup for the Google stuff).

On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin  wrote:

> I have indeed a branch with cherry picked commits. This branch is
> available in the apache repo. The branch is available as
> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> upgrade the google cloud operators/hooks as part of this release as the
> licenses were only added after these changes came in. I prefer this
> approach as we stay away from custom commits as much as possible then.
>
> - Bolke
>
> > Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli 
> > 
> het volgende geschreven:
> >
> > Agenda
> >
> >
> >   -
> >
> >   Committers sync-up: progress and plans
> >   -
> >
> >  Max to do a recap since last release
> >  -
> >
> >  Airbnb
> >  -
> >
> >  Anyone else?
> >  -
> >
> >   Cooperation Best Practices
> >   -
> >
> >   Solicit for feedback for Impersonation Design Review
> >   -
> >
> >   Release Schedule, Management
> >   -
> >
> >   Roadmap discussion
> >
> >
> > Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
> > Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur Wiedmer,
> > Gurer Kiratli
> >
> > Notes
> >
> >
> >   -
> >
> >   Recap for the overall project
> >   -
> >
> >  [Max]
> >
> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> >  -
> >
> >   Airbnb Recap
> >   -
> >
> >  [Paul] Airbnb is continuing work on features related to migrating
> >  more jobs onto Airflow - DB connection scalability, impersonation,
> >  refreshing web UI, DAG git versioning, task resource isolation
> >  -
> >
> >   Other Recaps
> >   -
> >
> >  [Chris] Improvements to Import functionality. Joy did some work
> >  around CLIs and mark success in collaboration with Bolke.
> >  -
> >
> >  [Jeremiah] Came up with the Merge tool. Focusing on the work around
> >  configuration.
> >  -
> >
> >   Meeting cadence
> >   -
> >
> >  [Max] We can have monthly meetings. Let’s start with monthly we can
> >  change the cadence. We should invite all contributors. We should
> > specify in
> >  the invite that everyone is welcome. Currently posting this on
> > the mailing
> >  list and wiki.
> >  -
> >
> >  *Action Item*  Gurer to schedule monthly meetings next one at
> Airbnb.
> >  Airbnb is welcoming folks to come onsite
> >  -
> >
> >   Blog/Documentation
> >   -
> >
> >  [Max, Chris, Jeremiah] Having a blog would be useful. We can
> >  structure our thoughts.
> >  -
> >
> >  [Andrew] Blog should have better language with screenshots and all.
> >  This raises the level of effort bar and this might mean that
> > nobody really
> >  does it.
> >  -
> >
> >  [Max] Maybe use Medium? We can try it out.
> >  -
> >
> >  [Max] Haven’t looked at the documentation for a long time. I don’t
> >  really know about its quality.
> >  -
> >
> >  [Arthur] Looking at the email list responses, seems like the
> >  documentation is missing certain elements.
> >  -
> >
> >  [Max] Maybe we can interview people who just started and identify
> >  gaps.
> >  -
> >
> >  *Action Item* [Jeremiah] We can also ask the mailing list about
> this.
> >  He will send this out.
> >  -
> >
> >   Cooperation guidelines
> >   -
> >
> >  [Max, Arthur]
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> >  -
> >
> >  [Chris] Let’s link the guidelines to contributor md . This should be
> >  discoverable
> >  -
> >
> >   Release
> >   -
> >
> >  [Chris, Max] Bolke might have branch with High charts and Licensing
> >  cherry picked. We can use this as a dry run. We can test the
> process.
> >  -
> >
> >  [Max] Big release is probably around September.
> >  -
> >
> >  [Andrew] If we want to have a true practice, we need to be
> diligently
> >  following Apache guidelines.
> >  -
> >
> >  [Max] We might want to revive the email thread about how we should
> do
> >  this.
> >  -
> >
> >  [Chris] It would be great for someone who hasn’t done this before to
> >  do this.
> >  -
> >
> >  [Andrew] Building a script for this is smart otherwise the manual
> >  work is painful from experience.
> >
> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
> >
> >
> > https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release
> >
> >   -
> >
> >  *Action Item* [Max] Max will own this. This will happen by 08/19.
> >  -
> >
> >  [Arthur] For the main release around mid September. We need to
> >  confirm this date.
> >  -
> >
> >   Roadmap
> >   -
> >
> >  [Max] Maybe each company puts their own section? Or should we have a
> >  more coh