Fwd: Cannot access https://cms.apache.org/incubator/publish

Kaxil, Can you try these steps and update the airflow wiki (committer guide) based on your findings? -s Sent from Sid's iPhone Begin forwarded message: > From: Martin Gainty > Date: May 15, 2018 at 4:21:13 AM PDT > To: "san...@apache.org" > Subject:

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

FYI, can anyone pictorially describe the release process (and post it on the apache airflow wiki)? I think that would eliminate a lot of confusion in the future and avoid a rehash of this email thread on the next release. -s On Wed, Jul 19, 2017 at 10:48 AM, Hitesh Shah

Re: Podling Report Reminder - July 2017

I've updated the Airflow report on https://wiki.apache.org/incubator/July2017 Do let me know if you have any questions. -s On Wed, Jul 5, 2017 at 7:00 AM, wrote: > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It

Question about updating podling report

Folks! As I joined a new role (at a new company), it means that I also started using a new laptop. I'd like to update the podling report (due tomorrow(, but don't remember the specifics for connecting via svn to the incubator repo. Anyone have instructions? I believe I need to generate a new SSH

Re: Podling Report Reminder - July 2017

Adding this to my to-do list. On Tue, Jun 27, 2017 at 4:54 PM, wrote: > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare your quarterly board report. > >

Re: DAGs dont get refreshed ?

There is a manual DAG refresh option in the UI for DAGs the UI is already aware of -- this will reload a DAG. But that's not a complete solution to the the more general DAG refresh problem. -s On Tue, Jun 20, 2017 at 6:36 PM, siddharth anand <san...@apache.org> wrote: > To clarify, at

Re: DAGs dont get refreshed ?

PM, siddharth anand <san...@apache.org> wrote: > We actually do restart both Web and Schedulers. I know the scheduler does > reparse the files in the dag folder, but the current state of the web ui > does require a restart. > > -s > > On Tue, Jun 20, 2017 at 6:07

Re: DAGs dont get refreshed ?

We actually do restart both Web and Schedulers. I know the scheduler does reparse the files in the dag folder, but the current state of the web ui does require a restart. -s On Tue, Jun 20, 2017 at 6:07 PM, Ashika Umanga Umagiliya < umanga@gmail.com> wrote: > Greetings, > > We are using

Re: Passing Variables

Ah.. I completely missed the question.. in my haste to do too many things. Assuming you have a DAG named process_my_data with 3 tasks : read__from_source_table --> transform --> write_to_new_table. This dag should have a @none schedule. You could write a script to read your list of source tables

Re: Passing Variables

We use Airflow variables heavily. from airflow.models import Variable # Load an environment variable as a string ENV = Variable.get('ENV').strip() # Load an environment variable as JSON and access a JSON field named PLATFORM PLATFORM = 'EP' SSH_KEY = Variable.get('ep_platform_ssh_keys',

Re: New Apache Airflow meetup : in Tokyo

ke > > Airflow is not so popular in Japan yet, and a part of its reason is > few information in Japanese, IMHO. > The above slides are written in Japanese and really valuable for the > (potential) Airflow users in Japan. > > Regards, > > Kengo Seki <sek...@apache.org

Re: Concurrent schedulers

I did run into "double SLA miss alarms" firing, but that was on 1.7x. I haven't tested if that is still an issue in 1.8x. -s On Tue, May 23, 2017 at 8:46 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Awesome. I wasn't aware of DagRun locking, this is even better! > > Max > > On

Re: Removing members from dev list?

Great. Thx Andrew. -s On Mon, May 22, 2017 at 5:23 AM, Andrew Phillips wrote: > Hi Siddarth > > How do we (PMC) remove a email recipient from the dev list? >> > > Anyone who is a moderator of the list should be able to request removal of > a subscriber by sending an email to

Removing members from dev list?

How do we (PMC) remove a email recipient from the dev list? I keep getting requests to moderate the following because "kerzh...@yahoo-inc.com" is no longer at the yahoo. "kerzh...@yahoo-inc.com is no longer with Yahoo! Inc." -s

Re: Article: Why Robinhood uses Airflow

Tweeted it using the Airflow account! -s On Thu, May 11, 2017 at 5:33 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8 > > Grateful to have you on board Robinhood! >

Re: IMPORTANT: I need your pypi usernames

Chris, Sorry for the delay, my user name is r39132 for both of those! -s On Tue, May 9, 2017 at 1:25 PM, Chris Riccomini wrote: > I have added the following: > > https://pypi.python.org/pypi/apache-airflow > artwr, aoen, mistercrunch > >

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

https://issues.apache.org/jira/browse/AIRFLOW-1121 is merged to fix the webserver pid issue.. thx Kengo! -s On Tue, Apr 18, 2017 at 6:15 PM, Hitesh Shah wrote: > -1. > > Not sure if these have been called out earlier. > > For all the bundled files with different licenses

Re: Best practices on Long running process over LB

Another approach : 1. Airflow calls webservice in a fire-and-forget fashion 2. Webservice updates a message bus/stream (e.g. SQS) with result 3. An airfllow sensor pulls updates off SQS and processes them This saves airflow from polling your webservice which would in turn poll your DB.

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

https://issues.apache.org/jira/browse/AIRFLOW-1121 Jira filed. On Tue, Apr 18, 2017 at 1:27 PM, siddharth anand <san...@apache.org> wrote: > Sure. As soon as I get out of my meetings. > > -s > > On Tue, Apr 18, 2017 at 1:01 PM Chris Riccomini <criccom...@apache.org>

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

Sure. As soon as I get out of my meetings. -s On Tue, Apr 18, 2017 at 1:01 PM Chris Riccomini <criccom...@apache.org> wrote: > @Sid, can you open JIRA(s), and assign them as blockers to 1.8.1? > > On Tue, Apr 18, 2017 at 12:39 PM, siddharth anand <san...@apache.org> &g

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

Hmn.. it always worked for me for any of the releases we installed. I install `pip install ` -s On Tue, Apr 18, 2017 at 10:44 AM, Chris Riccomini wrote: > @Sid, how do you enable the versioning? I've never been able to get this to > work in my environment. It always

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

Just installed the rc in our staging env and letting it bake. FYI, I noticed that the version is not available at the following URI : /admin/versionview/ Here's a screenshot : https://www.dropbox.com/s/shdzwadb8klqwt2/Screenshot%202017-04-17%2020.57.06.png?dl=0 I installed via pip! -s On Mon,

Re: Welcome @saguziel as a committer and PMC member!

Welcome Alex! On Fri, Apr 14, 2017 at 8:23 AM, Chris Riccomini wrote: > Congrats, Alex! Welcome. :) > > On Thu, Apr 13, 2017 at 7:06 PM, Dan Davydov invalid > > wrote: > > > Alex (@saguziel - AirBnB) has been making contributions and reviews for

New Apache Airflow meetup : in Tokyo

Live in Tokyo & want to contribute to @ApacheAirflow ? Check out our new Tokyo meetup : http://bit.ly/2o7jXWF . First meetup on May 11 : https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/events/238731591/ Thanks to Kengo

Re: Cleanup

Edgardo, This is a great question and something that requires functionality to address. As Airflow starts getting used for bigger workloads, we need a way to clean up defunct resources. - How do we delete a dag and its related resources? - Until the recent release, the way that I stopped

Re: 1.8.1 release

: https://issues.apache.org/jira/browse/AIRFLOW-1055 I have a PR that works for this, but will need to add unit tests for it as well as for AIRFLOW-1013. -s On Wed, Mar 29, 2017 at 3:24 PM, siddharth anand <san...@apache.org> wrote: > Didn't realize https://issues.apache.org/jira/browse/AIR

Re: 1.8.1 release

rowse/AIRFLOW-1003 > >> > https://issues.apache.org/jira/browse/AIRFLOW-1001 > >> > > >> > It looks like AIRFLOW-1017 is done, though the JIRA is not closed. > >> > > >> > The rest remain open. I will wait on the release until the remaining

Re: 1.8.1 release

Ruslan, Thanks for sharing this list. I can pick a few up. I agree we should aim to get some of them into 1.8.1. -s On Tue, Mar 21, 2017 at 2:29 PM, Ruslan Dautkhanov wrote: > Some of the issues I ran into while testing 1.8rc5 : > >

Re: [RESULT][VOTE]Release Airflow 1.8.0 based on Airflow 1.8.0rc5

n Thu, Mar 16, 2017 at 9:01 AM, Bolke de Bruin <bdbr...@gmail.com> > wrote: > >>> > >>>> Hello, > >>>> > >>>> Apache Airflow (incubating) 1.8.0 (RC5) has been accepted. > >>>> > >>>> 9 “+1” votes received: > &

Reminder : LatestOnlyOperator

With the Apache Airflow 1.8 release imminent, you may want to try out the *LatestOnlyOperator.* If you want your DAG to only run on the most recent scheduled slot, regardless of backlog, this operator will skip running downstream tasks for all DAG Runs prior to the current time slot. For

Re: Airflow Committers: Landscape checks doing more harm than good?

+1 for replacing it with travis linting. On Thu, Mar 16, 2017 at 7:59 PM, Jeremiah Lowin wrote: > FWIW I recently started using yapf (https://github.com/google/yapf) with a > slightly custom config to format all of my projects. Rather than alert to > discrete linting errors

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

t;> > >>>>> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbr...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> That is not an airflow error, but a Kerberos error. Try executing > the > >>>>>> kinit command on the command line by yourself. > >

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

> > > >> > `airflow kerberos` is broken in 1.8-rc5 > >> > https://issues.apache.org/jira/browse/AIRFLOW-987 > >> > Hopefully fix can be part of the 1.8 release. > >> > > >> > > >> > > >> > -- > >> > Ruslan Dautkh

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

is a major regression because it will force everyone to use the CLI for things that they would normally use the UI for. -s -s On Tue, Mar 14, 2017 at 1:32 PM, Daniel Huang <dxhu...@gmail.com> wrote: > +1 (non-binding)! > > On Tue, Mar 14, 2017 at 11:35 AM, siddharth anand &l

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

I'm going to deploy this to staging now. Fab work Bolke! -s On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov wrote: > I'll test this on staging as soon as I get a chance (the testing is > non-blocking on the rc5). Bolke very much in particular :). > > On Mon, Mar

Re: scheduler running on multiple nodes

I did run 2 or more schedulers with Local Executors up until mid last year. There have been enough changes to the code and feature additions that I don't think this is a recommended practice at this point. Also, there is not a lot of synchronization in the scheduler to ensure this will work. -s

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

nd here: > > >>>> > > >>>> http://mail-archives.apache.org/mod_mbox/incubator-general/ > > >>> 201702.mbox/% > > >>>> 3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3e < > > >>>> http://mail

Re: Meetup featuring an Airflow talk tomorrow

Nice! https://twitter.com/ApacheAirflow/status/834945481440546816 Do please share slides and video so we can post both on via twitter & wiki. -s On Wed, Feb 22, 2017 at 3:00 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hi, > > Just wanted to let you know that Arthur (one of our

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

+1 (binding). Thx Bolke! -s On Mon, Feb 20, 2017 at 2:51 PM, Alex Van Boxel wrote: > +1 (binding) > > On Mon, Feb 20, 2017 at 5:32 AM y...@yahoo-inc.com.INVALID > wrote: > > > > > +1 (non-binding) > > > > Thanks for all the work! > > > > YiOn

Re: Soliciting feedback: Using the Airflow CLI as a thin client

Hi Wilson, I'm a huge fan of the CLI and you are correct that the released current version of the CLI requires both a connection to the DB and access to the dag folder. In the new 1.8.0 release that is currently being driven by Bolke, the CLI uses the API. I'm not 100% sure that all CLIs commands

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Deployed to stage and will watch over the weekend before voting. -s On Sat, Feb 11, 2017 at 9:53 AM, Jeremiah Lowin wrote: > Boris, I submitted a PR to address your second point -- > https://github.com/apache/incubator-airflow/pull/2068. Thanks! > > On Sat, Feb 11, 2017 at

Re: Airflow Meetup @ Paypal (San Jose)

t 5:07 AM, Bolke de Bruin <bdbr...@gmail.com> > wrote: > > > > > I might. But I would maybe be more interested in a kind of round table > / > > > panel session to discuss directions? Does that make sense? Or would you > > > like me to talk about a spe

Re: Airflow Meetup 1Q17 Talk Videos

, Feb 6, 2017 at 2:27 PM, siddharth anand <san...@apache.org> wrote: > Thx George. > > <http://goog_936078273> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links > > I've added it to the page above and created a section for all Meetup > Videos. &g

Re: Airflow 1.8.0 Release Candidate 1

| boolean| not null Indexes: "dag_stats_pkey" PRIMARY KEY, btree (dag_id, state) The PKEY is a combination of 2 provided columns, so I'm wondering why Alembic is complaining here. On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand <san...@apache.org> wrote: > Actually, I

Re: Airflow 1.8.0 Release Candidate 1

table. I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any version greater than 0.9.8 -s On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand <san...@apache.org> wrote: > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install > https://dist.apache.org/repos/dist/

Re: Airflow 1.8.0 Release Candidate 1

I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow upgradedb didn't quite work. First, I thought it completed successfully, then saw errors some tables were indeed

Re: Airflow Meetup 1Q17 Talk Videos

Thx George. https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links I've added it to the page above and created a section for all Meetup Videos. Community members, For future reference, even if you don't plan to stream, I'd recommend recording your meetups so they can live on here.

Re: NYC Airflow Meetup

Great! Thanks for creating it - I've just joined so you can add me as an organizer. I've linked to it on : - https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements - https://twitter.com/ApacheAirflow/status/827743162789605382 -

Re: Airflow Meetup in NYC @ Blue Apron

Hope this went well. Feel free to share videos and slides. Also, it would be great if we could create a NY Apache Airflow meetup page. Would you be interested in setting one up? It would be easier to promote a meetup page on social media than an email on this list. -s On Fri, Jan 20, 2017 at

Re: Airflow Meetup @ Paypal (San Jose)

onl...@gmail.com> wrote: > Sure, > I have created event on Meetup : > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/ > 237412864/ > > Thanks for helping on this Siddharth. > Jayesh > > > > On Wed, Feb 1, 2017 at 7:50 PM, siddharth anand <

Re: Airflow Meetup @ Paypal (San Jose)

ybody else ? > > > > > > - Jayesh > > > > On Tue, Jan 31, 2017 at 8:16 PM, siddharth anand <san...@apache.org> > > wrote: > > > > > Jayesh, > > > Looks good. No need to vote. Just publish a new event with details on > the

Re: Airflow Meetup in NYC @ Blue Apron

Also, if you record a video, we'd be happy to place it on the wiki and promote it via our twitter feed, etc... -s On Mon, Jan 30, 2017 at 5:34 PM, Boris Tyukin wrote: > i hope you guys can share presentation slides at least for all of us who > are not in NYC > > On Mon,

Re: Airflow Meetup @ Paypal (San Jose)

; > > > > russell.jur...@gmail.com> > > > > > wrote: > > > > > > > > > > > Boris, would you be able to attend an evening meetup on the > nights > > of > > > > > 3/15 > > > > > > or 3/16? I think at

Re: Airflow Meetup @ Paypal (San Jose)

..@gmail.com > > > wrote: > > > Unfortunately, Strata has no room for us :( Paypal sounds like a great > > option. > > > > Jayesh, sounds like you're driving? :) > > > > On Tue, Jan 24, 2017 at 12:04 PM, siddharth anand <san...@apache.org&g

Re: Airflow Meetup @ Paypal (San Jose)

Russell, Let us know what you learn about Strata. Even if Strata offers up rooms to communities for free (based on information such as community size, etc...), I'm doubtful they would cover food and drinks. That cost would need to be carried by a sponsor -- i.e. you'd need to find a sponsor for

QCon London

Hi Folks! I will be attending QCon London Mar 5-8. Happy to meet locals and talk Airflow and data infrastructure if there is interest. FYI, I'm also a co-chair for QCon London and would be very interested in getting a deeper understanding of the local (London and environs) tech scene and potential

Re: Medium series: Airflow for Google Cloud

Looks like you don't have an account.. once you create one.. let me know and I will grant you admin perms on the wiki. -s On Fri, Jan 20, 2017 at 6:08 PM, siddharth anand <san...@apache.org> wrote: > I've added it to https://cwiki.apache.org/confluence/display/AIRFLOW/ > Airflow+Li

Re: Medium series: Airflow for Google Cloud

I've added it to https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links Feel free to add future posts to this page. You should have access. -s On Fri, Jan 20, 2017 at 3:23 PM, Alex Van Boxel wrote: > Hey all, > > now that 1.8 is nearing release. I finally started

Re: New book covers Airflow with PySpark: Agile Data Science 2.0 (O'Reilly, 2017) AND Airflow Meetup?

host it. > > When would be a good date, do you think? Late February? > > On Thu, Jan 19, 2017 at 5:19 PM, siddharth anand <san...@apache.org> > wrote: > > > Sounds like a great idea. We are looking for someone to host the next > one.. > > once one is announced, yo

Re: Airflow Github Mirror is not synchronizing

The repo mirror is syncing now. -s On Sat, Dec 31, 2016 at 11:22 AM siddharth anand <san...@apache.org> wrote: > FYI! > I've reopened my earlier JIRA issue. It looks like multiple Apache > Projects are reporting the same. > https://issues.apache.org/jira/browse/INFRA-1294

Airflow Github Mirror is not synchronizing

FYI! I've reopened my earlier JIRA issue. It looks like multiple Apache Projects are reporting the same. https://issues.apache.org/jira/browse/INFRA-12949 Newly merged changes won't be available to contributors/users until the mirroring issue is fixed.

[AIRFLOW-676] Do not allow Pools with 0 slots

https://github.com/apache/incubator-airflow/pull/1967 Hi Folks! Would appreciate your feedback on the following pull request. If you'd like this functionality, please provide a +1 on the PR itself. -s

Re: Podling Report Reminder - January 2017

I'll put this together. -s On Thu, Dec 29, 2016 at 6:31 PM, wrote: > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare your quarterly board report. > >

Re: Airflow 2.0

Max, Do you have time to summarize this thread? Perhaps, publish it on the Wiki! -s On Thu, Dec 1, 2016 at 12:27 PM, Van Klaveren, Brian N. < b...@slac.stanford.edu> wrote: > With the announcement of AWS Batch (https://aws.amazon.com/batch/), and > my own selfish needs, I think it'd be really

Re: Simple Feature Request

e to have it.. > > > > On Thu, Nov 24, 2016 at 3:51 AM, siddharth anand <san...@apache.org> > wrote: > > > If we support a test query dialog, we could just execute the query via > each > > hook's pre-existing execute method. > > > > -s > > >

Re: Simple Feature Request

st” function. > > > > Op 23 nov. 2016, om 21:28 heeft siddharth anand <san...@apache.org> het > volgende geschreven: > > > > Folks! > > Here's a nice (simple) feature request if someone would like to > implement it and file a PR. > > > > On the Admin--

Simple Feature Request

Folks! Here's a nice (simple) feature request if someone would like to implement it and file a PR. On the Admin-->Connections page, you are add a connection. But how do you know if it works? How about adding a "Test Connection" button on the edit page to test if the parameters you have specified

Re: Dynamic creation of DAG

Hi Max, Which part in the above PR is related to dynamic dags? When thinking about adding documentation about functionality, I propose the community bias towards adding working examples and test coverage. We offer a quick start (which by the way needs some updates - for example, why does it not

Re: Airflow 2.0

ompanies have a luxury to run all the systems > on UTC. > > > On Mon, Nov 21, 2016 at 5:26 PM, siddharth anand <san...@apache.org> > wrote: > > > Also, a survey will be a little less noisy and easier to summarize than > +1s > > in this email thread. > &

Idea for a new UI feature

A sort of obvious and missing feature of Airflow (awesome UI) is knowing whether your DAGs are missing SLAs. For example, say that you have an Hourly DAG and say that you have specified the dag run timeout or sla=timedelta(hours=2), then after 2 hours, you will receive an email of the SLA miss.

Re: Subsequent Airflow Meetup: 2017/01/11

Nov 10, 2016 at 10:17 AM, siddharth anand <san...@apache.org> > wrote: > > > Kevin, > > Here's a link to the 1Q17 meet-up. > > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/ > > 235259523/ > > > > Both upcoming meet-ups (next week at We

Re: Github Mirroring currently broken

This appears to be working again! -s On Sun, Nov 20, 2016 at 12:10 PM, siddharth anand <san...@apache.org> wrote: > Committers/Maintainers, > The Apache Airflow Github mirror is not synchronizing. I've filed a > ticket. It looks like, as of now, 2 other Apache projects (nifi &

Github Mirroring currently broken

Committers/Maintainers, The Apache Airflow Github mirror is not synchronizing. I've filed a ticket. It looks like, as of now, 2 other Apache projects (nifi & brooklyn-server) have reported the same issue. https://issues.apache.org/jira/browse/INFRA-12949 This means that although we are

Re: Airflow 2.0

David https://issues.apache.org/jira/browse/AIRFLOW-558 (i.e. http s://github.com/apache/incubator-airflow/pull/1830 ) Is on my plate.. have already gone through many rounds of reviews, testing, and fixes with the submitter and does not need to wait till 2.0. We should be able to merge it soon.

Re: Airflow installation error on ubuntu

Kapil, Please resubmit the email with links to the images. All embedded or attached objects are stripped from email before Apache mail servers forward them to recipients. -s On Fri, Nov 18, 2016 at 4:20 PM, Kapil Khandelwal wrote: > Hi, > > I am trying to run command pip

Priorities for the Airflow Community

As we near the last month of the year and the 9th month in Apache Incubation, I'd like to share some thoughts and gauge community feedback. At the beginning of this year, we saw a good deal of interest in Airbnb's Airflow project. As more companies started using it for business critical work, it

Re: Slides & Video Recording from yesterday's Airflow Talk

ions to edit the wiki > > On Thu, Nov 17, 2016 at 10:41 AM, Chris Riccomini <criccom...@apache.org> > wrote: > > > We'll be posting the video recording shortly. IT is working on it. :) > > > > Will post link on the meetup and mailing list. > >

Slides & Video Recording from yesterday's Airflow Talk

Chris and WePayEng, Thanks for hosting another great Airflow meet-up. Can all of the speakers post their slides online and add links to those talks in response to this email (and also on our Wiki)? -s

Cold Case PR Cleanup Status - Deadline Extended to Nov 30

Committers, please resolve any open Cold Case PRs that you still have on your list below : https://cwiki.apache.org/confluence/display/AIRFLOW/Cold-Case+PR+Resolution In the past 6 weeks, we've cleaned up over 70 (from over 110 down to 36! We have 16 more to go to get to our goal of ~20. Dan,

Re: Tasks getting Queued when Pool is full sometimes never get run

Not seeing this, but I am not running master in my staging or production envs currently. -s On Mon, Nov 14, 2016 at 12:38 PM, Ben Tallman wrote: > We are seeing an issue when running Master where tasks sometimes never run. > It seems that once they get marked as Dependencies

Re: BranchPython : failed stated not propagated to downstream taks

correct On Tue, Nov 15, 2016 at 1:11 AM wrote: Thanks siddharth for your answer. I'm going to look at the extra "failure processing branch" as you suggested. >It is intended that a DAGRun be deemed successful in all cases except for > failure. So, skipped nodes F and G, would

Using Airflow? Add your Company to the ReadMe!

https://github.com/apache/incubator-airflow/blob/master/README.md If you haven't yet, make sure to add your company to the list of companies using Airflow! Submit a PR to add your company to the list. -s

Re: Anyone noticed PR's for 41 & 137?

Hi Gerard, I have them both under my name on the following wiki though anyone from the community is welcome to help with the review and test it out as well. https://cwiki.apache.org/confluence/display/AIRFLOW/Cold-Case+PR+Resolution About 6 weeks ago, we had over 110 open PRs, many of which were

Re: Gantt chart broken on master?

en, right? > > > PS: link to screenshot > > https://www.dropbox.com/s/wql05icqgnewl0d/Screenshot%202016-11-14%2006.55.43.png?dl=0 > > > > > On Mon, Nov 14, 2016 at 12:30 PM, siddharth anand <san...@apache.org> > wrote: > > > Found the issue.. it seem

Re: Skip task

For cases like this, we (Agari) use the following approach : 1. Create a Variable in the UI of type boolean such as *enable_feature_x* 2. Use a ShortCircuitOperator (or BranchPythonOperator) to Skip downstream processing based on the value of *enable_feature_x* 3. Assuming that you

Re: Issue with latest versions of Celery & Kombu

I don't run celery. However, I would suggest that any solutions that you come up with be followed up by a PR with version updates to setup.py. On Mon, Nov 14, 2016 at 11:56 AM, Miller, Robin < robin.mil...@affiliate.oliverwyman.com> wrote: > Hi Nadeem, > > > We are using Celery with RabbitMQ.

Re: Issue with airflow upgradedb...

Ben, I ran into issues while maintaining my company's airflow fork and cherry-picking my changes into the fork, especially when my changes included db changes. I had to play with the alembic_version in the db and do some other magic that escapes me now. My best guidance for the future is to

Re: BranchPython : failed stated not propagated to downstream taks

A couple things are at play. The best practice is to ensure that your branch python operator does not fail. Catch the most general exception and return a value that would pick a "failure processing branch". If neither D nor E is your failure processing branch, then add one more branch -

Re: Cold Case PR Cleanup -- Current Status

Current status : we have < 50 open PRs presently! https://cwiki.apache.org/confluence/display/AIRFLOW/Cold-Case+PR+Resolution -s On Sat, Nov 5, 2016 at 8:06 PM, siddharth anand <san...@apache.org> wrote: > Committers, > If you have time this week, please make a push to get your

Re: [VOTE]: Using fractional seconds

SGTM On Sun, Nov 13, 2016 at 12:02 PM, Bolke de Bruin wrote: > Hi All, > > I count 3 positive votes, 0 negative ones. Therefore, I will finalize > https://github.com/apache/incubator-airflow/pull/1794 which implements > Option 1. > > Thanks! > Bolke > > > Op 9 nov. 2016, om

Gantt chart broken on master?

Gantt chart is broken for me on master. I think it's due to this merge. https://github.com/apache/incubator-airflow/commit/868bc83137adca0ebfd5780f0dff5a7bfdfaadf9 Why is an end_date needed? [image: Inline image 1] This is the tree view: [image: Inline image 2] Sumit, as the merger/committer,

Fwd: Cold Case PR Cleanup -- Current Status

Committers, If you have time this week, please make a push to get your cold case PR cleanup done. The deadline is Nov 15, just in time for our WePay meetup. I will be making an announcement there. -s -- Forwarded message -- From: *siddharth anand* <san...@apache.org> Date:

Re: Cold Case PR Cleanup -- Current Status

the maintainer group sticks to. BTW, I've seen a few contributors pitch in by reviewing PRs. That is extremely helpful and speeds up PR review/merge times. Please continue. Here's the current cold-case status. [image: Inline image 1] -s On Wed, Nov 2, 2016 at 12:50 PM, siddharth anand <

Re: Cold Case PR Cleanup -- Current Status

this endeavor started, we were at *110* open PRs. [image: Inline image 1] -s On Wed, Nov 2, 2016 at 12:30 AM, siddharth anand <san...@apache.org> wrote: > https://cwiki.apache.org/confluence/display/AIRFLOW/ > Cold-Case+PR+Resolution > > [image: Inline image 1] > > -s >

Re: Astronomer.io Airflow blog post

Merged. On Wed, Nov 2, 2016 at 11:43 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Here: > https://github.com/apache/incubator-airflow/pull/1863 > > > On Wed, Nov 2, 2016 at 10:51 AM, siddharth anand <san...@apache.org> > wrote: > > > Removi

Re: Astronomer.io Airflow blog post

Removing laurel's email address as it appears to be wrong. -s On Wed, Nov 2, 2016 at 10:49 AM, siddharth anand <san...@apache.org> wrote: > +1 for that idea. We should place all links on the wiki and just have the > project page point to the wiki! > (https://airflow.incub

Re: Astronomer.io Airflow blog post

+1 for that idea. We should place all links on the wiki and just have the project page point to the wiki! (https://airflow.incubator.apache.org/project.html). Gerard, would you like to file a quick PR for that change and I can approve/merge it? -s On Wed, Nov 2, 2016 at 10:37 AM, Gerard Toonstra

Cold Case PR Cleanup -- Current Status

https://cwiki.apache.org/confluence/display/AIRFLOW/Cold-Case+PR+Resolution [image: Inline image 1] -s

Re: The best place to initialize a db table whenever airflow starts ?

A DAG Run is a run for a particular dag at a particular time. For example, if your DAG were called FOO and if its schedule interval were @hourly, then you would have DAG runs : Foo @ noon, Foo @ 1p, Foo @ 2p, etc... If you need to seed a table prior to each run, just add a task at the start of

Re: Possible airflow-pool bug

Yes.. we have seen the over-subscription of pools. We do need a fix for it --- I don't believe there is one. We need someone to own and fix it.. happy to review a PR. We use pools at Agari for all of our needs. We are okay with mild oversubscription, so we do see numbers slightly higher, but our

  1   2   3   >