Fwd: Apache Airflow (incubating) PPMC Meeting Minutes - April 20, 2016

-- Forwarded message -- From: Dan Davydov Date: Wed, Apr 20, 2016 at 11:02 PM Subject: Re: Apache Airflow (incubating) PPMC Meeting Minutes - April 20, 2016 To: Siddharth Anand Ack, thanks for putting this together! Will sync up with Jlowin re the fixes and then do a release. I

Apache Airflow Talk at DataDaySeattle - July 23

Hi Folks!If you plan on being in the Seattle area on July 23, check out Data Day Seattle I'll be speaking about Apache Airflow. If there is enough interest, I can probably do a meet-up as well that weekend.  Talk Title : Introducing Apache Airflow (Incubating) - A Better Way to Build Data Pipel

Re: GitHub Issue #1427 takeaways

I have the following concerns, which may be best illustrated with examples:  Start date : 2016-01-01Schedule Interval : */20 * * * *  (run every 20 mins) Expectation : Runs on  - 20160101T00 - 20160101T002000 - 20160101T004000 - 20160101T014000 - ... Start dat

Apache Airflow (incubating) Meeting Minutes - April 27, 2016

*Attendees* : Max, Sid, Chris R, Bolke, Jeremiah *Topics* : - *Moving to Apache* - Steps, Owners, Milestones/Timelines with links to Jira issues are documented on - https://cwiki.apache.org/confluence/display/AIRFLOW/Migrating+to+Apache - Item missed :

Re: 1.7.1 release status

Btw, is anyone of the committers running 1.7.0 or later in any staging or production env? I have to say that given that 1.6.2 was the most stable release and is 4 or more months old does not say much for our release cadence or process. What's our plan for 1.7.1? Sent from Sid's iPhone > On Ap

Re: Board report

I can if someone wants to send me pointers. -s On Friday, April 29, 2016 1:18 PM, Chris Riccomini wrote: So, who wants to write the board report? :)

Re: Board report

l in a lot; the project is just starting, which is a fine response. -jg On 29 April 2016 at 13:23, Siddharth Anand wrote: > I can if someone wants to send me pointers. > -s > >    On Friday, April 29, 2016 1:18 PM, Chris Riccomini >wrote: > > >  So, who wants to write the board report? :) > > >

Apache Airflow (Incubating) : May Board Report Draft (comments welcome)

Folks!Please provide comments on the draft below. Airflow Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines. Airflow has been incubating since 2016-03-31. Three most important issues to address in the move towards graduation:   1. Continue

Re: Board report

https://wiki.apache.org/incubator/April2016 >> >> Here's where to edit it for May: https://wiki.apache.org/incubator/May2016 >> >> Don't feel pressure to fill in a lot; the project is just starting, >> which is a fine response. >> >> -jg >>

Re: Board report

I've updated the May Board Report for Apache Airflow (Incubating). Mentors, please let me know if you have any thoughts or questions. https://wiki.apache.org/incubator/May2016 On Friday, April 29, 2016 2:56 PM, Siddharth Anand wrote: Ok, you should see a separate email with a

Apache Airflow (Incubating) Twitter Account

Hi Folks,Interested in the latest news about the Apache Airflow (Incubating) project, follow our newly-minted twitter account! ApacheAirflow (@ApacheAirflow) | Twitter | | | | || | | | || ApacheAirflow (@ApacheAirflow) | Twitter The latest Tweets from ApacheAirflo

Re: AIRFLOW-20: Improving the scheduler by make dag runs more coherent

Bolke, Thanks for taking the lead on this. Do you mind creating a confluence wiki page with this proposal, so that we can link email, jiras, and the signal to it. We will need a roadmap linked to our main site as well. Sent from Sid's iPhone > On May 1, 2016, at 12:45 PM, Bolke de Bruin wrote

Add a Jira for all GitHub PRs

Thx to @jlowin for creating the Github label Missing JIRA Issue  Fellow maintainers, we need to ensure all PRs have an associated Jira before merging them - please continue to enforce this. At the end of the month, when the next Apache board report is due, it will be easier to use Jira as a sourc

Add a Jira for all GitHub PRs(2)

Thx to @jlowin for creating the Github label Missing JIRA Issue  Fellow maintainers, we need to ensure all PRs have an associated Jira before merging them - please continue to enforce this. At the end of the month, when the next Apache board report is due, it will be easier to use Jira as a sourc

Re: [jira] [Commented] (AIRFLOW-35) UI toggle to recursively set success for nested DAG operators

Merged the PR Sent from Sid's iPhone > On May 3, 2016, at 1:10 PM, Chris Riccomini (JIRA) wrote: > > >[ > https://issues.apache.org/jira/browse/AIRFLOW-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269483#comment-15269483 > ] > > Chris Ric

Status of Apache Migration

Committers/Maintainers :  If you have an open item on  https://cwiki.apache.org/confluence/display/AIRFLOW/Migrating+to+Apache, please send status/blockers on your items :  Specifically: - Max (repo migration)  - Jeremiah (travis migration) - I know you are waiting on Max -

Why do we need SQLite in Airflow?

>From time to time, we run into bugs with the SQLite dialect in SQLAlchemy and >close the bugs as "wont-fix" because we don't want to be in the business of >fixing such bug. We deem SQLite as a "non-serious" database that no one [in >his/her right mind] would run in his/her staging, qa, or produ

Re: Status of Apache Migration

xperiences migrating repos > on > > Github, the transition should be smooth where Github handles all of the > > redirection. Even Travis and other Github ecosystem tools should just > work. > > > > Chris, I assume the merge button will be greyed on PRs, wha

Re: Status of Apache Migration

That's right.. forgot about that. Cool. Thx. -s On Wednesday, May 4, 2016 7:04 PM, Maxime Beauchemin wrote: Github has a branch protection feature that prevents force pushes and branch deletion. I just turned it on for `master`. ​ On Wed, May 4, 2016 at 11:18 AM, Siddharth

Apache Infra Migration

Hi Folks!We've made good progress on the move to Apache.  Based on  https://cwiki.apache.org/confluence/display/AIRFLOW/Migrating+to+Apache, we still have the following remaining items :  - Creating a website : https://issues.apache.org/jira/browse/AIRFLOW-9  (Owner : Max) - Migrating to

Re: JIRA Account Creation

Gurer,We met at the first meet-up and welcome to Apache Airflow (Incubating) Please set accounts up on the following : * dev@airflow.incubator.apache.org (you can do this by sending an email to  dev-subscr...@airflow.incubator.apache.org)* comm...@airflow.incubator.apache.org (you can do this by

Blocker Bug on Latest airbnb_1.7.1 Release Candidate

Hi Folks!There is a bug on current Master that might be related to a commit on April 4. I need to confirm with the committer. If so, then we have another blocker bug on the release candidate.  https://issues.apache.org/jira/browse/AIRFLOW-106 In a nutshell:Task retries, on_failure callback, and

Blocker Bug on Latest airbnb_1.7.1 Release Candidate

Hi Folks! Resending an updated email version, because I sent it earlier with my non-apache email id.  There is a bug on current Master that might be related to a commit on April 4. I need to confirm with the committer. If so, then we have another blocker bug on the release candidate.  https://is

Voting Changes for Scheduler-related PRs/Commits

Hi Folks!As many of you know, Apache Airflow (incubating) came from Airbnb, where it currently still represents the largest Airflow deployment. Airflow entered the Apache Incubator shortly over a month ago but still depends on Airbnb's production deployment to vet its release candidates. As Airf

Re: Blocker Bug on Latest airbnb_1.7.1 Release Candidate

Dan, You may proceed. I have merged the change to master.-s On Thursday, May 12, 2016 6:27 AM, Dan Davydov wrote: Thanks for finding this Sid, will wait on this before staging. On May 11, 2016 8:04 PM, "Siddharth Anand" wrote: > Hi Folks!There is a bug on current Master

Re: Summary of committer meeting 2016-05-12

I mentioned this on the call yesterday as well. Going forward, all meetings will be community-inclusive. We can follow what Apache Beam is doing ( they have 10-15+ video windows at a time ) in this respect. We will need a topic and agenda for each meetings, so that they are not misconstrued as "

Re: Summary of committer meeting 2016-05-12

Bolke, Thanks for providing the document and for generally driving a path forward. Regarding Local vs. Celery I think the project benefits greatly from having multiple executors. Widespread adoption of Airflow involves keeping the barriers to adoption as low as possible. We ship Airflow with a S

Re: Summary of committer meeting 2016-05-12

SQS or whatever, great. > > I just want it easy and tested/stable. > > Cheers, > Chris > > On Fri, May 13, 2016 at 11:57 AM, Siddharth Anand > wrote: > >> Bolke, Thanks for providing the document and for generally driving a path >> forward. >> Re

[DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

Folks, It's a good time for us to discuss whether we will adopt a RTC or CTR policy towards software changes. In the RTC case, we need 3 +1 binding (a.k.a. committer) votes and 0 binding vetos for any change as RTC requires consensus approval: - http://www.apache.org/foundation/glossary.html#R

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

y, I don't have a strong preference either way, except that I > > strongly prefer not to have a separate development branch. If you force > me > > to pick, I'd pick RTC. I find that RTC encourages a shared understanding > of > > the code, which is useful. > >

New Airflow Contributors Guide

https://cwiki.apache.org/confluence/display/AIRFLOW/Contributor+Guide Please add to it.. also share thoughts on what is unclear or missing. -s

Re: How do you use pools?

Hi Lance! Yes, we do the same. Specifically, we have multiple DAGs that share access to a Spark cluster through the use of Pools. By setting the pool size to say 4, we remove the possibility of some backfill swamping the Spark cluster. BTW, there were some bugs with over-subscription of pools. It's

Fwd: [jira] [Updated] (AIRFLOW-151) trigger_rule='one_success' not allowing tasks downstream of a BranchPythonOperator to be executed

Mentors, Question. Why is that we needed to move to JIRA, yet INFRA has allowed for the creation (but not closing) of Github issues in Github? There is no workflow or way for us to close accidently created GitHub issues. -s -- Forwarded message -- From: Siddharth Anand (JIRA

Re: Turning off GH issues

Yes..pls.. I just complained about this as well on the github list. On Fri, May 20, 2016 at 3:32 AM, Chris Riccomini wrote: > Hey all, > > If no one objects, I'll file an INFRA ticket to turn off the "issues" tab > on our GitHub page. I've seen a number of folks open issues on here (it's > very

Apache Airflow Announcement : New Committer (Steven Yvinec-Kruyk)

Folks, We are happy to announce that Steven Yvinec-Kruyk (syvineckruyk ) joined the Apache Airflow Committer and PPMC group today. Please give him a hearty welcome and of course.. ask him to review you

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

PM, Jakob Homan wrote: > On 18 May 2016 at 16:06, siddharth anand wrote: > > Is there an automation we can use to ensure/enforce that PRs > > and JIRAs receive the +1 binding (from committers) votes needed before a > > merge can proceed > > Not that I know of. Since w

Re: MODERATE for dev@airflow.incubator.apache.org

Pavlo, Please follow the instructions in the Airflow Contributor Guide to create a mailing list account on dev and commits. This will avoid us from having to accept/reject your individual emails. When you email dev-subscr...@

Announcing the long awaited 1.7.1 Release

Hi Folks! Release 1.7.1 is out with large bunch of stability fixes & new features - 214 commits to be precise as per CHANGELOG.txt You can install via Pypi or via Git tag

Re: Turning off GH issues

1:22 AM, Chris Riccomini > wrote: > > > Doing this now. > > > > On Fri, May 20, 2016 at 9:27 AM, Maxime Beauchemin < > > maximebeauche...@gmail.com> wrote: > > > >> +1 > >> > >> On Thu, May 19, 2016 at 8:36 PM, siddharth anand > &g

Re: Announcing the long awaited 1.7.1 Release

d validation on their clusters! Very excited to try this out. > > > > Also, when should we start on the next release? This one took so long > that > > there's actually a fair amount of stuff already on master that's not in > > 1.7.1. Plus, if we expect 3-5 RCs, it

Contributors can now self-assign JIRAs

Contributors, Please feel free to assign JIRAs to yourselves. This has not been possible in the past due to some incorrect JIRA permissions, but that is now fixed . 70% of our issues were left unassigned due to this as shown

Re: Announcing the long awaited 1.7.1 Release

fair amount of stuff already on master that's not in > > 1.7.1. Plus, if we expect 3-5 RCs, it might be better to start earlier > > rather than later. > > > > On Fri, May 20, 2016 at 11:23 AM, siddharth anand > wrote: > > > >> Hi Folks! > >&

[VOTE] Review-then-commit (RTC) vs Commit-then-review (CTR)

Hi Folks! To formalize the RTC vs CTR policy going forward for committing, I'm putting it to a vote. "Commits need a +1 vote from a committer who is not the author with the exception of minor commits" Anyone can vote, but committer/PPMC votes are binding. http://www.apache.org/foundation/voting

Re: [VOTE] Review-then-commit (RTC) vs Commit-then-review (CTR)

+1 (binding) : FYI, Chris was binding as well. -s On Tue, May 24, 2016 at 3:00 PM, Chris Riccomini wrote: > +1 > > On Tue, May 24, 2016 at 12:50 PM, siddharth anand > wrote: > > > Hi Folks! > > To formalize the RTC vs CTR policy going forward for committing, I

Re: [VOTE] Review-then-commit (RTC) vs Commit-then-review (CTR)

Tue, May 24, 2016 at 2:31 PM, Hitesh Shah wrote: > > > +1 > > > > — Hitesh > > > > > On May 24, 2016, at 12:50 PM, siddharth anand > wrote: > > > > > > Hi Folks! > > > To formalize the RTC vs CTR policy going forward for com

Re: [VOTE] Review-then-commit (RTC) vs Commit-then-review (CTR)

Else continue voting :-) On Tuesday, May 24, 2016, siddharth anand wrote: > Actually, that "minor updates" statement should have been dropped. I'm > surprised no one else noticed it:-) > > "Commits need a +1 vote from a committer who is not the author" is w

Re: Upcoming Podling report, due July 1st

I can. Sent from Sid's iPhone > On May 25, 2016, at 7:18 PM, Jakob Homan wrote: > > Just a reminder that the second Podling report is imminent. After > three monthly reports (this is our second), we move to quarterly > reporting. Anyone want to volunteer? > > http://wiki.apache.org/incubato

[RESULT][VOTE] Review-then-commit (RTC) vs Commit-then-review (CTR)

Folks, Here is the result of the vote for the proposed change to our commit policy: *Proposed Change* : "Commits need a +1 vote from a committer who is not the author" +1 (binding) votes : 8 votes - Chris Riccomini - Max (mistercrunch) - Arthur (artwr) - Dan (aoen) - Bolke - Ja

Re: Upcoming Podling report, due July 1st

Folks, I have updated the June Podling report for Airflow https://wiki.apache.org/incubator/June2016 Please review and share any thoughts before June 1. -s On Wed, May 25, 2016 at 7:09 PM, siddharth anand wrote: > I can. > > Sent from Sid's iPhone > > > On May 25, 2016,

Re: June 14 meetup

Chris! Thanks for putting this together and rallying the troops. Looking forward to what promises to be a great get-together. -s On Fri, May 27, 2016 at 11:01 AM, Chris Riccomini wrote: > Hey all, > > I just wanted to send a reminder that the second Bay Area Airflow meetup is > coming up on Jun

Re: Committers meeting 2016-06-01@11AM PST

+1 and looking forward to it. I am going ahead and publicizing this meeting on twitter as well. -s On Thu, May 26, 2016 at 10:57 AM, Jakob Homan wrote: > This is awesome. Thanks for publicizing it. One last thing with > mentor hat on: we can't call it a committers meeting, since all > contrib

Re: Committers meeting 2016-06-01@11AM PST

Done! - https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-May27,2016 - https://twitter.com/ApacheAirflow/status/736391899318095872 -s On Fri, May 27, 2016 at 7:53 PM, siddharth anand wrote: > +1 and looking forward to it. I am going ahead

Re: Exception on /admin/airflow/clear

Cyril, Can you create a Jira for this? We need a way to reproduce this. Please add your backend db (e.g. sqlite or something else) and an example of your code -- the one you are trying to clear. Then assign the bug to me (r39132). This may be related to the recursive clear of subdags, which was me

Re: Exception on /admin/airflow/clear

Did you > make > a mistake ? > > > On May 27, 2016, at 23:24, siddharth anand wrote: > > > > Cyril, > > Can you create a Jira for this? We need a way to reproduce this. Please > add > > your backend db (e.g. sqlite or something else) and an example of your &

Re: Airflow open space at pycon?

Bence. Anytime you would like to spontaneously organize an Airflow meet-up or gathering, you are most welcome to. I don't think any of the committers are heading to PyCon. We encourage our community to help spread the word about Airflow - so, please do not be shy about presenting on the topic. If

Re: Upcoming Podling report, due July 1st

tion, etc. > > > — Hitesh > > > On May 27, 2016, at 5:35 PM, siddharth anand wrote: > > > > Folks, > > I have updated the June Podling report for Airflow > > https://wiki.apache.org/incubator/June2016 > > > > Please review and share any thoughts before Ju

Re: High res Airflow image?

LOL. Yeah, Chris and I were talking about how awesome it would be to have the hosting (meet-up) company print off shirts that say things like "Company X loves Apache Airflow". Also, one thing we are trying out in this upcoming meet-up and should continue is that there are 3 speakers : 1 from the h

Re: Upcoming Podling report, due July 1st

gt; A lot of those tasks are specific to the Champion. Also it's part of > > the Incubator infrastucture, which generally only mentors and champion > > have access to. > > > > On 31 May 2016 at 15:39, siddharth anand wrote: > > > Any takers? > > >

Airflow Contributors Meeting (June 01, 2016) : Minutes

Hi Folks! We held our first contributor meeting this morning. I was about 20 minutes late, but did ask others in attendance for their input before compiling these minutes. *Agenda* : https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-May27,2016 *Outcomes*: - We n

We need a homepage

Maybe something as simple as what arrow has : https://arrow.apache.org/ It is the only remaining item from our migration effort : https://cwiki.apache.org/confluence/display/AIRFLOW/Migrating+to+Apache Maxime, You have that item. Would you like to hand it off? -s

Re: Unit testing dags

We were discussing this today. There is a framework that you might be able to use to test your Dags. In addition, we would like to build a corpus of sample dags that we run this framework against (once a day) to catch regressions. Max, What do you want to do with the End-to-end DAG testing framewo

Re: We need a homepage

row. There's a JIRA for this ( > > https://issues.apache.org/jira/browse/AIRFLOW-9) but it's attached to > > Maxime right now. I'm going to unassign it in case others want to get > > involved. Max, feel free to re-grab it if you're working on it. > &

Re: Unit testing dags

; > Cheers, > Thoralf > > ___ > > *Thoralf Gutierrez* > > *Real**Impact* Analytics *| *Brussels Office > www.realimpactanalytics.com > > *BE *+32 477 68 94 65 *|* *Skype* thoralf.gutierrez > > On Thu, Jun 2, 2016 at 2:33 AM, siddharth anand wrote: > > >

ASF and Neutrality : A good read if you are not familiar with ASF

http://www.infoworld.com/article/3079813/open-source-tools/the-apache-foundations-incredible-rise.html

Re: Upgrade from 1.7.0 to 1.7.1.2

It's always recommended to run upgradedb. -s On Fri, Jun 10, 2016 at 2:40 AM, Sumit Maheshwari wrote: > afaik you have to run "airflow upgradedb" command, so any new migrations > will be applied. > > > On Fri, Jun 10, 2016 at 12:35 PM, Jason Chen > wrote: > > > Hi, > > > > I upgraded airflow f

Re: Streaming Airflow meetup

Thx Chris : Have also tweeted it via the Airflow twitter channel & added it to the announcements wiki : https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-June15,2016 On Wed, Jun 15, 2016 at 3:58 PM, Chris

Re: Airflow 1.7.1.3 UI status not corect

Resending with links since the apache mail server does not allow emails larger than ~1MB. -s On Mon, Jun 27, 2016 at 7:13 PM, siddharth anand wrote: > I found this to also be a little jarring at first, but the functionality > is an arguable improvement. > > For the benefit of ot

Re: Deadlocked Dag

Open a JIRA and provide details such as version of Airflow that you are running. Also, are you running on UTC everywhere? Also, please set up requisite accounts (including subscribing to the mailing lists so that admins don't need to moderate your emails) : https://cwiki.apache.org/confluence/disp

Re: July 2016 Podling Report -- Draft

Many thanks Gurer! I noticed that we reported adding a committer in the July Report that we actually added and mentioned in the June Report . As a matter of process, it's worth calling out the names of the com

Re: Aiming for an Apache release 1st week of September?

+1 On Thu, Jul 14, 2016 at 4:42 PM, Chris Riccomini wrote: > (might want to have a look at > http://incubator.apache.org/guides/releasemanagement.html as well) > > On Thu, Jul 14, 2016 at 4:41 PM, Chris Riccomini > wrote: > > @bolke, is the release ready for an RC? If so, I can ship to our > >

Re: execution_timeout ignored for PythonOperator in 1.7.1.2?

Alyssa, Are you running UTC everywhere? That is still an unfortunate requirement in some parts of airflow. I only ask because this is a common stumbling point -- one I have debugged for other before. You need to run your DB and your airflow servers in UTC. -s On Thu, Jun 9, 2016 at 9:03 AM, Alys

Re: Juggling with Github and JIRA

+1 from me. I don't think anyone feels Jira is better than GH Issues. What are next steps? -s On Fri, Jun 10, 2016 at 11:37 AM, Bolke de Bruin wrote: > +1 > > What Github provides is basically the review board of Apache, but then a > lot better. > > > > > > Op 10 jun. 2016, om 17:51 heeft Chris

Calling Airflow Enthusiasts and Users

Hi Folks! I will be speaking this coming Saturday at DataDaySeattle on Airflow and our use of it at Agari. I'd love to mention use-cases from the community as well. If you'd like to share your use-case, feel free to email me at r39...@apache.org, and I'd be ha

Introducing Apache Airflow : DataDaySeattle

http://bit.ly/2a2e9vw Link to slides from Data Day Seattle -s -- Sent from Gmail Mobile

Question for mentors : podling report frequency

Mentors, Now that we have filed 3 monthly podling reports, what is the expected cadence of our podling reports going forward? -s

Re: Upgrading to 1.7.1.2...

It would be good to split release notes into separate sections of fixes and features. I recall that we generate the release notes from a git log command. We might want to make that script a little smarter - e.g. separate commits into the features and fixes buckets. We are committing a lot of fixes

Re: 1.7.1.2 got broken, 1.7.1.3 is out

FYI, 1.4.0 does not in fact work for us, but 1.4.1 works on 1.7.1.3. It's weird that we are able to run 1.7.1.3 on 1.4.1. We are building using setup.py from the Git repo. -s On Mon, Jun 13, 2016 at 3:33 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hi there, > > The new flask-a

Re: Projects using GitHub issues

Mentors, Any updates here? I looked at the thread that Chris R had pasted to kick off this discussion. The thread meandered. I saw a comment from Jim J stating : Recall that issue trackers have not been, traditionally, held under the requirement of being under ASF infra nor infra control. If peopl

Re: DAG without a schedule interval?

As Max mentioned, you can write : dag = DAG('ep_reload_data', schedule_interval=None, default_args=default_args) -s On Fri, Jun 10, 2016 at 8:37 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > schedule_interval=None is supported and works nicely as an externally > triggered, "on d

Re: DAG scheduled for start_date of today and an interval of 7 days keeps getting scheduled for the past

The problem might be that the start_date does not get updated. I work around this by changing the name of my dag. I do lose history as well, but it works. My dags are named "some_dag_v1". When I change a start date, I update the version suffix to force a reload : "some_dag_v2" -s On Tue, Aug 2,

Re: DAG scheduled for start_date of today and an interval of 7 days keeps getting scheduled for the past

ed for the past before any dags or > start_dates existed. > > Cheers, > David > > On Tue, Aug 2, 2016 at 7:10 PM, siddharth anand wrote: > > > The problem might be that the start_date does not get updated. I work > > around this by changing the name of my dag. I d

Re: What is the best way to retry an entire DAG instead of just a single task?

Hi Arthur, It's not a dumb question. We don't have the ability to retry a DAG based on a task failure in a programatic way to the best of my knowledge. Also, we don't allow cyclic dependencies.. hence the Acyclic part of DAG. TriggerDagRunOperator won't work because the execution is async. The TDR

Re: DAG status still running when all its tasks are complete

Hi Nadeem, Can you open a JIRA, attach a DAG which I can run to reproduce your issue, and assign the JIRA to me? -s On Tue, Aug 2, 2016 at 8:40 PM, Nadeem Ahmed Nazeer wrote: > Could someone please shed some light on this DAG status? > > My airflow version is 1.7.0. This is the only version that

Re: Run input field in Gantt page

This makes sense. Go ahead and submit a PR if you haven't already and ping me to review it. -s On Fri, Jul 15, 2016 at 7:37 PM, wood stock wrote: > Hi All, > > My first post. I installed the latest release with "pip install airflow" > and it is running. Initial impression is very good. > > Quest

Re: Running a task from the Airflow UI

A REST api is long overdue. I suggest anyone in the community that has the cycles to start implementing.. your PRs would be welcome. Currently, we have a very powerful CLI that should ideally have similar functionality exposed via the API. The CLI's trigger_dag command is one of the first ones I'd

Re: Creating and accessing different variable values for each instance of DAG run

+1 for Laura's suggestion. We (Agari) also find ourselves in situations where dag runs end up overlapping. We pass the execution_date via Jinja templates to BashOperator to remote shell commands to Spark jobs to the final Spark output files on S3. The execution date is what ties the DAG Run to the

Re: Airflow SLA shows tasks that doesn't miss SLA

We have been using run the SLA_miss_callback feature for over 6 months now... for our hourly DAGs. We tend to miss SLAs (and rightly so) about 20% of the time. We haven't seen this bug. We are also running on the hour boundaries and not at 59 minutes from the hour. Is there are reason your start_da

Re: Handling running tasks

+1 on the uniqueness of the solution. Wondering how it worked! -s On Sun, Jul 10, 2016 at 9:02 AM, Cyril Scetbon wrote: > Interesting. Thanks for this solution Lance, gonna try it > > On Jul 6, 2016, at 19:11, Lance Norskog wrote: > > > > You could use the XCOM feature to post a semaphore at t

Re: "Run" pulldown menu in TreeView, TaskDuration, LandingTimes

That sounds correct to me. Is there a PR? -s On Sat, Jul 16, 2016 at 12:40 AM, wood stock wrote: > Hi, > I found an issue with the "Number of runs:" pulldown menu. Looks like we > intend to retrieve task instances for the last X runs until the "Base". > However, the logic has a flaw IMO. > > It

Re: Airflow Meetup at Stripe in September?

Jeff, That sounds great. We encourage any and all in the community to host Airflow meet-ups. We recommend the following meet-up format: https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups Though the format is a suggestion, we do prefer you shoot for 1 host speaker and 2 guest speakers.

Re: Airflow Meetup at Stripe in September?

You can also ask on this list if there are folks interested in speaking. Also, should you folks ever want to speak in other meet-ups too, it's fully welcome. -s On Thu, Aug 4, 2016 at 7:39 PM, Siddharth Anand wrote: > Jeff, > That sounds great. We encourage any and all in the c

Re: airflow initdb failed when dag file reference undefined Variables

Seems like a very reasonable question and a bug - feel free to file a bug (and fix it :-). The loading of a dag_bag happens in a few places and one of them seems to be getting triggered in this case. -s On Wed, Aug 3, 2016 at 12:58 AM, Wang Yajun wrote: > Hi > > the airflow initdb failed if dag

Does your company use Airflow?

Folks! Does your company use Airflow? If so, add your company (along with your GitHub handle) as links to https://github.com/apache/incubator-airflow/blob/master/README.md Then submit a Pull Request (and open a Jira as well) as per the Airflow Contributors' Guide

Re: Long pending PR

Looking at it now! -s On Wed, Aug 10, 2016 at 10:59 PM, Sumit Maheshwari wrote: > Hello guys, > > There has been a PR long pending for review, can some one please take a > look on it and merge it? > > https://github.com/apache/incubator-airflow/pull/1682/files > > There is a very small change

Re: Airflow Meetup at Stripe in September?

Thu, Aug 4, 2016 at 10:56 PM, Sumit Maheshwari wrote: > Hi Jeff, > > If possible I would also love to give a small talk remotely. > > > Thanks, > Sumit > > On Fri, Aug 5, 2016 at 10:39 AM, siddharth anand > wrote: > > > You can also ask on this list if t

Re: The default variable {{ var.value.my_var }} don't exists!

Please file a bug and provide an example (as you did above!). Also, you are free to assign the JIRA to yourself and submit a PR with the fix. -s On Wed, Aug 3, 2016 at 12:40 AM, Wang Yajun wrote: > Hi > > it seems that the default variable {{ var.value.my_var }} described in this > wiki

Shout out!

FYI! Just wanted to give a special shout-out for jlowin for writing a great merge tool for committers. Thx to this tool, merging your PR is super easy. -s

Re: Airflow Meetup at Stripe in September?

b613> > @apigee > <http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nM > JW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=https% > 3A%2F%2Ftwitter.com%2Fapigee&si=5141814536306688&pi= > de892aba-bd1f-4c0e-948f-454ffe35b613> > > > On Wed, Aug 10, 2016 at 11:20 PM, siddharth anand &g

Re: Airflow Meetup at Stripe in September?

w-dev/201608.mbox/%3CCAEdoVMW7cTyAQ%3Da%2Bwy%3DN7v1Cq0oYVbCwYG%2BN% > 2B8wuAAQR3KYH3A%40mail.gmail.com%3E > > On Thu, Aug 11, 2016 at 9:45 PM, siddharth anand > wrote: > > Haven't heard back on this thread. Ben, interested in hosting? Where is > > apigee? > > &

Re: Shout out!

ug 12, 2016 at 9:46 AM, Dan Davydov invalid > > wrote: > > > +1 > > > > On Aug 12, 2016 8:02 AM, "Chris Riccomini" > wrote: > > > > > Same. It's awesome. > > > > > > On Thu, Aug 11, 2016 at 7:28 PM, siddharth anand &

  1   2   3   4   >