Re: We've migrated to Github to repo!

We should ask Apache infra to send the GH notifs to another mailing list. Over at jclouds, we created a "notifications@" list for this purpose (well, actually we renamed "issues@" to "notifications@"), and send messages there: https://issues.apache.org/jira/browse/INFRA-7180 https://mail-arc

Re: singing to mailing list

Hi Martin How can I sign in for the mailing list? Try sending an email to "dev-subscribe@..." - you shouldn't need an apache.org email address to do that. See http://apache.org/foundation/mailinglists.html Regards ap

Re: programatically configure aws_default, emr_default connections?

Is there any way to programatically configure the aws_default, emr_default or other airflow connections? In a pre-1.8.x installation we used the following script: --- snip --- #!/usr/bin/python import json from airflow import models from airflow.utils import db as db_utils CONNECTIONS = [

Re: Mesos Marathon support in Airflow

Hi Niranda I know Airflow currently supports Mesos executor, but may I know if it supports Mesos Marathon? The idea is to use Airflow to run task as a container orchestrator. Could you explain in a bit more detail what you mean by "Airflow supporting Marathon"? Airflow certainly runs *inside*

Re: Removing members from dev list?

Hi Siddarth How do we (PMC) remove a email recipient from the dev list? Anyone who is a moderator of the list should be able to request removal of a subscriber by sending an email to [1]: {listname}-unsubscribe-badboy=menace@tlp.apache.org I.e. in this case dev-unsubscribe-kerzhner=ya

Re: Scheduler silently dies

We use celery and run into it from time to time. Bang goes my theory ;-) At least, assuming it's the same underlying cause... Regards ap

Re: Scheduler silently dies

Does anyone have any idea why this happens? It seems like a bug that should be fixed, but we're all just living with it instead of trying to fix it. From the little I understand, one of the main problems here is that it seems very difficult to reliably reproduce the issue. There are a bunch o

Re: NYC Meetup?

We at Blue Apron would be very interested. Same here. ap

Re: Merging the experimental API Framework

Just wanted to say this is very exciting, thank you Bolke :). Big +1 to that. Thanks, Bolke! ap

Re: String formatting

My understanding from the latest python documentation is that '.format' supersedes the use of '%'. Therefore, it seemed a little strange that the Landscape configuration was advising the use of '%', I think the thing here is that the "%" that the warning is talking about is *not* the same oper

Re: String formatting

Perhaps I stand corrected! -- though I don't see where it actually says this approach is preferred. In any case, the Python 3 docs explicitly state that the behavior is only maintained for backwards compatibility: https://docs.python.org/3/howto/logging.html#logging-variable-data Ah, interesti

Re: String formatting

"Use % formatting in logging functions and pass the % parameters as arguments" [...] Can anybody give a judgement on this? If the .format is preferred, then we'll look into changing the landscape.io settings. As far as I can tell from the Python 2 logging docs [1] at least, the following is

Re: Custom (mongodb) hook as a plugin

6. Went to UI/Connections but couldn't find my connector/hook The experts on the list will hopefully set me right I'm my understanding is incorrect, but from what I can see the list of hook types available *in the UI* is fixed: https://github.com/apache/incubator-airflow/blob/f360414774f1ecb

Re: Custom Hook development guideline

Are the above steps correct or am I missing something? Is something not working as expected, e.g. are you unable to *use* the plugin from your DAGs? If so, could you provide a few more details on the error? Regards ap

Re: Scheduler getting stuck - request for details

First and foremost I am assuming that getting “stuck” is only happening when using a CeleryExecutor. We have seen repeated instanced of the scheduler "dying" - i.e. no more scheduler threads in a ps output - with LocalExecutor too. If you feel this fits the description of "getting stuck", happ

Re: airflow is stuck

After a few hours, we noticed that no task is executed. We've run into a similar situation, which may or may not be related. In our case, the scheduler seems to die; there are suddenly no more active scheduler threads. Restarting the scheduler resolves the issue, although we sometimes need

Restarting the scheduler regularly - still current advice?

Hi all I just wanted to check to what extent the advice in [1] and [2], namely to restart the scheduler "every once in a while", is still considered accurate? "Restart your scheduler process to get a clean environment every once in a while. Use --num_runs N scheduler CLI option to make it st

Re: A question for the Airflow community

2. There is a feature 'XCom' which allows you to use the Airflow database as a key-value store. If you wish DAG and task instances to see particular data items, you can store them via 'XCom'. I have not done this. Thanks, Lance! An XCom is indeed what we are using now, as also discussed in [2

Re: A question for the Airflow community

I'd like to ask you all a question in turn: what do you know now that you wish you knew when you first deployed Airflow? Mainly questions around suggestions/best practice for addressing reasonably common (?) integration challenges: 1. Talking to Airflow from remote/external systems (e.g. on a

Re: Suggested way of passing "input parameters" to a DAG run?

On 2016-08-03 18:42, Joseph Napolitano wrote: What I can say is that we use it a lot, but very lightly. We basically use it to communicate the S3 key for a flat file between operators. Just to follow up on this: here's a Gist with some example usage based on the "store in XCom" approach:

Re: Suggested way of passing "input parameters" to a DAG run?

Let me know if that helps, or if I completely misunderstood :) That helps, indeed - thanks, Joe! We were in fact going down exactly this path as an alternative; we were just a bit hesitant to use XComs based on the following comment in the docs [1]: "If it absolutely can’t be avoided, Airflo

Suggested way of passing "input parameters" to a DAG run?

Hi all What is/are the suggested way(s) of passing "input parameters" to a DAG run (adding quotes since, as far as we can tell, that concept doesn't exist natively in Airflow, probably by design)? This would be information that is used by one or multiple operators in a DAG run and that shoul

Re: Running a task from the Airflow UI

A REST api is long overdue. I suggest anyone in the community that has the cycles to start implementing.. your PRs would be welcome. Currently, we have a very powerful CLI that should ideally have similar functionality exposed via the API. On the subject of "exposing the CLI via an API", thi

Re: Airflow REST Api support

variables/get?varName=foo http://host:8088/airflow/dags/trigger?dagId=foo&runId=bar ... Regards ap [1] https://github.com/msoap/shell2http -- Andrew Phillips Apache jclouds

Re: (Recommended) way to define a failure handler that *only* triggers on FAILED, not on UPSTREAM_FAILED?

I'm confused that a failed status on short_circuit doesn't list its associated DagRun as failed. I was under the impression a DagRun was given failed status if any of its tasks failed. Thanks for the comments, Laura! Yes, we were slightly surprised by the effect of the short circuit operator

Fwd: (Recommended) way to define a failure handler that *only* triggers on FAILED, not on UPSTREAM_FAILED?

Hi all Quick follow-up ping here - hoping someone has some thoughts/advice? Regards ap

(Recommended) way to define a failure handler that *only* triggers on FAILED, not on UPSTREAM_FAILED?

start_date=kwargs['execution_date'], end_date=kwargs['execution_date'])[0]... So we were wondering if we've perhaps missed an easier/more recommended way to do this? Thanks for a builing this great app and making it available, of course! Regards ap [1] https://groups.google.com/forum/#!topic/airbnb_airflow/6cLDFHUUzhE -- Andrew Phillips Apache jclouds