Re: Iterating task xcom keys inside template

Thanks Jeremiah! Looking at the code it seems like get_many can return only many xcom values from different task_ids and not different keys from the same task_id. But it doesn't really matter as I would really like to do the iteration somehow inside the template. I feel like building the template

Re: S3 connection

Hey Tyrone- I just set this up on 1.7.1.2 and found the documentation confusing too. Been meaning to improve the documentation. To get S3 logging configured I: (a) Set up an S3Connection (let's call it foo) with only the extra param set to the following json: { "s3_config_file": "/usr/local/

Re: Streaming Airflow meetup

Thx Chris : Have also tweeted it via the Airflow twitter channel & added it to the announcements wiki : https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-June15,2016 On Wed, Jun 15, 2016 at 3:58 PM, Chris

Re: Streaming Airflow meetup

Hey all, The airflow meetup recording is now available here: https://wepayinc.app.box.com/s/hf1chwmthuet29ux2a83f5quc8o5q18k A lot of good conversation on how Airflow is run, how different companies are using it, future roadmap work, etc. Cheers, Chris On Tue, Jun 14, 2016 at 6:39 PM, Chris

Re: S3 connection

@Jeremiah, http://pythonhosted.org/airflow/configuration.html#logs I used to log to s3 in 1.7.0, and my background .aws/credentials would take care of authenticating in the background. Now it appears that I need to set that "remote_log_conn_id" config field in order to continue logging to s3 in 1

running task from web ui fails intermittently with no error message

Hi, I'm occasionally trying to force the final task in a DAG by clicking on the task in the web UI and clicking 'Run' (with 'Force' and 'Ignore Dependencies' selected). The UI shows 'Sent to the message queue, it should start any moment now'. Going back to that task and selecting 'Log' I see so

Re: Getting to an Apache release

May as well. Technically headers have to be added to anything where human creativity was involved (ie, not in generated source code), so one could argue they're not necessary in __init__.pys. A lot of projects will use tools to verify that headers are present in source files, it'll probably be ea

Re: Getting to an Apache release

I can take a stab at writing a script and sending a pull request for #2. Should we add license headers to empty __init__.py as well? Shall I create a JIRA for it(I couldn't find an already existing one)? On Wed, Jun 15, 2016 at 1:17 AM, Jeremiah Lowin wrote: > Related to #2, I am dusting off AIR

Re: Running on master

Hey Lance, Thanks for the suggestions. A variation of your idea, which was suggested at the meetup yesterday was to snapshot the DB after a "successful" run of the integration tests. Then, on future runs, we can snapshot the DB again, and check that the past snapshot "matches" the current snapshot

Re: Iterating task xcom keys inside template

Hi Or, There is support for pulling multiple XComs at once (see XCom.get_many()) but it won't be directly available inside a jinja template, where you you are typically calling ti.xcom_pull() and returning a single XCom at a time, using a key you know in advance. I would suggest that you do this

Iterating task xcom keys inside template

Hi all, For the purpose of data validation tests I have two adjacent operators: Python - Parsing predefined json of tests and saves the result as an xcom where each key is a test id. Email - I'd like to be able to iterate over all keys from the previous task inside the predefined email template s