Re: Release Manager for 1.8.2?

Please also include https://github.com/apache/incubator-airflow/pull/2022 I have had reports it was also required for MySQL. Bolke Sent from my iPhone > On 22 Jun 2017, at 00:03, Maxime Beauchemin > wrote: > > Ok I'll try to cherry-pick all of these, though if there are any merge > conflict

Re: Airflow Logging Improvements

Responding to some of Bolke's concerns in the github PR for this change: > Mmm still not convinced. Especially on elastic search it is just easier to use the start_date to shard on. sharding on start_date isn't great because there is still some risk of collisions and it means that we are coupling

Airflow Logging Improvements

Hi, I am in the process of making airflow logging backed by Elasticsearch (more detail please check AIRFLOW-1325 ). Here are several more logging improvements we are considering: *1. Log streaming.* Auto-refresh the logs if tasks are running. *

Re: Release Manager for 1.8.2?

Ok I'll try to cherry-pick all of these, though if there are any merge conflicts I'll pass and move their Fix-Version to 1.9.0. Max On Wed, Jun 21, 2017 at 1:20 AM, Bolke de Bruin wrote: > Hi max, > > PR 2365 is merged into master and also into v1-8-test. > > I have added: > > https://issues.ap

Re: EMROperator templating

Actually it looks like while we allow for dicts and lists in template_fields, we won't recurse through them: https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L2318 Since it's in BaseOperator you could override to render_template_from_field recurse through dicts and lists i

Re: airflow with MSSQL db

I've been running it for a few months without issue. I'm pretty sure you'll need to make sure to include this commit if it hasn't made it onto 1.8 yet https://github.com/apache/incubator-airflow/pull/2022/commits/702d3abd0d84773ca27f608d26b1f7243abc7336 On Wed, Jun 21, 2017 at 12:09 PM, Bolke d

Re: airflow with MSSQL db

It is not officially supported but I know of some that run airflow on top of mssql. We do integrate fixes for it and did so in the past. The sql we run is not overly complex, so in general you should be fine. I would stress test it with a couple of examples and run the tests against it. Bolke

airflow with MSSQL db

Hello, Is MSSQL server supported as airflow metadata DB host? Have anybody tried to set it up? Any concerns? Thanks mC Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII W

Re: Release Manager for 1.8.2?

Hi max, PR 2365 is merged into master and also into v1-8-test. I have added: https://issues.apache.org/jira/browse/AIRFLOW-935 and https://issues.apache.org/jira/browse/AIRFLOW-860 To the l

Re: EMROperator templating

I already tried that, does not work. Seems that attribute job_flow_overrides http://boto3.readthedocs.io/en/latest/reference/services/emr.html#EMR.Client.run_job_flow I am not sure that jinja template can handle something like : EmrCreateJobFlowOperator(job_flow_overrides={ ... 'Steps': [