Re: Airflow 1.10.2b1 available now on PyPI - PLEASE TEST

2019-01-14 Thread Kaxil Naik
I would suggest if https://github.com/apache/airflow/pull/4513 needs more work and probably more time we can include it in the next release. We can have a patch release 1.10.3 after fixing this issue as currently, it is the only pending issue. Let me know your thoughts on this if you agree I will c

Re: Requesting AIP access

2019-01-14 Thread Tao Feng
James, I added you to the wiki. Please give a try and see if it works for you. Happy AIP! On Mon, Jan 14, 2019 at 9:15 AM James Meickle wrote: > Could I get AIP access for "eronarn"? Thank you! >

Requesting AIP access

2019-01-14 Thread James Meickle
Could I get AIP access for "eronarn"? Thank you!

Re: Further improvements to plugin handling

2019-01-14 Thread Björn Pollex
This is what we’ve been doing internally, and if it becomes the “official” way of doing things, then I’m more than happy. Cheers, Björn > On 14. Jan 2019, at 13:38, Ash Berlin-Taylor wrote: > > Operators and hooks are just normal python modules, and don't actually need > to be define

Re: Further improvements to plugin handling

2019-01-14 Thread Ash Berlin-Taylor
Operators and hooks are just normal python modules, and don't actually need to be defined under `airflow.` to work. My advice (and something I've been meaning to update the docs to say) is to just make your custom hooks/ops a normal class and do from myco.airflow.operators.my_op import MyOp

Re: A DAG with two schedules

2019-01-14 Thread Brian Greene
You could have 2 “trigger dags” on different schedules triggering the one that you’re interested in. If you needed task selection in the target based on schedule, you could branch around tasks based on the DagRun. Sent from a device with less than stellar autocorrect > On Jan 13, 2019, at 9

Further improvements to plugin handling

2019-01-14 Thread Björn Pollex
With the recent (awesome) new feature to [load plugins from entry points][1], I’m wondering if this could be taken further, to make handling plugins simpler and less magic. One pain point for me personally has been that, since plugins are loaded by Airflow magic, my IDE cannot give me any help (

Re: Airflow 1.10.2b1 available now on PyPI - PLEASE TEST

2019-01-14 Thread Kaxil Naik
Done. Merged. On Mon, Jan 14, 2019 at 2:53 AM Deng Xiaodong wrote: > Another improvement we should consider for 1.10.2 in plan: > > Currently our version for "requests" dependency is 'requests>=2.5.1, <3'. > But according to https://nvd.nist.gov/vuln/detail/CVE-2018-18074, the > Requests packag

Re: A DAG with two schedules

2019-01-14 Thread Ash Berlin-Taylor
No, the schedule is a property of the DAG, so you can't. If one schedule is a subset of the other you could look at using the branching operator to "skip" one of the tasks sometimes: https://airflow.apache.org/concepts.html?highlight=branch#branching https://airflow.apache.org/code.html?highligh