Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread James Meickle
I suggest not adopting pipenv. It has a nice "first five minutes" demo but it's simply not baked enough to depend on as a swap in pip replacement. We are in the process of removing it after finding several serious bugs in our POC of it. On Thu, Oct 4, 2018, 20:30 Alex Guziel wrote: > FWIW,

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Alex Guziel
FWIW, there's some value in using virtualenv with Docker to isolate yourself from your system's Python. It's worth noting that requirements files can link other requirements files, so that would make groups easier, but not that pip in one run has no guarantee of transitive dependencies not

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Driesprong, Fokko
Hi Jarek, Thanks for bringing this up. I missed the discussion on Slack since I'm on holiday, but I saw the thread and it was way too interesting, and therefore this email :) This is actually something that we need to address asap. Like you mention, we saw it earlier that specific transient

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Arthur Wiedmer
Thanks Jakob! I think that this is a huge risk of Slack. I am not against Slack as a support channel, but it is a slippery slope to have more and more decisions/conversations happening there, contrary to what we hope to achieve with the ASF. When we are starting to discuss issues of development,

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Jarek Potiuk
Thanks for pointing it out Jakob. I am still very fresh in the ASF community and learning the ropes and etiquette and code of conduct. Apologies for my ignorance. I re-read the conduct and FAQ now again - with more understanding and will pay more attention to wording in the future. As you

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Alex Guziel
You should run `pip check` to ensure no conflicts. Pip does not do this on its own. On Thu, Oct 4, 2018 at 9:20 AM Jarek Potiuk wrote: > Great that this discussion already happened :). Lots of useful things in > it. And yes - it means pinning in requirement.txt - this is how pip-tools > work. >

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Jarek Potiuk
Great that this discussion already happened :). Lots of useful things in it. And yes - it means pinning in requirement.txt - this is how pip-tools work. J. Principal Software Engineer Phone: +48660796129 On Thu, 4 Oct 2018, 18:14 Arthur Wiedmer, wrote: > Hi Jarek, > > I will +1 the discussion

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Arthur Wiedmer
Hi Jarek, I will +1 the discussion Dan is referring to and George's advice. I just want to double check we are talking about pinning in requirements.txt only. This offers the ability to pip install -r requirements.txt pip install --no-deps airflow For a guaranteed install which works. Several

Re: Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

2018-10-04 Thread Kyle Hamlin
If I remove the Flask-AppBuild pinning to 1.11.0 then it uncovers a Jinja2 conflict which is baffling because I don't see anywhere in the graph that jinja2 >=2.10 is required. Could not find a version that matches jinja2<2.9.0,>=2.10,>=2.4,>=2.5,>=2.7.3,>=2.8 Tried: 2.0, 2.1, 2.1.1, 2.2, 2.2.1,

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Dan Davydov
Relevant discussion about this: https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174 On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk wrote: > TL;DR; A change is coming in the way how dependencies/requirements are > specified for Apache Airflow - they will be fixed rather

Re: Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

2018-10-04 Thread Kyle Hamlin
Thank you for the response Ash. Even with your suggestion, there appear to be version conflicts all over the place. Can you get this Pipfile to install because I cannot? *Pipfile:* [[source]] url = "https://pypi.python.org/simple; [[source]] verify_ssl = true name = "pypi" [packages]

Re: Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

2018-10-04 Thread Kyle Hamlin
whoops remove the [[source]] at the end of the url = " https://pypi.python.org/simple; that is a typo. On Thu, Oct 4, 2018 at 11:26 AM Kyle Hamlin wrote: > Thank you for the response Ash. > > Even with your suggestion, there appear to be version conflicts all over > the place. Can you get this

Pinning dependencies for Apache Airflow

2018-10-04 Thread Jarek Potiuk
TL;DR; A change is coming in the way how dependencies/requirements are specified for Apache Airflow - they will be fixed rather than flexible (== rather than >=). This is follow up after Slack discussion we had with Ash and Kaxil - summarising what we propose we'll do. *Problem:* During last few

Manual validation operator

2018-10-04 Thread Björn Pollex
Hi all, In some of our workflows we require a manual validation step, where some generated data has to be reviewed by an authorised person before the workflow can continue. We currently model this by using a custom dummy operator that always fails. After the review, we manually mark it as

Re: Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

2018-10-04 Thread Ash Berlin-Taylor
We've committed a fix for this to master and will include it in a 1.10.1 https://github.com/apache/incubator-airflow/commit/fb5ffd146a5a33820cfa7541e5ce09098f3d541a For installing in the mea time pin `Flask-AppBuilder=1.11.0' > On 4 Oct 2018, at 00:41, Kyle Hamlin wrote: > > Hi, > > Today I

Re: Slides from London Airflow Meetup #1

2018-10-04 Thread Deng Xiaodong
Definitely there is ;-) Such slides & videos would be great supplement on the top of documentation. XD On Tue, Oct 2, 2018 at 17:01 Ash Berlin-Taylor wrote: > Is there demand for me to give a re-run of this talk (recorded this time), > probably over a Google Hangout or a YouTube stream? > >

Log all output of the S3 file transform script

2018-10-04 Thread Szymon Biliński
I've opened an Issue/PR a while back and could use some help getting it out the door: https://github.com/apache/incubator-airflow/pull/3914. It's a small change (I think), but I it would be useful for us, as we're currently reimplementing the same operator for this exact purpose. Some code