I understand what Ash and Jarek are saying. I actually use airflow with custom plugins to have a end-product that fully satisfy our needs and when writing new hooks and operators I don't want to see "airflow uses requirement foo=1 but you have foo=2" - but actually that sometimes also just works. So why not let the developer decide if he want to risk breaking it? I like the idea of firstly defining groups for core and non-core dependencies. I think only need to manage the core ones to always work will be a lot easier. So I would suggest for core ones we have ranges including minor releases and non-core ones getting pinned by default installation (and ci).
I think the more dependencies we add (which comes automatically as airflow is growing) the more we are willing to set to more specific version ranges and maybe even pin them at the end. Felix Sent from ProtonMail mobile -------- Original Message -------- On Aug 2, 2019, 06:10, Jarek Potiuk wrote: > Ash is totally right - that's exactly the difficulty we face. Airflow is > both a library and end product and this makes the usual advice (pin if you > are end-product, don't pin if you are library) not really useful. From the > very beginning of my adventures with Airflow I was for pinning of > everything (and using dependabot or similar - I use it for other projects), > but over time I realised that this is very short-sighted approach. it does > not take into account the "library" point of view.. Depending which user of > airflow you are, you have contradicting requirements. If you are user of > airflow who just want to use it as "end product" you want pinning. If you > want to develop your own operators or extend existing ones - you use > airflow as "library" and you do not want pinning. > > I also proposed at the beginning of that thread that we split core > requirements (and pin it) and non-core ones (and don't pin it). But it > ain't easy to separate those two sets in a clear way unfortunately. > > That's why the idea of choosing at the installation time (and not at build > time) whether you want to install "loose" or "frozen" dependencies is so > appealing. > Possibly the best solution could be that you 'pip install airflow' and you > get the pinned versions and then some other way to get the loose one. But I > think we are a bit on the mercy of pip - this does not seem to be possible. > > Then - it looks like using extras to add "pinning" mechanism is the next > best idea. > > I am not afraid about complexity. We can fully automate generating those > pinned requirements. I already have some ideas how we can make sure that we > keep those requirements in sync while developing and how they can end up > frozen in release. I would like to run a POC on that but in short it is > another "by-product" of the CI image we have now. Our CI image is the > perfect source of frozen requirements - we know those requirements in CI > image are ok and we can use them to generate the standard > "requirements.txt" file and keep it updated via some local update script > (and pre-commit hooks) + we can verify that they are updated in the CI. We > can then write custom setup.py that will use that requirements.txt and the > existing "extras" and generate "pinned" extras automatically. That sounds > like fully doable and with very limited maintenance effort. > > J. > > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <q...@scribd.com> > wrote: > >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <cix...@gmail.com> >> wrote: >> > It is sometimes hard to distinguish if it is a library or an application. >> > Take operator as an example, a non-tech people may think it as a >> well-built >> > application while an engineer may consider it as a library and extends >> > functionalities on it. >> > >> >> Yeah, I agree. Personally, I would consider operator to be a library >> due to the expectation that other people will import them in their own >> projects/source tree. >> >> For things like REST endpoint handlers and perhaps scheduler, it seems >> safe to assume all changes and improvements will happen within Airflow >> source tree. In that case, it's safe to classify that part of code as >> application and freeze all its dependencies. The current plugin system >> might make this slightly complicated because people can still extend >> the core with custom code. But again, in an ideal world, plugins >> should be self-contained and communicating with core through a well >> defined interface ;) >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: [+48 660 796 129](tel:+48660796129) <[+48660796129](tel:+48660796129)> > [image: Polidea] <https://www.polidea.com/>
