I think we have similar understanding. But let me just clarify because I think we think about we think about solving two different problems My proposal is not solving all problems with dependencies - quite the contrary, I want to solve just one specific "repeatability" problem - read on :)..
1. A potential source of confusion: using "-pinned" for installation but > using "non-pinned" for DAG development. > This could be confusing indeed - but they are the same in fact - just deps might be different over time. 2. Most of the users would still try to install "apache-airflow" package > that might have been broken for example because of a dependency release, > either way, we would still have to suggest them to use "pinned" version > True. I thought we might describe it in the README and make it prominently explained. Usually people look at the readme in PyPI when they are installing stuff and it does not work: https://pypi.org/project/apache-airflow/. Also - we could of course explain how to use requirements.txt from the released version when they are installing it. That would be an extra friction point though and maybe having "always installable" version of airflow is a better choice. 3. If they install "pinned" version, it is no longer a library again, > that is users won't be able to use new NumPy release or matplotlib for > example. In which case we are just circling back to the same problem, > "either we risk broken package" while releasing or we risk potentially > incompatible versions. > Yep. But maybe it's just a question of naming. Maybe even we could name this package differently to indicate that this version is a way to quickly install airflow but not to do any serious development with it. So speaking about THE problem I want to solve with the requirements.txt and apache-airflow-pinned package: I really only want to solve "first-time-user" experience here - nothing more. I definitely do not want to replace the current installation method for experienced users - for them using --constraint requirements.txt is exactly what they need. The only problem I am trying to solve with that is "repeatability" of installation. Maybe "apache-airflow-quickinstall" or something like that would be better than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or something like that. I think about it as a "flavour" of ariflow rather than anything else. I even originally implemented it as [pinned] extra where I pinned all requirements. Unfortunately I found that if you have main requirement without limits, adding the same requirement as extra with == does not make it pinned :(. That was my original plan. > Btw I have been on "we should have pinned dependency" camp as Airflow > should definitely install without breaking since day-1 but I think a > separate "-pinned" package won't solve that issue. > Ah yeah we went the same route. I do not think we can solve the "library vs. app" problem easily. This is a bit of "eat-and-have-cake" at the same time. I know people have problems with conflicting dependencies when they are trying to install libraries with different requirements. And I am not even trying to solve that problem now. Not even close. This requires some other solution (for example separate virtualenvs with different dependencies build from wheels on per-task basis). But that's something much further in the future (if at all). > > WDYT? Also please do let me know if I have misunderstood something > (definitely possible :D). > > Regards, > Kaxil
