Generally, I like the idea of having such a tool, but we need to make it pretty general so that when such a tight dependency issue comes up, the tool should just be able to pull it off by building the rest of the ecosystem while keeping "constraints".
Btw, I think this issue is also for intel machines and not just ARM installations. Ref: https://github.com/cython/cython/issues/5541#issuecomment-1641654183 Also, not sure if this can be fixed by constraining Cython, see thread: https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833/21 Thanks & Regards, Amogh Desai On Mon, Jul 31, 2023 at 3:57 PM Jarek Potiuk <[email protected]> wrote: > BTW. I will run the consensus question until Thursday 3rd of August > 2023 1pm noon. > > On Mon, Jul 31, 2023 at 12:20 PM Jarek Potiuk <[email protected]> wrote: > > > > Hello everyone, > > > > TL;DR; I have a small proposal - I would like to fix constraints for > > all Airflow 2.5.* and 2.6* by updating pymssql to (released today) > > 2.2.8 version from 2.2.7. > > > > I still have to wait for the "complete" release. They have not yet > > released linux binary variants of the packages for 2.2.8 and people > > watching it flagged it to the maintainers. but I wanted to get > > consensus on it before I start doing it. > > > > Currently users installing MSSQL provider for their ARM-based airflow > > are experiencing "build failed" when pymssql is installed. They have > > to use a workaround described here - > > https://github.com/apache/airflow/issues/32672#issuecomment-1647007726 > > and the proposal aims to fix it so that the workaround will not be > > needed when using constraints. There are already few issues about it > > in our repo: > > > > This is one of the extremely rare cases (happened already 2 times over > > last 2 years) where our "reproducible installation" stopped working > > for Python versions - because of the `pip` tooling update that we have > > no control over, but thanks to ability of updating constraints, we can > > fix it by updating constraints. > > > > If we get consensus I will use that opportunity to add some tooling to > > make it easier to do such updates in the future - it requires creating > > new branch for every versio and moving constraint tags - but this is > > easy to automate. And I will have an excuse to develop a small tool do > > help with that that - which we will be able to use in the future in > > simillar cases (I've done it manually before). > > > > Some more context: > > > > Two weeks ago, on 17th of July, Cython released a long-in-the-making > > 3.0.0 version with some backwards-incompatible changes, and while a > > lot of the packages have been made compatible, pymssql was one of the > > packages that was not. The issue did not affect x86 users, because > > pymssql binaries were pre-compiled in PyPI > > https://pypi.org/project/pymssql/2.2.7 but ARM users have problems > > installing it, because it needs to be compiled on-the-flight for them. > > > > It caused quite a bit of mayhem in Python ecosystem - especially for > > projects that are not as up-to-date as Airflow is with all our > > dependencies - most of our dependecies are automatically updated in > > the constraints as soon as new versions are released, and many of them > > have binary packages already. So given how big of the problem it was > > for some other projects, having just pymssql being problematic is > > quite cool and shows that our approach works :). > > > > Unfortunately we have no control over which version of Cython is used > > when compiling PyMSSQL (this is something described by pymssql package > > - and new versions of pip uses "build-isolation" enabled by default, > > so it's only up to the package itself to decide on the version of > > build tools that are used. There is a "mostly accidental" - I think - > > workaround with PIP_CONSTRAINTS environment variable but it is rather > > complexi-sh to pull, especialy in custom docker images based on the > > slim images. > > > > I've implemented the workaround for our ARM images last week to make > > them work - so you can see it's quite a bit complex-ish: > > https://github.com/apache/airflow/pull/32748 > > > > The 2.2.8 version of pymssql has only one change: > > > > > Version 2.2.8 - 2023-07-30 - Mikhail Terekhov > > > Compatibility with Cython. Thanks to matusvalo (Matus Valo) (fix #826). > > > > Why 2.5+ ? > > > > a) because ARM suppport for MsSQL has been introduced in 2.5.1 > > b) because 2.4 used 2.2.5 version of PyMSSQL and there were few more > > changes in 2.2.6 so there is a (low) risk it will break something > > else. > > > > Note, that we do NOT have to rebuild our images, when the pymssql > > 2.2.7 has been build before Cython 3.0.0, it is good to go. The fact > > that 2.2.8 only change is to make it works with CPython to build - we > > do not need to rebuild and re-release our images. > > > > Can we get consensus on it? Anyone has anything against it ? > > > > J. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
