BTW. I will run the consensus question until Thursday 3rd of August 2023 1pm noon.
On Mon, Jul 31, 2023 at 12:20 PM Jarek Potiuk <[email protected]> wrote: > > Hello everyone, > > TL;DR; I have a small proposal - I would like to fix constraints for > all Airflow 2.5.* and 2.6* by updating pymssql to (released today) > 2.2.8 version from 2.2.7. > > I still have to wait for the "complete" release. They have not yet > released linux binary variants of the packages for 2.2.8 and people > watching it flagged it to the maintainers. but I wanted to get > consensus on it before I start doing it. > > Currently users installing MSSQL provider for their ARM-based airflow > are experiencing "build failed" when pymssql is installed. They have > to use a workaround described here - > https://github.com/apache/airflow/issues/32672#issuecomment-1647007726 > and the proposal aims to fix it so that the workaround will not be > needed when using constraints. There are already few issues about it > in our repo: > > This is one of the extremely rare cases (happened already 2 times over > last 2 years) where our "reproducible installation" stopped working > for Python versions - because of the `pip` tooling update that we have > no control over, but thanks to ability of updating constraints, we can > fix it by updating constraints. > > If we get consensus I will use that opportunity to add some tooling to > make it easier to do such updates in the future - it requires creating > new branch for every versio and moving constraint tags - but this is > easy to automate. And I will have an excuse to develop a small tool do > help with that that - which we will be able to use in the future in > simillar cases (I've done it manually before). > > Some more context: > > Two weeks ago, on 17th of July, Cython released a long-in-the-making > 3.0.0 version with some backwards-incompatible changes, and while a > lot of the packages have been made compatible, pymssql was one of the > packages that was not. The issue did not affect x86 users, because > pymssql binaries were pre-compiled in PyPI > https://pypi.org/project/pymssql/2.2.7 but ARM users have problems > installing it, because it needs to be compiled on-the-flight for them. > > It caused quite a bit of mayhem in Python ecosystem - especially for > projects that are not as up-to-date as Airflow is with all our > dependencies - most of our dependecies are automatically updated in > the constraints as soon as new versions are released, and many of them > have binary packages already. So given how big of the problem it was > for some other projects, having just pymssql being problematic is > quite cool and shows that our approach works :). > > Unfortunately we have no control over which version of Cython is used > when compiling PyMSSQL (this is something described by pymssql package > - and new versions of pip uses "build-isolation" enabled by default, > so it's only up to the package itself to decide on the version of > build tools that are used. There is a "mostly accidental" - I think - > workaround with PIP_CONSTRAINTS environment variable but it is rather > complexi-sh to pull, especialy in custom docker images based on the > slim images. > > I've implemented the workaround for our ARM images last week to make > them work - so you can see it's quite a bit complex-ish: > https://github.com/apache/airflow/pull/32748 > > The 2.2.8 version of pymssql has only one change: > > > Version 2.2.8 - 2023-07-30 - Mikhail Terekhov > > Compatibility with Cython. Thanks to matusvalo (Matus Valo) (fix #826). > > Why 2.5+ ? > > a) because ARM suppport for MsSQL has been introduced in 2.5.1 > b) because 2.4 used 2.2.5 version of PyMSSQL and there were few more > changes in 2.2.6 so there is a (low) risk it will break something > else. > > Note, that we do NOT have to rebuild our images, when the pymssql > 2.2.7 has been build before Cython 3.0.0, it is good to go. The fact > that 2.2.8 only change is to make it works with CPython to build - we > do not need to rebuild and re-release our images. > > Can we get consensus on it? Anyone has anything against it ? > > J. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
