Hey everyone, We have the Bullseye - based builds successful - including all tests - so we are ready to switch:
Looking forward to the reviews - there are three PRs: 1) https://github.com/apache/airflow/pull/21546 - allows to switch easily between Buster/Bullseye 2) https://github.com/apache/airflow/pull/21378 - actually Switch to Debian 11 (bullseye) as base for our dockerfiles 3) https://github.com/apache/airflow/pull/21697 - proposed Draft to policy we should follow (based on the discussion above) Once we merge those and we consent on the policy, I will add another one where we will be building both Bullseye/Buster in our CI and run verification for both. We are close to switching to Python version of builds (as the internship work on switching Breeze to Python is close to conclusion) so I want to do it after we will actually switch in order to avoid duplication of the work. Looking forward to comments. J. On Wed, Feb 16, 2022 at 8:48 PM Jarek Potiuk <[email protected]> wrote: > See this PR: https://github.com/apache/airflow/pull/21378 (Particularly > last commit). > > Currently failing on pyodbc.OperationalError: ('08001', '[08001] > [Microsoft][ODBC Driver 18 for SQL Server]SSL Provider: [error:1416F086:SSL > routines:tls_process_server_certificate:certificate verify failed:self > signed certificate] (-1) (SQLDriverConnect)' but that will fixed in the nex > fixup. > > > On Wed, Feb 16, 2022 at 8:45 PM Alex Begg <[email protected]> wrote: > >> Not sure if this is done already, but I think places with "ODBC Driver 17 >> for SQL Server" hardcoded in might need to change from 17 to 18 as well: >> https://github.com/apache/airflow/search?q=ODBC+Driver+17+for+SQL+Server >> >> >> On Wed, Feb 16, 2022 at 12:06 AM Jarek Potiuk <[email protected]> wrote: >> >>> Yep. Already did :) >>> >>> On Wed, Feb 16, 2022 at 6:22 AM Alex Begg <[email protected]> wrote: >>> >>>> I wanted to point out that not only did Microsoft add Debian Bullseye >>>> support for MSSQL today, to do so they did a major version >>>> upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the >>>> URL used to populate mssql-release.list, you also need to change to >>>> msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code >>>> base. >>>> >>>> Instructions for installing msodbcsql18 in Debian are here: >>>> >>>> https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18 >>>> >>>> Alex >>>> >>>> >>>> On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <[email protected]> wrote: >>>> >>>>> Yeah. That would definitely be a simpler approach for us. >>>>> >>>>> What we can also do is we can still make sure that you can actually >>>>> "build" Buster images if you want and we could run the simple tests we >>>>> already have for those images (those tests just check very basic stuff >>>>> - like if expected packages are importable, if Airflow commands work, >>>>> if you can run airflow as root user etc.. >>>>> This would be very little to no complication of the CI and we could >>>>> provide a recipe for the users how to build their own buster image - >>>>> but without guarantees that everything works there. >>>>> >>>>> This way might provide the users with a "workable solution" while we >>>>> could keep it simple for us and put a strong emphasis on "If you want >>>>> to get full support - also upgrade to the latest stable OS". >>>>> >>>>> J. >>>>> >>>>> >>>>> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <[email protected]> >>>>> wrote: >>>>> > >>>>> > My vote is to just move forward and only release Bullseye images for >>>>> 2.3 onwards >>>>> > >>>>> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >> >>>>> >> Good news from today - I got the response from Microsoft and it >>>>> seems >>>>> >> that MSSQL Bullseye support is going to be released today :) . So at >>>>> >> least that problem is going to be solved. Still the question of >>>>> >> whether we should build/release both Bullseye and Buster ? >>>>> >> >>>>> >> J. >>>>> >> >>>>> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >>> >>>>> >>> >>>>> >>> And Python default images are Bullseye now. >>>>> >>> >>>>> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >>>> >>>>> >>>> >>>>> >>>> Buster end of life is August 2022 of course :) >>>>> >>>> >>>>> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Hello everyone, >>>>> >>>>> >>>>> >>>>> I am just about to complete preparation to make our images (and >>>>> CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks >>>>> (almost) good, but I would like to tap into the collective wisdom of the >>>>> group here to decide on next steps (and maybe propose a policy we might be >>>>> following in the future). The last change that allows to easily switch >>>>> between the two is in reviee: >>>>> https://github.com/apache/airflow/pull/21546 >>>>> >>>>> >>>>> >>>>> Context/where we are: >>>>> >>>>> >>>>> >>>>> For quite a while we used Buster (Debian 10) as our base image. >>>>> Buster is a stable release and it is being replaced with the next stable >>>>> release Bullseye (Debian 11). >>>>> >>>>> >>>>> >>>>> The Release schedule is here: >>>>> https://wiki.debian.org/DebianReleases >>>>> >>>>> >>>>> >>>>> The most important facts from those: >>>>> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has >>>>> no end-of-life yet >>>>> >>>>> * Buster was introduced in July 2019 and its end of life is ~ >>>>> August 2020 (approx. 6 months from now) >>>>> >>>>> >>>>> >>>>> Python default images are (as of a few months), However they >>>>> provide (as usual -buster and -bullseye) specific images. >>>>> >>>>> >>>>> >>>>> The Bullseye switch is really needed to support ARM (M1) images >>>>> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 >>>>> (still unresolved on Buster). >>>>> >>>>> >>>>> >>>>> It seems appropriate to switch to Bullseye now, however there >>>>> is one problem with it - MSSQL drivers do not yet support Bullseye. I've >>>>> reached out to the maintainers and await their answer when it might be >>>>> available: >>>>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131 >>>>> . This is the last blocker that prevents full switch (failing PR for a >>>>> full >>>>> switch is https://github.com/apache/airflow/pull/21378). All tests >>>>> pass except MSSQL. >>>>> >>>>> >>>>> >>>>> But I think we need to proceed, regardless of the timeline and >>>>> switch to Bullseye. >>>>> >>>>> >>>>> >>>>> Question: >>>>> >>>>> >>>>> >>>>> We need to decide on what support we give our users in the >>>>> images now, in the 2.3.0 release and in the future. We have no policies >>>>> for >>>>> that yet, so it might be a good time to decide now. Some of the users >>>>> might >>>>> "rely" on the Buster being used by the images as they might have some >>>>> incompatible libraries (similarly to MSSQL). And I believe we need to >>>>> support Buster till at least the end of life of it. >>>>> >>>>> >>>>> >>>>> This support might be twofold: >>>>> >>>>> >>>>> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow >>>>> our binary images with Buster and/or Bullseye >>>>> >>>>> * we can only publish one of those (and change from Buster to >>>>> Bullseye for 2.3.0) but users will be able to build their own custom >>>>> images >>>>> using our Dockerfile for either of those. >>>>> >>>>> >>>>> >>>>> Proposal: >>>>> >>>>> >>>>> >>>>> I think there are multiple reasons why we should support both >>>>> Bullseye and Buster for the coming 6 months. We might build both images in >>>>> CI and add a new "matrix" dimension and run most tests on Bullseye but >>>>> some >>>>> tests on Buster (specifically MSSQL until Bullseye is supported). >>>>> >>>>> >>>>> >>>>> We can publish both types of images (with -buster, -bullseye >>>>> suffixes same as Python) but the default image should be changed to >>>>> Bullseye. >>>>> >>>>> >>>>> >>>>> This will add some complexity (and build overhead) to our CI, >>>>> but I think it's worth it. >>>>> >>>>> >>>>> >>>>> Then we could drop support and release the -buster images after >>>>> it reaches end of life (but we will leave the users possibility of >>>>> building >>>>> their own buster images without guarantee that it will work). >>>>> >>>>> >>>>> >>>>> I think this might become our Policy also for the future (6 >>>>> months before end-of-life we switch by default to the new stable and >>>>> provide built images until end-of-life). >>>>> >>>>> >>>>> >>>>> WDYT? Does it look like a good proposal? >>>>> >>>>> >>>>> >>>>> J. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>
