Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Culmination of a great bit of effort Jarek! Great job!! 👏 -- Regards, Aritra Basu On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, wrote: > It has never been simpler to contribute to Airflow! Awesome job Jarek :) > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > Quick additional info - if you have in your repo a 'tests` or 'airflow' > > folder remaining in the root of the repo - because you had some extra > files > > in those (for example generated node_modules) - you should delete those > > two directories. They are now unused and any files remaining there can > and > > *SHOULD* be deleted > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk > napisał: > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > Most important - *please rebase all your work now to the latest main*. > > > Most PR will have conflicts and will require to be rebased anyway, but > you > > > will do you a favour if you do it manually first. > > > > > > Most likely those rebases will not work from the UI (they will just ask > > > you to do the rebase manual way and give some hints on how this can be > done. > > > > > > If you have apache airflow repo set as remote, (I have 'apache' > remote), > > > this can be usually done with: > > > > > > git fetch apache > > > git rebase --onto apache/main $(git merge-base) > > > > > > Of course you have to check it manually - but this one should take all > the > > > commits you locally committed when you worked on your PR and > 'transplant' > > > them on top of the main branch. > > > > > > Few things to take care of after: > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > breeze ci-image build > > > > > > 2. Make sure to resync your uv .venv including reinstallation: > > > > > > uv self upgrade > > > uv sync --reinstall > > > > > > This one will update your venv and make sure it gets reinstalled with > the > > > new packages and all necessary deps for core airflow. > > > > > > There are quite a few other variants of such sync you should be able to > > > use from now on: > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > uv sync > > > > > > This one will (after this change) install airflow core + all optional > > > dependencies of airflow + all pre installed providers locally (and > their > > > dependencies) . Which means that it should allow to run all > `airflow-core` > > > tests. In theory - we still have few tests in airflow that might > require > > > other providers - to be cleaned up later. I will modify our CI later to > > > also run using those limited, isolated environments to keep it this > way in > > > the future. > > > > > > You should be also able to run tests after regular activation of your > venv > > > (. ./.ven/bin/activate) and this is where your IDE should also have > your > > > python interpreter set - but uv has this cool `uv run` feature that > allows > > > you to run any command with automated activation of the venv: > > > > > > uv run pytest airflow-core/tests/ > > > > > > > > > Also this should work out of the box: > > > > > > uv run airflow > > > > > > Go figure :) > > > > > > > > > *Syncing dependencies for particular provider (and other dependent > > > providers)* > > > > > > In the root of Airflow repo > > > > > > uv sync --package apache-airflow-providers-amazon > > > > > > This will sync amazon and all necessary development deps + all the > > > providers that amazon depends on, this way you **should** be able to > run > > > all amazon provider tests (including transfers and all others) - what > > > Dennis asked about at the call yesterday. > > > > > > Similarly you can run your tests this way > > > > > > uv run --package apache-airflow-providers-amazon pytest > > > providers/amazon/unit/ > > > > > > *Alternative way of syncing provider dependencies * > > > > > > cd providers/amazon > > > uv sync > > > > > > In this case you should be able to also do this: > > > > > > uv run pytest tests/unit/ > > > > > > You soon will be able to do the same in `airflow-core` - once the tests > > > that are expecting providers are removed from "airflow-core". > > > > > > cd airflow-core > > > uv sync > > > > > > That's about it. All the rest should not change, Breeze tests, > > > start-airflow etc. should work as usual. > > > > > > > > > *Syncing all dependencies* > > > > > > This is equivalent to what `breeze` image has. I do not really > recommend > > > using it daily - syncing venv and swapping dependencies take > sub-seconds > > > with *uv, *also you should really treat the .venv in your repo as > > > disposable and something you can easily resync any time. > > > > > > uv sync --all-packages > > > > > > This should allow you to run everything > > > > > > uv run --all-packages pytest > > > > > > Have fun! > > > > > > I am here and on slack `#contributors` later today. Shoot me with any > > > questions and problems - happy to help (and encourage to help each > other > > > there too) > > > > > > *Bonus info* > > > > >
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
It has never been simpler to contribute to Airflow! Awesome job Jarek :) On 2025/03/21 13:50:05 Jarek Potiuk wrote: > Quick additional info - if you have in your repo a 'tests` or 'airflow' > folder remaining in the root of the repo - because you had some extra files > in those (for example generated node_modules) - you should delete those > two directories. They are now unused and any files remaining there can and > *SHOULD* be deleted > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk napisał: > > > Ok. Now the "airflow-core" change is merged. > > > > Most important - *please rebase all your work now to the latest main*. > > Most PR will have conflicts and will require to be rebased anyway, but you > > will do you a favour if you do it manually first. > > > > Most likely those rebases will not work from the UI (they will just ask > > you to do the rebase manual way and give some hints on how this can be done. > > > > If you have apache airflow repo set as remote, (I have 'apache' remote), > > this can be usually done with: > > > > git fetch apache > > git rebase --onto apache/main $(git merge-base) > > > > Of course you have to check it manually - but this one should take all the > > commits you locally committed when you worked on your PR and 'transplant' > > them on top of the main branch. > > > > Few things to take care of after: > > > > 1. Make sure to rebuild your breeze image: > > > > breeze ci-image build > > > > 2. Make sure to resync your uv .venv including reinstallation: > > > > uv self upgrade > > uv sync --reinstall > > > > This one will update your venv and make sure it gets reinstalled with the > > new packages and all necessary deps for core airflow. > > > > There are quite a few other variants of such sync you should be able to > > use from now on: > > > > *Syncing airflow core minimum dev dependencies * > > > > uv sync > > > > This one will (after this change) install airflow core + all optional > > dependencies of airflow + all pre installed providers locally (and their > > dependencies) . Which means that it should allow to run all `airflow-core` > > tests. In theory - we still have few tests in airflow that might require > > other providers - to be cleaned up later. I will modify our CI later to > > also run using those limited, isolated environments to keep it this way in > > the future. > > > > You should be also able to run tests after regular activation of your venv > > (. ./.ven/bin/activate) and this is where your IDE should also have your > > python interpreter set - but uv has this cool `uv run` feature that allows > > you to run any command with automated activation of the venv: > > > > uv run pytest airflow-core/tests/ > > > > > > Also this should work out of the box: > > > > uv run airflow > > > > Go figure :) > > > > > > *Syncing dependencies for particular provider (and other dependent > > providers)* > > > > In the root of Airflow repo > > > > uv sync --package apache-airflow-providers-amazon > > > > This will sync amazon and all necessary development deps + all the > > providers that amazon depends on, this way you **should** be able to run > > all amazon provider tests (including transfers and all others) - what > > Dennis asked about at the call yesterday. > > > > Similarly you can run your tests this way > > > > uv run --package apache-airflow-providers-amazon pytest > > providers/amazon/unit/ > > > > *Alternative way of syncing provider dependencies * > > > > cd providers/amazon > > uv sync > > > > In this case you should be able to also do this: > > > > uv run pytest tests/unit/ > > > > You soon will be able to do the same in `airflow-core` - once the tests > > that are expecting providers are removed from "airflow-core". > > > > cd airflow-core > > uv sync > > > > That's about it. All the rest should not change, Breeze tests, > > start-airflow etc. should work as usual. > > > > > > *Syncing all dependencies* > > > > This is equivalent to what `breeze` image has. I do not really recommend > > using it daily - syncing venv and swapping dependencies take sub-seconds > > with *uv, *also you should really treat the .venv in your repo as > > disposable and something you can easily resync any time. > > > > uv sync --all-packages > > > > This should allow you to run everything > > > > uv run --all-packages pytest > > > > Have fun! > > > > I am here and on slack `#contributors` later today. Shoot me with any > > questions and problems - happy to help (and encourage to help each other > > there too) > > > > *Bonus info* > > > > Actually you do not even need to do 'uv sync`. When you use uv run , uv > > automatically > > runs uv sync under the hood (applying the --package switches as > > appropriate) and you get the latest env resynced automatically ! > > > > Actually it's even more - you do not need python installed at all when you > > run `uv run` - uv will download and install (in seconds) the right version > > of Python for you automatically ! > > >
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Cool On Fri, 21 Mar 2025 at 20:46, Aritra Basu wrote: > Culmination of a great bit of effort Jarek! Great job!! 👏 > -- > Regards, > Aritra Basu > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, wrote: > > > It has never been simpler to contribute to Airflow! Awesome job Jarek :) > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > Quick additional info - if you have in your repo a 'tests` or 'airflow' > > > folder remaining in the root of the repo - because you had some extra > > files > > > in those (for example generated node_modules) - you should delete > those > > > two directories. They are now unused and any files remaining there can > > and > > > *SHOULD* be deleted > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk > > napisał: > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > Most important - *please rebase all your work now to the latest > main*. > > > > Most PR will have conflicts and will require to be rebased anyway, > but > > you > > > > will do you a favour if you do it manually first. > > > > > > > > Most likely those rebases will not work from the UI (they will just > ask > > > > you to do the rebase manual way and give some hints on how this can > be > > done. > > > > > > > > If you have apache airflow repo set as remote, (I have 'apache' > > remote), > > > > this can be usually done with: > > > > > > > > git fetch apache > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > Of course you have to check it manually - but this one should take > all > > the > > > > commits you locally committed when you worked on your PR and > > 'transplant' > > > > them on top of the main branch. > > > > > > > > Few things to take care of after: > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > breeze ci-image build > > > > > > > > 2. Make sure to resync your uv .venv including reinstallation: > > > > > > > > uv self upgrade > > > > uv sync --reinstall > > > > > > > > This one will update your venv and make sure it gets reinstalled with > > the > > > > new packages and all necessary deps for core airflow. > > > > > > > > There are quite a few other variants of such sync you should be able > to > > > > use from now on: > > > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > > > uv sync > > > > > > > > This one will (after this change) install airflow core + all optional > > > > dependencies of airflow + all pre installed providers locally (and > > their > > > > dependencies) . Which means that it should allow to run all > > `airflow-core` > > > > tests. In theory - we still have few tests in airflow that might > > require > > > > other providers - to be cleaned up later. I will modify our CI later > to > > > > also run using those limited, isolated environments to keep it this > > way in > > > > the future. > > > > > > > > You should be also able to run tests after regular activation of your > > venv > > > > (. ./.ven/bin/activate) and this is where your IDE should also have > > your > > > > python interpreter set - but uv has this cool `uv run` feature that > > allows > > > > you to run any command with automated activation of the venv: > > > > > > > > uv run pytest airflow-core/tests/ > > > > > > > > > > > > Also this should work out of the box: > > > > > > > > uv run airflow > > > > > > > > Go figure :) > > > > > > > > > > > > *Syncing dependencies for particular provider (and other dependent > > > > providers)* > > > > > > > > In the root of Airflow repo > > > > > > > > uv sync --package apache-airflow-providers-amazon > > > > > > > > This will sync amazon and all necessary development deps + all the > > > > providers that amazon depends on, this way you **should** be able to > > run > > > > all amazon provider tests (including transfers and all others) - what > > > > Dennis asked about at the call yesterday. > > > > > > > > Similarly you can run your tests this way > > > > > > > > uv run --package apache-airflow-providers-amazon pytest > > > > providers/amazon/unit/ > > > > > > > > *Alternative way of syncing provider dependencies * > > > > > > > > cd providers/amazon > > > > uv sync > > > > > > > > In this case you should be able to also do this: > > > > > > > > uv run pytest tests/unit/ > > > > > > > > You soon will be able to do the same in `airflow-core` - once the > tests > > > > that are expecting providers are removed from "airflow-core". > > > > > > > > cd airflow-core > > > > uv sync > > > > > > > > That's about it. All the rest should not change, Breeze tests, > > > > start-airflow etc. should work as usual. > > > > > > > > > > > > *Syncing all dependencies* > > > > > > > > This is equivalent to what `breeze` image has. I do not really > > recommend > > > > using it daily - syncing venv and swapping dependencies take > > sub-seconds > > > > with *uv, *also you should really treat the .venv in your repo as > > > > disposable and something you can easily resync any time. > > > > > > >
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Yeah. For a moment I considered this to be candidate for PR of the month - but we have so many candidates! On Tue, Mar 25, 2025 at 2:54 PM Vincent Beck wrote: > I love this script! Thanks Pavan for doing it, that saves me (and I guess > many people as well) a lot of time! > > On 2025/03/25 00:52:13 Jarek Potiuk wrote: > > Nice follow up ! > > > > Thanks to Pavan, we now have a very nice tooling to set up the Airflow > > multi-distribution project with PyCharm/IntelliJ. With a single command > uv > > run setup_idea.py you will have .idea/airflow.iml .idea/modules.xml > > generated for you with all the src and test files configured for you in > > PyCharm/IntelliJ. > > > > You do not have to configure them manually any more - and you can start > > hacking with airflow-core, task-sdk, airflow-ctl or any other providers > in > > IntelliJ in a matter of seconds. > > > > It’s nicely described in > > > https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst > > how to set up your project with PyCharm/ IntelliJ. > > > > If someone with VSCode needs would like to look for something similar - > > the path is cleared now :) > > > > J. > > > > > > > > On Sat, Mar 22, 2025 at 12:53 PM Amogh Desai > > wrote: > > > > > That is brilliant! > > > > > > I am loving the new repo structure and am absolutely amazed at how > few/none > > > teething issues > > > such a move has introduced! Well executed Jarek! > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Fri, Mar 21, 2025 at 11:35 PM Elad Kalif > wrote: > > > > > > > Amazing work! > > > > > > > > On Fri, Mar 21, 2025 at 7:50 PM Pavankumar Gopidesu < > > > > [email protected]> > > > > wrote: > > > > > > > > > Great work Jarek :) > > > > > > > > > > Pavan > > > > > > > > > > On Fri, Mar 21, 2025 at 5:42 PM Buğra Öztürk < > [email protected]> > > > > > wrote: > > > > > > > > > > > Amazing news and effort! Thanks Jarek! > > > > > > > > > > > > On Fri, 21 Mar 2025, 16:18 Kaxil Naik, > wrote: > > > > > > > > > > > > > Cool > > > > > > > > > > > > > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > > > > > > > -- > > > > > > > > Regards, > > > > > > > > Aritra Basu > > > > > > > > > > > > > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, < > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > It has never been simpler to contribute to Airflow! > Awesome job > > > > > Jarek > > > > > > > :) > > > > > > > > > > > > > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > > > > > > > Quick additional info - if you have in your repo a > 'tests` or > > > > > > > 'airflow' > > > > > > > > > > folder remaining in the root of the repo - because you > had > > > some > > > > > > extra > > > > > > > > > files > > > > > > > > > > in those (for example generated node_modules) - you > should > > > > > delete > > > > > > > > those > > > > > > > > > > two directories. They are now unused and any files > remaining > > > > > there > > > > > > > can > > > > > > > > > and > > > > > > > > > > *SHOULD* be deleted > > > > > > > > > > > > > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk < > > > > > [email protected]> > > > > > > > > > napisał: > > > > > > > > > > > > > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > > > > > > > > > > > > > Most important - *please rebase all your work now to > the > > > > latest > > > > > > > > main*. > > > > > > > > > > > Most PR will have conflicts and will require to be > rebased > > > > > > anyway, > > > > > > > > but > > > > > > > > > you > > > > > > > > > > > will do you a favour if you do it manually first. > > > > > > > > > > > > > > > > > > > > > > Most likely those rebases will not work from the UI > (they > > > > will > > > > > > just > > > > > > > > ask > > > > > > > > > > > you to do the rebase manual way and give some hints on > how > > > > this > > > > > > can > > > > > > > > be > > > > > > > > > done. > > > > > > > > > > > > > > > > > > > > > > If you have apache airflow repo set as remote, (I have > > > > 'apache' > > > > > > > > > remote), > > > > > > > > > > > this can be usually done with: > > > > > > > > > > > > > > > > > > > > > > git fetch apache > > > > > > > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > > > > > > > > > > > > > Of course you have to check it manually - but this one > > > should > > > > > > take > > > > > > > > all > > > > > > > > > the > > > > > > > > > > > commits you locally committed when you worked on your > PR > > > and > > > > > > > > > 'transplant' > > > > > > > > > > > them on top of the main branch. > > > > > > > > > > > > > > > > > > > > > > Few things to take care of after: > > > > > > > > > > > > > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > > >
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
I love this script! Thanks Pavan for doing it, that saves me (and I guess many people as well) a lot of time! On 2025/03/25 00:52:13 Jarek Potiuk wrote: > Nice follow up ! > > Thanks to Pavan, we now have a very nice tooling to set up the Airflow > multi-distribution project with PyCharm/IntelliJ. With a single command uv > run setup_idea.py you will have .idea/airflow.iml .idea/modules.xml > generated for you with all the src and test files configured for you in > PyCharm/IntelliJ. > > You do not have to configure them manually any more - and you can start > hacking with airflow-core, task-sdk, airflow-ctl or any other providers in > IntelliJ in a matter of seconds. > > It’s nicely described in > https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst > how to set up your project with PyCharm/ IntelliJ. > > If someone with VSCode needs would like to look for something similar - > the path is cleared now :) > > J. > > > > On Sat, Mar 22, 2025 at 12:53 PM Amogh Desai > wrote: > > > That is brilliant! > > > > I am loving the new repo structure and am absolutely amazed at how few/none > > teething issues > > such a move has introduced! Well executed Jarek! > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Fri, Mar 21, 2025 at 11:35 PM Elad Kalif wrote: > > > > > Amazing work! > > > > > > On Fri, Mar 21, 2025 at 7:50 PM Pavankumar Gopidesu < > > > [email protected]> > > > wrote: > > > > > > > Great work Jarek :) > > > > > > > > Pavan > > > > > > > > On Fri, Mar 21, 2025 at 5:42 PM Buğra Öztürk > > > > wrote: > > > > > > > > > Amazing news and effort! Thanks Jarek! > > > > > > > > > > On Fri, 21 Mar 2025, 16:18 Kaxil Naik, wrote: > > > > > > > > > > > Cool > > > > > > > > > > > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > > > > > > -- > > > > > > > Regards, > > > > > > > Aritra Basu > > > > > > > > > > > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, > > > > > wrote: > > > > > > > > > > > > > > > It has never been simpler to contribute to Airflow! Awesome job > > > > Jarek > > > > > > :) > > > > > > > > > > > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > > > > > > Quick additional info - if you have in your repo a 'tests` or > > > > > > 'airflow' > > > > > > > > > folder remaining in the root of the repo - because you had > > some > > > > > extra > > > > > > > > files > > > > > > > > > in those (for example generated node_modules) - you should > > > > delete > > > > > > > those > > > > > > > > > two directories. They are now unused and any files remaining > > > > there > > > > > > can > > > > > > > > and > > > > > > > > > *SHOULD* be deleted > > > > > > > > > > > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk < > > > > [email protected]> > > > > > > > > napisał: > > > > > > > > > > > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > > > > > > > > > > > Most important - *please rebase all your work now to the > > > latest > > > > > > > main*. > > > > > > > > > > Most PR will have conflicts and will require to be rebased > > > > > anyway, > > > > > > > but > > > > > > > > you > > > > > > > > > > will do you a favour if you do it manually first. > > > > > > > > > > > > > > > > > > > > Most likely those rebases will not work from the UI (they > > > will > > > > > just > > > > > > > ask > > > > > > > > > > you to do the rebase manual way and give some hints on how > > > this > > > > > can > > > > > > > be > > > > > > > > done. > > > > > > > > > > > > > > > > > > > > If you have apache airflow repo set as remote, (I have > > > 'apache' > > > > > > > > remote), > > > > > > > > > > this can be usually done with: > > > > > > > > > > > > > > > > > > > > git fetch apache > > > > > > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > > > > > > > > > > > Of course you have to check it manually - but this one > > should > > > > > take > > > > > > > all > > > > > > > > the > > > > > > > > > > commits you locally committed when you worked on your PR > > and > > > > > > > > 'transplant' > > > > > > > > > > them on top of the main branch. > > > > > > > > > > > > > > > > > > > > Few things to take care of after: > > > > > > > > > > > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > > > > > > > > > > > breeze ci-image build > > > > > > > > > > > > > > > > > > > > 2. Make sure to resync your uv .venv including > > > reinstallation: > > > > > > > > > > > > > > > > > > > > uv self upgrade > > > > > > > > > > uv sync --reinstall > > > > > > > > > > > > > > > > > > > > This one will update your venv and make sure it gets > > > > reinstalled > > > > > > with > > > > > > > > the > > > > > > > > > > new packages and all necessary deps for core airflow. > > > > > > > > > > > > > > > > > > > > There
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Nice follow up ! Thanks to Pavan, we now have a very nice tooling to set up the Airflow multi-distribution project with PyCharm/IntelliJ. With a single command uv run setup_idea.py you will have .idea/airflow.iml .idea/modules.xml generated for you with all the src and test files configured for you in PyCharm/IntelliJ. You do not have to configure them manually any more - and you can start hacking with airflow-core, task-sdk, airflow-ctl or any other providers in IntelliJ in a matter of seconds. It’s nicely described in https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst how to set up your project with PyCharm/ IntelliJ. If someone with VSCode needs would like to look for something similar - the path is cleared now :) J. On Sat, Mar 22, 2025 at 12:53 PM Amogh Desai wrote: > That is brilliant! > > I am loving the new repo structure and am absolutely amazed at how few/none > teething issues > such a move has introduced! Well executed Jarek! > > Thanks & Regards, > Amogh Desai > > > On Fri, Mar 21, 2025 at 11:35 PM Elad Kalif wrote: > > > Amazing work! > > > > On Fri, Mar 21, 2025 at 7:50 PM Pavankumar Gopidesu < > > [email protected]> > > wrote: > > > > > Great work Jarek :) > > > > > > Pavan > > > > > > On Fri, Mar 21, 2025 at 5:42 PM Buğra Öztürk > > > wrote: > > > > > > > Amazing news and effort! Thanks Jarek! > > > > > > > > On Fri, 21 Mar 2025, 16:18 Kaxil Naik, wrote: > > > > > > > > > Cool > > > > > > > > > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu < > [email protected]> > > > > > wrote: > > > > > > > > > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > > > > > -- > > > > > > Regards, > > > > > > Aritra Basu > > > > > > > > > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, > > > > wrote: > > > > > > > > > > > > > It has never been simpler to contribute to Airflow! Awesome job > > > Jarek > > > > > :) > > > > > > > > > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > > > > > Quick additional info - if you have in your repo a 'tests` or > > > > > 'airflow' > > > > > > > > folder remaining in the root of the repo - because you had > some > > > > extra > > > > > > > files > > > > > > > > in those (for example generated node_modules) - you should > > > delete > > > > > > those > > > > > > > > two directories. They are now unused and any files remaining > > > there > > > > > can > > > > > > > and > > > > > > > > *SHOULD* be deleted > > > > > > > > > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk < > > > [email protected]> > > > > > > > napisał: > > > > > > > > > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > > > > > > > > > Most important - *please rebase all your work now to the > > latest > > > > > > main*. > > > > > > > > > Most PR will have conflicts and will require to be rebased > > > > anyway, > > > > > > but > > > > > > > you > > > > > > > > > will do you a favour if you do it manually first. > > > > > > > > > > > > > > > > > > Most likely those rebases will not work from the UI (they > > will > > > > just > > > > > > ask > > > > > > > > > you to do the rebase manual way and give some hints on how > > this > > > > can > > > > > > be > > > > > > > done. > > > > > > > > > > > > > > > > > > If you have apache airflow repo set as remote, (I have > > 'apache' > > > > > > > remote), > > > > > > > > > this can be usually done with: > > > > > > > > > > > > > > > > > > git fetch apache > > > > > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > > > > > > > > > Of course you have to check it manually - but this one > should > > > > take > > > > > > all > > > > > > > the > > > > > > > > > commits you locally committed when you worked on your PR > and > > > > > > > 'transplant' > > > > > > > > > them on top of the main branch. > > > > > > > > > > > > > > > > > > Few things to take care of after: > > > > > > > > > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > > > > > > > > > breeze ci-image build > > > > > > > > > > > > > > > > > > 2. Make sure to resync your uv .venv including > > reinstallation: > > > > > > > > > > > > > > > > > > uv self upgrade > > > > > > > > > uv sync --reinstall > > > > > > > > > > > > > > > > > > This one will update your venv and make sure it gets > > > reinstalled > > > > > with > > > > > > > the > > > > > > > > > new packages and all necessary deps for core airflow. > > > > > > > > > > > > > > > > > > There are quite a few other variants of such sync you > should > > be > > > > > able > > > > > > to > > > > > > > > > use from now on: > > > > > > > > > > > > > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > > > > > > > > > > > > > uv sync > > > > > > > > > > > > > > > > > > This one will (after this change) install airflow core + > all > > > > > optional > > > > > > > > > dependencies of airflow + all pre installed providers > locall
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
That is brilliant! I am loving the new repo structure and am absolutely amazed at how few/none teething issues such a move has introduced! Well executed Jarek! Thanks & Regards, Amogh Desai On Fri, Mar 21, 2025 at 11:35 PM Elad Kalif wrote: > Amazing work! > > On Fri, Mar 21, 2025 at 7:50 PM Pavankumar Gopidesu < > [email protected]> > wrote: > > > Great work Jarek :) > > > > Pavan > > > > On Fri, Mar 21, 2025 at 5:42 PM Buğra Öztürk > > wrote: > > > > > Amazing news and effort! Thanks Jarek! > > > > > > On Fri, 21 Mar 2025, 16:18 Kaxil Naik, wrote: > > > > > > > Cool > > > > > > > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu > > > > wrote: > > > > > > > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > > > > -- > > > > > Regards, > > > > > Aritra Basu > > > > > > > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, > > > wrote: > > > > > > > > > > > It has never been simpler to contribute to Airflow! Awesome job > > Jarek > > > > :) > > > > > > > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > > > > Quick additional info - if you have in your repo a 'tests` or > > > > 'airflow' > > > > > > > folder remaining in the root of the repo - because you had some > > > extra > > > > > > files > > > > > > > in those (for example generated node_modules) - you should > > delete > > > > > those > > > > > > > two directories. They are now unused and any files remaining > > there > > > > can > > > > > > and > > > > > > > *SHOULD* be deleted > > > > > > > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk < > > [email protected]> > > > > > > napisał: > > > > > > > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > > > > > > > Most important - *please rebase all your work now to the > latest > > > > > main*. > > > > > > > > Most PR will have conflicts and will require to be rebased > > > anyway, > > > > > but > > > > > > you > > > > > > > > will do you a favour if you do it manually first. > > > > > > > > > > > > > > > > Most likely those rebases will not work from the UI (they > will > > > just > > > > > ask > > > > > > > > you to do the rebase manual way and give some hints on how > this > > > can > > > > > be > > > > > > done. > > > > > > > > > > > > > > > > If you have apache airflow repo set as remote, (I have > 'apache' > > > > > > remote), > > > > > > > > this can be usually done with: > > > > > > > > > > > > > > > > git fetch apache > > > > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > > > > > > > Of course you have to check it manually - but this one should > > > take > > > > > all > > > > > > the > > > > > > > > commits you locally committed when you worked on your PR and > > > > > > 'transplant' > > > > > > > > them on top of the main branch. > > > > > > > > > > > > > > > > Few things to take care of after: > > > > > > > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > > > > > > > breeze ci-image build > > > > > > > > > > > > > > > > 2. Make sure to resync your uv .venv including > reinstallation: > > > > > > > > > > > > > > > > uv self upgrade > > > > > > > > uv sync --reinstall > > > > > > > > > > > > > > > > This one will update your venv and make sure it gets > > reinstalled > > > > with > > > > > > the > > > > > > > > new packages and all necessary deps for core airflow. > > > > > > > > > > > > > > > > There are quite a few other variants of such sync you should > be > > > > able > > > > > to > > > > > > > > use from now on: > > > > > > > > > > > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > > > > > > > > > > > uv sync > > > > > > > > > > > > > > > > This one will (after this change) install airflow core + all > > > > optional > > > > > > > > dependencies of airflow + all pre installed providers locally > > > (and > > > > > > their > > > > > > > > dependencies) . Which means that it should allow to run all > > > > > > `airflow-core` > > > > > > > > tests. In theory - we still have few tests in airflow that > > might > > > > > > require > > > > > > > > other providers - to be cleaned up later. I will modify our > CI > > > > later > > > > > to > > > > > > > > also run using those limited, isolated environments to keep > it > > > this > > > > > > way in > > > > > > > > the future. > > > > > > > > > > > > > > > > You should be also able to run tests after regular activation > > of > > > > your > > > > > > venv > > > > > > > > (. ./.ven/bin/activate) and this is where your IDE should > also > > > have > > > > > > your > > > > > > > > python interpreter set - but uv has this cool `uv run` > feature > > > that > > > > > > allows > > > > > > > > you to run any command with automated activation of the venv: > > > > > > > > > > > > > > > > uv run pytest airflow-core/tests/ > > > > > > > > > > > > > > > > > > > > > > > > Also this should work out of the box: > > > > > > > > > > > > > > > > uv run airflow > > > > > > > > > > > > > > > > Go figure
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Quick additional info - if you have in your repo a 'tests` or 'airflow' folder remaining in the root of the repo - because you had some extra files in those (for example generated node_modules) - you should delete those two directories. They are now unused and any files remaining there can and *SHOULD* be deleted pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk napisał: > Ok. Now the "airflow-core" change is merged. > > Most important - *please rebase all your work now to the latest main*. > Most PR will have conflicts and will require to be rebased anyway, but you > will do you a favour if you do it manually first. > > Most likely those rebases will not work from the UI (they will just ask > you to do the rebase manual way and give some hints on how this can be done. > > If you have apache airflow repo set as remote, (I have 'apache' remote), > this can be usually done with: > > git fetch apache > git rebase --onto apache/main $(git merge-base) > > Of course you have to check it manually - but this one should take all the > commits you locally committed when you worked on your PR and 'transplant' > them on top of the main branch. > > Few things to take care of after: > > 1. Make sure to rebuild your breeze image: > > breeze ci-image build > > 2. Make sure to resync your uv .venv including reinstallation: > > uv self upgrade > uv sync --reinstall > > This one will update your venv and make sure it gets reinstalled with the > new packages and all necessary deps for core airflow. > > There are quite a few other variants of such sync you should be able to > use from now on: > > *Syncing airflow core minimum dev dependencies * > > uv sync > > This one will (after this change) install airflow core + all optional > dependencies of airflow + all pre installed providers locally (and their > dependencies) . Which means that it should allow to run all `airflow-core` > tests. In theory - we still have few tests in airflow that might require > other providers - to be cleaned up later. I will modify our CI later to > also run using those limited, isolated environments to keep it this way in > the future. > > You should be also able to run tests after regular activation of your venv > (. ./.ven/bin/activate) and this is where your IDE should also have your > python interpreter set - but uv has this cool `uv run` feature that allows > you to run any command with automated activation of the venv: > > uv run pytest airflow-core/tests/ > > > Also this should work out of the box: > > uv run airflow > > Go figure :) > > > *Syncing dependencies for particular provider (and other dependent > providers)* > > In the root of Airflow repo > > uv sync --package apache-airflow-providers-amazon > > This will sync amazon and all necessary development deps + all the > providers that amazon depends on, this way you **should** be able to run > all amazon provider tests (including transfers and all others) - what > Dennis asked about at the call yesterday. > > Similarly you can run your tests this way > > uv run --package apache-airflow-providers-amazon pytest > providers/amazon/unit/ > > *Alternative way of syncing provider dependencies * > > cd providers/amazon > uv sync > > In this case you should be able to also do this: > > uv run pytest tests/unit/ > > You soon will be able to do the same in `airflow-core` - once the tests > that are expecting providers are removed from "airflow-core". > > cd airflow-core > uv sync > > That's about it. All the rest should not change, Breeze tests, > start-airflow etc. should work as usual. > > > *Syncing all dependencies* > > This is equivalent to what `breeze` image has. I do not really recommend > using it daily - syncing venv and swapping dependencies take sub-seconds > with *uv, *also you should really treat the .venv in your repo as > disposable and something you can easily resync any time. > > uv sync --all-packages > > This should allow you to run everything > > uv run --all-packages pytest > > Have fun! > > I am here and on slack `#contributors` later today. Shoot me with any > questions and problems - happy to help (and encourage to help each other > there too) > > *Bonus info* > > Actually you do not even need to do 'uv sync`. When you use uv run , uv > automatically > runs uv sync under the hood (applying the --package switches as > appropriate) and you get the latest env resynced automatically ! > > Actually it's even more - you do not need python installed at all when you > run `uv run` - uv will download and install (in seconds) the right version > of Python for you automatically ! > > So really: > > * Install uv > * git clone > * uv run pytest > > Is absolutely all you need to start contributing to Airflow. > > And I absolutely love it. This has been 4 years in the making and it's > finally there! > > J > > > > On Thu, Mar 20, 2025 at 12:56 PM Jarek Potiuk wrote: > >> Ok. The PR https://github.com/apache/airflow/pull/47798 is "green" >> (minus failing main issue with
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Amazing work! On Fri, Mar 21, 2025 at 7:50 PM Pavankumar Gopidesu wrote: > Great work Jarek :) > > Pavan > > On Fri, Mar 21, 2025 at 5:42 PM Buğra Öztürk > wrote: > > > Amazing news and effort! Thanks Jarek! > > > > On Fri, 21 Mar 2025, 16:18 Kaxil Naik, wrote: > > > > > Cool > > > > > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu > > > wrote: > > > > > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > > > -- > > > > Regards, > > > > Aritra Basu > > > > > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, > > wrote: > > > > > > > > > It has never been simpler to contribute to Airflow! Awesome job > Jarek > > > :) > > > > > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > > > Quick additional info - if you have in your repo a 'tests` or > > > 'airflow' > > > > > > folder remaining in the root of the repo - because you had some > > extra > > > > > files > > > > > > in those (for example generated node_modules) - you should > delete > > > > those > > > > > > two directories. They are now unused and any files remaining > there > > > can > > > > > and > > > > > > *SHOULD* be deleted > > > > > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk < > [email protected]> > > > > > napisał: > > > > > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > > > > > Most important - *please rebase all your work now to the latest > > > > main*. > > > > > > > Most PR will have conflicts and will require to be rebased > > anyway, > > > > but > > > > > you > > > > > > > will do you a favour if you do it manually first. > > > > > > > > > > > > > > Most likely those rebases will not work from the UI (they will > > just > > > > ask > > > > > > > you to do the rebase manual way and give some hints on how this > > can > > > > be > > > > > done. > > > > > > > > > > > > > > If you have apache airflow repo set as remote, (I have 'apache' > > > > > remote), > > > > > > > this can be usually done with: > > > > > > > > > > > > > > git fetch apache > > > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > > > > > Of course you have to check it manually - but this one should > > take > > > > all > > > > > the > > > > > > > commits you locally committed when you worked on your PR and > > > > > 'transplant' > > > > > > > them on top of the main branch. > > > > > > > > > > > > > > Few things to take care of after: > > > > > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > > > > > breeze ci-image build > > > > > > > > > > > > > > 2. Make sure to resync your uv .venv including reinstallation: > > > > > > > > > > > > > > uv self upgrade > > > > > > > uv sync --reinstall > > > > > > > > > > > > > > This one will update your venv and make sure it gets > reinstalled > > > with > > > > > the > > > > > > > new packages and all necessary deps for core airflow. > > > > > > > > > > > > > > There are quite a few other variants of such sync you should be > > > able > > > > to > > > > > > > use from now on: > > > > > > > > > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > > > > > > > > > uv sync > > > > > > > > > > > > > > This one will (after this change) install airflow core + all > > > optional > > > > > > > dependencies of airflow + all pre installed providers locally > > (and > > > > > their > > > > > > > dependencies) . Which means that it should allow to run all > > > > > `airflow-core` > > > > > > > tests. In theory - we still have few tests in airflow that > might > > > > > require > > > > > > > other providers - to be cleaned up later. I will modify our CI > > > later > > > > to > > > > > > > also run using those limited, isolated environments to keep it > > this > > > > > way in > > > > > > > the future. > > > > > > > > > > > > > > You should be also able to run tests after regular activation > of > > > your > > > > > venv > > > > > > > (. ./.ven/bin/activate) and this is where your IDE should also > > have > > > > > your > > > > > > > python interpreter set - but uv has this cool `uv run` feature > > that > > > > > allows > > > > > > > you to run any command with automated activation of the venv: > > > > > > > > > > > > > > uv run pytest airflow-core/tests/ > > > > > > > > > > > > > > > > > > > > > Also this should work out of the box: > > > > > > > > > > > > > > uv run airflow > > > > > > > > > > > > > > Go figure :) > > > > > > > > > > > > > > > > > > > > > *Syncing dependencies for particular provider (and other > > dependent > > > > > > > providers)* > > > > > > > > > > > > > > In the root of Airflow repo > > > > > > > > > > > > > > uv sync --package apache-airflow-providers-amazon > > > > > > > > > > > > > > This will sync amazon and all necessary development deps + all > > the > > > > > > > providers that amazon depends on, this way you **should** be > able > > > to > > > > > run > > > > > > > all amazon provider tests (including transfers and all others) > - > > > what > > > > > > > Denni
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Great work Jarek :) Pavan On Fri, Mar 21, 2025 at 5:42 PM Buğra Öztürk wrote: > Amazing news and effort! Thanks Jarek! > > On Fri, 21 Mar 2025, 16:18 Kaxil Naik, wrote: > > > Cool > > > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu > > wrote: > > > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > > -- > > > Regards, > > > Aritra Basu > > > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, > wrote: > > > > > > > It has never been simpler to contribute to Airflow! Awesome job Jarek > > :) > > > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > > Quick additional info - if you have in your repo a 'tests` or > > 'airflow' > > > > > folder remaining in the root of the repo - because you had some > extra > > > > files > > > > > in those (for example generated node_modules) - you should delete > > > those > > > > > two directories. They are now unused and any files remaining there > > can > > > > and > > > > > *SHOULD* be deleted > > > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk > > > > napisał: > > > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > > > Most important - *please rebase all your work now to the latest > > > main*. > > > > > > Most PR will have conflicts and will require to be rebased > anyway, > > > but > > > > you > > > > > > will do you a favour if you do it manually first. > > > > > > > > > > > > Most likely those rebases will not work from the UI (they will > just > > > ask > > > > > > you to do the rebase manual way and give some hints on how this > can > > > be > > > > done. > > > > > > > > > > > > If you have apache airflow repo set as remote, (I have 'apache' > > > > remote), > > > > > > this can be usually done with: > > > > > > > > > > > > git fetch apache > > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > > > Of course you have to check it manually - but this one should > take > > > all > > > > the > > > > > > commits you locally committed when you worked on your PR and > > > > 'transplant' > > > > > > them on top of the main branch. > > > > > > > > > > > > Few things to take care of after: > > > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > > > breeze ci-image build > > > > > > > > > > > > 2. Make sure to resync your uv .venv including reinstallation: > > > > > > > > > > > > uv self upgrade > > > > > > uv sync --reinstall > > > > > > > > > > > > This one will update your venv and make sure it gets reinstalled > > with > > > > the > > > > > > new packages and all necessary deps for core airflow. > > > > > > > > > > > > There are quite a few other variants of such sync you should be > > able > > > to > > > > > > use from now on: > > > > > > > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > > > > > > > uv sync > > > > > > > > > > > > This one will (after this change) install airflow core + all > > optional > > > > > > dependencies of airflow + all pre installed providers locally > (and > > > > their > > > > > > dependencies) . Which means that it should allow to run all > > > > `airflow-core` > > > > > > tests. In theory - we still have few tests in airflow that might > > > > require > > > > > > other providers - to be cleaned up later. I will modify our CI > > later > > > to > > > > > > also run using those limited, isolated environments to keep it > this > > > > way in > > > > > > the future. > > > > > > > > > > > > You should be also able to run tests after regular activation of > > your > > > > venv > > > > > > (. ./.ven/bin/activate) and this is where your IDE should also > have > > > > your > > > > > > python interpreter set - but uv has this cool `uv run` feature > that > > > > allows > > > > > > you to run any command with automated activation of the venv: > > > > > > > > > > > > uv run pytest airflow-core/tests/ > > > > > > > > > > > > > > > > > > Also this should work out of the box: > > > > > > > > > > > > uv run airflow > > > > > > > > > > > > Go figure :) > > > > > > > > > > > > > > > > > > *Syncing dependencies for particular provider (and other > dependent > > > > > > providers)* > > > > > > > > > > > > In the root of Airflow repo > > > > > > > > > > > > uv sync --package apache-airflow-providers-amazon > > > > > > > > > > > > This will sync amazon and all necessary development deps + all > the > > > > > > providers that amazon depends on, this way you **should** be able > > to > > > > run > > > > > > all amazon provider tests (including transfers and all others) - > > what > > > > > > Dennis asked about at the call yesterday. > > > > > > > > > > > > Similarly you can run your tests this way > > > > > > > > > > > > uv run --package apache-airflow-providers-amazon pytest > > > > > > providers/amazon/unit/ > > > > > > > > > > > > *Alternative way of syncing provider dependencies * > > > > > > > > > > > > cd providers/amazon > > > > > > uv sync > > > > > > > > > > > > In this case you should be able to
Re: [ANNOUNCEMENT] airflow-core is there (finally!)
Amazing news and effort! Thanks Jarek! On Fri, 21 Mar 2025, 16:18 Kaxil Naik, wrote: > Cool > > On Fri, 21 Mar 2025 at 20:46, Aritra Basu > wrote: > > > Culmination of a great bit of effort Jarek! Great job!! 👏 > > -- > > Regards, > > Aritra Basu > > > > On Fri, 21 Mar 2025, 7:42 pm Vincent Beck, wrote: > > > > > It has never been simpler to contribute to Airflow! Awesome job Jarek > :) > > > > > > On 2025/03/21 13:50:05 Jarek Potiuk wrote: > > > > Quick additional info - if you have in your repo a 'tests` or > 'airflow' > > > > folder remaining in the root of the repo - because you had some extra > > > files > > > > in those (for example generated node_modules) - you should delete > > those > > > > two directories. They are now unused and any files remaining there > can > > > and > > > > *SHOULD* be deleted > > > > > > > > pt., 21 mar 2025, 14:28 użytkownik Jarek Potiuk > > > napisał: > > > > > > > > > Ok. Now the "airflow-core" change is merged. > > > > > > > > > > Most important - *please rebase all your work now to the latest > > main*. > > > > > Most PR will have conflicts and will require to be rebased anyway, > > but > > > you > > > > > will do you a favour if you do it manually first. > > > > > > > > > > Most likely those rebases will not work from the UI (they will just > > ask > > > > > you to do the rebase manual way and give some hints on how this can > > be > > > done. > > > > > > > > > > If you have apache airflow repo set as remote, (I have 'apache' > > > remote), > > > > > this can be usually done with: > > > > > > > > > > git fetch apache > > > > > git rebase --onto apache/main $(git merge-base) > > > > > > > > > > Of course you have to check it manually - but this one should take > > all > > > the > > > > > commits you locally committed when you worked on your PR and > > > 'transplant' > > > > > them on top of the main branch. > > > > > > > > > > Few things to take care of after: > > > > > > > > > > 1. Make sure to rebuild your breeze image: > > > > > > > > > > breeze ci-image build > > > > > > > > > > 2. Make sure to resync your uv .venv including reinstallation: > > > > > > > > > > uv self upgrade > > > > > uv sync --reinstall > > > > > > > > > > This one will update your venv and make sure it gets reinstalled > with > > > the > > > > > new packages and all necessary deps for core airflow. > > > > > > > > > > There are quite a few other variants of such sync you should be > able > > to > > > > > use from now on: > > > > > > > > > > *Syncing airflow core minimum dev dependencies * > > > > > > > > > > uv sync > > > > > > > > > > This one will (after this change) install airflow core + all > optional > > > > > dependencies of airflow + all pre installed providers locally (and > > > their > > > > > dependencies) . Which means that it should allow to run all > > > `airflow-core` > > > > > tests. In theory - we still have few tests in airflow that might > > > require > > > > > other providers - to be cleaned up later. I will modify our CI > later > > to > > > > > also run using those limited, isolated environments to keep it this > > > way in > > > > > the future. > > > > > > > > > > You should be also able to run tests after regular activation of > your > > > venv > > > > > (. ./.ven/bin/activate) and this is where your IDE should also have > > > your > > > > > python interpreter set - but uv has this cool `uv run` feature that > > > allows > > > > > you to run any command with automated activation of the venv: > > > > > > > > > > uv run pytest airflow-core/tests/ > > > > > > > > > > > > > > > Also this should work out of the box: > > > > > > > > > > uv run airflow > > > > > > > > > > Go figure :) > > > > > > > > > > > > > > > *Syncing dependencies for particular provider (and other dependent > > > > > providers)* > > > > > > > > > > In the root of Airflow repo > > > > > > > > > > uv sync --package apache-airflow-providers-amazon > > > > > > > > > > This will sync amazon and all necessary development deps + all the > > > > > providers that amazon depends on, this way you **should** be able > to > > > run > > > > > all amazon provider tests (including transfers and all others) - > what > > > > > Dennis asked about at the call yesterday. > > > > > > > > > > Similarly you can run your tests this way > > > > > > > > > > uv run --package apache-airflow-providers-amazon pytest > > > > > providers/amazon/unit/ > > > > > > > > > > *Alternative way of syncing provider dependencies * > > > > > > > > > > cd providers/amazon > > > > > uv sync > > > > > > > > > > In this case you should be able to also do this: > > > > > > > > > > uv run pytest tests/unit/ > > > > > > > > > > You soon will be able to do the same in `airflow-core` - once the > > tests > > > > > that are expecting providers are removed from "airflow-core". > > > > > > > > > > cd airflow-core > > > > > uv sync > > > > > > > > > > That's about it. All the rest should not change, Breeze tests, > > > > > start-airflow etc. shoul
