That's awesome Jarek, thank you for this :) Regards, Pavan
On Wed, Apr 2, 2025 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > You might also want to do *docker system prune* or even *docker system > prune --all* or run *breeze doctor* to clean-up some stale cache, images, > docker volumes > > Also IntelliJ/PyCharm users *uv run setup_idea.py* will add some missing > directories and regenerate your IntelliJ project configuration. > > > > > > On Wed, Apr 2, 2025 at 1:11 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Hello here, > > > > As part of the packaging work - I merged the > > https://github.com/apache/airflow/pull/48223 > > > > *TL;DR; Rebase all PRS, rebuild images and uv sync - and > > hopefully everything should work as before even if a lot of things moved. > > Hopefully the move will be largely transparent (except changing *include: > > in open PRs in docs). * > > > > *NOTE! UV is now mandatory and a lot of code is gone thanks to that. > > Breeze will also refuse to work if uv is not installed.* > > > > It took a bit of time, but we are in a much more standard and better > shape > > now - and as a side effect (which was intended but I had to implement it > as > > part of that monster PR to fix documentation) we now have a much simpler > > (more guidelines are coming) way to iterate on doc building. > > > > *Few important things first: * > > > > 1) Make sure to rebase your PRs, Run `breeze image build`, Rnu `uv sync`. > > Due to the way git handles things - you ** might ** have some dangling > > generated directories in your repo and they might cause some problems. > Run > > "git status" after rebase and see you have some files you need to delete > > (manually) > > > > 2) If you are brave enough - you might want to run `breeze doctor` and > > cleanup git repo - it should clean all files that should be removed, but > > also it might remove some of your custom configurations and files you > > created, > > > > 3) Generally everything should work as it worked before with breeze (for > > example `breeze build-docs` command works as before. But a number of > > folders/distributions/code (not airflow nore providers directly) were > > moved/updated. For now you can just continue to build docs as before - > with > > breeze. But simpler/faster ways are coming as follow up. > > > > 4) If you have some new examples or documentation included in your PRs > the > > doc build might start falling for you - but this is because `include::` > or > > `exampleinclude:` might need to be updated - look at other examples - I > > fixed the includes in all providers. More explanation in follow-up doc > > build improvement PR - in the meantime, feel free to ask on slack or PR > for > > help. > > > > *Generated provider_dependencies.json do not need to be updated* > > > > The "generated/provider_dependencies.json" is no longer committed to the > > repo - it is .gitignored. We are generating it as-needed on the flight. > It > > should be automatically regenerated when you run pre-commits locally and > > when you build the breeze image. > > > > There might be some cases when we add dependencies and you will need to > > regenerate it but that should happen automatically as needed. > > > > *New, updated folders* > > > > The change are mostly with these: > > > > ./dev/pyproject.toml > > ./devel-common/pyproject.toml > > ./doc > > ./docker-stack-docs > > ./providers-summary-docs > > > > *More explanation for distributions/folder changes* > > > > The dev is now a separate distribution with its own pyproject.toml > > dependencies that are used for all the release management and general dev > > housekeeping. This is different from "devel-common" which is a common > > package with a lot of common code reused for tests and builds (including > > doc builds scripts are using). > > > > We might want to change names later, as they are confusing but I will > > leave that discussion for later when we complete all the isolation work - > > with those changes I implemented it will be very easy (as opposed to how > it > > was) to move those distributions around - and if we will want to > > restructure it again, that will be a very simple move. > > > > *Improved doc structure* > > > > This is because we finally made the last step - where the "doc" code is > > moved to "devel-common" and we are importing it from there, also each > > distribution has it's own "conf.py" and that makes modifying docs > building > > and our sphinx building scripts MUCH more readable ana mangeable. The > "doc" > > code now contains just spelling wordlist and README.md explaining where > to > > look for the documentation. > > > > Those new top-level folders ("docker-stack-docs" and > > "providers-summary-docs") - contain the two independent pieces of our > > documentation ("docker-stack" and "apache-airflow-providers" package > names > > from "breeze build-docs" command. > > > > The REALLY nice thing now is that the doc files are not copied between > > places and each of the docs folder (including each of the 90 providers > docs > > folder) has it's own "conf.py", which makes the "docs" folder for them > > "isolated" and "fully self-contained" - which will open up the ways on > much > > more friendly sphinx workflows. Wait for it ! Coming **just in time** > when > > we most need it - when we want to start updating the docs for Airflow 3. > > > > J. > > > > > > > > > > > > > > >
