Thank you for the effort Jarek! I am looking forward to having a simpler life building docs. Personally it was a nightmare for me earlier. Will shout out if / when I get to that stage.
Thanks & Regards, Amogh Desai On Wed, Apr 2, 2025 at 10:37 PM Buğra Öztürk <ozturkbugr...@gmail.com> wrote: > Amazing news! Thanks for the huge effort Jarek! :) > > On Wed, 2 Apr 2025, 18:45 Pavankumar Gopidesu, <gopidesupa...@gmail.com> > wrote: > > > That's awesome Jarek, thank you for this :) > > > > Regards, > > Pavan > > > > On Wed, Apr 2, 2025 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > You might also want to do *docker system prune* or even *docker system > > > prune --all* or run *breeze doctor* to clean-up some stale cache, > > images, > > > docker volumes > > > > > > Also IntelliJ/PyCharm users *uv run setup_idea.py* will add some > missing > > > directories and regenerate your IntelliJ project configuration. > > > > > > > > > > > > > > > > > > On Wed, Apr 2, 2025 at 1:11 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > Hello here, > > > > > > > > As part of the packaging work - I merged the > > > > https://github.com/apache/airflow/pull/48223 > > > > > > > > *TL;DR; Rebase all PRS, rebuild images and uv sync - and > > > > hopefully everything should work as before even if a lot of things > > moved. > > > > Hopefully the move will be largely transparent (except changing > > *include: > > > > in open PRs in docs). * > > > > > > > > *NOTE! UV is now mandatory and a lot of code is gone thanks to that. > > > > Breeze will also refuse to work if uv is not installed.* > > > > > > > > It took a bit of time, but we are in a much more standard and better > > > shape > > > > now - and as a side effect (which was intended but I had to implement > > it > > > as > > > > part of that monster PR to fix documentation) we now have a much > > simpler > > > > (more guidelines are coming) way to iterate on doc building. > > > > > > > > *Few important things first: * > > > > > > > > 1) Make sure to rebase your PRs, Run `breeze image build`, Rnu `uv > > sync`. > > > > Due to the way git handles things - you ** might ** have some > dangling > > > > generated directories in your repo and they might cause some > problems. > > > Run > > > > "git status" after rebase and see you have some files you need to > > delete > > > > (manually) > > > > > > > > 2) If you are brave enough - you might want to run `breeze doctor` > and > > > > cleanup git repo - it should clean all files that should be removed, > > but > > > > also it might remove some of your custom configurations and files you > > > > created, > > > > > > > > 3) Generally everything should work as it worked before with breeze > > (for > > > > example `breeze build-docs` command works as before. But a number of > > > > folders/distributions/code (not airflow nore providers directly) were > > > > moved/updated. For now you can just continue to build docs as before > - > > > with > > > > breeze. But simpler/faster ways are coming as follow up. > > > > > > > > 4) If you have some new examples or documentation included in your > PRs > > > the > > > > doc build might start falling for you - but this is because > `include::` > > > or > > > > `exampleinclude:` might need to be updated - look at other examples > - > > I > > > > fixed the includes in all providers. More explanation in follow-up > doc > > > > build improvement PR - in the meantime, feel free to ask on slack or > PR > > > for > > > > help. > > > > > > > > *Generated provider_dependencies.json do not need to be updated* > > > > > > > > The "generated/provider_dependencies.json" is no longer committed to > > the > > > > repo - it is .gitignored. We are generating it as-needed on the > flight. > > > It > > > > should be automatically regenerated when you run pre-commits locally > > and > > > > when you build the breeze image. > > > > > > > > There might be some cases when we add dependencies and you will need > to > > > > regenerate it but that should happen automatically as needed. > > > > > > > > *New, updated folders* > > > > > > > > The change are mostly with these: > > > > > > > > ./dev/pyproject.toml > > > > ./devel-common/pyproject.toml > > > > ./doc > > > > ./docker-stack-docs > > > > ./providers-summary-docs > > > > > > > > *More explanation for distributions/folder changes* > > > > > > > > The dev is now a separate distribution with its own pyproject.toml > > > > dependencies that are used for all the release management and general > > dev > > > > housekeeping. This is different from "devel-common" which is a common > > > > package with a lot of common code reused for tests and builds > > (including > > > > doc builds scripts are using). > > > > > > > > We might want to change names later, as they are confusing but I will > > > > leave that discussion for later when we complete all the isolation > > work - > > > > with those changes I implemented it will be very easy (as opposed to > > how > > > it > > > > was) to move those distributions around - and if we will want to > > > > restructure it again, that will be a very simple move. > > > > > > > > *Improved doc structure* > > > > > > > > This is because we finally made the last step - where the "doc" code > is > > > > moved to "devel-common" and we are importing it from there, also each > > > > distribution has it's own "conf.py" and that makes modifying docs > > > building > > > > and our sphinx building scripts MUCH more readable ana mangeable. The > > > "doc" > > > > code now contains just spelling wordlist and README.md explaining > where > > > to > > > > look for the documentation. > > > > > > > > Those new top-level folders ("docker-stack-docs" and > > > > "providers-summary-docs") - contain the two independent pieces of our > > > > documentation ("docker-stack" and "apache-airflow-providers" package > > > names > > > > from "breeze build-docs" command. > > > > > > > > The REALLY nice thing now is that the doc files are not copied > between > > > > places and each of the docs folder (including each of the 90 > providers > > > docs > > > > folder) has it's own "conf.py", which makes the "docs" folder for > them > > > > "isolated" and "fully self-contained" - which will open up the ways > on > > > much > > > > more friendly sphinx workflows. Wait for it ! Coming **just in time** > > > when > > > > we most need it - when we want to start updating the docs for Airflow > > 3. > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
