Amazing news! Thanks for the huge effort Jarek! :) On Wed, 2 Apr 2025, 18:45 Pavankumar Gopidesu, <gopidesupa...@gmail.com> wrote:
> That's awesome Jarek, thank you for this :) > > Regards, > Pavan > > On Wed, Apr 2, 2025 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > You might also want to do *docker system prune* or even *docker system > > prune --all* or run *breeze doctor* to clean-up some stale cache, > images, > > docker volumes > > > > Also IntelliJ/PyCharm users *uv run setup_idea.py* will add some missing > > directories and regenerate your IntelliJ project configuration. > > > > > > > > > > > > On Wed, Apr 2, 2025 at 1:11 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Hello here, > > > > > > As part of the packaging work - I merged the > > > https://github.com/apache/airflow/pull/48223 > > > > > > *TL;DR; Rebase all PRS, rebuild images and uv sync - and > > > hopefully everything should work as before even if a lot of things > moved. > > > Hopefully the move will be largely transparent (except changing > *include: > > > in open PRs in docs). * > > > > > > *NOTE! UV is now mandatory and a lot of code is gone thanks to that. > > > Breeze will also refuse to work if uv is not installed.* > > > > > > It took a bit of time, but we are in a much more standard and better > > shape > > > now - and as a side effect (which was intended but I had to implement > it > > as > > > part of that monster PR to fix documentation) we now have a much > simpler > > > (more guidelines are coming) way to iterate on doc building. > > > > > > *Few important things first: * > > > > > > 1) Make sure to rebase your PRs, Run `breeze image build`, Rnu `uv > sync`. > > > Due to the way git handles things - you ** might ** have some dangling > > > generated directories in your repo and they might cause some problems. > > Run > > > "git status" after rebase and see you have some files you need to > delete > > > (manually) > > > > > > 2) If you are brave enough - you might want to run `breeze doctor` and > > > cleanup git repo - it should clean all files that should be removed, > but > > > also it might remove some of your custom configurations and files you > > > created, > > > > > > 3) Generally everything should work as it worked before with breeze > (for > > > example `breeze build-docs` command works as before. But a number of > > > folders/distributions/code (not airflow nore providers directly) were > > > moved/updated. For now you can just continue to build docs as before - > > with > > > breeze. But simpler/faster ways are coming as follow up. > > > > > > 4) If you have some new examples or documentation included in your PRs > > the > > > doc build might start falling for you - but this is because `include::` > > or > > > `exampleinclude:` might need to be updated - look at other examples - > I > > > fixed the includes in all providers. More explanation in follow-up doc > > > build improvement PR - in the meantime, feel free to ask on slack or PR > > for > > > help. > > > > > > *Generated provider_dependencies.json do not need to be updated* > > > > > > The "generated/provider_dependencies.json" is no longer committed to > the > > > repo - it is .gitignored. We are generating it as-needed on the flight. > > It > > > should be automatically regenerated when you run pre-commits locally > and > > > when you build the breeze image. > > > > > > There might be some cases when we add dependencies and you will need to > > > regenerate it but that should happen automatically as needed. > > > > > > *New, updated folders* > > > > > > The change are mostly with these: > > > > > > ./dev/pyproject.toml > > > ./devel-common/pyproject.toml > > > ./doc > > > ./docker-stack-docs > > > ./providers-summary-docs > > > > > > *More explanation for distributions/folder changes* > > > > > > The dev is now a separate distribution with its own pyproject.toml > > > dependencies that are used for all the release management and general > dev > > > housekeeping. This is different from "devel-common" which is a common > > > package with a lot of common code reused for tests and builds > (including > > > doc builds scripts are using). > > > > > > We might want to change names later, as they are confusing but I will > > > leave that discussion for later when we complete all the isolation > work - > > > with those changes I implemented it will be very easy (as opposed to > how > > it > > > was) to move those distributions around - and if we will want to > > > restructure it again, that will be a very simple move. > > > > > > *Improved doc structure* > > > > > > This is because we finally made the last step - where the "doc" code is > > > moved to "devel-common" and we are importing it from there, also each > > > distribution has it's own "conf.py" and that makes modifying docs > > building > > > and our sphinx building scripts MUCH more readable ana mangeable. The > > "doc" > > > code now contains just spelling wordlist and README.md explaining where > > to > > > look for the documentation. > > > > > > Those new top-level folders ("docker-stack-docs" and > > > "providers-summary-docs") - contain the two independent pieces of our > > > documentation ("docker-stack" and "apache-airflow-providers" package > > names > > > from "breeze build-docs" command. > > > > > > The REALLY nice thing now is that the doc files are not copied between > > > places and each of the docs folder (including each of the 90 providers > > docs > > > folder) has it's own "conf.py", which makes the "docs" folder for them > > > "isolated" and "fully self-contained" - which will open up the ways on > > much > > > more friendly sphinx workflows. Wait for it ! Coming **just in time** > > when > > > we most need it - when we want to start updating the docs for Airflow > 3. > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > >
