Although there have been some comments on the associated PRs, and I’ve merged
some of them, I’m slightly disappointed that there has been no discussion here,
although there have been complaints about one of the problems this is intended
to help with (broken website builds).
I mention in future work (3) that it might be possible to adapt some of this
for docs PR builds. Here’s what I have in mind:
1. Something in the CI detects that there’s a change impacting the docs (or we
just run this for every PR).
2. CI for the PR extracts the main repo, the PR repo, the original branch, and
the PR branch and sends them (how?) to camel-website CI.
3. CI for camel-website takes this information and constructs a snippet like
this:
```
- require: '@djencks/antora-source-map'
source-map:
- url: ‘$ORIGINAL_REPO'
mapped-url: ‘$PR_REPO'
branches:
- branch: $ORIGINAL_BRANCH
mapped-branch: $PR_BRANCH
```
This gets appended to the antora playbook and the antora build runs. (or a full
build)
To make it quicker, if a current site-manifest is available, we could also
append something for @djencks/antora-source-watch and
@djencks/antora-site-manifest to only build the stuff from the changed repo
(although this can often be tricky due to interrelationships between projects,
e.g. camel and camel-spring-boot).
Conceptually this seems pretty simple, but I don’t know enough about GitHub
actions to know whether it can be done or how to do it. I’ve found the GitHub
actions docs hard to understand. If anyone wants to implement this or provide
advice about how to set up the information transfer and triggers that would
make it much more likely to happen.
David Jencks
> On Dec 14, 2021, at 6:38 PM, David Jencks <david.a.jen...@gmail.com> wrote:
>
> I have a POC for a reliable way to locally build small parts of the
> (Antora-built) website, e.g. a subproject.
>
> Prerequisites:
>
> 1. camel-website must be cloned next to the subproject: e.g. if the “main”
> directory is camel-projects, you’ll have camel-projects/camel-website and
> camel-projects/camel-quarkus.
> Note: it would be possible to have camel-website under the subproject, but
> that would make working on more than one subproject implausible.
>
> 2. You’re using linux, macOS, or another unix-like OS…. there’s a shell
> script.
>
> What to do:
>
> 1. Initially build the camel ui. This is easy on linux systems and difficult
> on macOS; you have to run yarn to recompile some dependencies. It may be
> easier to build in docker.
>
> 2. In the subproject, initially run `./loca-build.sh full` once to perform a
> full Antora build that includes your local subproject state, to generate the
> ‘site-manifest’ needed for the partial build.
>
> 3. After this, run `./local-build.sh` to continuously update the full build
> with changes in the subproject. This starts a local web server which is
> slightly more capable than looking at the file urls.
> This does an initial partial build, and then watches for file changes. The
> whole site is viewable.
>
> Limitations:
>
> 1. At the moment, xref checks from the subproject to the rest of the site are
> validated correctly, but xrefs into the subproject from the rest of the site
> are not. I’m hoping to extend the site-manifest to fix this.
>
> 2. You have to do a full build locally once to construct the site manifest
> and the local content from the parts of the site outside the subproject. This
> takes about 7 minutes on my laptop. If there are significant changes to the
> website you need to redo this local build, and there’s no way to detect when
> this is needed.
>
> 3. You have to build the camel ui once which is difficult on non-linux
> systems. This could easily be fixed by checking in the built ui bundle
> whenever it changes.
>
> Possible changes/future work:
>
> 1. The site-manifest can be built at minimal cost during the official website
> build, and served from the website. It’s about 2.5 MB, so I don’t think it’s
> a significant drain on resources. Doing just this would eliminate the need
> for step (2), building the full site locally, and you’d still have good xref
> checks, but with just this step you’d get a site where only the subproject
> pages are local, and links out of the subproject go to the main site; links
> into the subproject would stay on the main site. I don’t like this much, but
> perhaps it’s completely adequate for most site development.
>
> 2. In addition to (1), the zip-publisher could be added to the build, to zip
> up the whole Antora part of the site; this could be downloaded instead of
> step (2). This would be pretty easy to set up, but the resulting zip is about
> 110MB. I don’t know what infra would think of serving this from the root of
> the website. This would completely eliminate the need for step (2) unless
> you wanted to do a full local build with your changes, perhaps to check xrefs
> into the subproject or to check other unusual interactions.
>
> 3. Some of this may be adaptable to building PRs that change the site
> content, with just a subproject PR and not an additional camel-website PR.
> This depends on how much information can be sent from one GitHub action to
> another in a different repository.
>
> Current state:
>
> I’ve set this up for camel-quarkus main in its current state and
> camel-kamelets main showing how I propose to generate the kamelet pages on
> the fly with Antora. I can set this up for other subprojects on request.
>
> How it works:
>
> There are three Antora extensions, some per-subproject playbook additions,
> and a shell script.
>
> @djencks/antora-source-map uses configuration to modify the playbook
> source/branch info, for each configured source/branch replacing the apache
> GitHub repo with the local clone and branch. (this is what would be used in
> "future (3)’)
>
> @djencks/antora-site-manifest, which is an adaptation and modification of
> something Dan Allen developed, generates the site-manifest in the full build
> and imports it in the partial build.
>
> @djencks/antora-source-watch restricts which files are loaded into the
> content catalog (rather than being represented by the site-manifest entry)
> and sets up a file watcher to detect changes and a web server to show the
> site; on a file change it rebuilds the site. browser-watch refreshes any
> pages you may be looking at from the site.
>
> camel-website has a playbook addition that configures the site manifest
> generation on a full build. This could be added to the playbook if we
> decided to always generate the site-manifest (future (1)).
> There are also a couple of yarn scripts to do full and partial antora builds.
>
> Each subproject has two playbook additions:
> - source-map.yml to configure the source-map extension
> - source-watch.yml to configure the source-watch extension
>
> and a script that concatenates the additions to the main Antora playbook to
> result in local playbooks for full and partial builds, and runs the builds.
>
> Where is it?!?!?
>
> Camel-website issue: https://github.com/apache/camel-website/issues/720
> <https://github.com/apache/camel-website/issues/720>
> camel-website PR https://github.com/apache/camel-website/pull/721
> <https://github.com/apache/camel-website/pull/721>
> camel-quarkus PR https://github.com/apache/camel-quarkus/pull/3385
> <https://github.com/apache/camel-quarkus/pull/3385>
> camel-kamelets PR https://github.com/apache/camel-kamelets/pull/630
> <https://github.com/apache/camel-kamelets/pull/630>
>
> I think you can try this out by making sure your clones are next to one
> another as in prerequisite (1), pulling down my branches, and following the
> instructions.
>
> Comments?
>
> David Jencks
