Hi Zoran, inline after extracting a couple of concrete proposals. 1. I’m going to open a website issue and PR for publishing the site manifest with the site, to enable immediate quick local “xref check” builds without needing to build the full site.
2. I’m going to open a website issue proposing checking in the build UI bundle. This is a different solution to the problem than PR#537[1], and is certainly open to debate. 2. I’m going to open a website issue to move the site publishing to a separate repo 3. I’m going to start on a PR for the user manual to document how to do local builds. My comments on why below... > On Dec 21, 2021, at 1:55 AM, Zoran Regvart <zo...@regvart.com> wrote: > > Hi David, > > On Mon, Dec 20, 2021 at 10:23 PM David Jencks <david.a.jen...@gmail.com> > wrote: >> >> (Mostly general info, everyone please read) >> >> Hi Antonin, >> >> Sorry, somehow I missed this earlier! >> >> This work should help with that issue. I’m finding it really great for >> stuff I do, it’s also really easy to adapt to create a website PR for >> integrating lots of branch changes, cf. >> https://github.com/apache/camel-website/pull/729 >> <https://github.com/apache/camel-website/pull/729>. >> >> I’m going to be working on my “referential integrity” idea next (unless more >> documentation fires break out :-). This will touch every active branch of >> every subproject, and unless someone objects I’m going to install the >> local-build stuff everywhere to make it easier for myself :-), so it should >> soon be pretty easy for everyone to try it on their favorite parts of camel. > > +1 this is great work and it'll help us in the long run. I think > you're doing it carefully and conscientiously taking into account not > to be disruptive. Did we already update the README's, so folk > understand how to use `local-build.sh`? Not yet, but I think this is going well enough to assume we'll keep it. I think the best will be if I have detailed instructions in the “working on the website” page in the user manual and very short directions in each subproject. Would it be a good idea to make the local instructions named so they are obviously instructions, maybe README_local_build.adoc? I never know what to expect from a README :-) > >> It’s not a problem for me since I basically only work on the website, but I >> would imagine two major annoyances of the current system for everyone else >> are: >> >> (1) You have to build the full site at least once, which takes annoyingly >> long (7 minutes?). >> (2) You have to build the UI locally, which is tricky on non-linux systems. >> >> For (1), building the site manifest as part of the regular website build so >> the latest can always be downloaded would make it so you can always do a >> quick local partial build, although you won’t get a very usable local >> website: just the local stuff will be there locally. It would detect the >> same errors as a the current local build, but not xref problems into the >> local changes: this (currently) requires a full build. > > Still, I think this is a great step forward, we can detect some issues > early and quickly, I'm sure we can improve to detect most issues in > subsequent refinements I think there are 3 cases for local builds: 1. You just want to check for errors as quickly as possible. I think this is what most people want to do most of the time. We might even be able to get maven to run this. Having the site manifest served from the live site (or some other location) is needed for this. 2. You are actually working on the docs and want a quick continuous build of a subproject, and don’t mind having to do a full build first (maybe 7-8 minutes). This is what the current `./local-build.sh` does, after a `./local-build.sh full`. This detects xref problems out of the subproject but not into the subproject. 3. You want to check for all xref and other problems with your local changes. This is what the current `.local-build.sh full` does. We don’t support (1) yet, likely by far the most common case. So, I propose we start by publishing the site manifest. I can’t see any downside…. we can always stop. > >> For (2), one simple option is just checking the built UI bundle into git. >> >> A further annoyance that I experience is that the built website is checked >> into a branch of camel-website. This means whenever you pull camel-website, >> you get the latest published website which is a lot of changes, along with >> the tiny changes that might or might not have occurred since your last >> update. > > Nicola touched upon this in PR#537[1], would that be a better option? #537 is about publishing the UI bundle, not whether the asf-site branch to publish the website is in the same repo as the antora and hugo build machinery. I think that #537 would be materially less convenient and slower and harder for everyone. It’s definitely a possibility. Here’s my comparison, checking in bundle vs. publishing bundle - The main build can continue to rely on the yarn workspace coordinates for the bundle, as can local builds… vs…. either the main build and local builds use different playbooks or the main build has to deal with getting the released bundle. - A camel-website clone or pull/update will come with the latest (or at least recent enough) UI for local builds… vs… an additional fetch operation is needed for all builds, which is slower for everyone. However, IMO some way of doing a local build without having to locally build the UI bundle is needed. I think by far the easiest is to check in the built bundle, but I don’t really care how we do it. > >> I would like us to move the published website to a different repo. This >> would also allow us to keep the website history, which would partly solve >> the problem of docs for outdated component versions that we’ve dropped from >> the active website. I also think from some comments about the old svn site >> publishing system that infra expects website history to be preserved, >> although I haven’t asked explicitly. I’ve set this up in a couple other >> projects, and it’s pretty easy to do. > > I've chatted with INFRA a while back about the size of the git > repository that was causing issues with the publish, and the > workaround for that issue was to do what we do now and squash the > commits before pushing. If we preserve history I think we'll end up > with the same issue again. I don't understand the need of having the > history of the website build result preserved, given that the output > is optimized, changes can be seen much clearer and easier in the > website sources. And I don't think there is much difference in having > a separate repository vs a separate branch for publishing. > My main motivation for proposing this now is that we’re going to have everyone working on the docs cloning or pulling the camel-website repo, and no one needs the publishing branch. Even if there are no changes in the antora or hugo stuff, there’s still a several-second download of the latest site content. This is, at least to me, annoying and confusing. I’d like it if, when there are no changes to the stuff that builds the site, nothing is pulled: I can be confident that I don’t have to investigate anything further. Therefore I want to move the published website to a separate repo. If we do this, we have the option of keeping history or continuing to squash it. - Either way, we only need to get the repo at depth 1, with no history. Therefore I think the git traffic and speed would be the same. Does Jenkins have some caching that invalidates this argument? - We’ve discussed somewhere the problem that we want to keep the website build reasonable fast but by pruning old doc versions we make old releases undocumented. If we keep the website history then one solution to this is to check out an appropriate vintage commit and look at it with a local httpd. There’s no practical way to build an old version of the site, and this would give the docs as they existed. Something else we might do is to tag published site commits just before we remove a doc version. Even if we squashed history, this would keep those specific versions available. If we kept history, tags would provide some way to find the revision you are interested in :-). So, I want to move the publishing to a separate repo to make local builds quicker: doing so also provides room for discussing keeping history, which might or might not be a good idea. > 2c > > zoran > > [1] https://github.com/apache/camel-website/pull/537 > -- > Zoran Regvart
