Re: [qubes-devel] Should we migrate the documentation to another platform?
On Tue, Sep 28, 2021 at 02:59:03PM +0200, Wojtek Porczyk wrote: > On Mon, Sep 27, 2021 at 04:46:28AM -0700, Andrew David Wong wrote: > > RTD: > > RTD, apart from sphinx, also supports mkdocs, though I've never used it. > > > + Built-in release-specific doc support. > > + Built-in localization support. > > + Supports Git for version control and history. (Presumably, we could > > continue to use signed commits and tags for user and content > > authentication.) > > + GitHub integration. > > + Supports Markdown (natively, AFAIK, though many seem to use RST). We > > currently use GitHub-flavored Markdown, so some conversion may be necessary > > but shouldn't be too bad. > > + Built-in support for downloading/exporting docs in multiple formats for > > offline reading. Direct access to human-readable Markdown source files in > > Git repos should also continue to be possible. > > + Includes search functionality. > > + Widespread use among open-source projects. Established history. Probably > > not going away anytime soon (though it could eventually, e.g., if the > > project behind it ever shuts down). > > + Can be self-hosted. > > ? Unclear exactly how easy it'll be for users to contribute. (Almost > > certainly no more difficult than it is now.) > > "Edit on Github" will be less friendly, because rST is a second-class citizen > on github. Also, markdown is second-class citizen in sphinx. There are many > things that are possible only in rST. > > You can mix both markup languages in a single project (not in single page), > but that would require explicit policy, what should be written in which > language. I've once migrated one github wiki to sphinx just by copying all GFM > pages to spinx and then rewrote them one by one. > > > ? [Subjective] Some people think RTD looks better than a wiki. > > ? Might look different from the rest of our website. Might require a > > subdomain. > > Those two points very much depend on sphinx' theme. Wouldn't you want to port > the current layout to sphinx theme? > > We already use RTD for dev.qubes-os.org (with sphinx, rST and one of the stock > themes) and it works. > > > With sphinx it's sometimes tricky to get rendering reproducible, i.e. `make > html` on your local system renders slightly differently than the "official" > build on rtfd.io. Those differences are largely limited to roles and > glossaries, but are still noticeable. (At least that's my experience with > sphinx-rtd-theme in another project. For compatibility with Ubuntu 18.04, said > project is still stuck with Sphinx 1.8, as available in apt repo, because we > also render manpages using sphinx). > > >From my limited experience, it will be *more* difficult for users to contribute, unless we accept text rewrite by email, and process it ourselves. I agree with Wojtek: on the basis of my limited use, there can be significant differences between local and RTD builds. I'd also ask Nina for her view on RTD and what theme to consider - there is a useful comparison page at https://rtd-sphinx-theme-sample-project.readthedocs.io/en/latest/ with some decent themes. -- You received this message because you are subscribed to the Google Groups "qubes-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/YVMgq/odQ/f2p/TZ%40thirdeyesecurity.org.
Re: [qubes-devel] Should we migrate the documentation to another platform?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Mon, Sep 27, 2021 at 04:46:28AM -0700, Andrew David Wong wrote: > RTD: RTD, apart from sphinx, also supports mkdocs, though I've never used it. > + Built-in release-specific doc support. > + Built-in localization support. > + Supports Git for version control and history. (Presumably, we could > continue to use signed commits and tags for user and content > authentication.) > + GitHub integration. > + Supports Markdown (natively, AFAIK, though many seem to use RST). We > currently use GitHub-flavored Markdown, so some conversion may be necessary > but shouldn't be too bad. > + Built-in support for downloading/exporting docs in multiple formats for > offline reading. Direct access to human-readable Markdown source files in > Git repos should also continue to be possible. > + Includes search functionality. > + Widespread use among open-source projects. Established history. Probably > not going away anytime soon (though it could eventually, e.g., if the > project behind it ever shuts down). > + Can be self-hosted. > ? Unclear exactly how easy it'll be for users to contribute. (Almost > certainly no more difficult than it is now.) "Edit on Github" will be less friendly, because rST is a second-class citizen on github. Also, markdown is second-class citizen in sphinx. There are many things that are possible only in rST. You can mix both markup languages in a single project (not in single page), but that would require explicit policy, what should be written in which language. I've once migrated one github wiki to sphinx just by copying all GFM pages to spinx and then rewrote them one by one. > ? [Subjective] Some people think RTD looks better than a wiki. > ? Might look different from the rest of our website. Might require a > subdomain. Those two points very much depend on sphinx' theme. Wouldn't you want to port the current layout to sphinx theme? We already use RTD for dev.qubes-os.org (with sphinx, rST and one of the stock themes) and it works. With sphinx it's sometimes tricky to get rendering reproducible, i.e. `make html` on your local system renders slightly differently than the "official" build on rtfd.io. Those differences are largely limited to roles and glossaries, but are still noticeable. (At least that's my experience with sphinx-rtd-theme in another project. For compatibility with Ubuntu 18.04, said project is still stuck with Sphinx 1.8, as available in apt repo, because we also render manpages using sphinx). - -- pozdrawiam / best regards Wojtek Porczyk Gramine / Invisible Things Lab I do not fear computers, I fear lack of them. -- Isaac Asimov -BEGIN PGP SIGNATURE- iQIzBAEBCgAdFiEEaO0VFfpr0tEF6hYkv2vZMhA6I1EFAmFTEZYACgkQv2vZMhA6 I1FGhA/+InLJxfraA0+Ohq5EZD90Ljt4Dv+Bj/Wi9/48VkkX9LPE2teYuMJCTf+x QsDMVz5zZopWwFwnuPN8WQJLsQ5Ah5luOM8l0i2zA5UgYdG1gPycGHnJRZTLLlkr 7GvlSmfMjtwj4AZuTqQfrVkUWTQhlBZTSLkuu7/QtJeRaSlWWb990+CDecB35gxo x1XUE6LTbLa/QJPVhREjSMDreAoyfyQEQIGLxytj3Czkl7J1tnsrJ37k8ptHcLPC XyYYEVYIO6tURaIEF5Clh6Rxq5vh7lA7Yjs0q7i7AxTohJQBviQMhR6xuqK0FCnN bEPuRBnqLFK34sEnMHBCxLCRl8vfRUkU0B4WuBhxycgkDMtaF3qDyOKo/lxAn5Pq 11zsjxXvz9zx2Ujmawqi1M09POeNUjx3PfDwVGTCCFeRB0eoar0dpuLRktVarA21 wcZqFCTVy2dcIe2YDK5Hos5dQu1+bZUigSmIbYDptZ3k0ozH60D8WNx8lMa+w4Mm xRs0m5/XPSV/hSci3T86YY2aK0w3eFNInXjbW5Cw7nWGzdqXrDGh860YUD6ip2KB vkZfHNokO173KcwPBnaxciakYjakNMq3/bSf1PwZ01cOIcqULUwi8m3dEaAb+d2F ghaAXQGz43LAOZmQ0MQMcKqNMHXBMtgTvQsdu52mS6aAx02kh2Q= =jGwU -END PGP SIGNATURE- -- You received this message because you are subscribed to the Google Groups "qubes-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/YVMRl1oP7CRf/Nmo%40invisiblethingslab.com.
Re: [qubes-devel] Should we migrate the documentation to another platform?
On Mon, Sep 27, 2021 at 04:46:28AM -0700, Andrew David Wong wrote: > Wiki: > + Familiar to almost everyone thanks to Wikipedia and specialized wikis. > + Easy for users to contribute. > + Built-in version control and history. > + Includes search functionality. > + Even more widespread use throughout the web. Even longer established > history. Practically ubiquitous. Definitely not going away anytime soon. > + Can be self-hosted. > ? Unclear how good the security would be with respect to user and content > authentication. > ? Unclear to what extent localization support is built-in. There are > probably plugins. Unclear how easy Transifex integration would be. > ? Would probably look different from the rest of our website. Might require > a subdomain. > ? [Subjective] Some people think wikis look bad. > - No built-in support for release-specific docs (AFAIK). > - No (common, standard, native) Git support. > - Native wiki syntax is different from Markdown. Markdown plugins are > available but can be problematic. Markdown can be converted into wiki syntax > using, e.g., pandoc, but will probably still require some manual cleanup. > However, once this conversion is done, contributions could be written > natively in wiki syntax with no ongoing conversions required. > - Probably wouldn't easily support direct access to human-readable source > files. Offline docs might require a local wiki instance. - Attracts lots of spammers, such that you either have to spend a lot of time managing approved edits and approved editors, largely defeating the purpose of a wiki. > Overall, I'm leaning toward RTD, but I don't yet know what it'll be like to > maintain the docs there, so I'd like to experiment with it a bit before > forming an opinion. You can self-host RTD-built documentation, but this does defeat a lot of builtin nice-to-haves like github integration. At least with RTD you always have an option of switching to self-rendering without any loss of functionality. I heartily recommend RTD. We stopped accepting new wikis on the kernel.org side of things because they almost inevitably become a spam-haven [1]. I believe wikis as a concept are largely dead except for very large projects (wikipedia, wikia, etc) that have the resources to keep spammers away. -K [1] https://korg.docs.kernel.org/docs.html -- You received this message because you are subscribed to the Google Groups "qubes-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/20210927125841.4w5slyz375chhe2y%40meerkat.local.
[qubes-devel] Should we migrate the documentation to another platform?
## Summary Currently, the Qubes documentation is written in plain text Markdown files, version controlled with Git, rendered using Jekyll, and hosted from GitHub Pages. This setup has served us well over the years, but as the demands on the documentation have grown, it has become increasingly difficult for the current setup to meet our needs. Therefore, it may be time to consider migrating to a dedicated platform, such as Read the Docs or a standard wiki. The purpose of this thread is to help us make the best decision for our users and for the project as a whole. ## Pros and cons of the status quo There are good reasons for our current setup: 1. History of changes is transparent and immutable thanks to Git. 2. Authenticity can be verified thanks to PGP-signed commits and tags. 3. Unauthorized changes are prohibited thanks to signature-checking. 4. Plain text files can be read directly, offline, even in a terminal. 5. Full control over web-view styling with custom CSS. And more... However, there are also challenges: 6. Difficult for many users to contribute. Requires use of GitHub's web editor at the very least or local command-line usage for full control. No WYSIWYG editor. (GitHub offers a Markdown preview, but you have to serve the site locally to see how your changes will really look on the live website.) 7. Serving the website on one's local machine is cumbersome and error-prone. Upstream changes often break the workflow. Even users who have all the required software installed and follow the exact same steps we do sometimes can't get it to work. We used to instruct users to install Jekyll and run it directly to serve the website, but they often couldn't get it to work. We thought that switching to Docker/Podman would solve this "works on my machine but doesn't work on yours" problem, but it still often doesn't work out in practice. (Currently, even I'm having trouble getting it to work consistently on my own development machine.) 8. Requires a complex backend to achieve basic functionality that's standard elsewhere. Since we're hosted from GitHub Pages, we can't use most Jekyll plugins, which means we have to write and maintain custom code for basic things like per-page tables of contents, language switching, and version switching. To support translations, we've had to drastically increase the complexity of all the layouts, includes, and data files that constitute the guts of a Jekyll website. We've had to add a lot of Liquid code in these files to do the kinds of processing we need, and we have to hook it all up to complicated series of Git repos, subrepos, and branches. This makes it very difficult to understand and maintain. Anyone who wants to contribute or help out with maintenance faces a steep learning curve, which discourages contributions and changes. Even an existing expert in Git, Jekyll, Liquid, and Markdown would have to become acquainted with all of our custom code. And most of it remains undocumented, as many of the people who built it had to rush off to other tasks due to lack of time. 9. Related to the previous point, adding new functionality becomes increasingly difficult as the system grows more complex. The latest example is release-specific documentation. There are many small design decisions that have to be made, and it all has to be done in a way that's compatible with translations. We can't simply create a new branch and call it a day, because all the customized guts of the Jekyll system have to be changed to account for multiple versions of documentation, and the documentation files themselves are a mix of release-specific and non-release-specific files which have to be reorganized and classified with new metadata. Jekyll doesn't support variables in YAML, so we can't just add a release variable to our existing YAML files and headers. We have to either duplicate them for every release or fundamentally redesign large swaths of the system with layers of indirection to work around such limitations. Even bulk-manipulation of the YAML headers in all the doc files requires us to write our own special tools, which themselves then have to be maintained. And all of this only gets us to the point where different doc sets can simply *exist* without everything breaking. Exposing them to users in a friendly way would be an entire further stage of the process, in which we'd probably have to write a lot more custom code just to get a simple dropdown menu that allows users to switch between versions. All of this work would probably result in something that's still not as good as existing off-the-shelf solutions backed by dedicated teams who specialize in developing documentation management systems. ## Why now? We've had the current setup for a long time. Why should we consider migrating now? We've considered it in the past and decided against it at the time, so what's changed? The answer is that there are
