Re: [PATCH] kernel-doc: better handle '::' sequences
On Tue, Mar 30, 2021 at 1:07 PM Jani Nikula wrote: > > FWIW, and this should be obvious, I think going with what's natural for > documenting Rust source code is the right choice. Markdown as parsed by > rustdoc. People will expect Rust documentation comments to just work, > without some kernel specific gotchas. Don't try to reinvent the wheel > here, it's a dead end. Agreed! > The interesting question is, I think, figuring out if rustdoc output > could be incorporated into Sphinx documentation, and how. It would be > pretty disappointing if we ended up with two documentation silos based > on the module implementation language. I want to have the Rust docs linked from Sphinx and uploaded as usual to kernel.org etc. However, please note that the implementation language implies a lot of things, not just the "implementation language", if that makes sense. For instance, if you write your module in Rust, the idea is that you use the Rust infrastructure and exposed abstractions -- not that you call C kernel functions on your own. > At the moment it seems to me rustdoc can only output HTML, and that > seems pretty deeply ingrained in the tool. AFAICT, there isn't an > intermediate phase where it would be trivial to output the documentation > in Markdown (though I don't really know Rust and I only had a cursory > look at librustdoc). And even if it were possible, with Markdown you'd > have the issues with conflicting Markdown flavours, what's supported by > rustdoc vs. commonmark.py used by Sphinx. Please note that `rustdoc` generates HTML that is intended for Rust code, i.e. generating an intermediate format to then generate HTML from Sphinx would make the Rust docs worse, unless the mapping is perfect (but, at that point, why not just keep the standard Rust docs?). > Perhaps the bare minimum is running rustdoc first, and generating the > results into Sphinx static pages [1], to make them part of the > whole. Even if the HTML style might be different. Perhaps it would be I don't think it is the "bare minimum", I think this is the optimal solution! :-) It is also not just about the style. The Rust docs are organized for Rust code, the search functionality is meant for it, etc. > possible to come up with a Sphinx extensions to make it convenient to > reference content in the rustdoc generated HTML from reStructuredText. For C -> Rust links, the plan I suggested to Jonathan was to have Sphinx generate a text file with (reference, URL) pairs that then `rustdoc` can use as links (e.g. to link to, say, the docs for `printk`). I also discussed it with the `rustdoc` maintainers, and they thought it would be an interesting proposal, so I agreed to make an RFC for it (note that it can be useful not just for Rust docs that need to refer to C code, but also for any other kind of external content, e.g. imagine a math library referencing a set of papers, books, etc. without having to re-write the URL everywhere). Cheers, Miguel
Re: [PATCH] kernel-doc: better handle '::' sequences
On Tue, Mar 30, 2021 at 02:43:45PM +0200, Markus Heiser wrote: > > Am 30.03.21 um 13:35 schrieb Jani Nikula: > > > If the introduction were "/*rST" instead of "/**", would we have > > > consensus? It gives us a path to let people intermix kernel-doc and > > > hawkmoth comments in the same file, which would be amazing. > > > If you want to allow two syntaxes for documentation comments (current > > kernel-doc and pure reStructuredText with just the comment markers and > > indentation removed) I think the natural first step would be to modify > > kernel-doc the perl script to support that. It would probably even be > > trivial. > > My 2cent: to tag the markup of the documentation, in python they > use a variable named __docformat__ [PEP-258] / e.g.: > > __docformat__ = "restructuredtext en" > > [PEP-258] https://www.python.org/dev/peps/pep-0258/#choice-of-docstring-format I don't think we need to do that. We can use .. kernel-doc:: foo.c to indicate the comments are in kernel-doc format and .. hawkmoth:: bar.c to indicate the comments are in hawkmoth format. Of course, that means we have to choose for an entire .c file whether it's in hawkmoth or kernel-doc format, but that's also true for pep-258. > > Perhaps the bare minimum is running rustdoc first, and generating the > > results into Sphinx static pages [1], to make them part of the > > whole. Even if the HTML style might be different. > > Cross referencing will be problematic, I think. That would be a second step. I'd rather see the rst files gain the ability to have: .. rustdoc:: quux.rs to bring the markdown into the sphinx system.
Re: [PATCH] kernel-doc: better handle '::' sequences
Am 30.03.21 um 13:35 schrieb Jani Nikula: If the introduction were "/*rST" instead of "/**", would we have consensus? It gives us a path to let people intermix kernel-doc and hawkmoth comments in the same file, which would be amazing. If you want to allow two syntaxes for documentation comments (current kernel-doc and pure reStructuredText with just the comment markers and indentation removed) I think the natural first step would be to modify kernel-doc the perl script to support that. It would probably even be trivial. My 2cent: to tag the markup of the documentation, in python they use a variable named __docformat__ [PEP-258] / e.g.: __docformat__ = "restructuredtext en" [PEP-258] https://www.python.org/dev/peps/pep-0258/#choice-of-docstring-format > Perhaps the bare minimum is running rustdoc first, and generating the > results into Sphinx static pages [1], to make them part of the > whole. Even if the HTML style might be different. Cross referencing will be problematic, I think. -- Markus --
Re: [PATCH] kernel-doc: better handle '::' sequences
On Mon, 29 Mar 2021, Matthew Wilcox wrote: > On Mon, Mar 29, 2021 at 09:33:30PM +0300, Jani Nikula wrote: >> On Mon, 29 Mar 2021, Matthew Wilcox wrote: >> > So here's my "modest proposal": >> > >> > - Similar to our ".. kernel-doc::" invocation in .rst files, handle >> >".. rustdoc::" (insert weeks of hacking here) >> > - Now add ".. rst-doc::" which parses .c files like [1] kernel-doc >> >does, but interprets a different style of comment and actually does >> >most of the repetitive boring bits for you. >> >> As a hobby, I've written a Sphinx extension to use Clang to parse the >> code and extract pure reStructuredText documentation comments with >> minimal conversions [1]. No additional syntax. Just use reStructuredText >> for everything instead of inventing your own. >> >> I'm not proposing to use that in kernel, at all. It was more like a >> diversion from the kernel documentation. > > Actually, that looks like my proposal, except that it uses the same /** > as kernel-doc, so you can't tell whether a comment is intended to be > interpreted by kernel-doc or hawkmoth. > > https://github.com/jnikula/hawkmoth/blob/master/test/example-70-function.c > > If the introduction were "/*rST" instead of "/**", would we have > consensus? It gives us a path to let people intermix kernel-doc and > hawkmoth comments in the same file, which would be amazing. If you want to allow two syntaxes for documentation comments (current kernel-doc and pure reStructuredText with just the comment markers and indentation removed) I think the natural first step would be to modify kernel-doc the perl script to support that. It would probably even be trivial. Hawkmoth uses Clang for parsing, with none of the kernel specific stuff that kernel-doc has, such as EXPORT_SYMBOL(). It makes sense for a pet project with a clean break. I don't know if anyone has the bandwidth or desire to re-implement the kernel specific stuff on top of Clang. (I know I don't, I started the project because I wanted that clean break to begin with!) The real question is, is it a good idea to support multiple formats at all? (N.b. I'm not a fan of extending the kernel-doc syntax either.) BR, Jani. > >> But based on my experience with the old and new kernel documentation >> systems and the hobby one, the one takeaway is to not create new >> syntaxes, grammars, parsers, or preprocessors to be maintained by the >> kernel community. Just don't. Take what's working and supported by other >> projects, and add the minimal glue using Sphinx extensions to put it >> together, and no more. >> >> Of course, we couldn't ditch kernel-doc the script, but we managed to >> trim it down quite a bit. OTOH, there have been a number of additions >> outside of Sphinx in Makefiles and custom tools in various languages >> that I'm really not happy about. It's all too reminiscient of the old >> DocBook toolchain, while Sphinx was supposed to be the one tool to tie >> it all together, partially chosen because of the extension support. >> >> >> BR, >> Jani. >> >> >> [1] https://github.com/jnikula/hawkmoth >> >> >> > >> > For example, xa_load: >> > >> > /** >> > * xa_load() - Load an entry from an XArray. >> > * @xa: XArray. >> > * @index: index into array. >> > * >> > * Context: Any context. Takes and releases the RCU lock. >> > * Return: The entry at @index in @xa. >> > */ >> > void *xa_load(struct xarray *xa, unsigned long index) >> > >> > //rST >> > // Load an entry from an XArray. >> > // >> > // :Context: Any context. Takes and releases the RCU lock. >> > // :Return: The entry in `xa` at `index`. >> > void *xa_load(struct xarray *xa, unsigned long index) >> > >> > (more complex example below [2]) >> > >> > Things I considered: >> > >> > - Explicitly document that this is rST markup instead of Markdown or >> >whatever. >> > - Don't repeat the name of the function. The tool can figure it out. >> > - Don't force documenting each parameter. Often they are obvious >> >and there's really nothing interesting to say about the parameter. >> >Witness the number of '@foo: The foo' (of type struct foo) that we >> >have scattered throughout the tree. It's not that the documenter is >> >lazy, it's that there's genuinely nothing to say here. >> > - Use `interpreted text` to refer to parameters instead of *emphasis* or >> >**strong emphasis**. The tool can turn that into whatever markup >> >is appropriate. >> > - Use field lists for Context and Return instead of sections. The markup >> >is simpler to use, and I think the rendered output is better. >> > >> > [1] by which i mean "in a completely different way from, but similar in >> > concept" >> > >> > [2] More complex example: >> > >> > /** >> > * xa_store() - Store this entry in the XArray. >> > * @xa: XArray. >> > * @index: Index into array. >> > * @entry: New entry. >> > * @gfp: Memory allocation flags. >> > * >> > * After this function returns, loads from
Re: [PATCH] kernel-doc: better handle '::' sequences
On Mon, 29 Mar 2021, Miguel Ojeda wrote: > On Thu, Mar 25, 2021 at 11:18 PM Matthew Wilcox wrote: >> >> The rust code is alredy coming though ... >> >> rust/kernel/buffer.rs:/// A pre-allocated buffer that implements >> [`core::fmt::Write`]. >> >> so now we have three formats. Markdown and RST are _very_ similar, but >> not identical [1]. Oh, and even better we now have three distinct tools -- >> kerneldoc, rustdoc and sphinx. Have the rust people reached out to you >> about integrating the various docs? > > Yeah, I reached out to Jonathan a few weeks ago to discuss how we will > approach this, because I knew using `rustdoc` and Markdown could be > contentious ;-) > > This is the solution we decided to go for the RFC but, of course, > nothing is set in stone: > > 1. The "out-of-line" docs in `Documentation/rust/`: these will be in > RST as usual [*]. However, please note this does not include APIs or > anything like that, as it is done in the C side. Only a few "general > documents" that don't fit anywhere else are kept here. > > 2. The "inline" docs in Rust source files: these will be parsed by > `rustdoc` and written in Markdown. These will contain the majority of > the Rust documentation. `rustdoc` is designed for Rust code, and > internally uses the Rust compiler, which comes with a number of > advantages. > > The generated HTML docs will be showcased in the RFC. It is my hope > that this way we get feedback on them and see if people agree this > approach is worth keeping. We have put an effort (and I have been > annoying contributors enough to that end :-) to provide high-quality > documentation from the get-go. > > Please note that we chose this way knowing well that inconsistency and > adding "yet one more tool" needs to come with big advantages to > offset. We think it is the best approach nevertheless! > > [*] I discussed with Jonathan using Markdown since Sphinx supports it. > The main advantage would be easier refactoring of comments between the > out- and inline docs. But this is very minor, thus mixing two formats > inside `Documentation/` does not seem like worth it. FWIW, and this should be obvious, I think going with what's natural for documenting Rust source code is the right choice. Markdown as parsed by rustdoc. People will expect Rust documentation comments to just work, without some kernel specific gotchas. Don't try to reinvent the wheel here, it's a dead end. The interesting question is, I think, figuring out if rustdoc output could be incorporated into Sphinx documentation, and how. It would be pretty disappointing if we ended up with two documentation silos based on the module implementation language. At the moment it seems to me rustdoc can only output HTML, and that seems pretty deeply ingrained in the tool. AFAICT, there isn't an intermediate phase where it would be trivial to output the documentation in Markdown (though I don't really know Rust and I only had a cursory look at librustdoc). And even if it were possible, with Markdown you'd have the issues with conflicting Markdown flavours, what's supported by rustdoc vs. commonmark.py used by Sphinx. Perhaps the bare minimum is running rustdoc first, and generating the results into Sphinx static pages [1], to make them part of the whole. Even if the HTML style might be different. Perhaps it would be possible to come up with a Sphinx extensions to make it convenient to reference content in the rustdoc generated HTML from reStructuredText. BR, Jani. [1] https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_static_path -- Jani Nikula, Intel Open Source Graphics Center
Re: [PATCH] kernel-doc: better handle '::' sequences
On Thu, Mar 25, 2021 at 11:18 PM Matthew Wilcox wrote: > > The rust code is alredy coming though ... > > rust/kernel/buffer.rs:/// A pre-allocated buffer that implements > [`core::fmt::Write`]. > > so now we have three formats. Markdown and RST are _very_ similar, but > not identical [1]. Oh, and even better we now have three distinct tools -- > kerneldoc, rustdoc and sphinx. Have the rust people reached out to you > about integrating the various docs? Yeah, I reached out to Jonathan a few weeks ago to discuss how we will approach this, because I knew using `rustdoc` and Markdown could be contentious ;-) This is the solution we decided to go for the RFC but, of course, nothing is set in stone: 1. The "out-of-line" docs in `Documentation/rust/`: these will be in RST as usual [*]. However, please note this does not include APIs or anything like that, as it is done in the C side. Only a few "general documents" that don't fit anywhere else are kept here. 2. The "inline" docs in Rust source files: these will be parsed by `rustdoc` and written in Markdown. These will contain the majority of the Rust documentation. `rustdoc` is designed for Rust code, and internally uses the Rust compiler, which comes with a number of advantages. The generated HTML docs will be showcased in the RFC. It is my hope that this way we get feedback on them and see if people agree this approach is worth keeping. We have put an effort (and I have been annoying contributors enough to that end :-) to provide high-quality documentation from the get-go. Please note that we chose this way knowing well that inconsistency and adding "yet one more tool" needs to come with big advantages to offset. We think it is the best approach nevertheless! [*] I discussed with Jonathan using Markdown since Sphinx supports it. The main advantage would be easier refactoring of comments between the out- and inline docs. But this is very minor, thus mixing two formats inside `Documentation/` does not seem like worth it. Cheers, Miguel
Re: kerneldoc and rust (was [PATCH] kernel-doc: better handle '::' sequences)
On Mon, Mar 29, 2021 at 9:34 PM Jonathan Corbet wrote: > > If we're going to talk about incorporating Rust into the doc system, we > should probably include some Rust folks - thus, I'm adding Miguel. Thanks a lot Jonathan for the Cc! I will read and answer the other emails. Cheers, Miguel
Re: kerneldoc and rust (was [PATCH] kernel-doc: better handle '::' sequences)
Matthew Wilcox writes: If we're going to talk about incorporating Rust into the doc system, we should probably include some Rust folks - thus, I'm adding Miguel. > On Thu, Mar 25, 2021 at 04:30:32PM -0600, Jonathan Corbet wrote: >> Matthew Wilcox writes: >> >> We did come to the mutual agreement that teaching kernel-doc to parse >> Rust code as well was not an ideal solution. Probably there will be >> some sort of tool to translate between rustdoc and our sphinx setup. >> Beyond that, we'll see how it goes. > > In the spirit of groping around for the best solution, I did some looking > around at various options, including using rustdoc for .c files (that > uses Markdown, which appears to be strictly worse than rST for our > purposes). > > So here's my "modest proposal": > > - Similar to our ".. kernel-doc::" invocation in .rst files, handle >".. rustdoc::" (insert weeks of hacking here) > - Now add ".. rst-doc::" which parses .c files like [1] kernel-doc >does, but interprets a different style of comment and actually does >most of the repetitive boring bits for you. > > For example, xa_load: > > /** > * xa_load() - Load an entry from an XArray. > * @xa: XArray. > * @index: index into array. > * > * Context: Any context. Takes and releases the RCU lock. > * Return: The entry at @index in @xa. > */ > void *xa_load(struct xarray *xa, unsigned long index) > > //rST > // Load an entry from an XArray. > // > // :Context: Any context. Takes and releases the RCU lock. > // :Return: The entry in `xa` at `index`. > void *xa_load(struct xarray *xa, unsigned long index) > > (more complex example below [2]) > > Things I considered: > > - Explicitly document that this is rST markup instead of Markdown or >whatever. > - Don't repeat the name of the function. The tool can figure it out. That worries me a wee bit just because a common source of problems is kerneldoc comments becoming separated from the functions they describe over time. We finally have tooling to notice that; this seems like a step in the other direction. > - Don't force documenting each parameter. Often they are obvious >and there's really nothing interesting to say about the parameter. >Witness the number of '@foo: The foo' (of type struct foo) that we >have scattered throughout the tree. It's not that the documenter is >lazy, it's that there's genuinely nothing to say here. ...another failure mode is developers adding parameters and not documenting them; this would mask that problem too. > - Use `interpreted text` to refer to parameters instead of *emphasis* or >**strong emphasis**. The tool can turn that into whatever markup >is appropriate. > - Use field lists for Context and Return instead of sections. The markup >is simpler to use, and I think the rendered output is better. > > [1] by which i mean "in a completely different way from, but similar in > concept" > > [2] More complex example: > > /** > * xa_store() - Store this entry in the XArray. > * @xa: XArray. > * @index: Index into array. > * @entry: New entry. > * @gfp: Memory allocation flags. > * > * After this function returns, loads from this index will return @entry. > * Storing into an existing multi-index entry updates the entry of every > index. > * The marks associated with @index are unaffected unless @entry is %NULL. > * > * Context: Any context. Takes and releases the xa_lock. > * May sleep if the @gfp flags permit. > * Return: The old entry at this index on success, xa_err(-EINVAL) if @entry > * cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation > * failed. > */ > void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) > > //rST > // Store an entry in the XArray. > // > // After this function returns, loads from `index` will return `entry`. > // Storing into an existing multi-index entry updates the entry of every > index. > // The marks associated with `index` are unaffected unless `entry` is > ``NULL``. > // > // :Context: Any context. Takes and releases the xa_lock. > //May sleep if the `gfp` flags permit. > // :Return: The old entry at this index on success, xa_err(-EINVAL) if `entry` > //cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation > //failed. > void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) Thanks, jon
Re: [PATCH] kernel-doc: better handle '::' sequences
On Mon, Mar 29, 2021 at 09:33:30PM +0300, Jani Nikula wrote: > On Mon, 29 Mar 2021, Matthew Wilcox wrote: > > So here's my "modest proposal": > > > > - Similar to our ".. kernel-doc::" invocation in .rst files, handle > >".. rustdoc::" (insert weeks of hacking here) > > - Now add ".. rst-doc::" which parses .c files like [1] kernel-doc > >does, but interprets a different style of comment and actually does > >most of the repetitive boring bits for you. > > As a hobby, I've written a Sphinx extension to use Clang to parse the > code and extract pure reStructuredText documentation comments with > minimal conversions [1]. No additional syntax. Just use reStructuredText > for everything instead of inventing your own. > > I'm not proposing to use that in kernel, at all. It was more like a > diversion from the kernel documentation. Actually, that looks like my proposal, except that it uses the same /** as kernel-doc, so you can't tell whether a comment is intended to be interpreted by kernel-doc or hawkmoth. https://github.com/jnikula/hawkmoth/blob/master/test/example-70-function.c If the introduction were "/*rST" instead of "/**", would we have consensus? It gives us a path to let people intermix kernel-doc and hawkmoth comments in the same file, which would be amazing. > But based on my experience with the old and new kernel documentation > systems and the hobby one, the one takeaway is to not create new > syntaxes, grammars, parsers, or preprocessors to be maintained by the > kernel community. Just don't. Take what's working and supported by other > projects, and add the minimal glue using Sphinx extensions to put it > together, and no more. > > Of course, we couldn't ditch kernel-doc the script, but we managed to > trim it down quite a bit. OTOH, there have been a number of additions > outside of Sphinx in Makefiles and custom tools in various languages > that I'm really not happy about. It's all too reminiscient of the old > DocBook toolchain, while Sphinx was supposed to be the one tool to tie > it all together, partially chosen because of the extension support. > > > BR, > Jani. > > > [1] https://github.com/jnikula/hawkmoth > > > > > > For example, xa_load: > > > > /** > > * xa_load() - Load an entry from an XArray. > > * @xa: XArray. > > * @index: index into array. > > * > > * Context: Any context. Takes and releases the RCU lock. > > * Return: The entry at @index in @xa. > > */ > > void *xa_load(struct xarray *xa, unsigned long index) > > > > //rST > > // Load an entry from an XArray. > > // > > // :Context: Any context. Takes and releases the RCU lock. > > // :Return: The entry in `xa` at `index`. > > void *xa_load(struct xarray *xa, unsigned long index) > > > > (more complex example below [2]) > > > > Things I considered: > > > > - Explicitly document that this is rST markup instead of Markdown or > >whatever. > > - Don't repeat the name of the function. The tool can figure it out. > > - Don't force documenting each parameter. Often they are obvious > >and there's really nothing interesting to say about the parameter. > >Witness the number of '@foo: The foo' (of type struct foo) that we > >have scattered throughout the tree. It's not that the documenter is > >lazy, it's that there's genuinely nothing to say here. > > - Use `interpreted text` to refer to parameters instead of *emphasis* or > >**strong emphasis**. The tool can turn that into whatever markup > >is appropriate. > > - Use field lists for Context and Return instead of sections. The markup > >is simpler to use, and I think the rendered output is better. > > > > [1] by which i mean "in a completely different way from, but similar in > > concept" > > > > [2] More complex example: > > > > /** > > * xa_store() - Store this entry in the XArray. > > * @xa: XArray. > > * @index: Index into array. > > * @entry: New entry. > > * @gfp: Memory allocation flags. > > * > > * After this function returns, loads from this index will return @entry. > > * Storing into an existing multi-index entry updates the entry of every > > index. > > * The marks associated with @index are unaffected unless @entry is %NULL. > > * > > * Context: Any context. Takes and releases the xa_lock. > > * May sleep if the @gfp flags permit. > > * Return: The old entry at this index on success, xa_err(-EINVAL) if @entry > > * cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation > > * failed. > > */ > > void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t > > gfp) > > > > //rST > > // Store an entry in the XArray. > > // > > // After this function returns, loads from `index` will return `entry`. > > // Storing into an existing multi-index entry updates the entry of every > > index. > > // The marks associated with `index` are unaffected unless `entry` is > > ``NULL``. > > // > > // :Context: Any context. Takes and releases the xa_lock. > > //
Re: [PATCH] kernel-doc: better handle '::' sequences
On Mon, 29 Mar 2021, Matthew Wilcox wrote: > On Thu, Mar 25, 2021 at 04:30:32PM -0600, Jonathan Corbet wrote: >> Matthew Wilcox writes: >> >> > The rust code is alredy coming though ... >> > >> > rust/kernel/buffer.rs:/// A pre-allocated buffer that implements >> > [`core::fmt::Write`]. >> > >> > so now we have three formats. Markdown and RST are _very_ similar, but >> > not identical [1]. Oh, and even better we now have three distinct tools -- >> > kerneldoc, rustdoc and sphinx. Have the rust people reached out to you >> > about integrating the various docs? >> >> I have talked with them a bit, yes, but without any clear conclusions at >> this point. The Rust world has its own way of doing things with regard >> to documentation, and I don't want to tell them they can't use it in the >> kernel context. So I think there's going to be a certain amount of >> groping around for the best solution. >> >> We did come to the mutual agreement that teaching kernel-doc to parse >> Rust code as well was not an ideal solution. Probably there will be >> some sort of tool to translate between rustdoc and our sphinx setup. >> Beyond that, we'll see how it goes. > > In the spirit of groping around for the best solution, I did some looking > around at various options, including using rustdoc for .c files (that > uses Markdown, which appears to be strictly worse than rST for our > purposes). > > So here's my "modest proposal": > > - Similar to our ".. kernel-doc::" invocation in .rst files, handle >".. rustdoc::" (insert weeks of hacking here) > - Now add ".. rst-doc::" which parses .c files like [1] kernel-doc >does, but interprets a different style of comment and actually does >most of the repetitive boring bits for you. As a hobby, I've written a Sphinx extension to use Clang to parse the code and extract pure reStructuredText documentation comments with minimal conversions [1]. No additional syntax. Just use reStructuredText for everything instead of inventing your own. I'm not proposing to use that in kernel, at all. It was more like a diversion from the kernel documentation. But based on my experience with the old and new kernel documentation systems and the hobby one, the one takeaway is to not create new syntaxes, grammars, parsers, or preprocessors to be maintained by the kernel community. Just don't. Take what's working and supported by other projects, and add the minimal glue using Sphinx extensions to put it together, and no more. Of course, we couldn't ditch kernel-doc the script, but we managed to trim it down quite a bit. OTOH, there have been a number of additions outside of Sphinx in Makefiles and custom tools in various languages that I'm really not happy about. It's all too reminiscient of the old DocBook toolchain, while Sphinx was supposed to be the one tool to tie it all together, partially chosen because of the extension support. BR, Jani. [1] https://github.com/jnikula/hawkmoth > > For example, xa_load: > > /** > * xa_load() - Load an entry from an XArray. > * @xa: XArray. > * @index: index into array. > * > * Context: Any context. Takes and releases the RCU lock. > * Return: The entry at @index in @xa. > */ > void *xa_load(struct xarray *xa, unsigned long index) > > //rST > // Load an entry from an XArray. > // > // :Context: Any context. Takes and releases the RCU lock. > // :Return: The entry in `xa` at `index`. > void *xa_load(struct xarray *xa, unsigned long index) > > (more complex example below [2]) > > Things I considered: > > - Explicitly document that this is rST markup instead of Markdown or >whatever. > - Don't repeat the name of the function. The tool can figure it out. > - Don't force documenting each parameter. Often they are obvious >and there's really nothing interesting to say about the parameter. >Witness the number of '@foo: The foo' (of type struct foo) that we >have scattered throughout the tree. It's not that the documenter is >lazy, it's that there's genuinely nothing to say here. > - Use `interpreted text` to refer to parameters instead of *emphasis* or >**strong emphasis**. The tool can turn that into whatever markup >is appropriate. > - Use field lists for Context and Return instead of sections. The markup >is simpler to use, and I think the rendered output is better. > > [1] by which i mean "in a completely different way from, but similar in > concept" > > [2] More complex example: > > /** > * xa_store() - Store this entry in the XArray. > * @xa: XArray. > * @index: Index into array. > * @entry: New entry. > * @gfp: Memory allocation flags. > * > * After this function returns, loads from this index will return @entry. > * Storing into an existing multi-index entry updates the entry of every > index. > * The marks associated with @index are unaffected unless @entry is %NULL. > * > * Context: Any context. Takes and releases the xa_lock. > * May sleep if the @gfp flags
Re: [PATCH] kernel-doc: better handle '::' sequences
On Thu, Mar 25, 2021 at 04:30:32PM -0600, Jonathan Corbet wrote: > Matthew Wilcox writes: > > > The rust code is alredy coming though ... > > > > rust/kernel/buffer.rs:/// A pre-allocated buffer that implements > > [`core::fmt::Write`]. > > > > so now we have three formats. Markdown and RST are _very_ similar, but > > not identical [1]. Oh, and even better we now have three distinct tools -- > > kerneldoc, rustdoc and sphinx. Have the rust people reached out to you > > about integrating the various docs? > > I have talked with them a bit, yes, but without any clear conclusions at > this point. The Rust world has its own way of doing things with regard > to documentation, and I don't want to tell them they can't use it in the > kernel context. So I think there's going to be a certain amount of > groping around for the best solution. > > We did come to the mutual agreement that teaching kernel-doc to parse > Rust code as well was not an ideal solution. Probably there will be > some sort of tool to translate between rustdoc and our sphinx setup. > Beyond that, we'll see how it goes. In the spirit of groping around for the best solution, I did some looking around at various options, including using rustdoc for .c files (that uses Markdown, which appears to be strictly worse than rST for our purposes). So here's my "modest proposal": - Similar to our ".. kernel-doc::" invocation in .rst files, handle ".. rustdoc::" (insert weeks of hacking here) - Now add ".. rst-doc::" which parses .c files like [1] kernel-doc does, but interprets a different style of comment and actually does most of the repetitive boring bits for you. For example, xa_load: /** * xa_load() - Load an entry from an XArray. * @xa: XArray. * @index: index into array. * * Context: Any context. Takes and releases the RCU lock. * Return: The entry at @index in @xa. */ void *xa_load(struct xarray *xa, unsigned long index) //rST // Load an entry from an XArray. // // :Context: Any context. Takes and releases the RCU lock. // :Return: The entry in `xa` at `index`. void *xa_load(struct xarray *xa, unsigned long index) (more complex example below [2]) Things I considered: - Explicitly document that this is rST markup instead of Markdown or whatever. - Don't repeat the name of the function. The tool can figure it out. - Don't force documenting each parameter. Often they are obvious and there's really nothing interesting to say about the parameter. Witness the number of '@foo: The foo' (of type struct foo) that we have scattered throughout the tree. It's not that the documenter is lazy, it's that there's genuinely nothing to say here. - Use `interpreted text` to refer to parameters instead of *emphasis* or **strong emphasis**. The tool can turn that into whatever markup is appropriate. - Use field lists for Context and Return instead of sections. The markup is simpler to use, and I think the rendered output is better. [1] by which i mean "in a completely different way from, but similar in concept" [2] More complex example: /** * xa_store() - Store this entry in the XArray. * @xa: XArray. * @index: Index into array. * @entry: New entry. * @gfp: Memory allocation flags. * * After this function returns, loads from this index will return @entry. * Storing into an existing multi-index entry updates the entry of every index. * The marks associated with @index are unaffected unless @entry is %NULL. * * Context: Any context. Takes and releases the xa_lock. * May sleep if the @gfp flags permit. * Return: The old entry at this index on success, xa_err(-EINVAL) if @entry * cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation * failed. */ void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) //rST // Store an entry in the XArray. // // After this function returns, loads from `index` will return `entry`. // Storing into an existing multi-index entry updates the entry of every index. // The marks associated with `index` are unaffected unless `entry` is ``NULL``. // // :Context: Any context. Takes and releases the xa_lock. //May sleep if the `gfp` flags permit. // :Return: The old entry at this index on success, xa_err(-EINVAL) if `entry` //cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation //failed. void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp)
Re: [PATCH] kernel-doc: better handle '::' sequences
Matthew Wilcox writes: > The rust code is alredy coming though ... > > rust/kernel/buffer.rs:/// A pre-allocated buffer that implements > [`core::fmt::Write`]. > > so now we have three formats. Markdown and RST are _very_ similar, but > not identical [1]. Oh, and even better we now have three distinct tools -- > kerneldoc, rustdoc and sphinx. Have the rust people reached out to you > about integrating the various docs? I have talked with them a bit, yes, but without any clear conclusions at this point. The Rust world has its own way of doing things with regard to documentation, and I don't want to tell them they can't use it in the kernel context. So I think there's going to be a certain amount of groping around for the best solution. We did come to the mutual agreement that teaching kernel-doc to parse Rust code as well was not an ideal solution. Probably there will be some sort of tool to translate between rustdoc and our sphinx setup. Beyond that, we'll see how it goes. jon
Re: [PATCH] kernel-doc: better handle '::' sequences
On Thu, Mar 25, 2021 at 03:04:00PM -0600, Jonathan Corbet wrote: > Matthew Wilcox writes: > > Well ... > > > > If somebody wants to write a new tool (*) that extracts documentation > > written in a different format, I think that could be done. Because the > > hard part of writing documentation is getting the person who knows the > > code to get everything that's in their brain into words, not really > > the formatting. > > > > If somebody did want to write such a tool, I think we'd also want a > > tool that turns the existing kernel-doc into the new format, because > > maintaining two function-doc formats would be awful. > > Yeah, the thing is that, as long as we're documenting code with > something other than RST, we *do* have two formats, and they interact > with each other in surprising and unwelcome ways. > > I don't really see a fix, though. Even if we come up with the Perfect > New Format™, I don't want to be the one trying to push through the > patches changing tens of thousands of kerneldoc comments over... I can't argue with either of your points. The rust code is alredy coming though ... rust/kernel/buffer.rs:/// A pre-allocated buffer that implements [`core::fmt::Write`]. so now we have three formats. Markdown and RST are _very_ similar, but not identical [1]. Oh, and even better we now have three distinct tools -- kerneldoc, rustdoc and sphinx. Have the rust people reached out to you about integrating the various docs? [1] https://en.wikipedia.org/wiki/Lightweight_markup_language#Comparison_of_lightweight_markup_language_syntax
Re: [PATCH] kernel-doc: better handle '::' sequences
Matthew Wilcox writes: > On Thu, Mar 25, 2021 at 12:51:25PM -0600, Jonathan Corbet wrote: >> Mauro Carvalho Chehab writes: >> >> > Right now, if one of the following headers end with a '::', the >> > kernel-doc script will do the wrong thing: >> > >> >description|context|returns?|notes?|examples? >> > >> > The real issue is with examples, as people could try to write >> > something like: >> > >> >example:: >> > >> >/* Some C code */ >> > >> > and this won't be properly evaluated. So, improve the regex >> > to not catch '\w+::' regex for the above identifiers. >> > >> > Signed-off-by: Mauro Carvalho Chehab >> > --- >> > scripts/kernel-doc | 2 +- >> > 1 file changed, 1 insertion(+), 1 deletion(-) >> >> Ahwouldn't it be nice if kerneldoc comments had just been RST from >> the beginning? I don't think we're fixing that at this point, though, >> so this makes sense; applied. > > Well ... > > If somebody wants to write a new tool (*) that extracts documentation > written in a different format, I think that could be done. Because the > hard part of writing documentation is getting the person who knows the > code to get everything that's in their brain into words, not really > the formatting. > > If somebody did want to write such a tool, I think we'd also want a > tool that turns the existing kernel-doc into the new format, because > maintaining two function-doc formats would be awful. Yeah, the thing is that, as long as we're documenting code with something other than RST, we *do* have two formats, and they interact with each other in surprising and unwelcome ways. I don't really see a fix, though. Even if we come up with the Perfect New Format™, I don't want to be the one trying to push through the patches changing tens of thousands of kerneldoc comments over... jon
Re: [PATCH] kernel-doc: better handle '::' sequences
On Thu, Mar 25, 2021 at 12:51:25PM -0600, Jonathan Corbet wrote: > Mauro Carvalho Chehab writes: > > > Right now, if one of the following headers end with a '::', the > > kernel-doc script will do the wrong thing: > > > > description|context|returns?|notes?|examples? > > > > The real issue is with examples, as people could try to write > > something like: > > > > example:: > > > > /* Some C code */ > > > > and this won't be properly evaluated. So, improve the regex > > to not catch '\w+::' regex for the above identifiers. > > > > Signed-off-by: Mauro Carvalho Chehab > > --- > > scripts/kernel-doc | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > Ahwouldn't it be nice if kerneldoc comments had just been RST from > the beginning? I don't think we're fixing that at this point, though, > so this makes sense; applied. Well ... If somebody wants to write a new tool (*) that extracts documentation written in a different format, I think that could be done. Because the hard part of writing documentation is getting the person who knows the code to get everything that's in their brain into words, not really the formatting. If somebody did want to write such a tool, I think we'd also want a tool that turns the existing kernel-doc into the new format, because maintaining two function-doc formats would be awful. https://blog.golang.org/godoc would be my preferred format ... the less information repeated from the actual function, the better. But if we're actually going to have rust in the kernel, perhaps rustdoc is better. https://doc.rust-lang.org/beta/rust-by-example/meta/doc.html (*) because nobody actually likes kernel-doc.pl, right?
Re: [PATCH] kernel-doc: better handle '::' sequences
Mauro Carvalho Chehab writes: > Right now, if one of the following headers end with a '::', the > kernel-doc script will do the wrong thing: > > description|context|returns?|notes?|examples? > > The real issue is with examples, as people could try to write > something like: > > example:: > > /* Some C code */ > > and this won't be properly evaluated. So, improve the regex > to not catch '\w+::' regex for the above identifiers. > > Signed-off-by: Mauro Carvalho Chehab > --- > scripts/kernel-doc | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Ahwouldn't it be nice if kerneldoc comments had just been RST from the beginning? I don't think we're fixing that at this point, though, so this makes sense; applied. Thanks, jon
[PATCH] kernel-doc: better handle '::' sequences
Right now, if one of the following headers end with a '::', the kernel-doc script will do the wrong thing: description|context|returns?|notes?|examples? The real issue is with examples, as people could try to write something like: example:: /* Some C code */ and this won't be properly evaluated. So, improve the regex to not catch '\w+::' regex for the above identifiers. Signed-off-by: Mauro Carvalho Chehab --- scripts/kernel-doc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/kernel-doc b/scripts/kernel-doc index cb92d0e1e932..0ecd71477a16 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -392,7 +392,7 @@ my $doc_com_body = '\s*\* ?'; my $doc_decl = $doc_com . '(\w+)'; # @params and a strictly limited set of supported section names my $doc_sect = $doc_com . - '\s*(\@[.\w]+|\@\.\.\.|description|context|returns?|notes?|examples?)\s*:(.*)'; + '\s*(\@[.\w]+|\@\.\.\.|description|context|returns?|notes?|examples?)\s*:([^:]*)$'; my $doc_content = $doc_com_body . '(.*)'; my $doc_block = $doc_com . 'DOC:\s*(.*)?'; my $doc_inline_start = '^\s*/\*\*\s*$'; -- 2.30.2
