Re: Kernel docs: muddying the waters a bit
Em Fri, 6 May 2016 18:26:10 +0200 Markus Heiser escreveu: > Hi Mauro, > > Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab > : > > Yeah, it looks better, however table truncation seem to be > > happening also on other parts, like the tables on this page: > > > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html > > (original table: > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html) > > This table should contain 32 bits, but only the first 7 bits are shown > > > > and those (among others): > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > > > > Hmm... after looking more carefully, it added a horizontal scroll bar. > > That looks ugly, IMHO, and makes harder to understand its contents. The > > last one, in particular > > (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html), > > is a big table on both horiz and vert dimensions. Try to read how the > > bits are packed on a random line in the middle of the table, like > > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean. > > I know what you mean ;-) ... I'am also unhappy, but I will address this point > later when it goes to finish the layout. > > Currently lets focus on contend and (the two)extensions. OK. > > The table here looks weird (although it is correct): > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html > > (original table: > > https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html) > > > > I validated this and the other tables you mentioned above ... these are > all correct migrated ... it is 1:1 translated from DocBook ... they > might be show different to this what you know from your docbook > toolchain, because in the docbook-html you have no table grids and > wrong cellspans are not clear ... sometimes, like in the last example > you gave: > > > > > > a colspec might ambiguous ... so there is no clear role to migrate. Ok, that's what I was thinking. Ok, this can be fixed later manually, where needed. Of course one way would be to disable grids on those tables, but I would instead fix it. > > > It seems that Sphinx is assuming something like "A4 portrait" for > > the margins, while those big tables would only fit (in PDF) as > > "A4 landscape". > > No, no, no ;-) > > Sphinx assumes nothing about the layout, sphinx and the underlying > docutils mostly juggling with nodes and the writers in e.g. the > html-writer, outputs a clear HTML without any style but with classified > HTML tags. Styling is done in the presentation layer, in HTML, it is > done in CSS. Hmm... Then there's something deadly wrong at CSS template, as it is shown texts only at half of my horizontal res (1920). Probably this is the culpit: .container { margin: 50px auto 40px auto; width: 600px; text-align: center; } width is set to 600px, instead of using a percentage, like 100% (or 90%). > > > I guess the better would be to not limit the right > > margin or to change it where those big tables happen, in order to > > allow PDF generation. > > Generating PDF has nothing to do with generating HTML. To generate > PDF there is a other writer, the latex2e writer, which produce > LaTeX markup from which you build PDF or other printed-like medias. Ok. > > > > > There are also some tables that went wrong. See the Color Sample > > Location table at: > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html > > I'm pretty sure we'll need to fix some cases like that manually. > > I didn't check, but perhaps, in this case, the DocBook were using > > empty columns just to make the table bigger. If so, this is not a > > problem with the conversion, and should be manually fixed later. > > +1 > > Yes, lets do it manually later ... what I have in my POC is a automated > process, it is hard to consider individuals in an automatic process. > Making details *nicer* and making ambiguous markups clear is manually > done in minutes where I need hours to implement this in a automated > process. Yeah, we should not try to fix everything via auto-scripts, and spending time right now with manual fixes will be wasted, as we need to run it at the latest media documentation, as changes might have happened upstream. > > Aside, from: http://docutils.sourceforge.net/docs/peps/pep-0258.html > > Docutils Project Model -- Project components and data flow: > > +---+ > |Docutils: | > | docutils.core.Publisher, | > | docutils.core.publish_*() | >
Re: Kernel docs: muddying the waters a bit
Hi Mauro, Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab : > Yeah, it looks better, however table truncation seem to be > happening also on other parts, like the tables on this page: > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html > (original table: > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html) > This table should contain 32 bits, but only the first 7 bits are shown > > and those (among others): > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > > Hmm... after looking more carefully, it added a horizontal scroll bar. > That looks ugly, IMHO, and makes harder to understand its contents. The > last one, in particular > (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html), > is a big table on both horiz and vert dimensions. Try to read how the > bits are packed on a random line in the middle of the table, like > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean. I know what you mean ;-) ... I'am also unhappy, but I will address this point later when it goes to finish the layout. Currently lets focus on contend and (the two)extensions. > The table here looks weird (although it is correct): > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html > (original table: > https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html) > I validated this and the other tables you mentioned above ... these are all correct migrated ... it is 1:1 translated from DocBook ... they might be show different to this what you know from your docbook toolchain, because in the docbook-html you have no table grids and wrong cellspans are not clear ... sometimes, like in the last example you gave: a colspec might ambiguous ... so there is no clear role to migrate. > It seems that Sphinx is assuming something like "A4 portrait" for > the margins, while those big tables would only fit (in PDF) as > "A4 landscape". No, no, no ;-) Sphinx assumes nothing about the layout, sphinx and the underlying docutils mostly juggling with nodes and the writers in e.g. the html-writer, outputs a clear HTML without any style but with classified HTML tags. Styling is done in the presentation layer, in HTML, it is done in CSS. > I guess the better would be to not limit the right > margin or to change it where those big tables happen, in order to > allow PDF generation. Generating PDF has nothing to do with generating HTML. To generate PDF there is a other writer, the latex2e writer, which produce LaTeX markup from which you build PDF or other printed-like medias. > > There are also some tables that went wrong. See the Color Sample > Location table at: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html > I'm pretty sure we'll need to fix some cases like that manually. > I didn't check, but perhaps, in this case, the DocBook were using > empty columns just to make the table bigger. If so, this is not a > problem with the conversion, and should be manually fixed later. +1 Yes, lets do it manually later ... what I have in my POC is a automated process, it is hard to consider individuals in an automatic process. Making details *nicer* and making ambiguous markups clear is manually done in minutes where I need hours to implement this in a automated process. Aside, from: http://docutils.sourceforge.net/docs/peps/pep-0258.html Docutils Project Model -- Project components and data flow: +---+ |Docutils: | | docutils.core.Publisher, | | docutils.core.publish_*() | +---+ /|\ / | \ 1,3,5 /6 | \ 7 ++ +-+ ++ | READER | > | TRANSFORMER | > | WRITER | ++ +-+ ++ / \\ | / \\ | 2/ 4 \\ 8 | +---+ ++++ | INPUT | | PARSER || OUTPUT | +---+ ++++ This is a bit simplified, because we use sphinx, which has "builders" and sits on top of this architecture. But it might help to see, that processes like reading, transforming and writing are discrete. In short: readers (the reST file reader) are creating node trees, which are transformed by a
Re: Kernel docs: muddying the waters a bit
On Fri, 06 May 2016, Markus Heiser wrote: > Am 06.05.2016 um 17:06 schrieb Jani Nikula : > >> On Fri, 06 May 2016, Markus Heiser wrote: >>> @Jonathan: what do you think? Should I prepare a patch >>> with a basic reST (sphinx) build infrastructure, including >>> >>> * a folder for sphinx docs: >>> >>> ./Documentation/sphinx/ >> >> I'm already working on a patch series taking a different approach. I >> don't think we should hide the documentation under an extra folder named >> after a tool. Actually, I'm strongly opposed to that. > > Could you post a link to a repo? / thanks Very much a work-in-progress https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx I was hoping to polish it a bit more before showing it to the world. > There is no need for concurrency, let's work together on your repo. > Within my POC I realized similar building processes we will need in the > kernel sources ... where you have cascading configuration. A base > configuration which fits for all common cases and (if needed) a > *per-book* configuration. > > At the end, when it comes to generate pdf books/articles, man pages > and e.g. texinfo files out of a sphinx-project you will need a build > infrastructure like this. ... > You will need on sphinx-project for each DocBook and one single > sphinx-project where you collect the .txt to .rst migrated files. Surely you know more about Sphinx than I do, but I specifically would like to include e.g. gpu documentation in the main build. I'm really hoping we can have *additional* configuration files for special cases (only) as needed. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Am 06.05.2016 um 17:06 schrieb Jani Nikula : > On Fri, 06 May 2016, Markus Heiser wrote: >> @Jonathan: what do you think? Should I prepare a patch >> with a basic reST (sphinx) build infrastructure, including >> >> * a folder for sphinx docs: >> >> ./Documentation/sphinx/ > > I'm already working on a patch series taking a different approach. I > don't think we should hide the documentation under an extra folder named > after a tool. Actually, I'm strongly opposed to that. Could you post a link to a repo? / thanks There is no need for concurrency, let's work together on your repo. Within my POC I realized similar building processes we will need in the kernel sources ... where you have cascading configuration. A base configuration which fits for all common cases and (if needed) a *per-book* configuration. At the end, when it comes to generate pdf books/articles, man pages and e.g. texinfo files out of a sphinx-project you will need a build infrastructure like this. > Instead, we should place the Sphinx stuff directly under Documentation, > and have Sphinx recursively pick up all the *.rst files. We should > promote gradually switching to lightweight markup and integration of the > documents into one system. This process should be as little disruptive > as possible. > > If someone wants to convert a .txt document to .rst and get it processed > by Sphinx, it should be as simple as renaming the file, doing the > necessary edits, and adding it to a toctree. Imagine gradually > converting the files under, say, Documentation/kbuild. Why should the > .rst files be moved under another directory? They should stay alongside > the .txt files under the same directory. There's bound to be a lot of > people who'll never use Sphinx, and will expect to find the good old > plain text files (albeit with some markup) where they always were. > Ok, I agree with you in the fact that a additional "sphinx" folder is unrewarding. This means (e.g.) a migrated Documentation/DocBook/gpu book should placed in Documentation/gpu ... but don' try to merge all (Doc-)Books and .txt-files into one sphinx project! You will need on sphinx-project for each DocBook and one single sphinx-project where you collect the .txt to .rst migrated files. Am 06.05.2016 um 17:23 schrieb Mauro Carvalho Chehab : > We won't avoid the need of moving things among directories, as we > have lots of stuff under DocBook/ dir (btw, named after the toolchain). Yes, it is named by the toolchain, but no one reads xml-files. Reading text files is common. > Ok, if we put the .rst files at Documentation, we very likely reduce > the number of renames, but we'll increase the Makefile complexity, > and the risk of breakages. I don't see a great potential of breakages ... if we place every book in a separated folder and have one project which collects the .txt files (see above). --Markus--
Re: Kernel docs: muddying the waters a bit
Em Fri, 6 May 2016 18:06:49 +0300 Jani Nikula escreveu: > On Fri, 06 May 2016, Markus Heiser wrote: > > @Jonathan: what do you think? Should I prepare a patch > > with a basic reST (sphinx) build infrastructure, including > > > > * a folder for sphinx docs: > > > > ./Documentation/sphinx/ > > I'm already working on a patch series taking a different approach. I > don't think we should hide the documentation under an extra folder named > after a tool. Actually, I'm strongly opposed to that. > > Instead, we should place the Sphinx stuff directly under Documentation, > and have Sphinx recursively pick up all the *.rst files. We should > promote gradually switching to lightweight markup and integration of the > documents into one system. This process should be as little disruptive > as possible. We won't avoid the need of moving things among directories, as we have lots of stuff under DocBook/ dir (btw, named after the toolchain). Ok, if we put the .rst files at Documentation, we very likely reduce the number of renames, but we'll increase the Makefile complexity, and the risk of breakages. One alternative would be to put the sphinx stuff on a separate Makefile, but using multiple makefiles on a single dir is not standard at the Kernel. > If someone wants to convert a .txt document to .rst and get it processed > by Sphinx, it should be as simple as renaming the file, doing the > necessary edits, and adding it to a toctree. Imagine gradually > converting the files under, say, Documentation/kbuild. Why should the > .rst files be moved under another directory? They should stay alongside > the .txt files under the same directory. There's bound to be a lot of > people who'll never use Sphinx, and will expect to find the good old > plain text files (albeit with some markup) where they always were. Well, git will show the change as a rename, no matter if the directory name changes or not (except if we keep the rst files with .txt extension), but I agree with you that people will expect to see text files at Documentation, and most will just read it without caring to run Sphinx. -- Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
Em Fri, 6 May 2016 16:27:21 +0200 Markus Heiser escreveu: > Hi all, hi Jonathan, > > Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab > : > > > Em Fri, 6 May 2016 15:32:35 +0200 > > Markus Heiser escreveu: > > > >> Hi Mauro, > >> > >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab > >> : > >> > >>> Markus, > >>> > >>> Em Fri, 6 May 2016 13:23:06 +0200 > >>> Markus Heiser escreveu: > >>> > > In this conf.py you have to *register* your folder with the extensions. > A few words about the flat-table extension and a (future) kernel-doc > one: > >>> > >>> ... > >>> > The flat-table is a pure docutils (the layer below sphinx) extension > which > is not application specific, so I will ask for moving it to the docutils > upstream. > >>> > >>> So, if I understood well, your proposal is to have a conf.py and the > >>> flat-table (plus other extensions) at the Kernel tree. > >> > >> Each book (better call it root-node) is a Sphinx-project, each > >> Sphinx-project need a conf.py file (the build configuration file) > >> in its reST-source tree. > >> > >> * http://www.sphinx-doc.org/en/stable/config.html > >> > >>> Assuming that docutils upstream receives the flat-table extension > >>> (and eventually modifies it), while the new version doesn't arrive > >>> all distros, we'll end by having some developers using the newer > >>> docutils with the extension, plus others using the in-tree one. > >>> > >>> Is there a way to specify at the conf.py what extension variant > >>> should it use, in case of both the in-tree or the docutils have > >>> the same? > >> > >> The build configuration file is a regular python file, you can > >> implement conditions whatever you want/need. > >> > >>> This could be trickier if they end by modifying the extension, > >>> but we can always backport the latest version, if they change the > >>> API. > >> > >> As far as i know, the docutils API is stable since 2002. In the > >> meantime there has been so many application build on it that > >> it is not realistic, you will see a not backward compatibly > >> change. > >> > >> The docutils project is conservative, very conservative, IMO to > >> conservative. > >> > >> Today I'am doubtful if it isn't better I would merge it sphinx > >> upstream. I have to discuss this with some maintainers, but > >> before I have to persuade myself, that all aspects are covered > >> and the implementations are robust. We are at the beginning and > >> we should not fear about every bit which could happen in the future. > >> > >> The sphinx / docutils bottom plate gives us a number of degrees > >> of freedom to find answers to question we have not yet asked. ;-) > > > > Ok. So, from what I understand, once Sphinx support is added at > > Kernel upstream, we could convert the media docbook to > > reST+flat-table extension, adding such extension either on a shared > > place or only for the media DocBook build, together with its > > conf.py. > > > > Yes, in your media-conf you could decide to use a extension. > > > Once it reaches upstream (either sphinx or docutils), we can > > work to make it integrate better with upstream as needed. > > > > Right? > > yes, right :-) > > > If so, I'm ok with merging it as soon as possible. > > If we advice a merge of the flat-table directive we should > bundle this with the (to implement) "kernel-doc" directive ... > > > In reST the directive might look like: > > > > - > > Device Instance and Driver Handling > > === > > > > .. kernel-doc:: drivers/gpu/drm/drm_drv.c > > :doc: driver instance overview > > :exported: > > > > - > > and the patches from my kernel-doc perl script to produce > reST from source code comments. > > With this bundle within the kernel tree we have a good starting > point to compose reST documents from scratch and to migrate book > by book from DocBook to reST. > > I insist to migrate book by book, because there are some > broken books. Broken by that, that some sources have changed > but not the corresponding documentation which use the comments > of these sources ... grap "Ooops" in the builded (xml or rst) docs. > > E.g. I greped the .rst file and found the following Oops in the migrated > books: > > ./books/mtdnand/pubfunctions-000-012.rst:13:Oops > ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops > ./books/s390-drivers/ccw-000-004-003.rst:13:Oops > ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops > ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops > ./books/device-drivers/Basics-000-001-002.rst:13:Oops > ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops > ./books/device-drivers/Basics-000-009-032.rst:13:Oops > ./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops > ./books/gadget/api-000-011-005.rst:13:Oops > ./books/gadget/api-000-011-009.rst:13:Oops > ./books/gadget/api-000-011-007.rst:13:
Re: Kernel docs: muddying the waters a bit
On Fri, 06 May 2016, Markus Heiser wrote: > @Jonathan: what do you think? Should I prepare a patch > with a basic reST (sphinx) build infrastructure, including > > * a folder for sphinx docs: > > ./Documentation/sphinx/ I'm already working on a patch series taking a different approach. I don't think we should hide the documentation under an extra folder named after a tool. Actually, I'm strongly opposed to that. Instead, we should place the Sphinx stuff directly under Documentation, and have Sphinx recursively pick up all the *.rst files. We should promote gradually switching to lightweight markup and integration of the documents into one system. This process should be as little disruptive as possible. If someone wants to convert a .txt document to .rst and get it processed by Sphinx, it should be as simple as renaming the file, doing the necessary edits, and adding it to a toctree. Imagine gradually converting the files under, say, Documentation/kbuild. Why should the .rst files be moved under another directory? They should stay alongside the .txt files under the same directory. There's bound to be a lot of people who'll never use Sphinx, and will expect to find the good old plain text files (albeit with some markup) where they always were. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Hi all, hi Jonathan, Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab : > Em Fri, 6 May 2016 15:32:35 +0200 > Markus Heiser escreveu: > >> Hi Mauro, >> >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab >> : >> >>> Markus, >>> >>> Em Fri, 6 May 2016 13:23:06 +0200 >>> Markus Heiser escreveu: >>> In this conf.py you have to *register* your folder with the extensions. A few words about the flat-table extension and a (future) kernel-doc one: >>> >>> ... >>> The flat-table is a pure docutils (the layer below sphinx) extension which is not application specific, so I will ask for moving it to the docutils upstream. >>> >>> So, if I understood well, your proposal is to have a conf.py and the >>> flat-table (plus other extensions) at the Kernel tree. >> >> Each book (better call it root-node) is a Sphinx-project, each >> Sphinx-project need a conf.py file (the build configuration file) >> in its reST-source tree. >> >> * http://www.sphinx-doc.org/en/stable/config.html >> >>> Assuming that docutils upstream receives the flat-table extension >>> (and eventually modifies it), while the new version doesn't arrive >>> all distros, we'll end by having some developers using the newer >>> docutils with the extension, plus others using the in-tree one. >>> >>> Is there a way to specify at the conf.py what extension variant >>> should it use, in case of both the in-tree or the docutils have >>> the same? >> >> The build configuration file is a regular python file, you can >> implement conditions whatever you want/need. >> >>> This could be trickier if they end by modifying the extension, >>> but we can always backport the latest version, if they change the >>> API. >> >> As far as i know, the docutils API is stable since 2002. In the >> meantime there has been so many application build on it that >> it is not realistic, you will see a not backward compatibly >> change. >> >> The docutils project is conservative, very conservative, IMO to >> conservative. >> >> Today I'am doubtful if it isn't better I would merge it sphinx >> upstream. I have to discuss this with some maintainers, but >> before I have to persuade myself, that all aspects are covered >> and the implementations are robust. We are at the beginning and >> we should not fear about every bit which could happen in the future. >> >> The sphinx / docutils bottom plate gives us a number of degrees >> of freedom to find answers to question we have not yet asked. ;-) > > Ok. So, from what I understand, once Sphinx support is added at > Kernel upstream, we could convert the media docbook to > reST+flat-table extension, adding such extension either on a shared > place or only for the media DocBook build, together with its > conf.py. > Yes, in your media-conf you could decide to use a extension. > Once it reaches upstream (either sphinx or docutils), we can > work to make it integrate better with upstream as needed. > > Right? yes, right :-) > If so, I'm ok with merging it as soon as possible. If we advice a merge of the flat-table directive we should bundle this with the (to implement) "kernel-doc" directive ... > In reST the directive might look like: > > - > Device Instance and Driver Handling > === > > .. kernel-doc:: drivers/gpu/drm/drm_drv.c > :doc: driver instance overview > :exported: > > - and the patches from my kernel-doc perl script to produce reST from source code comments. With this bundle within the kernel tree we have a good starting point to compose reST documents from scratch and to migrate book by book from DocBook to reST. I insist to migrate book by book, because there are some broken books. Broken by that, that some sources have changed but not the corresponding documentation which use the comments of these sources ... grap "Ooops" in the builded (xml or rst) docs. E.g. I greped the .rst file and found the following Oops in the migrated books: ./books/mtdnand/pubfunctions-000-012.rst:13:Oops ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops ./books/s390-drivers/ccw-000-004-003.rst:13:Oops ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops ./books/device-drivers/Basics-000-001-002.rst:13:Oops ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops ./books/device-drivers/Basics-000-009-032.rst:13:Oops ./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops ./books/gadget/api-000-011-005.rst:13:Oops ./books/gadget/api-000-011-009.rst:13:Oops ./books/gadget/api-000-011-007.rst:13:Oops ./books/gadget/api-000-011-003.rst:13:Oops ./books/gadget/api-000-011-011.rst:13:Oops ./books/genericirq/intfunctions-000-009.rst:13:Oops - Summarize @Jonathan: what do you think? Should I prepare a patch with a basic reST (sphinx) build infrastructure, including * a folder for sphinx docs: ./Documentation/sphinx/ * flat-table & kernel-doc extensi
Re: Kernel docs: muddying the waters a bit
Hi Jani, I forget to mentioning, with a local copy of my kernel-doc script: https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc You could do reST markup in the source code comments and extract them. This might be a interim workaround which helps you not to edit source code comments twice. -- Markus-- Am 06.05.2016 um 13:23 schrieb Markus Heiser : > > Hy Jani, > > Am 04.05.2016 um 18:13 schrieb Jani Nikula : > >>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : >>> I think all of this makes sense. It would be really nice to have the directives in the native sphinx language like that. I *don't* think we need to aim for that at the outset; the docproc approach works until we can properly get rid of it. What would be *really* nice would be to get support for the kernel-doc directive into the sphinx upstream. >>> >>> No need for kernel-doc directive in sphinx upstream, later it will be >>> an extension which could be installed by a simple command like >>> "pip install kernel-doc-extensions" or similar. >>> >>> I develop these required extension (and more) within my proof of concept >>> on github ... this takes time ... if I finished all my tests and all is >>> well, I will build the *kernel-doc-extensions* package and deploy it >>> on https://pypi.python.org/pypi from where everyone could install this >>> with "pip". >> >> I think we should go for vanilla sphinx at first, to make the setup step >> as easy as possible for everyone. Even if it means still doing that ugly >> docproc step to call kernel-doc. We can improve from there, and I >> definitely appreciate your work on making this work with sphinx >> extensions. > > +1 > >> That said, how would it work to include the kernel-doc extension in the >> kernel source tree? Having things just work if sphinx is installed is >> preferred over requiring installation of something extra from pypi. (I >> know this may sound backwards for a lot of projects, but for kernel I'm >> pretty sure this is how it should be done.) > > Thats all right. Lets talk about the extension infrastructure by example: > > First we have to chose a folder where we place all the *sphinx-documentation* > I recommending: > > /share/linux/Documentation/sphinx > > Next we have to chose a folder where reST-extensions should take place, I > would prefer ... or similar: > > /share/linux/Documentation/sphinx/extensions > > Lets say, you wan't to get in use of the "flat-table" extension. > > Copy (only) the rstFlatTable.py file from my POC extension folder (ignore > other extensions which might be there) ... > > https://github.com/return42/sphkerneldoc/tree/master/doc/extensions > > Now lets say you are writing on a gpu book, it wold be placed in the folder: > > /share/linux/Documentation/sphinx/gpu > > In this gpu-folder you have to place the conf.py config file, needed to > setup the sphinx build environment. > > /share/linux/Documentation/sphinx/gpu/conf.py > > In this conf.py you have to *register* your folder with the extensions. > > > >import os.path, sys > >EXT_PATH = "../extensions" # the path of extension folder relative to > the conf.py file >sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH))) > ># now import the "flat-table" extension, it will be self-registering to > docutils > >import rstFlatTable > > > > Thats all, you can run your sphinx-build command and the flat-tables in your > reST sources should be handled as common tables. > > ASIDE: > > You will find similar parts in your conf.py which you have created > with the sphinx-quickstart command. There, you will also find a block > looks like ... > > extensions = [ >'sphinx.ext.autodoc' > > ] > > Don't try to add flat-table extension to this list. This list is a list > of sphinx extensions, we will use it later for other *real* sphinx > extensions. > > A few words about the flat-table extension and a (future) kernel-doc one: > > The flat-table is a pure docutils (the layer below sphinx) extension which > is not application specific, so I will ask for moving it to the docutils > upstream. > > The kernel-doc extension on the other side is a very (very) kernel specific > application, this would never go to sphinx nor docutils upstream. > > --Markus-- >
Re: Kernel docs: muddying the waters a bit
Hy Jani, Am 04.05.2016 um 18:13 schrieb Jani Nikula : >> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : >> >>> I think all of this makes sense. It would be really nice to have the >>> directives in the native sphinx language like that. I *don't* think we >>> need to aim for that at the outset; the docproc approach works until we can >>> properly get rid of it. What would be *really* nice would be to get >>> support for the kernel-doc directive into the sphinx upstream. >> >> No need for kernel-doc directive in sphinx upstream, later it will be >> an extension which could be installed by a simple command like >> "pip install kernel-doc-extensions" or similar. >> >> I develop these required extension (and more) within my proof of concept >> on github ... this takes time ... if I finished all my tests and all is >> well, I will build the *kernel-doc-extensions* package and deploy it >> on https://pypi.python.org/pypi from where everyone could install this >> with "pip". > > I think we should go for vanilla sphinx at first, to make the setup step > as easy as possible for everyone. Even if it means still doing that ugly > docproc step to call kernel-doc. We can improve from there, and I > definitely appreciate your work on making this work with sphinx > extensions. +1 > That said, how would it work to include the kernel-doc extension in the > kernel source tree? Having things just work if sphinx is installed is > preferred over requiring installation of something extra from pypi. (I > know this may sound backwards for a lot of projects, but for kernel I'm > pretty sure this is how it should be done.) Thats all right. Lets talk about the extension infrastructure by example: First we have to chose a folder where we place all the *sphinx-documentation* I recommending: /share/linux/Documentation/sphinx Next we have to chose a folder where reST-extensions should take place, I would prefer ... or similar: /share/linux/Documentation/sphinx/extensions Lets say, you wan't to get in use of the "flat-table" extension. Copy (only) the rstFlatTable.py file from my POC extension folder (ignore other extensions which might be there) ... https://github.com/return42/sphkerneldoc/tree/master/doc/extensions Now lets say you are writing on a gpu book, it wold be placed in the folder: /share/linux/Documentation/sphinx/gpu In this gpu-folder you have to place the conf.py config file, needed to setup the sphinx build environment. /share/linux/Documentation/sphinx/gpu/conf.py In this conf.py you have to *register* your folder with the extensions. import os.path, sys EXT_PATH = "../extensions" # the path of extension folder relative to the conf.py file sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH))) # now import the "flat-table" extension, it will be self-registering to docutils import rstFlatTable Thats all, you can run your sphinx-build command and the flat-tables in your reST sources should be handled as common tables. ASIDE: You will find similar parts in your conf.py which you have created with the sphinx-quickstart command. There, you will also find a block looks like ... extensions = [ 'sphinx.ext.autodoc' ] Don't try to add flat-table extension to this list. This list is a list of sphinx extensions, we will use it later for other *real* sphinx extensions. A few words about the flat-table extension and a (future) kernel-doc one: The flat-table is a pure docutils (the layer below sphinx) extension which is not application specific, so I will ask for moving it to the docutils upstream. The kernel-doc extension on the other side is a very (very) kernel specific application, this would never go to sphinx nor docutils upstream. --Markus--
Re: Kernel docs: muddying the waters a bit
Hi Mauro, Am 04.05.2016 um 18:15 schrieb Mauro Carvalho Chehab : > Em Wed, 4 May 2016 11:34:08 +0200 > Markus Heiser escreveu: > >> Hi all, (hi Jonathan, please take note of my offer below) >> >> Am 03.05.2016 um 16:31 schrieb Daniel Vetter : >> >>> Hi all, >>> >>> So sounds like moving ahead with rst/sphinx is the option that should >>> allow us to address everyone's concerns eventually? Of course the >>> first one won't have it all (media seems really tricky), ... >> >> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... >> For this, I wrote the "flat-table" reST-directive, which adds >> missing cells automatically: >> >> doc: >> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table >> source: >> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py > > Yeah, this should address the lack of a proper way to markup cell/row > spans, providing the additional bits for the tables we have at media. > > Yet, there are some issues with table conversions. See below. > Some tables, like the one here: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html > > are truncated (tested with Mozilla and Chrome), and part of the information is > lost due to that. Not a problem of rendering. This was a bug in the magration from DocBook to reST. You might remember that we have discussed, that some of the tables are better marked-up as definition lists. This was (the last) one I forgot to convert to a definition list ... I hope it was the last one, if not and you find more or other broken parts, please inform me (on the linux-tv mailing, or direct). --Markus--
Re: Kernel docs: muddying the waters a bit
Em Thu, 5 May 2016 07:02:10 -0600 Jonathan Corbet escreveu: > On Wed, 4 May 2016 14:57:38 -0300 > Mauro Carvalho Chehab wrote: > > > Also, media documentation is not just one more documentation. It is > > the biggest one we have, and that has more changes than any other > > documentation under Documentation/DocBook: > > > > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l > > 116 > > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type > > f`|wc -l > > 179 > > > > It also is more than twice the size of the other DocBook docs: > > > > $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1 > > 82275 total > > $ wc -l $(ls *.tmpl|grep -v media)|tail -1 > > 29568 total > > > > E. g. media corresponds to 60% of the number of patches and 73% of > > the DocBook stuff. > > These numbers are not entirely representative, I have to say. You're > ignoring the kerneldoc comments - which is what much of the "DocBook" > documents are made of, and which is the focus of much of this activity. If > you could find a way to count those, you'd get a different picture. Yeah, if we take the big picture, I'm pretty sure that there are more stuff written using kerneldoc. However, what is written at the kerneldoc comments don't use DocBook markup, but its own markup language. Assuming that we'll keep using kerneldoc script to convert from the Kerneldoc NANO markup, it means that any changes at the kerneldoc backend to use DocBook/Markdown/asciidoc/reST/... won't change anything for the developers nor will require them to actually use the new markups. So, they can gradually start using new markups as they wish/learn, if they want to improve the documentation. Lazy developers could just ignore reST if they want, as everything will work as before. However, for the files written directly in DocBook, there's no option: at the moment the doc is converted, the developer will need to submit patches using the new markup language. So, we need to be sure that such transition will happen in a way that won't cause more harm than needed. I would prefer to convert all of them at one, on a single Kernel release (or two), in order to minimize the impact for the developers. > > But I don't think that really matters; there doesn't seem to be *that* much > disagreement here. > > The media book is important; we want it to be a part of the overall kernel > documentation suite and not stuck in some DocBook ghetto. I agree that we > should have an idea for a plausible path for *all* of our documentation. > But I'm also concerned about delaying this work yet again; we have > developers trying to push forward with improved documentation, and they've > had to wait a year for this stuff - so far. Yeah, I understand. Regards, Mauro
Re: Kernel docs: muddying the waters a bit
On Wed, 4 May 2016 14:57:38 -0300 Mauro Carvalho Chehab wrote: > Also, media documentation is not just one more documentation. It is > the biggest one we have, and that has more changes than any other > documentation under Documentation/DocBook: > > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l > 116 > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc > -l > 179 > > It also is more than twice the size of the other DocBook docs: > > $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1 > 82275 total > $ wc -l $(ls *.tmpl|grep -v media)|tail -1 > 29568 total > > E. g. media corresponds to 60% of the number of patches and 73% of > the DocBook stuff. These numbers are not entirely representative, I have to say. You're ignoring the kerneldoc comments - which is what much of the "DocBook" documents are made of, and which is the focus of much of this activity. If you could find a way to count those, you'd get a different picture. But I don't think that really matters; there doesn't seem to be *that* much disagreement here. The media book is important; we want it to be a part of the overall kernel documentation suite and not stuck in some DocBook ghetto. I agree that we should have an idea for a plausible path for *all* of our documentation. But I'm also concerned about delaying this work yet again; we have developers trying to push forward with improved documentation, and they've had to wait a year for this stuff - so far. Thanks, jon
Re: Kernel docs: muddying the waters a bit
Em Wed, 4 May 2016 10:59:36 -0600 Jonathan Corbet escreveu: > On Wed, 4 May 2016 13:50:35 -0300 > Mauro Carvalho Chehab wrote: > > > Em Wed, 4 May 2016 19:13:21 +0300 > > Jani Nikula escreveu: > > > I think we should go for vanilla sphinx at first, to make the setup step > > > as easy as possible for everyone. > > > > Vanilla Sphinx doesn't work, as reST markup language is too limited > > to support the docs we have. So, we should either push the needed > > extensions to Sphinx reST or find a way to put it at Kernel tree > > without causing too much pain for the developers, e. g. as something > > that just doing "make htmldoc" would automatically use such extensions > > without needing to actually install the extensions. > > It works for everything except the extended media book, right? Or is there > something I'm missing here? As far as I know, yes, but I didn't check the other documentation to be sure. > I am very much interested in having one system > for *all* our docs, but we wouldn't attempt to convert a document can't be > expressed in sphinx. Looking from the other side, keeping only one document as DocBook and having everything else converted to some markup solution seem equally a bad solution, as, in the end of the day, we'll become dependent of two different tool chains to produce document. Also, media documentation is not just one more documentation. It is the biggest one we have, and that has more changes than any other documentation under Documentation/DocBook: $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l 116 $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l 179 It also is more than twice the size of the other DocBook docs: $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1 82275 total $ wc -l $(ls *.tmpl|grep -v media)|tail -1 29568 total E. g. media corresponds to 60% of the number of patches and 73% of the DocBook stuff. Not converting it to the new "official" Kernel markup language would would mean that developers that work on more than the media subsystem would need to know two different, incompatible dialects to produce documentation (plus kernel-doc). That sounds messy and it is an extra penalty to media developers. Some may even start to think that that writing bad docs would be a good thing. > > > Even if it means still doing that ugly > > > docproc step to call kernel-doc. We can improve from there, and I > > > definitely appreciate your work on making this work with sphinx > > > extensions. > > > > I disagree: We should not cause documentation regressions by > > producing incomplete documentation and losing tables because of > > such conversion. > > > The quality of the documentation after the change should be *at least* > > equal to the one we current produce, never worse. > > Agreed; that's why I think the existing DocBook mechanism should stay in > place until that document will be improved by the change. But I'd rather > not hold up the entire process for one book, especially since pushing that > process forward can only help in dealing with those final problems. Yeah, in practice, we won't be converting everything on just one Kernel version. Yet, we need at least an strategy and a plan that would allow converting everything to the new markup language. Provided that we would merge Sphinx support on 4.8, I would like to convert the media documentation just after merging the new toolchain upstream, in order to minimize the conversion impact and to not lose developers and good developer's documentation in the process. However, such work can only be started if we have a way to work with the required extensions, as we're deciding to use a markup language that has incomplete table support. So, IMHO, we should have a way to run the required extensions in a painless way before doing the merge. > > > That said, how would it work to include the kernel-doc extension in the > > > kernel source tree? Having things just work if sphinx is installed is > > > preferred over requiring installation of something extra from pypi. (I > > > know this may sound backwards for a lot of projects, but for kernel I'm > > > pretty sure this is how it should be done.) > > > > Yeah, using pypi seems an additional undesired step. Aren't there > > any way to make python to use an additional extension at the > > Kernel tree without needing to install it? > > Well, sphinx has to come from somewhere too. But yes, we should be able to > carry extensions in the kernel tree. But I would still rather upstream it > (to sphinx) if possible, so we're not stuck trying to maintain it across > several sphinx releases. I don't know how volatile their interfaces are, > but it would be, in any case, the analog of an out-of-tree module... Yeah, the best would be to put everything we need at Sphinx upstream, but assuming that they agree today to add the extensions we need, we would still need to carry those exten
Re: Kernel docs: muddying the waters a bit
On Wed, 4 May 2016 13:50:35 -0300 Mauro Carvalho Chehab wrote: > Em Wed, 4 May 2016 19:13:21 +0300 > Jani Nikula escreveu: > > I think we should go for vanilla sphinx at first, to make the setup step > > as easy as possible for everyone. > > Vanilla Sphinx doesn't work, as reST markup language is too limited > to support the docs we have. So, we should either push the needed > extensions to Sphinx reST or find a way to put it at Kernel tree > without causing too much pain for the developers, e. g. as something > that just doing "make htmldoc" would automatically use such extensions > without needing to actually install the extensions. It works for everything except the extended media book, right? Or is there something I'm missing here? I am very much interested in having one system for *all* our docs, but we wouldn't attempt to convert a document can't be expressed in sphinx. > > Even if it means still doing that ugly > > docproc step to call kernel-doc. We can improve from there, and I > > definitely appreciate your work on making this work with sphinx > > extensions. > > I disagree: We should not cause documentation regressions by > producing incomplete documentation and losing tables because of > such conversion. > The quality of the documentation after the change should be *at least* > equal to the one we current produce, never worse. Agreed; that's why I think the existing DocBook mechanism should stay in place until that document will be improved by the change. But I'd rather not hold up the entire process for one book, especially since pushing that process forward can only help in dealing with those final problems. > > That said, how would it work to include the kernel-doc extension in the > > kernel source tree? Having things just work if sphinx is installed is > > preferred over requiring installation of something extra from pypi. (I > > know this may sound backwards for a lot of projects, but for kernel I'm > > pretty sure this is how it should be done.) > > Yeah, using pypi seems an additional undesired step. Aren't there > any way to make python to use an additional extension at the > Kernel tree without needing to install it? Well, sphinx has to come from somewhere too. But yes, we should be able to carry extensions in the kernel tree. But I would still rather upstream it (to sphinx) if possible, so we're not stuck trying to maintain it across several sphinx releases. I don't know how volatile their interfaces are, but it would be, in any case, the analog of an out-of-tree module... jon
Re: Kernel docs: muddying the waters a bit
Em Wed, 4 May 2016 19:13:21 +0300 Jani Nikula escreveu: > On Wed, 04 May 2016, Markus Heiser wrote: > > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in > > particular with your "MARKDOWNREADY := gpu.xml" process. > > > > As I understood, you convert markdown to docbook within the kernel-doc > > script using pandoc's executable? ... I don't face this topic. With my > > modification of kernel-doc I produced pure reST markup from standardize > > kernel-doc markup, no intermediate steps or tools required. > > That pandoc thing is a dead end. Forget about it. I think we've all > pretty much agreed we should have kernel-doc produce the lightweight > markup directly since [1]. > > [1] > http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com > > > Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : > > > >> I think all of this makes sense. It would be really nice to have the > >> directives in the native sphinx language like that. I *don't* think we > >> need to aim for that at the outset; the docproc approach works until we can > >> properly get rid of it. What would be *really* nice would be to get > >> support for the kernel-doc directive into the sphinx upstream. > > > > No need for kernel-doc directive in sphinx upstream, later it will be > > an extension which could be installed by a simple command like > > "pip install kernel-doc-extensions" or similar. > > > > I develop these required extension (and more) within my proof of concept > > on github ... this takes time ... if I finished all my tests and all is > > well, I will build the *kernel-doc-extensions* package and deploy it > > on https://pypi.python.org/pypi from where everyone could install this > > with "pip". > > I think we should go for vanilla sphinx at first, to make the setup step > as easy as possible for everyone. Vanilla Sphinx doesn't work, as reST markup language is too limited to support the docs we have. So, we should either push the needed extensions to Sphinx reST or find a way to put it at Kernel tree without causing too much pain for the developers, e. g. as something that just doing "make htmldoc" would automatically use such extensions without needing to actually install the extensions. > Even if it means still doing that ugly > docproc step to call kernel-doc. We can improve from there, and I > definitely appreciate your work on making this work with sphinx > extensions. I disagree: We should not cause documentation regressions by producing incomplete documentation and losing tables because of such conversion. The quality of the documentation after the change should be *at least* equal to the one we current produce, never worse. > That said, how would it work to include the kernel-doc extension in the > kernel source tree? Having things just work if sphinx is installed is > preferred over requiring installation of something extra from pypi. (I > know this may sound backwards for a lot of projects, but for kernel I'm > pretty sure this is how it should be done.) Yeah, using pypi seems an additional undesired step. Aren't there any way to make python to use an additional extension at the Kernel tree without needing to install it? Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
Em Wed, 4 May 2016 08:57:13 -0600 Jonathan Corbet escreveu: > On Wed, 4 May 2016 16:18:27 +0200 > Daniel Vetter wrote: > > > > I'd really like to converge on the markup question, so that we can start > > > using all the cool stuff with impunity in gpu documentations. > > > > Aside: If we decide this now I could send in a pull request for the > > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal > > markdown/asciidoc code I still have). That would be really awesome ... > > Sorry for my relative absence...I'm still busy dealing with bureaucracy > an ocean away from home. I hope to begin emerging from this mess in the > near future. > > So ... there's the code you have, the work I (+Jani) did, and the work > Markus has done. Which would you have me push into 4.7? IMHO, Markus approach is the one that is providing a better result, as it added support for missing features that we require for the media DocBook. Still, it seems premature to merge it for 4.7. Markus is getting real progress on converting the media docs to work with Sphynx, but there are still troubles when the table is big, as, currently, it is truncating everything that passes the right margin on some tables. So, IMHO, the next action item would be to be sure that big tables will not be truncated. Assuming that he can fix that in time, we can merge it for 4.8, and then start porting the documentation to use it. Btw, converting the DocBook/media Makefile will require an extra step, as part of the documentation is generated via scripts (some C file examples, and include/uapi headers). Those scripts also warrant that (almost) every API at include/uapi is synchronized with the DocBook. I use this on my patch review process, in order to reject patches that don't add the proper documentation updates. Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
On Wed, May 4, 2016 at 5:02 PM, Daniel Vetter wrote: > On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote: >> On Wed, 4 May 2016 16:18:27 +0200 >> Daniel Vetter wrote: >> >> > > I'd really like to converge on the markup question, so that we can start >> > > using all the cool stuff with impunity in gpu documentations. >> > >> > Aside: If we decide this now I could send in a pull request for the >> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal >> > markdown/asciidoc code I still have). That would be really awesome ... >> >> Sorry for my relative absence...I'm still busy dealing with bureaucracy >> an ocean away from home. I hope to begin emerging from this mess in the >> near future. >> >> So ... there's the code you have, the work I (+Jani) did, and the work >> Markus has done. Which would you have me push into 4.7? >> >> The sphinx/rst approach does seem, to me, to be the right one, with the >> existing DocBook structure remaining in place for those who want/need >> it. I'm inclined toward my stuff as a base to work with, obviously :) But >> it's hackish at best and needs a lot of cleaning up. It's a proof of >> concept, but it's hardly finished (one might say it's barely begun...) >> >> In the end, I guess, I feel that anything we might try to push for 4.7 is >> going to look rushed and not ready, and Linus might react accordingly. >> I'd be more comfortable aiming for 4.8. I *will* have more time to focus >> on things in that time frame... I suspect you're pretty well fed up with >> this stuff being pushed back, and rightly so. All I can do is apologize. >> >> That said, if you do think there's something out there that is good >> enough to consider pushing in a week or two, do tell and we can all take >> a look. > > Well I'd just have taken the asciidoc hacks I have currently in my > topic/kerneldoc branch, converted to sphinx and looked how it fares. It > should be fairly minimal, and I think the first step we want to do for the > long-term plan. I hope I can ready something, and then we can look whether > it's rushed for 4.7 or not. Ok, discussed this a bit more with Jani on IRC and he really doesn't like the old design of that branch (it calls the converter for every kerneldoc comment). So I guess nothing rushed for 4.7, but hopefully something for 4.8. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
Em Wed, 4 May 2016 11:34:08 +0200 Markus Heiser escreveu: > Hi all, (hi Jonathan, please take note of my offer below) > > Am 03.05.2016 um 16:31 schrieb Daniel Vetter : > > > Hi all, > > > > So sounds like moving ahead with rst/sphinx is the option that should > > allow us to address everyone's concerns eventually? Of course the > > first one won't have it all (media seems really tricky), ... > > BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... > > Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab > : > > > With that sense, the "List tables" format is also not good, as > > one row addition would generate several hunks (one for each column > > of the table), making harder to review the patch by just looking at > > the diff. > > For this, I wrote the "flat-table" reST-directive, which adds > missing cells automatically: > > doc: > http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table > source: > https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py Yeah, this should address the lack of a proper way to markup cell/row spans, providing the additional bits for the tables we have at media. Yet, there are some issues with table conversions. See below. > > I used "flat-table" to migrate all DocBook-XML documents to reST. With this > directive, I also managed to migrate the complete media book (no more TODOs) > incl. the large tables like them from subdev-formats: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > > (Rendering large tables is a general discussion which should not take place > in this MT) Some tables, like the one here: https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html are truncated (tested with Mozilla and Chrome), and part of the information is lost due to that. Regards, Mauro
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016, Markus Heiser wrote: > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in > particular with your "MARKDOWNREADY := gpu.xml" process. > > As I understood, you convert markdown to docbook within the kernel-doc > script using pandoc's executable? ... I don't face this topic. With my > modification of kernel-doc I produced pure reST markup from standardize > kernel-doc markup, no intermediate steps or tools required. That pandoc thing is a dead end. Forget about it. I think we've all pretty much agreed we should have kernel-doc produce the lightweight markup directly since [1]. [1] http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com > Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : > >> I think all of this makes sense. It would be really nice to have the >> directives in the native sphinx language like that. I *don't* think we >> need to aim for that at the outset; the docproc approach works until we can >> properly get rid of it. What would be *really* nice would be to get >> support for the kernel-doc directive into the sphinx upstream. > > No need for kernel-doc directive in sphinx upstream, later it will be > an extension which could be installed by a simple command like > "pip install kernel-doc-extensions" or similar. > > I develop these required extension (and more) within my proof of concept > on github ... this takes time ... if I finished all my tests and all is > well, I will build the *kernel-doc-extensions* package and deploy it > on https://pypi.python.org/pypi from where everyone could install this > with "pip". I think we should go for vanilla sphinx at first, to make the setup step as easy as possible for everyone. Even if it means still doing that ugly docproc step to call kernel-doc. We can improve from there, and I definitely appreciate your work on making this work with sphinx extensions. That said, how would it work to include the kernel-doc extension in the kernel source tree? Having things just work if sphinx is installed is preferred over requiring installation of something extra from pypi. (I know this may sound backwards for a lot of projects, but for kernel I'm pretty sure this is how it should be done.) BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Am 04.05.2016 um 15:43 schrieb Daniel Vetter : > On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote: >>> On Wed, 04 May 2016, Markus Heiser wrote: >>> I'd be *very* hesitant about adding the kind of things you do in >>> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc >>> comments must be as simple as possible with basically pass-through of >>> the comment blocks to sphinx. >> >> Right, but you forgot, that the current markup in the source code comments >> is based on the kernel-doc-nano-HOWTO and I guess no one will migrate all >> these source code comments to reST markup ;-) >> >> So there is a need to differentiate between *vintage* kernel-doc markup >> and reST markup. >> >> My suggestion is to add a tag to the comments, here a short example >> what this might look like. >> >> -- >> /** >> * drm_minor_release - Release DRM minor >> * @minor: Pointer to DRM minor object >> * >> * Release a minor that was previously acquired via drm_minor_acquire(). >> */ >> -- >> >> in short: the vintage style does not need any change and >> comments with reST markup has a tag ":reST:" in the first >> line(s) ... >> >> -- >> /** >> * :reST: >> * >> * .. c:function:: void drm_minor_release(struct drm_minor *minor) >> * >> *Release DRM minor >> * >> *:param struct drm_minor \*minor: Pointer to DRM minor object >> * >> */ >> -- >> >> Comments with the ":reST:" tag could be exported and pass-through >> to sphinx. >> >>> Specifically, do not attempt to detect and >>> parse elements like lists in kernel-doc. >> >> If your markup in the comments is plain reST, no need to parse, but there >> are markups in the (vintage) comments which made use of individual >> text-markups, e.g. the markup of lists or ASCII-art. >> >> This individual text-markups has not been converted/rendered in the docbook >> output, but I wanted to convert this individuals to reST, to render them in >> Sphinx. > > I think we need to unconfuse what's current standardize kerneldoc markup. > There's three bits: > - The header with the one-liner and parameters, i.e. OK, forget my example, I don't really want to promote this way ... I agree, it is better to stay with standardize kernel-doc markup and provide a "pass through" for the section-content (even if it is mixing markups). Am 04.05.2016 um 15:41 schrieb Jani Nikula : > Kernel-doc the tool should continue to parse kernel-doc comments at the > high level like they currently are, according to the kernel-doc > howto. The requirement was unclear for me, thanks to Daniel and Jani for clarifying this. Am 04.05.2016 um 15:43 schrieb Daniel Vetter : > This is already used widely in kerneldoc included by gpu.tmpl, and > currently it's asciidoc. Those lists and asciiarts though are _not_ > standard kerneldoc at all. But imo any doc toolchain improvement should > integrate along those lines. > > For reference, this is what it's supposed to look like with the asciidoc > support enabled: > > https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html > > Fyi those hacked-up patches to make this happen are available in > > https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in particular with your "MARKDOWNREADY := gpu.xml" process. As I understood, you convert markdown to docbook within the kernel-doc script using pandoc's executable? ... I don't face this topic. With my modification of kernel-doc I produced pure reST markup from standardize kernel-doc markup, no intermediate steps or tools required. Am 04.05.2016 um 15:41 schrieb Jani Nikula : > Overall I think we should promote fixing those in the kernel-doc > comments. Trying to guesswork in kernel-doc tool will leave the source > littered with bad examples that get proliferated all around. Instead of > gradually fixing things, we'll end up gradually messing things up even > more. I agree with you, lets drop the reformat_block_rst from my kernel-doc script: * https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736 and we should have a plain "pass through". Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : > I think all of this makes sense. It would be really nice to have the > directives in the native sphinx language like that. I *don't* think we > need to aim for that at the outset; the docproc approach works until we can > properly get rid of it. What would be *really* nice would be to get > support for the kernel-doc directive into the sphinx upstream. No need for kernel-doc directive in sphinx upstream, later it will be an extension which could be installed by a simple command like "pip install kernel-doc-extensions" or similar. I develop these required extension (and more) within my proof of concept on github ... this takes time ... if I finished all my tests and all is well, I will build the *kernel-doc-extensions*
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016, Jonathan Corbet wrote: > The sphinx/rst approach does seem, to me, to be the right one, with the > existing DocBook structure remaining in place for those who want/need > it. I'm inclined toward my stuff as a base to work with, obviously :) But > it's hackish at best and needs a lot of cleaning up. It's a proof of > concept, but it's hardly finished (one might say it's barely begun...) Thanks. I'll start looking at how to make it less hackish and more upstreamable. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote: > On Wed, 4 May 2016 16:18:27 +0200 > Daniel Vetter wrote: > > > > I'd really like to converge on the markup question, so that we can start > > > using all the cool stuff with impunity in gpu documentations. > > > > Aside: If we decide this now I could send in a pull request for the > > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal > > markdown/asciidoc code I still have). That would be really awesome ... > > Sorry for my relative absence...I'm still busy dealing with bureaucracy > an ocean away from home. I hope to begin emerging from this mess in the > near future. > > So ... there's the code you have, the work I (+Jani) did, and the work > Markus has done. Which would you have me push into 4.7? > > The sphinx/rst approach does seem, to me, to be the right one, with the > existing DocBook structure remaining in place for those who want/need > it. I'm inclined toward my stuff as a base to work with, obviously :) But > it's hackish at best and needs a lot of cleaning up. It's a proof of > concept, but it's hardly finished (one might say it's barely begun...) > > In the end, I guess, I feel that anything we might try to push for 4.7 is > going to look rushed and not ready, and Linus might react accordingly. > I'd be more comfortable aiming for 4.8. I *will* have more time to focus > on things in that time frame... I suspect you're pretty well fed up with > this stuff being pushed back, and rightly so. All I can do is apologize. > > That said, if you do think there's something out there that is good > enough to consider pushing in a week or two, do tell and we can all take > a look. Well I'd just have taken the asciidoc hacks I have currently in my topic/kerneldoc branch, converted to sphinx and looked how it fares. It should be fairly minimal, and I think the first step we want to do for the long-term plan. I hope I can ready something, and then we can look whether it's rushed for 4.7 or not. Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016 16:41:50 +0300 Jani Nikula wrote: > On Wed, 04 May 2016, Markus Heiser wrote: > > In reST the directive might look like: > > > > - > > Device Instance and Driver Handling > > === > > > > .. kernel-doc:: drivers/gpu/drm/drm_drv.c > >:doc: driver instance overview > >:exported: > > > > - > > Yes, I think something like this, parsed by sphinx directly (via some > extension perhaps), should be the end goal. I am not sure if it should > be the immediate first goal though or whether we should reuse the > existing docproc for now. I think all of this makes sense. It would be really nice to have the directives in the native sphinx language like that. I *don't* think we need to aim for that at the outset; the docproc approach works until we can properly get rid of it. What would be *really* nice would be to get support for the kernel-doc directive into the sphinx upstream. > > -- > > > > Comments with the ":reST:" tag could be exported and pass-through > > to sphinx. > > Disagreed on most of the above. Agreed with the disagreement here. We can't be adding ":reST:" tags to comments; I anticipate a wee bit of pushback if we try. It needs to work with the comments as they are now. It seems that should be possible. Thanks, jon
Re: Kernel docs: muddying the waters a bit
On Wed, 4 May 2016 16:18:27 +0200 Daniel Vetter wrote: > > I'd really like to converge on the markup question, so that we can start > > using all the cool stuff with impunity in gpu documentations. > > Aside: If we decide this now I could send in a pull request for the > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal > markdown/asciidoc code I still have). That would be really awesome ... Sorry for my relative absence...I'm still busy dealing with bureaucracy an ocean away from home. I hope to begin emerging from this mess in the near future. So ... there's the code you have, the work I (+Jani) did, and the work Markus has done. Which would you have me push into 4.7? The sphinx/rst approach does seem, to me, to be the right one, with the existing DocBook structure remaining in place for those who want/need it. I'm inclined toward my stuff as a base to work with, obviously :) But it's hackish at best and needs a lot of cleaning up. It's a proof of concept, but it's hardly finished (one might say it's barely begun...) In the end, I guess, I feel that anything we might try to push for 4.7 is going to look rushed and not ready, and Linus might react accordingly. I'd be more comfortable aiming for 4.8. I *will* have more time to focus on things in that time frame... I suspect you're pretty well fed up with this stuff being pushed back, and rightly so. All I can do is apologize. That said, if you do think there's something out there that is good enough to consider pushing in a week or two, do tell and we can all take a look. Thanks, jon
Re: Kernel docs: muddying the waters a bit
On Wed, May 4, 2016 at 3:43 PM, Daniel Vetter wrote: > I'd really like to converge on the markup question, so that we can start > using all the cool stuff with impunity in gpu documentations. Aside: If we decide this now I could send in a pull request for the rst/sphinx kernel-doc support still for 4.7 (based upon the minimal markdown/asciidoc code I still have). That would be really awesome ... Jon? Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote: > > On Wed, 04 May 2016, Markus Heiser wrote: > > I'd be *very* hesitant about adding the kind of things you do in > > reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc > > comments must be as simple as possible with basically pass-through of > > the comment blocks to sphinx. > > Right, but you forgot, that the current markup in the source code comments > is based on the kernel-doc-nano-HOWTO and I guess no one will migrate all > these source code comments to reST markup ;-) > > So there is a need to differentiate between *vintage* kernel-doc markup > and reST markup. > > My suggestion is to add a tag to the comments, here a short example > what this might look like. > > -- > /** > * drm_minor_release - Release DRM minor > * @minor: Pointer to DRM minor object > * > * Release a minor that was previously acquired via drm_minor_acquire(). > */ > -- > > in short: the vintage style does not need any change and > comments with reST markup has a tag ":reST:" in the first > line(s) ... > > -- > /** > * :reST: > * > * .. c:function:: void drm_minor_release(struct drm_minor *minor) > * > *Release DRM minor > * > *:param struct drm_minor \*minor: Pointer to DRM minor object > * > */ > -- > > Comments with the ":reST:" tag could be exported and pass-through > to sphinx. > > > Specifically, do not attempt to detect and > > parse elements like lists in kernel-doc. > > If your markup in the comments is plain reST, no need to parse, but there > are markups in the (vintage) comments which made use of individual > text-markups, e.g. the markup of lists or ASCII-art. > > This individual text-markups has not been converted/rendered in the docbook > output, but I wanted to convert this individuals to reST, to render them in > Sphinx. I think we need to unconfuse what's current standardize kerneldoc markup. There's three bits: - The header with the one-liner and parameters, i.e. /** * drm_minor_release - Release DRM minor * @minor: Pointer to DRM minor object There's thousands of those, and we cant realistically change them. This means we need to teach kernel-doc to parse those and convert them to rest/shpinx layout. Any approach that expects a conversion of all the existing comments over to sphinx/rst is imo unworkable. - References for function(), &structures, #constants and @parameters. Again for the same reasons of being here already, we can't change those at all. On top of that these markers are also used to generate hyperlinks in the final docs. The kernel-doc script therefor not only needs to generate the right markup, but also correct links. Without generating links to functions/structures outside of a given document. - kernel-doc also does paragraph splitting and minimalistic headers like * Returns: * * This will be under a separate Returns: heading in the text. afaict at least the paragraph splitting would be identical between all proposed markup languages. Now what all current proposals have done is simply allow pass-through of asciidoc or rst markup in those paragraphs, so that the final processor could make stuff pretty. This is already used widely in kerneldoc included by gpu.tmpl, and currently it's asciidoc. Those lists and asciiarts though are _not_ standard kerneldoc at all. But imo any doc toolchain improvement should integrate along those lines. For reference, this is what it's supposed to look like with the asciidoc support enabled: https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html Fyi those hacked-up patches to make this happen are available in https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc Note that this isn't upstream, but we did start using this support all over in gpu documentation simply because we really, really need it. And I'm willing to throw all the comments into a converter to end up with whatever markup we decide upon. But what will not work is to throw the existing kernel-doc standard into the shredder. So all the stuff with @, (), & and # must keep working. > E.g. compare the member & description section of struct drm-display-mode ... > > DocBook: > > * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html > > kernel-doc reST with the additional reformat_block_rst function: > > * > http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode So we have this already. The two things I thought this entire discussion was about: - also throw out the docbook .xml and replace it with something else. That was jani's prototype on top of asciidoc, and it sounds like we could do something similar with rst/sphinx. Everyone pretty much agrees that this would be a second step, and the first step would be to simply integrate more advanced markup into the existing toolchain.
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016, Markus Heiser wrote: >> What do you mean by ".tmpl workflow"? > > Sorry for bad naming, I addressed the DOCPROC build process which builds > the .xml files from .tmpl files ... Yeah, I know more about this part than I care. I was just wondering what you refer to with that in the sphinx context. > In reST the directive might look like: > > - > Device Instance and Driver Handling > === > > .. kernel-doc:: drivers/gpu/drm/drm_drv.c >:doc: driver instance overview >:exported: > > - Yes, I think something like this, parsed by sphinx directly (via some extension perhaps), should be the end goal. I am not sure if it should be the immediate first goal though or whether we should reuse the existing docproc for now. > Right, but you forgot, that the current markup in the source code comments > is based on the kernel-doc-nano-HOWTO and I guess no one will migrate all > these source code comments to reST markup ;-) > > So there is a need to differentiate between *vintage* kernel-doc markup > and reST markup. > > My suggestion is to add a tag to the comments, here a short example > what this might look like. > > -- > /** > * drm_minor_release - Release DRM minor > * @minor: Pointer to DRM minor object > * > * Release a minor that was previously acquired via drm_minor_acquire(). > */ > -- > > in short: the vintage style does not need any change and > comments with reST markup has a tag ":reST:" in the first > line(s) ... > > -- > /** > * :reST: > * > * .. c:function:: void drm_minor_release(struct drm_minor *minor) > * > *Release DRM minor > * > *:param struct drm_minor \*minor: Pointer to DRM minor object > * > */ > -- > > Comments with the ":reST:" tag could be exported and pass-through > to sphinx. Disagreed on most of the above. Maybe I was unclear in my previous mail, and I've probably worked on this so much previously that I thought it was clear how we should handle lightweight markup in kernel-doc comments. Kernel-doc the tool should continue to parse kernel-doc comments at the high level like they currently are, according to the kernel-doc howto. The first heading line, the parameter/member lines with @param:, and references within the text with @param, function(), &struct structures, etc. For those, kernel-doc should produce appropriate rst elements. Beyond that, kernel-doc should *not* try to make guesses about the formatting of the documentation comments. It should just pass the rest of the formatting through. If the ad-hoc lists etc. in the current comments don't turn into proper rst lists, then so be it. If it bugs people, the comments will gradually be converted to proper rst. The worst we could do is promote a sloppy style that kernel-doc fixes for you, possibly "fixing" things you'd want to pass through to sphinx. What you have in example above will not fly. The documentation comment style must remain as is defined in the kernel-doc how, i.e. example. It is imperative that the kernel-doc comments remain as readable as they currently are in the source code. >> Specifically, do not attempt to detect and >> parse elements like lists in kernel-doc. > > If your markup in the comments is plain reST, no need to parse, but there > are markups in the (vintage) comments which made use of individual > text-markups, e.g. the markup of lists or ASCII-art. > > This individual text-markups has not been converted/rendered in the docbook > output, but I wanted to convert this individuals to reST, to render them in > Sphinx. > > E.g. compare the member & description section of struct drm-display-mode ... > > DocBook: > > * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html > > kernel-doc reST with the additional reformat_block_rst function: > > * > http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode Overall I think we should promote fixing those in the kernel-doc comments. Trying to guesswork in kernel-doc tool will leave the source littered with bad examples that get proliferated all around. Instead of gradually fixing things, we'll end up gradually messing things up even more. For those specific examples, IIRC the last time I played with that, simply leaving the prefix whitespace in place went a long way (just eat " * " from the lines). The asciiart just worked. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Hi Jani, Am 04.05.2016 um 11:58 schrieb Jani Nikula : > On Wed, 04 May 2016, Markus Heiser wrote: >> but I think this will not by very helpful, as long as you miss >> a similar ".tmpl" workflow for reST documents. >> >> I'am working on a reST directive (named: "kernel-doc") to provide a >> similar ".tmpl" workflow within plain reST. The first step towards >> is done with (my) modified kernel-doc script ... >> >> * >> https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736 >> >> which produce reST from source code comments. E.g. this content is >> generated with it. > > What do you mean by ".tmpl workflow"? Sorry for bad naming, I addressed the DOCPROC build process which builds the .xml files from .tmpl files ... ### # The build process is as follows (targets): # (xmldocs) [by docproc] # file.tmpl --> file.xml +--> file.ps (psdocs) [by db2ps or xmlto] #+--> file.pdf (pdfdocs) [by db2pdf or xmlto] #+--> DIR=file (htmldocs) [by xmlto] #+--> man/ (mandocs) [by xmlto] ### # DOCPROC is used for two purposes: # 1) To generate a dependency list for a .tmpl file # 2) To preprocess a .tmpl file and call kernel-doc with # appropriate parameters. # The following rules are used to generate the .xml documentation # required to generate the final targets. (ps, pdf, html). - The DOCPROC markup directives are described in the kernel-doc-nano-HOWTO: https://github.com/torvalds/linux/blob/master/Documentation/kernel-doc-nano-HOWTO.txt#L332 My aim is to implement a reST-directive which is similar. E.g: - Device Instance and Driver Handling !Pdrivers/gpu/drm/drm_drv.c driver instance overview !Edrivers/gpu/drm/drm_drv.c - In reST the directive might look like: - Device Instance and Driver Handling === .. kernel-doc:: drivers/gpu/drm/drm_drv.c :doc: driver instance overview :exported: - > I'd be *very* hesitant about adding the kind of things you do in > reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc > comments must be as simple as possible with basically pass-through of > the comment blocks to sphinx. Right, but you forgot, that the current markup in the source code comments is based on the kernel-doc-nano-HOWTO and I guess no one will migrate all these source code comments to reST markup ;-) So there is a need to differentiate between *vintage* kernel-doc markup and reST markup. My suggestion is to add a tag to the comments, here a short example what this might look like. -- /** * drm_minor_release - Release DRM minor * @minor: Pointer to DRM minor object * * Release a minor that was previously acquired via drm_minor_acquire(). */ -- in short: the vintage style does not need any change and comments with reST markup has a tag ":reST:" in the first line(s) ... -- /** * :reST: * * .. c:function:: void drm_minor_release(struct drm_minor *minor) * *Release DRM minor * *:param struct drm_minor \*minor: Pointer to DRM minor object * */ -- Comments with the ":reST:" tag could be exported and pass-through to sphinx. > Specifically, do not attempt to detect and > parse elements like lists in kernel-doc. If your markup in the comments is plain reST, no need to parse, but there are markups in the (vintage) comments which made use of individual text-markups, e.g. the markup of lists or ASCII-art. This individual text-markups has not been converted/rendered in the docbook output, but I wanted to convert this individuals to reST, to render them in Sphinx. E.g. compare the member & description section of struct drm-display-mode ... DocBook: * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html kernel-doc reST with the additional reformat_block_rst function: * http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode -- Markus-- > -- > Jani Nikula, Intel Open Source Technology Center > -- > To unsubscribe from this list: send the line "unsubscribe linux-media" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016, Markus Heiser wrote: > but I think this will not by very helpful, as long as you miss > a similar ".tmpl" workflow for reST documents. > > I'am working on a reST directive (named: "kernel-doc") to provide a > similar ".tmpl" workflow within plain reST. The first step towards > is done with (my) modified kernel-doc script ... > > * > https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736 > > which produce reST from source code comments. E.g. this content is > generated with it. What do you mean by ".tmpl workflow"? I'd be *very* hesitant about adding the kind of things you do in reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc comments must be as simple as possible with basically pass-through of the comment blocks to sphinx. Specifically, do not attempt to detect and parse elements like lists in kernel-doc. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Hi all, (hi Jonathan, please take note of my offer below) Am 03.05.2016 um 16:31 schrieb Daniel Vetter : > Hi all, > > So sounds like moving ahead with rst/sphinx is the option that should > allow us to address everyone's concerns eventually? Of course the > first one won't have it all (media seems really tricky), ... BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab : > With that sense, the "List tables" format is also not good, as > one row addition would generate several hunks (one for each column > of the table), making harder to review the patch by just looking at > the diff. For this, I wrote the "flat-table" reST-directive, which adds missing cells automatically: doc: http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table source: https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py I used "flat-table" to migrate all DocBook-XML documents to reST. With this directive, I also managed to migrate the complete media book (no more TODOs) incl. the large tables like them from subdev-formats: https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html (Rendering large tables is a general discussion which should not take place in this MT) > but I'd like > to get something awesome in this area closer to mainline. I'm stalling > on typing more fancyful gpu docs right now (with tables, pictures and > stuff) since that would require a pile of needless work to redo when > we switch a few times more ;-) Are your "fancyful gpu" written from scratch, or will it be an rewrite of the Documentation/DocBook/gpu.tmpl? If it is the last; as starting point you can use a copy of: https://github.com/return42/sphkerneldoc/tree/master/doc/books/gpu but I think this will not by very helpful, as long as you miss a similar ".tmpl" workflow for reST documents. I'am working on a reST directive (named: "kernel-doc") to provide a similar ".tmpl" workflow within plain reST. The first step towards is done with (my) modified kernel-doc script ... * https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736 which produce reST from source code comments. E.g. this content is generated with it. * http://return42.github.io/sphkerneldoc/linux_src_doc/index.html > And Jani also wants to get this in, but he doesn't really want to > spend more effort on a system that can't be merged. > > So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov? My recommendation is to start with reST-markup if you are writing new docs from scratch. Workflows like the .tmpl one can be added to the kernel build process step by step and there is no need to drop the DocBook-XML process. Every author should be free to decide by himself, when it is time to migrate his DocBook sources into a reST building process. I'am new to the kernel, my aim is to support the doc maintainer (contact me), e.g. to migrate existing DocBooks to reST, elaborate (build) procedures for improved reST support within the kernel sources and to give recommendation to infrastructures. @Jonathan: I know, you have no time right now. If it is OK for you, I elaborate all these reST stuff within my POC at github (this will take some time). Later, if you have time, we can merge the made experience and tools to the kernel build process .. is that okay for you? -- Markus -- > > Cheers, Daniel > > On Wed, Apr 27, 2016 at 4:28 PM, Grant Likely > wrote: >> On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet wrote: >>> On Fri, 8 Apr 2016 17:12:27 +0200 >>> Markus Heiser wrote: >>> motivated by this MT, I implemented a toolchain to migrate the kernel’s DocBook XML documentation to reST markup. It converts 99% of the docs well ... to gain an impression how kernel-docs could benefit from, visit my sphkerneldoc project page on github: http://return42.github.io/sphkerneldoc/ >>> >>> So I've obviously been pretty quiet on this recently. Apologies...I've >>> been dealing with an extended death-in-the-family experience, and there is >>> still a fair amount of cleanup to be done. >>> >>> Looking quickly at this work, it seems similar to the results I got. But >>> there's a lot of code there that came from somewhere? I'd put together a >>> fairly simple conversion using pandoc and a couple of short sed scripts; >>> is there a reason for a more complex solution? >>> >>> Thanks for looking into this, anyway; I hope to be able to focus more on >>> it shortly. >> >> Hi Jon, >> >> Thanks for digging into this. FWIW, here is my $0.02. I've been >> working on restarting the devicetree specification, and after looking >> at both reStructuredText and Asciidoc(tor) I thought I liked the >> Asciidoc markup better, so chose that. I then proceeded to spend weeks >> trying to get reasonable output from the toolchain. When I got fed up >> and gave Sphinx
Re: Kernel docs: muddying the waters a bit
Daniel Vetter writes: > So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer > pov? I think the right answer for today is to use sphinx to generate docs From inline comments, to encourage outline docs to give it a try but to allow doc writers to use whatever works for them. That will leave the media docs using the existing system so that their tables don't get wrecked, while providing guidance to new doc writers and allowing inline docs to include markup. -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
Hi all, So sounds like moving ahead with rst/sphinx is the option that should allow us to address everyone's concerns eventually? Of course the first one won't have it all (media seems really tricky), but I'd like to get something awesome in this area closer to mainline. I'm stalling on typing more fancyful gpu docs right now (with tables, pictures and stuff) since that would require a pile of needless work to redo when we switch a few times more ;-) And Jani also wants to get this in, but he doesn't really want to spend more effort on a system that can't be merged. So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov? Cheers, Daniel On Wed, Apr 27, 2016 at 4:28 PM, Grant Likely wrote: > On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet wrote: >> On Fri, 8 Apr 2016 17:12:27 +0200 >> Markus Heiser wrote: >> >>> motivated by this MT, I implemented a toolchain to migrate the kernel’s >>> DocBook XML documentation to reST markup. >>> >>> It converts 99% of the docs well ... to gain an impression how >>> kernel-docs could benefit from, visit my sphkerneldoc project page >>> on github: >>> >>> http://return42.github.io/sphkerneldoc/ >> >> So I've obviously been pretty quiet on this recently. Apologies...I've >> been dealing with an extended death-in-the-family experience, and there is >> still a fair amount of cleanup to be done. >> >> Looking quickly at this work, it seems similar to the results I got. But >> there's a lot of code there that came from somewhere? I'd put together a >> fairly simple conversion using pandoc and a couple of short sed scripts; >> is there a reason for a more complex solution? >> >> Thanks for looking into this, anyway; I hope to be able to focus more on >> it shortly. > > Hi Jon, > > Thanks for digging into this. FWIW, here is my $0.02. I've been > working on restarting the devicetree specification, and after looking > at both reStructuredText and Asciidoc(tor) I thought I liked the > Asciidoc markup better, so chose that. I then proceeded to spend weeks > trying to get reasonable output from the toolchain. When I got fed up > and gave Sphinx a try, I was up and running with reasonable PDF and > HTML output in a day and a half. > > Honestly, in the end I think we could make either tool do what is > needed of it. However, my impression after trying to do a document > that needs to have nice publishable output with both tools is that > Sphinx is easier to work with, simpler to extend, better supported. My > vote is firmly behind Sphinx/reStructuredText. > > g. -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet wrote: > On Fri, 8 Apr 2016 17:12:27 +0200 > Markus Heiser wrote: > >> motivated by this MT, I implemented a toolchain to migrate the kernel’s >> DocBook XML documentation to reST markup. >> >> It converts 99% of the docs well ... to gain an impression how >> kernel-docs could benefit from, visit my sphkerneldoc project page >> on github: >> >> http://return42.github.io/sphkerneldoc/ > > So I've obviously been pretty quiet on this recently. Apologies...I've > been dealing with an extended death-in-the-family experience, and there is > still a fair amount of cleanup to be done. > > Looking quickly at this work, it seems similar to the results I got. But > there's a lot of code there that came from somewhere? I'd put together a > fairly simple conversion using pandoc and a couple of short sed scripts; > is there a reason for a more complex solution? > > Thanks for looking into this, anyway; I hope to be able to focus more on > it shortly. Hi Jon, Thanks for digging into this. FWIW, here is my $0.02. I've been working on restarting the devicetree specification, and after looking at both reStructuredText and Asciidoc(tor) I thought I liked the Asciidoc markup better, so chose that. I then proceeded to spend weeks trying to get reasonable output from the toolchain. When I got fed up and gave Sphinx a try, I was up and running with reasonable PDF and HTML output in a day and a half. Honestly, in the end I think we could make either tool do what is needed of it. However, my impression after trying to do a document that needs to have nice publishable output with both tools is that Sphinx is easier to work with, simpler to extend, better supported. My vote is firmly behind Sphinx/reStructuredText. g.
Re: Kernel docs: muddying the waters a bit
Em Mon, 18 Apr 2016 10:10:17 +0200 Markus Heiser escreveu: > Hi Mauro, hi kernel-doc authors, > > Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab > : > > > Em Fri, 8 Apr 2016 17:12:27 +0200 > > Markus Heiser escreveu: > > > >> Hi kernel-doc authors, > >> > >> motivated by this MT, I implemented a toolchain to migrate the kernel’s > >> DocBook XML documentation to reST markup. > >> > >> It converts 99% of the docs well ... to gain an impression how > >> kernel-docs could benefit from, visit my sphkerneldoc project page > >> on github: > >> > >> http://return42.github.io/sphkerneldoc/ > >> > >> The sources available at: > >> > >> https://github.com/return42/sphkerneldoc > >> > >> The work is underway, suggestions are welcome! > >> > >> .. have a nice weekend .. > > > > Hi Markus, > > > > Thanks for your work. I'm basically worried about the media docbook, > > with has some complexities that I was not able to figure out a way to > > convert it to reST. > > > > So, let me pinpoint some issues that I noticed there, on a quick > > look. > > > > The first thing I noticed is that the index doesn't match what's > > there, when generated from DocBook: > > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html > > > > So, for instance, "Interfaces" is at the same level as "Input/Output". > > > > This sounds like an something went wrong when dealing with the title > > indentation levels during the conversion. > > > > Yes, the hierarchical structure was broken by two causes. First was, that the > *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) > placed > all sections and sub(-sub)-sections at the same level (direct under chapter). > This was > not only broken in the linux_tv book, in the other books also. > > Thanks for pointing and sorry that I have overlooked this (I was so much > focused on > on converting single elements to reST, that I not see the wood for the trees). > > It is now fixed, may you give it a new try. Thanks! It looks good now. > > I would also like to see the titles numbered in chapters (and, if > > possible, in sections and items) and to be properly indented at the > > index. > > BTW a few words about differences between DockBook and reST (Sphinx). > > With DocBook you write *books*, the protocol (the DocBook application) has > no facility to *chunk* and interconnect several documents. The external > ENTITY > is a workaround on the SGML layer, not on XML nor on the DB-application layer. > Thats the reason, why so many XML-tools don't handle this entities and > many DocBook to (e.g.) reST tools are fail. > > With **standard** reST it is nearly the same, except there is a "include" > directive on the application layer. But this directive is very simple, > comparable to the C preprocessor "#include" directive. > > With the **superset** reST-markup of Sphinx-doc you get a the "toctree" > directive, > which lets you control how a document-tree should be build. > > http://www.sphinx-doc.org/en/stable/markup/toctree.html > > @Mauro: you mentioned a docutils (rst2*) experience in your mail > http://marc.info/?l=linux-doc&m=145735316012094&w=2 > > Because the "toctree" directive -- and other directives > we use -- are a part of a superset of the **standard** > reST, the standard docutils (like rst2*) will not work. > > OK, back to your requirements: within the toctree directive you can > set options like "maxdepth" and "numbered". It is a decision, how > deep TOCs should go and if they should be numbered. IMO, in a > HTML rendered page, with a proper navigation bar on the side, deep > TOCs in the running text have no pros, they only blow up the running > text and bring more scrolling with. In my sense numbering chapters > make only sense in books, not in HTML pages, where you have hyperlinks. > > Just for demonstration, I added numbering in the linux-tv book: > > https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3 > > May be, I will drop it later, because all reST sources are generated > by a make target and I always commit the whole reST tree. As I said, > it is a decision which might be made later, when the migration takes > places. This is the uAPI spec DocBook, that we modify frequently, as we add more features to the Kernel, and as we make sure that all drivers will behave the same. So, from time to time, we need to clarify some topics at the documentation. By having a numeration, it is easier for us to discuss things like: "1.2.10.14. V4L2_PIX_FMT_VYUY (‘VYUY’)" is not properly described and requires some sort of clarification. Ok, one could also refer to it via a hyperlink, but several Kernel media maintainers prefer to generate a single big html file, as it makes easier to locate everything it is needed on it. So, with the item number, they can just seek for "1.2.10.14. " string to find the item that is under discussion. > > Also
Re: Kernel docs: muddying the waters a bit
Hi Jonahtan, Am 12.04.2016 um 17:46 schrieb Jonathan Corbet : > On Fri, 8 Apr 2016 17:12:27 +0200 > Markus Heiser wrote: > >> motivated by this MT, I implemented a toolchain to migrate the kernel’s >> DocBook XML documentation to reST markup. >> >> It converts 99% of the docs well ... to gain an impression how >> kernel-docs could benefit from, visit my sphkerneldoc project page >> on github: >> >> http://return42.github.io/sphkerneldoc/ > > So I've obviously been pretty quiet on this recently. Apologies...I've > been dealing with an extended death-in-the-family experience, and there is > still a fair amount of cleanup to be done. > > Looking quickly at this work, it seems similar to the results I got. But > there's a lot of code there that came from somewhere? >From me? ... except the kernel-doc script which is a fork from your git://git.lwn.net/linux.git doc/sphinx > I'd put together a > fairly simple conversion using pandoc and a couple of short sed scripts; > is there a reason for a more complex solution? It depends. If you have a simple DocBook with less various markup, maybe not. May you want to read my remarks about migration tools and especially pandoc: * https://return42.github.io/sphkerneldoc/articles/dbtools.html#remarks-on-pandoc A few more words about, what I have done: I wrote a lib of XML filters which might be also usefully in other migration projects (dbxml). * https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py It uses a xml-parser, pandoc, pandoc-filters and regular expressions. Because I did not implemented a whole converter, I hacked around pandoc. Thats why conversion is done in several steps: 1. copy xml file(s) to a cache space 2. substitude unsolved internal and external entities 3. filter all xml files * run custom hooks on every node * apply filters on every node and inject reST into the XML-tree where pandoc fails. https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py#L515 4. convert intermediary XML result with pandoc to json (needed by pandoc filters) 5. apply pandoc-filter and clean up the injected reST markup from step3 6. convert filtered json to reST 7. fix the produce reST with regular expression ... the last step is similar to your sed scripts. And I wrote a commandline Interface to use this lib (see func db2rst): * https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L146 With this db2rst all kernel DB-XML books could be migrated, except the linux-tv book, which has much more complexity. For this, there is a separated commandline called media2rst * https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L107 The media2rst needs several special handlings, which is implemented in hooks (the dbxml interface method) * https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py Summarize, why should one prefer this tools over pandoc + sed? * Pandoc coverage is less on reading and writing, this is where dbxml comes into play - reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23 - writing reST has many bugs and leaks (you fixed some of them with sed) * Pandoc does not support external entities (linux-tv), covered by dbxml * dbxml brings the ability to chunk one large XML book into small reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking * dbxml lets you manipulate the XML source before you convert it to reST this might helpfull e.g. if you have to convert single-column informal-tables to lists or other things ... in short; dbxml and it's hooks are the key to hack everything you need in a full automated DocBook-->reST migration workflow. --Markus-- > Thanks for looking into this, anyway; I hope to be able to focus more on > it shortly. > > jon > -- > To unsubscribe from this list: send the line "unsubscribe linux-media" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Hi Mauro, hi kernel-doc authors, Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab : > Em Fri, 8 Apr 2016 17:12:27 +0200 > Markus Heiser escreveu: > >> Hi kernel-doc authors, >> >> motivated by this MT, I implemented a toolchain to migrate the kernel’s >> DocBook XML documentation to reST markup. >> >> It converts 99% of the docs well ... to gain an impression how >> kernel-docs could benefit from, visit my sphkerneldoc project page >> on github: >> >> http://return42.github.io/sphkerneldoc/ >> >> The sources available at: >> >> https://github.com/return42/sphkerneldoc >> >> The work is underway, suggestions are welcome! >> >> .. have a nice weekend .. > > Hi Markus, > > Thanks for your work. I'm basically worried about the media docbook, > with has some complexities that I was not able to figure out a way to > convert it to reST. > > So, let me pinpoint some issues that I noticed there, on a quick > look. > > The first thing I noticed is that the index doesn't match what's > there, when generated from DocBook: > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html > > So, for instance, "Interfaces" is at the same level as "Input/Output". > > This sounds like an something went wrong when dealing with the title > indentation levels during the conversion. > Yes, the hierarchical structure was broken by two causes. First was, that the *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) placed all sections and sub(-sub)-sections at the same level (direct under chapter). This was not only broken in the linux_tv book, in the other books also. Thanks for pointing and sorry that I have overlooked this (I was so much focused on on converting single elements to reST, that I not see the wood for the trees). It is now fixed, may you give it a new try. > I would also like to see the titles numbered in chapters (and, if > possible, in sections and items) and to be properly indented at the > index. BTW a few words about differences between DockBook and reST (Sphinx). With DocBook you write *books*, the protocol (the DocBook application) has no facility to *chunk* and interconnect several documents. The external ENTITY is a workaround on the SGML layer, not on XML nor on the DB-application layer. Thats the reason, why so many XML-tools don't handle this entities and many DocBook to (e.g.) reST tools are fail. With **standard** reST it is nearly the same, except there is a "include" directive on the application layer. But this directive is very simple, comparable to the C preprocessor "#include" directive. With the **superset** reST-markup of Sphinx-doc you get a the "toctree" directive, which lets you control how a document-tree should be build. http://www.sphinx-doc.org/en/stable/markup/toctree.html @Mauro: you mentioned a docutils (rst2*) experience in your mail http://marc.info/?l=linux-doc&m=145735316012094&w=2 Because the "toctree" directive -- and other directives we use -- are a part of a superset of the **standard** reST, the standard docutils (like rst2*) will not work. OK, back to your requirements: within the toctree directive you can set options like "maxdepth" and "numbered". It is a decision, how deep TOCs should go and if they should be numbered. IMO, in a HTML rendered page, with a proper navigation bar on the side, deep TOCs in the running text have no pros, they only blow up the running text and bring more scrolling with. In my sense numbering chapters make only sense in books, not in HTML pages, where you have hyperlinks. Just for demonstration, I added numbering in the linux-tv book: https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3 May be, I will drop it later, because all reST sources are generated by a make target and I always commit the whole reST tree. As I said, it is a decision which might be made later, when the migration takes places. > Also, it seems that there's still a lot of work to do, as there are several > tables that are missing conversion, like the table for "struct > v4l2_pix_format": > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html > > and the big tables at: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > Yes, I marked them as TODO: https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262 aspect *authoring tables* (see below) > Also, some tables that are not so big like: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html > > don't look nice, at least on my browser, as the "white" area is too small, > and some cells seem to be broken in two, because of the color changes in the > middle
Re: Kernel docs: muddying the waters a bit
On Fri, 8 Apr 2016 17:12:27 +0200 Markus Heiser wrote: > motivated by this MT, I implemented a toolchain to migrate the kernel’s > DocBook XML documentation to reST markup. > > It converts 99% of the docs well ... to gain an impression how > kernel-docs could benefit from, visit my sphkerneldoc project page > on github: > > http://return42.github.io/sphkerneldoc/ So I've obviously been pretty quiet on this recently. Apologies...I've been dealing with an extended death-in-the-family experience, and there is still a fair amount of cleanup to be done. Looking quickly at this work, it seems similar to the results I got. But there's a lot of code there that came from somewhere? I'd put together a fairly simple conversion using pandoc and a couple of short sed scripts; is there a reason for a more complex solution? Thanks for looking into this, anyway; I hope to be able to focus more on it shortly. jon
Re: Kernel docs: muddying the waters a bit
Hi Markus, On 04/08/16 17:12, Markus Heiser wrote: > Hi kernel-doc authors, > > motivated by this MT, I implemented a toolchain to migrate the kernel’s > DocBook XML documentation to reST markup. > > It converts 99% of the docs well ... to gain an impression how > kernel-docs could benefit from, visit my sphkerneldoc project page > on github: > > http://return42.github.io/sphkerneldoc/ > > The sources available at: > > https://github.com/return42/sphkerneldoc > > The work is underway, suggestions are welcome! I have to admit that this looks pretty good :-) My main remark based on my quick scan through the docs is that anything in typewriter font seems to be shown as red text with a rectangle around it. That's quite jarring for me and I think it should be just shown as normal text, just using a non-proportional font, just like in the original. I also noticed that the 'title' of tables ends with a '¶' character for some reason. See e.g. the struct v4l2_audioout table in http://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/vidioc-g-audioout.html Regards, Hans > > .. have a nice weekend .. > > --M-- > > > Am 13.03.2016 um 16:33 schrieb Markus Heiser : > >> >> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab >> : >> >>> Em Thu, 10 Mar 2016 12:25:58 +0200 >>> Jani Nikula escreveu: >>> TL;DR? Skip to the last paragraph. On Wed, 09 Mar 2016, Mauro Carvalho Chehab wrote: > I guess the conversion to asciidoc format is now in good shape, > at least to demonstrate that it is possible to use this format for the > media docbook. Still, there are lots of broken references. Getting references right with asciidoc is a big problem in the kernel-doc side. As I wrote before, the proofs of concept only worked because everything was processed as one big file (via includes). The Asciidoctor inter-document references won't help, because we won't know the target document name while processing kernel-doc. >>> >>> I was able to produce chunked htmls here with: >>> >>> asciidoctor -b docbook45 media_api.adoc >>> xmlto -o html-dir html media_api.xml >>> >>> The results are at: >>> >>> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/ >>> >>> But yeah, all references seem to be broken there. It could be due to some >>> conversion issue (I didn't actually tried to check what's wrong there), >>> but I think that there's something not ok with docbook45 >>> output for multi-part documents (on both AsciiDoc and Asciidoctor). >>> Sphinx is massively better at handling cross references for kernel-doc. We can use domains (C language) and roles (e.g. functions, types, etc.) for the references, which provide kind of namespaces. Sphinx warns for referencing non-existing targets, but doesn't generate broken links in the result like Asciidoctor does. For example, in the documentation for a function that has struct foo as parameter or return type, a cross reference to struct foo is added automagically, but only if documentation for struct foo actually exists. In Asciidoctor, we would have to blindly generate the references ourselves, and try to resolve broken links ourselves by somehow post-processing the result. > Yet, from my side, if we're willing to get rid of DocBook, then > Asciidoctor seems to be the *only* alternative so far to parse the > complex media documents. I think you mean, "get rid of DocBook as source format", not altogether? I'm yet to be convinved we could rely on Asciidoctor's native formats. >>> >>> What I mean is that, right now, I see only two alternatives for the >>> media uAPI documentation: >>> 1) keep using DocBook; >>> 2) AsciiDoc/Asciidoctor. >>> >>> Sphinx doesn't have what's needed to support the complexity of the >>> media books, specially since cell span seems to be possible only >>> by using asciiArt formats. Writing a big table using asciiArt is >>> something that is a *real pain*. Also, as tested, if the table is >>> too big, it fails to parse such asciiArt tables. So, while Sphinx >>> doesn't have a decent way to describe tables, we can't use it. >> >> >> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc, >> (mostly) you have more then one choice .. e.g. import csv tables .. >> but this should be discussed by example ... >> >> >>> If it starts implementing it, then we can check if the other >>> features used by the media documentation are also supported. >>> Probably, multi-part books would be another pain with Sphinx. >>> We have actually 4 books inside a common body. A few chapters >>> (like book licensing, bibliography, error codes) are shared >>> by all 4 documents. >>> >>> But, so far, I can't see any way to port media books without >>> lots of lot of work to develop new features at the Sphinx code. >> >> >> may I can help you ... >> >>
Re: Kernel docs: muddying the waters a bit
Hi kernel-doc authors, motivated by this MT, I implemented a toolchain to migrate the kernel’s DocBook XML documentation to reST markup. It converts 99% of the docs well ... to gain an impression how kernel-docs could benefit from, visit my sphkerneldoc project page on github: http://return42.github.io/sphkerneldoc/ The sources available at: https://github.com/return42/sphkerneldoc The work is underway, suggestions are welcome! .. have a nice weekend .. --M-- Am 13.03.2016 um 16:33 schrieb Markus Heiser : > > Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab > : > >> Em Thu, 10 Mar 2016 12:25:58 +0200 >> Jani Nikula escreveu: >> >>> TL;DR? Skip to the last paragraph. >>> >>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab wrote: I guess the conversion to asciidoc format is now in good shape, at least to demonstrate that it is possible to use this format for the media docbook. Still, there are lots of broken references. >>> >>> Getting references right with asciidoc is a big problem in the >>> kernel-doc side. As I wrote before, the proofs of concept only worked >>> because everything was processed as one big file (via includes). The >>> Asciidoctor inter-document references won't help, because we won't know >>> the target document name while processing kernel-doc. >> >> I was able to produce chunked htmls here with: >> >> asciidoctor -b docbook45 media_api.adoc >> xmlto -o html-dir html media_api.xml >> >> The results are at: >> >> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/ >> >> But yeah, all references seem to be broken there. It could be due to some >> conversion issue (I didn't actually tried to check what's wrong there), >> but I think that there's something not ok with docbook45 >> output for multi-part documents (on both AsciiDoc and Asciidoctor). >> >>> Sphinx is massively better at handling cross references for >>> kernel-doc. We can use domains (C language) and roles (e.g. functions, >>> types, etc.) for the references, which provide kind of >>> namespaces. Sphinx warns for referencing non-existing targets, but >>> doesn't generate broken links in the result like Asciidoctor does. >>> >>> For example, in the documentation for a function that has struct foo as >>> parameter or return type, a cross reference to struct foo is added >>> automagically, but only if documentation for struct foo actually >>> exists. In Asciidoctor, we would have to blindly generate the references >>> ourselves, and try to resolve broken links ourselves by somehow >>> post-processing the result. >>> Yet, from my side, if we're willing to get rid of DocBook, then Asciidoctor seems to be the *only* alternative so far to parse the complex media documents. >>> >>> I think you mean, "get rid of DocBook as source format", not altogether? >>> I'm yet to be convinved we could rely on Asciidoctor's native formats. >> >> What I mean is that, right now, I see only two alternatives for the >> media uAPI documentation: >> 1) keep using DocBook; >> 2) AsciiDoc/Asciidoctor. >> >> Sphinx doesn't have what's needed to support the complexity of the >> media books, specially since cell span seems to be possible only >> by using asciiArt formats. Writing a big table using asciiArt is >> something that is a *real pain*. Also, as tested, if the table is >> too big, it fails to parse such asciiArt tables. So, while Sphinx >> doesn't have a decent way to describe tables, we can't use it. > > > Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc, > (mostly) you have more then one choice .. e.g. import csv tables .. > but this should be discussed by example ... > > >> If it starts implementing it, then we can check if the other >> features used by the media documentation are also supported. >> Probably, multi-part books would be another pain with Sphinx. >> We have actually 4 books inside a common body. A few chapters >> (like book licensing, bibliography, error codes) are shared >> by all 4 documents. >> >> But, so far, I can't see any way to port media books without >> lots of lot of work to develop new features at the Sphinx code. > > > may I can help you ... > > >>> The toolchain gets faster, easier to debug and simplified a lot with >>> DocBook out of the equation completely. Sphinx itself is stable, widely >>> available, and well documented. IMO there's sufficient native output >>> format support. There are plenty of really nice extensions >>> available. There's a possibility of doing kernel-doc as an extension in >>> the future (either by calling current kernel-doc from the extension or >>> by rewriting it). >> >> Well, if we go to Sphinx for kernel-doc, that means that we'll need >> 2 different tools for the documentation: >> - Sphinx for kernel-doc >> - either DocBook or Asciidoctor/AsciiDoc for media. >> >> IMHO, this is the worse scenario, as we'll keep depending on >>
Re: Kernel docs: muddying the waters a bit
Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab : > Em Thu, 10 Mar 2016 12:25:58 +0200 > Jani Nikula escreveu: > >> TL;DR? Skip to the last paragraph. >> >> On Wed, 09 Mar 2016, Mauro Carvalho Chehab wrote: >>> I guess the conversion to asciidoc format is now in good shape, >>> at least to demonstrate that it is possible to use this format for the >>> media docbook. Still, there are lots of broken references. >> >> Getting references right with asciidoc is a big problem in the >> kernel-doc side. As I wrote before, the proofs of concept only worked >> because everything was processed as one big file (via includes). The >> Asciidoctor inter-document references won't help, because we won't know >> the target document name while processing kernel-doc. > > I was able to produce chunked htmls here with: > > asciidoctor -b docbook45 media_api.adoc > xmlto -o html-dir html media_api.xml > > The results are at: > > https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/ > > But yeah, all references seem to be broken there. It could be due to some > conversion issue (I didn't actually tried to check what's wrong there), > but I think that there's something not ok with docbook45 > output for multi-part documents (on both AsciiDoc and Asciidoctor). > >> Sphinx is massively better at handling cross references for >> kernel-doc. We can use domains (C language) and roles (e.g. functions, >> types, etc.) for the references, which provide kind of >> namespaces. Sphinx warns for referencing non-existing targets, but >> doesn't generate broken links in the result like Asciidoctor does. >> >> For example, in the documentation for a function that has struct foo as >> parameter or return type, a cross reference to struct foo is added >> automagically, but only if documentation for struct foo actually >> exists. In Asciidoctor, we would have to blindly generate the references >> ourselves, and try to resolve broken links ourselves by somehow >> post-processing the result. >> >>> Yet, from my side, if we're willing to get rid of DocBook, then >>> Asciidoctor seems to be the *only* alternative so far to parse the >>> complex media documents. >> >> I think you mean, "get rid of DocBook as source format", not altogether? >> I'm yet to be convinved we could rely on Asciidoctor's native formats. > > What I mean is that, right now, I see only two alternatives for the > media uAPI documentation: > 1) keep using DocBook; > 2) AsciiDoc/Asciidoctor. > > Sphinx doesn't have what's needed to support the complexity of the > media books, specially since cell span seems to be possible only > by using asciiArt formats. Writing a big table using asciiArt is > something that is a *real pain*. Also, as tested, if the table is > too big, it fails to parse such asciiArt tables. So, while Sphinx > doesn't have a decent way to describe tables, we can't use it. Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc, (mostly) you have more then one choice .. e.g. import csv tables .. but this should be discussed by example ... > If it starts implementing it, then we can check if the other > features used by the media documentation are also supported. > Probably, multi-part books would be another pain with Sphinx. > We have actually 4 books inside a common body. A few chapters > (like book licensing, bibliography, error codes) are shared > by all 4 documents. > > But, so far, I can't see any way to port media books without > lots of lot of work to develop new features at the Sphinx code. may I can help you ... >> The toolchain gets faster, easier to debug and simplified a lot with >> DocBook out of the equation completely. Sphinx itself is stable, widely >> available, and well documented. IMO there's sufficient native output >> format support. There are plenty of really nice extensions >> available. There's a possibility of doing kernel-doc as an extension in >> the future (either by calling current kernel-doc from the extension or >> by rewriting it). > > Well, if we go to Sphinx for kernel-doc, that means that we'll need > 2 different tools for the documentation: > - Sphinx for kernel-doc > - either DocBook or Asciidoctor/AsciiDoc for media. > > IMHO, this is the worse scenario, as we'll keep depending on > DocBook plus requiring Sphinx, but it is up to Jon to decide. > The migration of kernel-doc is a long term project, not a one shot job. The scope of documents to migrate is not limited to the files with DocBook markup in, most documents have not a real markup. Please take a look at my thoughts and efforts about migration. * https://sphkerneldoc.readthedocs.org * https://github.com/return42/sphkerneldoc.git sphkerneldoc.git is a small project started this weekend, within this project I show you, how migration could be done and we can discuss concerns like "tables and cell-spans" by example. Believe me, most concerns discussed
Re: Kernel docs: muddying the waters a bit
Em Thu, 10 Mar 2016 12:25:58 +0200 Jani Nikula escreveu: > TL;DR? Skip to the last paragraph. > > On Wed, 09 Mar 2016, Mauro Carvalho Chehab wrote: > > I guess the conversion to asciidoc format is now in good shape, > > at least to demonstrate that it is possible to use this format for the > > media docbook. Still, there are lots of broken references. > > Getting references right with asciidoc is a big problem in the > kernel-doc side. As I wrote before, the proofs of concept only worked > because everything was processed as one big file (via includes). The > Asciidoctor inter-document references won't help, because we won't know > the target document name while processing kernel-doc. I was able to produce chunked htmls here with: asciidoctor -b docbook45 media_api.adoc xmlto -o html-dir html media_api.xml The results are at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/ But yeah, all references seem to be broken there. It could be due to some conversion issue (I didn't actually tried to check what's wrong there), but I think that there's something not ok with docbook45 output for multi-part documents (on both AsciiDoc and Asciidoctor). > Sphinx is massively better at handling cross references for > kernel-doc. We can use domains (C language) and roles (e.g. functions, > types, etc.) for the references, which provide kind of > namespaces. Sphinx warns for referencing non-existing targets, but > doesn't generate broken links in the result like Asciidoctor does. > > For example, in the documentation for a function that has struct foo as > parameter or return type, a cross reference to struct foo is added > automagically, but only if documentation for struct foo actually > exists. In Asciidoctor, we would have to blindly generate the references > ourselves, and try to resolve broken links ourselves by somehow > post-processing the result. > > > Yet, from my side, if we're willing to get rid of DocBook, then > > Asciidoctor seems to be the *only* alternative so far to parse the > > complex media documents. > > I think you mean, "get rid of DocBook as source format", not altogether? > I'm yet to be convinved we could rely on Asciidoctor's native formats. What I mean is that, right now, I see only two alternatives for the media uAPI documentation: 1) keep using DocBook; 2) AsciiDoc/Asciidoctor. Sphinx doesn't have what's needed to support the complexity of the media books, specially since cell span seems to be possible only by using asciiArt formats. Writing a big table using asciiArt is something that is a *real pain*. Also, as tested, if the table is too big, it fails to parse such asciiArt tables. So, while Sphinx doesn't have a decent way to describe tables, we can't use it. If it starts implementing it, then we can check if the other features used by the media documentation are also supported. Probably, multi-part books would be another pain with Sphinx. We have actually 4 books inside a common body. A few chapters (like book licensing, bibliography, error codes) are shared by all 4 documents. But, so far, I can't see any way to port media books without lots of lot of work to develop new features at the Sphinx code. > --- > > Mauro, I truly appreciate your efforts at evaluating both > alternatives. I also appreciate Dan's inputs on Asciidoctor. > > Despite your evaluation that Asciidoctor is the only alternative for > media documents, it is my opinion that we should go with Sphinx. > > It's an opinion, it's subjective, it's from my perspective, especially > from the kernel-doc POV, so please don't take it as a slap in the face > after all the work you've done. With that out of the way, here's why. > > For starters, Jon's Sphinx proof-of-concept at > http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and > usable. Cross references work, there are no broken links (I hacked a bit > more on kernel-doc and it gets even better). There's embedded search > (and if this gets exported to https://readthedocs.org/ the search is > even better). The API documentation is sensible and the headings aren't > mixed up with other headings. It's all there. It's what we've been > looking for. > > The toolchain gets faster, easier to debug and simplified a lot with > DocBook out of the equation completely. Sphinx itself is stable, widely > available, and well documented. IMO there's sufficient native output > format support. There are plenty of really nice extensions > available. There's a possibility of doing kernel-doc as an extension in > the future (either by calling current kernel-doc from the extension or > by rewriting it). Well, if we go to Sphinx for kernel-doc, that means that we'll need 2 different tools for the documentation: - Sphinx for kernel-doc - either DocBook or Asciidoctor/AsciiDoc for media. IMHO, this is the worse scenario, as we'll keep depending on DocBook plus requiring Sph
Re: Kernel docs: muddying the waters a bit
TL;DR? Skip to the last paragraph. On Wed, 09 Mar 2016, Mauro Carvalho Chehab wrote: > I guess the conversion to asciidoc format is now in good shape, > at least to demonstrate that it is possible to use this format for the > media docbook. Still, there are lots of broken references. Getting references right with asciidoc is a big problem in the kernel-doc side. As I wrote before, the proofs of concept only worked because everything was processed as one big file (via includes). The Asciidoctor inter-document references won't help, because we won't know the target document name while processing kernel-doc. Sphinx is massively better at handling cross references for kernel-doc. We can use domains (C language) and roles (e.g. functions, types, etc.) for the references, which provide kind of namespaces. Sphinx warns for referencing non-existing targets, but doesn't generate broken links in the result like Asciidoctor does. For example, in the documentation for a function that has struct foo as parameter or return type, a cross reference to struct foo is added automagically, but only if documentation for struct foo actually exists. In Asciidoctor, we would have to blindly generate the references ourselves, and try to resolve broken links ourselves by somehow post-processing the result. > Yet, from my side, if we're willing to get rid of DocBook, then > Asciidoctor seems to be the *only* alternative so far to parse the > complex media documents. I think you mean, "get rid of DocBook as source format", not altogether? I'm yet to be convinved we could rely on Asciidoctor's native formats. --- Mauro, I truly appreciate your efforts at evaluating both alternatives. I also appreciate Dan's inputs on Asciidoctor. Despite your evaluation that Asciidoctor is the only alternative for media documents, it is my opinion that we should go with Sphinx. It's an opinion, it's subjective, it's from my perspective, especially from the kernel-doc POV, so please don't take it as a slap in the face after all the work you've done. With that out of the way, here's why. For starters, Jon's Sphinx proof-of-concept at http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and usable. Cross references work, there are no broken links (I hacked a bit more on kernel-doc and it gets even better). There's embedded search (and if this gets exported to https://readthedocs.org/ the search is even better). The API documentation is sensible and the headings aren't mixed up with other headings. It's all there. It's what we've been looking for. The toolchain gets faster, easier to debug and simplified a lot with DocBook out of the equation completely. Sphinx itself is stable, widely available, and well documented. IMO there's sufficient native output format support. There are plenty of really nice extensions available. There's a possibility of doing kernel-doc as an extension in the future (either by calling current kernel-doc from the extension or by rewriting it). Dan keeps bringing up the active community in Asciidoctor, and how they're fixing things up as we speak... which is great, but Sphinx is here now, packaged and shipping in distros ready to use. It seems that of the two, an Asciidoctor based toolchain is currently more in need of hacking and extending to meet our needs. Which brings us to the implementation language, Python vs. Ruby. I won't make the mistake of comparing the relative merits of the languages, but I'll boldly claim the set of kernel developers who know Python is likely larger than the set of kernel developers who know Ruby [citation needed]. AFAICT there are no Ruby tools in the kernel tree, but there is a bunch of Python. My own very limited and subjective experience with other tools around the kernel is that Python is much more popular than Ruby. So my claim here is that we're in a better position to hack on Sphinx extensions ourselves than Asciidoctor. My conclusion is that Sphinx covers the vast majority of the needs of our documentation producers and consumers, in an amazing way, out of the box, better than Asciidoctor. Which brings us to the minority and the parts where Sphinx falls short, media documentation in particular. It's complex documentation, with very specific requirements on the output, especially that many things remain exactly as they are now. It also feels like the target is more to have standalone media documentation, and not so much to be aligned with and be part of the rest of the kernel documentation. I want to question the need to have all kernel documentation use tools that meet the strict requirements of the outlier, when there's a better alternative for the vast majority of the documentation. Especially when Asciidoctor isn't a ready solution for media documentation either. In summary, my proposal is to go with Sphinx, leave media docs as DocBook for now, and see if and how they can be converted to Sphinx/reStructuredText later on when we have everything else in pla
Re: Kernel docs: muddying the waters a bit
Em Tue, 8 Mar 2016 12:39:21 -0300 Mauro Carvalho Chehab escreveu: > Pandoc failed to fully convert it, but at least it left all the texts, > with prevented rewriting it from scratch. This is the manual fix > I applied to it: > > https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc > > FYI, we have 133 xml files at the media uAPI doc with refmeta. I used pandoc to convert from the html files and manually edited it. I also fixed lots of other issues with the conversion. I guess the conversion to asciidoc format is now in good shape, at least to demonstrate that it is possible to use this format for the media docbook. Still, there are lots of broken references. The proof of concept html file is at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html I also added the ascii doc files there, at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/ And I'm keeping the git tree, with helps to identify the work that was needed to make it work: https://git.linuxtv.org/mchehab/asciidoc-poc.git In summary, AsciiDoc, formatted via AsciiDoctor worked fine to produce an html file. PROBLEMS 1) I was not able to produce outputs on any other format. For example, when trying to generate docbook45 output, it sems that part of the trouble was due to pandoc conversion. It produces links like: link:#ftn.id-1.4.11.43.5.11.2.7.2.6.2[^[a]^] Which causes errors with DocBook parsers, like xmllint: media_api.xml:32300: parser error : Opening and ending tag mismatch: superscript line 32300 and ulink [a] ^ I suspect that this is fixable. I may try to fix it later. 2) It seems that Asciidoctor doesn't allow annexes per document part. It numberates them as chapters, instead of using A, B, C, ... 3) Even producing the html without troubles, it produces an error: asciidoctor: ERROR: media_api.adoc: line 57: invalid part, must have at least one section (e.g., chapter, appendix, etc.) 4) There are some things that got lost during the conversion, like copyright notes and revision notes. This could be simply a problem with pandoc conversion. Nothing serious, I guess, as we could insert the lost data manually. Yet, it means that, to move from the PoC to the Kernel, there are still lots of work to do. I was unable do discover why, nor to suppress this error message. Yet, from my side, if we're willing to get rid of DocBook, then Asciidoctor seems to be the *only* alternative so far to parse the complex media documents. Regards, Mauro
Re: Kernel docs: muddying the waters a bit
On Wed, 09 Mar 2016, Dan Allen wrote: > On Tue, Mar 8, 2016 at 6:58 AM, Jani Nikula wrote: > >> I need to look into this again. Is there a specific option or directive >> to produce split output for includes? When I tried this, the result was >> just one big output file. (And indeed we'd need both. Some includes we >> want embedded, some includes should produce separate outputs.) >> > > Nope. What I'm saying is that you run Asciidoctor on each sub-master > include file (an include that manages a part or chapter). That gives you > your individual part/chapter files. Then you need to make an index page, > probably by using the Asciidoctor API to itemize all the chapters as a list > or something. Bummer. Getting the inter-document cross references right may become tricky. We'll be generating plenty of snippets of lightweight markup from source code documentation comments. At the time of processing, we won't know where e.g. a specific function to be cross referenced is documented, if at all. We can't require the documentation comment writers to figure that out either; it's too burdensome, too ugly in the code, and they'll bitrot quickly. Cross referencing in the asciidoc proofs of concept have worked because they've all done the processing as a single single unit, with includes. These hacks have also ignored any broken links, and there have been > Yes, it does require some thinking about cross references. There is a lot > more we can do out of the box, but all those references can be fixed with a > little bit of post-processing in the meantime. It seems to me Sphinx provides much better support regarding cross references, out of the box, within documents and to external documents (intersphinx), with target roles and domains, including validation and not creating broken links in the output. Looking at the current hacks we have for post-processing references, I'm really not thrilled about the prospect of keeping or redoing that. See how this works in Jon's Sphinx test [1]. At the time of generating the markup from source comments, there is no idea if and where gem_init_hw() and intel_guc_ucode_init() are documented. Indeed, documentation for the former does not exist, but there's no broken link. BR, Jani. [1] http://static.lwn.net/kerneldoc/gpu.html#c.intel_guc_ucode_load -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Em Tue, 8 Mar 2016 10:39:22 -0300 Mauro Carvalho Chehab escreveu: > Em Tue, 08 Mar 2016 05:13:13 -0700 > Dan Allen escreveu: > > > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab < > > mche...@osg.samsung.com> wrote: > > > > > pandoc did a really crap job on the conversion. To convert this > > > into something useful, we'll need to spend a lot of time, as it lost > > > most of the cross-references, as they were defined via DocBook macros. > > > > > > > I agree pandoc creates crappy AsciiDoc. We have a much better converter in > > the works called DocBookRx. > > > > https://github.com/opendevise/docbookrx > > > > It has converted several very serious DocBook documents and we're > > continuing to improve it. It's also a lot easier to hack than pandoc. > > Didn't work: > > $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. I tried to use docbookrx for the bits that were not properly converted, like the manpage-like pages: $ ../docbookrx/bin/docbookrx Documentation/DocBook/media/v4l/func-ioctl.xml No visitor defined for ! Skipping. Dan, if you want to take a look on what's going wrong here, the XML I'm trying to convert is: https://git.linuxtv.org/media_tree.git/tree/Documentation/DocBook/media/v4l/func-ioctl.xml If this would work, it should be generating something like: https://git.linuxtv.org/mchehab/asciidoc-poc.git/tree/func-ioctl.adoc Pandoc failed to fully convert it, but at least it left all the texts, with prevented rewriting it from scratch. This is the manual fix I applied to it: https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc FYI, we have 133 xml files at the media uAPI doc with refmeta. -- Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
On Tue, 08 Mar 2016, Dan Allen wrote: > That's not entirely true. First, you can pre-split at the source level > using includes and generate output for each of the masters. That's what I > tend to do and it works really well since these are logical split points. I need to look into this again. Is there a specific option or directive to produce split output for includes? When I tried this, the result was just one big output file. (And indeed we'd need both. Some includes we want embedded, some includes should produce separate outputs.) >> That actually makes choosing asciidoc harder, because >> requiring another language environment complicates, not simplifies, the >> toolchain. I'd really like to lower the bar for building the >> documentation, for everyone, so much so that it becomes part of the >> normal checks for patch inclusion. > > Pardon my bluntness here, but I don't buy that argument. This is Linux. > Installing software couldn't be simpler, and we're talking about an > extremely well supported language (Ruby). Granted, that part works for me. I'm not so sensitive to the dependencies; others may disagree. > I think it's a huge exaggeration to say that Asciidoctor is any harder to > install than AsciiDoc Python. It's also a heck of a lot smaller in size > since AsciiDoc Python pulls in hundreds of MB of LaTeX packages. For me, the comparison is really between Sphinx and Asciidoctor, not so much doc vs. doctor. The native output format and extension support in Sphinx is appealing; I am not yet convinced we could manage with Asciidoctor but without DocBook. The extension offering seems better in Sphinx. > Whatever you decide, I wish you all the best with your documentation > efforts! Thanks! BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Em Tue, 08 Mar 2016 05:13:13 -0700 Dan Allen escreveu: > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab < > mche...@osg.samsung.com> wrote: > > > pandoc did a really crap job on the conversion. To convert this > > into something useful, we'll need to spend a lot of time, as it lost > > most of the cross-references, as they were defined via DocBook macros. > > > > I agree pandoc creates crappy AsciiDoc. We have a much better converter in > the works called DocBookRx. > > https://github.com/opendevise/docbookrx > > It has converted several very serious DocBook documents and we're > continuing to improve it. It's also a lot easier to hack than pandoc. Didn't work: $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml No visitor defined for ! Skipping. No visitor defined for ! Skipping. No visitor defined for ! Skipping. No visitor defined for ! Skipping. No visitor defined for ! Skipping. > > -Dan > > -- Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
Em Tue, 08 Mar 2016 05:09:40 -0700 Dan Allen escreveu: > Jani wrote: > > > there was no support for chunked, or split > > to chapters, HTML, and the single page result was simply way too big. > > > > That's not entirely true. First, you can pre-split at the source level > using includes and generate output for each of the masters. That's what I > tend to do and it works really well since these are logical split points. The problem on pre-splitting the documents and process them in separate is that this will break cross-references. At the media uAPI Docbook, we use a lot of cross references. Btw, we use a lot of includes. Currently, it has 187 separate files. We even parse the header files looking for typedefs, structs, enums, #defines and functions, in order to produce a document that will cross-reference the documentation. > Second, there is a custom converter in the works to split post-generate > (which is really what we're talking about when we compare it to the DocBook > toolchain). > > https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb > > It's just a prototype, but proves it is possible by design. I didn't test it, but I saw some comments at the web that the part that would handle cross-references between files is not ready. > Personally, I don't like most chunked HTML approaches because they split > arbitrarily. We are trying to find the right balance so that the output is > actually sensible. There's still work to do, but there are options in the > meantime. Well, if it is capable of creating one chunk per include file, and do cross-references between chunks, this would work for media UAPI book. Yet, it would be good to have the multi-chunk extension packaged on major distros, as I don't like the idea of installing it without using my distro's package manager. -- Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
Em Tue, 08 Mar 2016 11:49:35 +0200 Jani Nikula escreveu: > On Tue, 08 Mar 2016, Dan Allen wrote: > > One of the key goals of the Asciidoctor project is to be able to directly > > produce a wide variety of outputs from the same source (without DocBook). > > We've added flexibility and best practices into the syntax and matured the > > converter mechanism to bridge this (sometimes very wide) gap. > > I think our conclusion so far was that the native AsciiDoc (and > Asciidoctor) outputs fell short of our needs, forcing us to use the > DocBook pipeline. I, for one, was hoping we could eventually simplify > the toolchain. For example, there was no support for chunked, or split > to chapters, HTML, and the single page result was simply way too big. > > > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers > > acknowledge that (including the original creator). > > Thanks for the input. We've touched the topic of AsciiDoc > vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of > AsciiDoc. That actually makes choosing asciidoc harder, because > requiring another language environment complicates, not simplifies, the > toolchain. I'd really like to lower the bar for building the > documentation, for everyone, so much so that it becomes part of the > normal checks for patch inclusion. As I failed to find a way to solve the issues with Sphinx/RST, I started a PoC using Asciidoctor for the Media uAPI docbook conversion, in order to see if this would work for us. I'm not a big fan of using a language that I don't domain, like Ruby (and the same applies to Python), but, as I said before, provided that the toolchain works, can easily be installed on Fedora and Debian, and provide the functionality I need, I'm ok with that. On my tests, Asciidoctor is really fast. It takes a fraction of the time require to build from DocBook, with is a good thing. Yet, I suspect that it doesn't have the strict checks that xmllint have. For example, I didn't see any warning about a missing cross-ref. We use those warnings to discover if something is added at the code, but were not documented. Its error reports are also crappy, as it doesn't tell where the problem is. For example: $ asciidoctor -n -b docbook media_api.adoc asciidoctor: WARNING: tables must have at least one body row On a document with 33793 lines (this is the size of the uAPI doc when converted to AsciiDoc format), the above warning doesn't help. So, I'm actually planning to use AsciiDoc/xmllint to check for documentation troubles, if possible. Even with AsciiDoc/Asciidoctor, there are several features at the media documents that aren't well supported. For example, we widely use the Docbook's tags to generate a manpage-like description: V4L2 open() &manvol; v4l2-open Open a V4L2 device #includeint open const char *device_name int flags Asciidoctor doesn't produce anything like that, if the booktype is not "manpage". Well, the media documentation is a multi-part book. So, all those tags should be manually converted (actually, pandoc made a mess with those tags - so manual work is required anyway, no matter what markdown language we use). It will be a hard work, but, at least, this is doable. Also, it doesn't accept images with PDF format. We have a few PDF images, although I we have them also in GIF. So, not a big issue here. I also did lots of table conversions, to see if our complex tables would fit. The answer is: Yes: on all cases I converted, it worked fine. There are two ways of doing that: - Nested tables - Merging cells Some tables we use won't work fine with nested tables, as they have cells merging the entire line of such tables. Also, converting some tables to nested tables would be incredible hard, as we have really big tables there that would need to add dozens of nested tables inside, like this one: https://linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html#id-1.4.4.14.6.4 Using merging cells works fine, tough, and it is not hard to use. The tables are easy to edit. The Asciidoctor syntax (not sure if AsciiDoc also accepts this) is: .MFC 5.1 Control IDs [width="100%",cols="7%,40%,13%,40%",options="header",] |=== |ID |Type 2+| Description ... The "2+|" tells that the next content should be merged into two cells. One of the good things is that we're not forced to use asciiart, with would make it really hard to handle the tables. For those wanting to see the results so far: The html is at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html The testing tree is at: https://git.linuxtv.org/mchehab/asciidoc-poc.git/ PS.: pandoc did a really crap job on the conversion. To convert this into something useful, we'll need to
Re: Kernel docs: muddying the waters a bit
On Tue, 08 Mar 2016, Dan Allen wrote: > One of the key goals of the Asciidoctor project is to be able to directly > produce a wide variety of outputs from the same source (without DocBook). > We've added flexibility and best practices into the syntax and matured the > converter mechanism to bridge this (sometimes very wide) gap. I think our conclusion so far was that the native AsciiDoc (and Asciidoctor) outputs fell short of our needs, forcing us to use the DocBook pipeline. I, for one, was hoping we could eventually simplify the toolchain. For example, there was no support for chunked, or split to chapters, HTML, and the single page result was simply way too big. > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers > acknowledge that (including the original creator). Thanks for the input. We've touched the topic of AsciiDoc vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of AsciiDoc. That actually makes choosing asciidoc harder, because requiring another language environment complicates, not simplifies, the toolchain. I'd really like to lower the bar for building the documentation, for everyone, so much so that it becomes part of the normal checks for patch inclusion. BR, Jani. [1] http://mid.gmane.org/86pow31ddj@hiro.keithp.com -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Fri, 2016-03-04 at 09:46 +0200, Jani Nikula wrote: > […] > If we're talking about the same asciidoctor (http://asciidoctor.org/) > it's written in ruby but you can apparently run it in JVM using > JRuby. Calling it Java-based is misleading. Indeed, I was somewhat imprecise. Thanks to the work mostly of Charles Nutter, JRuby is invariably a faster platform for Ruby code than Ruby is. So yes ASCIIDoctor is JVM-based via JRuby, not Java-based. The real point here is that in a move from DocBook/XML as a documentation source, ASCIIDoctor is an excellent choice. -- Russel.=Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.ukLondon SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: Kernel docs: muddying the waters a bit
Em Mon, 7 Mar 2016 00:29:08 +0100 Johannes Stezenbach escreveu: > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > > > I converted one of the big tables to CSV. At least now it recognized > > it as a table. Yet, the table was very badly formated: > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > > > This is how this table should look like: > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > > > Also, as this table has merged cells at the legend. I've no idea how > > to tell sphinx to do that on csv format. > > > > The RST files are on this git tree: > > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ > > Yeah, seems it can't do merged cells in csv. Attached patch converts it > back to grid table format and fixes the table definition. > The html output looks usable, but clearly it is no fun to > work with tables in Sphinx. Yes, the output is OK, but, as you said, working with tables in Sphinx is hard, and using asciiart for the kind of tables we have is not nice. > > Sphinx' latex writer can't handle nested tables, though. Yeah, this is a big trouble that need to be solved if you're willing to use Sphinx. Btw, it crashes when trying to generate man pages: Exception occurred: File "/usr/lib/python2.7/site-packages/docutils/writers/manpage.py", line 627, in depart_entry self._active_table.append_cell(self.body[start:]) AttributeError: 'NoneType' object has no attribute 'append_cell' The full traceback has been saved in /tmp/sphinx-err-04qRMz.log, if you want to report the issue to the developers. So, if we're willing to use sphinx, someone should either fix it to produce latex nexted table and fix it to generate manpages, or we'll need to stick with just html output. > Python's docutils rst2latex can, but that doesn't help here. > rst2pdf also supports it. At least here, rst2* scripts were unable to identify that the index.rst had links to other rst documents. In the specific case of rst2latex, I got several errors like: index.rst:21: (ERROR/3) Unknown interpreted text role "ref". > But I have doubts such a large > table would render OK in pdf without using landscape orientation. Yeah, in the past, when we had pdf enabled for DocBook (e. g. when media development was using a separate mercurial tree), I guess we had tags changing the text orientation on a few tables that would otherwise won't diplay fine, but I can't remember the dirty details anymore. > I have not tried because I used python3-sphinx but rst2pdf > is only availble for Python2 in Debian so it does not integrate > with Sphinx. > > > Johannes -- Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
Em Mon, 7 Mar 2016 09:48:26 +0100 Johannes Stezenbach escreveu: > On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote: > > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > > > > > I converted one of the big tables to CSV. At least now it recognized > > > it as a table. Yet, the table was very badly formated: > > > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > > > > > This is how this table should look like: > > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > > > > > Also, as this table has merged cells at the legend. I've no idea how > > > to tell sphinx to do that on csv format. > > > > > > The RST files are on this git tree: > > > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ > > > > Yeah, seems it can't do merged cells in csv. Attached patch converts it > > back to grid table format and fixes the table definition. > > The html output looks usable, but clearly it is no fun to > > work with tables in Sphinx. > > > > Sphinx' latex writer can't handle nested tables, though. > > Python's docutils rst2latex can, but that doesn't help here. > > rst2pdf also supports it. But I have doubts such a large > > table would render OK in pdf without using landscape orientation. > > I have not tried because I used python3-sphinx but rst2pdf > > is only availble for Python2 in Debian so it does not integrate > > with Sphinx. > > Just a quick idea: > Perhaps one alternative would be to use Graphviz to render > the problematic tables, it supports a HTML-like syntax > and can be embedded in Spinx documents: > > http://www.sphinx-doc.org/en/stable/ext/graphviz.html > http://www.graphviz.org/content/node-shapes#html > http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables That could work, but it is scary... Graphviz is great to generate diagrams, but it really sucks when one wants to put a graph element on a specific place, as it loves to reorder elements putting them on unexpected places. Btw, I converted all docs from our uAPI docbook to rst using pandoc. It was a brainless conversion, except for a few fixes. The output is at: https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/ I added it on the top of my PoC tree at: https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ Besides tables, I noticed some other bad things that needs to be corrected somehow: 1) Document divisions are not numbered. We need that. It should be broken into: - Document divisions - one per documented API: - V4L2 - Remote Controllers - DVB - Media Controller - Chapters - Sessions Everything should be numbered, as, when discussing API improvements, it is usual the need of pinpoint to an specific chapter and section. Tables and images should also be numbered, and we need a way to use references for table/image numbers. 2) Images Most images didn't popup. We have images on different file formats: - SVG - GIF - PDF - PNG 3) References It could be a conversion issue, but there are lots of missing references at the documentation. 4) We need to have some way to tell sphinx to not put some things at the lateral ToC bar. For example, at the V4L2 "Changes" section, we don't want to have one entry per version at the ToC bar. Giving that, I suspect that we'll have huge headaches to address if we use sphinx, as it seems too limited to handle complex documents. We should try to use some other tool that would give us better results. Regards, Mauro
Re: Kernel docs: muddying the waters a bit
On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote: > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > > > I converted one of the big tables to CSV. At least now it recognized > > it as a table. Yet, the table was very badly formated: > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > > > This is how this table should look like: > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > > > Also, as this table has merged cells at the legend. I've no idea how > > to tell sphinx to do that on csv format. > > > > The RST files are on this git tree: > > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ > > Yeah, seems it can't do merged cells in csv. Attached patch converts it > back to grid table format and fixes the table definition. > The html output looks usable, but clearly it is no fun to > work with tables in Sphinx. > > Sphinx' latex writer can't handle nested tables, though. > Python's docutils rst2latex can, but that doesn't help here. > rst2pdf also supports it. But I have doubts such a large > table would render OK in pdf without using landscape orientation. > I have not tried because I used python3-sphinx but rst2pdf > is only availble for Python2 in Debian so it does not integrate > with Sphinx. Just a quick idea: Perhaps one alternative would be to use Graphviz to render the problematic tables, it supports a HTML-like syntax and can be embedded in Spinx documents: http://www.sphinx-doc.org/en/stable/ext/graphviz.html http://www.graphviz.org/content/node-shapes#html http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables Johannes
Re: Kernel docs: muddying the waters a bit
On Thu, 03 Mar 2016 16:03:14 +0200 Jani Nikula wrote: > This stalled a bit, but the waters are still muddy... So I've been messing with this a bit; wanted to do a proper patch posting but I'm fried and mostly out of time for the moment. The results I'm getting now can be seen at: http://static.lwn.net/kerneldoc/ I've pulled in a few templates (including gpu.tmpl), converted them, and built them into some reasonable-looking HTML, modulo a fair number of glitches. There's lots of details to deal with, but the broad shape of it is there. If you look, you'll see that things like cross-file cross-references, a feature we've never had before, work nicely. I can also get good EPUB and PDF output - except that rst2pdf is currently crashing on me, which is a little discouraging. Man page output will take more work. What I have so far can be pulled from: git://git.lwn.net/linux.git doc/sphinx It's still based on using docproc because that was easiest (for me). The kernel-doc part is Jani's asciidoc stuff, hacked up and made uglier. I'm not sure that any of it is worth more than a demonstration of what can be done; I'm not particularly proud of (or tied to) any of it. But it's a start. I've not looked at the table situation at all; soon. jon
Re: Kernel docs: muddying the waters a bit
On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > I converted one of the big tables to CSV. At least now it recognized > it as a table. Yet, the table was very badly formated: > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > This is how this table should look like: > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > Also, as this table has merged cells at the legend. I've no idea how > to tell sphinx to do that on csv format. > > The RST files are on this git tree: > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ Yeah, seems it can't do merged cells in csv. Attached patch converts it back to grid table format and fixes the table definition. The html output looks usable, but clearly it is no fun to work with tables in Sphinx. Sphinx' latex writer can't handle nested tables, though. Python's docutils rst2latex can, but that doesn't help here. rst2pdf also supports it. But I have doubts such a large table would render OK in pdf without using landscape orientation. I have not tried because I used python3-sphinx but rst2pdf is only availble for Python2 in Debian so it does not integrate with Sphinx. Johannes >From 61674b398e778bd5ff644ffd493d5ff1cfaca0ef Mon Sep 17 00:00:00 2001 From: Johannes Stezenbach Date: Sun, 6 Mar 2016 23:55:19 +0100 Subject: [PATCH] some progress for html output --- _static/borderless.css | 3 -- _static/v4l2tables.css | 9 + _templates/layout.html | 9 + packed-rgb.rst | 88 +- pixfmt-yuyv.rst| 2 +- v4l-table-within-table.rst | 72 +++-- 6 files changed, 105 insertions(+), 78 deletions(-) delete mode 100644 _static/borderless.css create mode 100644 _static/v4l2tables.css diff --git a/_static/borderless.css b/_static/borderless.css deleted file mode 100644 index bfd4b01..000 --- a/_static/borderless.css +++ /dev/null @@ -1,3 +0,0 @@ -table#table-borderless { -border: 1px solid black; -} diff --git a/_static/v4l2tables.css b/_static/v4l2tables.css new file mode 100644 index 000..c045e45 --- /dev/null +++ b/_static/v4l2tables.css @@ -0,0 +1,9 @@ +table.noborder { +border: 1px solid black; +background: white; +white-space: nowrap; +} + +table.noborder td, table.noborder th { +padding: 0px; +} diff --git a/_templates/layout.html b/_templates/layout.html index b6bf12b..637332d 100644 --- a/_templates/layout.html +++ b/_templates/layout.html @@ -1,9 +1,2 @@ {% extends "!layout.html" %} -{% block tables %} - -table#table-borderless { -border: 1px solid black; -} - -{{ super() }} -{% endblock %} +{% set css_files = css_files + ["_static/v4l2tables.css"] %} diff --git a/packed-rgb.rst b/packed-rgb.rst index 352b91c..b4fcf3e 100644 --- a/packed-rgb.rst +++ b/packed-rgb.rst @@ -9,25 +9,46 @@ graphics frame buffers. They occupy 8, 16, 24 or 32 bits per pixel. These are all packed-pixel formats, meaning all the data for a pixel lie next to each other in memory. -.. csv-table:: Table: Packed RGB Image Formats - :header: Identifier,Code, ,Byte 0 in memory,Byte 1,Byte 2,Byte 3 +.. table:: Packed RGB Image Formats + :class: noborder - ``V4L2_PIX_FMT_RGB332``,'RGB1',,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`1`,b\ :sub:`0` - ``V4L2_PIX_FMT_ARGB444``,'AR12',,g\ :sub:`3`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,a\ :sub:`3`,a\ :sub:`2`,a\ :sub:`1`,a\ :sub:`0`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0` - ``V4L2_PIX_FMT_XRGB444``,'XR12',,g\ :sub:`3`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,-,-,-,-,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0` - ``V4L2_PIX_FMT_ARGB555``,'AR15',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,a,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3` - ``V4L2_PIX_FMT_XRGB555``,'XR15',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,-,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3` - ``V4L2_PIX_FMT_RGB565``,'RGBP',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`5`,g\ :sub:`4`,g\ :sub:`3` - ``V4L2_PIX_FMT_ARGB555X``,'AR15' | (1<<31),,a,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`,,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0` - ``V4L2_PIX_FMT_XRGB555X``,'XR15' | (1<<31),,-,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`,,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0` - ``V4L2_PIX_FMT_RGB565X``,'RGBR',,r\ :sub:`4`,r\
Re: Kernel docs: muddying the waters a bit
Em Fri, 04 Mar 2016 15:09:09 +0100 Johannes Stezenbach escreveu: > On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote: > > > > 3) I tried to use a .. cssclass, as Johannes suggested, but > > I was not able to include the CSS file. I suspect that this is > > easy to fix, but I want to see if the cssclass will also work for > > the pdf output as well. > > "cssclass" was (I think) a custom role defined in the example, > unless you also have defined a custom role you can use plain "class". > I have not looked deeper into the theming and template stuff. Well, it accepted cssclass for html (well, it didn't find the templates - so I guess it is just me failing to understand how to tell sphinx to get the stylesheet), but it rejects it for latexPDF. > > > 4) It seems that it can't produce nested tables in pdf: > > > > Markup is unsupported in LaTeX: > > v4l-table-within-table:: nested tables are not yet implemented. > > Makefile:115: recipe for target 'latexpdf' failed > > This: > http://www.sphinx-doc.org/en/stable/markup/misc.html#tables > > suggests you need to add the tabularcolumns directive > for complex tables. > > BTW, as an alternative to the ASCII-art input > there is also support for CSV and list tables: > http://docutils.sourceforge.net/docs/ref/rst/directives.html#table I converted one of the big tables to CSV. At least now it recognized it as a table. Yet, the table was very badly formated: https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html This is how this table should look like: https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html Also, as this table has merged cells at the legend. I've no idea how to tell sphinx to do that on csv format. The RST files are on this git tree: https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ Regards, Mauro
Re: Kernel docs: muddying the waters a bit
On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote: > > 3) I tried to use a .. cssclass, as Johannes suggested, but > I was not able to include the CSS file. I suspect that this is > easy to fix, but I want to see if the cssclass will also work for > the pdf output as well. "cssclass" was (I think) a custom role defined in the example, unless you also have defined a custom role you can use plain "class". I have not looked deeper into the theming and template stuff. > 4) It seems that it can't produce nested tables in pdf: > > Markup is unsupported in LaTeX: > v4l-table-within-table:: nested tables are not yet implemented. > Makefile:115: recipe for target 'latexpdf' failed This: http://www.sphinx-doc.org/en/stable/markup/misc.html#tables suggests you need to add the tabularcolumns directive for complex tables. BTW, as an alternative to the ASCII-art input there is also support for CSV and list tables: http://docutils.sourceforge.net/docs/ref/rst/directives.html#table Johannes
Re: Kernel docs: muddying the waters a bit
Em Fri, 04 Mar 2016 10:29:08 +0200 Jani Nikula escreveu: > On Fri, 04 Mar 2016, Mauro Carvalho Chehab wrote: > > Em Thu, 03 Mar 2016 15:23:23 -0800 > > Keith Packard escreveu: > > > >> Mauro Carvalho Chehab writes: > >> > >> > On my tests, Sphinix seemed too limited to format tables. Asciidoc > >> > produced an output that worked better. > >> > >> Yes, asciidoc has much more flexibility in table formatting, including > >> the ability to control text layout within cells and full control over > >> borders. > >> > >> However, I think asciidoc has two serious problems: > >> > >> 1) the python version (asciidoc) appears to have been abandoned in > >> favor of the ruby version. > >> > >> 2) It really is just a docbook pre-processor. Native html/latex output > >> is poorly supported at best, and exposes only a small subset of the > >> full capabilities of the input language. > >> > >> As such, we would have to commit to using the ruby version and either > >> committing to fixing the native html output backend or continuing to use > >> the rest of the docbook toolchain. > >> > >> We could insist on using the python version, of course. I spent a bit of > >> time hacking that up to add 'real' support for a table-of-contents in > >> the native HTML backend and it looks like getting those changes > >> upstreamed would be reasonably straightforward. However, we'd end up > >> 'owning' the code, and I'm not sure we want to. > > > > I'm a way more concerned about using a tool that fulfill our needs > > than to look for something that won't use the docbook toolchain or > > require to install ruby. > > I think you meant that to be the other way round, or I fail at parsing > you. ;) I mean: I'm a way more concerned about using a tool that fulfill our needs than on toolchain it uses. > > > In the case of Docbook, we know it works and we know already its > > issues. Please correct me if I'm wrong, but the big problem we > > have is not due to the DocBook toolchain, but due to the lack of > > features at the kernel-doc script. Also, xmlto is already installed > > by the ones that build the kernel docs. So, keeping use it won't > > require to install a weird toolchain by hand. > > I think kernel-doc is just a small part of the puzzle. It's a problem, > but a small one at that, and we've already made it output asciidoc, rst > and docbook as part of this exercise. For real, as in code, not as in > talk. > > The reasons I'm involved in this is that I want to make writing > documentation and rich kernel-doc comments easier (using lightweight > markup) and I want to make building the documentation easier (using a > straightforward toolchain with not too many dependencies). I'm hoping > the former makes writing documentation more attractive and the latter > keeps the documentation and the toolchain in a better shape through > having more people actually build the documentation. I don't think the toolchain is a problem. We don't attract too many people because developers don't like writing documentation. Ok, using a markup language can be easier than writing DocBook tags directly, but people usually don't even add C comments on the code they submit. > IMHO docbook is problematic because the toolchain gets too long and > fragile. You need plenty of tools installed to build the documentation, > it's fussy to get working, and people just won't. Like code, > documentation bitrots too when it's not used. On most distros, a single command installs all that it is needed. > The documentation build is broken too often. This is indeed a problem, but the way I solved this at the media subsystem is that I rebuild the documentation every time a media file either at Documentation/Docbook or included at device-drivers.tmpl is touched. If the script produces error, I nack the patch. So, IMHO, this is not a toolchain fault, but the lack of a process. The kernel build robot is now producing e-mails when the documentation has new troubles, so I guess this will help a lot to avoid adding new documentation breakages. > Debugging formatting issues through the entire > pipeline gets hard; I already faced some of this when playing with the > kernel-doc->asciidoc->docbook->html chain. > > In short, I don't think the docbook toolchain fills all of our needs > either. > > > So, to be frank, it doesn't scary me to use either pyhton or > > ruby script + docbook. > > > > Of course, having to own the code has a cost that should be evaluated. > > > > If, on the other hand, we decide to use RST, we'll very likely need to > > patch it to fulfill our needs in order to add proper table support. > > I've no idea how easy/difficult would be to do that, nor if Sphinx > > upstream would accept such changes. > > > > So, at the end of the day, we may end by having to carry on our own > > version of Sphinx inside our tree, with doesn't sound good, specially > > since it is not just a script, but a package with hu
Re: Kernel docs: muddying the waters a bit
On Fri, Mar 04, 2016 at 10:29:08AM +0200, Jani Nikula wrote: > On Fri, 04 Mar 2016, Mauro Carvalho Chehab wrote: > > > > If, on the other hand, we decide to use RST, we'll very likely need to > > patch it to fulfill our needs in order to add proper table support. > > I've no idea how easy/difficult would be to do that, nor if Sphinx > > upstream would accept such changes. > > > > So, at the end of the day, we may end by having to carry on our own > > version of Sphinx inside our tree, with doesn't sound good, specially > > since it is not just a script, but a package with hundreds of > > files. > > If we end up having to modify Sphinx, it has a powerful extension > mechanism for this. We wouldn't have to worry about getting it merged to > Sphinx upstream, and we wouldn't have to carry a local version of all of > Sphinx. (In fact, the extension mechanism provides a future path for > doing kernel-doc within Sphinx instead of as a preprocessing step.) > > I know none of this alleviates your concerns with table supports right > now. I'll try to have a look at that a bit more. FWIW, I think table formatting in Sphinx works via style sheets. The mechanism is documented in the Python docutils docs that Sphinx is built upon. Basically you use the "class" or "role" directive and define the corresponding CSS or LaTeX (or rst2pdf) style. Here is one example (using a custom "cssclass" role): https://pythonhosted.org/sphinxjp.themes.basicstrap/sample.html Directives (especially role and class): http://www.sphinx-doc.org/en/stable/rest.html#directives LaTeX styling: http://docutils.readthedocs.org/en/sphinx-docs/user/latex.html#custom-interpreted-text-roles HTH, Johannes
Re: Kernel docs: muddying the waters a bit
On Fri, 04 Mar 2016, Mauro Carvalho Chehab wrote: > Em Thu, 03 Mar 2016 15:23:23 -0800 > Keith Packard escreveu: > >> Mauro Carvalho Chehab writes: >> >> > On my tests, Sphinix seemed too limited to format tables. Asciidoc >> > produced an output that worked better. >> >> Yes, asciidoc has much more flexibility in table formatting, including >> the ability to control text layout within cells and full control over >> borders. >> >> However, I think asciidoc has two serious problems: >> >> 1) the python version (asciidoc) appears to have been abandoned in >> favor of the ruby version. >> >> 2) It really is just a docbook pre-processor. Native html/latex output >> is poorly supported at best, and exposes only a small subset of the >> full capabilities of the input language. >> >> As such, we would have to commit to using the ruby version and either >> committing to fixing the native html output backend or continuing to use >> the rest of the docbook toolchain. >> >> We could insist on using the python version, of course. I spent a bit of >> time hacking that up to add 'real' support for a table-of-contents in >> the native HTML backend and it looks like getting those changes >> upstreamed would be reasonably straightforward. However, we'd end up >> 'owning' the code, and I'm not sure we want to. > > I'm a way more concerned about using a tool that fulfill our needs > than to look for something that won't use the docbook toolchain or > require to install ruby. I think you meant that to be the other way round, or I fail at parsing you. ;) > In the case of Docbook, we know it works and we know already its > issues. Please correct me if I'm wrong, but the big problem we > have is not due to the DocBook toolchain, but due to the lack of > features at the kernel-doc script. Also, xmlto is already installed > by the ones that build the kernel docs. So, keeping use it won't > require to install a weird toolchain by hand. I think kernel-doc is just a small part of the puzzle. It's a problem, but a small one at that, and we've already made it output asciidoc, rst and docbook as part of this exercise. For real, as in code, not as in talk. The reasons I'm involved in this is that I want to make writing documentation and rich kernel-doc comments easier (using lightweight markup) and I want to make building the documentation easier (using a straightforward toolchain with not too many dependencies). I'm hoping the former makes writing documentation more attractive and the latter keeps the documentation and the toolchain in a better shape through having more people actually build the documentation. IMHO docbook is problematic because the toolchain gets too long and fragile. You need plenty of tools installed to build the documentation, it's fussy to get working, and people just won't. Like code, documentation bitrots too when it's not used. The documentation build is broken too often. Debugging formatting issues through the entire pipeline gets hard; I already faced some of this when playing with the kernel-doc->asciidoc->docbook->html chain. In short, I don't think the docbook toolchain fills all of our needs either. > So, to be frank, it doesn't scary me to use either pyhton or > ruby script + docbook. > > Of course, having to own the code has a cost that should be evaluated. > > If, on the other hand, we decide to use RST, we'll very likely need to > patch it to fulfill our needs in order to add proper table support. > I've no idea how easy/difficult would be to do that, nor if Sphinx > upstream would accept such changes. > > So, at the end of the day, we may end by having to carry on our own > version of Sphinx inside our tree, with doesn't sound good, specially > since it is not just a script, but a package with hundreds of > files. If we end up having to modify Sphinx, it has a powerful extension mechanism for this. We wouldn't have to worry about getting it merged to Sphinx upstream, and we wouldn't have to carry a local version of all of Sphinx. (In fact, the extension mechanism provides a future path for doing kernel-doc within Sphinx instead of as a preprocessing step.) I know none of this alleviates your concerns with table supports right now. I'll try to have a look at that a bit more. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Fri, 04 Mar 2016, Russel Winder wrote: > On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote: >> 1) the python version (asciidoc) appears to have been abandoned in >> favor of the ruby version. > > This is I think true, however the Java-based tool chain Asciidoctor is > I believe the standard bearer for ASCIIdoc these days, albeit called > ASCIIdoctor. If we're talking about the same asciidoctor (http://asciidoctor.org/) it's written in ruby but you can apparently run it in JVM using JRuby. Calling it Java-based is misleading. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote: > […] > However, I think asciidoc has two serious problems: > > 1) the python version (asciidoc) appears to have been abandoned in > favor of the ruby version. This is I think true, however the Java-based tool chain Asciidoctor is I believe the standard bearer for ASCIIdoc these days, albeit called ASCIIdoctor. > 2) It really is just a docbook pre-processor. Native html/latex > output > is poorly supported at best, and exposes only a small subset of > the > full capabilities of the input language. This is not true. Yes ASCIIDoc started as a DocBook/XML frontend so as to use a sane :-) markup language rather than XML (XML is a notation for consenting computers only), but the current ASCIIDoctor toolchain deals very well in direct HTML and PDF generation, without needing a DocBook/XML toolchain. > As such, we would have to commit to using the ruby version and either > committing to fixing the native html output backend or continuing to > use > the rest of the docbook toolchain. Or trial the JVM-based ASCIIdoctor which is what the projects I am involved with chose to use. Perhaps as an example I can give you http:/ /gpars.website (it's a redirector) all the HTML and PDF is generated from ASCIIDoc source using ASCIIDoctor driven with a Gradle build system. This is still very much a work in progress (by Jim Northrop, not me currently), but I like it. > We could insist on using the python version, of course. I spent a bit > of > time hacking that up to add 'real' support for a table-of-contents in > the native HTML backend and it looks like getting those changes > upstreamed would be reasonably straightforward. However, we'd end up > 'owning' the code, and I'm not sure we want to. If the Python version is really not being maintained, I would suggest that unless you want to take over the project and be it's maintainer, you would be better advised to use a different version. -- Russel.=Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.ukLondon SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: Kernel docs: muddying the waters a bit
Em Thu, 03 Mar 2016 15:23:23 -0800 Keith Packard escreveu: > Mauro Carvalho Chehab writes: > > > On my tests, Sphinix seemed too limited to format tables. Asciidoc > > produced an output that worked better. > > Yes, asciidoc has much more flexibility in table formatting, including > the ability to control text layout within cells and full control over > borders. > > However, I think asciidoc has two serious problems: > > 1) the python version (asciidoc) appears to have been abandoned in > favor of the ruby version. > > 2) It really is just a docbook pre-processor. Native html/latex output > is poorly supported at best, and exposes only a small subset of the > full capabilities of the input language. > > As such, we would have to commit to using the ruby version and either > committing to fixing the native html output backend or continuing to use > the rest of the docbook toolchain. > > We could insist on using the python version, of course. I spent a bit of > time hacking that up to add 'real' support for a table-of-contents in > the native HTML backend and it looks like getting those changes > upstreamed would be reasonably straightforward. However, we'd end up > 'owning' the code, and I'm not sure we want to. I'm a way more concerned about using a tool that fulfill our needs than to look for something that won't use the docbook toolchain or require to install ruby. In the case of Docbook, we know it works and we know already its issues. Please correct me if I'm wrong, but the big problem we have is not due to the DocBook toolchain, but due to the lack of features at the kernel-doc script. Also, xmlto is already installed by the ones that build the kernel docs. So, keeping use it won't require to install a weird toolchain by hand. So, to be frank, it doesn't scary me to use either pyhton or ruby script + docbook. Of course, having to own the code has a cost that should be evaluated. If, on the other hand, we decide to use RST, we'll very likely need to patch it to fulfill our needs in order to add proper table support. I've no idea how easy/difficult would be to do that, nor if Sphinx upstream would accept such changes. So, at the end of the day, we may end by having to carry on our own version of Sphinx inside our tree, with doesn't sound good, specially since it is not just a script, but a package with hundreds of files. Thanks, Mauro pgpItSf1ILqWF.pgp Description: Assinatura digital OpenPGP
Re: Kernel docs: muddying the waters a bit
Mauro Carvalho Chehab writes: > On my tests, Sphinix seemed too limited to format tables. Asciidoc > produced an output that worked better. Yes, asciidoc has much more flexibility in table formatting, including the ability to control text layout within cells and full control over borders. However, I think asciidoc has two serious problems: 1) the python version (asciidoc) appears to have been abandoned in favor of the ruby version. 2) It really is just a docbook pre-processor. Native html/latex output is poorly supported at best, and exposes only a small subset of the full capabilities of the input language. As such, we would have to commit to using the ruby version and either committing to fixing the native html output backend or continuing to use the rest of the docbook toolchain. We could insist on using the python version, of course. I spent a bit of time hacking that up to add 'real' support for a table-of-contents in the native HTML backend and it looks like getting those changes upstreamed would be reasonably straightforward. However, we'd end up 'owning' the code, and I'm not sure we want to. -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
Em Thu, 03 Mar 2016 07:13:05 -0700 Jonathan Corbet escreveu: > On Thu, 03 Mar 2016 16:03:14 +0200 > Jani Nikula wrote: > > > This stalled a bit, but the waters are still muddy... > > I've been dealing with real-world obnoxiousness, something which won't > come to an immediate end, unfortunately. But I have been taking some time > to mess with things, and hope to have some more POC patches to send out > soon. > > > Is the Sphinx/reStructuredText table support adequate for media/v4l > > documentation? > > That's perhaps the biggest question. My sense is "yes", but this needs a > bit more assurance than that. On my tests, Sphinix seemed too limited to format tables. Asciidoc produced an output that worked better. Please notice that we tried to convert only one type of table. The result with RST was not beautiful, but worked. However, we use tables also to show how bits appear at the video formats, like the tables at: https://linuxtv.org/downloads/v4l-dvb-apis/subdev.html#v4l2-mbus-format For those tables to look nice, we should be able to remove borders and grids from the table. I was unable to find a way to control the tables format with RST to do things like grid/border removal. > > Are the Sphinx output formats adequate in general? Specifically, is the > > lack of DocBook support, and the flexibility it provides, a blocker? > > DocBook is a means to an end; nobody really wants DocBook itself as far > as I can tell. I've been messing with rst2pdf a bit; it's not hard to get > reasonable output, and, with some effort, we could probably get really > nice output. HTML and EPUB are easily covered, still haven't really played > around with man pages yet. And there's LaTeX if we really need it. I > kind of think we're covered there, unless I've missed something? > > > Otherwise, I think Sphinx is promising. > > > > Jon, I think we need a roll of dice, err, a well-thought-out decision > > from the maintainer to go with one or the other, so we can make some > > real progress. > > My inclination at the moment is very much in the Sphinx direction. I had > some vague thoughts of pushing a throwaway experimental directory with a > couple of docs for 4.6 that would just let people play with it easily; > then we'd see how many screams we get. We'll see if the world lets me get > there. I'm not against having a staging/Documentation for us to play with, provided, of course, that whatever tool chosen would allow converting what we have today. Regards, Mauro
Re: Kernel docs: muddying the waters a bit
On Thu, Mar 3, 2016 at 4:17 PM, Jonathan Corbet wrote: > I assume you're referring to gtk-doc? It's web page > (http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to > setup and use"; they recommend looking at Doxygen instead. So I guess I'm > not really sure what it offers that merits throwing another option into > the mix now? What am I missing? We use gtk-doc for the i915 testcase and tooling repo in userspace (intel-gpu-tools). The setup is somewhat arcane (some build-fu that is fumbly, and xml files to tie everything together). But it looks pretty and works well otherwise. It should be at https://01.org/linuxgraphics/gfx-docs/igt/ but our autobuilder seems to be screwed up right now. Of course I considered it as an option, but like doxygen it has it's own strong opinion about how in-code comments should look like, and those differ from kerneldoc syntax. Beyond that I don't really see benefits over any of the solutions proposed here already (either sphinx or rst or horror! even the hackfest I still carry around in drm-intel.git branches). Btw for igt we went with gtkdoc over docygen because a few people on our team had "doxygen only over my corpse" level kind of strong opinions. Everyone just loves their own color choice for this bikeshed ;-) -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Thu, 3 Mar 2016 14:34:25 + One Thousand Gnomes wrote: > We only have docbook because it was the tool of choice rather a lot of > years ago to then get useful output formats. It was just inherited when > borrowed the original scripts from Gnome/Gtk. It's still the most > effective way IMHO of building big structured documents out of the kernel. ...except that we haven't used it that way. Instead, we make a whole bunch of smaller, partially structured document silos. > The Gtk people long ago rewrote the original document script into a real > tool so they have some different and maintained tools that are close to > equivalent and already have some markdown support. Before we go off and > re-invent the wheel it might be worth just borrowing their wheel and > tweaking it as needed ? In particular they can generate help indexes so > that the entire output becomes nicely browsable with an HTML based help > browser. Well, not inventing the wheel was kind of the motivation behind much of this effort; I got kind of worried watching us trying to cobble more functionality into our existing house-of-cards documentation system. Sphinx is a well-established, heavily used, and well supported system; using it would not be an exercise in wheel reinvention. As far as I can tell, it does everything we need (with some open questions about table support), lets us drop the whole DocBook toolchain dependency, and move to a much better-supported setup than we have now. Plus we get much nicer output, index generation, cross-references between documents, and the ability to write documents in a lightweight markup language. Seems like a win. I assume you're referring to gtk-doc? It's web page (http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to setup and use"; they recommend looking at Doxygen instead. So I guess I'm not really sure what it offers that merits throwing another option into the mix now? What am I missing? Thanks, jon
Re: Kernel docs: muddying the waters a bit
> DocBook is a means to an end; nobody really wants DocBook itself as far > as I can tell. We only have docbook because it was the tool of choice rather a lot of years ago to then get useful output formats. It was just inherited when borrowed the original scripts from Gnome/Gtk. It's still the most effective way IMHO of building big structured documents out of the kernel. The Gtk people long ago rewrote the original document script into a real tool so they have some different and maintained tools that are close to equivalent and already have some markdown support. Before we go off and re-invent the wheel it might be worth just borrowing their wheel and tweaking it as needed ? In particular they can generate help indexes so that the entire output becomes nicely browsable with an HTML based help browser. Alan
Re: Kernel docs: muddying the waters a bit
On Thu, 03 Mar 2016 16:03:14 +0200 Jani Nikula wrote: > This stalled a bit, but the waters are still muddy... I've been dealing with real-world obnoxiousness, something which won't come to an immediate end, unfortunately. But I have been taking some time to mess with things, and hope to have some more POC patches to send out soon. > Is the Sphinx/reStructuredText table support adequate for media/v4l > documentation? That's perhaps the biggest question. My sense is "yes", but this needs a bit more assurance than that. > Are the Sphinx output formats adequate in general? Specifically, is the > lack of DocBook support, and the flexibility it provides, a blocker? DocBook is a means to an end; nobody really wants DocBook itself as far as I can tell. I've been messing with rst2pdf a bit; it's not hard to get reasonable output, and, with some effort, we could probably get really nice output. HTML and EPUB are easily covered, still haven't really played around with man pages yet. And there's LaTeX if we really need it. I kind of think we're covered there, unless I've missed something? > Otherwise, I think Sphinx is promising. > > Jon, I think we need a roll of dice, err, a well-thought-out decision > from the maintainer to go with one or the other, so we can make some > real progress. My inclination at the moment is very much in the Sphinx direction. I had some vague thoughts of pushing a throwaway experimental directory with a couple of docs for 4.6 that would just let people play with it easily; then we'd see how many screams we get. We'll see if the world lets me get there. Thanks, jon
Re: Kernel docs: muddying the waters a bit
On Sat, 13 Feb 2016, Jonathan Corbet wrote: > So can we discuss? I'm not saying we have to use Sphinx, but, should we > choose not to, we should do so with open eyes and good reasons for the > course we do take. What do you all think? This stalled a bit, but the waters are still muddy... Is the Sphinx/reStructuredText table support adequate for media/v4l documentation? Are the Sphinx output formats adequate in general? Specifically, is the lack of DocBook support, and the flexibility it provides, a blocker? Otherwise, I think Sphinx is promising. Jon, I think we need a roll of dice, err, a well-thought-out decision from the maintainer to go with one or the other, so we can make some real progress. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Thu, Feb 18, 2016 at 2:01 PM, Jonathan Corbet wrote: > On Thu, 18 Feb 2016 10:24:04 +0100 > Daniel Vetter wrote: > >> > Worth noting is that, AFAICT, in all of the proposals, including the >> > original where kernel-doc produces docbook, this autoreferencing only >> > works within parts processed by kernel-doc. Not in the template >> > documents themselves. (You can still use the markup's more verbose cross >> > referencing keywords.) >> >> It works everywhere, even in the docbook template, as long as you mark >> it up correctly. Which in docbook means func. > > In Sphinx that's :func:`function_name`. Not a thing of beauty, but it > works. Compared to function_name already an improvement. I can very much live with this ;-) -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Thu, 18 Feb 2016 10:44:34 -0200 Mauro Carvalho Chehab wrote: > > > It is workable, but I guess nested tables produced a better > > > result. > > > > > > I did myself a test with nested tables with asciidoc too: > > > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii > > > > > > With looks very decent to me. > > > > It does, except for the vertical alignment of the third column (at > > least when viewed with google chrome). > > Not sure what you mean. Here, it looks fine on both Firefox and > Chrome, except that the second colum size could be smaller. If this > is what you're meaning this can be fixed by changing the second line > from: I think Hans' problem (I see it as well) is coming from css-style of "paragraph" which is: margin-top: 0.5em; margin-bottom: 0.5em; This makes the third column non-vertical-aligned -- Patrick
Re: Kernel docs: muddying the waters a bit
On Thu, 18 Feb 2016 10:24:04 +0100 Daniel Vetter wrote: > > Worth noting is that, AFAICT, in all of the proposals, including the > > original where kernel-doc produces docbook, this autoreferencing only > > works within parts processed by kernel-doc. Not in the template > > documents themselves. (You can still use the markup's more verbose cross > > referencing keywords.) > > It works everywhere, even in the docbook template, as long as you mark > it up correctly. Which in docbook means func. In Sphinx that's :func:`function_name`. Not a thing of beauty, but it works. jon
Re: Kernel docs: muddying the waters a bit
Em Thu, 18 Feb 2016 13:07:03 +0100 Hans Verkuil escreveu: > On 02/18/16 13:04, Mauro Carvalho Chehab wrote: > > Em Thu, 18 Feb 2016 13:23:37 +0200 > > Jani Nikula escreveu: > > > >> On Thu, 18 Feb 2016, Mauro Carvalho Chehab > >> wrote: > >>> For simple documents like the one produced by kernel-doc, I guess > >>> all markup languages would work equally. > >>> > >>> The problem is for complex documents like the media kAPI one, where > >>> the document was written to produce a book. So, it uses some complex > >>> features found at DocBook. One of such features we use extensively > >>> is the capability of having a table with per-line columns. This way, > >>> we can produce things like: > >>> > >>> V4L2_CID_COLOR_KILLER boolean Enable the color killer (i. e. force a > >>> black & white image in case of a weak video signal). > >>> V4L2_CID_COLORFX enumSelects a color effect. The following values > >>> are defined: > >>> V4L2_COLORFX_NONE Color effect is > >>> disabled. > >>> V4L2_COLORFX_ANTIQUEAn aging (old > >>> photo) effect. > >>> V4L2_COLORFX_ART_FREEZE Frost color > >>> effect. > >>> > >>> In the above example, we have a main 3 columns table, and we embed > >>> a 2 columns table at the third field of V4L2_CID_COLORFX to represent > >>> possible values for this menu control. > >>> > >>> See https://linuxtv.org/downloads/v4l-dvb-apis/control.html for the > >>> complete output of it. > >>> > >>> This is used extensively inside the media DocBook, and properly > >>> supporting it is one of our major concerns. > >>> > >>> Are there any way to represent those things with the markup > >>> languages currently being analyzed? > >>> > >>> Converting those tables will likely require manual work, as I don't > >>> think automatic tools will properly handle it, specially since we > >>> use some DocBook macros to help creating such tables. > >> > >> Since I've let myself be told that asciidoc handles tables better than > >> reStructuredText, I tested this a bit with the presumably inferior one. > >> > >> rst has two table types, simple tables and grid tables [1]. It seems > >> like grid tables can do pretty much anything, but they can be cumbersome > >> to work with. So I tried to check what can be done with simple tables. > >> > >> Here's a sample, converted using rst2html (Sphinx will be prettier, but > >> rst2html works for simple things like this): > >> > >> https://people.freedesktop.org/~jani/v4l-table-within-table.rst > >> https://people.freedesktop.org/~jani/v4l-table-within-table.html > > > > Yes, this would work. Can we remove the border from the main table? > > I guess it would be nicer. > > > >> > >> Rather than using nested tables, you might want to consider using > >> definition lists within tables: > >> > >> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst > >> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html > >> > >> You be the judge, but I think this is workable. > > > > It is workable, but I guess nested tables produced a better result. > > > > I did myself a test with nested tables with asciidoc too: > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii > > > > With looks very decent to me. > > It does, except for the vertical alignment of the third column (at least when > viewed > with google chrome). Not sure what you mean. Here, it looks fine on both Firefox and Chrome, except that the second colum size could be smaller. If this is what you're meaning this can be fixed by changing the second line from: [width="100%",cols="2,1,10a",options="header",frame="none", grid="none"] to: [width="100%",cols="3,1,30a",options="header",frame="none", grid="none"] With regards to ReStructured Text, I've no idea how to control the format of a table in order to do things like hiding the borders and changing the column spacing. So, at least on a first glance, asciidoc seems to fit better. Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
On 02/18/16 13:04, Mauro Carvalho Chehab wrote: > Em Thu, 18 Feb 2016 13:23:37 +0200 > Jani Nikula escreveu: > >> On Thu, 18 Feb 2016, Mauro Carvalho Chehab wrote: >>> For simple documents like the one produced by kernel-doc, I guess >>> all markup languages would work equally. >>> >>> The problem is for complex documents like the media kAPI one, where >>> the document was written to produce a book. So, it uses some complex >>> features found at DocBook. One of such features we use extensively >>> is the capability of having a table with per-line columns. This way, >>> we can produce things like: >>> >>> V4L2_CID_COLOR_KILLER boolean Enable the color killer (i. e. force a >>> black & white image in case of a weak video signal). >>> V4L2_CID_COLORFXenumSelects a color effect. The following values >>> are defined: >>> V4L2_COLORFX_NONE Color effect is >>> disabled. >>> V4L2_COLORFX_ANTIQUEAn aging (old >>> photo) effect. >>> V4L2_COLORFX_ART_FREEZE Frost color >>> effect. >>> >>> In the above example, we have a main 3 columns table, and we embed >>> a 2 columns table at the third field of V4L2_CID_COLORFX to represent >>> possible values for this menu control. >>> >>> See https://linuxtv.org/downloads/v4l-dvb-apis/control.html for the >>> complete output of it. >>> >>> This is used extensively inside the media DocBook, and properly >>> supporting it is one of our major concerns. >>> >>> Are there any way to represent those things with the markup >>> languages currently being analyzed? >>> >>> Converting those tables will likely require manual work, as I don't >>> think automatic tools will properly handle it, specially since we >>> use some DocBook macros to help creating such tables. >> >> Since I've let myself be told that asciidoc handles tables better than >> reStructuredText, I tested this a bit with the presumably inferior one. >> >> rst has two table types, simple tables and grid tables [1]. It seems >> like grid tables can do pretty much anything, but they can be cumbersome >> to work with. So I tried to check what can be done with simple tables. >> >> Here's a sample, converted using rst2html (Sphinx will be prettier, but >> rst2html works for simple things like this): >> >> https://people.freedesktop.org/~jani/v4l-table-within-table.rst >> https://people.freedesktop.org/~jani/v4l-table-within-table.html > > Yes, this would work. Can we remove the border from the main table? > I guess it would be nicer. > >> >> Rather than using nested tables, you might want to consider using >> definition lists within tables: >> >> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst >> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html >> >> You be the judge, but I think this is workable. > > It is workable, but I guess nested tables produced a better result. > > I did myself a test with nested tables with asciidoc too: > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii > > With looks very decent to me. It does, except for the vertical alignment of the third column (at least when viewed with google chrome). Hans > > I had to manually add the nested table, as pandoc conversion sent the > DocBook's nested table to /dev/null. > > Thanks, > Mauro >
Re: Kernel docs: muddying the waters a bit
Em Thu, 18 Feb 2016 13:23:37 +0200 Jani Nikula escreveu: > On Thu, 18 Feb 2016, Mauro Carvalho Chehab wrote: > > For simple documents like the one produced by kernel-doc, I guess > > all markup languages would work equally. > > > > The problem is for complex documents like the media kAPI one, where > > the document was written to produce a book. So, it uses some complex > > features found at DocBook. One of such features we use extensively > > is the capability of having a table with per-line columns. This way, > > we can produce things like: > > > > V4L2_CID_COLOR_KILLER boolean Enable the color killer (i. e. force a > > black & white image in case of a weak video signal). > > V4L2_CID_COLORFXenumSelects a color effect. The following values > > are defined: > > V4L2_COLORFX_NONE Color effect is > > disabled. > > V4L2_COLORFX_ANTIQUEAn aging (old > > photo) effect. > > V4L2_COLORFX_ART_FREEZE Frost color > > effect. > > > > In the above example, we have a main 3 columns table, and we embed > > a 2 columns table at the third field of V4L2_CID_COLORFX to represent > > possible values for this menu control. > > > > See https://linuxtv.org/downloads/v4l-dvb-apis/control.html for the > > complete output of it. > > > > This is used extensively inside the media DocBook, and properly > > supporting it is one of our major concerns. > > > > Are there any way to represent those things with the markup > > languages currently being analyzed? > > > > Converting those tables will likely require manual work, as I don't > > think automatic tools will properly handle it, specially since we > > use some DocBook macros to help creating such tables. > > Since I've let myself be told that asciidoc handles tables better than > reStructuredText, I tested this a bit with the presumably inferior one. > > rst has two table types, simple tables and grid tables [1]. It seems > like grid tables can do pretty much anything, but they can be cumbersome > to work with. So I tried to check what can be done with simple tables. > > Here's a sample, converted using rst2html (Sphinx will be prettier, but > rst2html works for simple things like this): > > https://people.freedesktop.org/~jani/v4l-table-within-table.rst > https://people.freedesktop.org/~jani/v4l-table-within-table.html Yes, this would work. Can we remove the border from the main table? I guess it would be nicer. > > Rather than using nested tables, you might want to consider using > definition lists within tables: > > https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst > https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html > > You be the judge, but I think this is workable. It is workable, but I guess nested tables produced a better result. I did myself a test with nested tables with asciidoc too: https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii With looks very decent to me. I had to manually add the nested table, as pandoc conversion sent the DocBook's nested table to /dev/null. Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
On Thu, 18 Feb 2016, Mauro Carvalho Chehab wrote: > For simple documents like the one produced by kernel-doc, I guess > all markup languages would work equally. > > The problem is for complex documents like the media kAPI one, where > the document was written to produce a book. So, it uses some complex > features found at DocBook. One of such features we use extensively > is the capability of having a table with per-line columns. This way, > we can produce things like: > > V4L2_CID_COLOR_KILLER boolean Enable the color killer (i. e. force a black & > white image in case of a weak video signal). > V4L2_CID_COLORFX enumSelects a color effect. The following values > are defined: > V4L2_COLORFX_NONE Color effect is > disabled. > V4L2_COLORFX_ANTIQUEAn aging (old > photo) effect. > V4L2_COLORFX_ART_FREEZE Frost color > effect. > > In the above example, we have a main 3 columns table, and we embed > a 2 columns table at the third field of V4L2_CID_COLORFX to represent > possible values for this menu control. > > See https://linuxtv.org/downloads/v4l-dvb-apis/control.html for the > complete output of it. > > This is used extensively inside the media DocBook, and properly > supporting it is one of our major concerns. > > Are there any way to represent those things with the markup > languages currently being analyzed? > > Converting those tables will likely require manual work, as I don't > think automatic tools will properly handle it, specially since we > use some DocBook macros to help creating such tables. Since I've let myself be told that asciidoc handles tables better than reStructuredText, I tested this a bit with the presumably inferior one. rst has two table types, simple tables and grid tables [1]. It seems like grid tables can do pretty much anything, but they can be cumbersome to work with. So I tried to check what can be done with simple tables. Here's a sample, converted using rst2html (Sphinx will be prettier, but rst2html works for simple things like this): https://people.freedesktop.org/~jani/v4l-table-within-table.rst https://people.freedesktop.org/~jani/v4l-table-within-table.html Rather than using nested tables, you might want to consider using definition lists within tables: https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html You be the judge, but I think this is workable. BR, Jani. [1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#tables -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Em Thu, 18 Feb 2016 10:24:04 +0100 Daniel Vetter escreveu: > On Thu, Feb 18, 2016 at 10:11 AM, Jani Nikula wrote: > > On Thu, 18 Feb 2016, Daniel Vetter wrote: > >> On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet wrote: > >>> On Sun, 14 Feb 2016 13:27:04 +0100 > >>> Daniel Vetter wrote: > >>> > One concern/open I have for pro/cons are the hyperlinks from kerneldoc > comments. Currently we have the postproc hack, iirc Jani's patches > generated links native when extracting the kerneldoc. What's the > solution with spinx? > >>> > >>> So I've been trying to figure out what this refers to. Is this the > >>> cross-reference links within the document? When I did my sphinx hack it > >>> used a technique that, shall we say, strongly resembles what Jani's > >>> patches did. One difference is that Sphinx has the concept of > >>> "functions" built into it, so I use function references for those. > >> > >> That's what I meant. As long as I can type in stuff like func(), > >> &struct and similar and get a link for it automatically (plus anywhere > >> else in the templated stuff for function headers) I'm really happy. > > > > I think that could be made to work in rst just as well as > > asciidoc. Which is to say, kernel-doc may generate broken refs in both, > > since it doesn't know if the link target exists outside of the > > file. Also, in theory, it's possible to generate non-unique targets in > > the end result if there are same named enums, structs, static functions > > etc. but I think that's less of a problem. Asciidoc just ignores these > > issues, I don't know what sphinx does. > > > > I think some of that could be alleviated by making the kernel-doc > > inclusion a directive through a sphinx extension. It could at the very > > least provide informative error messages. But that's distant future. > > > > Worth noting is that, AFAICT, in all of the proposals, including the > > original where kernel-doc produces docbook, this autoreferencing only > > works within parts processed by kernel-doc. Not in the template > > documents themselves. (You can still use the markup's more verbose cross > > referencing keywords.) > > It works everywhere, even in the docbook template, as long as you mark > it up correctly. Which in docbook means func. > That's because it's a post-proc path over the entire doc. But then the > entire point here is to move the overview sections all into kerneldoc, > so making the links in the templates more verbose shouldn't be a big > deal. For simple documents like the one produced by kernel-doc, I guess all markup languages would work equally. The problem is for complex documents like the media kAPI one, where the document was written to produce a book. So, it uses some complex features found at DocBook. One of such features we use extensively is the capability of having a table with per-line columns. This way, we can produce things like: V4L2_CID_COLOR_KILLER boolean Enable the color killer (i. e. force a black & white image in case of a weak video signal). V4L2_CID_COLORFXenumSelects a color effect. The following values are defined: V4L2_COLORFX_NONE Color effect is disabled. V4L2_COLORFX_ANTIQUEAn aging (old photo) effect. V4L2_COLORFX_ART_FREEZE Frost color effect. In the above example, we have a main 3 columns table, and we embed a 2 columns table at the third field of V4L2_CID_COLORFX to represent possible values for this menu control. See https://linuxtv.org/downloads/v4l-dvb-apis/control.html for the complete output of it. This is used extensively inside the media DocBook, and properly supporting it is one of our major concerns. Are there any way to represent those things with the markup languages currently being analyzed? Converting those tables will likely require manual work, as I don't think automatic tools will properly handle it, specially since we use some DocBook macros to help creating such tables. -- Thanks, Mauro
Re: Kernel docs: muddying the waters a bit
On Thu, Feb 18, 2016 at 10:11 AM, Jani Nikula wrote: > On Thu, 18 Feb 2016, Daniel Vetter wrote: >> On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet wrote: >>> On Sun, 14 Feb 2016 13:27:04 +0100 >>> Daniel Vetter wrote: >>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc comments. Currently we have the postproc hack, iirc Jani's patches generated links native when extracting the kerneldoc. What's the solution with spinx? >>> >>> So I've been trying to figure out what this refers to. Is this the >>> cross-reference links within the document? When I did my sphinx hack it >>> used a technique that, shall we say, strongly resembles what Jani's >>> patches did. One difference is that Sphinx has the concept of >>> "functions" built into it, so I use function references for those. >> >> That's what I meant. As long as I can type in stuff like func(), >> &struct and similar and get a link for it automatically (plus anywhere >> else in the templated stuff for function headers) I'm really happy. > > I think that could be made to work in rst just as well as > asciidoc. Which is to say, kernel-doc may generate broken refs in both, > since it doesn't know if the link target exists outside of the > file. Also, in theory, it's possible to generate non-unique targets in > the end result if there are same named enums, structs, static functions > etc. but I think that's less of a problem. Asciidoc just ignores these > issues, I don't know what sphinx does. > > I think some of that could be alleviated by making the kernel-doc > inclusion a directive through a sphinx extension. It could at the very > least provide informative error messages. But that's distant future. > > Worth noting is that, AFAICT, in all of the proposals, including the > original where kernel-doc produces docbook, this autoreferencing only > works within parts processed by kernel-doc. Not in the template > documents themselves. (You can still use the markup's more verbose cross > referencing keywords.) It works everywhere, even in the docbook template, as long as you mark it up correctly. Which in docbook means func. That's because it's a post-proc path over the entire doc. But then the entire point here is to move the overview sections all into kerneldoc, so making the links in the templates more verbose shouldn't be a big deal. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Thu, 18 Feb 2016, Daniel Vetter wrote: > On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet wrote: >> On Sun, 14 Feb 2016 13:27:04 +0100 >> Daniel Vetter wrote: >> >>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc >>> comments. Currently we have the postproc hack, iirc Jani's patches >>> generated links native when extracting the kerneldoc. What's the >>> solution with spinx? >> >> So I've been trying to figure out what this refers to. Is this the >> cross-reference links within the document? When I did my sphinx hack it >> used a technique that, shall we say, strongly resembles what Jani's >> patches did. One difference is that Sphinx has the concept of >> "functions" built into it, so I use function references for those. > > That's what I meant. As long as I can type in stuff like func(), > &struct and similar and get a link for it automatically (plus anywhere > else in the templated stuff for function headers) I'm really happy. I think that could be made to work in rst just as well as asciidoc. Which is to say, kernel-doc may generate broken refs in both, since it doesn't know if the link target exists outside of the file. Also, in theory, it's possible to generate non-unique targets in the end result if there are same named enums, structs, static functions etc. but I think that's less of a problem. Asciidoc just ignores these issues, I don't know what sphinx does. I think some of that could be alleviated by making the kernel-doc inclusion a directive through a sphinx extension. It could at the very least provide informative error messages. But that's distant future. Worth noting is that, AFAICT, in all of the proposals, including the original where kernel-doc produces docbook, this autoreferencing only works within parts processed by kernel-doc. Not in the template documents themselves. (You can still use the markup's more verbose cross referencing keywords.) BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet wrote: > On Sun, 14 Feb 2016 13:27:04 +0100 > Daniel Vetter wrote: > >> One concern/open I have for pro/cons are the hyperlinks from kerneldoc >> comments. Currently we have the postproc hack, iirc Jani's patches >> generated links native when extracting the kerneldoc. What's the >> solution with spinx? > > So I've been trying to figure out what this refers to. Is this the > cross-reference links within the document? When I did my sphinx hack it > used a technique that, shall we say, strongly resembles what Jani's > patches did. One difference is that Sphinx has the concept of > "functions" built into it, so I use function references for those. That's what I meant. As long as I can type in stuff like func(), &struct and similar and get a link for it automatically (plus anywhere else in the templated stuff for function headers) I'm really happy. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
On Sun, 14 Feb 2016 13:27:04 +0100 Daniel Vetter wrote: > One concern/open I have for pro/cons are the hyperlinks from kerneldoc > comments. Currently we have the postproc hack, iirc Jani's patches > generated links native when extracting the kerneldoc. What's the > solution with spinx? So I've been trying to figure out what this refers to. Is this the cross-reference links within the document? When I did my sphinx hack it used a technique that, shall we say, strongly resembles what Jani's patches did. One difference is that Sphinx has the concept of "functions" built into it, so I use function references for those. If you mean links to the wider world, I'm not sure I see that in Jani's patches. It's going to be easily managed in whatever markup language we use, though. jon
Re: Kernel docs: muddying the waters a bit
Jonathan Corbet writes: > Indeed, I doubt many people want the DocBook itself. Might be nice to actually have a set of requirements before anyone tries to select a suitable system then :-) Here's my current set: asciidocsphinx htmlvia docbook native native (kinda) pdf via docbook via rst2pdf epubvia docbook?native man ? native? -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
On Tue, 16 Feb 2016 11:13:29 -0800 Keith Packard wrote: > https://github.com/HolgerPeters/sphinxcontrib-docbook > > which appears to provide docbook output for sphinx, but I haven't tested > this at all. Yup, that's the one I found, the one that says "Very much work in progress". It looks like that work stopped almost a year ago. > Of course, docbook is just a means to an end -- the question should > probably be what final output formats are actually required and whether > the tools generate those in a reasonable fashion. Indeed, I doubt many people want the DocBook itself. jon
Re: Kernel docs: muddying the waters a bit
Jani Nikula writes: > However I didn't think Sphinx could produce docbook, and a quick search > doesn't convince me otherwise. Do you have some links to back this up? > Would the lack of docbook be a showstopper? (Of course, the pandoc > swiss-army knife can handle rst->docbook if needed.) A quick search yields: https://github.com/HolgerPeters/sphinxcontrib-docbook which appears to provide docbook output for sphinx, but I haven't tested this at all. Of course, docbook is just a means to an end -- the question should probably be what final output formats are actually required and whether the tools generate those in a reasonable fashion. -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
On Tue, 16 Feb 2016, Jonathan Corbet wrote: > Whether this is a show-stopper is indeed a good question. I doubt many > people wanted the DocBook for its own sake, it's a matter of where you > can go from there. But yes, it would be good to be sure on this point. So the question is, are HTML, latex (for pdf and presumably ps), epub, texinfo, and man pages enough? My subjective opinion is they are enough, and whoever wants more can make it work for them using pandoc. It's not like we had this great publishing framework before that we'd be breaking. Quite the opposite, and it would be awesome to have a robust set of outputs actually working for the majority of people. > There's a certain elegance to it that I like, but it is an idea that > needs to actually be demonstrated. It could also come later on, though, > with the docproc or include mechanisms used for now. Oh, totally agreed, I was just musing on potential follow-up work. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
On Tue, 16 Feb 2016 10:25:49 +0200 Jani Nikula wrote: > However I didn't think Sphinx could produce docbook, and a quick search > doesn't convince me otherwise. Do you have some links to back this up? Somehow I was really sure of it, but I'm not finding it now. There is an extension out there, but it warns about being a "work in progress," so I'm not sure we can count it. Whether this is a show-stopper is indeed a good question. I doubt many people wanted the DocBook for its own sake, it's a matter of where you can go from there. But yes, it would be good to be sure on this point. > Sphinx might offer a way to drop docproc through the extension > mechanism, without resorting to the "separate-file approach". It might > be a more sensible approach as a whole. There's a certain elegance to it that I like, but it is an idea that needs to actually be demonstrated. It could also come later on, though, with the docproc or include mechanisms used for now. Thanks, jon
Re: Kernel docs: muddying the waters a bit
On Sat, 13 Feb 2016, Jonathan Corbet wrote: > So can we discuss? I'm not saying we have to use Sphinx, but, should we > choose not to, we should do so with open eyes and good reasons for the > course we do take. What do you all think? FWIW I was in favor of reStructuredText to begin with, but decided I'd rather have any lightweight markup than a bikeshed fest over which one. Switching is a non-issue to me. It does seem like the tools available for each markup carry more weight than the markup itself, as long as there aren't any huge gaps. Sphinx seems like a reasonable, actively developed choice. However I didn't think Sphinx could produce docbook, and a quick search doesn't convince me otherwise. Do you have some links to back this up? Would the lack of docbook be a showstopper? (Of course, the pandoc swiss-army knife can handle rst->docbook if needed.) Sphinx might offer a way to drop docproc through the extension mechanism, without resorting to the "separate-file approach". It might be a more sensible approach as a whole. I'll have a more in-depth look into this. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center
Re: Kernel docs: muddying the waters a bit
Daniel Vetter writes: > The other one is graphs - Keith showed me some neat stuff that > asciidoc can do, and I definitely wanted to integrate something like > that as a follow-up into the kerneldoc toolchain. Often a diagram is a > lot more helpful than lots of words. Can sphinx gives us that too? .. graphviz:: digraph foo { "bar" -> "baz"; } Even better than asciidoc -- svg output is supported in both html and pdf (when using rst2pdf). I had to hack asciidoc to add support for svg output when using docbook. > Wrt reformatting: I'm not going to like it, but I hope that with a bit > of sed we can fix up any of the asciidoc comments we have already > easily - right now we don't (yet) use much of the more sophisticated > markup yet. So much better to change now than 1 year down the road. I used pandoc on the docbook output from asciidoc to get a 100 page document converted here. It wasn't perfect -- all of the internal links were busted, and labels for tables were mis-positioned. It might be that a few minor fixes to pandoc could be done to add 'sphinx'-specific rst support that could fix this? I spent (too much) time yesterday playing with sphinx and generated a new html theme. Here's the result: http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html Here's the PDF output from rst2pdf, a python-based PDF output which doesn't use docbook *or* latex: http://keithp.com/~keithp/altusmetrum-sphinx/Altus%20Metrum.pdf I need to spend some quality time building my own PDF theme; the default provided by rst2pdf isn't great. It does, however, use fontconfig, so switching fonts is *way* easier than with docbook... There's currently an incompatibility between the rst2pdf and sphnix packages in debian (and upstream) which I hacked around to generate that output, but otherwise I'm using packaged bits. So, another pro for sphinx appears to be native PDF generation... -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
On Sun, Feb 14, 2016 at 1:57 AM, Keith Packard wrote: > Jonathan Corbet writes: > >> Asciidoc is a credible solution to the formatted documentation problem, >> but it's not the only such; I'd like to be sure that we pick the right >> one. I worry that asciidoc seems to be aimed mostly at small documents, >> and that the project itself seems a little lifeless - it's not a good >> sign when your main page's link to the repository has been dead for a long >> time. (Asciidoctor seems more active, with the Github folks behind it, >> but that means bringing Ruby into the picture). > > I was surprised when one of the asciidoctor developers said that > asciidoc itself was 'in maintenance mode for existing users'. I've tried > asciidoctor but never got it to the point where I was happy with the > results. Having two tools using the same nominal format doesn't seem > like a great idea to me. > > It's also clear from my hacking in asciidoc that docbook is the expected > target for that tool. I've managed to make direct HTML output usable, > but LaTeX doesn't work at all. Something which focuses on direct HTML > (and ePub) output would be pretty nice. > >> An alternative we haven't really looked at yet is ReStructuredText (or >> "RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is >> YA simple markup scheme, remarkably similar to Markdown or Asciidoc; >> Sphinx is a fairly sophisticated documentation system that uses RST. > > I've installed debian's python3-sphinx package; it looks like it doesn't > have a huge dependency chain below it, which is a nice change. > > I translated a fairly long document from asciidoc to rst using pandoc by > using the docbook output from asciidoc -- pandoc doesn't have a native > asciidoc reader, only a writer. The result didn't totally suck, although > I haven't messed with fixing the css or using a different theme at all. > > http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html > > I installed the sphinxcontrib.fulltoc extension so that the whole TOC > was visible from each section; this made navigating a lot easier. Having > search included (if you have javascript) seems like a nice feature. > >> Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain >> requirements there. > > Having functional native latex output means that even PDF generation is > lighterweight though. > >> It produces integrated, multi-file HTML natively, >> with a TOC, an index, cross-file cross references, and more. It can make >> things like function indexes. It claims output in epub, docbook, and man >> (I've not yet messed with those). The path to PDF is via latex; clearly >> the docbook path could be used too. > > I've tried epub and latex backends; epub seems just fine (it's just > html, after all). LaTeX works, and generates functional PDF, but I'm > going to have to spend a bunch of time making it looks nice. > > http://keithp.com/~keithp/altusmetrum-sphinx/AltusMetrum.pdf > >> So can we discuss? I'm not saying we have to use Sphinx, but, should we >> choose not to, we should do so with open eyes and good reasons for the >> course we do take. What do you all think? > > Having spent the afternoon playing with it, I'm definitely > impressed. I've spent a ton of time getting asciidoc to generate html > and pdf that I can tolerate; far too much of that involved hacking XML > files related to the docbook backend. > > Pros > > * Credible HTML output without docbook > > * Credible PDF output without docbook. > > * Constructs a unified set of documents across >multiple files. > > * Written in Python (2 or 3) > > * PanDoc already supports rst for both input and output. So, if we get >bored with RST, we've got a way out. > > Cons > > * Table formatting doesn't seem as sophisticated as asciidoc > > Questions > > * Conditional text appears to be harder to manage (I haven't managed to >make it work at all). > > * Takes over a directory making building more than one >document in a directory hard/impossible? The config file must be >named 'conf.py'? One concern/open I have for pro/cons are the hyperlinks from kerneldoc comments. Currently we have the postproc hack, iirc Jani's patches generated links native when extracting the kerneldoc. What's the solution with spinx? The other one is graphs - Keith showed me some neat stuff that asciidoc can do, and I definitely wanted to integrate something like that as a follow-up into the kerneldoc toolchain. Often a diagram is a lot more helpful than lots of words. Can sphinx gives us that too? Wrt reformatting: I'm not going to like it, but I hope that with a bit of sed we can fix up any of the asciidoc comments we have already easily - right now we don't (yet) use much of the more sophisticated markup yet. So much better to change now than 1 year down the road. Cheers, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: Kernel docs: muddying the waters a bit
Jonathan Corbet writes: > Asciidoc is a credible solution to the formatted documentation problem, > but it's not the only such; I'd like to be sure that we pick the right > one. I worry that asciidoc seems to be aimed mostly at small documents, > and that the project itself seems a little lifeless - it's not a good > sign when your main page's link to the repository has been dead for a long > time. (Asciidoctor seems more active, with the Github folks behind it, > but that means bringing Ruby into the picture). I was surprised when one of the asciidoctor developers said that asciidoc itself was 'in maintenance mode for existing users'. I've tried asciidoctor but never got it to the point where I was happy with the results. Having two tools using the same nominal format doesn't seem like a great idea to me. It's also clear from my hacking in asciidoc that docbook is the expected target for that tool. I've managed to make direct HTML output usable, but LaTeX doesn't work at all. Something which focuses on direct HTML (and ePub) output would be pretty nice. > An alternative we haven't really looked at yet is ReStructuredText (or > "RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is > YA simple markup scheme, remarkably similar to Markdown or Asciidoc; > Sphinx is a fairly sophisticated documentation system that uses RST. I've installed debian's python3-sphinx package; it looks like it doesn't have a huge dependency chain below it, which is a nice change. I translated a fairly long document from asciidoc to rst using pandoc by using the docbook output from asciidoc -- pandoc doesn't have a native asciidoc reader, only a writer. The result didn't totally suck, although I haven't messed with fixing the css or using a different theme at all. http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html I installed the sphinxcontrib.fulltoc extension so that the whole TOC was visible from each section; this made navigating a lot easier. Having search included (if you have javascript) seems like a nice feature. > Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain > requirements there. Having functional native latex output means that even PDF generation is lighterweight though. > It produces integrated, multi-file HTML natively, > with a TOC, an index, cross-file cross references, and more. It can make > things like function indexes. It claims output in epub, docbook, and man > (I've not yet messed with those). The path to PDF is via latex; clearly > the docbook path could be used too. I've tried epub and latex backends; epub seems just fine (it's just html, after all). LaTeX works, and generates functional PDF, but I'm going to have to spend a bunch of time making it looks nice. http://keithp.com/~keithp/altusmetrum-sphinx/AltusMetrum.pdf > So can we discuss? I'm not saying we have to use Sphinx, but, should we > choose not to, we should do so with open eyes and good reasons for the > course we do take. What do you all think? Having spent the afternoon playing with it, I'm definitely impressed. I've spent a ton of time getting asciidoc to generate html and pdf that I can tolerate; far too much of that involved hacking XML files related to the docbook backend. Pros * Credible HTML output without docbook * Credible PDF output without docbook. * Constructs a unified set of documents across multiple files. * Written in Python (2 or 3) * PanDoc already supports rst for both input and output. So, if we get bored with RST, we've got a way out. Cons * Table formatting doesn't seem as sophisticated as asciidoc Questions * Conditional text appears to be harder to manage (I haven't managed to make it work at all). * Takes over a directory making building more than one document in a directory hard/impossible? The config file must be named 'conf.py'? -- -keith signature.asc Description: PGP signature
Kernel docs: muddying the waters a bit
So I fear you all are going to hate me for this... Asciidoc is a credible solution to the formatted documentation problem, but it's not the only such; I'd like to be sure that we pick the right one. I worry that asciidoc seems to be aimed mostly at small documents, and that the project itself seems a little lifeless - it's not a good sign when your main page's link to the repository has been dead for a long time. (Asciidoctor seems more active, with the Github folks behind it, but that means bringing Ruby into the picture). An alternative we haven't really looked at yet is ReStructuredText (or "RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is YA simple markup scheme, remarkably similar to Markdown or Asciidoc; Sphinx is a fairly sophisticated documentation system that uses RST. I spent a few hours reworking the asciidoc patches to do RST instead, then built a few template files' worth of docs. The result can be seen at: http://static.lwn.net/kerneldoc/ It's very much a POC (however you might want to define the term), there's lots of glitches, I chose a theme pretty much at random, etc. But it shows that it can be done. Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain requirements there. It produces integrated, multi-file HTML natively, with a TOC, an index, cross-file cross references, and more. It can make things like function indexes. It claims output in epub, docbook, and man (I've not yet messed with those). The path to PDF is via latex; clearly the docbook path could be used too. I used my same docproc hack to extract the comments here, mostly because I had it at hand. We could go with Jani's separate-file approach if we wanted. There's also a tool out there (called "breathe") that's meant to turn doxygen-style comments into RST; I haven't had a chance to mess with it. We *could* also write an extension to pull the comments directly in Sphinx if there were a compelling reason to do so. If anybody's curious, the work done to get this far is in: git://git.lwn.net/linux.git doc/sphinx but it looks suspiciously like the previous asciidoc patches, and, in any case, it would have to be thrown out, publicly disowned, and replaced before going any further with this, should that be what we decide to do. So can we discuss? I'm not saying we have to use Sphinx, but, should we choose not to, we should do so with open eyes and good reasons for the course we do take. What do you all think? jon