Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:26:10 +0200
Markus Heiser  escreveu:

> Hi Mauro,
> 
> Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab 
> :
> > Yeah, it looks better, however table truncation seem to be
> > happening also on other parts, like the tables on this page:
> > 
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> > (original table: 
> > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> > This table should contain 32 bits, but only the first 7 bits are shown
> > 
> > and those (among others):
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> > 
> > Hmm... after looking more carefully, it added a horizontal scroll bar.
> > That looks ugly, IMHO, and makes harder to understand its contents. The
> > last one, in particular 
> > (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> > is a big table on both horiz and vert dimensions. Try to read how the
> > bits are packed on a random line in the middle of the table, like
> > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.  
> 
> I know what you mean ;-) ... I'am also unhappy, but I will address this point
> later when it goes to finish the layout.
> 
> Currently lets focus on contend and (the two)extensions.

OK.

> > The table here looks weird (although it is correct):
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
> > (original table: 
> > https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
> >   
> 
> I validated this and the other tables you mentioned above ... these are 
> all correct migrated ... it is 1:1 translated from DocBook ... they
> might be show different to this what you know from your docbook
> toolchain, because in the docbook-html you have no table grids and 
> wrong cellspans are not clear ... sometimes, like in the last example 
> you gave:
> 
>   
> 
> 
> 
> a colspec might ambiguous ... so there is no clear role to migrate.

Ok, that's what I was thinking. Ok, this can be fixed later manually,
where needed. Of course one way would be to disable grids on those
tables, but I would instead fix it.

> 
> > It seems that Sphinx is assuming something like "A4 portrait" for
> > the margins, while those big tables would only fit (in PDF) as
> > "A4 landscape".  
> 
> No, no, no ;-)
> 
> Sphinx assumes nothing about the layout, sphinx and the underlying
> docutils mostly juggling with nodes and the writers in e.g. the
> html-writer, outputs a clear HTML without any style but with classified 
> HTML tags. Styling is done in the presentation layer, in HTML, it is 
> done in CSS.

Hmm... Then there's something deadly wrong at CSS template, as it is
shown texts only at half of my horizontal res (1920).

Probably this is the culpit:

  .container { margin: 50px auto 40px auto; width: 600px; text-align: 
center; }

width is set to 600px, instead of using a percentage, like 100%
(or 90%).

> 
> > I guess the better would be to not limit the right
> > margin or to change it where those big tables happen, in order to
> > allow PDF generation.  
> 
> Generating PDF has nothing to do with generating HTML. To generate
> PDF there is a other writer, the latex2e writer, which produce 
> LaTeX markup from which you build PDF or other printed-like medias.

Ok.

>  
> > 
> > There are also some tables that went wrong. See the Color Sample
> > Location table at:
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> > I'm pretty sure we'll need to fix some cases like that manually.
> > I didn't check, but perhaps, in this case, the DocBook were using 
> > empty columns just to make the table bigger. If so, this is not a
> > problem with the conversion, and should be manually fixed later.  
> 
> +1
> 
> Yes, lets do it manually later ... what I have in my POC is a automated
> process, it is hard to consider individuals in an automatic process.
> Making details *nicer* and making ambiguous markups clear is manually
> done in minutes where I need hours to implement this in a automated
> process.

Yeah, we should not try to fix everything via auto-scripts, and
spending time right now with manual fixes will be wasted, as we need
to run it at the latest media documentation, as changes might have
happened upstream.

> 
> Aside, from:  http://docutils.sourceforge.net/docs/peps/pep-0258.html
> 
> Docutils Project Model -- Project components and data flow:
> 
>  +---+
>  |Docutils:  |
>  | docutils.core.Publisher,  |
> 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser  wrote:
> Am 06.05.2016 um 17:06 schrieb Jani Nikula :
>
>> On Fri, 06 May 2016, Markus Heiser  wrote:
>>> @Jonathan: what do you think? Should I prepare a patch
>>> with a basic reST (sphinx) build infrastructure, including
>>> 
>>> * a folder for sphinx docs:
>>> 
>>>  ./Documentation/sphinx/
>> 
>> I'm already working on a patch series taking a different approach. I
>> don't think we should hide the documentation under an extra folder named
>> after a tool. Actually, I'm strongly opposed to that.
>
> Could you post a link to a repo? / thanks

Very much a work-in-progress
https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx

I was hoping to polish it a bit more before showing it to the world.

> There is no need for concurrency, let's work together on your repo. 
> Within my POC I realized similar building processes we will need in the
> kernel sources ... where you have cascading configuration. A base 
> configuration which fits for all common cases and (if needed) a 
> *per-book* configuration.
>
> At the end, when it comes to generate pdf books/articles, man pages 
> and e.g. texinfo files out of a sphinx-project you will need a build
> infrastructure like this.

...

> You will need on sphinx-project for each DocBook and one single 
> sphinx-project where you collect the .txt to .rst migrated files.

Surely you know more about Sphinx than I do, but I specifically would
like to include e.g. gpu documentation in the main build. I'm really
hoping we can have *additional* configuration files for special cases
(only) as needed.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:06:49 +0300
Jani Nikula  escreveu:

> On Fri, 06 May 2016, Markus Heiser  wrote:
> > @Jonathan: what do you think? Should I prepare a patch
> > with a basic reST (sphinx) build infrastructure, including
> >
> > * a folder for sphinx docs:
> >
> >   ./Documentation/sphinx/  
> 
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.
> 
> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.

We won't avoid the need of moving things among directories, as we
have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Ok, if we put the .rst files at Documentation, we very likely reduce
the number of renames, but we'll increase the Makefile complexity,
and the risk of breakages. One alternative would be to put the sphinx
stuff on a separate Makefile, but using multiple makefiles on a single
dir is not standard at the Kernel.

> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.


Well, git will show the change as a rename, no matter if the
directory name changes or not (except if we keep the rst files
with .txt extension), but I agree with you that people will expect
to see text files at Documentation, and most will just read it without
caring to run Sphinx.

-- 
Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 16:27:21 +0200
Markus Heiser  escreveu:

> Hi all, hi Jonathan,
> 
> Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab 
> :
> 
> > Em Fri, 6 May 2016 15:32:35 +0200
> > Markus Heiser  escreveu:
> >   
> >> Hi Mauro,
> >> 
> >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab 
> >> :
> >>   
> >>> Markus,
> >>> 
> >>> Em Fri, 6 May 2016 13:23:06 +0200
> >>> Markus Heiser  escreveu:
> >>>   
>  
>  In this conf.py you have to *register* your folder with the extensions.
>  A few words about the flat-table extension and a (future) kernel-doc 
>  one:
> >>> 
> >>> ...
> >>>   
>  The flat-table is a pure docutils (the layer below sphinx) extension 
>  which
>  is not application specific, so I will ask for moving it to the docutils 
>  upstream. 
> >>> 
> >>> So, if I understood well, your proposal is to have a conf.py and the
> >>> flat-table (plus other extensions) at the Kernel tree.
> >> 
> >> Each book (better call it root-node) is a Sphinx-project, each 
> >> Sphinx-project need a conf.py file (the build configuration file)
> >> in its reST-source tree.
> >> 
> >> * http://www.sphinx-doc.org/en/stable/config.html
> >>   
> >>> Assuming that docutils upstream receives the flat-table extension
> >>> (and eventually modifies it), while the new version doesn't arrive
> >>> all distros, we'll end by having some developers using the newer
> >>> docutils with the extension, plus others using the in-tree one.
> >>> 
> >>> Is there a way to specify at the conf.py what extension variant
> >>> should it use, in case of both the in-tree or the docutils have
> >>> the same?
> >> 
> >> The build configuration file is a regular python file, you can 
> >> implement conditions whatever you want/need.
> >>   
> >>> This could be trickier if they end by modifying the extension,
> >>> but we can always backport the latest version, if they change the
> >>> API.
> >> 
> >> As far as i know, the docutils API is stable since 2002. In the
> >> meantime there has been so many application build on it that
> >> it is not realistic, you will see a not backward compatibly 
> >> change.
> >> 
> >> The docutils project is conservative, very conservative, IMO to
> >> conservative.
> >> 
> >> Today I'am doubtful if it isn't better I would merge it sphinx
> >> upstream. I have to discuss this with some maintainers, but 
> >> before I have to persuade myself, that all aspects are covered
> >> and the implementations are robust. We are at the beginning and
> >> we should not fear about every bit which could happen in the future.
> >> 
> >> The sphinx / docutils bottom plate gives us a number of degrees 
> >> of freedom to find answers to question we have not yet asked. ;-)  
> > 
> > Ok. So, from what I understand, once Sphinx support is added at
> > Kernel upstream, we could convert the media docbook to
> > reST+flat-table extension, adding such extension either on a shared
> > place or only for the media DocBook build, together with its
> > conf.py.
> >   
> 
> Yes, in your media-conf you could decide to use a extension.
> 
> > Once it reaches upstream (either sphinx or docutils), we can
> > work to make it integrate better with upstream as needed.
> > 
> > Right?  
> 
> yes, right :-)
> 
> > If so, I'm ok with merging it as soon as possible.  
> 
> If we advice a merge of the flat-table directive we should 
> bundle this with the (to implement) "kernel-doc" directive ...
> 
> > In reST the directive might look like:
> > 
> >  -
> > Device Instance and Driver Handling
> > ===
> > 
> > .. kernel-doc::  drivers/gpu/drm/drm_drv.c
> >   :doc:  driver instance overview
> >   :exported:
> > 
> >  -  
> 
> and the patches from my kernel-doc perl script to produce
> reST from source code comments.
> 
> With this bundle within the kernel tree we have a good starting
> point to compose reST documents from scratch and to migrate book
> by book from DocBook to reST.
> 
> I insist to migrate book by book, because there are some
> broken books. Broken by that, that some sources have changed
> but not the corresponding documentation which use the comments
> of these sources ... grap "Ooops" in the builded (xml or rst) docs.
> 
> E.g. I greped the .rst file and found the following Oops in the migrated 
> books:
> 
> ./books/mtdnand/pubfunctions-000-012.rst:13:Oops
> ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
> ./books/s390-drivers/ccw-000-004-003.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
> ./books/device-drivers/Basics-000-001-002.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
> ./books/device-drivers/Basics-000-009-032.rst:13:Oops
> 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi all, hi Jonathan,

Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab :

> Em Fri, 6 May 2016 15:32:35 +0200
> Markus Heiser  escreveu:
> 
>> Hi Mauro,
>> 
>> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab 
>> :
>> 
>>> Markus,
>>> 
>>> Em Fri, 6 May 2016 13:23:06 +0200
>>> Markus Heiser  escreveu:
>>> 
 
 In this conf.py you have to *register* your folder with the extensions.
 A few words about the flat-table extension and a (future) kernel-doc one:  
>>> 
>>> ...
>>> 
 The flat-table is a pure docutils (the layer below sphinx) extension which
 is not application specific, so I will ask for moving it to the docutils 
 upstream.   
>>> 
>>> So, if I understood well, your proposal is to have a conf.py and the
>>> flat-table (plus other extensions) at the Kernel tree.  
>> 
>> Each book (better call it root-node) is a Sphinx-project, each 
>> Sphinx-project need a conf.py file (the build configuration file)
>> in its reST-source tree.
>> 
>> * http://www.sphinx-doc.org/en/stable/config.html
>> 
>>> Assuming that docutils upstream receives the flat-table extension
>>> (and eventually modifies it), while the new version doesn't arrive
>>> all distros, we'll end by having some developers using the newer
>>> docutils with the extension, plus others using the in-tree one.
>>> 
>>> Is there a way to specify at the conf.py what extension variant
>>> should it use, in case of both the in-tree or the docutils have
>>> the same?  
>> 
>> The build configuration file is a regular python file, you can 
>> implement conditions whatever you want/need.
>> 
>>> This could be trickier if they end by modifying the extension,
>>> but we can always backport the latest version, if they change the
>>> API.  
>> 
>> As far as i know, the docutils API is stable since 2002. In the
>> meantime there has been so many application build on it that
>> it is not realistic, you will see a not backward compatibly 
>> change.
>> 
>> The docutils project is conservative, very conservative, IMO to
>> conservative.
>> 
>> Today I'am doubtful if it isn't better I would merge it sphinx
>> upstream. I have to discuss this with some maintainers, but 
>> before I have to persuade myself, that all aspects are covered
>> and the implementations are robust. We are at the beginning and
>> we should not fear about every bit which could happen in the future.
>> 
>> The sphinx / docutils bottom plate gives us a number of degrees 
>> of freedom to find answers to question we have not yet asked. ;-)
> 
> Ok. So, from what I understand, once Sphinx support is added at
> Kernel upstream, we could convert the media docbook to
> reST+flat-table extension, adding such extension either on a shared
> place or only for the media DocBook build, together with its
> conf.py.
> 

Yes, in your media-conf you could decide to use a extension.

> Once it reaches upstream (either sphinx or docutils), we can
> work to make it integrate better with upstream as needed.
> 
> Right?

yes, right :-)

> If so, I'm ok with merging it as soon as possible.

If we advice a merge of the flat-table directive we should 
bundle this with the (to implement) "kernel-doc" directive ...

> In reST the directive might look like:
> 
>  -
> Device Instance and Driver Handling
> ===
> 
> .. kernel-doc::  drivers/gpu/drm/drm_drv.c
>   :doc:  driver instance overview
>   :exported:
> 
>  -

and the patches from my kernel-doc perl script to produce
reST from source code comments.

With this bundle within the kernel tree we have a good starting
point to compose reST documents from scratch and to migrate book
by book from DocBook to reST.

I insist to migrate book by book, because there are some
broken books. Broken by that, that some sources have changed
but not the corresponding documentation which use the comments
of these sources ... grap "Ooops" in the builded (xml or rst) docs.

E.g. I greped the .rst file and found the following Oops in the migrated books:

./books/mtdnand/pubfunctions-000-012.rst:13:Oops
./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
./books/s390-drivers/ccw-000-004-003.rst:13:Oops
./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
./books/device-drivers/Basics-000-001-002.rst:13:Oops
./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
./books/device-drivers/Basics-000-009-032.rst:13:Oops
./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops
./books/gadget/api-000-011-005.rst:13:Oops
./books/gadget/api-000-011-009.rst:13:Oops
./books/gadget/api-000-011-007.rst:13:Oops
./books/gadget/api-000-011-003.rst:13:Oops
./books/gadget/api-000-011-011.rst:13:Oops
./books/genericirq/intfunctions-000-009.rst:13:Oops

- Summarize 

@Jonathan: what do you think? Should I prepare a patch
with a basic reST (sphinx) build 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jani,

I forget to mentioning, with a local copy of my kernel-doc script:

https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc

You could do reST markup in the source code comments and extract them.

This might be a interim workaround which helps you not to edit source 
code comments twice.

-- Markus--



Am 06.05.2016 um 13:23 schrieb Markus Heiser :

> 
> Hy Jani,
> 
> Am 04.05.2016 um 18:13 schrieb Jani Nikula :
> 
>>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>>> 
 I think all of this makes sense.  It would be really nice to have the
 directives in the native sphinx language like that.  I *don't* think we
 need to aim for that at the outset; the docproc approach works until we can
 properly get rid of it.  What would be *really* nice would be to get
 support for the kernel-doc directive into the sphinx upstream.
>>> 
>>> No need for kernel-doc directive in sphinx upstream, later it will be 
>>> an extension which could be installed by a simple command like 
>>> "pip install kernel-doc-extensions" or similar.
>>> 
>>> I develop these required extension (and more) within my proof of concept
>>> on github ... this takes time ... if I finished all my tests and all is
>>> well, I will build the *kernel-doc-extensions* package and deploy it
>>> on https://pypi.python.org/pypi from where everyone could install this 
>>> with "pip".
>> 
>> I think we should go for vanilla sphinx at first, to make the setup step
>> as easy as possible for everyone. Even if it means still doing that ugly
>> docproc step to call kernel-doc. We can improve from there, and I
>> definitely appreciate your work on making this work with sphinx
>> extensions.
> 
> +1 
> 
>> That said, how would it work to include the kernel-doc extension in the
>> kernel source tree? Having things just work if sphinx is installed is
>> preferred over requiring installation of something extra from pypi. (I
>> know this may sound backwards for a lot of projects, but for kernel I'm
>> pretty sure this is how it should be done.)
> 
> Thats all right. Lets talk about the extension infrastructure by example:
> 
> First we have to chose a folder where we place all the *sphinx-documentation*
> I recommending:
> 
> /share/linux/Documentation/sphinx
> 
> Next we have to chose a folder where reST-extensions should take place, I
> would prefer ... or similar:
> 
> /share/linux/Documentation/sphinx/extensions
> 
> Lets say, you wan't to get in use of the "flat-table" extension.
> 
> Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
> other extensions which might be there) ...
> 
> https://github.com/return42/sphkerneldoc/tree/master/doc/extensions
> 
> Now lets say you are writing on a gpu book, it wold be placed in the folder:
> 
> /share/linux/Documentation/sphinx/gpu
> 
> In this gpu-folder you have to place the conf.py config file, needed to
> setup the sphinx build environment.
> 
> /share/linux/Documentation/sphinx/gpu/conf.py
> 
> In this conf.py you have to *register* your folder with the extensions.
> 
>  
> 
>import os.path, sys
> 
>EXT_PATH  = "../extensions"  # the path of extension folder relative to 
> the conf.py file
>sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))
> 
># now import the "flat-table" extension, it will be self-registering to 
> docutils
> 
>import rstFlatTable
> 
>  
> 
> Thats all, you can run your sphinx-build command and the flat-tables in your
> reST sources should be handled as common tables.
> 
> ASIDE: 
> 
> You will find similar parts in your conf.py which you have created 
> with the sphinx-quickstart command. There, you will also find a block 
> looks like ...
> 
> extensions = [
>'sphinx.ext.autodoc'
> 
> ]
> 
> Don't try to add flat-table extension to this list. This list is a list
> of sphinx extensions, we will use it later for other *real* sphinx 
> extensions.
> 
> A few words about the flat-table extension and a (future) kernel-doc one:
> 
> The flat-table is a pure docutils (the layer below sphinx) extension which
> is not application specific, so I will ask for moving it to the docutils 
> upstream. 
> 
> The kernel-doc extension on the other side is a very (very) kernel specific
> application, this would never go to sphinx nor docutils upstream.
> 
> --Markus--
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hy Jani,

Am 04.05.2016 um 18:13 schrieb Jani Nikula :

>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>> 
>>> I think all of this makes sense.  It would be really nice to have the
>>> directives in the native sphinx language like that.  I *don't* think we
>>> need to aim for that at the outset; the docproc approach works until we can
>>> properly get rid of it.  What would be *really* nice would be to get
>>> support for the kernel-doc directive into the sphinx upstream.
>> 
>> No need for kernel-doc directive in sphinx upstream, later it will be 
>> an extension which could be installed by a simple command like 
>> "pip install kernel-doc-extensions" or similar.
>> 
>> I develop these required extension (and more) within my proof of concept
>> on github ... this takes time ... if I finished all my tests and all is
>> well, I will build the *kernel-doc-extensions* package and deploy it
>> on https://pypi.python.org/pypi from where everyone could install this 
>> with "pip".
> 
> I think we should go for vanilla sphinx at first, to make the setup step
> as easy as possible for everyone. Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

+1 

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Thats all right. Lets talk about the extension infrastructure by example:

First we have to chose a folder where we place all the *sphinx-documentation*
I recommending:

 /share/linux/Documentation/sphinx

Next we have to chose a folder where reST-extensions should take place, I
would prefer ... or similar:
 
 /share/linux/Documentation/sphinx/extensions

Lets say, you wan't to get in use of the "flat-table" extension.

Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
other extensions which might be there) ...

 https://github.com/return42/sphkerneldoc/tree/master/doc/extensions

Now lets say you are writing on a gpu book, it wold be placed in the folder:

 /share/linux/Documentation/sphinx/gpu

In this gpu-folder you have to place the conf.py config file, needed to
setup the sphinx build environment.

 /share/linux/Documentation/sphinx/gpu/conf.py

In this conf.py you have to *register* your folder with the extensions.

 

import os.path, sys

EXT_PATH  = "../extensions"  # the path of extension folder relative to the 
conf.py file
sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))

# now import the "flat-table" extension, it will be self-registering to 
docutils

import rstFlatTable

 

Thats all, you can run your sphinx-build command and the flat-tables in your
reST sources should be handled as common tables.

ASIDE: 

You will find similar parts in your conf.py which you have created 
with the sphinx-quickstart command. There, you will also find a block 
looks like ...

extensions = [
'sphinx.ext.autodoc'

]

Don't try to add flat-table extension to this list. This list is a list
of sphinx extensions, we will use it later for other *real* sphinx 
extensions.

A few words about the flat-table extension and a (future) kernel-doc one:

The flat-table is a pure docutils (the layer below sphinx) extension which
is not application specific, so I will ask for moving it to the docutils 
upstream. 

The kernel-doc extension on the other side is a very (very) kernel specific
application, this would never go to sphinx nor docutils upstream.

--Markus--

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro,

Am 04.05.2016 um 18:15 schrieb Mauro Carvalho Chehab :

> Em Wed, 4 May 2016 11:34:08 +0200
> Markus Heiser  escreveu:
> 
>> Hi all, (hi Jonathan, please take note of my offer below)
>> 
>> Am 03.05.2016 um 16:31 schrieb Daniel Vetter :
>> 
>>> Hi all,
>>> 
>>> So sounds like moving ahead with rst/sphinx is the option that should
>>> allow us to address everyone's concerns eventually? Of course the
>>> first one won't have it all (media seems really tricky), ...  
>> 
>> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 
>> For this, I wrote the "flat-table" reST-directive, which adds 
>> missing cells automatically:
>> 
>> doc:
>> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
>> source: 
>> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py
> 
> Yeah, this should address the lack of a proper way to markup cell/row
> spans, providing the additional bits for the tables we have at media.
> 
> Yet, there are some issues with table conversions. See below.
> Some tables, like the one here:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html
> 
> are truncated (tested with Mozilla and Chrome), and part of the information is
> lost due to that.

Not a problem of rendering. This was a bug in the magration from DocBook to 
reST. 
You might remember that we have discussed, that some of the tables are better
marked-up as definition lists. This was (the last) one I forgot to convert to 
a definition list ... I hope it was the last one, if not and you find more 
or other broken parts, please inform me (on the linux-tv mailing, or direct).

--Markus--

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Thu, 5 May 2016 07:02:10 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 14:57:38 -0300
> Mauro Carvalho Chehab  wrote:
> 
> > Also, media documentation is not just one more documentation. It is
> > the biggest one we have, and that has more changes than any other
> > documentation under Documentation/DocBook:
> > 
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> > 116
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type 
> > f`|wc -l
> > 179
> > 
> > It also is more than twice the size of the other DocBook docs:
> > 
> > $ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
> >   82275 total
> > $ wc -l $(ls *.tmpl|grep -v media)|tail -1
> >   29568 total
> > 
> > E. g. media corresponds to 60% of the number of patches and 73% of
> > the DocBook stuff.  
> 
> These numbers are not entirely representative, I have to say.  You're
> ignoring the kerneldoc comments - which is what much of the "DocBook"
> documents are made of, and which is the focus of much of this activity.  If
> you could find a way to count those, you'd get a different picture.

Yeah, if we take the big picture, I'm pretty sure that there are more
stuff written using kerneldoc. However, what is written at the 
kerneldoc comments don't use DocBook markup, but its own markup
language.

Assuming that we'll keep using kerneldoc script to convert from the
Kerneldoc NANO markup, it means that any changes at the kerneldoc
backend to use DocBook/Markdown/asciidoc/reST/... won't change anything
for the developers nor will require them to actually use the new markups.
So, they can gradually start using new markups as they wish/learn, if
they want to improve the documentation. Lazy developers could just ignore
reST if they want, as everything will work as before.

However, for the files written directly in DocBook, there's no option:
at the moment the doc is converted, the developer will need to submit
patches using the new markup language. So, we need to be sure that
such transition will happen in a way that won't cause more harm than
needed.  I would prefer to convert all of them at one, on a single Kernel
release (or two), in order to minimize the impact for the developers.

> 
> But I don't think that really matters; there doesn't seem to be *that* much
> disagreement here.
> 
> The media book is important; we want it to be a part of the overall kernel
> documentation suite and not stuck in some DocBook ghetto.  I agree that we
> should have an idea for a plausible path for *all* of our documentation.
> But I'm also concerned about delaying this work yet again; we have
> developers trying to push forward with improved documentation, and they've
> had to wait a year for this stuff - so far.

Yeah, I understand.

Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 14:57:38 -0300
Mauro Carvalho Chehab  wrote:

> Also, media documentation is not just one more documentation. It is
> the biggest one we have, and that has more changes than any other
> documentation under Documentation/DocBook:
> 
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> 116
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc 
> -l
> 179
> 
> It also is more than twice the size of the other DocBook docs:
> 
> $ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
>   82275 total
> $ wc -l $(ls *.tmpl|grep -v media)|tail -1
>   29568 total
> 
> E. g. media corresponds to 60% of the number of patches and 73% of
> the DocBook stuff.

These numbers are not entirely representative, I have to say.  You're
ignoring the kerneldoc comments - which is what much of the "DocBook"
documents are made of, and which is the focus of much of this activity.  If
you could find a way to count those, you'd get a different picture.

But I don't think that really matters; there doesn't seem to be *that* much
disagreement here.

The media book is important; we want it to be a part of the overall kernel
documentation suite and not stuck in some DocBook ghetto.  I agree that we
should have an idea for a plausible path for *all* of our documentation.
But I'm also concerned about delaying this work yet again; we have
developers trying to push forward with improved documentation, and they've
had to wait a year for this stuff - so far.

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 13:50:35 -0300
Mauro Carvalho Chehab  wrote:

> Em Wed, 4 May 2016 19:13:21 +0300
> Jani Nikula  escreveu:
> > I think we should go for vanilla sphinx at first, to make the setup step
> > as easy as possible for everyone.  
> 
> Vanilla Sphinx doesn't work, as reST markup language is too limited
> to support the docs we have. So, we should either push the needed
> extensions to Sphinx reST or find a way to put it at Kernel tree
> without causing too much pain for the developers, e. g. as something
> that just doing "make htmldoc" would automatically use such extensions
> without needing to actually install the extensions.

It works for everything except the extended media book, right?  Or is there
something I'm missing here?  I am very much interested in having one system
for *all* our docs, but we wouldn't attempt to convert a document can't be
expressed in sphinx.

> > Even if it means still doing that ugly
> > docproc step to call kernel-doc. We can improve from there, and I
> > definitely appreciate your work on making this work with sphinx
> > extensions.  
> 
> I disagree: We should not cause documentation regressions by
> producing incomplete documentation and losing tables because of
> such conversion.

> The quality of the documentation after the change should be *at least*
> equal to the one we current produce, never worse.

Agreed; that's why I think the existing DocBook mechanism should stay in
place until that document will be improved by the change.  But I'd rather
not hold up the entire process for one book, especially since pushing that
process forward can only help in dealing with those final problems.

> > That said, how would it work to include the kernel-doc extension in the
> > kernel source tree? Having things just work if sphinx is installed is
> > preferred over requiring installation of something extra from pypi. (I
> > know this may sound backwards for a lot of projects, but for kernel I'm
> > pretty sure this is how it should be done.)  
> 
> Yeah, using pypi seems an additional undesired step. Aren't there
> any way to make python to use an additional extension at the
> Kernel tree without needing to install it?

Well, sphinx has to come from somewhere too.  But yes, we should be able to
carry extensions in the kernel tree.  But I would still rather upstream it
(to sphinx) if possible, so we're not stuck trying to maintain it across
several sphinx releases.  I don't know how volatile their interfaces are,
but it would be, in any case, the analog of an out-of-tree module...

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 08:57:13 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter  wrote:
> 
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.
> > 
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...  
> 
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home.  I hope to begin emerging from this mess in the
> near future.
> 
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done.  Which would you have me push into 4.7?

IMHO, Markus approach is the one that is providing a better result,
as it added support for missing features that we require for the
media DocBook.

Still, it seems premature to merge it for 4.7.

Markus is getting real progress on converting the media docs to work
with Sphynx, but there are still troubles when the table is big, as,
currently, it is truncating everything that passes the right margin
on some tables.

So, IMHO, the next action item would be to be sure that big tables 
will not be truncated. Assuming that he can fix that in time, we
can merge it for 4.8, and then start porting the documentation to
use it.

Btw, converting the DocBook/media Makefile will require an extra
step, as part of the documentation is generated via scripts (some
C file examples, and include/uapi headers). Those scripts also
warrant that (almost) every API at include/uapi is synchronized
with the DocBook. I use this on my patch review process, in order
to reject patches that don't add the proper documentation updates.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 4, 2016 at 5:02 PM, Daniel Vetter  wrote:
> On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote:
>> On Wed, 4 May 2016 16:18:27 +0200
>> Daniel Vetter  wrote:
>>
>> > > I'd really like to converge on the markup question, so that we can start
>> > > using all the cool stuff with impunity in gpu documentations.
>> >
>> > Aside: If we decide this now I could send in a pull request for the
>> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
>> > markdown/asciidoc code I still have). That would be really awesome ...
>>
>> Sorry for my relative absence...I'm still busy dealing with bureaucracy
>> an ocean away from home.  I hope to begin emerging from this mess in the
>> near future.
>>
>> So ... there's the code you have, the work I (+Jani) did, and the work
>> Markus has done.  Which would you have me push into 4.7?
>>
>> The sphinx/rst approach does seem, to me, to be the right one, with the
>> existing DocBook structure remaining in place for those who want/need
>> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
>> it's hackish at best and needs a lot of cleaning up.  It's a proof of
>> concept, but it's hardly finished (one might say it's barely begun...)
>>
>> In the end, I guess, I feel that anything we might try to push for 4.7 is
>> going to look rushed and not ready, and Linus might react accordingly.
>> I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
>> on things in that time frame...  I suspect you're pretty well fed up with
>> this stuff being pushed back, and rightly so.  All I can do is apologize.
>>
>> That said, if you do think there's something out there that is good
>> enough to consider pushing in a week or two, do tell and we can all take
>> a look.
>
> Well I'd just have taken the asciidoc hacks I have currently in my
> topic/kerneldoc branch, converted to sphinx and looked how it fares. It
> should be fairly minimal, and I think the first step we want to do for the
> long-term plan. I hope I can ready something, and then we can look whether
> it's rushed for 4.7 or not.

Ok, discussed this a bit more with Jani on IRC and he really doesn't
like the old design of that branch (it calls the converter for every
kerneldoc comment). So I guess nothing rushed for 4.7, but hopefully
something for 4.8.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 11:34:08 +0200
Markus Heiser  escreveu:

> Hi all, (hi Jonathan, please take note of my offer below)
> 
> Am 03.05.2016 um 16:31 schrieb Daniel Vetter :
> 
> > Hi all,
> > 
> > So sounds like moving ahead with rst/sphinx is the option that should
> > allow us to address everyone's concerns eventually? Of course the
> > first one won't have it all (media seems really tricky), ...  
> 
> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 
> 
> Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab 
> :
> 
> > With that sense, the "List tables" format is also not good, as
> > one row addition would generate several hunks (one for each column
> > of the table), making harder to review the patch by just looking at
> > the diff.  
> 
> For this, I wrote the "flat-table" reST-directive, which adds 
> missing cells automatically:
> 
> doc:
> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
> source: 
> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

Yeah, this should address the lack of a proper way to markup cell/row
spans, providing the additional bits for the tables we have at media.

Yet, there are some issues with table conversions. See below.

> 
> I used "flat-table" to migrate all DocBook-XML documents to reST. With this
> directive, I also managed to migrate the complete media book (no more TODOs)
> incl. the large tables like them from subdev-formats:
> 
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 
> (Rendering large tables is a general discussion which should not take place 
> in this MT)

Some tables, like the one here:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html

are truncated (tested with Mozilla and Chrome), and part of the information is
lost due to that.

Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
> Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
> particular with your "MARKDOWNREADY := gpu.xml" process.
>
> As I understood, you convert markdown to docbook within the kernel-doc 
> script using pandoc's executable? ... I don't face this topic. With my 
> modification of kernel-doc I produced pure reST markup from standardize
> kernel-doc markup, no intermediate steps or tools required.

That pandoc thing is a dead end. Forget about it. I think we've all
pretty much agreed we should have kernel-doc produce the lightweight
markup directly since [1].

[1] http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com

> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>
>> I think all of this makes sense.  It would be really nice to have the
>> directives in the native sphinx language like that.  I *don't* think we
>> need to aim for that at the outset; the docproc approach works until we can
>> properly get rid of it.  What would be *really* nice would be to get
>> support for the kernel-doc directive into the sphinx upstream.
>
> No need for kernel-doc directive in sphinx upstream, later it will be 
> an extension which could be installed by a simple command like 
> "pip install kernel-doc-extensions" or similar.
>
> I develop these required extension (and more) within my proof of concept
> on github ... this takes time ... if I finished all my tests and all is
> well, I will build the *kernel-doc-extensions* package and deploy it
> on https://pypi.python.org/pypi from where everyone could install this 
> with "pip".

I think we should go for vanilla sphinx at first, to make the setup step
as easy as possible for everyone. Even if it means still doing that ugly
docproc step to call kernel-doc. We can improve from there, and I
definitely appreciate your work on making this work with sphinx
extensions.

That said, how would it work to include the kernel-doc extension in the
kernel source tree? Having things just work if sphinx is installed is
preferred over requiring installation of something extra from pypi. (I
know this may sound backwards for a lot of projects, but for kernel I'm
pretty sure this is how it should be done.)

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 04.05.2016 um 15:43 schrieb Daniel Vetter :

> On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote:
>>> On Wed, 04 May 2016, Markus Heiser  wrote:
>>> I'd be *very* hesitant about adding the kind of things you do in
>>> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
>>> comments must be as simple as possible with basically pass-through of
>>> the comment blocks to sphinx.
>> 
>> Right, but you forgot, that the current markup in the source code comments
>> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
>> these source code comments to reST markup ;-)
>> 
>> So there is a need to differentiate between *vintage* kernel-doc markup 
>> and reST markup.
>> 
>> My suggestion is to add a tag to the comments, here a short example
>> what this might look like.
>> 
>>  --
>> /**
>> * drm_minor_release - Release DRM minor
>> * @minor: Pointer to DRM minor object
>> *
>> * Release a minor that was previously acquired via drm_minor_acquire().
>> */
>>  --
>> 
>> in short: the vintage style does not need any change and 
>> comments with reST markup has a tag ":reST:" in the first 
>> line(s) ...
>> 
>>  --
>> /**
>> * :reST:
>> *
>> * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>> *
>> *Release DRM minor
>> *
>> *:param struct drm_minor \*minor: Pointer to DRM minor object
>> *
>> */
>>  --
>> 
>> Comments with the ":reST:" tag could be exported and pass-through
>> to sphinx.
>> 
>>> Specifically, do not attempt to detect and
>>> parse elements like lists in kernel-doc.
>> 
>> If your markup in the comments is plain reST, no need to parse, but there
>> are markups in the (vintage) comments which made use of individual 
>> text-markups, e.g. the markup of lists or ASCII-art. 
>> 
>> This individual text-markups has not been converted/rendered in the docbook
>> output, but I wanted to convert this individuals to reST, to render them in
>> Sphinx.
> 
> I think we need to unconfuse what's current standardize kerneldoc markup.
> There's three bits:
> - The header with the one-liner and parameters, i.e.

OK, forget my  example, I don't really want to promote
this way ... I agree, it is better to stay with standardize kernel-doc markup
and provide a "pass through" for the section-content (even if it is mixing 
markups).

Am 04.05.2016 um 15:41 schrieb Jani Nikula :

> Kernel-doc the tool should continue to parse kernel-doc comments at the
> high level like they currently are, according to the kernel-doc
> howto. 

The requirement was unclear for me, thanks to Daniel and 
Jani for clarifying this.


Am 04.05.2016 um 15:43 schrieb Daniel Vetter :

> This is already used widely in kerneldoc included by gpu.tmpl, and
> currently it's asciidoc. Those lists and asciiarts though are _not_
> standard kerneldoc at all. But imo any doc toolchain improvement should
> integrate along those lines.
> 
> For reference, this is what it's supposed to look like with the asciidoc
> support enabled:
> 
> https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html
> 
> Fyi those hacked-up patches to make this happen are available in
> 
> https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc
> 

Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
particular with your "MARKDOWNREADY := gpu.xml" process.

As I understood, you convert markdown to docbook within the kernel-doc 
script using pandoc's executable? ... I don't face this topic. With my 
modification of kernel-doc I produced pure reST markup from standardize
kernel-doc markup, no intermediate steps or tools required.

Am 04.05.2016 um 15:41 schrieb Jani Nikula :

> Overall I think we should promote fixing those in the kernel-doc
> comments. Trying to guesswork in kernel-doc tool will leave the source
> littered with bad examples that get proliferated all around. Instead of
> gradually fixing things, we'll end up gradually messing things up even
> more.

I agree with you, lets drop the reformat_block_rst from my kernel-doc script:

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

and we should have a plain "pass through".

Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :

> I think all of this makes sense.  It would be really nice to have the
> directives in the native sphinx language like that.  I *don't* think we
> need to aim for that at the outset; the docproc approach works until we can
> properly get rid of it.  What would be *really* nice would be to get
> support for the kernel-doc directive into the sphinx upstream.

No need for kernel-doc directive in sphinx upstream, later it will be 
an extension which could be installed by a simple command like 
"pip install kernel-doc-extensions" or similar.

I develop these required extension (and more) within my proof of 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Jonathan Corbet  wrote:
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up.  It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)

Thanks. I'll start looking at how to make it less hackish and more
upstreamable.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote:
> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter  wrote:
> 
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.  
> > 
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...
> 
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home.  I hope to begin emerging from this mess in the
> near future.
> 
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done.  Which would you have me push into 4.7?
> 
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up.  It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)
> 
> In the end, I guess, I feel that anything we might try to push for 4.7 is
> going to look rushed and not ready, and Linus might react accordingly.
> I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
> on things in that time frame...  I suspect you're pretty well fed up with
> this stuff being pushed back, and rightly so.  All I can do is apologize.
> 
> That said, if you do think there's something out there that is good
> enough to consider pushing in a week or two, do tell and we can all take
> a look.

Well I'd just have taken the asciidoc hacks I have currently in my
topic/kerneldoc branch, converted to sphinx and looked how it fares. It
should be fairly minimal, and I think the first step we want to do for the
long-term plan. I hope I can ready something, and then we can look whether
it's rushed for 4.7 or not.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 16:18:27 +0200
Daniel Vetter  wrote:

> > I'd really like to converge on the markup question, so that we can start
> > using all the cool stuff with impunity in gpu documentations.  
> 
> Aside: If we decide this now I could send in a pull request for the
> rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> markdown/asciidoc code I still have). That would be really awesome ...

Sorry for my relative absence...I'm still busy dealing with bureaucracy
an ocean away from home.  I hope to begin emerging from this mess in the
near future.

So ... there's the code you have, the work I (+Jani) did, and the work
Markus has done.  Which would you have me push into 4.7?

The sphinx/rst approach does seem, to me, to be the right one, with the
existing DocBook structure remaining in place for those who want/need
it.  I'm inclined toward my stuff as a base to work with, obviously :) But
it's hackish at best and needs a lot of cleaning up.  It's a proof of
concept, but it's hardly finished (one might say it's barely begun...)

In the end, I guess, I feel that anything we might try to push for 4.7 is
going to look rushed and not ready, and Linus might react accordingly.
I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
on things in that time frame...  I suspect you're pretty well fed up with
this stuff being pushed back, and rightly so.  All I can do is apologize.

That said, if you do think there's something out there that is good
enough to consider pushing in a week or two, do tell and we can all take
a look.

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 4, 2016 at 3:43 PM, Daniel Vetter  wrote:
> I'd really like to converge on the markup question, so that we can start
> using all the cool stuff with impunity in gpu documentations.

Aside: If we decide this now I could send in a pull request for the
rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
markdown/asciidoc code I still have). That would be really awesome ...

Jon?

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Grant Likely]

On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  wrote:
>
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>> DocBook XML documentation to reST markup.
>>
>> It converts 99% of the docs well ... to gain an impression how
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>>
>>   http://return42.github.io/sphkerneldoc/
>
> So I've obviously been pretty quiet on this recently.  Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
>
> Looking quickly at this work, it seems similar to the results I got.  But
> there's a lot of code there that came from somewhere?  I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?
>
> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.

Hi Jon,

Thanks for digging into this. FWIW, here is my $0.02. I've been
working on restarting the devicetree specification, and after looking
at both reStructuredText and Asciidoc(tor) I thought I liked the
Asciidoc markup better, so chose that. I then proceeded to spend weeks
trying to get reasonable output from the toolchain. When I got fed up
and gave Sphinx a try, I was up and running with reasonable PDF and
HTML output in a day and a half.

Honestly, in the end I think we could make either tool do what is
needed of it. However, my impression after trying to do a document
that needs to have nice publishable output with both tools is that
Sphinx is easier to work with, simpler to extend, better supported. My
vote is firmly behind Sphinx/reStructuredText.

g.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Mon, 18 Apr 2016 10:10:17 +0200
Markus Heiser  escreveu:

> Hi Mauro, hi kernel-doc authors, 
> 
> Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab 
> :
> 
> > Em Fri, 8 Apr 2016 17:12:27 +0200
> > Markus Heiser  escreveu:
> >   
> >> Hi kernel-doc authors,
> >> 
> >> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> >> DocBook XML documentation to reST markup. 
> >> 
> >> It converts 99% of the docs well ... to gain an impression how 
> >> kernel-docs could benefit from, visit my sphkerneldoc project page
> >> on github:
> >> 
> >> http://return42.github.io/sphkerneldoc/
> >> 
> >> The sources available at:
> >> 
> >> https://github.com/return42/sphkerneldoc
> >> 
> >> The work is underway, suggestions are welcome!
> >> 
> >> .. have a nice weekend ..  
> > 
> > Hi Markus,
> > 
> > Thanks for your work. I'm basically worried about the media docbook,
> > with has some complexities that I was not able to figure out a way to
> > convert it to reST.
> > 
> > So, let me pinpoint some issues that I noticed there, on a quick
> > look.
> > 
> > The first thing I noticed is that the index doesn't match what's
> > there, when generated from DocBook:
> > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
> > 
> > So, for instance, "Interfaces" is at the same level as "Input/Output".
> > 
> > This sounds like an something went wrong when dealing with the title
> > indentation levels during the conversion.
> >   
> 
> Yes, the hierarchical structure was broken by two causes. First was, that the
> *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) 
> placed
> all sections and sub(-sub)-sections at the same level (direct under chapter). 
> This was
> not only broken in the linux_tv book, in the other books also.
> 
> Thanks for pointing and sorry that I have overlooked this (I was so much 
> focused on
> on converting single elements to reST, that I not see the wood for the trees).
> 
> It is now fixed, may you give it a new try.

Thanks! It looks good now.

> > I would also like to see the titles numbered in chapters (and, if
> > possible, in sections and items) and to be properly indented at the 
> > index.  
> 
> BTW a few words about differences between DockBook and reST (Sphinx).
> 
> With DocBook you write *books*, the protocol (the DocBook application) has
> no facility to *chunk* and interconnect several documents. The external 
> ENTITY 
> is a workaround on the SGML layer, not on XML nor on the DB-application layer.
> Thats the reason, why so many XML-tools don't handle this entities and
> many DocBook to (e.g.) reST tools are fail.
> 
> With **standard** reST it is nearly the same, except there is a "include"
> directive on the application layer. But this directive is very simple,
> comparable to the C preprocessor "#include" directive.
> 
> With the **superset** reST-markup of Sphinx-doc you get a the "toctree" 
> directive,
> which lets you control how a document-tree should be build.
> 
>  http://www.sphinx-doc.org/en/stable/markup/toctree.html
> 
> @Mauro: you mentioned a docutils (rst2*) experience in your mail 
>   http://marc.info/?l=linux-doc=145735316012094=2
> 
>   Because the "toctree" directive -- and other directives
>   we use -- are a part of a superset of the **standard** 
>   reST, the standard docutils (like rst2*) will not work.
> 
> OK, back to your requirements: within the toctree directive you can
> set options like "maxdepth" and "numbered". It is a decision, how
> deep TOCs should go and if they should be numbered. IMO, in a
> HTML rendered page, with a proper navigation bar on the side, deep 
> TOCs in the running text have no pros, they only blow up the running
> text and bring more scrolling with. In my sense numbering chapters
> make only sense in books, not in HTML pages, where you have hyperlinks.
> 
> Just for demonstration, I added numbering in the linux-tv book:
> 
> https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3
> 
> May be, I will drop it later, because all reST sources are generated
> by a make target and I always commit the whole reST tree. As I said, 
> it is a decision which might be made later, when the migration takes 
> places.

This is the uAPI spec DocBook, that we modify frequently, as we add
more features to the Kernel, and as we make sure that all drivers will
behave the same. So, from time to time, we need to clarify some topics
at the documentation. By having a numeration, it is easier for us to
discuss things like:
"1.2.10.14. V4L2_PIX_FMT_VYUY (‘VYUY’)" is not properly
described and requires some sort of clarification.

Ok, one could also refer to it via a hyperlink, but several Kernel
media maintainers prefer to generate a single big html file, as it
makes easier to locate everything it is needed on it.

So, with the item number, they can just seek 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jonahtan,

Am 12.04.2016 um 17:46 schrieb Jonathan Corbet :

> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  wrote:
> 
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
>> DocBook XML documentation to reST markup. 
>> 
>> It converts 99% of the docs well ... to gain an impression how 
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>> 
>>  http://return42.github.io/sphkerneldoc/
> 
> So I've obviously been pretty quiet on this recently.  Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
> 
> Looking quickly at this work, it seems similar to the results I got.  But
> there's a lot of code there that came from somewhere?

>From me? ... except the kernel-doc script which is a fork from your 

  git://git.lwn.net/linux.git doc/sphinx


>  I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?

It depends. If you have a simple DocBook with less various markup, maybe not.
May you want to read my remarks about migration tools and especially pandoc:

* 
https://return42.github.io/sphkerneldoc/articles/dbtools.html#remarks-on-pandoc

A few more words about, what I have done:

I wrote a lib of XML filters which might be also usefully in other
migration projects (dbxml). 

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py

It uses a xml-parser, pandoc, pandoc-filters and regular expressions. Because
I did not implemented a whole converter, I hacked around pandoc. Thats why
conversion is done in several steps:

1. copy xml file(s) to a cache space

2. substitude unsolved internal and external entities

3. filter all xml files

   * run custom hooks on every node 

   * apply filters on every node and inject reST into the XML-tree where pandoc 
fails.
 https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py#L515

4. convert intermediary XML result with pandoc to json (needed by pandoc 
filters)

5. apply pandoc-filter and clean up the injected reST markup from step3

6. convert filtered json to reST

7. fix the produce reST with regular expression

... the last step is similar to your sed scripts.


And I wrote a commandline Interface to use this lib (see func db2rst):

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L146
 
With this db2rst all kernel DB-XML books could be migrated, except the linux-tv
book, which has much more complexity. For this, there is a separated commandline
called media2rst

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L107

The media2rst needs several special handlings, which is implemented in 
hooks (the dbxml interface method)

* https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py

Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where 
  dbxml comes into play

  - reading DocBook: 

https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23
  
  - writing reST has many bugs and leaks 
(you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

* dbxml brings the ability to chunk one large XML book into small 
  reST chunks e.g. kernel-hacking book:


https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

  this might helpfull e.g. if you have to convert single-column informal-tables 
  to lists or other things ... in short; dbxml and it's hooks are the key to 
hack
  everything you need in a full automated DocBook-->reST migration workflow.


--Markus--

> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.
> 
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro, hi kernel-doc authors, 

Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab :

> Em Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  escreveu:
> 
>> Hi kernel-doc authors,
>> 
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
>> DocBook XML documentation to reST markup. 
>> 
>> It converts 99% of the docs well ... to gain an impression how 
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>> 
>> http://return42.github.io/sphkerneldoc/
>> 
>> The sources available at:
>> 
>> https://github.com/return42/sphkerneldoc
>> 
>> The work is underway, suggestions are welcome!
>> 
>> .. have a nice weekend ..
> 
> Hi Markus,
> 
> Thanks for your work. I'm basically worried about the media docbook,
> with has some complexities that I was not able to figure out a way to
> convert it to reST.
> 
> So, let me pinpoint some issues that I noticed there, on a quick
> look.
> 
> The first thing I noticed is that the index doesn't match what's
> there, when generated from DocBook:
>   https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
> 
> So, for instance, "Interfaces" is at the same level as "Input/Output".
> 
> This sounds like an something went wrong when dealing with the title
> indentation levels during the conversion.
> 

Yes, the hierarchical structure was broken by two causes. First was, that the
*chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) 
placed
all sections and sub(-sub)-sections at the same level (direct under chapter). 
This was
not only broken in the linux_tv book, in the other books also.

Thanks for pointing and sorry that I have overlooked this (I was so much 
focused on
on converting single elements to reST, that I not see the wood for the trees).

It is now fixed, may you give it a new try.


> I would also like to see the titles numbered in chapters (and, if
> possible, in sections and items) and to be properly indented at the 
> index.

BTW a few words about differences between DockBook and reST (Sphinx).

With DocBook you write *books*, the protocol (the DocBook application) has
no facility to *chunk* and interconnect several documents. The external ENTITY 
is a workaround on the SGML layer, not on XML nor on the DB-application layer.
Thats the reason, why so many XML-tools don't handle this entities and
many DocBook to (e.g.) reST tools are fail.

With **standard** reST it is nearly the same, except there is a "include"
directive on the application layer. But this directive is very simple,
comparable to the C preprocessor "#include" directive.

With the **superset** reST-markup of Sphinx-doc you get a the "toctree" 
directive,
which lets you control how a document-tree should be build.

 http://www.sphinx-doc.org/en/stable/markup/toctree.html

@Mauro: you mentioned a docutils (rst2*) experience in your mail 
  http://marc.info/?l=linux-doc=145735316012094=2

  Because the "toctree" directive -- and other directives
  we use -- are a part of a superset of the **standard** 
  reST, the standard docutils (like rst2*) will not work.

OK, back to your requirements: within the toctree directive you can
set options like "maxdepth" and "numbered". It is a decision, how
deep TOCs should go and if they should be numbered. IMO, in a
HTML rendered page, with a proper navigation bar on the side, deep 
TOCs in the running text have no pros, they only blow up the running
text and bring more scrolling with. In my sense numbering chapters
make only sense in books, not in HTML pages, where you have hyperlinks.

Just for demonstration, I added numbering in the linux-tv book:

https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3

May be, I will drop it later, because all reST sources are generated
by a make target and I always commit the whole reST tree. As I said, 
it is a decision which might be made later, when the migration takes 
places.

> Also, it seems that there's still a lot of work to do, as there are several
> tables that are missing conversion, like the table for "struct 
> v4l2_pix_format":
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html
> 
> and the big tables at:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 

Yes, I marked them as TODO: 

 https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262

aspect *authoring tables* (see below)

> Also, some tables that are not so big like:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> 
> don't look nice, at least on my browser, as the "white" area is too small,
> and some cells seem to be broken in 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi kernel-doc authors,

motivated by this MT, I implemented a toolchain to migrate the kernel’s 
DocBook XML documentation to reST markup. 

It converts 99% of the docs well ... to gain an impression how 
kernel-docs could benefit from, visit my sphkerneldoc project page
on github:

  http://return42.github.io/sphkerneldoc/

The sources available at:

  https://github.com/return42/sphkerneldoc

The work is underway, suggestions are welcome!

.. have a nice weekend ..

--M--


Am 13.03.2016 um 16:33 schrieb Markus Heiser :

> 
> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab 
> :
> 
>> Em Thu, 10 Mar 2016 12:25:58 +0200
>> Jani Nikula  escreveu:
>> 
>>> TL;DR? Skip to the last paragraph.
>>> 
>>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
 I guess the conversion to asciidoc format is now in good shape,
 at least to demonstrate that it is possible to use this format for the
 media docbook. Still, there are lots of broken references.  
>>> 
>>> Getting references right with asciidoc is a big problem in the
>>> kernel-doc side. As I wrote before, the proofs of concept only worked
>>> because everything was processed as one big file (via includes). The
>>> Asciidoctor inter-document references won't help, because we won't know
>>> the target document name while processing kernel-doc.
>> 
>> I was able to produce chunked htmls here with:
>> 
>>  asciidoctor -b docbook45 media_api.adoc
>>  xmlto -o html-dir html media_api.xml
>> 
>> The results are at:
>>  
>> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
>> 
>> But yeah, all references seem to be broken there. It could be due to some
>> conversion issue (I didn't actually tried to check what's wrong there),
>> but I think that there's something not ok with docbook45
>> output for multi-part documents (on both AsciiDoc and Asciidoctor).
>> 
>>> Sphinx is massively better at handling cross references for
>>> kernel-doc. We can use domains (C language) and roles (e.g. functions,
>>> types, etc.) for the references, which provide kind of
>>> namespaces. Sphinx warns for referencing non-existing targets, but
>>> doesn't generate broken links in the result like Asciidoctor does.
>>> 
>>> For example, in the documentation for a function that has struct foo as
>>> parameter or return type, a cross reference to struct foo is added
>>> automagically, but only if documentation for struct foo actually
>>> exists. In Asciidoctor, we would have to blindly generate the references
>>> ourselves, and try to resolve broken links ourselves by somehow
>>> post-processing the result.
>>> 
 Yet, from my side, if we're willing to get rid of DocBook, then
 Asciidoctor seems to be the *only* alternative so far to parse the
 complex media documents.  
>>> 
>>> I think you mean, "get rid of DocBook as source format", not altogether?
>>> I'm yet to be convinved we could rely on Asciidoctor's native formats.
>> 
>> What I mean is that, right now, I see only two alternatives for the
>> media uAPI documentation:
>>  1) keep using DocBook;
>>  2) AsciiDoc/Asciidoctor.
>> 
>> Sphinx doesn't have what's needed to support the complexity of the
>> media books, specially since cell span seems to be possible only
>> by using asciiArt formats. Writing a big table using asciiArt is
>> something that is a *real pain*. Also, as tested, if the table is
>> too big, it fails to parse such asciiArt tables. So, while Sphinx
>> doesn't have a decent way to describe tables, we can't use it.
> 
> 
> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
> (mostly) you have more then one choice .. e.g. import csv tables .. 
> but this should be discussed by example ...
> 
> 
>> If it starts implementing it, then we can check if the other
>> features used by the media documentation are also supported.
>> Probably, multi-part books would be another pain with Sphinx.
>> We have actually 4 books inside a common body. A few chapters
>> (like book licensing, bibliography, error codes) are shared
>> by all 4 documents.
>> 
>> But, so far, I can't see any way to port media books without
>> lots of lot of work to develop new features at the Sphinx code.
> 
> 
> may I can help you ...
> 
> 
>>> The toolchain gets faster, easier to debug and simplified a lot with
>>> DocBook out of the equation completely. Sphinx itself is stable, widely
>>> available, and well documented. IMO there's sufficient native output
>>> format support. There are plenty of really nice extensions
>>> available. There's a possibility of doing kernel-doc as an extension in
>>> the future (either by calling current kernel-doc from the extension or
>>> by rewriting it).
>> 
>> Well, if we go to Sphinx for kernel-doc, that means that we'll need
>> 2 different tools for the documentation:
>>  - Sphinx for kernel-doc
>>  - either DocBook or 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 12:39:21 -0300
Mauro Carvalho Chehab  escreveu:

> Pandoc failed to fully convert it, but at least it left all the texts,
> with prevented rewriting it from scratch. This is the manual fix
> I applied to it:
>   
> https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc
> 
> FYI, we have 133 xml files at the media uAPI doc with refmeta.

I used pandoc to convert from the html files and manually edited it.
I also fixed lots of other issues with the conversion.

I guess the conversion to asciidoc format is now in good shape,
at least to demonstrate that it is possible to use this format for the
media docbook. Still, there are lots of broken references.

The proof of concept html file is at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

I also added the ascii doc files there, at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/   

And I'm keeping the git tree, with helps to identify the work that was
needed to make it work:
https://git.linuxtv.org/mchehab/asciidoc-poc.git

In summary, AsciiDoc, formatted via AsciiDoctor worked fine to produce
an html file.

PROBLEMS


1)

I was not able to produce outputs on any other format.

For example, when trying to generate docbook45 output, it sems that
part of the trouble was due to pandoc conversion. It produces
links like:

link:#ftn.id-1.4.11.43.5.11.2.7.2.6.2[^[a]^]

Which causes errors with DocBook parsers, like xmllint:

media_api.xml:32300: parser error : Opening and ending tag mismatch: 
superscript line 32300 and ulink
[a]
 ^

I suspect that this is fixable. I may try to fix it later.

2) It seems that Asciidoctor doesn't allow annexes per document part.
It numberates them as chapters, instead of using A, B, C, ...

3) Even producing the html without troubles, it produces an error:
asciidoctor: ERROR: media_api.adoc: line 57: invalid part, must have at 
least one section (e.g., chapter, appendix, etc.)

4) There are some things that got lost during the conversion, like
copyright notes and revision notes. This could be simply a problem
with pandoc conversion. Nothing serious, I guess, as we could insert
the lost data manually. Yet, it means that, to move from the PoC to
the Kernel, there are still lots of work to do.

I was unable do discover why, nor to suppress this error message.

Yet, from my side, if we're willing to get rid of DocBook, then
Asciidoctor seems to be the *only* alternative so far to parse the
complex media documents.

Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 09 Mar 2016, Dan Allen  wrote:
> On Tue, Mar 8, 2016 at 6:58 AM, Jani Nikula  wrote:
>
>> I need to look into this again. Is there a specific option or directive
>> to produce split output for includes? When I tried this, the result was
>> just one big output file. (And indeed we'd need both. Some includes we
>> want embedded, some includes should produce separate outputs.)
>>
>
> Nope. What I'm saying is that you run Asciidoctor on each sub-master
> include file (an include that manages a part or chapter). That gives you
> your individual part/chapter files. Then you need to make an index page,
> probably by using the Asciidoctor API to itemize all the chapters as a list
> or something.

Bummer.

Getting the inter-document cross references right may become
tricky. We'll be generating plenty of snippets of lightweight markup
from source code documentation comments. At the time of processing, we
won't know where e.g. a specific function to be cross referenced is
documented, if at all. We can't require the documentation comment
writers to figure that out either; it's too burdensome, too ugly in the
code, and they'll bitrot quickly.

Cross referencing in the asciidoc proofs of concept have worked because
they've all done the processing as a single single unit, with
includes. These hacks have also ignored any broken links, and there have
been

> Yes, it does require some thinking about cross references. There is a lot
> more we can do out of the box, but all those references can be fixed with a
> little bit of post-processing in the meantime.

It seems to me Sphinx provides much better support regarding cross
references, out of the box, within documents and to external documents
(intersphinx), with target roles and domains, including validation and
not creating broken links in the output.

Looking at the current hacks we have for post-processing references, I'm
really not thrilled about the prospect of keeping or redoing that.

See how this works in Jon's Sphinx test [1]. At the time of generating
the markup from source comments, there is no idea if and where
gem_init_hw() and intel_guc_ucode_init() are documented. Indeed,
documentation for the former does not exist, but there's no broken link.

BR,
Jani.


[1] http://static.lwn.net/kerneldoc/gpu.html#c.intel_guc_ucode_load


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 10:39:22 -0300
Mauro Carvalho Chehab  escreveu:

> Em Tue, 08 Mar 2016 05:13:13 -0700
> Dan Allen  escreveu:
> 
> > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <  
> > mche...@osg.samsung.com> wrote:
> >   
> > > pandoc did a really crap job on the conversion. To convert this
> > > into something useful, we'll need to spend a lot of time, as it lost
> > > most of the cross-references, as they were defined via DocBook macros.
> > >
> > 
> > I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> > the works called DocBookRx.
> > 
> > https://github.com/opendevise/docbookrx
> > 
> > It has converted several very serious DocBook documents and we're
> > continuing to improve it. It's also a lot easier to hack than pandoc.  
> 
> Didn't work:
> 
> $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml 
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.

I tried to use docbookrx for the bits that were not properly converted,
like the manpage-like pages:

$  ../docbookrx/bin/docbookrx Documentation/DocBook/media/v4l/func-ioctl.xml
No visitor defined for ! Skipping.

Dan, if you want to take a look on what's going wrong here, 
the XML I'm trying to convert is:


https://git.linuxtv.org/media_tree.git/tree/Documentation/DocBook/media/v4l/func-ioctl.xml

If this would work, it should be generating something like:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/tree/func-ioctl.adoc

Pandoc failed to fully convert it, but at least it left all the texts,
with prevented rewriting it from scratch. This is the manual fix
I applied to it:

https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc

FYI, we have 133 xml files at the media uAPI doc with refmeta.

-- 
Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Tue, 08 Mar 2016, Dan Allen  wrote:
> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

I need to look into this again. Is there a specific option or directive
to produce split output for includes? When I tried this, the result was
just one big output file. (And indeed we'd need both. Some includes we
want embedded, some includes should produce separate outputs.)

>> That actually makes choosing asciidoc harder, because
>> requiring another language environment complicates, not simplifies, the
>> toolchain. I'd really like to lower the bar for building the
>> documentation, for everyone, so much so that it becomes part of the
>> normal checks for patch inclusion.
>
> Pardon my bluntness here, but I don't buy that argument. This is Linux.
> Installing software couldn't be simpler, and we're talking about an
> extremely well supported language (Ruby).

Granted, that part works for me. I'm not so sensitive to the
dependencies; others may disagree.

> I think it's a huge exaggeration to say that Asciidoctor is any harder to
> install than AsciiDoc Python. It's also a heck of a lot smaller in size
> since AsciiDoc Python pulls in hundreds of MB of LaTeX packages.

For me, the comparison is really between Sphinx and Asciidoctor, not so
much doc vs. doctor. The native output format and extension support in
Sphinx is appealing; I am not yet convinced we could manage with
Asciidoctor but without DocBook. The extension offering seems better in
Sphinx.

> Whatever you decide, I wish you all the best with your documentation
> efforts!

Thanks!

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:13:13 -0700
Dan Allen  escreveu:

> On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <
> mche...@osg.samsung.com> wrote:  
> 
> > pandoc did a really crap job on the conversion. To convert this
> > into something useful, we'll need to spend a lot of time, as it lost
> > most of the cross-references, as they were defined via DocBook macros.
> >  
> 
> I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> the works called DocBookRx.
> 
> https://github.com/opendevise/docbookrx
> 
> It has converted several very serious DocBook documents and we're
> continuing to improve it. It's also a lot easier to hack than pandoc.

Didn't work:

$ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml 
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.


> 
> -Dan
> 
> 


-- 
Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:09:40 -0700
Dan Allen  escreveu:

> Jani wrote:
> 
> > there was no support for chunked, or split
> > to chapters, HTML, and the single page result was simply way too big.
> >  
> 
> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

The problem on pre-splitting the documents and process them in separate
is that this will break cross-references. At the media uAPI Docbook,
we use a lot of cross references.

Btw, we use a lot of includes. Currently, it has 187 separate files.

We even parse the header files looking for typedefs, structs, enums,
#defines and functions, in order to produce a document that will
cross-reference the documentation.

> Second, there is a custom converter in the works to split post-generate
> (which is really what we're talking about when we compare it to the DocBook
> toolchain).
> 
> https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb
> 
> It's just a prototype, but proves it is possible by design.

I didn't test it, but I saw some comments at the web that the part
that would handle cross-references between files is not ready.

> Personally, I don't like most chunked HTML approaches because they split
> arbitrarily. We are trying to find the right balance so that the output is
> actually sensible. There's still work to do, but there are options in the
> meantime.

Well, if it is capable of creating one chunk per include file, and do
cross-references between chunks, this would work for media UAPI book.

Yet, it would be good to have the multi-chunk extension packaged
on major distros, as I don't like the idea of installing it without
using my distro's package manager.

-- 
Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 11:49:35 +0200
Jani Nikula  escreveu:

> On Tue, 08 Mar 2016, Dan Allen  wrote:
> > One of the key goals of the Asciidoctor project is to be able to directly
> > produce a wide variety of outputs from the same source (without DocBook).
> > We've added flexibility and best practices into the syntax and matured the
> > converter mechanism to bridge this (sometimes very wide) gap.  
> 
> I think our conclusion so far was that the native AsciiDoc (and
> Asciidoctor) outputs fell short of our needs, forcing us to use the
> DocBook pipeline. I, for one, was hoping we could eventually simplify
> the toolchain. For example, there was no support for chunked, or split
> to chapters, HTML, and the single page result was simply way too big.
> 
> > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers
> > acknowledge that (including the original creator).  
> 
> Thanks for the input. We've touched the topic of AsciiDoc
> vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of
> AsciiDoc. That actually makes choosing asciidoc harder, because
> requiring another language environment complicates, not simplifies, the
> toolchain. I'd really like to lower the bar for building the
> documentation, for everyone, so much so that it becomes part of the
> normal checks for patch inclusion.

As I failed to find a way to solve the issues with Sphinx/RST, I started
a PoC using Asciidoctor for the Media uAPI docbook conversion, in order
to see if this would work for us.

I'm not a big fan of using a language that I don't domain, like
Ruby (and the same applies to Python), but, as I said before,
provided that the toolchain works, can easily be installed on
Fedora and Debian, and provide the functionality I need, I'm
ok with that. 

On my tests, Asciidoctor is really fast. It takes a fraction of the
time require to build from DocBook, with is a good thing.

Yet, I suspect that it doesn't have the strict checks that xmllint
have. For example, I didn't see any warning about a missing cross-ref.
We use those warnings to discover if something is added at the code,
but were not documented. Its error reports are also crappy, as it
doesn't tell where the problem is. For example:

$ asciidoctor -n -b docbook media_api.adoc
asciidoctor: WARNING: tables must have at least one body row

On a document with 33793 lines (this is the size of the uAPI doc
when converted to AsciiDoc format), the above warning doesn't help.

So, I'm actually planning to use AsciiDoc/xmllint to check for
documentation troubles, if possible.

Even with AsciiDoc/Asciidoctor, there are several features at the
media documents that aren't well supported. For example, we widely
use the Docbook's tags to generate a manpage-like description:


  
V4L2 open()

  

  
v4l2-open
Open a V4L2 device
  

  

  #include fcntl.h
  
int open
const char *device_name
int flags
  

  

Asciidoctor doesn't produce anything like that, if the booktype is not
"manpage". Well, the media documentation is a multi-part book.

So, all those tags should be manually converted (actually, pandoc
made a mess with those tags - so manual work is required anyway,
no matter what markdown language we use). It will be a hard work,
but, at least, this is doable. 

Also, it doesn't accept images with PDF format. We have a few PDF
images, although I we have them also in GIF. So, not a big issue here.

I also did lots of table conversions, to see if our complex tables
would fit. The answer is: Yes: on all cases I converted, it worked
fine.

There are two ways of doing that:
- Nested tables
- Merging cells

Some tables we use won't work fine with nested tables, as they have
cells merging the entire line of such tables.

Also, converting some tables to nested tables would be incredible hard,
as we have really big tables there that would need to add dozens of nested
tables inside, like this one:


https://linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html#id-1.4.4.14.6.4

Using merging cells works fine, tough, and it is not hard to use.
The tables are easy to edit. The Asciidoctor syntax (not sure if AsciiDoc
also accepts this) is:

.MFC 5.1 Control IDs
[width="100%",cols="7%,40%,13%,40%",options="header",]
|===
|ID |Type 2+| Description

...

The "2+|" tells that the next content should be merged into two cells.

One of the good things is that we're not forced to use asciiart, with
would make it really hard to handle the tables.

For those wanting to see the results so far:

The html is at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

The testing tree is at:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/

PS.: pandoc did a really crap job on the conversion. To convert this

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Tue, 08 Mar 2016, Dan Allen  wrote:
> One of the key goals of the Asciidoctor project is to be able to directly
> produce a wide variety of outputs from the same source (without DocBook).
> We've added flexibility and best practices into the syntax and matured the
> converter mechanism to bridge this (sometimes very wide) gap.

I think our conclusion so far was that the native AsciiDoc (and
Asciidoctor) outputs fell short of our needs, forcing us to use the
DocBook pipeline. I, for one, was hoping we could eventually simplify
the toolchain. For example, there was no support for chunked, or split
to chapters, HTML, and the single page result was simply way too big.

> Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers
> acknowledge that (including the original creator).

Thanks for the input. We've touched the topic of AsciiDoc
vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of
AsciiDoc. That actually makes choosing asciidoc harder, because
requiring another language environment complicates, not simplifies, the
toolchain. I'd really like to lower the bar for building the
documentation, for everyone, so much so that it becomes part of the
normal checks for patch inclusion.


BR,
Jani.


[1] http://mid.gmane.org/86pow31ddj@hiro.keithp.com


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Russel Winder]

On Fri, 2016-03-04 at 09:46 +0200, Jani Nikula wrote:
> […]
> If we're talking about the same asciidoctor (http://asciidoctor.org/)
> it's written in ruby but you can apparently run it in JVM using
> JRuby. Calling it Java-based is misleading.

Indeed, I was somewhat imprecise. Thanks to the work mostly of Charles
Nutter, JRuby is invariably a faster platform for Ruby code than Ruby
is. So yes ASCIIDoctor is JVM-based via JRuby, not Java-based.

The real point here is that in a move from DocBook/XML as a
documentation source, ASCIIDoctor is an excellent choice.

-- 
Russel.=Dr
 Russel Winder  t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net41 
Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@winder.org.ukLondon SW11 
1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Mon, 7 Mar 2016 00:29:08 +0100
Johannes Stezenbach  escreveu:

> On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> > 
> > I converted one of the big tables to CSV. At least now it recognized
> > it as a table. Yet, the table was very badly formated:
> > 
> > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > 
> > This is how this table should look like:
> > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > 
> > Also, as this table has merged cells at the legend. I've no idea how
> > to tell sphinx to do that on csv format.
> > 
> > The RST files are on this git tree:
> > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  
> 
> Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> back to grid table format and fixes the table definition.
> The html output looks usable, but clearly it is no fun to
> work with tables in Sphinx.

Yes, the output is OK, but, as you said, working with tables in
Sphinx is hard, and using asciiart for the kind of tables we have
is not nice.

> 
> Sphinx' latex writer can't handle nested tables, though.

Yeah, this is a big trouble that need to be solved if you're
willing to use Sphinx.

Btw, it crashes when trying to generate man pages:

Exception occurred:
  File "/usr/lib/python2.7/site-packages/docutils/writers/manpage.py", 
line 627, in depart_entry
self._active_table.append_cell(self.body[start:])
AttributeError: 'NoneType' object has no attribute 'append_cell'
The full traceback has been saved in /tmp/sphinx-err-04qRMz.log, if you 
want to report the issue to the developers.

So, if we're willing to use sphinx, someone should either fix
it to produce latex nexted table and fix it to generate manpages,
or we'll need to stick with just html output.

> Python's docutils rst2latex can, but that doesn't help here.
> rst2pdf also supports it.

At least here, rst2* scripts were unable to identify that the
index.rst had links to other rst documents. 

In the specific case of rst2latex, I got several errors like:

index.rst:21: (ERROR/3) Unknown interpreted text role "ref".


> But I have doubts such a large
> table would render OK in pdf without using landscape orientation.

Yeah, in the past, when we had pdf enabled for DocBook (e. g. when
media development was using a separate mercurial tree), I guess
we had tags changing the text orientation on a few tables that
would otherwise won't diplay fine, but I can't remember the dirty
details anymore.

> I have not tried because I used python3-sphinx but rst2pdf
> is only availble for Python2 in Debian so it does not integrate
> with Sphinx.
> 
> 
> Johannes


-- 
Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Mon, 7 Mar 2016 09:48:26 +0100
Johannes Stezenbach  escreveu:

> On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote:
> > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:  
> > > 
> > > I converted one of the big tables to CSV. At least now it recognized
> > > it as a table. Yet, the table was very badly formated:
> > >   
> > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > > 
> > > This is how this table should look like:
> > >   https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > > 
> > > Also, as this table has merged cells at the legend. I've no idea how
> > > to tell sphinx to do that on csv format.
> > > 
> > > The RST files are on this git tree:
> > >   https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  
> > 
> > Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> > back to grid table format and fixes the table definition.
> > The html output looks usable, but clearly it is no fun to
> > work with tables in Sphinx.
> > 
> > Sphinx' latex writer can't handle nested tables, though.
> > Python's docutils rst2latex can, but that doesn't help here.
> > rst2pdf also supports it.  But I have doubts such a large
> > table would render OK in pdf without using landscape orientation.
> > I have not tried because I used python3-sphinx but rst2pdf
> > is only availble for Python2 in Debian so it does not integrate
> > with Sphinx.  
> 
> Just a quick idea:
> Perhaps one alternative would be to use Graphviz to render
> the problematic tables, it supports a HTML-like syntax
> and can be embedded in Spinx documents:
> 
> http://www.sphinx-doc.org/en/stable/ext/graphviz.html
> http://www.graphviz.org/content/node-shapes#html
> http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables

That could work, but it is scary... Graphviz is great to generate
diagrams, but it really sucks when one wants to put a graph element
on a specific place, as it loves to reorder elements putting them on
unexpected places.

Btw, 

I converted all docs from our uAPI docbook to rst using pandoc.
It was a brainless conversion, except for a few fixes.

The output is at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/

I added it on the top of my PoC tree at:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  

Besides tables, I noticed some other bad things that needs to be
corrected somehow:

1) Document divisions are not numbered. We need that. It should be
broken into:
- Document divisions - one per documented API:
- V4L2
- Remote Controllers
- DVB
- Media Controller

- Chapters
- Sessions

Everything should be numbered, as, when discussing API improvements,
it is usual the need of pinpoint to an specific chapter and section.

Tables and images should also be numbered, and we need a way to
use references for table/image numbers.

2) Images

Most images didn't popup. We have images on different file formats:
- SVG
- GIF
- PDF
- PNG

3) References

It could be a conversion issue, but there are lots of missing 
references at the documentation.

4) We need to have some way to tell sphinx to not put some things
at the lateral ToC bar. For example, at the V4L2 "Changes" section,
we don't want to have one entry per version at the ToC bar.

Giving that, I suspect that we'll have huge headaches to address
if we use sphinx, as it seems too limited to handle complex
documents. We should try to use some other tool that would give
us better results.


Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Johannes Stezenbach]

On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> 
> I converted one of the big tables to CSV. At least now it recognized
> it as a table. Yet, the table was very badly formated:
>   
> https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> 
> This is how this table should look like:
>   https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> 
> Also, as this table has merged cells at the legend. I've no idea how
> to tell sphinx to do that on csv format.
> 
> The RST files are on this git tree:
>   https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Yeah, seems it can't do merged cells in csv.  Attached patch converts it
back to grid table format and fixes the table definition.
The html output looks usable, but clearly it is no fun to
work with tables in Sphinx.

Sphinx' latex writer can't handle nested tables, though.
Python's docutils rst2latex can, but that doesn't help here.
rst2pdf also supports it.  But I have doubts such a large
table would render OK in pdf without using landscape orientation.
I have not tried because I used python3-sphinx but rst2pdf
is only availble for Python2 in Debian so it does not integrate
with Sphinx.


Johannes
>From 61674b398e778bd5ff644ffd493d5ff1cfaca0ef Mon Sep 17 00:00:00 2001
From: Johannes Stezenbach 
Date: Sun, 6 Mar 2016 23:55:19 +0100
Subject: [PATCH] some progress for html output

---
 _static/borderless.css |  3 --
 _static/v4l2tables.css |  9 +
 _templates/layout.html |  9 +
 packed-rgb.rst | 88 +-
 pixfmt-yuyv.rst|  2 +-
 v4l-table-within-table.rst | 72 +++--
 6 files changed, 105 insertions(+), 78 deletions(-)
 delete mode 100644 _static/borderless.css
 create mode 100644 _static/v4l2tables.css

diff --git a/_static/borderless.css b/_static/borderless.css
deleted file mode 100644
index bfd4b01..000
--- a/_static/borderless.css
+++ /dev/null
@@ -1,3 +0,0 @@
-table#table-borderless {
-border: 1px solid black;
-}
diff --git a/_static/v4l2tables.css b/_static/v4l2tables.css
new file mode 100644
index 000..c045e45
--- /dev/null
+++ b/_static/v4l2tables.css
@@ -0,0 +1,9 @@
+table.noborder {
+border: 1px solid black;
+background: white;
+white-space: nowrap;
+}
+
+table.noborder td, table.noborder th {
+padding: 0px;
+}
diff --git a/_templates/layout.html b/_templates/layout.html
index b6bf12b..637332d 100644
--- a/_templates/layout.html
+++ b/_templates/layout.html
@@ -1,9 +1,2 @@
 {% extends "!layout.html" %}
-{% block tables %}
-
-table#table-borderless {
-border: 1px solid black;
-}
-
-{{ super() }}
-{% endblock %}
+{% set css_files = css_files + ["_static/v4l2tables.css"] %}
diff --git a/packed-rgb.rst b/packed-rgb.rst
index 352b91c..b4fcf3e 100644
--- a/packed-rgb.rst
+++ b/packed-rgb.rst
@@ -9,25 +9,46 @@ graphics frame buffers. They occupy 8, 16, 24 or 32 bits per pixel.
 These are all packed-pixel formats, meaning all the data for a pixel lie
 next to each other in memory.
 
-.. csv-table:: Table: Packed RGB Image Formats
-  :header: Identifier,Code, ,Byte 0 in memory,Byte 1,Byte 2,Byte 3
+.. table:: Packed RGB Image Formats
+   :class: noborder
 
-  ``V4L2_PIX_FMT_RGB332``,'RGB1',,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`1`,b\ :sub:`0`
-  ``V4L2_PIX_FMT_ARGB444``,'AR12',,g\ :sub:`3`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,a\ :sub:`3`,a\ :sub:`2`,a\ :sub:`1`,a\ :sub:`0`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`
-  ``V4L2_PIX_FMT_XRGB444``,'XR12',,g\ :sub:`3`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,-,-,-,-,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`
-  ``V4L2_PIX_FMT_ARGB555``,'AR15',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,a,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`
-  ``V4L2_PIX_FMT_XRGB555``,'XR15',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,-,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`
-  ``V4L2_PIX_FMT_RGB565``,'RGBP',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`5`,g\ :sub:`4`,g\ :sub:`3`
-  ``V4L2_PIX_FMT_ARGB555X``,'AR15' | (1<<31),,a,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`,,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`
-  ``V4L2_PIX_FMT_XRGB555X``,'XR15' | (1<<31),,-,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`,,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`
-  

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 04 Mar 2016 15:09:09 +0100
Johannes Stezenbach  escreveu:

> On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
> > 
> > 3) I tried to use a .. cssclass, as Johannes suggested, but
> > I was not able to include the CSS file. I suspect that this is
> > easy to fix, but I want to see if the cssclass will also work for
> > the pdf output as well.  
> 
> "cssclass" was (I think) a custom role defined in the example,
> unless you also have defined a custom role you can use plain "class".
> I have not looked deeper into the theming and template stuff.

Well, it accepted cssclass for html (well, it didn't find the
templates - so I guess it is just me failing to understand how to tell
sphinx to get the stylesheet), but it rejects it for latexPDF.

> 
> > 4) It seems that it can't produce nested tables in pdf:
> > 
> > Markup is unsupported in LaTeX:
> > v4l-table-within-table:: nested tables are not yet implemented.
> > Makefile:115: recipe for target 'latexpdf' failed  
> 
> This:
> http://www.sphinx-doc.org/en/stable/markup/misc.html#tables
> 
> suggests you need to add the tabularcolumns directive
> for complex tables.
> 
> BTW, as an alternative to the ASCII-art input
> there is also support for CSV and list tables:
> http://docutils.sourceforge.net/docs/ref/rst/directives.html#table

I converted one of the big tables to CSV. At least now it recognized
it as a table. Yet, the table was very badly formated:

https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html

This is how this table should look like:
https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html

Also, as this table has merged cells at the legend. I've no idea how
to tell sphinx to do that on csv format.

The RST files are on this git tree:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Johannes Stezenbach]

On Fri, Mar 04, 2016 at 10:29:08AM +0200, Jani Nikula wrote:
> On Fri, 04 Mar 2016, Mauro Carvalho Chehab  wrote:
> >
> > If, on the other hand, we decide to use RST, we'll very likely need to
> > patch it to fulfill our needs in order to add proper table support.
> > I've no idea how easy/difficult would be to do that, nor if Sphinx
> > upstream would accept such changes.
> >
> > So, at the end of the day, we may end by having to carry on our own
> > version of Sphinx inside our tree, with doesn't sound good, specially
> > since it is not just a script, but a package with hundreds of
> > files.
> 
> If we end up having to modify Sphinx, it has a powerful extension
> mechanism for this. We wouldn't have to worry about getting it merged to
> Sphinx upstream, and we wouldn't have to carry a local version of all of
> Sphinx. (In fact, the extension mechanism provides a future path for
> doing kernel-doc within Sphinx instead of as a preprocessing step.)
> 
> I know none of this alleviates your concerns with table supports right
> now. I'll try to have a look at that a bit more.

FWIW, I think table formatting in Sphinx works via style sheets.
The mechanism is documented in the Python docutils docs that
Sphinx is built upon.
Basically you use the "class" or "role" directive and define
the corresponding CSS or LaTeX (or rst2pdf) style.

Here is one example (using a custom "cssclass" role):
https://pythonhosted.org/sphinxjp.themes.basicstrap/sample.html

Directives (especially role and class):
http://www.sphinx-doc.org/en/stable/rest.html#directives

LaTeX styling:
http://docutils.readthedocs.org/en/sphinx-docs/user/latex.html#custom-interpreted-text-roles


HTH,
Johannes
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 04 Mar 2016, Mauro Carvalho Chehab  wrote:
> Em Thu, 03 Mar 2016 15:23:23 -0800
> Keith Packard  escreveu:
>
>> Mauro Carvalho Chehab  writes:
>> 
>> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
>> > produced an output that worked better.  
>> 
>> Yes, asciidoc has much more flexibility in table formatting, including
>> the ability to control text layout within cells and full control over
>> borders.
>> 
>> However, I think asciidoc has two serious problems:
>> 
>>   1) the python version (asciidoc) appears to have been abandoned in
>>  favor of the ruby version. 
>> 
>>   2) It really is just a docbook pre-processor. Native html/latex output
>>  is poorly supported at best, and exposes only a small subset of the
>>  full capabilities of the input language.
>> 
>> As such, we would have to commit to using the ruby version and either
>> committing to fixing the native html output backend or continuing to use
>> the rest of the docbook toolchain.
>> 
>> We could insist on using the python version, of course. I spent a bit of
>> time hacking that up to add 'real' support for a table-of-contents in
>> the native HTML backend and it looks like getting those changes
>> upstreamed would be reasonably straightforward. However, we'd end up
>> 'owning' the code, and I'm not sure we want to.
>
> I'm a way more concerned about using a tool that fulfill our needs
> than to look for something that won't use the docbook toolchain or
> require to install ruby.

I think you meant that to be the other way round, or I fail at parsing
you. ;)

> In the case of Docbook, we know it works and we know already its
> issues. Please correct me if I'm wrong, but the big problem we
> have is not due to the DocBook toolchain, but due to the lack of
> features at the kernel-doc script. Also, xmlto is already installed
> by the ones that build the kernel docs. So, keeping use it won't
> require to install a weird toolchain by hand.

I think kernel-doc is just a small part of the puzzle. It's a problem,
but a small one at that, and we've already made it output asciidoc, rst
and docbook as part of this exercise. For real, as in code, not as in
talk.

The reasons I'm involved in this is that I want to make writing
documentation and rich kernel-doc comments easier (using lightweight
markup) and I want to make building the documentation easier (using a
straightforward toolchain with not too many dependencies). I'm hoping
the former makes writing documentation more attractive and the latter
keeps the documentation and the toolchain in a better shape through
having more people actually build the documentation.

IMHO docbook is problematic because the toolchain gets too long and
fragile. You need plenty of tools installed to build the documentation,
it's fussy to get working, and people just won't. Like code,
documentation bitrots too when it's not used. The documentation build is
broken too often. Debugging formatting issues through the entire
pipeline gets hard; I already faced some of this when playing with the
kernel-doc->asciidoc->docbook->html chain.

In short, I don't think the docbook toolchain fills all of our needs
either.

> So, to be frank, it doesn't scary me to use either pyhton or
> ruby script + docbook.
>
> Of course, having to own the code has a cost that should be evaluated.
>
> If, on the other hand, we decide to use RST, we'll very likely need to
> patch it to fulfill our needs in order to add proper table support.
> I've no idea how easy/difficult would be to do that, nor if Sphinx
> upstream would accept such changes.
>
> So, at the end of the day, we may end by having to carry on our own
> version of Sphinx inside our tree, with doesn't sound good, specially
> since it is not just a script, but a package with hundreds of
> files.

If we end up having to modify Sphinx, it has a powerful extension
mechanism for this. We wouldn't have to worry about getting it merged to
Sphinx upstream, and we wouldn't have to carry a local version of all of
Sphinx. (In fact, the extension mechanism provides a future path for
doing kernel-doc within Sphinx instead of as a preprocessing step.)

I know none of this alleviates your concerns with table supports right
now. I'll try to have a look at that a bit more.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 04 Mar 2016, Russel Winder  wrote:
> On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
>>   1) the python version (asciidoc) appears to have been abandoned in
>>  favor of the ruby version. 
>
> This is I think true, however the Java-based tool chain Asciidoctor is
> I believe the standard bearer for ASCIIdoc these days, albeit called
> ASCIIdoctor.

If we're talking about the same asciidoctor (http://asciidoctor.org/)
it's written in ruby but you can apparently run it in JVM using
JRuby. Calling it Java-based is misleading.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Russel Winder]

On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
> 
[…]
> However, I think asciidoc has two serious problems:
> 
>   1) the python version (asciidoc) appears to have been abandoned in
>  favor of the ruby version. 

This is I think true, however the Java-based tool chain Asciidoctor is
I believe the standard bearer for ASCIIdoc these days, albeit called
ASCIIdoctor.

>   2) It really is just a docbook pre-processor. Native html/latex
> output
>  is poorly supported at best, and exposes only a small subset of
> the
>  full capabilities of the input language.

This is not true. Yes ASCIIDoc started as a DocBook/XML frontend so as
to use a sane :-) markup language rather than XML (XML is a notation
for consenting computers only), but the current ASCIIDoctor toolchain
deals very well in direct HTML and PDF generation, without needing a
DocBook/XML toolchain. 

> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to
> use
> the rest of the docbook toolchain.

Or trial the JVM-based ASCIIdoctor which is what the projects I am
involved with chose to use. Perhaps as an example I can give you http:/
/gpars.website (it's a redirector) all the HTML and PDF is generated
from ASCIIDoc source using ASCIIDoctor driven with a Gradle build
system. This is still very much a work in progress (by Jim Northrop,
not me currently), but I like it.

> We could insist on using the python version, of course. I spent a bit
> of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

If the Python version is really not being maintained, I would suggest
that unless you want to take over the project and be it's maintainer,
you would be better advised to use a different version.

-- 
Russel.=Dr
 Russel Winder  t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net41 
Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@winder.org.ukLondon SW11 
1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Thu, 03 Mar 2016 15:23:23 -0800
Keith Packard  escreveu:

> Mauro Carvalho Chehab  writes:
> 
> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
> > produced an output that worked better.  
> 
> Yes, asciidoc has much more flexibility in table formatting, including
> the ability to control text layout within cells and full control over
> borders.
> 
> However, I think asciidoc has two serious problems:
> 
>   1) the python version (asciidoc) appears to have been abandoned in
>  favor of the ruby version. 
> 
>   2) It really is just a docbook pre-processor. Native html/latex output
>  is poorly supported at best, and exposes only a small subset of the
>  full capabilities of the input language.
> 
> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to use
> the rest of the docbook toolchain.
> 
> We could insist on using the python version, of course. I spent a bit of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

I'm a way more concerned about using a tool that fulfill our needs
than to look for something that won't use the docbook toolchain or
require to install ruby.

In the case of Docbook, we know it works and we know already its
issues. Please correct me if I'm wrong, but the big problem we
have is not due to the DocBook toolchain, but due to the lack of
features at the kernel-doc script. Also, xmlto is already installed
by the ones that build the kernel docs. So, keeping use it won't
require to install a weird toolchain by hand.

So, to be frank, it doesn't scary me to use either pyhton or
ruby script + docbook.

Of course, having to own the code has a cost that should be evaluated.

If, on the other hand, we decide to use RST, we'll very likely need to
patch it to fulfill our needs in order to add proper table support.
I've no idea how easy/difficult would be to do that, nor if Sphinx
upstream would accept such changes.

So, at the end of the day, we may end by having to carry on our own
version of Sphinx inside our tree, with doesn't sound good, specially
since it is not just a script, but a package with hundreds of
files.

Thanks,
Mauro


pgp9vItgvYAsJ.pgp
Description: Assinatura digital OpenPGP

Re: Kernel docs: muddying the waters a bit

[Keith Packard]

Mauro Carvalho Chehab  writes:

> On my tests, Sphinix seemed too limited to format tables. Asciidoc
> produced an output that worked better.

Yes, asciidoc has much more flexibility in table formatting, including
the ability to control text layout within cells and full control over
borders.

However, I think asciidoc has two serious problems:

  1) the python version (asciidoc) appears to have been abandoned in
 favor of the ruby version. 

  2) It really is just a docbook pre-processor. Native html/latex output
 is poorly supported at best, and exposes only a small subset of the
 full capabilities of the input language.

As such, we would have to commit to using the ruby version and either
committing to fixing the native html output backend or continuing to use
the rest of the docbook toolchain.

We could insist on using the python version, of course. I spent a bit of
time hacking that up to add 'real' support for a table-of-contents in
the native HTML backend and it looks like getting those changes
upstreamed would be reasonably straightforward. However, we'd end up
'owning' the code, and I'm not sure we want to.

-- 
-keith


signature.asc
Description: PGP signature

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Thu, 3 Mar 2016 14:34:25 +
One Thousand Gnomes  wrote:

> We only have docbook because it was the tool of choice rather a lot of
> years ago to then get useful output formats. It was just inherited when
> borrowed the original scripts from Gnome/Gtk. It's still the most
> effective way IMHO of building big structured documents out of the kernel.

...except that we haven't used it that way.  Instead, we make a whole
bunch of smaller, partially structured document silos.

> The Gtk people long ago rewrote the original document script into a real
> tool so they have some different and maintained tools that are close to
> equivalent and already have some markdown support. Before we go off and
> re-invent the wheel it might be worth just borrowing their wheel and
> tweaking it as needed ? In particular they can generate help indexes so
> that the entire output becomes nicely browsable with an HTML based help
> browser.

Well, not inventing the wheel was kind of the motivation behind much of
this effort; I got kind of worried watching us trying to cobble more
functionality into our existing house-of-cards documentation system.

Sphinx is a well-established, heavily used, and well supported system;
using it would not be an exercise in wheel reinvention.  As far as I can
tell, it does everything we need (with some open questions about table
support), lets us drop the whole DocBook toolchain dependency, and move to
a much better-supported setup than we have now.  Plus we get much nicer
output, index generation, cross-references between documents, and the
ability to write documents in a lightweight markup language.  Seems like a
win.

I assume you're referring to gtk-doc?  It's web page
(http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
setup and use"; they recommend looking at Doxygen instead.  So I guess I'm
not really sure what it offers that merits throwing another option into
the mix now?  What am I missing?

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[One Thousand Gnomes]

> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell. 

We only have docbook because it was the tool of choice rather a lot of
years ago to then get useful output formats. It was just inherited when
borrowed the original scripts from Gnome/Gtk. It's still the most
effective way IMHO of building big structured documents out of the kernel.

The Gtk people long ago rewrote the original document script into a real
tool so they have some different and maintained tools that are close to
equivalent and already have some markdown support. Before we go off and
re-invent the wheel it might be worth just borrowing their wheel and
tweaking it as needed ? In particular they can generate help indexes so
that the entire output becomes nicely browsable with an HTML based help
browser.

Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Thu, 03 Mar 2016 16:03:14 +0200
Jani Nikula  wrote:

> This stalled a bit, but the waters are still muddy...

I've been dealing with real-world obnoxiousness, something which won't
come to an immediate end, unfortunately.  But I have been taking some time
to mess with things, and hope to have some more POC patches to send out
soon.

> Is the Sphinx/reStructuredText table support adequate for media/v4l
> documentation?

That's perhaps the biggest question.  My sense is "yes", but this needs a
bit more assurance than that.

> Are the Sphinx output formats adequate in general? Specifically, is the
> lack of DocBook support, and the flexibility it provides, a blocker?

DocBook is a means to an end; nobody really wants DocBook itself as far
as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
reasonable output, and, with some effort, we could probably get really
nice output. HTML and EPUB are easily covered, still haven't really played
around with man pages yet.  And there's LaTeX if we really need it.  I
kind of think we're covered there, unless I've missed something?

> Otherwise, I think Sphinx is promising.
> 
> Jon, I think we need a roll of dice, err, a well-thought-out decision
> from the maintainer to go with one or the other, so we can make some
> real progress.

My inclination at the moment is very much in the Sphinx direction.  I had
some vague thoughts of pushing a throwaway experimental directory with a
couple of docs for 4.6 that would just let people play with it easily;
then we'd see how many screams we get.  We'll see if the world lets me get
there.

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Sat, 13 Feb 2016, Jonathan Corbet  wrote:
> So can we discuss?  I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take.  What do you all think?

This stalled a bit, but the waters are still muddy...

Is the Sphinx/reStructuredText table support adequate for media/v4l
documentation?

Are the Sphinx output formats adequate in general? Specifically, is the
lack of DocBook support, and the flexibility it provides, a blocker?

Otherwise, I think Sphinx is promising.

Jon, I think we need a roll of dice, err, a well-thought-out decision
from the maintainer to go with one or the other, so we can make some
real progress.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Thu, Feb 18, 2016 at 10:11 AM, Jani Nikula  wrote:
> On Thu, 18 Feb 2016, Daniel Vetter  wrote:
>> On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet  wrote:
>>> On Sun, 14 Feb 2016 13:27:04 +0100
>>> Daniel Vetter  wrote:
>>>
 One concern/open I have for pro/cons are the hyperlinks from kerneldoc
 comments. Currently we have the postproc hack, iirc Jani's patches
 generated links native when extracting the kerneldoc. What's the
 solution with spinx?
>>>
>>> So I've been trying to figure out what this refers to.  Is this the
>>> cross-reference links within the document?  When I did my sphinx hack it
>>> used a technique that, shall we say, strongly resembles what Jani's
>>> patches did.  One difference is that Sphinx has the concept of
>>> "functions" built into it, so I use function references for those.
>>
>> That's what I meant. As long as I can type in stuff like func(),
>>  and similar and get a link for it automatically (plus anywhere
>> else in the templated stuff for function headers) I'm really happy.
>
> I think that could be made to work in rst just as well as
> asciidoc. Which is to say, kernel-doc may generate broken refs in both,
> since it doesn't know if the link target exists outside of the
> file. Also, in theory, it's possible to generate non-unique targets in
> the end result if there are same named enums, structs, static functions
> etc. but I think that's less of a problem. Asciidoc just ignores these
> issues, I don't know what sphinx does.
>
> I think some of that could be alleviated by making the kernel-doc
> inclusion a directive through a sphinx extension. It could at the very
> least provide informative error messages. But that's distant future.
>
> Worth noting is that, AFAICT, in all of the proposals, including the
> original where kernel-doc produces docbook, this autoreferencing only
> works within parts processed by kernel-doc. Not in the template
> documents themselves. (You can still use the markup's more verbose cross
> referencing keywords.)

It works everywhere, even in the docbook template, as long as you mark
it up correctly. Which in docbook means func.
That's because it's a post-proc path over the entire doc. But then the
entire point here is to move the overview sections all into kerneldoc,
so making the links in the templates more verbose shouldn't be a big
deal.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Thu, 18 Feb 2016, Daniel Vetter  wrote:
> On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet  wrote:
>> On Sun, 14 Feb 2016 13:27:04 +0100
>> Daniel Vetter  wrote:
>>
>>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc
>>> comments. Currently we have the postproc hack, iirc Jani's patches
>>> generated links native when extracting the kerneldoc. What's the
>>> solution with spinx?
>>
>> So I've been trying to figure out what this refers to.  Is this the
>> cross-reference links within the document?  When I did my sphinx hack it
>> used a technique that, shall we say, strongly resembles what Jani's
>> patches did.  One difference is that Sphinx has the concept of
>> "functions" built into it, so I use function references for those.
>
> That's what I meant. As long as I can type in stuff like func(),
>  and similar and get a link for it automatically (plus anywhere
> else in the templated stuff for function headers) I'm really happy.

I think that could be made to work in rst just as well as
asciidoc. Which is to say, kernel-doc may generate broken refs in both,
since it doesn't know if the link target exists outside of the
file. Also, in theory, it's possible to generate non-unique targets in
the end result if there are same named enums, structs, static functions
etc. but I think that's less of a problem. Asciidoc just ignores these
issues, I don't know what sphinx does.

I think some of that could be alleviated by making the kernel-doc
inclusion a directive through a sphinx extension. It could at the very
least provide informative error messages. But that's distant future.

Worth noting is that, AFAICT, in all of the proposals, including the
original where kernel-doc produces docbook, this autoreferencing only
works within parts processed by kernel-doc. Not in the template
documents themselves. (You can still use the markup's more verbose cross
referencing keywords.)


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet  wrote:
> On Sun, 14 Feb 2016 13:27:04 +0100
> Daniel Vetter  wrote:
>
>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc
>> comments. Currently we have the postproc hack, iirc Jani's patches
>> generated links native when extracting the kerneldoc. What's the
>> solution with spinx?
>
> So I've been trying to figure out what this refers to.  Is this the
> cross-reference links within the document?  When I did my sphinx hack it
> used a technique that, shall we say, strongly resembles what Jani's
> patches did.  One difference is that Sphinx has the concept of
> "functions" built into it, so I use function references for those.

That's what I meant. As long as I can type in stuff like func(),
 and similar and get a link for it automatically (plus anywhere
else in the templated stuff for function headers) I'm really happy.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Keith Packard]

Jonathan Corbet  writes:

> Indeed, I doubt many people want the DocBook itself.

Might be nice to actually have a set of requirements before anyone tries
to select a suitable system then :-)

Here's my current set:

asciidocsphinx

htmlvia docbook native
native (kinda)
pdf via docbook via rst2pdf
epubvia docbook?native
man ?   native?

-- 
-keith


signature.asc
Description: PGP signature

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Tue, 16 Feb 2016, Jonathan Corbet  wrote:
> Whether this is a show-stopper is indeed a good question.  I doubt many
> people wanted the DocBook for its own sake, it's a matter of where you
> can go from there.  But yes, it would be good to be sure on this point.

So the question is, are HTML, latex (for pdf and presumably ps), epub,
texinfo, and man pages enough?

My subjective opinion is they are enough, and whoever wants more can
make it work for them using pandoc. It's not like we had this great
publishing framework before that we'd be breaking. Quite the opposite,
and it would be awesome to have a robust set of outputs actually working
for the majority of people.

> There's a certain elegance to it that I like, but it is an idea that
> needs to actually be demonstrated.  It could also come later on, though,
> with the docproc or include mechanisms used for now.

Oh, totally agreed, I was just musing on potential follow-up work.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Tue, 16 Feb 2016 10:25:49 +0200
Jani Nikula  wrote:

> However I didn't think Sphinx could produce docbook, and a quick search
> doesn't convince me otherwise. Do you have some links to back this up?

Somehow I was really sure of it, but I'm not finding it now.  There is an
extension out there, but it warns about being a "work in progress," so
I'm not sure we can count it.

Whether this is a show-stopper is indeed a good question.  I doubt many
people wanted the DocBook for its own sake, it's a matter of where you
can go from there.  But yes, it would be good to be sure on this point.

> Sphinx might offer a way to drop docproc through the extension
> mechanism, without resorting to the "separate-file approach". It might
> be a more sensible approach as a whole.

There's a certain elegance to it that I like, but it is an idea that
needs to actually be demonstrated.  It could also come later on, though,
with the docproc or include mechanisms used for now.

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Kernel docs: muddying the waters a bit

[Jonathan Corbet]

So I fear you all are going to hate me for this...

Asciidoc is a credible solution to the formatted documentation problem,
but it's not the only such; I'd like to be sure that we pick the right
one.  I worry that asciidoc seems to be aimed mostly at small documents,
and that the project itself seems a little lifeless - it's not a good
sign when your main page's link to the repository has been dead for a long
time.  (Asciidoctor seems more active, with the Github folks behind it,
but that means bringing Ruby into the picture).

An alternative we haven't really looked at yet is ReStructuredText (or
"RST") and the Sphinx system (sphinx-doc.org) built on top of it.  RST is
YA simple markup scheme, remarkably similar to Markdown or Asciidoc;
Sphinx is a fairly sophisticated documentation system that uses RST.

I spent a few hours reworking the asciidoc patches to do RST instead, then
built a few template files' worth of docs.  The result can be seen at:

http://static.lwn.net/kerneldoc/

It's very much a POC (however you might want to define the term), there's
lots of glitches, I chose a theme pretty much at random, etc.  But it
shows that it can be done.

Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain
requirements there.  It produces integrated, multi-file HTML natively,
with a TOC, an index, cross-file cross references, and more.  It can make
things like function indexes.  It claims output in epub, docbook, and man
(I've not yet messed with those).  The path to PDF is via latex; clearly
the docbook path could be used too.

I used my same docproc hack to extract the comments here, mostly because I
had it at hand.  We could go with Jani's separate-file approach if we
wanted.  There's also a tool out there (called "breathe") that's meant to
turn doxygen-style comments into RST; I haven't had a chance to mess with
it.  We *could* also write an extension to pull the comments directly in
Sphinx if there were a compelling reason to do so.

If anybody's curious, the work done to get this far is in:

git://git.lwn.net/linux.git doc/sphinx

but it looks suspiciously like the previous asciidoc patches, and, in any
case, it would have to be thrown out, publicly disowned, and replaced
before going any further with this, should that be what we decide to do.

So can we discuss?  I'm not saying we have to use Sphinx, but, should we
choose not to, we should do so with open eyes and good reasons for the
course we do take.  What do you all think?

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Keith Packard]

Jonathan Corbet  writes:

> Asciidoc is a credible solution to the formatted documentation problem,
> but it's not the only such; I'd like to be sure that we pick the right
> one.  I worry that asciidoc seems to be aimed mostly at small documents,
> and that the project itself seems a little lifeless - it's not a good
> sign when your main page's link to the repository has been dead for a long
> time.  (Asciidoctor seems more active, with the Github folks behind it,
> but that means bringing Ruby into the picture).

I was surprised when one of the asciidoctor developers said that
asciidoc itself was 'in maintenance mode for existing users'. I've tried
asciidoctor but never got it to the point where I was happy with the
results. Having two tools using the same nominal format doesn't seem
like a great idea to me.

It's also clear from my hacking in asciidoc that docbook is the expected
target for that tool. I've managed to make direct HTML output usable,
but LaTeX doesn't work at all. Something which focuses on direct HTML
(and ePub) output would be pretty nice.

> An alternative we haven't really looked at yet is ReStructuredText (or
> "RST") and the Sphinx system (sphinx-doc.org) built on top of it.  RST is
> YA simple markup scheme, remarkably similar to Markdown or Asciidoc;
> Sphinx is a fairly sophisticated documentation system that uses RST.

I've installed debian's python3-sphinx package; it looks like it doesn't
have a huge dependency chain below it, which is a nice change.

I translated a fairly long document from asciidoc to rst using pandoc by
using the docbook output from asciidoc -- pandoc doesn't have a native
asciidoc reader, only a writer. The result didn't totally suck, although
I haven't messed with fixing the css or using a different theme at all.

http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html

I installed the sphinxcontrib.fulltoc extension so that the whole TOC
was visible from each section; this made navigating a lot easier. Having
search included (if you have javascript) seems like a nice feature.

> Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain
> requirements there.

Having functional native latex output means that even PDF generation is
lighterweight though.

> It produces integrated, multi-file HTML natively,
> with a TOC, an index, cross-file cross references, and more.  It can make
> things like function indexes.  It claims output in epub, docbook, and man
> (I've not yet messed with those).  The path to PDF is via latex; clearly
> the docbook path could be used too.

I've tried epub and latex backends; epub seems just fine (it's just
html, after all). LaTeX works, and generates functional PDF, but I'm
going to have to spend a bunch of time making it looks nice.

http://keithp.com/~keithp/altusmetrum-sphinx/AltusMetrum.pdf

> So can we discuss?  I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take.  What do you all think?

Having spent the afternoon playing with it, I'm definitely
impressed. I've spent a ton of time getting asciidoc to generate html
and pdf that I can tolerate; far too much of that involved hacking XML
files related to the docbook backend.

Pros

 * Credible HTML output without docbook

 * Credible PDF output without docbook.

 * Constructs a unified set of documents across
   multiple files.

 * Written in Python (2 or 3)

 * PanDoc already supports rst for both input and output. So, if we get
   bored with RST, we've got a way out.

Cons

 * Table formatting doesn't seem as sophisticated as asciidoc

Questions

 * Conditional text appears to be harder to manage (I haven't managed to
   make it work at all).

 * Takes over a directory making building more than one
   document in a directory hard/impossible? The config file must be
   named 'conf.py'?
   
-- 
-keith


signature.asc
Description: PGP signature