Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:26:10 +0200
Markus Heiser  escreveu:

> Hi Mauro,
> 
> Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab 
> :
> > Yeah, it looks better, however table truncation seem to be
> > happening also on other parts, like the tables on this page:
> > 
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> > (original table: 
> > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> > This table should contain 32 bits, but only the first 7 bits are shown
> > 
> > and those (among others):
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> > 
> > Hmm... after looking more carefully, it added a horizontal scroll bar.
> > That looks ugly, IMHO, and makes harder to understand its contents. The
> > last one, in particular 
> > (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> > is a big table on both horiz and vert dimensions. Try to read how the
> > bits are packed on a random line in the middle of the table, like
> > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.  
> 
> I know what you mean ;-) ... I'am also unhappy, but I will address this point
> later when it goes to finish the layout.
> 
> Currently lets focus on contend and (the two)extensions.

OK.

> > The table here looks weird (although it is correct):
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
> > (original table: 
> > https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
> >   
> 
> I validated this and the other tables you mentioned above ... these are 
> all correct migrated ... it is 1:1 translated from DocBook ... they
> might be show different to this what you know from your docbook
> toolchain, because in the docbook-html you have no table grids and 
> wrong cellspans are not clear ... sometimes, like in the last example 
> you gave:
> 
>   
> 
> 
> 
> a colspec might ambiguous ... so there is no clear role to migrate.

Ok, that's what I was thinking. Ok, this can be fixed later manually,
where needed. Of course one way would be to disable grids on those
tables, but I would instead fix it.

> 
> > It seems that Sphinx is assuming something like "A4 portrait" for
> > the margins, while those big tables would only fit (in PDF) as
> > "A4 landscape".  
> 
> No, no, no ;-)
> 
> Sphinx assumes nothing about the layout, sphinx and the underlying
> docutils mostly juggling with nodes and the writers in e.g. the
> html-writer, outputs a clear HTML without any style but with classified 
> HTML tags. Styling is done in the presentation layer, in HTML, it is 
> done in CSS.

Hmm... Then there's something deadly wrong at CSS template, as it is
shown texts only at half of my horizontal res (1920).

Probably this is the culpit:

  .container { margin: 50px auto 40px auto; width: 600px; text-align: 
center; }

width is set to 600px, instead of using a percentage, like 100%
(or 90%).

> 
> > I guess the better would be to not limit the right
> > margin or to change it where those big tables happen, in order to
> > allow PDF generation.  
> 
> Generating PDF has nothing to do with generating HTML. To generate
> PDF there is a other writer, the latex2e writer, which produce 
> LaTeX markup from which you build PDF or other printed-like medias.

Ok.

>  
> > 
> > There are also some tables that went wrong. See the Color Sample
> > Location table at:
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> > I'm pretty sure we'll need to fix some cases like that manually.
> > I didn't check, but perhaps, in this case, the DocBook were using 
> > empty columns just to make the table bigger. If so, this is not a
> > problem with the conversion, and should be manually fixed later.  
> 
> +1
> 
> Yes, lets do it manually later ... what I have in my POC is a automated
> process, it is hard to consider individuals in an automatic process.
> Making details *nicer* and making ambiguous markups clear is manually
> done in minutes where I need hours to implement this in a automated
> process.

Yeah, we should not try to fix everything via auto-scripts, and
spending time right now with manual fixes will be wasted, as we need
to run it at the latest media documentation, as changes might have
happened upstream.

> 
> Aside, from:  http://docutils.sourceforge.net/docs/peps/pep-0258.html
> 
> Docutils Project Model -- Project components and data flow:
> 
>  +---+
>  |Docutils:  |
>  | docutils.core.Publisher,  |
> 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:26:10 +0200
Markus Heiser  escreveu:

> Hi Mauro,
> 
> Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab 
> :
> > Yeah, it looks better, however table truncation seem to be
> > happening also on other parts, like the tables on this page:
> > 
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> > (original table: 
> > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> > This table should contain 32 bits, but only the first 7 bits are shown
> > 
> > and those (among others):
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> > 
> > Hmm... after looking more carefully, it added a horizontal scroll bar.
> > That looks ugly, IMHO, and makes harder to understand its contents. The
> > last one, in particular 
> > (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> > is a big table on both horiz and vert dimensions. Try to read how the
> > bits are packed on a random line in the middle of the table, like
> > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.  
> 
> I know what you mean ;-) ... I'am also unhappy, but I will address this point
> later when it goes to finish the layout.
> 
> Currently lets focus on contend and (the two)extensions.

OK.

> > The table here looks weird (although it is correct):
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
> > (original table: 
> > https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
> >   
> 
> I validated this and the other tables you mentioned above ... these are 
> all correct migrated ... it is 1:1 translated from DocBook ... they
> might be show different to this what you know from your docbook
> toolchain, because in the docbook-html you have no table grids and 
> wrong cellspans are not clear ... sometimes, like in the last example 
> you gave:
> 
>   
> 
> 
> 
> a colspec might ambiguous ... so there is no clear role to migrate.

Ok, that's what I was thinking. Ok, this can be fixed later manually,
where needed. Of course one way would be to disable grids on those
tables, but I would instead fix it.

> 
> > It seems that Sphinx is assuming something like "A4 portrait" for
> > the margins, while those big tables would only fit (in PDF) as
> > "A4 landscape".  
> 
> No, no, no ;-)
> 
> Sphinx assumes nothing about the layout, sphinx and the underlying
> docutils mostly juggling with nodes and the writers in e.g. the
> html-writer, outputs a clear HTML without any style but with classified 
> HTML tags. Styling is done in the presentation layer, in HTML, it is 
> done in CSS.

Hmm... Then there's something deadly wrong at CSS template, as it is
shown texts only at half of my horizontal res (1920).

Probably this is the culpit:

  .container { margin: 50px auto 40px auto; width: 600px; text-align: 
center; }

width is set to 600px, instead of using a percentage, like 100%
(or 90%).

> 
> > I guess the better would be to not limit the right
> > margin or to change it where those big tables happen, in order to
> > allow PDF generation.  
> 
> Generating PDF has nothing to do with generating HTML. To generate
> PDF there is a other writer, the latex2e writer, which produce 
> LaTeX markup from which you build PDF or other printed-like medias.

Ok.

>  
> > 
> > There are also some tables that went wrong. See the Color Sample
> > Location table at:
> > 
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> > I'm pretty sure we'll need to fix some cases like that manually.
> > I didn't check, but perhaps, in this case, the DocBook were using 
> > empty columns just to make the table bigger. If so, this is not a
> > problem with the conversion, and should be manually fixed later.  
> 
> +1
> 
> Yes, lets do it manually later ... what I have in my POC is a automated
> process, it is hard to consider individuals in an automatic process.
> Making details *nicer* and making ambiguous markups clear is manually
> done in minutes where I need hours to implement this in a automated
> process.

Yeah, we should not try to fix everything via auto-scripts, and
spending time right now with manual fixes will be wasted, as we need
to run it at the latest media documentation, as changes might have
happened upstream.

> 
> Aside, from:  http://docutils.sourceforge.net/docs/peps/pep-0258.html
> 
> Docutils Project Model -- Project components and data flow:
> 
>  +---+
>  |Docutils:  |
>  | docutils.core.Publisher,  |
>  | docutils.core.publish_*() |
>

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro,

Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab :
> Yeah, it looks better, however table truncation seem to be
> happening also on other parts, like the tables on this page:
> 
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
>   (original table: 
> https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> This table should contain 32 bits, but only the first 7 bits are shown
> 
> and those (among others):
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 
> Hmm... after looking more carefully, it added a horizontal scroll bar.
> That looks ugly, IMHO, and makes harder to understand its contents. The
> last one, in particular 
> (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> is a big table on both horiz and vert dimensions. Try to read how the
> bits are packed on a random line in the middle of the table, like
> MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.

I know what you mean ;-) ... I'am also unhappy, but I will address this point
later when it goes to finish the layout.

Currently lets focus on contend and (the two)extensions.

> The table here looks weird (although it is correct):
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
>   (original table: 
> https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
> 

I validated this and the other tables you mentioned above ... these are 
all correct migrated ... it is 1:1 translated from DocBook ... they
might be show different to this what you know from your docbook
toolchain, because in the docbook-html you have no table grids and 
wrong cellspans are not clear ... sometimes, like in the last example 
you gave:


  
  

a colspec might ambiguous ... so there is no clear role to migrate.

> It seems that Sphinx is assuming something like "A4 portrait" for
> the margins, while those big tables would only fit (in PDF) as
> "A4 landscape".

No, no, no ;-)

Sphinx assumes nothing about the layout, sphinx and the underlying
docutils mostly juggling with nodes and the writers in e.g. the
html-writer, outputs a clear HTML without any style but with classified 
HTML tags. Styling is done in the presentation layer, in HTML, it is 
done in CSS.

> I guess the better would be to not limit the right
> margin or to change it where those big tables happen, in order to
> allow PDF generation.

Generating PDF has nothing to do with generating HTML. To generate
PDF there is a other writer, the latex2e writer, which produce 
LaTeX markup from which you build PDF or other printed-like medias.
 
> 
> There are also some tables that went wrong. See the Color Sample
> Location table at:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> I'm pretty sure we'll need to fix some cases like that manually.
> I didn't check, but perhaps, in this case, the DocBook were using 
> empty columns just to make the table bigger. If so, this is not a
> problem with the conversion, and should be manually fixed later.

+1

Yes, lets do it manually later ... what I have in my POC is a automated
process, it is hard to consider individuals in an automatic process.
Making details *nicer* and making ambiguous markups clear is manually
done in minutes where I need hours to implement this in a automated
process.

Aside, from:  http://docutils.sourceforge.net/docs/peps/pep-0258.html

Docutils Project Model -- Project components and data flow:

 +---+
 |Docutils:  |
 | docutils.core.Publisher,  |
 | docutils.core.publish_*() |
 +---+
  /|\
 / | \
1,3,5   /6 |  \ 7
   ++   +-+   ++
   | READER | > | TRANSFORMER | > | WRITER |
   ++   +-+   ++
/ \\  |
   /   \\ |
 2/  4  \\ 8  |
+---+   ++++
| INPUT |   | PARSER || OUTPUT |
+---+   ++++


This is a bit simplified, because we use sphinx, which 
has "builders" and sits on top of this architecture.
But it might help to see, that processes like reading,
transforming and writing are discrete.

In short: readers (the reST file reader) are creating node trees,

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro,

Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab :
> Yeah, it looks better, however table truncation seem to be
> happening also on other parts, like the tables on this page:
> 
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
>   (original table: 
> https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> This table should contain 32 bits, but only the first 7 bits are shown
> 
> and those (among others):
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 
> Hmm... after looking more carefully, it added a horizontal scroll bar.
> That looks ugly, IMHO, and makes harder to understand its contents. The
> last one, in particular 
> (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> is a big table on both horiz and vert dimensions. Try to read how the
> bits are packed on a random line in the middle of the table, like
> MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.

I know what you mean ;-) ... I'am also unhappy, but I will address this point
later when it goes to finish the layout.

Currently lets focus on contend and (the two)extensions.

> The table here looks weird (although it is correct):
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
>   (original table: 
> https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
> 

I validated this and the other tables you mentioned above ... these are 
all correct migrated ... it is 1:1 translated from DocBook ... they
might be show different to this what you know from your docbook
toolchain, because in the docbook-html you have no table grids and 
wrong cellspans are not clear ... sometimes, like in the last example 
you gave:


  
  

a colspec might ambiguous ... so there is no clear role to migrate.

> It seems that Sphinx is assuming something like "A4 portrait" for
> the margins, while those big tables would only fit (in PDF) as
> "A4 landscape".

No, no, no ;-)

Sphinx assumes nothing about the layout, sphinx and the underlying
docutils mostly juggling with nodes and the writers in e.g. the
html-writer, outputs a clear HTML without any style but with classified 
HTML tags. Styling is done in the presentation layer, in HTML, it is 
done in CSS.

> I guess the better would be to not limit the right
> margin or to change it where those big tables happen, in order to
> allow PDF generation.

Generating PDF has nothing to do with generating HTML. To generate
PDF there is a other writer, the latex2e writer, which produce 
LaTeX markup from which you build PDF or other printed-like medias.
 
> 
> There are also some tables that went wrong. See the Color Sample
> Location table at:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> I'm pretty sure we'll need to fix some cases like that manually.
> I didn't check, but perhaps, in this case, the DocBook were using 
> empty columns just to make the table bigger. If so, this is not a
> problem with the conversion, and should be manually fixed later.

+1

Yes, lets do it manually later ... what I have in my POC is a automated
process, it is hard to consider individuals in an automatic process.
Making details *nicer* and making ambiguous markups clear is manually
done in minutes where I need hours to implement this in a automated
process.

Aside, from:  http://docutils.sourceforge.net/docs/peps/pep-0258.html

Docutils Project Model -- Project components and data flow:

 +---+
 |Docutils:  |
 | docutils.core.Publisher,  |
 | docutils.core.publish_*() |
 +---+
  /|\
 / | \
1,3,5   /6 |  \ 7
   ++   +-+   ++
   | READER | > | TRANSFORMER | > | WRITER |
   ++   +-+   ++
/ \\  |
   /   \\ |
 2/  4  \\ 8  |
+---+   ++++
| INPUT |   | PARSER || OUTPUT |
+---+   ++++


This is a bit simplified, because we use sphinx, which 
has "builders" and sits on top of this architecture.
But it might help to see, that processes like reading,
transforming and writing are discrete.

In short: readers (the reST file reader) are creating node trees,
which are transformed by a 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser  wrote:
> Am 06.05.2016 um 17:06 schrieb Jani Nikula :
>
>> On Fri, 06 May 2016, Markus Heiser  wrote:
>>> @Jonathan: what do you think? Should I prepare a patch
>>> with a basic reST (sphinx) build infrastructure, including
>>> 
>>> * a folder for sphinx docs:
>>> 
>>>  ./Documentation/sphinx/
>> 
>> I'm already working on a patch series taking a different approach. I
>> don't think we should hide the documentation under an extra folder named
>> after a tool. Actually, I'm strongly opposed to that.
>
> Could you post a link to a repo? / thanks

Very much a work-in-progress
https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx

I was hoping to polish it a bit more before showing it to the world.

> There is no need for concurrency, let's work together on your repo. 
> Within my POC I realized similar building processes we will need in the
> kernel sources ... where you have cascading configuration. A base 
> configuration which fits for all common cases and (if needed) a 
> *per-book* configuration.
>
> At the end, when it comes to generate pdf books/articles, man pages 
> and e.g. texinfo files out of a sphinx-project you will need a build
> infrastructure like this.

...

> You will need on sphinx-project for each DocBook and one single 
> sphinx-project where you collect the .txt to .rst migrated files.

Surely you know more about Sphinx than I do, but I specifically would
like to include e.g. gpu documentation in the main build. I'm really
hoping we can have *additional* configuration files for special cases
(only) as needed.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser  wrote:
> Am 06.05.2016 um 17:06 schrieb Jani Nikula :
>
>> On Fri, 06 May 2016, Markus Heiser  wrote:
>>> @Jonathan: what do you think? Should I prepare a patch
>>> with a basic reST (sphinx) build infrastructure, including
>>> 
>>> * a folder for sphinx docs:
>>> 
>>>  ./Documentation/sphinx/
>> 
>> I'm already working on a patch series taking a different approach. I
>> don't think we should hide the documentation under an extra folder named
>> after a tool. Actually, I'm strongly opposed to that.
>
> Could you post a link to a repo? / thanks

Very much a work-in-progress
https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx

I was hoping to polish it a bit more before showing it to the world.

> There is no need for concurrency, let's work together on your repo. 
> Within my POC I realized similar building processes we will need in the
> kernel sources ... where you have cascading configuration. A base 
> configuration which fits for all common cases and (if needed) a 
> *per-book* configuration.
>
> At the end, when it comes to generate pdf books/articles, man pages 
> and e.g. texinfo files out of a sphinx-project you will need a build
> infrastructure like this.

...

> You will need on sphinx-project for each DocBook and one single 
> sphinx-project where you collect the .txt to .rst migrated files.

Surely you know more about Sphinx than I do, but I specifically would
like to include e.g. gpu documentation in the main build. I'm really
hoping we can have *additional* configuration files for special cases
(only) as needed.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 06.05.2016 um 17:06 schrieb Jani Nikula :

> On Fri, 06 May 2016, Markus Heiser  wrote:
>> @Jonathan: what do you think? Should I prepare a patch
>> with a basic reST (sphinx) build infrastructure, including
>> 
>> * a folder for sphinx docs:
>> 
>>  ./Documentation/sphinx/
> 
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.

Could you post a link to a repo? / thanks 

There is no need for concurrency, let's work together on your repo. 
Within my POC I realized similar building processes we will need in the
kernel sources ... where you have cascading configuration. A base 
configuration which fits for all common cases and (if needed) a 
*per-book* configuration.

At the end, when it comes to generate pdf books/articles, man pages 
and e.g. texinfo files out of a sphinx-project you will need a build
infrastructure like this.

> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.
> 
> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.
> 

Ok, I agree with you in the fact that a additional "sphinx" folder
is unrewarding.

This means (e.g.) a migrated Documentation/DocBook/gpu book should placed
in Documentation/gpu ... but don' try to merge all (Doc-)Books and .txt-files
into one sphinx project!

You will need on sphinx-project for each DocBook and one single 
sphinx-project where you collect the .txt to .rst migrated files.

Am 06.05.2016 um 17:23 schrieb Mauro Carvalho Chehab :

> We won't avoid the need of moving things among directories, as we
> have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Yes, it is named by the toolchain, but no one reads xml-files. Reading
text files is common.

> Ok, if we put the .rst files at Documentation, we very likely reduce
> the number of renames, but we'll increase the Makefile complexity,
> and the risk of breakages.

I don't see a great potential of breakages ...  if we place every book
in a separated folder and have one project which collects the .txt files
(see above).


--Markus--

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 06.05.2016 um 17:06 schrieb Jani Nikula :

> On Fri, 06 May 2016, Markus Heiser  wrote:
>> @Jonathan: what do you think? Should I prepare a patch
>> with a basic reST (sphinx) build infrastructure, including
>> 
>> * a folder for sphinx docs:
>> 
>>  ./Documentation/sphinx/
> 
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.

Could you post a link to a repo? / thanks 

There is no need for concurrency, let's work together on your repo. 
Within my POC I realized similar building processes we will need in the
kernel sources ... where you have cascading configuration. A base 
configuration which fits for all common cases and (if needed) a 
*per-book* configuration.

At the end, when it comes to generate pdf books/articles, man pages 
and e.g. texinfo files out of a sphinx-project you will need a build
infrastructure like this.

> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.
> 
> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.
> 

Ok, I agree with you in the fact that a additional "sphinx" folder
is unrewarding.

This means (e.g.) a migrated Documentation/DocBook/gpu book should placed
in Documentation/gpu ... but don' try to merge all (Doc-)Books and .txt-files
into one sphinx project!

You will need on sphinx-project for each DocBook and one single 
sphinx-project where you collect the .txt to .rst migrated files.

Am 06.05.2016 um 17:23 schrieb Mauro Carvalho Chehab :

> We won't avoid the need of moving things among directories, as we
> have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Yes, it is named by the toolchain, but no one reads xml-files. Reading
text files is common.

> Ok, if we put the .rst files at Documentation, we very likely reduce
> the number of renames, but we'll increase the Makefile complexity,
> and the risk of breakages.

I don't see a great potential of breakages ...  if we place every book
in a separated folder and have one project which collects the .txt files
(see above).


--Markus--

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:06:49 +0300
Jani Nikula  escreveu:

> On Fri, 06 May 2016, Markus Heiser  wrote:
> > @Jonathan: what do you think? Should I prepare a patch
> > with a basic reST (sphinx) build infrastructure, including
> >
> > * a folder for sphinx docs:
> >
> >   ./Documentation/sphinx/  
> 
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.
> 
> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.

We won't avoid the need of moving things among directories, as we
have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Ok, if we put the .rst files at Documentation, we very likely reduce
the number of renames, but we'll increase the Makefile complexity,
and the risk of breakages. One alternative would be to put the sphinx
stuff on a separate Makefile, but using multiple makefiles on a single
dir is not standard at the Kernel.

> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.


Well, git will show the change as a rename, no matter if the
directory name changes or not (except if we keep the rst files
with .txt extension), but I agree with you that people will expect
to see text files at Documentation, and most will just read it without
caring to run Sphinx.

-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:06:49 +0300
Jani Nikula  escreveu:

> On Fri, 06 May 2016, Markus Heiser  wrote:
> > @Jonathan: what do you think? Should I prepare a patch
> > with a basic reST (sphinx) build infrastructure, including
> >
> > * a folder for sphinx docs:
> >
> >   ./Documentation/sphinx/  
> 
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.
> 
> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.

We won't avoid the need of moving things among directories, as we
have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Ok, if we put the .rst files at Documentation, we very likely reduce
the number of renames, but we'll increase the Makefile complexity,
and the risk of breakages. One alternative would be to put the sphinx
stuff on a separate Makefile, but using multiple makefiles on a single
dir is not standard at the Kernel.

> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.


Well, git will show the change as a rename, no matter if the
directory name changes or not (except if we keep the rst files
with .txt extension), but I agree with you that people will expect
to see text files at Documentation, and most will just read it without
caring to run Sphinx.

-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 16:27:21 +0200
Markus Heiser  escreveu:

> Hi all, hi Jonathan,
> 
> Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab 
> :
> 
> > Em Fri, 6 May 2016 15:32:35 +0200
> > Markus Heiser  escreveu:
> >   
> >> Hi Mauro,
> >> 
> >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab 
> >> :
> >>   
> >>> Markus,
> >>> 
> >>> Em Fri, 6 May 2016 13:23:06 +0200
> >>> Markus Heiser  escreveu:
> >>>   
>  
>  In this conf.py you have to *register* your folder with the extensions.
>  A few words about the flat-table extension and a (future) kernel-doc 
>  one:
> >>> 
> >>> ...
> >>>   
>  The flat-table is a pure docutils (the layer below sphinx) extension 
>  which
>  is not application specific, so I will ask for moving it to the docutils 
>  upstream. 
> >>> 
> >>> So, if I understood well, your proposal is to have a conf.py and the
> >>> flat-table (plus other extensions) at the Kernel tree.
> >> 
> >> Each book (better call it root-node) is a Sphinx-project, each 
> >> Sphinx-project need a conf.py file (the build configuration file)
> >> in its reST-source tree.
> >> 
> >> * http://www.sphinx-doc.org/en/stable/config.html
> >>   
> >>> Assuming that docutils upstream receives the flat-table extension
> >>> (and eventually modifies it), while the new version doesn't arrive
> >>> all distros, we'll end by having some developers using the newer
> >>> docutils with the extension, plus others using the in-tree one.
> >>> 
> >>> Is there a way to specify at the conf.py what extension variant
> >>> should it use, in case of both the in-tree or the docutils have
> >>> the same?
> >> 
> >> The build configuration file is a regular python file, you can 
> >> implement conditions whatever you want/need.
> >>   
> >>> This could be trickier if they end by modifying the extension,
> >>> but we can always backport the latest version, if they change the
> >>> API.
> >> 
> >> As far as i know, the docutils API is stable since 2002. In the
> >> meantime there has been so many application build on it that
> >> it is not realistic, you will see a not backward compatibly 
> >> change.
> >> 
> >> The docutils project is conservative, very conservative, IMO to
> >> conservative.
> >> 
> >> Today I'am doubtful if it isn't better I would merge it sphinx
> >> upstream. I have to discuss this with some maintainers, but 
> >> before I have to persuade myself, that all aspects are covered
> >> and the implementations are robust. We are at the beginning and
> >> we should not fear about every bit which could happen in the future.
> >> 
> >> The sphinx / docutils bottom plate gives us a number of degrees 
> >> of freedom to find answers to question we have not yet asked. ;-)  
> > 
> > Ok. So, from what I understand, once Sphinx support is added at
> > Kernel upstream, we could convert the media docbook to
> > reST+flat-table extension, adding such extension either on a shared
> > place or only for the media DocBook build, together with its
> > conf.py.
> >   
> 
> Yes, in your media-conf you could decide to use a extension.
> 
> > Once it reaches upstream (either sphinx or docutils), we can
> > work to make it integrate better with upstream as needed.
> > 
> > Right?  
> 
> yes, right :-)
> 
> > If so, I'm ok with merging it as soon as possible.  
> 
> If we advice a merge of the flat-table directive we should 
> bundle this with the (to implement) "kernel-doc" directive ...
> 
> > In reST the directive might look like:
> > 
> >  -
> > Device Instance and Driver Handling
> > ===
> > 
> > .. kernel-doc::  drivers/gpu/drm/drm_drv.c
> >   :doc:  driver instance overview
> >   :exported:
> > 
> >  -  
> 
> and the patches from my kernel-doc perl script to produce
> reST from source code comments.
> 
> With this bundle within the kernel tree we have a good starting
> point to compose reST documents from scratch and to migrate book
> by book from DocBook to reST.
> 
> I insist to migrate book by book, because there are some
> broken books. Broken by that, that some sources have changed
> but not the corresponding documentation which use the comments
> of these sources ... grap "Ooops" in the builded (xml or rst) docs.
> 
> E.g. I greped the .rst file and found the following Oops in the migrated 
> books:
> 
> ./books/mtdnand/pubfunctions-000-012.rst:13:Oops
> ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
> ./books/s390-drivers/ccw-000-004-003.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
> ./books/device-drivers/Basics-000-001-002.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
> ./books/device-drivers/Basics-000-009-032.rst:13:Oops
> 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 16:27:21 +0200
Markus Heiser  escreveu:

> Hi all, hi Jonathan,
> 
> Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab 
> :
> 
> > Em Fri, 6 May 2016 15:32:35 +0200
> > Markus Heiser  escreveu:
> >   
> >> Hi Mauro,
> >> 
> >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab 
> >> :
> >>   
> >>> Markus,
> >>> 
> >>> Em Fri, 6 May 2016 13:23:06 +0200
> >>> Markus Heiser  escreveu:
> >>>   
>  
>  In this conf.py you have to *register* your folder with the extensions.
>  A few words about the flat-table extension and a (future) kernel-doc 
>  one:
> >>> 
> >>> ...
> >>>   
>  The flat-table is a pure docutils (the layer below sphinx) extension 
>  which
>  is not application specific, so I will ask for moving it to the docutils 
>  upstream. 
> >>> 
> >>> So, if I understood well, your proposal is to have a conf.py and the
> >>> flat-table (plus other extensions) at the Kernel tree.
> >> 
> >> Each book (better call it root-node) is a Sphinx-project, each 
> >> Sphinx-project need a conf.py file (the build configuration file)
> >> in its reST-source tree.
> >> 
> >> * http://www.sphinx-doc.org/en/stable/config.html
> >>   
> >>> Assuming that docutils upstream receives the flat-table extension
> >>> (and eventually modifies it), while the new version doesn't arrive
> >>> all distros, we'll end by having some developers using the newer
> >>> docutils with the extension, plus others using the in-tree one.
> >>> 
> >>> Is there a way to specify at the conf.py what extension variant
> >>> should it use, in case of both the in-tree or the docutils have
> >>> the same?
> >> 
> >> The build configuration file is a regular python file, you can 
> >> implement conditions whatever you want/need.
> >>   
> >>> This could be trickier if they end by modifying the extension,
> >>> but we can always backport the latest version, if they change the
> >>> API.
> >> 
> >> As far as i know, the docutils API is stable since 2002. In the
> >> meantime there has been so many application build on it that
> >> it is not realistic, you will see a not backward compatibly 
> >> change.
> >> 
> >> The docutils project is conservative, very conservative, IMO to
> >> conservative.
> >> 
> >> Today I'am doubtful if it isn't better I would merge it sphinx
> >> upstream. I have to discuss this with some maintainers, but 
> >> before I have to persuade myself, that all aspects are covered
> >> and the implementations are robust. We are at the beginning and
> >> we should not fear about every bit which could happen in the future.
> >> 
> >> The sphinx / docutils bottom plate gives us a number of degrees 
> >> of freedom to find answers to question we have not yet asked. ;-)  
> > 
> > Ok. So, from what I understand, once Sphinx support is added at
> > Kernel upstream, we could convert the media docbook to
> > reST+flat-table extension, adding such extension either on a shared
> > place or only for the media DocBook build, together with its
> > conf.py.
> >   
> 
> Yes, in your media-conf you could decide to use a extension.
> 
> > Once it reaches upstream (either sphinx or docutils), we can
> > work to make it integrate better with upstream as needed.
> > 
> > Right?  
> 
> yes, right :-)
> 
> > If so, I'm ok with merging it as soon as possible.  
> 
> If we advice a merge of the flat-table directive we should 
> bundle this with the (to implement) "kernel-doc" directive ...
> 
> > In reST the directive might look like:
> > 
> >  -
> > Device Instance and Driver Handling
> > ===
> > 
> > .. kernel-doc::  drivers/gpu/drm/drm_drv.c
> >   :doc:  driver instance overview
> >   :exported:
> > 
> >  -  
> 
> and the patches from my kernel-doc perl script to produce
> reST from source code comments.
> 
> With this bundle within the kernel tree we have a good starting
> point to compose reST documents from scratch and to migrate book
> by book from DocBook to reST.
> 
> I insist to migrate book by book, because there are some
> broken books. Broken by that, that some sources have changed
> but not the corresponding documentation which use the comments
> of these sources ... grap "Ooops" in the builded (xml or rst) docs.
> 
> E.g. I greped the .rst file and found the following Oops in the migrated 
> books:
> 
> ./books/mtdnand/pubfunctions-000-012.rst:13:Oops
> ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
> ./books/s390-drivers/ccw-000-004-003.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
> ./books/device-drivers/Basics-000-001-002.rst:13:Oops
> ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
> ./books/device-drivers/Basics-000-009-032.rst:13:Oops
> ./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops
> ./books/gadget/api-000-011-005.rst:13:Oops
> ./books/gadget/api-000-011-009.rst:13:Oops
> 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser  wrote:
> @Jonathan: what do you think? Should I prepare a patch
> with a basic reST (sphinx) build infrastructure, including
>
> * a folder for sphinx docs:
>
>   ./Documentation/sphinx/

I'm already working on a patch series taking a different approach. I
don't think we should hide the documentation under an extra folder named
after a tool. Actually, I'm strongly opposed to that.

Instead, we should place the Sphinx stuff directly under Documentation,
and have Sphinx recursively pick up all the *.rst files. We should
promote gradually switching to lightweight markup and integration of the
documents into one system. This process should be as little disruptive
as possible.

If someone wants to convert a .txt document to .rst and get it processed
by Sphinx, it should be as simple as renaming the file, doing the
necessary edits, and adding it to a toctree. Imagine gradually
converting the files under, say, Documentation/kbuild. Why should the
.rst files be moved under another directory? They should stay alongside
the .txt files under the same directory. There's bound to be a lot of
people who'll never use Sphinx, and will expect to find the good old
plain text files (albeit with some markup) where they always were.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser  wrote:
> @Jonathan: what do you think? Should I prepare a patch
> with a basic reST (sphinx) build infrastructure, including
>
> * a folder for sphinx docs:
>
>   ./Documentation/sphinx/

I'm already working on a patch series taking a different approach. I
don't think we should hide the documentation under an extra folder named
after a tool. Actually, I'm strongly opposed to that.

Instead, we should place the Sphinx stuff directly under Documentation,
and have Sphinx recursively pick up all the *.rst files. We should
promote gradually switching to lightweight markup and integration of the
documents into one system. This process should be as little disruptive
as possible.

If someone wants to convert a .txt document to .rst and get it processed
by Sphinx, it should be as simple as renaming the file, doing the
necessary edits, and adding it to a toctree. Imagine gradually
converting the files under, say, Documentation/kbuild. Why should the
.rst files be moved under another directory? They should stay alongside
the .txt files under the same directory. There's bound to be a lot of
people who'll never use Sphinx, and will expect to find the good old
plain text files (albeit with some markup) where they always were.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi all, hi Jonathan,

Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab :

> Em Fri, 6 May 2016 15:32:35 +0200
> Markus Heiser  escreveu:
> 
>> Hi Mauro,
>> 
>> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab 
>> :
>> 
>>> Markus,
>>> 
>>> Em Fri, 6 May 2016 13:23:06 +0200
>>> Markus Heiser  escreveu:
>>> 
 
 In this conf.py you have to *register* your folder with the extensions.
 A few words about the flat-table extension and a (future) kernel-doc one:  
>>> 
>>> ...
>>> 
 The flat-table is a pure docutils (the layer below sphinx) extension which
 is not application specific, so I will ask for moving it to the docutils 
 upstream.   
>>> 
>>> So, if I understood well, your proposal is to have a conf.py and the
>>> flat-table (plus other extensions) at the Kernel tree.  
>> 
>> Each book (better call it root-node) is a Sphinx-project, each 
>> Sphinx-project need a conf.py file (the build configuration file)
>> in its reST-source tree.
>> 
>> * http://www.sphinx-doc.org/en/stable/config.html
>> 
>>> Assuming that docutils upstream receives the flat-table extension
>>> (and eventually modifies it), while the new version doesn't arrive
>>> all distros, we'll end by having some developers using the newer
>>> docutils with the extension, plus others using the in-tree one.
>>> 
>>> Is there a way to specify at the conf.py what extension variant
>>> should it use, in case of both the in-tree or the docutils have
>>> the same?  
>> 
>> The build configuration file is a regular python file, you can 
>> implement conditions whatever you want/need.
>> 
>>> This could be trickier if they end by modifying the extension,
>>> but we can always backport the latest version, if they change the
>>> API.  
>> 
>> As far as i know, the docutils API is stable since 2002. In the
>> meantime there has been so many application build on it that
>> it is not realistic, you will see a not backward compatibly 
>> change.
>> 
>> The docutils project is conservative, very conservative, IMO to
>> conservative.
>> 
>> Today I'am doubtful if it isn't better I would merge it sphinx
>> upstream. I have to discuss this with some maintainers, but 
>> before I have to persuade myself, that all aspects are covered
>> and the implementations are robust. We are at the beginning and
>> we should not fear about every bit which could happen in the future.
>> 
>> The sphinx / docutils bottom plate gives us a number of degrees 
>> of freedom to find answers to question we have not yet asked. ;-)
> 
> Ok. So, from what I understand, once Sphinx support is added at
> Kernel upstream, we could convert the media docbook to
> reST+flat-table extension, adding such extension either on a shared
> place or only for the media DocBook build, together with its
> conf.py.
> 

Yes, in your media-conf you could decide to use a extension.

> Once it reaches upstream (either sphinx or docutils), we can
> work to make it integrate better with upstream as needed.
> 
> Right?

yes, right :-)

> If so, I'm ok with merging it as soon as possible.

If we advice a merge of the flat-table directive we should 
bundle this with the (to implement) "kernel-doc" directive ...

> In reST the directive might look like:
> 
>  -
> Device Instance and Driver Handling
> ===
> 
> .. kernel-doc::  drivers/gpu/drm/drm_drv.c
>   :doc:  driver instance overview
>   :exported:
> 
>  -

and the patches from my kernel-doc perl script to produce
reST from source code comments.

With this bundle within the kernel tree we have a good starting
point to compose reST documents from scratch and to migrate book
by book from DocBook to reST.

I insist to migrate book by book, because there are some
broken books. Broken by that, that some sources have changed
but not the corresponding documentation which use the comments
of these sources ... grap "Ooops" in the builded (xml or rst) docs.

E.g. I greped the .rst file and found the following Oops in the migrated books:

./books/mtdnand/pubfunctions-000-012.rst:13:Oops
./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
./books/s390-drivers/ccw-000-004-003.rst:13:Oops
./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
./books/device-drivers/Basics-000-001-002.rst:13:Oops
./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
./books/device-drivers/Basics-000-009-032.rst:13:Oops
./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops
./books/gadget/api-000-011-005.rst:13:Oops
./books/gadget/api-000-011-009.rst:13:Oops
./books/gadget/api-000-011-007.rst:13:Oops
./books/gadget/api-000-011-003.rst:13:Oops
./books/gadget/api-000-011-011.rst:13:Oops
./books/genericirq/intfunctions-000-009.rst:13:Oops

- Summarize 

@Jonathan: what do you think? Should I prepare a patch
with a basic reST (sphinx) build 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi all, hi Jonathan,

Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab :

> Em Fri, 6 May 2016 15:32:35 +0200
> Markus Heiser  escreveu:
> 
>> Hi Mauro,
>> 
>> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab 
>> :
>> 
>>> Markus,
>>> 
>>> Em Fri, 6 May 2016 13:23:06 +0200
>>> Markus Heiser  escreveu:
>>> 
 
 In this conf.py you have to *register* your folder with the extensions.
 A few words about the flat-table extension and a (future) kernel-doc one:  
>>> 
>>> ...
>>> 
 The flat-table is a pure docutils (the layer below sphinx) extension which
 is not application specific, so I will ask for moving it to the docutils 
 upstream.   
>>> 
>>> So, if I understood well, your proposal is to have a conf.py and the
>>> flat-table (plus other extensions) at the Kernel tree.  
>> 
>> Each book (better call it root-node) is a Sphinx-project, each 
>> Sphinx-project need a conf.py file (the build configuration file)
>> in its reST-source tree.
>> 
>> * http://www.sphinx-doc.org/en/stable/config.html
>> 
>>> Assuming that docutils upstream receives the flat-table extension
>>> (and eventually modifies it), while the new version doesn't arrive
>>> all distros, we'll end by having some developers using the newer
>>> docutils with the extension, plus others using the in-tree one.
>>> 
>>> Is there a way to specify at the conf.py what extension variant
>>> should it use, in case of both the in-tree or the docutils have
>>> the same?  
>> 
>> The build configuration file is a regular python file, you can 
>> implement conditions whatever you want/need.
>> 
>>> This could be trickier if they end by modifying the extension,
>>> but we can always backport the latest version, if they change the
>>> API.  
>> 
>> As far as i know, the docutils API is stable since 2002. In the
>> meantime there has been so many application build on it that
>> it is not realistic, you will see a not backward compatibly 
>> change.
>> 
>> The docutils project is conservative, very conservative, IMO to
>> conservative.
>> 
>> Today I'am doubtful if it isn't better I would merge it sphinx
>> upstream. I have to discuss this with some maintainers, but 
>> before I have to persuade myself, that all aspects are covered
>> and the implementations are robust. We are at the beginning and
>> we should not fear about every bit which could happen in the future.
>> 
>> The sphinx / docutils bottom plate gives us a number of degrees 
>> of freedom to find answers to question we have not yet asked. ;-)
> 
> Ok. So, from what I understand, once Sphinx support is added at
> Kernel upstream, we could convert the media docbook to
> reST+flat-table extension, adding such extension either on a shared
> place or only for the media DocBook build, together with its
> conf.py.
> 

Yes, in your media-conf you could decide to use a extension.

> Once it reaches upstream (either sphinx or docutils), we can
> work to make it integrate better with upstream as needed.
> 
> Right?

yes, right :-)

> If so, I'm ok with merging it as soon as possible.

If we advice a merge of the flat-table directive we should 
bundle this with the (to implement) "kernel-doc" directive ...

> In reST the directive might look like:
> 
>  -
> Device Instance and Driver Handling
> ===
> 
> .. kernel-doc::  drivers/gpu/drm/drm_drv.c
>   :doc:  driver instance overview
>   :exported:
> 
>  -

and the patches from my kernel-doc perl script to produce
reST from source code comments.

With this bundle within the kernel tree we have a good starting
point to compose reST documents from scratch and to migrate book
by book from DocBook to reST.

I insist to migrate book by book, because there are some
broken books. Broken by that, that some sources have changed
but not the corresponding documentation which use the comments
of these sources ... grap "Ooops" in the builded (xml or rst) docs.

E.g. I greped the .rst file and found the following Oops in the migrated books:

./books/mtdnand/pubfunctions-000-012.rst:13:Oops
./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
./books/s390-drivers/ccw-000-004-003.rst:13:Oops
./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
./books/device-drivers/Basics-000-001-002.rst:13:Oops
./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
./books/device-drivers/Basics-000-009-032.rst:13:Oops
./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops
./books/gadget/api-000-011-005.rst:13:Oops
./books/gadget/api-000-011-009.rst:13:Oops
./books/gadget/api-000-011-007.rst:13:Oops
./books/gadget/api-000-011-003.rst:13:Oops
./books/gadget/api-000-011-011.rst:13:Oops
./books/genericirq/intfunctions-000-009.rst:13:Oops

- Summarize 

@Jonathan: what do you think? Should I prepare a patch
with a basic reST (sphinx) build infrastructure, including

* a folder for sphinx docs:

  ./Documentation/sphinx/

* flat-table & kernel-doc 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jani,

I forget to mentioning, with a local copy of my kernel-doc script:

https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc

You could do reST markup in the source code comments and extract them.

This might be a interim workaround which helps you not to edit source 
code comments twice.

-- Markus--



Am 06.05.2016 um 13:23 schrieb Markus Heiser :

> 
> Hy Jani,
> 
> Am 04.05.2016 um 18:13 schrieb Jani Nikula :
> 
>>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>>> 
 I think all of this makes sense.  It would be really nice to have the
 directives in the native sphinx language like that.  I *don't* think we
 need to aim for that at the outset; the docproc approach works until we can
 properly get rid of it.  What would be *really* nice would be to get
 support for the kernel-doc directive into the sphinx upstream.
>>> 
>>> No need for kernel-doc directive in sphinx upstream, later it will be 
>>> an extension which could be installed by a simple command like 
>>> "pip install kernel-doc-extensions" or similar.
>>> 
>>> I develop these required extension (and more) within my proof of concept
>>> on github ... this takes time ... if I finished all my tests and all is
>>> well, I will build the *kernel-doc-extensions* package and deploy it
>>> on https://pypi.python.org/pypi from where everyone could install this 
>>> with "pip".
>> 
>> I think we should go for vanilla sphinx at first, to make the setup step
>> as easy as possible for everyone. Even if it means still doing that ugly
>> docproc step to call kernel-doc. We can improve from there, and I
>> definitely appreciate your work on making this work with sphinx
>> extensions.
> 
> +1 
> 
>> That said, how would it work to include the kernel-doc extension in the
>> kernel source tree? Having things just work if sphinx is installed is
>> preferred over requiring installation of something extra from pypi. (I
>> know this may sound backwards for a lot of projects, but for kernel I'm
>> pretty sure this is how it should be done.)
> 
> Thats all right. Lets talk about the extension infrastructure by example:
> 
> First we have to chose a folder where we place all the *sphinx-documentation*
> I recommending:
> 
> /share/linux/Documentation/sphinx
> 
> Next we have to chose a folder where reST-extensions should take place, I
> would prefer ... or similar:
> 
> /share/linux/Documentation/sphinx/extensions
> 
> Lets say, you wan't to get in use of the "flat-table" extension.
> 
> Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
> other extensions which might be there) ...
> 
> https://github.com/return42/sphkerneldoc/tree/master/doc/extensions
> 
> Now lets say you are writing on a gpu book, it wold be placed in the folder:
> 
> /share/linux/Documentation/sphinx/gpu
> 
> In this gpu-folder you have to place the conf.py config file, needed to
> setup the sphinx build environment.
> 
> /share/linux/Documentation/sphinx/gpu/conf.py
> 
> In this conf.py you have to *register* your folder with the extensions.
> 
>  
> 
>import os.path, sys
> 
>EXT_PATH  = "../extensions"  # the path of extension folder relative to 
> the conf.py file
>sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))
> 
># now import the "flat-table" extension, it will be self-registering to 
> docutils
> 
>import rstFlatTable
> 
>  
> 
> Thats all, you can run your sphinx-build command and the flat-tables in your
> reST sources should be handled as common tables.
> 
> ASIDE: 
> 
> You will find similar parts in your conf.py which you have created 
> with the sphinx-quickstart command. There, you will also find a block 
> looks like ...
> 
> extensions = [
>'sphinx.ext.autodoc'
> 
> ]
> 
> Don't try to add flat-table extension to this list. This list is a list
> of sphinx extensions, we will use it later for other *real* sphinx 
> extensions.
> 
> A few words about the flat-table extension and a (future) kernel-doc one:
> 
> The flat-table is a pure docutils (the layer below sphinx) extension which
> is not application specific, so I will ask for moving it to the docutils 
> upstream. 
> 
> The kernel-doc extension on the other side is a very (very) kernel specific
> application, this would never go to sphinx nor docutils upstream.
> 
> --Markus--
> 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jani,

I forget to mentioning, with a local copy of my kernel-doc script:

https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc

You could do reST markup in the source code comments and extract them.

This might be a interim workaround which helps you not to edit source 
code comments twice.

-- Markus--



Am 06.05.2016 um 13:23 schrieb Markus Heiser :

> 
> Hy Jani,
> 
> Am 04.05.2016 um 18:13 schrieb Jani Nikula :
> 
>>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>>> 
 I think all of this makes sense.  It would be really nice to have the
 directives in the native sphinx language like that.  I *don't* think we
 need to aim for that at the outset; the docproc approach works until we can
 properly get rid of it.  What would be *really* nice would be to get
 support for the kernel-doc directive into the sphinx upstream.
>>> 
>>> No need for kernel-doc directive in sphinx upstream, later it will be 
>>> an extension which could be installed by a simple command like 
>>> "pip install kernel-doc-extensions" or similar.
>>> 
>>> I develop these required extension (and more) within my proof of concept
>>> on github ... this takes time ... if I finished all my tests and all is
>>> well, I will build the *kernel-doc-extensions* package and deploy it
>>> on https://pypi.python.org/pypi from where everyone could install this 
>>> with "pip".
>> 
>> I think we should go for vanilla sphinx at first, to make the setup step
>> as easy as possible for everyone. Even if it means still doing that ugly
>> docproc step to call kernel-doc. We can improve from there, and I
>> definitely appreciate your work on making this work with sphinx
>> extensions.
> 
> +1 
> 
>> That said, how would it work to include the kernel-doc extension in the
>> kernel source tree? Having things just work if sphinx is installed is
>> preferred over requiring installation of something extra from pypi. (I
>> know this may sound backwards for a lot of projects, but for kernel I'm
>> pretty sure this is how it should be done.)
> 
> Thats all right. Lets talk about the extension infrastructure by example:
> 
> First we have to chose a folder where we place all the *sphinx-documentation*
> I recommending:
> 
> /share/linux/Documentation/sphinx
> 
> Next we have to chose a folder where reST-extensions should take place, I
> would prefer ... or similar:
> 
> /share/linux/Documentation/sphinx/extensions
> 
> Lets say, you wan't to get in use of the "flat-table" extension.
> 
> Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
> other extensions which might be there) ...
> 
> https://github.com/return42/sphkerneldoc/tree/master/doc/extensions
> 
> Now lets say you are writing on a gpu book, it wold be placed in the folder:
> 
> /share/linux/Documentation/sphinx/gpu
> 
> In this gpu-folder you have to place the conf.py config file, needed to
> setup the sphinx build environment.
> 
> /share/linux/Documentation/sphinx/gpu/conf.py
> 
> In this conf.py you have to *register* your folder with the extensions.
> 
>  
> 
>import os.path, sys
> 
>EXT_PATH  = "../extensions"  # the path of extension folder relative to 
> the conf.py file
>sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))
> 
># now import the "flat-table" extension, it will be self-registering to 
> docutils
> 
>import rstFlatTable
> 
>  
> 
> Thats all, you can run your sphinx-build command and the flat-tables in your
> reST sources should be handled as common tables.
> 
> ASIDE: 
> 
> You will find similar parts in your conf.py which you have created 
> with the sphinx-quickstart command. There, you will also find a block 
> looks like ...
> 
> extensions = [
>'sphinx.ext.autodoc'
> 
> ]
> 
> Don't try to add flat-table extension to this list. This list is a list
> of sphinx extensions, we will use it later for other *real* sphinx 
> extensions.
> 
> A few words about the flat-table extension and a (future) kernel-doc one:
> 
> The flat-table is a pure docutils (the layer below sphinx) extension which
> is not application specific, so I will ask for moving it to the docutils 
> upstream. 
> 
> The kernel-doc extension on the other side is a very (very) kernel specific
> application, this would never go to sphinx nor docutils upstream.
> 
> --Markus--
> 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hy Jani,

Am 04.05.2016 um 18:13 schrieb Jani Nikula :

>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>> 
>>> I think all of this makes sense.  It would be really nice to have the
>>> directives in the native sphinx language like that.  I *don't* think we
>>> need to aim for that at the outset; the docproc approach works until we can
>>> properly get rid of it.  What would be *really* nice would be to get
>>> support for the kernel-doc directive into the sphinx upstream.
>> 
>> No need for kernel-doc directive in sphinx upstream, later it will be 
>> an extension which could be installed by a simple command like 
>> "pip install kernel-doc-extensions" or similar.
>> 
>> I develop these required extension (and more) within my proof of concept
>> on github ... this takes time ... if I finished all my tests and all is
>> well, I will build the *kernel-doc-extensions* package and deploy it
>> on https://pypi.python.org/pypi from where everyone could install this 
>> with "pip".
> 
> I think we should go for vanilla sphinx at first, to make the setup step
> as easy as possible for everyone. Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

+1 

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Thats all right. Lets talk about the extension infrastructure by example:

First we have to chose a folder where we place all the *sphinx-documentation*
I recommending:

 /share/linux/Documentation/sphinx

Next we have to chose a folder where reST-extensions should take place, I
would prefer ... or similar:
 
 /share/linux/Documentation/sphinx/extensions

Lets say, you wan't to get in use of the "flat-table" extension.

Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
other extensions which might be there) ...

 https://github.com/return42/sphkerneldoc/tree/master/doc/extensions

Now lets say you are writing on a gpu book, it wold be placed in the folder:

 /share/linux/Documentation/sphinx/gpu

In this gpu-folder you have to place the conf.py config file, needed to
setup the sphinx build environment.

 /share/linux/Documentation/sphinx/gpu/conf.py

In this conf.py you have to *register* your folder with the extensions.

 

import os.path, sys

EXT_PATH  = "../extensions"  # the path of extension folder relative to the 
conf.py file
sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))

# now import the "flat-table" extension, it will be self-registering to 
docutils

import rstFlatTable

 

Thats all, you can run your sphinx-build command and the flat-tables in your
reST sources should be handled as common tables.

ASIDE: 

You will find similar parts in your conf.py which you have created 
with the sphinx-quickstart command. There, you will also find a block 
looks like ...

extensions = [
'sphinx.ext.autodoc'

]

Don't try to add flat-table extension to this list. This list is a list
of sphinx extensions, we will use it later for other *real* sphinx 
extensions.

A few words about the flat-table extension and a (future) kernel-doc one:

The flat-table is a pure docutils (the layer below sphinx) extension which
is not application specific, so I will ask for moving it to the docutils 
upstream. 

The kernel-doc extension on the other side is a very (very) kernel specific
application, this would never go to sphinx nor docutils upstream.

--Markus--

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hy Jani,

Am 04.05.2016 um 18:13 schrieb Jani Nikula :

>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>> 
>>> I think all of this makes sense.  It would be really nice to have the
>>> directives in the native sphinx language like that.  I *don't* think we
>>> need to aim for that at the outset; the docproc approach works until we can
>>> properly get rid of it.  What would be *really* nice would be to get
>>> support for the kernel-doc directive into the sphinx upstream.
>> 
>> No need for kernel-doc directive in sphinx upstream, later it will be 
>> an extension which could be installed by a simple command like 
>> "pip install kernel-doc-extensions" or similar.
>> 
>> I develop these required extension (and more) within my proof of concept
>> on github ... this takes time ... if I finished all my tests and all is
>> well, I will build the *kernel-doc-extensions* package and deploy it
>> on https://pypi.python.org/pypi from where everyone could install this 
>> with "pip".
> 
> I think we should go for vanilla sphinx at first, to make the setup step
> as easy as possible for everyone. Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

+1 

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Thats all right. Lets talk about the extension infrastructure by example:

First we have to chose a folder where we place all the *sphinx-documentation*
I recommending:

 /share/linux/Documentation/sphinx

Next we have to chose a folder where reST-extensions should take place, I
would prefer ... or similar:
 
 /share/linux/Documentation/sphinx/extensions

Lets say, you wan't to get in use of the "flat-table" extension.

Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
other extensions which might be there) ...

 https://github.com/return42/sphkerneldoc/tree/master/doc/extensions

Now lets say you are writing on a gpu book, it wold be placed in the folder:

 /share/linux/Documentation/sphinx/gpu

In this gpu-folder you have to place the conf.py config file, needed to
setup the sphinx build environment.

 /share/linux/Documentation/sphinx/gpu/conf.py

In this conf.py you have to *register* your folder with the extensions.

 

import os.path, sys

EXT_PATH  = "../extensions"  # the path of extension folder relative to the 
conf.py file
sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))

# now import the "flat-table" extension, it will be self-registering to 
docutils

import rstFlatTable

 

Thats all, you can run your sphinx-build command and the flat-tables in your
reST sources should be handled as common tables.

ASIDE: 

You will find similar parts in your conf.py which you have created 
with the sphinx-quickstart command. There, you will also find a block 
looks like ...

extensions = [
'sphinx.ext.autodoc'

]

Don't try to add flat-table extension to this list. This list is a list
of sphinx extensions, we will use it later for other *real* sphinx 
extensions.

A few words about the flat-table extension and a (future) kernel-doc one:

The flat-table is a pure docutils (the layer below sphinx) extension which
is not application specific, so I will ask for moving it to the docutils 
upstream. 

The kernel-doc extension on the other side is a very (very) kernel specific
application, this would never go to sphinx nor docutils upstream.

--Markus--

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro,

Am 04.05.2016 um 18:15 schrieb Mauro Carvalho Chehab :

> Em Wed, 4 May 2016 11:34:08 +0200
> Markus Heiser  escreveu:
> 
>> Hi all, (hi Jonathan, please take note of my offer below)
>> 
>> Am 03.05.2016 um 16:31 schrieb Daniel Vetter :
>> 
>>> Hi all,
>>> 
>>> So sounds like moving ahead with rst/sphinx is the option that should
>>> allow us to address everyone's concerns eventually? Of course the
>>> first one won't have it all (media seems really tricky), ...  
>> 
>> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 
>> For this, I wrote the "flat-table" reST-directive, which adds 
>> missing cells automatically:
>> 
>> doc:
>> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
>> source: 
>> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py
> 
> Yeah, this should address the lack of a proper way to markup cell/row
> spans, providing the additional bits for the tables we have at media.
> 
> Yet, there are some issues with table conversions. See below.
> Some tables, like the one here:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html
> 
> are truncated (tested with Mozilla and Chrome), and part of the information is
> lost due to that.

Not a problem of rendering. This was a bug in the magration from DocBook to 
reST. 
You might remember that we have discussed, that some of the tables are better
marked-up as definition lists. This was (the last) one I forgot to convert to 
a definition list ... I hope it was the last one, if not and you find more 
or other broken parts, please inform me (on the linux-tv mailing, or direct).

--Markus--

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro,

Am 04.05.2016 um 18:15 schrieb Mauro Carvalho Chehab :

> Em Wed, 4 May 2016 11:34:08 +0200
> Markus Heiser  escreveu:
> 
>> Hi all, (hi Jonathan, please take note of my offer below)
>> 
>> Am 03.05.2016 um 16:31 schrieb Daniel Vetter :
>> 
>>> Hi all,
>>> 
>>> So sounds like moving ahead with rst/sphinx is the option that should
>>> allow us to address everyone's concerns eventually? Of course the
>>> first one won't have it all (media seems really tricky), ...  
>> 
>> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 
>> For this, I wrote the "flat-table" reST-directive, which adds 
>> missing cells automatically:
>> 
>> doc:
>> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
>> source: 
>> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py
> 
> Yeah, this should address the lack of a proper way to markup cell/row
> spans, providing the additional bits for the tables we have at media.
> 
> Yet, there are some issues with table conversions. See below.
> Some tables, like the one here:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html
> 
> are truncated (tested with Mozilla and Chrome), and part of the information is
> lost due to that.

Not a problem of rendering. This was a bug in the magration from DocBook to 
reST. 
You might remember that we have discussed, that some of the tables are better
marked-up as definition lists. This was (the last) one I forgot to convert to 
a definition list ... I hope it was the last one, if not and you find more 
or other broken parts, please inform me (on the linux-tv mailing, or direct).

--Markus--

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Thu, 5 May 2016 07:02:10 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 14:57:38 -0300
> Mauro Carvalho Chehab  wrote:
> 
> > Also, media documentation is not just one more documentation. It is
> > the biggest one we have, and that has more changes than any other
> > documentation under Documentation/DocBook:
> > 
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> > 116
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type 
> > f`|wc -l
> > 179
> > 
> > It also is more than twice the size of the other DocBook docs:
> > 
> > $ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
> >   82275 total
> > $ wc -l $(ls *.tmpl|grep -v media)|tail -1
> >   29568 total
> > 
> > E. g. media corresponds to 60% of the number of patches and 73% of
> > the DocBook stuff.  
> 
> These numbers are not entirely representative, I have to say.  You're
> ignoring the kerneldoc comments - which is what much of the "DocBook"
> documents are made of, and which is the focus of much of this activity.  If
> you could find a way to count those, you'd get a different picture.

Yeah, if we take the big picture, I'm pretty sure that there are more
stuff written using kerneldoc. However, what is written at the 
kerneldoc comments don't use DocBook markup, but its own markup
language.

Assuming that we'll keep using kerneldoc script to convert from the
Kerneldoc NANO markup, it means that any changes at the kerneldoc
backend to use DocBook/Markdown/asciidoc/reST/... won't change anything
for the developers nor will require them to actually use the new markups.
So, they can gradually start using new markups as they wish/learn, if
they want to improve the documentation. Lazy developers could just ignore
reST if they want, as everything will work as before.

However, for the files written directly in DocBook, there's no option:
at the moment the doc is converted, the developer will need to submit
patches using the new markup language. So, we need to be sure that
such transition will happen in a way that won't cause more harm than
needed.  I would prefer to convert all of them at one, on a single Kernel
release (or two), in order to minimize the impact for the developers.

> 
> But I don't think that really matters; there doesn't seem to be *that* much
> disagreement here.
> 
> The media book is important; we want it to be a part of the overall kernel
> documentation suite and not stuck in some DocBook ghetto.  I agree that we
> should have an idea for a plausible path for *all* of our documentation.
> But I'm also concerned about delaying this work yet again; we have
> developers trying to push forward with improved documentation, and they've
> had to wait a year for this stuff - so far.

Yeah, I understand.

Regards,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Thu, 5 May 2016 07:02:10 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 14:57:38 -0300
> Mauro Carvalho Chehab  wrote:
> 
> > Also, media documentation is not just one more documentation. It is
> > the biggest one we have, and that has more changes than any other
> > documentation under Documentation/DocBook:
> > 
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> > 116
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type 
> > f`|wc -l
> > 179
> > 
> > It also is more than twice the size of the other DocBook docs:
> > 
> > $ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
> >   82275 total
> > $ wc -l $(ls *.tmpl|grep -v media)|tail -1
> >   29568 total
> > 
> > E. g. media corresponds to 60% of the number of patches and 73% of
> > the DocBook stuff.  
> 
> These numbers are not entirely representative, I have to say.  You're
> ignoring the kerneldoc comments - which is what much of the "DocBook"
> documents are made of, and which is the focus of much of this activity.  If
> you could find a way to count those, you'd get a different picture.

Yeah, if we take the big picture, I'm pretty sure that there are more
stuff written using kerneldoc. However, what is written at the 
kerneldoc comments don't use DocBook markup, but its own markup
language.

Assuming that we'll keep using kerneldoc script to convert from the
Kerneldoc NANO markup, it means that any changes at the kerneldoc
backend to use DocBook/Markdown/asciidoc/reST/... won't change anything
for the developers nor will require them to actually use the new markups.
So, they can gradually start using new markups as they wish/learn, if
they want to improve the documentation. Lazy developers could just ignore
reST if they want, as everything will work as before.

However, for the files written directly in DocBook, there's no option:
at the moment the doc is converted, the developer will need to submit
patches using the new markup language. So, we need to be sure that
such transition will happen in a way that won't cause more harm than
needed.  I would prefer to convert all of them at one, on a single Kernel
release (or two), in order to minimize the impact for the developers.

> 
> But I don't think that really matters; there doesn't seem to be *that* much
> disagreement here.
> 
> The media book is important; we want it to be a part of the overall kernel
> documentation suite and not stuck in some DocBook ghetto.  I agree that we
> should have an idea for a plausible path for *all* of our documentation.
> But I'm also concerned about delaying this work yet again; we have
> developers trying to push forward with improved documentation, and they've
> had to wait a year for this stuff - so far.

Yeah, I understand.

Regards,
Mauro

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 14:57:38 -0300
Mauro Carvalho Chehab  wrote:

> Also, media documentation is not just one more documentation. It is
> the biggest one we have, and that has more changes than any other
> documentation under Documentation/DocBook:
> 
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> 116
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc 
> -l
> 179
> 
> It also is more than twice the size of the other DocBook docs:
> 
> $ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
>   82275 total
> $ wc -l $(ls *.tmpl|grep -v media)|tail -1
>   29568 total
> 
> E. g. media corresponds to 60% of the number of patches and 73% of
> the DocBook stuff.

These numbers are not entirely representative, I have to say.  You're
ignoring the kerneldoc comments - which is what much of the "DocBook"
documents are made of, and which is the focus of much of this activity.  If
you could find a way to count those, you'd get a different picture.

But I don't think that really matters; there doesn't seem to be *that* much
disagreement here.

The media book is important; we want it to be a part of the overall kernel
documentation suite and not stuck in some DocBook ghetto.  I agree that we
should have an idea for a plausible path for *all* of our documentation.
But I'm also concerned about delaying this work yet again; we have
developers trying to push forward with improved documentation, and they've
had to wait a year for this stuff - so far.

Thanks,

jon

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 14:57:38 -0300
Mauro Carvalho Chehab  wrote:

> Also, media documentation is not just one more documentation. It is
> the biggest one we have, and that has more changes than any other
> documentation under Documentation/DocBook:
> 
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> 116
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc 
> -l
> 179
> 
> It also is more than twice the size of the other DocBook docs:
> 
> $ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
>   82275 total
> $ wc -l $(ls *.tmpl|grep -v media)|tail -1
>   29568 total
> 
> E. g. media corresponds to 60% of the number of patches and 73% of
> the DocBook stuff.

These numbers are not entirely representative, I have to say.  You're
ignoring the kerneldoc comments - which is what much of the "DocBook"
documents are made of, and which is the focus of much of this activity.  If
you could find a way to count those, you'd get a different picture.

But I don't think that really matters; there doesn't seem to be *that* much
disagreement here.

The media book is important; we want it to be a part of the overall kernel
documentation suite and not stuck in some DocBook ghetto.  I agree that we
should have an idea for a plausible path for *all* of our documentation.
But I'm also concerned about delaying this work yet again; we have
developers trying to push forward with improved documentation, and they've
had to wait a year for this stuff - so far.

Thanks,

jon

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 10:59:36 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 13:50:35 -0300
> Mauro Carvalho Chehab  wrote:
> 
> > Em Wed, 4 May 2016 19:13:21 +0300
> > Jani Nikula  escreveu:  
> > > I think we should go for vanilla sphinx at first, to make the setup step
> > > as easy as possible for everyone.
> > 
> > Vanilla Sphinx doesn't work, as reST markup language is too limited
> > to support the docs we have. So, we should either push the needed
> > extensions to Sphinx reST or find a way to put it at Kernel tree
> > without causing too much pain for the developers, e. g. as something
> > that just doing "make htmldoc" would automatically use such extensions
> > without needing to actually install the extensions.  
> 
> It works for everything except the extended media book, right?  Or is there
> something I'm missing here?

As far as I know, yes, but I didn't check the other documentation
to be sure.

>  I am very much interested in having one system
> for *all* our docs, but we wouldn't attempt to convert a document can't be
> expressed in sphinx.

Looking from the other side, keeping only one document as DocBook
and having everything else converted to some markup solution seem
equally a bad solution, as, in the end of the day, we'll become
dependent of two different tool chains to produce document.

Also, media documentation is not just one more documentation. It is
the biggest one we have, and that has more changes than any other
documentation under Documentation/DocBook:

$ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
116
$ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l
179

It also is more than twice the size of the other DocBook docs:

$ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
  82275 total
$ wc -l $(ls *.tmpl|grep -v media)|tail -1
  29568 total

E. g. media corresponds to 60% of the number of patches and 73% of
the DocBook stuff.

Not converting it to the new "official" Kernel markup language
would would mean that developers that work on more than the
media subsystem would need to know two different, incompatible
dialects to produce documentation (plus kernel-doc). That sounds
messy and it is an extra penalty to media developers. Some may
even start to think that that writing bad docs would be a good thing.

> > > Even if it means still doing that ugly
> > > docproc step to call kernel-doc. We can improve from there, and I
> > > definitely appreciate your work on making this work with sphinx
> > > extensions.
> > 
> > I disagree: We should not cause documentation regressions by
> > producing incomplete documentation and losing tables because of
> > such conversion.  
> 
> > The quality of the documentation after the change should be *at least*
> > equal to the one we current produce, never worse.  
> 
> Agreed; that's why I think the existing DocBook mechanism should stay in
> place until that document will be improved by the change. But I'd rather
> not hold up the entire process for one book, especially since pushing that
> process forward can only help in dealing with those final problems.

Yeah, in practice, we won't be converting everything on just one
Kernel version. Yet, we need at least an strategy and a plan that would
allow converting everything to the new markup language.

Provided that we would merge Sphinx support on 4.8, I would like to convert
the media documentation just after merging the new toolchain upstream, 
in order to minimize the conversion impact and to not lose developers
and good developer's documentation in the process.

However, such work can only be started if we have a way to work with the
required extensions, as we're deciding to use a markup language that has
incomplete table support.

So, IMHO, we should have a way to run the required extensions in a
painless way before doing the merge.

> > > That said, how would it work to include the kernel-doc extension in the
> > > kernel source tree? Having things just work if sphinx is installed is
> > > preferred over requiring installation of something extra from pypi. (I
> > > know this may sound backwards for a lot of projects, but for kernel I'm
> > > pretty sure this is how it should be done.)
> > 
> > Yeah, using pypi seems an additional undesired step. Aren't there
> > any way to make python to use an additional extension at the
> > Kernel tree without needing to install it?  
> 
> Well, sphinx has to come from somewhere too.  But yes, we should be able to
> carry extensions in the kernel tree.  But I would still rather upstream it
> (to sphinx) if possible, so we're not stuck trying to maintain it across
> several sphinx releases.  I don't know how volatile their interfaces are,
> but it would be, in any case, the analog of an out-of-tree module...

Yeah, the best would be to put everything we need at Sphinx upstream,
but assuming that they agree today to add 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 10:59:36 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 13:50:35 -0300
> Mauro Carvalho Chehab  wrote:
> 
> > Em Wed, 4 May 2016 19:13:21 +0300
> > Jani Nikula  escreveu:  
> > > I think we should go for vanilla sphinx at first, to make the setup step
> > > as easy as possible for everyone.
> > 
> > Vanilla Sphinx doesn't work, as reST markup language is too limited
> > to support the docs we have. So, we should either push the needed
> > extensions to Sphinx reST or find a way to put it at Kernel tree
> > without causing too much pain for the developers, e. g. as something
> > that just doing "make htmldoc" would automatically use such extensions
> > without needing to actually install the extensions.  
> 
> It works for everything except the extended media book, right?  Or is there
> something I'm missing here?

As far as I know, yes, but I didn't check the other documentation
to be sure.

>  I am very much interested in having one system
> for *all* our docs, but we wouldn't attempt to convert a document can't be
> expressed in sphinx.

Looking from the other side, keeping only one document as DocBook
and having everything else converted to some markup solution seem
equally a bad solution, as, in the end of the day, we'll become
dependent of two different tool chains to produce document.

Also, media documentation is not just one more documentation. It is
the biggest one we have, and that has more changes than any other
documentation under Documentation/DocBook:

$ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
116
$ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l
179

It also is more than twice the size of the other DocBook docs:

$ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
  82275 total
$ wc -l $(ls *.tmpl|grep -v media)|tail -1
  29568 total

E. g. media corresponds to 60% of the number of patches and 73% of
the DocBook stuff.

Not converting it to the new "official" Kernel markup language
would would mean that developers that work on more than the
media subsystem would need to know two different, incompatible
dialects to produce documentation (plus kernel-doc). That sounds
messy and it is an extra penalty to media developers. Some may
even start to think that that writing bad docs would be a good thing.

> > > Even if it means still doing that ugly
> > > docproc step to call kernel-doc. We can improve from there, and I
> > > definitely appreciate your work on making this work with sphinx
> > > extensions.
> > 
> > I disagree: We should not cause documentation regressions by
> > producing incomplete documentation and losing tables because of
> > such conversion.  
> 
> > The quality of the documentation after the change should be *at least*
> > equal to the one we current produce, never worse.  
> 
> Agreed; that's why I think the existing DocBook mechanism should stay in
> place until that document will be improved by the change. But I'd rather
> not hold up the entire process for one book, especially since pushing that
> process forward can only help in dealing with those final problems.

Yeah, in practice, we won't be converting everything on just one
Kernel version. Yet, we need at least an strategy and a plan that would
allow converting everything to the new markup language.

Provided that we would merge Sphinx support on 4.8, I would like to convert
the media documentation just after merging the new toolchain upstream, 
in order to minimize the conversion impact and to not lose developers
and good developer's documentation in the process.

However, such work can only be started if we have a way to work with the
required extensions, as we're deciding to use a markup language that has
incomplete table support.

So, IMHO, we should have a way to run the required extensions in a
painless way before doing the merge.

> > > That said, how would it work to include the kernel-doc extension in the
> > > kernel source tree? Having things just work if sphinx is installed is
> > > preferred over requiring installation of something extra from pypi. (I
> > > know this may sound backwards for a lot of projects, but for kernel I'm
> > > pretty sure this is how it should be done.)
> > 
> > Yeah, using pypi seems an additional undesired step. Aren't there
> > any way to make python to use an additional extension at the
> > Kernel tree without needing to install it?  
> 
> Well, sphinx has to come from somewhere too.  But yes, we should be able to
> carry extensions in the kernel tree.  But I would still rather upstream it
> (to sphinx) if possible, so we're not stuck trying to maintain it across
> several sphinx releases.  I don't know how volatile their interfaces are,
> but it would be, in any case, the analog of an out-of-tree module...

Yeah, the best would be to put everything we need at Sphinx upstream,
but assuming that they agree today to add the extensions we need,
we would still need to carry those 

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 13:50:35 -0300
Mauro Carvalho Chehab  wrote:

> Em Wed, 4 May 2016 19:13:21 +0300
> Jani Nikula  escreveu:
> > I think we should go for vanilla sphinx at first, to make the setup step
> > as easy as possible for everyone.  
> 
> Vanilla Sphinx doesn't work, as reST markup language is too limited
> to support the docs we have. So, we should either push the needed
> extensions to Sphinx reST or find a way to put it at Kernel tree
> without causing too much pain for the developers, e. g. as something
> that just doing "make htmldoc" would automatically use such extensions
> without needing to actually install the extensions.

It works for everything except the extended media book, right?  Or is there
something I'm missing here?  I am very much interested in having one system
for *all* our docs, but we wouldn't attempt to convert a document can't be
expressed in sphinx.

> > Even if it means still doing that ugly
> > docproc step to call kernel-doc. We can improve from there, and I
> > definitely appreciate your work on making this work with sphinx
> > extensions.  
> 
> I disagree: We should not cause documentation regressions by
> producing incomplete documentation and losing tables because of
> such conversion.

> The quality of the documentation after the change should be *at least*
> equal to the one we current produce, never worse.

Agreed; that's why I think the existing DocBook mechanism should stay in
place until that document will be improved by the change.  But I'd rather
not hold up the entire process for one book, especially since pushing that
process forward can only help in dealing with those final problems.

> > That said, how would it work to include the kernel-doc extension in the
> > kernel source tree? Having things just work if sphinx is installed is
> > preferred over requiring installation of something extra from pypi. (I
> > know this may sound backwards for a lot of projects, but for kernel I'm
> > pretty sure this is how it should be done.)  
> 
> Yeah, using pypi seems an additional undesired step. Aren't there
> any way to make python to use an additional extension at the
> Kernel tree without needing to install it?

Well, sphinx has to come from somewhere too.  But yes, we should be able to
carry extensions in the kernel tree.  But I would still rather upstream it
(to sphinx) if possible, so we're not stuck trying to maintain it across
several sphinx releases.  I don't know how volatile their interfaces are,
but it would be, in any case, the analog of an out-of-tree module...

jon

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 13:50:35 -0300
Mauro Carvalho Chehab  wrote:

> Em Wed, 4 May 2016 19:13:21 +0300
> Jani Nikula  escreveu:
> > I think we should go for vanilla sphinx at first, to make the setup step
> > as easy as possible for everyone.  
> 
> Vanilla Sphinx doesn't work, as reST markup language is too limited
> to support the docs we have. So, we should either push the needed
> extensions to Sphinx reST or find a way to put it at Kernel tree
> without causing too much pain for the developers, e. g. as something
> that just doing "make htmldoc" would automatically use such extensions
> without needing to actually install the extensions.

It works for everything except the extended media book, right?  Or is there
something I'm missing here?  I am very much interested in having one system
for *all* our docs, but we wouldn't attempt to convert a document can't be
expressed in sphinx.

> > Even if it means still doing that ugly
> > docproc step to call kernel-doc. We can improve from there, and I
> > definitely appreciate your work on making this work with sphinx
> > extensions.  
> 
> I disagree: We should not cause documentation regressions by
> producing incomplete documentation and losing tables because of
> such conversion.

> The quality of the documentation after the change should be *at least*
> equal to the one we current produce, never worse.

Agreed; that's why I think the existing DocBook mechanism should stay in
place until that document will be improved by the change.  But I'd rather
not hold up the entire process for one book, especially since pushing that
process forward can only help in dealing with those final problems.

> > That said, how would it work to include the kernel-doc extension in the
> > kernel source tree? Having things just work if sphinx is installed is
> > preferred over requiring installation of something extra from pypi. (I
> > know this may sound backwards for a lot of projects, but for kernel I'm
> > pretty sure this is how it should be done.)  
> 
> Yeah, using pypi seems an additional undesired step. Aren't there
> any way to make python to use an additional extension at the
> Kernel tree without needing to install it?

Well, sphinx has to come from somewhere too.  But yes, we should be able to
carry extensions in the kernel tree.  But I would still rather upstream it
(to sphinx) if possible, so we're not stuck trying to maintain it across
several sphinx releases.  I don't know how volatile their interfaces are,
but it would be, in any case, the analog of an out-of-tree module...

jon

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 19:13:21 +0300
Jani Nikula  escreveu:

> On Wed, 04 May 2016, Markus Heiser  wrote:
> > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
> > particular with your "MARKDOWNREADY := gpu.xml" process.
> >
> > As I understood, you convert markdown to docbook within the kernel-doc 
> > script using pandoc's executable? ... I don't face this topic. With my 
> > modification of kernel-doc I produced pure reST markup from standardize
> > kernel-doc markup, no intermediate steps or tools required.  
> 
> That pandoc thing is a dead end. Forget about it. I think we've all
> pretty much agreed we should have kernel-doc produce the lightweight
> markup directly since [1].
> 
> [1] 
> http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com
> 
> > Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
> >  
> >> I think all of this makes sense.  It would be really nice to have the
> >> directives in the native sphinx language like that.  I *don't* think we
> >> need to aim for that at the outset; the docproc approach works until we can
> >> properly get rid of it.  What would be *really* nice would be to get
> >> support for the kernel-doc directive into the sphinx upstream.  
> >
> > No need for kernel-doc directive in sphinx upstream, later it will be 
> > an extension which could be installed by a simple command like 
> > "pip install kernel-doc-extensions" or similar.
> >
> > I develop these required extension (and more) within my proof of concept
> > on github ... this takes time ... if I finished all my tests and all is
> > well, I will build the *kernel-doc-extensions* package and deploy it
> > on https://pypi.python.org/pypi from where everyone could install this 
> > with "pip".  
> 
> I think we should go for vanilla sphinx at first, to make the setup step
> as easy as possible for everyone.

Vanilla Sphinx doesn't work, as reST markup language is too limited
to support the docs we have. So, we should either push the needed
extensions to Sphinx reST or find a way to put it at Kernel tree
without causing too much pain for the developers, e. g. as something
that just doing "make htmldoc" would automatically use such extensions
without needing to actually install the extensions.

> Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

I disagree: We should not cause documentation regressions by
producing incomplete documentation and losing tables because of
such conversion.

The quality of the documentation after the change should be *at least*
equal to the one we current produce, never worse.

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Yeah, using pypi seems an additional undesired step. Aren't there
any way to make python to use an additional extension at the
Kernel tree without needing to install it?

Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 19:13:21 +0300
Jani Nikula  escreveu:

> On Wed, 04 May 2016, Markus Heiser  wrote:
> > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
> > particular with your "MARKDOWNREADY := gpu.xml" process.
> >
> > As I understood, you convert markdown to docbook within the kernel-doc 
> > script using pandoc's executable? ... I don't face this topic. With my 
> > modification of kernel-doc I produced pure reST markup from standardize
> > kernel-doc markup, no intermediate steps or tools required.  
> 
> That pandoc thing is a dead end. Forget about it. I think we've all
> pretty much agreed we should have kernel-doc produce the lightweight
> markup directly since [1].
> 
> [1] 
> http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com
> 
> > Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
> >  
> >> I think all of this makes sense.  It would be really nice to have the
> >> directives in the native sphinx language like that.  I *don't* think we
> >> need to aim for that at the outset; the docproc approach works until we can
> >> properly get rid of it.  What would be *really* nice would be to get
> >> support for the kernel-doc directive into the sphinx upstream.  
> >
> > No need for kernel-doc directive in sphinx upstream, later it will be 
> > an extension which could be installed by a simple command like 
> > "pip install kernel-doc-extensions" or similar.
> >
> > I develop these required extension (and more) within my proof of concept
> > on github ... this takes time ... if I finished all my tests and all is
> > well, I will build the *kernel-doc-extensions* package and deploy it
> > on https://pypi.python.org/pypi from where everyone could install this 
> > with "pip".  
> 
> I think we should go for vanilla sphinx at first, to make the setup step
> as easy as possible for everyone.

Vanilla Sphinx doesn't work, as reST markup language is too limited
to support the docs we have. So, we should either push the needed
extensions to Sphinx reST or find a way to put it at Kernel tree
without causing too much pain for the developers, e. g. as something
that just doing "make htmldoc" would automatically use such extensions
without needing to actually install the extensions.

> Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

I disagree: We should not cause documentation regressions by
producing incomplete documentation and losing tables because of
such conversion.

The quality of the documentation after the change should be *at least*
equal to the one we current produce, never worse.

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Yeah, using pypi seems an additional undesired step. Aren't there
any way to make python to use an additional extension at the
Kernel tree without needing to install it?

Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 08:57:13 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter  wrote:
> 
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.
> > 
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...  
> 
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home.  I hope to begin emerging from this mess in the
> near future.
> 
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done.  Which would you have me push into 4.7?

IMHO, Markus approach is the one that is providing a better result,
as it added support for missing features that we require for the
media DocBook.

Still, it seems premature to merge it for 4.7.

Markus is getting real progress on converting the media docs to work
with Sphynx, but there are still troubles when the table is big, as,
currently, it is truncating everything that passes the right margin
on some tables.

So, IMHO, the next action item would be to be sure that big tables 
will not be truncated. Assuming that he can fix that in time, we
can merge it for 4.8, and then start porting the documentation to
use it.

Btw, converting the DocBook/media Makefile will require an extra
step, as part of the documentation is generated via scripts (some
C file examples, and include/uapi headers). Those scripts also
warrant that (almost) every API at include/uapi is synchronized
with the DocBook. I use this on my patch review process, in order
to reject patches that don't add the proper documentation updates.

Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 08:57:13 -0600
Jonathan Corbet  escreveu:

> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter  wrote:
> 
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.
> > 
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...  
> 
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home.  I hope to begin emerging from this mess in the
> near future.
> 
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done.  Which would you have me push into 4.7?

IMHO, Markus approach is the one that is providing a better result,
as it added support for missing features that we require for the
media DocBook.

Still, it seems premature to merge it for 4.7.

Markus is getting real progress on converting the media docs to work
with Sphynx, but there are still troubles when the table is big, as,
currently, it is truncating everything that passes the right margin
on some tables.

So, IMHO, the next action item would be to be sure that big tables 
will not be truncated. Assuming that he can fix that in time, we
can merge it for 4.8, and then start porting the documentation to
use it.

Btw, converting the DocBook/media Makefile will require an extra
step, as part of the documentation is generated via scripts (some
C file examples, and include/uapi headers). Those scripts also
warrant that (almost) every API at include/uapi is synchronized
with the DocBook. I use this on my patch review process, in order
to reject patches that don't add the proper documentation updates.

Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 4, 2016 at 5:02 PM, Daniel Vetter  wrote:
> On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote:
>> On Wed, 4 May 2016 16:18:27 +0200
>> Daniel Vetter  wrote:
>>
>> > > I'd really like to converge on the markup question, so that we can start
>> > > using all the cool stuff with impunity in gpu documentations.
>> >
>> > Aside: If we decide this now I could send in a pull request for the
>> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
>> > markdown/asciidoc code I still have). That would be really awesome ...
>>
>> Sorry for my relative absence...I'm still busy dealing with bureaucracy
>> an ocean away from home.  I hope to begin emerging from this mess in the
>> near future.
>>
>> So ... there's the code you have, the work I (+Jani) did, and the work
>> Markus has done.  Which would you have me push into 4.7?
>>
>> The sphinx/rst approach does seem, to me, to be the right one, with the
>> existing DocBook structure remaining in place for those who want/need
>> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
>> it's hackish at best and needs a lot of cleaning up.  It's a proof of
>> concept, but it's hardly finished (one might say it's barely begun...)
>>
>> In the end, I guess, I feel that anything we might try to push for 4.7 is
>> going to look rushed and not ready, and Linus might react accordingly.
>> I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
>> on things in that time frame...  I suspect you're pretty well fed up with
>> this stuff being pushed back, and rightly so.  All I can do is apologize.
>>
>> That said, if you do think there's something out there that is good
>> enough to consider pushing in a week or two, do tell and we can all take
>> a look.
>
> Well I'd just have taken the asciidoc hacks I have currently in my
> topic/kerneldoc branch, converted to sphinx and looked how it fares. It
> should be fairly minimal, and I think the first step we want to do for the
> long-term plan. I hope I can ready something, and then we can look whether
> it's rushed for 4.7 or not.

Ok, discussed this a bit more with Jani on IRC and he really doesn't
like the old design of that branch (it calls the converter for every
kerneldoc comment). So I guess nothing rushed for 4.7, but hopefully
something for 4.8.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 4, 2016 at 5:02 PM, Daniel Vetter  wrote:
> On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote:
>> On Wed, 4 May 2016 16:18:27 +0200
>> Daniel Vetter  wrote:
>>
>> > > I'd really like to converge on the markup question, so that we can start
>> > > using all the cool stuff with impunity in gpu documentations.
>> >
>> > Aside: If we decide this now I could send in a pull request for the
>> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
>> > markdown/asciidoc code I still have). That would be really awesome ...
>>
>> Sorry for my relative absence...I'm still busy dealing with bureaucracy
>> an ocean away from home.  I hope to begin emerging from this mess in the
>> near future.
>>
>> So ... there's the code you have, the work I (+Jani) did, and the work
>> Markus has done.  Which would you have me push into 4.7?
>>
>> The sphinx/rst approach does seem, to me, to be the right one, with the
>> existing DocBook structure remaining in place for those who want/need
>> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
>> it's hackish at best and needs a lot of cleaning up.  It's a proof of
>> concept, but it's hardly finished (one might say it's barely begun...)
>>
>> In the end, I guess, I feel that anything we might try to push for 4.7 is
>> going to look rushed and not ready, and Linus might react accordingly.
>> I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
>> on things in that time frame...  I suspect you're pretty well fed up with
>> this stuff being pushed back, and rightly so.  All I can do is apologize.
>>
>> That said, if you do think there's something out there that is good
>> enough to consider pushing in a week or two, do tell and we can all take
>> a look.
>
> Well I'd just have taken the asciidoc hacks I have currently in my
> topic/kerneldoc branch, converted to sphinx and looked how it fares. It
> should be fairly minimal, and I think the first step we want to do for the
> long-term plan. I hope I can ready something, and then we can look whether
> it's rushed for 4.7 or not.

Ok, discussed this a bit more with Jani on IRC and he really doesn't
like the old design of that branch (it calls the converter for every
kerneldoc comment). So I guess nothing rushed for 4.7, but hopefully
something for 4.8.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 11:34:08 +0200
Markus Heiser  escreveu:

> Hi all, (hi Jonathan, please take note of my offer below)
> 
> Am 03.05.2016 um 16:31 schrieb Daniel Vetter :
> 
> > Hi all,
> > 
> > So sounds like moving ahead with rst/sphinx is the option that should
> > allow us to address everyone's concerns eventually? Of course the
> > first one won't have it all (media seems really tricky), ...  
> 
> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 
> 
> Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab 
> :
> 
> > With that sense, the "List tables" format is also not good, as
> > one row addition would generate several hunks (one for each column
> > of the table), making harder to review the patch by just looking at
> > the diff.  
> 
> For this, I wrote the "flat-table" reST-directive, which adds 
> missing cells automatically:
> 
> doc:
> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
> source: 
> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

Yeah, this should address the lack of a proper way to markup cell/row
spans, providing the additional bits for the tables we have at media.

Yet, there are some issues with table conversions. See below.

> 
> I used "flat-table" to migrate all DocBook-XML documents to reST. With this
> directive, I also managed to migrate the complete media book (no more TODOs)
> incl. the large tables like them from subdev-formats:
> 
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 
> (Rendering large tables is a general discussion which should not take place 
> in this MT)

Some tables, like the one here:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html

are truncated (tested with Mozilla and Chrome), and part of the information is
lost due to that.

Regards,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 11:34:08 +0200
Markus Heiser  escreveu:

> Hi all, (hi Jonathan, please take note of my offer below)
> 
> Am 03.05.2016 um 16:31 schrieb Daniel Vetter :
> 
> > Hi all,
> > 
> > So sounds like moving ahead with rst/sphinx is the option that should
> > allow us to address everyone's concerns eventually? Of course the
> > first one won't have it all (media seems really tricky), ...  
> 
> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 
> 
> Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab 
> :
> 
> > With that sense, the "List tables" format is also not good, as
> > one row addition would generate several hunks (one for each column
> > of the table), making harder to review the patch by just looking at
> > the diff.  
> 
> For this, I wrote the "flat-table" reST-directive, which adds 
> missing cells automatically:
> 
> doc:
> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
> source: 
> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

Yeah, this should address the lack of a proper way to markup cell/row
spans, providing the additional bits for the tables we have at media.

Yet, there are some issues with table conversions. See below.

> 
> I used "flat-table" to migrate all DocBook-XML documents to reST. With this
> directive, I also managed to migrate the complete media book (no more TODOs)
> incl. the large tables like them from subdev-formats:
> 
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 
> (Rendering large tables is a general discussion which should not take place 
> in this MT)

Some tables, like the one here:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html

are truncated (tested with Mozilla and Chrome), and part of the information is
lost due to that.

Regards,
Mauro

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
> Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
> particular with your "MARKDOWNREADY := gpu.xml" process.
>
> As I understood, you convert markdown to docbook within the kernel-doc 
> script using pandoc's executable? ... I don't face this topic. With my 
> modification of kernel-doc I produced pure reST markup from standardize
> kernel-doc markup, no intermediate steps or tools required.

That pandoc thing is a dead end. Forget about it. I think we've all
pretty much agreed we should have kernel-doc produce the lightweight
markup directly since [1].

[1] http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com

> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>
>> I think all of this makes sense.  It would be really nice to have the
>> directives in the native sphinx language like that.  I *don't* think we
>> need to aim for that at the outset; the docproc approach works until we can
>> properly get rid of it.  What would be *really* nice would be to get
>> support for the kernel-doc directive into the sphinx upstream.
>
> No need for kernel-doc directive in sphinx upstream, later it will be 
> an extension which could be installed by a simple command like 
> "pip install kernel-doc-extensions" or similar.
>
> I develop these required extension (and more) within my proof of concept
> on github ... this takes time ... if I finished all my tests and all is
> well, I will build the *kernel-doc-extensions* package and deploy it
> on https://pypi.python.org/pypi from where everyone could install this 
> with "pip".

I think we should go for vanilla sphinx at first, to make the setup step
as easy as possible for everyone. Even if it means still doing that ugly
docproc step to call kernel-doc. We can improve from there, and I
definitely appreciate your work on making this work with sphinx
extensions.

That said, how would it work to include the kernel-doc extension in the
kernel source tree? Having things just work if sphinx is installed is
preferred over requiring installation of something extra from pypi. (I
know this may sound backwards for a lot of projects, but for kernel I'm
pretty sure this is how it should be done.)

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
> Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
> particular with your "MARKDOWNREADY := gpu.xml" process.
>
> As I understood, you convert markdown to docbook within the kernel-doc 
> script using pandoc's executable? ... I don't face this topic. With my 
> modification of kernel-doc I produced pure reST markup from standardize
> kernel-doc markup, no intermediate steps or tools required.

That pandoc thing is a dead end. Forget about it. I think we've all
pretty much agreed we should have kernel-doc produce the lightweight
markup directly since [1].

[1] http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com

> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :
>
>> I think all of this makes sense.  It would be really nice to have the
>> directives in the native sphinx language like that.  I *don't* think we
>> need to aim for that at the outset; the docproc approach works until we can
>> properly get rid of it.  What would be *really* nice would be to get
>> support for the kernel-doc directive into the sphinx upstream.
>
> No need for kernel-doc directive in sphinx upstream, later it will be 
> an extension which could be installed by a simple command like 
> "pip install kernel-doc-extensions" or similar.
>
> I develop these required extension (and more) within my proof of concept
> on github ... this takes time ... if I finished all my tests and all is
> well, I will build the *kernel-doc-extensions* package and deploy it
> on https://pypi.python.org/pypi from where everyone could install this 
> with "pip".

I think we should go for vanilla sphinx at first, to make the setup step
as easy as possible for everyone. Even if it means still doing that ugly
docproc step to call kernel-doc. We can improve from there, and I
definitely appreciate your work on making this work with sphinx
extensions.

That said, how would it work to include the kernel-doc extension in the
kernel source tree? Having things just work if sphinx is installed is
preferred over requiring installation of something extra from pypi. (I
know this may sound backwards for a lot of projects, but for kernel I'm
pretty sure this is how it should be done.)

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 04.05.2016 um 15:43 schrieb Daniel Vetter :

> On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote:
>>> On Wed, 04 May 2016, Markus Heiser  wrote:
>>> I'd be *very* hesitant about adding the kind of things you do in
>>> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
>>> comments must be as simple as possible with basically pass-through of
>>> the comment blocks to sphinx.
>> 
>> Right, but you forgot, that the current markup in the source code comments
>> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
>> these source code comments to reST markup ;-)
>> 
>> So there is a need to differentiate between *vintage* kernel-doc markup 
>> and reST markup.
>> 
>> My suggestion is to add a tag to the comments, here a short example
>> what this might look like.
>> 
>>  --
>> /**
>> * drm_minor_release - Release DRM minor
>> * @minor: Pointer to DRM minor object
>> *
>> * Release a minor that was previously acquired via drm_minor_acquire().
>> */
>>  --
>> 
>> in short: the vintage style does not need any change and 
>> comments with reST markup has a tag ":reST:" in the first 
>> line(s) ...
>> 
>>  --
>> /**
>> * :reST:
>> *
>> * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>> *
>> *Release DRM minor
>> *
>> *:param struct drm_minor \*minor: Pointer to DRM minor object
>> *
>> */
>>  --
>> 
>> Comments with the ":reST:" tag could be exported and pass-through
>> to sphinx.
>> 
>>> Specifically, do not attempt to detect and
>>> parse elements like lists in kernel-doc.
>> 
>> If your markup in the comments is plain reST, no need to parse, but there
>> are markups in the (vintage) comments which made use of individual 
>> text-markups, e.g. the markup of lists or ASCII-art. 
>> 
>> This individual text-markups has not been converted/rendered in the docbook
>> output, but I wanted to convert this individuals to reST, to render them in
>> Sphinx.
> 
> I think we need to unconfuse what's current standardize kerneldoc markup.
> There's three bits:
> - The header with the one-liner and parameters, i.e.

OK, forget my  example, I don't really want to promote
this way ... I agree, it is better to stay with standardize kernel-doc markup
and provide a "pass through" for the section-content (even if it is mixing 
markups).

Am 04.05.2016 um 15:41 schrieb Jani Nikula :

> Kernel-doc the tool should continue to parse kernel-doc comments at the
> high level like they currently are, according to the kernel-doc
> howto. 

The requirement was unclear for me, thanks to Daniel and 
Jani for clarifying this.


Am 04.05.2016 um 15:43 schrieb Daniel Vetter :

> This is already used widely in kerneldoc included by gpu.tmpl, and
> currently it's asciidoc. Those lists and asciiarts though are _not_
> standard kerneldoc at all. But imo any doc toolchain improvement should
> integrate along those lines.
> 
> For reference, this is what it's supposed to look like with the asciidoc
> support enabled:
> 
> https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html
> 
> Fyi those hacked-up patches to make this happen are available in
> 
> https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc
> 

Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
particular with your "MARKDOWNREADY := gpu.xml" process.

As I understood, you convert markdown to docbook within the kernel-doc 
script using pandoc's executable? ... I don't face this topic. With my 
modification of kernel-doc I produced pure reST markup from standardize
kernel-doc markup, no intermediate steps or tools required.

Am 04.05.2016 um 15:41 schrieb Jani Nikula :

> Overall I think we should promote fixing those in the kernel-doc
> comments. Trying to guesswork in kernel-doc tool will leave the source
> littered with bad examples that get proliferated all around. Instead of
> gradually fixing things, we'll end up gradually messing things up even
> more.

I agree with you, lets drop the reformat_block_rst from my kernel-doc script:

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

and we should have a plain "pass through".

Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :

> I think all of this makes sense.  It would be really nice to have the
> directives in the native sphinx language like that.  I *don't* think we
> need to aim for that at the outset; the docproc approach works until we can
> properly get rid of it.  What would be *really* nice would be to get
> support for the kernel-doc directive into the sphinx upstream.

No need for kernel-doc directive in sphinx upstream, later it will be 
an extension which could be installed by a simple command like 
"pip install kernel-doc-extensions" or similar.

I develop these required extension (and more) within my proof of 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 04.05.2016 um 15:43 schrieb Daniel Vetter :

> On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote:
>>> On Wed, 04 May 2016, Markus Heiser  wrote:
>>> I'd be *very* hesitant about adding the kind of things you do in
>>> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
>>> comments must be as simple as possible with basically pass-through of
>>> the comment blocks to sphinx.
>> 
>> Right, but you forgot, that the current markup in the source code comments
>> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
>> these source code comments to reST markup ;-)
>> 
>> So there is a need to differentiate between *vintage* kernel-doc markup 
>> and reST markup.
>> 
>> My suggestion is to add a tag to the comments, here a short example
>> what this might look like.
>> 
>>  --
>> /**
>> * drm_minor_release - Release DRM minor
>> * @minor: Pointer to DRM minor object
>> *
>> * Release a minor that was previously acquired via drm_minor_acquire().
>> */
>>  --
>> 
>> in short: the vintage style does not need any change and 
>> comments with reST markup has a tag ":reST:" in the first 
>> line(s) ...
>> 
>>  --
>> /**
>> * :reST:
>> *
>> * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>> *
>> *Release DRM minor
>> *
>> *:param struct drm_minor \*minor: Pointer to DRM minor object
>> *
>> */
>>  --
>> 
>> Comments with the ":reST:" tag could be exported and pass-through
>> to sphinx.
>> 
>>> Specifically, do not attempt to detect and
>>> parse elements like lists in kernel-doc.
>> 
>> If your markup in the comments is plain reST, no need to parse, but there
>> are markups in the (vintage) comments which made use of individual 
>> text-markups, e.g. the markup of lists or ASCII-art. 
>> 
>> This individual text-markups has not been converted/rendered in the docbook
>> output, but I wanted to convert this individuals to reST, to render them in
>> Sphinx.
> 
> I think we need to unconfuse what's current standardize kerneldoc markup.
> There's three bits:
> - The header with the one-liner and parameters, i.e.

OK, forget my  example, I don't really want to promote
this way ... I agree, it is better to stay with standardize kernel-doc markup
and provide a "pass through" for the section-content (even if it is mixing 
markups).

Am 04.05.2016 um 15:41 schrieb Jani Nikula :

> Kernel-doc the tool should continue to parse kernel-doc comments at the
> high level like they currently are, according to the kernel-doc
> howto. 

The requirement was unclear for me, thanks to Daniel and 
Jani for clarifying this.


Am 04.05.2016 um 15:43 schrieb Daniel Vetter :

> This is already used widely in kerneldoc included by gpu.tmpl, and
> currently it's asciidoc. Those lists and asciiarts though are _not_
> standard kerneldoc at all. But imo any doc toolchain improvement should
> integrate along those lines.
> 
> For reference, this is what it's supposed to look like with the asciidoc
> support enabled:
> 
> https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html
> 
> Fyi those hacked-up patches to make this happen are available in
> 
> https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc
> 

Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
particular with your "MARKDOWNREADY := gpu.xml" process.

As I understood, you convert markdown to docbook within the kernel-doc 
script using pandoc's executable? ... I don't face this topic. With my 
modification of kernel-doc I produced pure reST markup from standardize
kernel-doc markup, no intermediate steps or tools required.

Am 04.05.2016 um 15:41 schrieb Jani Nikula :

> Overall I think we should promote fixing those in the kernel-doc
> comments. Trying to guesswork in kernel-doc tool will leave the source
> littered with bad examples that get proliferated all around. Instead of
> gradually fixing things, we'll end up gradually messing things up even
> more.

I agree with you, lets drop the reformat_block_rst from my kernel-doc script:

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

and we should have a plain "pass through".

Am 04.05.2016 um 17:09 schrieb Jonathan Corbet :

> I think all of this makes sense.  It would be really nice to have the
> directives in the native sphinx language like that.  I *don't* think we
> need to aim for that at the outset; the docproc approach works until we can
> properly get rid of it.  What would be *really* nice would be to get
> support for the kernel-doc directive into the sphinx upstream.

No need for kernel-doc directive in sphinx upstream, later it will be 
an extension which could be installed by a simple command like 
"pip install kernel-doc-extensions" or similar.

I develop these required extension (and more) within my proof of concept
on github ... this takes time ... if I finished all my tests and all is
well, I will build the *kernel-doc-extensions* 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Jonathan Corbet  wrote:
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up.  It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)

Thanks. I'll start looking at how to make it less hackish and more
upstreamable.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Jonathan Corbet  wrote:
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up.  It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)

Thanks. I'll start looking at how to make it less hackish and more
upstreamable.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote:
> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter  wrote:
> 
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.  
> > 
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...
> 
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home.  I hope to begin emerging from this mess in the
> near future.
> 
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done.  Which would you have me push into 4.7?
> 
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up.  It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)
> 
> In the end, I guess, I feel that anything we might try to push for 4.7 is
> going to look rushed and not ready, and Linus might react accordingly.
> I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
> on things in that time frame...  I suspect you're pretty well fed up with
> this stuff being pushed back, and rightly so.  All I can do is apologize.
> 
> That said, if you do think there's something out there that is good
> enough to consider pushing in a week or two, do tell and we can all take
> a look.

Well I'd just have taken the asciidoc hacks I have currently in my
topic/kerneldoc branch, converted to sphinx and looked how it fares. It
should be fairly minimal, and I think the first step we want to do for the
long-term plan. I hope I can ready something, and then we can look whether
it's rushed for 4.7 or not.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote:
> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter  wrote:
> 
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.  
> > 
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...
> 
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home.  I hope to begin emerging from this mess in the
> near future.
> 
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done.  Which would you have me push into 4.7?
> 
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it.  I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up.  It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)
> 
> In the end, I guess, I feel that anything we might try to push for 4.7 is
> going to look rushed and not ready, and Linus might react accordingly.
> I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
> on things in that time frame...  I suspect you're pretty well fed up with
> this stuff being pushed back, and rightly so.  All I can do is apologize.
> 
> That said, if you do think there's something out there that is good
> enough to consider pushing in a week or two, do tell and we can all take
> a look.

Well I'd just have taken the asciidoc hacks I have currently in my
topic/kerneldoc branch, converted to sphinx and looked how it fares. It
should be fairly minimal, and I think the first step we want to do for the
long-term plan. I hope I can ready something, and then we can look whether
it's rushed for 4.7 or not.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 04 May 2016 16:41:50 +0300
Jani Nikula  wrote:

> On Wed, 04 May 2016, Markus Heiser  wrote:
> > In reST the directive might look like:
> >
> >  -
> > Device Instance and Driver Handling
> > ===
> >
> > .. kernel-doc::  drivers/gpu/drm/drm_drv.c
> >:doc:  driver instance overview
> >:exported:
> >
> >  -  
> 
> Yes, I think something like this, parsed by sphinx directly (via some
> extension perhaps), should be the end goal. I am not sure if it should
> be the immediate first goal though or whether we should reuse the
> existing docproc for now.

I think all of this makes sense.  It would be really nice to have the
directives in the native sphinx language like that.  I *don't* think we
need to aim for that at the outset; the docproc approach works until we can
properly get rid of it.  What would be *really* nice would be to get
support for the kernel-doc directive into the sphinx upstream.

> >  --
> >
> > Comments with the ":reST:" tag could be exported and pass-through
> > to sphinx.  
> 
> Disagreed on most of the above.

Agreed with the disagreement here.  We can't be adding ":reST:" tags to
comments; I anticipate a wee bit of pushback if we try.  It needs to work
with the comments as they are now.  It seems that should be possible.

Thanks,

jon

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 04 May 2016 16:41:50 +0300
Jani Nikula  wrote:

> On Wed, 04 May 2016, Markus Heiser  wrote:
> > In reST the directive might look like:
> >
> >  -
> > Device Instance and Driver Handling
> > ===
> >
> > .. kernel-doc::  drivers/gpu/drm/drm_drv.c
> >:doc:  driver instance overview
> >:exported:
> >
> >  -  
> 
> Yes, I think something like this, parsed by sphinx directly (via some
> extension perhaps), should be the end goal. I am not sure if it should
> be the immediate first goal though or whether we should reuse the
> existing docproc for now.

I think all of this makes sense.  It would be really nice to have the
directives in the native sphinx language like that.  I *don't* think we
need to aim for that at the outset; the docproc approach works until we can
properly get rid of it.  What would be *really* nice would be to get
support for the kernel-doc directive into the sphinx upstream.

> >  --
> >
> > Comments with the ":reST:" tag could be exported and pass-through
> > to sphinx.  
> 
> Disagreed on most of the above.

Agreed with the disagreement here.  We can't be adding ":reST:" tags to
comments; I anticipate a wee bit of pushback if we try.  It needs to work
with the comments as they are now.  It seems that should be possible.

Thanks,

jon

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 16:18:27 +0200
Daniel Vetter  wrote:

> > I'd really like to converge on the markup question, so that we can start
> > using all the cool stuff with impunity in gpu documentations.  
> 
> Aside: If we decide this now I could send in a pull request for the
> rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> markdown/asciidoc code I still have). That would be really awesome ...

Sorry for my relative absence...I'm still busy dealing with bureaucracy
an ocean away from home.  I hope to begin emerging from this mess in the
near future.

So ... there's the code you have, the work I (+Jani) did, and the work
Markus has done.  Which would you have me push into 4.7?

The sphinx/rst approach does seem, to me, to be the right one, with the
existing DocBook structure remaining in place for those who want/need
it.  I'm inclined toward my stuff as a base to work with, obviously :) But
it's hackish at best and needs a lot of cleaning up.  It's a proof of
concept, but it's hardly finished (one might say it's barely begun...)

In the end, I guess, I feel that anything we might try to push for 4.7 is
going to look rushed and not ready, and Linus might react accordingly.
I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
on things in that time frame...  I suspect you're pretty well fed up with
this stuff being pushed back, and rightly so.  All I can do is apologize.

That said, if you do think there's something out there that is good
enough to consider pushing in a week or two, do tell and we can all take
a look.

Thanks,

jon

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Wed, 4 May 2016 16:18:27 +0200
Daniel Vetter  wrote:

> > I'd really like to converge on the markup question, so that we can start
> > using all the cool stuff with impunity in gpu documentations.  
> 
> Aside: If we decide this now I could send in a pull request for the
> rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> markdown/asciidoc code I still have). That would be really awesome ...

Sorry for my relative absence...I'm still busy dealing with bureaucracy
an ocean away from home.  I hope to begin emerging from this mess in the
near future.

So ... there's the code you have, the work I (+Jani) did, and the work
Markus has done.  Which would you have me push into 4.7?

The sphinx/rst approach does seem, to me, to be the right one, with the
existing DocBook structure remaining in place for those who want/need
it.  I'm inclined toward my stuff as a base to work with, obviously :) But
it's hackish at best and needs a lot of cleaning up.  It's a proof of
concept, but it's hardly finished (one might say it's barely begun...)

In the end, I guess, I feel that anything we might try to push for 4.7 is
going to look rushed and not ready, and Linus might react accordingly.
I'd be more comfortable aiming for 4.8.  I *will* have more time to focus
on things in that time frame...  I suspect you're pretty well fed up with
this stuff being pushed back, and rightly so.  All I can do is apologize.

That said, if you do think there's something out there that is good
enough to consider pushing in a week or two, do tell and we can all take
a look.

Thanks,

jon

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 4, 2016 at 3:43 PM, Daniel Vetter  wrote:
> I'd really like to converge on the markup question, so that we can start
> using all the cool stuff with impunity in gpu documentations.

Aside: If we decide this now I could send in a pull request for the
rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
markdown/asciidoc code I still have). That would be really awesome ...

Jon?

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 4, 2016 at 3:43 PM, Daniel Vetter  wrote:
> I'd really like to converge on the markup question, so that we can start
> using all the cool stuff with impunity in gpu documentations.

Aside: If we decide this now I could send in a pull request for the
rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
markdown/asciidoc code I still have). That would be really awesome ...

Jon?

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote:
> > On Wed, 04 May 2016, Markus Heiser  wrote:
> > I'd be *very* hesitant about adding the kind of things you do in
> > reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
> > comments must be as simple as possible with basically pass-through of
> > the comment blocks to sphinx.
> 
> Right, but you forgot, that the current markup in the source code comments
> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
> these source code comments to reST markup ;-)
> 
> So there is a need to differentiate between *vintage* kernel-doc markup 
> and reST markup.
> 
> My suggestion is to add a tag to the comments, here a short example
> what this might look like.
> 
>  --
> /**
>  * drm_minor_release - Release DRM minor
>  * @minor: Pointer to DRM minor object
>  *
>  * Release a minor that was previously acquired via drm_minor_acquire().
>  */
>  --
> 
> in short: the vintage style does not need any change and 
> comments with reST markup has a tag ":reST:" in the first 
> line(s) ...
> 
>  --
> /**
>  * :reST:
>  *
>  * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>  *
>  *Release DRM minor
>  *
>  *:param struct drm_minor \*minor: Pointer to DRM minor object
>  *
>  */
>  --
> 
> Comments with the ":reST:" tag could be exported and pass-through
> to sphinx.
> 
> > Specifically, do not attempt to detect and
> > parse elements like lists in kernel-doc.
> 
> If your markup in the comments is plain reST, no need to parse, but there
> are markups in the (vintage) comments which made use of individual 
> text-markups, e.g. the markup of lists or ASCII-art. 
> 
> This individual text-markups has not been converted/rendered in the docbook
> output, but I wanted to convert this individuals to reST, to render them in
> Sphinx.

I think we need to unconfuse what's current standardize kerneldoc markup.
There's three bits:
- The header with the one-liner and parameters, i.e.

/**
 * drm_minor_release - Release DRM minor
 * @minor: Pointer to DRM minor object

  There's thousands of those, and we cant realistically change them. This
  means we need to teach kernel-doc to parse those and convert them to
  rest/shpinx layout. Any approach that expects a conversion of all the
  existing comments over to sphinx/rst is imo unworkable.

- References for function(), , #constants and @parameters.
  Again for the same reasons of being here already, we can't change those
  at all. On top of that these markers are also used to generate
  hyperlinks in the final docs. The kernel-doc script therefor not only
  needs to generate the right markup, but also correct links. Without
  generating links to functions/structures outside of a given document.

- kernel-doc also does paragraph splitting and minimalistic headers like

 * Returns:
 * 
 * This will be under a separate Returns: heading in the text.

  afaict at least the paragraph splitting would be identical between all
  proposed markup languages.

Now what all current proposals have done is simply allow pass-through of
asciidoc or rst markup in those paragraphs, so that the final processor
could make stuff pretty.

This is already used widely in kerneldoc included by gpu.tmpl, and
currently it's asciidoc. Those lists and asciiarts though are _not_
standard kerneldoc at all. But imo any doc toolchain improvement should
integrate along those lines.
 
For reference, this is what it's supposed to look like with the asciidoc
support enabled:

https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html

Fyi those hacked-up patches to make this happen are available in

https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc

Note that this isn't upstream, but we did start using this support all
over in gpu documentation simply because we really, really need it. And
I'm willing to throw all the comments into a converter to end up with
whatever markup we decide upon.

But what will not work is to throw the existing kernel-doc standard into
the shredder. So all the stuff with @, (), & and # must keep working.

> E.g. compare the member & description section of struct drm-display-mode ...
> 
> DocBook: 
> 
>   * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html
> 
> kernel-doc reST with the additional reformat_block_rst function:
> 
>   * 
> http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode

So we have this already. The two things I thought this entire discussion
was about:

- also throw out the docbook .xml and replace it with something else. That
  was jani's prototype on top of asciidoc, and it sounds like we could do
  something similar with rst/sphinx. Everyone pretty much agrees that
  this would be a second step, and the first step would be to simply
  integrate more advanced markup into the 

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote:
> > On Wed, 04 May 2016, Markus Heiser  wrote:
> > I'd be *very* hesitant about adding the kind of things you do in
> > reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
> > comments must be as simple as possible with basically pass-through of
> > the comment blocks to sphinx.
> 
> Right, but you forgot, that the current markup in the source code comments
> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
> these source code comments to reST markup ;-)
> 
> So there is a need to differentiate between *vintage* kernel-doc markup 
> and reST markup.
> 
> My suggestion is to add a tag to the comments, here a short example
> what this might look like.
> 
>  --
> /**
>  * drm_minor_release - Release DRM minor
>  * @minor: Pointer to DRM minor object
>  *
>  * Release a minor that was previously acquired via drm_minor_acquire().
>  */
>  --
> 
> in short: the vintage style does not need any change and 
> comments with reST markup has a tag ":reST:" in the first 
> line(s) ...
> 
>  --
> /**
>  * :reST:
>  *
>  * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>  *
>  *Release DRM minor
>  *
>  *:param struct drm_minor \*minor: Pointer to DRM minor object
>  *
>  */
>  --
> 
> Comments with the ":reST:" tag could be exported and pass-through
> to sphinx.
> 
> > Specifically, do not attempt to detect and
> > parse elements like lists in kernel-doc.
> 
> If your markup in the comments is plain reST, no need to parse, but there
> are markups in the (vintage) comments which made use of individual 
> text-markups, e.g. the markup of lists or ASCII-art. 
> 
> This individual text-markups has not been converted/rendered in the docbook
> output, but I wanted to convert this individuals to reST, to render them in
> Sphinx.

I think we need to unconfuse what's current standardize kerneldoc markup.
There's three bits:
- The header with the one-liner and parameters, i.e.

/**
 * drm_minor_release - Release DRM minor
 * @minor: Pointer to DRM minor object

  There's thousands of those, and we cant realistically change them. This
  means we need to teach kernel-doc to parse those and convert them to
  rest/shpinx layout. Any approach that expects a conversion of all the
  existing comments over to sphinx/rst is imo unworkable.

- References for function(), , #constants and @parameters.
  Again for the same reasons of being here already, we can't change those
  at all. On top of that these markers are also used to generate
  hyperlinks in the final docs. The kernel-doc script therefor not only
  needs to generate the right markup, but also correct links. Without
  generating links to functions/structures outside of a given document.

- kernel-doc also does paragraph splitting and minimalistic headers like

 * Returns:
 * 
 * This will be under a separate Returns: heading in the text.

  afaict at least the paragraph splitting would be identical between all
  proposed markup languages.

Now what all current proposals have done is simply allow pass-through of
asciidoc or rst markup in those paragraphs, so that the final processor
could make stuff pretty.

This is already used widely in kerneldoc included by gpu.tmpl, and
currently it's asciidoc. Those lists and asciiarts though are _not_
standard kerneldoc at all. But imo any doc toolchain improvement should
integrate along those lines.
 
For reference, this is what it's supposed to look like with the asciidoc
support enabled:

https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html

Fyi those hacked-up patches to make this happen are available in

https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc

Note that this isn't upstream, but we did start using this support all
over in gpu documentation simply because we really, really need it. And
I'm willing to throw all the comments into a converter to end up with
whatever markup we decide upon.

But what will not work is to throw the existing kernel-doc standard into
the shredder. So all the stuff with @, (), & and # must keep working.

> E.g. compare the member & description section of struct drm-display-mode ...
> 
> DocBook: 
> 
>   * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html
> 
> kernel-doc reST with the additional reformat_block_rst function:
> 
>   * 
> http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode

So we have this already. The two things I thought this entire discussion
was about:

- also throw out the docbook .xml and replace it with something else. That
  was jani's prototype on top of asciidoc, and it sounds like we could do
  something similar with rst/sphinx. Everyone pretty much agrees that
  this would be a second step, and the first step would be to simply
  integrate more advanced markup into the existing toolchain.

- which 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
>> What do you mean by ".tmpl workflow"?
>
> Sorry for bad naming, I addressed the DOCPROC build process which builds
> the .xml files from .tmpl files ...

Yeah, I know more about this part than I care. I was just wondering what
you refer to with that in the sphinx context.

> In reST the directive might look like:
>
>  -
> Device Instance and Driver Handling
> ===
>
> .. kernel-doc::  drivers/gpu/drm/drm_drv.c
>:doc:  driver instance overview
>:exported:
>
>  -

Yes, I think something like this, parsed by sphinx directly (via some
extension perhaps), should be the end goal. I am not sure if it should
be the immediate first goal though or whether we should reuse the
existing docproc for now.

> Right, but you forgot, that the current markup in the source code comments
> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
> these source code comments to reST markup ;-)
>
> So there is a need to differentiate between *vintage* kernel-doc markup 
> and reST markup.
>
> My suggestion is to add a tag to the comments, here a short example
> what this might look like.
>
>  --
> /**
>  * drm_minor_release - Release DRM minor
>  * @minor: Pointer to DRM minor object
>  *
>  * Release a minor that was previously acquired via drm_minor_acquire().
>  */
>  --
>
> in short: the vintage style does not need any change and 
> comments with reST markup has a tag ":reST:" in the first 
> line(s) ...
>
>  --
> /**
>  * :reST:
>  *
>  * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>  *
>  *Release DRM minor
>  *
>  *:param struct drm_minor \*minor: Pointer to DRM minor object
>  *
>  */
>  --
>
> Comments with the ":reST:" tag could be exported and pass-through
> to sphinx.

Disagreed on most of the above.

Maybe I was unclear in my previous mail, and I've probably worked on
this so much previously that I thought it was clear how we should handle
lightweight markup in kernel-doc comments.

Kernel-doc the tool should continue to parse kernel-doc comments at the
high level like they currently are, according to the kernel-doc
howto. The first heading line, the parameter/member lines with @param:,
and references within the text with @param, function(), 
structures, etc. For those, kernel-doc should produce appropriate rst
elements.

Beyond that, kernel-doc should *not* try to make guesses about the
formatting of the documentation comments. It should just pass the rest
of the formatting through. If the ad-hoc lists etc. in the current
comments don't turn into proper rst lists, then so be it. If it bugs
people, the comments will gradually be converted to proper rst. The
worst we could do is promote a sloppy style that kernel-doc fixes for
you, possibly "fixing" things you'd want to pass through to sphinx.

What you have in  example above will not fly. The
documentation comment style must remain as is defined in the kernel-doc
how, i.e.  example. It is imperative that the
kernel-doc comments remain as readable as they currently are in the
source code.

>> Specifically, do not attempt to detect and
>> parse elements like lists in kernel-doc.
>
> If your markup in the comments is plain reST, no need to parse, but there
> are markups in the (vintage) comments which made use of individual 
> text-markups, e.g. the markup of lists or ASCII-art. 
>
> This individual text-markups has not been converted/rendered in the docbook
> output, but I wanted to convert this individuals to reST, to render them in
> Sphinx.
>
> E.g. compare the member & description section of struct drm-display-mode ...
>
> DocBook: 
>
>   * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html
>
> kernel-doc reST with the additional reformat_block_rst function:
>
>   * 
> http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode

Overall I think we should promote fixing those in the kernel-doc
comments. Trying to guesswork in kernel-doc tool will leave the source
littered with bad examples that get proliferated all around. Instead of
gradually fixing things, we'll end up gradually messing things up even
more.

For those specific examples, IIRC the last time I played with that,
simply leaving the prefix whitespace in place went a long way (just eat
" * " from the lines). The asciiart just worked.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
>> What do you mean by ".tmpl workflow"?
>
> Sorry for bad naming, I addressed the DOCPROC build process which builds
> the .xml files from .tmpl files ...

Yeah, I know more about this part than I care. I was just wondering what
you refer to with that in the sphinx context.

> In reST the directive might look like:
>
>  -
> Device Instance and Driver Handling
> ===
>
> .. kernel-doc::  drivers/gpu/drm/drm_drv.c
>:doc:  driver instance overview
>:exported:
>
>  -

Yes, I think something like this, parsed by sphinx directly (via some
extension perhaps), should be the end goal. I am not sure if it should
be the immediate first goal though or whether we should reuse the
existing docproc for now.

> Right, but you forgot, that the current markup in the source code comments
> is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
> these source code comments to reST markup ;-)
>
> So there is a need to differentiate between *vintage* kernel-doc markup 
> and reST markup.
>
> My suggestion is to add a tag to the comments, here a short example
> what this might look like.
>
>  --
> /**
>  * drm_minor_release - Release DRM minor
>  * @minor: Pointer to DRM minor object
>  *
>  * Release a minor that was previously acquired via drm_minor_acquire().
>  */
>  --
>
> in short: the vintage style does not need any change and 
> comments with reST markup has a tag ":reST:" in the first 
> line(s) ...
>
>  --
> /**
>  * :reST:
>  *
>  * .. c:function:: void drm_minor_release(struct drm_minor *minor)
>  *
>  *Release DRM minor
>  *
>  *:param struct drm_minor \*minor: Pointer to DRM minor object
>  *
>  */
>  --
>
> Comments with the ":reST:" tag could be exported and pass-through
> to sphinx.

Disagreed on most of the above.

Maybe I was unclear in my previous mail, and I've probably worked on
this so much previously that I thought it was clear how we should handle
lightweight markup in kernel-doc comments.

Kernel-doc the tool should continue to parse kernel-doc comments at the
high level like they currently are, according to the kernel-doc
howto. The first heading line, the parameter/member lines with @param:,
and references within the text with @param, function(), 
structures, etc. For those, kernel-doc should produce appropriate rst
elements.

Beyond that, kernel-doc should *not* try to make guesses about the
formatting of the documentation comments. It should just pass the rest
of the formatting through. If the ad-hoc lists etc. in the current
comments don't turn into proper rst lists, then so be it. If it bugs
people, the comments will gradually be converted to proper rst. The
worst we could do is promote a sloppy style that kernel-doc fixes for
you, possibly "fixing" things you'd want to pass through to sphinx.

What you have in  example above will not fly. The
documentation comment style must remain as is defined in the kernel-doc
how, i.e.  example. It is imperative that the
kernel-doc comments remain as readable as they currently are in the
source code.

>> Specifically, do not attempt to detect and
>> parse elements like lists in kernel-doc.
>
> If your markup in the comments is plain reST, no need to parse, but there
> are markups in the (vintage) comments which made use of individual 
> text-markups, e.g. the markup of lists or ASCII-art. 
>
> This individual text-markups has not been converted/rendered in the docbook
> output, but I wanted to convert this individuals to reST, to render them in
> Sphinx.
>
> E.g. compare the member & description section of struct drm-display-mode ...
>
> DocBook: 
>
>   * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html
>
> kernel-doc reST with the additional reformat_block_rst function:
>
>   * 
> http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode

Overall I think we should promote fixing those in the kernel-doc
comments. Trying to guesswork in kernel-doc tool will leave the source
littered with bad examples that get proliferated all around. Instead of
gradually fixing things, we'll end up gradually messing things up even
more.

For those specific examples, IIRC the last time I played with that,
simply leaving the prefix whitespace in place went a long way (just eat
" * " from the lines). The asciiart just worked.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jani,

Am 04.05.2016 um 11:58 schrieb Jani Nikula :

> On Wed, 04 May 2016, Markus Heiser  wrote:
>> but I think this will not by very helpful, as long as you miss 
>> a similar ".tmpl" workflow for reST documents.
>> 
>> I'am working on a reST directive (named: "kernel-doc") to provide a
>> similar ".tmpl" workflow within plain reST. The first step towards
>> is done with (my) modified kernel-doc script ...
>> 
>> * 
>> https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736
>> 
>> which produce reST from source code comments. E.g. this content is 
>> generated with it.
> 
> What do you mean by ".tmpl workflow"?

Sorry for bad naming, I addressed the DOCPROC build process which builds
the .xml files from .tmpl files ...

 
###
# The build process is as follows (targets):
#  (xmldocs) [by docproc]
# file.tmpl --> file.xml +--> file.ps   (psdocs)   [by db2ps or xmlto]
#+--> file.pdf  (pdfdocs)  [by db2pdf or xmlto]
#+--> DIR=file  (htmldocs) [by xmlto]
#+--> man/  (mandocs)  [by xmlto]

###
# DOCPROC is used for two purposes:
# 1) To generate a dependency list for a .tmpl file
# 2) To preprocess a .tmpl file and call kernel-doc with
# appropriate parameters.
# The following rules are used to generate the .xml documentation
# required to generate the final targets. (ps, pdf, html).
 -


The DOCPROC markup directives are described in the kernel-doc-nano-HOWTO:

https://github.com/torvalds/linux/blob/master/Documentation/kernel-doc-nano-HOWTO.txt#L332

My aim is to implement a reST-directive which is similar. E.g: 
 
 -

  Device Instance and Driver Handling
!Pdrivers/gpu/drm/drm_drv.c driver instance overview
!Edrivers/gpu/drm/drm_drv.c


 -

In reST the directive might look like:

 -
Device Instance and Driver Handling
===

.. kernel-doc::  drivers/gpu/drm/drm_drv.c
   :doc:  driver instance overview
   :exported:

 -


> I'd be *very* hesitant about adding the kind of things you do in
> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
> comments must be as simple as possible with basically pass-through of
> the comment blocks to sphinx.

Right, but you forgot, that the current markup in the source code comments
is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
these source code comments to reST markup ;-)

So there is a need to differentiate between *vintage* kernel-doc markup 
and reST markup.

My suggestion is to add a tag to the comments, here a short example
what this might look like.

 --
/**
 * drm_minor_release - Release DRM minor
 * @minor: Pointer to DRM minor object
 *
 * Release a minor that was previously acquired via drm_minor_acquire().
 */
 --

in short: the vintage style does not need any change and 
comments with reST markup has a tag ":reST:" in the first 
line(s) ...

 --
/**
 * :reST:
 *
 * .. c:function:: void drm_minor_release(struct drm_minor *minor)
 *
 *Release DRM minor
 *
 *:param struct drm_minor \*minor: Pointer to DRM minor object
 *
 */
 --

Comments with the ":reST:" tag could be exported and pass-through
to sphinx.

> Specifically, do not attempt to detect and
> parse elements like lists in kernel-doc.

If your markup in the comments is plain reST, no need to parse, but there
are markups in the (vintage) comments which made use of individual 
text-markups, e.g. the markup of lists or ASCII-art. 

This individual text-markups has not been converted/rendered in the docbook
output, but I wanted to convert this individuals to reST, to render them in
Sphinx.

E.g. compare the member & description section of struct drm-display-mode ...

DocBook: 

  * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html

kernel-doc reST with the additional reformat_block_rst function:

  * 
http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode


-- Markus--

> -- 
> Jani Nikula, Intel Open Source Technology Center
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jani,

Am 04.05.2016 um 11:58 schrieb Jani Nikula :

> On Wed, 04 May 2016, Markus Heiser  wrote:
>> but I think this will not by very helpful, as long as you miss 
>> a similar ".tmpl" workflow for reST documents.
>> 
>> I'am working on a reST directive (named: "kernel-doc") to provide a
>> similar ".tmpl" workflow within plain reST. The first step towards
>> is done with (my) modified kernel-doc script ...
>> 
>> * 
>> https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736
>> 
>> which produce reST from source code comments. E.g. this content is 
>> generated with it.
> 
> What do you mean by ".tmpl workflow"?

Sorry for bad naming, I addressed the DOCPROC build process which builds
the .xml files from .tmpl files ...

 
###
# The build process is as follows (targets):
#  (xmldocs) [by docproc]
# file.tmpl --> file.xml +--> file.ps   (psdocs)   [by db2ps or xmlto]
#+--> file.pdf  (pdfdocs)  [by db2pdf or xmlto]
#+--> DIR=file  (htmldocs) [by xmlto]
#+--> man/  (mandocs)  [by xmlto]

###
# DOCPROC is used for two purposes:
# 1) To generate a dependency list for a .tmpl file
# 2) To preprocess a .tmpl file and call kernel-doc with
# appropriate parameters.
# The following rules are used to generate the .xml documentation
# required to generate the final targets. (ps, pdf, html).
 -


The DOCPROC markup directives are described in the kernel-doc-nano-HOWTO:

https://github.com/torvalds/linux/blob/master/Documentation/kernel-doc-nano-HOWTO.txt#L332

My aim is to implement a reST-directive which is similar. E.g: 
 
 -

  Device Instance and Driver Handling
!Pdrivers/gpu/drm/drm_drv.c driver instance overview
!Edrivers/gpu/drm/drm_drv.c


 -

In reST the directive might look like:

 -
Device Instance and Driver Handling
===

.. kernel-doc::  drivers/gpu/drm/drm_drv.c
   :doc:  driver instance overview
   :exported:

 -


> I'd be *very* hesitant about adding the kind of things you do in
> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
> comments must be as simple as possible with basically pass-through of
> the comment blocks to sphinx.

Right, but you forgot, that the current markup in the source code comments
is based on the  kernel-doc-nano-HOWTO and I guess no one will migrate all
these source code comments to reST markup ;-)

So there is a need to differentiate between *vintage* kernel-doc markup 
and reST markup.

My suggestion is to add a tag to the comments, here a short example
what this might look like.

 --
/**
 * drm_minor_release - Release DRM minor
 * @minor: Pointer to DRM minor object
 *
 * Release a minor that was previously acquired via drm_minor_acquire().
 */
 --

in short: the vintage style does not need any change and 
comments with reST markup has a tag ":reST:" in the first 
line(s) ...

 --
/**
 * :reST:
 *
 * .. c:function:: void drm_minor_release(struct drm_minor *minor)
 *
 *Release DRM minor
 *
 *:param struct drm_minor \*minor: Pointer to DRM minor object
 *
 */
 --

Comments with the ":reST:" tag could be exported and pass-through
to sphinx.

> Specifically, do not attempt to detect and
> parse elements like lists in kernel-doc.

If your markup in the comments is plain reST, no need to parse, but there
are markups in the (vintage) comments which made use of individual 
text-markups, e.g. the markup of lists or ASCII-art. 

This individual text-markups has not been converted/rendered in the docbook
output, but I wanted to convert this individuals to reST, to render them in
Sphinx.

E.g. compare the member & description section of struct drm-display-mode ...

DocBook: 

  * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html

kernel-doc reST with the additional reformat_block_rst function:

  * 
http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode


-- Markus--

> -- 
> Jani Nikula, Intel Open Source Technology Center
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
> but I think this will not by very helpful, as long as you miss 
> a similar ".tmpl" workflow for reST documents.
>
> I'am working on a reST directive (named: "kernel-doc") to provide a
> similar ".tmpl" workflow within plain reST. The first step towards
> is done with (my) modified kernel-doc script ...
>
> * 
> https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736
>
> which produce reST from source code comments. E.g. this content is 
> generated with it.

What do you mean by ".tmpl workflow"?

I'd be *very* hesitant about adding the kind of things you do in
reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
comments must be as simple as possible with basically pass-through of
the comment blocks to sphinx. Specifically, do not attempt to detect and
parse elements like lists in kernel-doc.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser  wrote:
> but I think this will not by very helpful, as long as you miss 
> a similar ".tmpl" workflow for reST documents.
>
> I'am working on a reST directive (named: "kernel-doc") to provide a
> similar ".tmpl" workflow within plain reST. The first step towards
> is done with (my) modified kernel-doc script ...
>
> * 
> https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736
>
> which produce reST from source code comments. E.g. this content is 
> generated with it.

What do you mean by ".tmpl workflow"?

I'd be *very* hesitant about adding the kind of things you do in
reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
comments must be as simple as possible with basically pass-through of
the comment blocks to sphinx. Specifically, do not attempt to detect and
parse elements like lists in kernel-doc.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi all, (hi Jonathan, please take note of my offer below)

Am 03.05.2016 um 16:31 schrieb Daniel Vetter :

> Hi all,
> 
> So sounds like moving ahead with rst/sphinx is the option that should
> allow us to address everyone's concerns eventually? Of course the
> first one won't have it all (media seems really tricky), ...

BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 

Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab :

> With that sense, the "List tables" format is also not good, as
> one row addition would generate several hunks (one for each column
> of the table), making harder to review the patch by just looking at
> the diff.

For this, I wrote the "flat-table" reST-directive, which adds 
missing cells automatically:

doc:
http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
source: 
https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

I used "flat-table" to migrate all DocBook-XML documents to reST. With this
directive, I also managed to migrate the complete media book (no more TODOs)
incl. the large tables like them from subdev-formats:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html

(Rendering large tables is a general discussion which should not take place in 
this MT)

>  but I'd like
> to get something awesome in this area closer to mainline. I'm stalling
> on typing more fancyful gpu docs right now (with tables, pictures and
> stuff) since that would require a pile of needless work to redo when
> we switch a few times more ;-)

Are your "fancyful gpu" written from scratch, or will it be an rewrite
of the Documentation/DocBook/gpu.tmpl?

If it is the last; as starting point you can use a copy of:

https://github.com/return42/sphkerneldoc/tree/master/doc/books/gpu

but I think this will not by very helpful, as long as you miss 
a similar ".tmpl" workflow for reST documents.

I'am working on a reST directive (named: "kernel-doc") to provide a
similar ".tmpl" workflow within plain reST. The first step towards
is done with (my) modified kernel-doc script ...

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

which produce reST from source code comments. E.g. this content is 
generated with it.

* http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

> And Jani also wants to get this in, but he doesn't really want to
> spend more effort on a system that can't be merged.
> 
> So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov?

My recommendation is to start with reST-markup if you are 
writing new docs from scratch. Workflows like the .tmpl one
can be added to the kernel build process step by step and
there is no need to drop the DocBook-XML process. Every author
should be free to decide by himself, when it is time to
migrate his DocBook sources into a reST building process.

I'am new to the kernel, my aim is to support the doc maintainer
(contact me), e.g. to migrate existing DocBooks to reST,
elaborate (build) procedures for improved reST support
within the kernel sources and to give recommendation to 
infrastructures.

@Jonathan: I know, you have no time right now. If it is OK
for you, I elaborate all these reST stuff within my POC
at github (this will take some time). Later, if you have
time, we can merge the made experience and tools to the
kernel build process .. is that okay for you?

-- Markus --


> 
> Cheers, Daniel
> 
> On Wed, Apr 27, 2016 at 4:28 PM, Grant Likely  
> wrote:
>> On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
>>> On Fri, 8 Apr 2016 17:12:27 +0200
>>> Markus Heiser  wrote:
>>> 
 motivated by this MT, I implemented a toolchain to migrate the kernel’s
 DocBook XML documentation to reST markup.
 
 It converts 99% of the docs well ... to gain an impression how
 kernel-docs could benefit from, visit my sphkerneldoc project page
 on github:
 
  http://return42.github.io/sphkerneldoc/
>>> 
>>> So I've obviously been pretty quiet on this recently.  Apologies...I've
>>> been dealing with an extended death-in-the-family experience, and there is
>>> still a fair amount of cleanup to be done.
>>> 
>>> Looking quickly at this work, it seems similar to the results I got.  But
>>> there's a lot of code there that came from somewhere?  I'd put together a
>>> fairly simple conversion using pandoc and a couple of short sed scripts;
>>> is there a reason for a more complex solution?
>>> 
>>> Thanks for looking into this, anyway; I hope to be able to focus more on
>>> it shortly.
>> 
>> Hi Jon,
>> 
>> Thanks for digging into this. FWIW, here is my $0.02. I've been
>> working on restarting the devicetree specification, and after looking
>> at both reStructuredText and Asciidoc(tor) I thought I liked the
>> Asciidoc markup better, so chose that. I 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi all, (hi Jonathan, please take note of my offer below)

Am 03.05.2016 um 16:31 schrieb Daniel Vetter :

> Hi all,
> 
> So sounds like moving ahead with rst/sphinx is the option that should
> allow us to address everyone's concerns eventually? Of course the
> first one won't have it all (media seems really tricky), ...

BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... 

Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab :

> With that sense, the "List tables" format is also not good, as
> one row addition would generate several hunks (one for each column
> of the table), making harder to review the patch by just looking at
> the diff.

For this, I wrote the "flat-table" reST-directive, which adds 
missing cells automatically:

doc:
http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
source: 
https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

I used "flat-table" to migrate all DocBook-XML documents to reST. With this
directive, I also managed to migrate the complete media book (no more TODOs)
incl. the large tables like them from subdev-formats:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html

(Rendering large tables is a general discussion which should not take place in 
this MT)

>  but I'd like
> to get something awesome in this area closer to mainline. I'm stalling
> on typing more fancyful gpu docs right now (with tables, pictures and
> stuff) since that would require a pile of needless work to redo when
> we switch a few times more ;-)

Are your "fancyful gpu" written from scratch, or will it be an rewrite
of the Documentation/DocBook/gpu.tmpl?

If it is the last; as starting point you can use a copy of:

https://github.com/return42/sphkerneldoc/tree/master/doc/books/gpu

but I think this will not by very helpful, as long as you miss 
a similar ".tmpl" workflow for reST documents.

I'am working on a reST directive (named: "kernel-doc") to provide a
similar ".tmpl" workflow within plain reST. The first step towards
is done with (my) modified kernel-doc script ...

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

which produce reST from source code comments. E.g. this content is 
generated with it.

* http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

> And Jani also wants to get this in, but he doesn't really want to
> spend more effort on a system that can't be merged.
> 
> So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov?

My recommendation is to start with reST-markup if you are 
writing new docs from scratch. Workflows like the .tmpl one
can be added to the kernel build process step by step and
there is no need to drop the DocBook-XML process. Every author
should be free to decide by himself, when it is time to
migrate his DocBook sources into a reST building process.

I'am new to the kernel, my aim is to support the doc maintainer
(contact me), e.g. to migrate existing DocBooks to reST,
elaborate (build) procedures for improved reST support
within the kernel sources and to give recommendation to 
infrastructures.

@Jonathan: I know, you have no time right now. If it is OK
for you, I elaborate all these reST stuff within my POC
at github (this will take some time). Later, if you have
time, we can merge the made experience and tools to the
kernel build process .. is that okay for you?

-- Markus --


> 
> Cheers, Daniel
> 
> On Wed, Apr 27, 2016 at 4:28 PM, Grant Likely  
> wrote:
>> On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
>>> On Fri, 8 Apr 2016 17:12:27 +0200
>>> Markus Heiser  wrote:
>>> 
 motivated by this MT, I implemented a toolchain to migrate the kernel’s
 DocBook XML documentation to reST markup.
 
 It converts 99% of the docs well ... to gain an impression how
 kernel-docs could benefit from, visit my sphkerneldoc project page
 on github:
 
  http://return42.github.io/sphkerneldoc/
>>> 
>>> So I've obviously been pretty quiet on this recently.  Apologies...I've
>>> been dealing with an extended death-in-the-family experience, and there is
>>> still a fair amount of cleanup to be done.
>>> 
>>> Looking quickly at this work, it seems similar to the results I got.  But
>>> there's a lot of code there that came from somewhere?  I'd put together a
>>> fairly simple conversion using pandoc and a couple of short sed scripts;
>>> is there a reason for a more complex solution?
>>> 
>>> Thanks for looking into this, anyway; I hope to be able to focus more on
>>> it shortly.
>> 
>> Hi Jon,
>> 
>> Thanks for digging into this. FWIW, here is my $0.02. I've been
>> working on restarting the devicetree specification, and after looking
>> at both reStructuredText and Asciidoc(tor) I thought I liked the
>> Asciidoc markup better, so chose that. I then proceeded to spend weeks
>> trying to get reasonable output from the toolchain. When I got fed up
>> and gave Sphinx 

Re: Kernel docs: muddying the waters a bit

[Keith Packard]

Daniel Vetter  writes:

> So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer
> pov?

I think the right answer for today is to use sphinx to generate docs
From inline comments, to encourage outline docs to give it a try but to
allow doc writers to use whatever works for them.

That will leave the media docs using the existing system so that their
tables don't get wrecked, while providing guidance to new doc writers
and allowing inline docs to include markup.

-- 
-keith


signature.asc
Description: PGP signature

Re: Kernel docs: muddying the waters a bit

[Keith Packard]

Daniel Vetter  writes:

> So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer
> pov?

I think the right answer for today is to use sphinx to generate docs
From inline comments, to encourage outline docs to give it a try but to
allow doc writers to use whatever works for them.

That will leave the media docs using the existing system so that their
tables don't get wrecked, while providing guidance to new doc writers
and allowing inline docs to include markup.

-- 
-keith


signature.asc
Description: PGP signature

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

Hi all,

So sounds like moving ahead with rst/sphinx is the option that should
allow us to address everyone's concerns eventually? Of course the
first one won't have it all (media seems really tricky), but I'd like
to get something awesome in this area closer to mainline. I'm stalling
on typing more fancyful gpu docs right now (with tables, pictures and
stuff) since that would require a pile of needless work to redo when
we switch a few times more ;-)

And Jani also wants to get this in, but he doesn't really want to
spend more effort on a system that can't be merged.

So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov?

Cheers, Daniel

On Wed, Apr 27, 2016 at 4:28 PM, Grant Likely  wrote:
> On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
>> On Fri, 8 Apr 2016 17:12:27 +0200
>> Markus Heiser  wrote:
>>
>>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>>> DocBook XML documentation to reST markup.
>>>
>>> It converts 99% of the docs well ... to gain an impression how
>>> kernel-docs could benefit from, visit my sphkerneldoc project page
>>> on github:
>>>
>>>   http://return42.github.io/sphkerneldoc/
>>
>> So I've obviously been pretty quiet on this recently.  Apologies...I've
>> been dealing with an extended death-in-the-family experience, and there is
>> still a fair amount of cleanup to be done.
>>
>> Looking quickly at this work, it seems similar to the results I got.  But
>> there's a lot of code there that came from somewhere?  I'd put together a
>> fairly simple conversion using pandoc and a couple of short sed scripts;
>> is there a reason for a more complex solution?
>>
>> Thanks for looking into this, anyway; I hope to be able to focus more on
>> it shortly.
>
> Hi Jon,
>
> Thanks for digging into this. FWIW, here is my $0.02. I've been
> working on restarting the devicetree specification, and after looking
> at both reStructuredText and Asciidoc(tor) I thought I liked the
> Asciidoc markup better, so chose that. I then proceeded to spend weeks
> trying to get reasonable output from the toolchain. When I got fed up
> and gave Sphinx a try, I was up and running with reasonable PDF and
> HTML output in a day and a half.
>
> Honestly, in the end I think we could make either tool do what is
> needed of it. However, my impression after trying to do a document
> that needs to have nice publishable output with both tools is that
> Sphinx is easier to work with, simpler to extend, better supported. My
> vote is firmly behind Sphinx/reStructuredText.
>
> g.



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Daniel Vetter]

Hi all,

So sounds like moving ahead with rst/sphinx is the option that should
allow us to address everyone's concerns eventually? Of course the
first one won't have it all (media seems really tricky), but I'd like
to get something awesome in this area closer to mainline. I'm stalling
on typing more fancyful gpu docs right now (with tables, pictures and
stuff) since that would require a pile of needless work to redo when
we switch a few times more ;-)

And Jani also wants to get this in, but he doesn't really want to
spend more effort on a system that can't be merged.

So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov?

Cheers, Daniel

On Wed, Apr 27, 2016 at 4:28 PM, Grant Likely  wrote:
> On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
>> On Fri, 8 Apr 2016 17:12:27 +0200
>> Markus Heiser  wrote:
>>
>>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>>> DocBook XML documentation to reST markup.
>>>
>>> It converts 99% of the docs well ... to gain an impression how
>>> kernel-docs could benefit from, visit my sphkerneldoc project page
>>> on github:
>>>
>>>   http://return42.github.io/sphkerneldoc/
>>
>> So I've obviously been pretty quiet on this recently.  Apologies...I've
>> been dealing with an extended death-in-the-family experience, and there is
>> still a fair amount of cleanup to be done.
>>
>> Looking quickly at this work, it seems similar to the results I got.  But
>> there's a lot of code there that came from somewhere?  I'd put together a
>> fairly simple conversion using pandoc and a couple of short sed scripts;
>> is there a reason for a more complex solution?
>>
>> Thanks for looking into this, anyway; I hope to be able to focus more on
>> it shortly.
>
> Hi Jon,
>
> Thanks for digging into this. FWIW, here is my $0.02. I've been
> working on restarting the devicetree specification, and after looking
> at both reStructuredText and Asciidoc(tor) I thought I liked the
> Asciidoc markup better, so chose that. I then proceeded to spend weeks
> trying to get reasonable output from the toolchain. When I got fed up
> and gave Sphinx a try, I was up and running with reasonable PDF and
> HTML output in a day and a half.
>
> Honestly, in the end I think we could make either tool do what is
> needed of it. However, my impression after trying to do a document
> that needs to have nice publishable output with both tools is that
> Sphinx is easier to work with, simpler to extend, better supported. My
> vote is firmly behind Sphinx/reStructuredText.
>
> g.



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: Kernel docs: muddying the waters a bit

[Grant Likely]

On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  wrote:
>
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>> DocBook XML documentation to reST markup.
>>
>> It converts 99% of the docs well ... to gain an impression how
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>>
>>   http://return42.github.io/sphkerneldoc/
>
> So I've obviously been pretty quiet on this recently.  Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
>
> Looking quickly at this work, it seems similar to the results I got.  But
> there's a lot of code there that came from somewhere?  I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?
>
> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.

Hi Jon,

Thanks for digging into this. FWIW, here is my $0.02. I've been
working on restarting the devicetree specification, and after looking
at both reStructuredText and Asciidoc(tor) I thought I liked the
Asciidoc markup better, so chose that. I then proceeded to spend weeks
trying to get reasonable output from the toolchain. When I got fed up
and gave Sphinx a try, I was up and running with reasonable PDF and
HTML output in a day and a half.

Honestly, in the end I think we could make either tool do what is
needed of it. However, my impression after trying to do a document
that needs to have nice publishable output with both tools is that
Sphinx is easier to work with, simpler to extend, better supported. My
vote is firmly behind Sphinx/reStructuredText.

g.

Re: Kernel docs: muddying the waters a bit

[Grant Likely]

On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbet  wrote:
> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  wrote:
>
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>> DocBook XML documentation to reST markup.
>>
>> It converts 99% of the docs well ... to gain an impression how
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>>
>>   http://return42.github.io/sphkerneldoc/
>
> So I've obviously been pretty quiet on this recently.  Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
>
> Looking quickly at this work, it seems similar to the results I got.  But
> there's a lot of code there that came from somewhere?  I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?
>
> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.

Hi Jon,

Thanks for digging into this. FWIW, here is my $0.02. I've been
working on restarting the devicetree specification, and after looking
at both reStructuredText and Asciidoc(tor) I thought I liked the
Asciidoc markup better, so chose that. I then proceeded to spend weeks
trying to get reasonable output from the toolchain. When I got fed up
and gave Sphinx a try, I was up and running with reasonable PDF and
HTML output in a day and a half.

Honestly, in the end I think we could make either tool do what is
needed of it. However, my impression after trying to do a document
that needs to have nice publishable output with both tools is that
Sphinx is easier to work with, simpler to extend, better supported. My
vote is firmly behind Sphinx/reStructuredText.

g.

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Mon, 18 Apr 2016 10:10:17 +0200
Markus Heiser  escreveu:

> Hi Mauro, hi kernel-doc authors, 
> 
> Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab 
> :
> 
> > Em Fri, 8 Apr 2016 17:12:27 +0200
> > Markus Heiser  escreveu:
> >   
> >> Hi kernel-doc authors,
> >> 
> >> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> >> DocBook XML documentation to reST markup. 
> >> 
> >> It converts 99% of the docs well ... to gain an impression how 
> >> kernel-docs could benefit from, visit my sphkerneldoc project page
> >> on github:
> >> 
> >> http://return42.github.io/sphkerneldoc/
> >> 
> >> The sources available at:
> >> 
> >> https://github.com/return42/sphkerneldoc
> >> 
> >> The work is underway, suggestions are welcome!
> >> 
> >> .. have a nice weekend ..  
> > 
> > Hi Markus,
> > 
> > Thanks for your work. I'm basically worried about the media docbook,
> > with has some complexities that I was not able to figure out a way to
> > convert it to reST.
> > 
> > So, let me pinpoint some issues that I noticed there, on a quick
> > look.
> > 
> > The first thing I noticed is that the index doesn't match what's
> > there, when generated from DocBook:
> > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
> > 
> > So, for instance, "Interfaces" is at the same level as "Input/Output".
> > 
> > This sounds like an something went wrong when dealing with the title
> > indentation levels during the conversion.
> >   
> 
> Yes, the hierarchical structure was broken by two causes. First was, that the
> *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) 
> placed
> all sections and sub(-sub)-sections at the same level (direct under chapter). 
> This was
> not only broken in the linux_tv book, in the other books also.
> 
> Thanks for pointing and sorry that I have overlooked this (I was so much 
> focused on
> on converting single elements to reST, that I not see the wood for the trees).
> 
> It is now fixed, may you give it a new try.

Thanks! It looks good now.

> > I would also like to see the titles numbered in chapters (and, if
> > possible, in sections and items) and to be properly indented at the 
> > index.  
> 
> BTW a few words about differences between DockBook and reST (Sphinx).
> 
> With DocBook you write *books*, the protocol (the DocBook application) has
> no facility to *chunk* and interconnect several documents. The external 
> ENTITY 
> is a workaround on the SGML layer, not on XML nor on the DB-application layer.
> Thats the reason, why so many XML-tools don't handle this entities and
> many DocBook to (e.g.) reST tools are fail.
> 
> With **standard** reST it is nearly the same, except there is a "include"
> directive on the application layer. But this directive is very simple,
> comparable to the C preprocessor "#include" directive.
> 
> With the **superset** reST-markup of Sphinx-doc you get a the "toctree" 
> directive,
> which lets you control how a document-tree should be build.
> 
>  http://www.sphinx-doc.org/en/stable/markup/toctree.html
> 
> @Mauro: you mentioned a docutils (rst2*) experience in your mail 
>   http://marc.info/?l=linux-doc=145735316012094=2
> 
>   Because the "toctree" directive -- and other directives
>   we use -- are a part of a superset of the **standard** 
>   reST, the standard docutils (like rst2*) will not work.
> 
> OK, back to your requirements: within the toctree directive you can
> set options like "maxdepth" and "numbered". It is a decision, how
> deep TOCs should go and if they should be numbered. IMO, in a
> HTML rendered page, with a proper navigation bar on the side, deep 
> TOCs in the running text have no pros, they only blow up the running
> text and bring more scrolling with. In my sense numbering chapters
> make only sense in books, not in HTML pages, where you have hyperlinks.
> 
> Just for demonstration, I added numbering in the linux-tv book:
> 
> https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3
> 
> May be, I will drop it later, because all reST sources are generated
> by a make target and I always commit the whole reST tree. As I said, 
> it is a decision which might be made later, when the migration takes 
> places.

This is the uAPI spec DocBook, that we modify frequently, as we add
more features to the Kernel, and as we make sure that all drivers will
behave the same. So, from time to time, we need to clarify some topics
at the documentation. By having a numeration, it is easier for us to
discuss things like:
"1.2.10.14. V4L2_PIX_FMT_VYUY (‘VYUY’)" is not properly
described and requires some sort of clarification.

Ok, one could also refer to it via a hyperlink, but several Kernel
media maintainers prefer to generate a single big html file, as it
makes easier to locate everything it is needed on it.

So, with the item number, they can just seek 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Mon, 18 Apr 2016 10:10:17 +0200
Markus Heiser  escreveu:

> Hi Mauro, hi kernel-doc authors, 
> 
> Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab 
> :
> 
> > Em Fri, 8 Apr 2016 17:12:27 +0200
> > Markus Heiser  escreveu:
> >   
> >> Hi kernel-doc authors,
> >> 
> >> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> >> DocBook XML documentation to reST markup. 
> >> 
> >> It converts 99% of the docs well ... to gain an impression how 
> >> kernel-docs could benefit from, visit my sphkerneldoc project page
> >> on github:
> >> 
> >> http://return42.github.io/sphkerneldoc/
> >> 
> >> The sources available at:
> >> 
> >> https://github.com/return42/sphkerneldoc
> >> 
> >> The work is underway, suggestions are welcome!
> >> 
> >> .. have a nice weekend ..  
> > 
> > Hi Markus,
> > 
> > Thanks for your work. I'm basically worried about the media docbook,
> > with has some complexities that I was not able to figure out a way to
> > convert it to reST.
> > 
> > So, let me pinpoint some issues that I noticed there, on a quick
> > look.
> > 
> > The first thing I noticed is that the index doesn't match what's
> > there, when generated from DocBook:
> > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
> > 
> > So, for instance, "Interfaces" is at the same level as "Input/Output".
> > 
> > This sounds like an something went wrong when dealing with the title
> > indentation levels during the conversion.
> >   
> 
> Yes, the hierarchical structure was broken by two causes. First was, that the
> *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) 
> placed
> all sections and sub(-sub)-sections at the same level (direct under chapter). 
> This was
> not only broken in the linux_tv book, in the other books also.
> 
> Thanks for pointing and sorry that I have overlooked this (I was so much 
> focused on
> on converting single elements to reST, that I not see the wood for the trees).
> 
> It is now fixed, may you give it a new try.

Thanks! It looks good now.

> > I would also like to see the titles numbered in chapters (and, if
> > possible, in sections and items) and to be properly indented at the 
> > index.  
> 
> BTW a few words about differences between DockBook and reST (Sphinx).
> 
> With DocBook you write *books*, the protocol (the DocBook application) has
> no facility to *chunk* and interconnect several documents. The external 
> ENTITY 
> is a workaround on the SGML layer, not on XML nor on the DB-application layer.
> Thats the reason, why so many XML-tools don't handle this entities and
> many DocBook to (e.g.) reST tools are fail.
> 
> With **standard** reST it is nearly the same, except there is a "include"
> directive on the application layer. But this directive is very simple,
> comparable to the C preprocessor "#include" directive.
> 
> With the **superset** reST-markup of Sphinx-doc you get a the "toctree" 
> directive,
> which lets you control how a document-tree should be build.
> 
>  http://www.sphinx-doc.org/en/stable/markup/toctree.html
> 
> @Mauro: you mentioned a docutils (rst2*) experience in your mail 
>   http://marc.info/?l=linux-doc=145735316012094=2
> 
>   Because the "toctree" directive -- and other directives
>   we use -- are a part of a superset of the **standard** 
>   reST, the standard docutils (like rst2*) will not work.
> 
> OK, back to your requirements: within the toctree directive you can
> set options like "maxdepth" and "numbered". It is a decision, how
> deep TOCs should go and if they should be numbered. IMO, in a
> HTML rendered page, with a proper navigation bar on the side, deep 
> TOCs in the running text have no pros, they only blow up the running
> text and bring more scrolling with. In my sense numbering chapters
> make only sense in books, not in HTML pages, where you have hyperlinks.
> 
> Just for demonstration, I added numbering in the linux-tv book:
> 
> https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3
> 
> May be, I will drop it later, because all reST sources are generated
> by a make target and I always commit the whole reST tree. As I said, 
> it is a decision which might be made later, when the migration takes 
> places.

This is the uAPI spec DocBook, that we modify frequently, as we add
more features to the Kernel, and as we make sure that all drivers will
behave the same. So, from time to time, we need to clarify some topics
at the documentation. By having a numeration, it is easier for us to
discuss things like:
"1.2.10.14. V4L2_PIX_FMT_VYUY (‘VYUY’)" is not properly
described and requires some sort of clarification.

Ok, one could also refer to it via a hyperlink, but several Kernel
media maintainers prefer to generate a single big html file, as it
makes easier to locate everything it is needed on it.

So, with the item number, they can just seek for "1.2.10.14. " string
to find the item that is under discussion.

> > Also, 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jonahtan,

Am 12.04.2016 um 17:46 schrieb Jonathan Corbet :

> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  wrote:
> 
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
>> DocBook XML documentation to reST markup. 
>> 
>> It converts 99% of the docs well ... to gain an impression how 
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>> 
>>  http://return42.github.io/sphkerneldoc/
> 
> So I've obviously been pretty quiet on this recently.  Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
> 
> Looking quickly at this work, it seems similar to the results I got.  But
> there's a lot of code there that came from somewhere?

>From me? ... except the kernel-doc script which is a fork from your 

  git://git.lwn.net/linux.git doc/sphinx


>  I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?

It depends. If you have a simple DocBook with less various markup, maybe not.
May you want to read my remarks about migration tools and especially pandoc:

* 
https://return42.github.io/sphkerneldoc/articles/dbtools.html#remarks-on-pandoc

A few more words about, what I have done:

I wrote a lib of XML filters which might be also usefully in other
migration projects (dbxml). 

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py

It uses a xml-parser, pandoc, pandoc-filters and regular expressions. Because
I did not implemented a whole converter, I hacked around pandoc. Thats why
conversion is done in several steps:

1. copy xml file(s) to a cache space

2. substitude unsolved internal and external entities

3. filter all xml files

   * run custom hooks on every node 

   * apply filters on every node and inject reST into the XML-tree where pandoc 
fails.
 https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py#L515

4. convert intermediary XML result with pandoc to json (needed by pandoc 
filters)

5. apply pandoc-filter and clean up the injected reST markup from step3

6. convert filtered json to reST

7. fix the produce reST with regular expression

... the last step is similar to your sed scripts.


And I wrote a commandline Interface to use this lib (see func db2rst):

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L146
 
With this db2rst all kernel DB-XML books could be migrated, except the linux-tv
book, which has much more complexity. For this, there is a separated commandline
called media2rst

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L107

The media2rst needs several special handlings, which is implemented in 
hooks (the dbxml interface method)

* https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py

Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where 
  dbxml comes into play

  - reading DocBook: 

https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23
  
  - writing reST has many bugs and leaks 
(you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

* dbxml brings the ability to chunk one large XML book into small 
  reST chunks e.g. kernel-hacking book:


https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

  this might helpfull e.g. if you have to convert single-column informal-tables 
  to lists or other things ... in short; dbxml and it's hooks are the key to 
hack
  everything you need in a full automated DocBook-->reST migration workflow.


--Markus--

> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.
> 
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Jonahtan,

Am 12.04.2016 um 17:46 schrieb Jonathan Corbet :

> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  wrote:
> 
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
>> DocBook XML documentation to reST markup. 
>> 
>> It converts 99% of the docs well ... to gain an impression how 
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>> 
>>  http://return42.github.io/sphkerneldoc/
> 
> So I've obviously been pretty quiet on this recently.  Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
> 
> Looking quickly at this work, it seems similar to the results I got.  But
> there's a lot of code there that came from somewhere?

>From me? ... except the kernel-doc script which is a fork from your 

  git://git.lwn.net/linux.git doc/sphinx


>  I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?

It depends. If you have a simple DocBook with less various markup, maybe not.
May you want to read my remarks about migration tools and especially pandoc:

* 
https://return42.github.io/sphkerneldoc/articles/dbtools.html#remarks-on-pandoc

A few more words about, what I have done:

I wrote a lib of XML filters which might be also usefully in other
migration projects (dbxml). 

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py

It uses a xml-parser, pandoc, pandoc-filters and regular expressions. Because
I did not implemented a whole converter, I hacked around pandoc. Thats why
conversion is done in several steps:

1. copy xml file(s) to a cache space

2. substitude unsolved internal and external entities

3. filter all xml files

   * run custom hooks on every node 

   * apply filters on every node and inject reST into the XML-tree where pandoc 
fails.
 https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py#L515

4. convert intermediary XML result with pandoc to json (needed by pandoc 
filters)

5. apply pandoc-filter and clean up the injected reST markup from step3

6. convert filtered json to reST

7. fix the produce reST with regular expression

... the last step is similar to your sed scripts.


And I wrote a commandline Interface to use this lib (see func db2rst):

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L146
 
With this db2rst all kernel DB-XML books could be migrated, except the linux-tv
book, which has much more complexity. For this, there is a separated commandline
called media2rst

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L107

The media2rst needs several special handlings, which is implemented in 
hooks (the dbxml interface method)

* https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py

Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where 
  dbxml comes into play

  - reading DocBook: 

https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23
  
  - writing reST has many bugs and leaks 
(you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

* dbxml brings the ability to chunk one large XML book into small 
  reST chunks e.g. kernel-hacking book:


https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

  this might helpfull e.g. if you have to convert single-column informal-tables 
  to lists or other things ... in short; dbxml and it's hooks are the key to 
hack
  everything you need in a full automated DocBook-->reST migration workflow.


--Markus--

> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.
> 
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro, hi kernel-doc authors, 

Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab :

> Em Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  escreveu:
> 
>> Hi kernel-doc authors,
>> 
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
>> DocBook XML documentation to reST markup. 
>> 
>> It converts 99% of the docs well ... to gain an impression how 
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>> 
>> http://return42.github.io/sphkerneldoc/
>> 
>> The sources available at:
>> 
>> https://github.com/return42/sphkerneldoc
>> 
>> The work is underway, suggestions are welcome!
>> 
>> .. have a nice weekend ..
> 
> Hi Markus,
> 
> Thanks for your work. I'm basically worried about the media docbook,
> with has some complexities that I was not able to figure out a way to
> convert it to reST.
> 
> So, let me pinpoint some issues that I noticed there, on a quick
> look.
> 
> The first thing I noticed is that the index doesn't match what's
> there, when generated from DocBook:
>   https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
> 
> So, for instance, "Interfaces" is at the same level as "Input/Output".
> 
> This sounds like an something went wrong when dealing with the title
> indentation levels during the conversion.
> 

Yes, the hierarchical structure was broken by two causes. First was, that the
*chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) 
placed
all sections and sub(-sub)-sections at the same level (direct under chapter). 
This was
not only broken in the linux_tv book, in the other books also.

Thanks for pointing and sorry that I have overlooked this (I was so much 
focused on
on converting single elements to reST, that I not see the wood for the trees).

It is now fixed, may you give it a new try.


> I would also like to see the titles numbered in chapters (and, if
> possible, in sections and items) and to be properly indented at the 
> index.

BTW a few words about differences between DockBook and reST (Sphinx).

With DocBook you write *books*, the protocol (the DocBook application) has
no facility to *chunk* and interconnect several documents. The external ENTITY 
is a workaround on the SGML layer, not on XML nor on the DB-application layer.
Thats the reason, why so many XML-tools don't handle this entities and
many DocBook to (e.g.) reST tools are fail.

With **standard** reST it is nearly the same, except there is a "include"
directive on the application layer. But this directive is very simple,
comparable to the C preprocessor "#include" directive.

With the **superset** reST-markup of Sphinx-doc you get a the "toctree" 
directive,
which lets you control how a document-tree should be build.

 http://www.sphinx-doc.org/en/stable/markup/toctree.html

@Mauro: you mentioned a docutils (rst2*) experience in your mail 
  http://marc.info/?l=linux-doc=145735316012094=2

  Because the "toctree" directive -- and other directives
  we use -- are a part of a superset of the **standard** 
  reST, the standard docutils (like rst2*) will not work.

OK, back to your requirements: within the toctree directive you can
set options like "maxdepth" and "numbered". It is a decision, how
deep TOCs should go and if they should be numbered. IMO, in a
HTML rendered page, with a proper navigation bar on the side, deep 
TOCs in the running text have no pros, they only blow up the running
text and bring more scrolling with. In my sense numbering chapters
make only sense in books, not in HTML pages, where you have hyperlinks.

Just for demonstration, I added numbering in the linux-tv book:

https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3

May be, I will drop it later, because all reST sources are generated
by a make target and I always commit the whole reST tree. As I said, 
it is a decision which might be made later, when the migration takes 
places.

> Also, it seems that there's still a lot of work to do, as there are several
> tables that are missing conversion, like the table for "struct 
> v4l2_pix_format":
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html
> 
> and the big tables at:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 

Yes, I marked them as TODO: 

 https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262

aspect *authoring tables* (see below)

> Also, some tables that are not so big like:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> 
> don't look nice, at least on my browser, as the "white" area is too small,
> and some cells seem to be broken in 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi Mauro, hi kernel-doc authors, 

Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab :

> Em Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser  escreveu:
> 
>> Hi kernel-doc authors,
>> 
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
>> DocBook XML documentation to reST markup. 
>> 
>> It converts 99% of the docs well ... to gain an impression how 
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>> 
>> http://return42.github.io/sphkerneldoc/
>> 
>> The sources available at:
>> 
>> https://github.com/return42/sphkerneldoc
>> 
>> The work is underway, suggestions are welcome!
>> 
>> .. have a nice weekend ..
> 
> Hi Markus,
> 
> Thanks for your work. I'm basically worried about the media docbook,
> with has some complexities that I was not able to figure out a way to
> convert it to reST.
> 
> So, let me pinpoint some issues that I noticed there, on a quick
> look.
> 
> The first thing I noticed is that the index doesn't match what's
> there, when generated from DocBook:
>   https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
> 
> So, for instance, "Interfaces" is at the same level as "Input/Output".
> 
> This sounds like an something went wrong when dealing with the title
> indentation levels during the conversion.
> 

Yes, the hierarchical structure was broken by two causes. First was, that the
*chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) 
placed
all sections and sub(-sub)-sections at the same level (direct under chapter). 
This was
not only broken in the linux_tv book, in the other books also.

Thanks for pointing and sorry that I have overlooked this (I was so much 
focused on
on converting single elements to reST, that I not see the wood for the trees).

It is now fixed, may you give it a new try.


> I would also like to see the titles numbered in chapters (and, if
> possible, in sections and items) and to be properly indented at the 
> index.

BTW a few words about differences between DockBook and reST (Sphinx).

With DocBook you write *books*, the protocol (the DocBook application) has
no facility to *chunk* and interconnect several documents. The external ENTITY 
is a workaround on the SGML layer, not on XML nor on the DB-application layer.
Thats the reason, why so many XML-tools don't handle this entities and
many DocBook to (e.g.) reST tools are fail.

With **standard** reST it is nearly the same, except there is a "include"
directive on the application layer. But this directive is very simple,
comparable to the C preprocessor "#include" directive.

With the **superset** reST-markup of Sphinx-doc you get a the "toctree" 
directive,
which lets you control how a document-tree should be build.

 http://www.sphinx-doc.org/en/stable/markup/toctree.html

@Mauro: you mentioned a docutils (rst2*) experience in your mail 
  http://marc.info/?l=linux-doc=145735316012094=2

  Because the "toctree" directive -- and other directives
  we use -- are a part of a superset of the **standard** 
  reST, the standard docutils (like rst2*) will not work.

OK, back to your requirements: within the toctree directive you can
set options like "maxdepth" and "numbered". It is a decision, how
deep TOCs should go and if they should be numbered. IMO, in a
HTML rendered page, with a proper navigation bar on the side, deep 
TOCs in the running text have no pros, they only blow up the running
text and bring more scrolling with. In my sense numbering chapters
make only sense in books, not in HTML pages, where you have hyperlinks.

Just for demonstration, I added numbering in the linux-tv book:

https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3

May be, I will drop it later, because all reST sources are generated
by a make target and I always commit the whole reST tree. As I said, 
it is a decision which might be made later, when the migration takes 
places.

> Also, it seems that there's still a lot of work to do, as there are several
> tables that are missing conversion, like the table for "struct 
> v4l2_pix_format":
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html
> 
> and the big tables at:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> 

Yes, I marked them as TODO: 

 https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262

aspect *authoring tables* (see below)

> Also, some tables that are not so big like:
>   
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> 
> don't look nice, at least on my browser, as the "white" area is too small,
> and some cells seem to be broken in two, because of the color changes in the 
> middle
> 

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Fri, 8 Apr 2016 17:12:27 +0200
Markus Heiser  wrote:

> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> DocBook XML documentation to reST markup. 
> 
> It converts 99% of the docs well ... to gain an impression how 
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
> 
>   http://return42.github.io/sphkerneldoc/

So I've obviously been pretty quiet on this recently.  Apologies...I've
been dealing with an extended death-in-the-family experience, and there is
still a fair amount of cleanup to be done.

Looking quickly at this work, it seems similar to the results I got.  But
there's a lot of code there that came from somewhere?  I'd put together a
fairly simple conversion using pandoc and a couple of short sed scripts;
is there a reason for a more complex solution?

Thanks for looking into this, anyway; I hope to be able to focus more on
it shortly.

jon

Re: Kernel docs: muddying the waters a bit

[Jonathan Corbet]

On Fri, 8 Apr 2016 17:12:27 +0200
Markus Heiser  wrote:

> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> DocBook XML documentation to reST markup. 
> 
> It converts 99% of the docs well ... to gain an impression how 
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
> 
>   http://return42.github.io/sphkerneldoc/

So I've obviously been pretty quiet on this recently.  Apologies...I've
been dealing with an extended death-in-the-family experience, and there is
still a fair amount of cleanup to be done.

Looking quickly at this work, it seems similar to the results I got.  But
there's a lot of code there that came from somewhere?  I'd put together a
fairly simple conversion using pandoc and a couple of short sed scripts;
is there a reason for a more complex solution?

Thanks for looking into this, anyway; I hope to be able to focus more on
it shortly.

jon

Re: Kernel docs: muddying the waters a bit

[Hans Verkuil]

Hi Markus,

On 04/08/16 17:12, Markus Heiser wrote:
> Hi kernel-doc authors,
> 
> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> DocBook XML documentation to reST markup. 
> 
> It converts 99% of the docs well ... to gain an impression how 
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
> 
>   http://return42.github.io/sphkerneldoc/
> 
> The sources available at:
> 
>   https://github.com/return42/sphkerneldoc
> 
> The work is underway, suggestions are welcome!

I have to admit that this looks pretty good :-)

My main remark based on my quick scan through the docs is that anything in
typewriter font seems to be shown as red text with a rectangle around it.
That's quite jarring for me and I think it should be just shown as normal
text, just using a non-proportional font, just like in the original.

I also noticed that the 'title' of tables ends with a '¶' character for
some reason.

See e.g. the struct v4l2_audioout table in
http://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/vidioc-g-audioout.html

Regards,

Hans

> 
> .. have a nice weekend ..
> 
> --M--
> 
> 
> Am 13.03.2016 um 16:33 schrieb Markus Heiser :
> 
>>
>> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab 
>> :
>>
>>> Em Thu, 10 Mar 2016 12:25:58 +0200
>>> Jani Nikula  escreveu:
>>>
 TL;DR? Skip to the last paragraph.

 On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
> I guess the conversion to asciidoc format is now in good shape,
> at least to demonstrate that it is possible to use this format for the
> media docbook. Still, there are lots of broken references.  

 Getting references right with asciidoc is a big problem in the
 kernel-doc side. As I wrote before, the proofs of concept only worked
 because everything was processed as one big file (via includes). The
 Asciidoctor inter-document references won't help, because we won't know
 the target document name while processing kernel-doc.
>>>
>>> I was able to produce chunked htmls here with:
>>>
>>> asciidoctor -b docbook45 media_api.adoc
>>> xmlto -o html-dir html media_api.xml
>>>
>>> The results are at:
>>> 
>>> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
>>>
>>> But yeah, all references seem to be broken there. It could be due to some
>>> conversion issue (I didn't actually tried to check what's wrong there),
>>> but I think that there's something not ok with docbook45
>>> output for multi-part documents (on both AsciiDoc and Asciidoctor).
>>>
 Sphinx is massively better at handling cross references for
 kernel-doc. We can use domains (C language) and roles (e.g. functions,
 types, etc.) for the references, which provide kind of
 namespaces. Sphinx warns for referencing non-existing targets, but
 doesn't generate broken links in the result like Asciidoctor does.

 For example, in the documentation for a function that has struct foo as
 parameter or return type, a cross reference to struct foo is added
 automagically, but only if documentation for struct foo actually
 exists. In Asciidoctor, we would have to blindly generate the references
 ourselves, and try to resolve broken links ourselves by somehow
 post-processing the result.

> Yet, from my side, if we're willing to get rid of DocBook, then
> Asciidoctor seems to be the *only* alternative so far to parse the
> complex media documents.  

 I think you mean, "get rid of DocBook as source format", not altogether?
 I'm yet to be convinved we could rely on Asciidoctor's native formats.
>>>
>>> What I mean is that, right now, I see only two alternatives for the
>>> media uAPI documentation:
>>> 1) keep using DocBook;
>>> 2) AsciiDoc/Asciidoctor.
>>>
>>> Sphinx doesn't have what's needed to support the complexity of the
>>> media books, specially since cell span seems to be possible only
>>> by using asciiArt formats. Writing a big table using asciiArt is
>>> something that is a *real pain*. Also, as tested, if the table is
>>> too big, it fails to parse such asciiArt tables. So, while Sphinx
>>> doesn't have a decent way to describe tables, we can't use it.
>>
>>
>> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
>> (mostly) you have more then one choice .. e.g. import csv tables .. 
>> but this should be discussed by example ...
>>
>>
>>> If it starts implementing it, then we can check if the other
>>> features used by the media documentation are also supported.
>>> Probably, multi-part books would be another pain with Sphinx.
>>> We have actually 4 books inside a common body. A few chapters
>>> (like book licensing, bibliography, error codes) are shared
>>> by all 4 documents.
>>>
>>> But, so far, I can't see any way to port media books without
>>> 

Re: Kernel docs: muddying the waters a bit

[Hans Verkuil]

Hi Markus,

On 04/08/16 17:12, Markus Heiser wrote:
> Hi kernel-doc authors,
> 
> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> DocBook XML documentation to reST markup. 
> 
> It converts 99% of the docs well ... to gain an impression how 
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
> 
>   http://return42.github.io/sphkerneldoc/
> 
> The sources available at:
> 
>   https://github.com/return42/sphkerneldoc
> 
> The work is underway, suggestions are welcome!

I have to admit that this looks pretty good :-)

My main remark based on my quick scan through the docs is that anything in
typewriter font seems to be shown as red text with a rectangle around it.
That's quite jarring for me and I think it should be just shown as normal
text, just using a non-proportional font, just like in the original.

I also noticed that the 'title' of tables ends with a '¶' character for
some reason.

See e.g. the struct v4l2_audioout table in
http://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/vidioc-g-audioout.html

Regards,

Hans

> 
> .. have a nice weekend ..
> 
> --M--
> 
> 
> Am 13.03.2016 um 16:33 schrieb Markus Heiser :
> 
>>
>> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab 
>> :
>>
>>> Em Thu, 10 Mar 2016 12:25:58 +0200
>>> Jani Nikula  escreveu:
>>>
 TL;DR? Skip to the last paragraph.

 On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
> I guess the conversion to asciidoc format is now in good shape,
> at least to demonstrate that it is possible to use this format for the
> media docbook. Still, there are lots of broken references.  

 Getting references right with asciidoc is a big problem in the
 kernel-doc side. As I wrote before, the proofs of concept only worked
 because everything was processed as one big file (via includes). The
 Asciidoctor inter-document references won't help, because we won't know
 the target document name while processing kernel-doc.
>>>
>>> I was able to produce chunked htmls here with:
>>>
>>> asciidoctor -b docbook45 media_api.adoc
>>> xmlto -o html-dir html media_api.xml
>>>
>>> The results are at:
>>> 
>>> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
>>>
>>> But yeah, all references seem to be broken there. It could be due to some
>>> conversion issue (I didn't actually tried to check what's wrong there),
>>> but I think that there's something not ok with docbook45
>>> output for multi-part documents (on both AsciiDoc and Asciidoctor).
>>>
 Sphinx is massively better at handling cross references for
 kernel-doc. We can use domains (C language) and roles (e.g. functions,
 types, etc.) for the references, which provide kind of
 namespaces. Sphinx warns for referencing non-existing targets, but
 doesn't generate broken links in the result like Asciidoctor does.

 For example, in the documentation for a function that has struct foo as
 parameter or return type, a cross reference to struct foo is added
 automagically, but only if documentation for struct foo actually
 exists. In Asciidoctor, we would have to blindly generate the references
 ourselves, and try to resolve broken links ourselves by somehow
 post-processing the result.

> Yet, from my side, if we're willing to get rid of DocBook, then
> Asciidoctor seems to be the *only* alternative so far to parse the
> complex media documents.  

 I think you mean, "get rid of DocBook as source format", not altogether?
 I'm yet to be convinved we could rely on Asciidoctor's native formats.
>>>
>>> What I mean is that, right now, I see only two alternatives for the
>>> media uAPI documentation:
>>> 1) keep using DocBook;
>>> 2) AsciiDoc/Asciidoctor.
>>>
>>> Sphinx doesn't have what's needed to support the complexity of the
>>> media books, specially since cell span seems to be possible only
>>> by using asciiArt formats. Writing a big table using asciiArt is
>>> something that is a *real pain*. Also, as tested, if the table is
>>> too big, it fails to parse such asciiArt tables. So, while Sphinx
>>> doesn't have a decent way to describe tables, we can't use it.
>>
>>
>> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
>> (mostly) you have more then one choice .. e.g. import csv tables .. 
>> but this should be discussed by example ...
>>
>>
>>> If it starts implementing it, then we can check if the other
>>> features used by the media documentation are also supported.
>>> Probably, multi-part books would be another pain with Sphinx.
>>> We have actually 4 books inside a common body. A few chapters
>>> (like book licensing, bibliography, error codes) are shared
>>> by all 4 documents.
>>>
>>> But, so far, I can't see any way to port media books without
>>> lots of lot of work to develop new features at the Sphinx code.
>>
>>
>> may I can help you ...
>>

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi kernel-doc authors,

motivated by this MT, I implemented a toolchain to migrate the kernel’s 
DocBook XML documentation to reST markup. 

It converts 99% of the docs well ... to gain an impression how 
kernel-docs could benefit from, visit my sphkerneldoc project page
on github:

  http://return42.github.io/sphkerneldoc/

The sources available at:

  https://github.com/return42/sphkerneldoc

The work is underway, suggestions are welcome!

.. have a nice weekend ..

--M--


Am 13.03.2016 um 16:33 schrieb Markus Heiser :

> 
> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab 
> :
> 
>> Em Thu, 10 Mar 2016 12:25:58 +0200
>> Jani Nikula  escreveu:
>> 
>>> TL;DR? Skip to the last paragraph.
>>> 
>>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
 I guess the conversion to asciidoc format is now in good shape,
 at least to demonstrate that it is possible to use this format for the
 media docbook. Still, there are lots of broken references.  
>>> 
>>> Getting references right with asciidoc is a big problem in the
>>> kernel-doc side. As I wrote before, the proofs of concept only worked
>>> because everything was processed as one big file (via includes). The
>>> Asciidoctor inter-document references won't help, because we won't know
>>> the target document name while processing kernel-doc.
>> 
>> I was able to produce chunked htmls here with:
>> 
>>  asciidoctor -b docbook45 media_api.adoc
>>  xmlto -o html-dir html media_api.xml
>> 
>> The results are at:
>>  
>> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
>> 
>> But yeah, all references seem to be broken there. It could be due to some
>> conversion issue (I didn't actually tried to check what's wrong there),
>> but I think that there's something not ok with docbook45
>> output for multi-part documents (on both AsciiDoc and Asciidoctor).
>> 
>>> Sphinx is massively better at handling cross references for
>>> kernel-doc. We can use domains (C language) and roles (e.g. functions,
>>> types, etc.) for the references, which provide kind of
>>> namespaces. Sphinx warns for referencing non-existing targets, but
>>> doesn't generate broken links in the result like Asciidoctor does.
>>> 
>>> For example, in the documentation for a function that has struct foo as
>>> parameter or return type, a cross reference to struct foo is added
>>> automagically, but only if documentation for struct foo actually
>>> exists. In Asciidoctor, we would have to blindly generate the references
>>> ourselves, and try to resolve broken links ourselves by somehow
>>> post-processing the result.
>>> 
 Yet, from my side, if we're willing to get rid of DocBook, then
 Asciidoctor seems to be the *only* alternative so far to parse the
 complex media documents.  
>>> 
>>> I think you mean, "get rid of DocBook as source format", not altogether?
>>> I'm yet to be convinved we could rely on Asciidoctor's native formats.
>> 
>> What I mean is that, right now, I see only two alternatives for the
>> media uAPI documentation:
>>  1) keep using DocBook;
>>  2) AsciiDoc/Asciidoctor.
>> 
>> Sphinx doesn't have what's needed to support the complexity of the
>> media books, specially since cell span seems to be possible only
>> by using asciiArt formats. Writing a big table using asciiArt is
>> something that is a *real pain*. Also, as tested, if the table is
>> too big, it fails to parse such asciiArt tables. So, while Sphinx
>> doesn't have a decent way to describe tables, we can't use it.
> 
> 
> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
> (mostly) you have more then one choice .. e.g. import csv tables .. 
> but this should be discussed by example ...
> 
> 
>> If it starts implementing it, then we can check if the other
>> features used by the media documentation are also supported.
>> Probably, multi-part books would be another pain with Sphinx.
>> We have actually 4 books inside a common body. A few chapters
>> (like book licensing, bibliography, error codes) are shared
>> by all 4 documents.
>> 
>> But, so far, I can't see any way to port media books without
>> lots of lot of work to develop new features at the Sphinx code.
> 
> 
> may I can help you ...
> 
> 
>>> The toolchain gets faster, easier to debug and simplified a lot with
>>> DocBook out of the equation completely. Sphinx itself is stable, widely
>>> available, and well documented. IMO there's sufficient native output
>>> format support. There are plenty of really nice extensions
>>> available. There's a possibility of doing kernel-doc as an extension in
>>> the future (either by calling current kernel-doc from the extension or
>>> by rewriting it).
>> 
>> Well, if we go to Sphinx for kernel-doc, that means that we'll need
>> 2 different tools for the documentation:
>>  - Sphinx for kernel-doc
>>  - either DocBook or 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi kernel-doc authors,

motivated by this MT, I implemented a toolchain to migrate the kernel’s 
DocBook XML documentation to reST markup. 

It converts 99% of the docs well ... to gain an impression how 
kernel-docs could benefit from, visit my sphkerneldoc project page
on github:

  http://return42.github.io/sphkerneldoc/

The sources available at:

  https://github.com/return42/sphkerneldoc

The work is underway, suggestions are welcome!

.. have a nice weekend ..

--M--


Am 13.03.2016 um 16:33 schrieb Markus Heiser :

> 
> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab 
> :
> 
>> Em Thu, 10 Mar 2016 12:25:58 +0200
>> Jani Nikula  escreveu:
>> 
>>> TL;DR? Skip to the last paragraph.
>>> 
>>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
 I guess the conversion to asciidoc format is now in good shape,
 at least to demonstrate that it is possible to use this format for the
 media docbook. Still, there are lots of broken references.  
>>> 
>>> Getting references right with asciidoc is a big problem in the
>>> kernel-doc side. As I wrote before, the proofs of concept only worked
>>> because everything was processed as one big file (via includes). The
>>> Asciidoctor inter-document references won't help, because we won't know
>>> the target document name while processing kernel-doc.
>> 
>> I was able to produce chunked htmls here with:
>> 
>>  asciidoctor -b docbook45 media_api.adoc
>>  xmlto -o html-dir html media_api.xml
>> 
>> The results are at:
>>  
>> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
>> 
>> But yeah, all references seem to be broken there. It could be due to some
>> conversion issue (I didn't actually tried to check what's wrong there),
>> but I think that there's something not ok with docbook45
>> output for multi-part documents (on both AsciiDoc and Asciidoctor).
>> 
>>> Sphinx is massively better at handling cross references for
>>> kernel-doc. We can use domains (C language) and roles (e.g. functions,
>>> types, etc.) for the references, which provide kind of
>>> namespaces. Sphinx warns for referencing non-existing targets, but
>>> doesn't generate broken links in the result like Asciidoctor does.
>>> 
>>> For example, in the documentation for a function that has struct foo as
>>> parameter or return type, a cross reference to struct foo is added
>>> automagically, but only if documentation for struct foo actually
>>> exists. In Asciidoctor, we would have to blindly generate the references
>>> ourselves, and try to resolve broken links ourselves by somehow
>>> post-processing the result.
>>> 
 Yet, from my side, if we're willing to get rid of DocBook, then
 Asciidoctor seems to be the *only* alternative so far to parse the
 complex media documents.  
>>> 
>>> I think you mean, "get rid of DocBook as source format", not altogether?
>>> I'm yet to be convinved we could rely on Asciidoctor's native formats.
>> 
>> What I mean is that, right now, I see only two alternatives for the
>> media uAPI documentation:
>>  1) keep using DocBook;
>>  2) AsciiDoc/Asciidoctor.
>> 
>> Sphinx doesn't have what's needed to support the complexity of the
>> media books, specially since cell span seems to be possible only
>> by using asciiArt formats. Writing a big table using asciiArt is
>> something that is a *real pain*. Also, as tested, if the table is
>> too big, it fails to parse such asciiArt tables. So, while Sphinx
>> doesn't have a decent way to describe tables, we can't use it.
> 
> 
> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
> (mostly) you have more then one choice .. e.g. import csv tables .. 
> but this should be discussed by example ...
> 
> 
>> If it starts implementing it, then we can check if the other
>> features used by the media documentation are also supported.
>> Probably, multi-part books would be another pain with Sphinx.
>> We have actually 4 books inside a common body. A few chapters
>> (like book licensing, bibliography, error codes) are shared
>> by all 4 documents.
>> 
>> But, so far, I can't see any way to port media books without
>> lots of lot of work to develop new features at the Sphinx code.
> 
> 
> may I can help you ...
> 
> 
>>> The toolchain gets faster, easier to debug and simplified a lot with
>>> DocBook out of the equation completely. Sphinx itself is stable, widely
>>> available, and well documented. IMO there's sufficient native output
>>> format support. There are plenty of really nice extensions
>>> available. There's a possibility of doing kernel-doc as an extension in
>>> the future (either by calling current kernel-doc from the extension or
>>> by rewriting it).
>> 
>> Well, if we go to Sphinx for kernel-doc, that means that we'll need
>> 2 different tools for the documentation:
>>  - Sphinx for kernel-doc
>>  - either DocBook or Asciidoctor/AsciiDoc for media.
>> 
>> IMHO, this is the worse scenario, as we'll keep depending on
>> 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab :

> Em Thu, 10 Mar 2016 12:25:58 +0200
> Jani Nikula  escreveu:
> 
>> TL;DR? Skip to the last paragraph.
>> 
>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
>>> I guess the conversion to asciidoc format is now in good shape,
>>> at least to demonstrate that it is possible to use this format for the
>>> media docbook. Still, there are lots of broken references.  
>> 
>> Getting references right with asciidoc is a big problem in the
>> kernel-doc side. As I wrote before, the proofs of concept only worked
>> because everything was processed as one big file (via includes). The
>> Asciidoctor inter-document references won't help, because we won't know
>> the target document name while processing kernel-doc.
> 
> I was able to produce chunked htmls here with:
> 
>   asciidoctor -b docbook45 media_api.adoc
>   xmlto -o html-dir html media_api.xml
> 
> The results are at:
>   
> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
> 
> But yeah, all references seem to be broken there. It could be due to some
> conversion issue (I didn't actually tried to check what's wrong there),
> but I think that there's something not ok with docbook45
> output for multi-part documents (on both AsciiDoc and Asciidoctor).
> 
>> Sphinx is massively better at handling cross references for
>> kernel-doc. We can use domains (C language) and roles (e.g. functions,
>> types, etc.) for the references, which provide kind of
>> namespaces. Sphinx warns for referencing non-existing targets, but
>> doesn't generate broken links in the result like Asciidoctor does.
>> 
>> For example, in the documentation for a function that has struct foo as
>> parameter or return type, a cross reference to struct foo is added
>> automagically, but only if documentation for struct foo actually
>> exists. In Asciidoctor, we would have to blindly generate the references
>> ourselves, and try to resolve broken links ourselves by somehow
>> post-processing the result.
>> 
>>> Yet, from my side, if we're willing to get rid of DocBook, then
>>> Asciidoctor seems to be the *only* alternative so far to parse the
>>> complex media documents.  
>> 
>> I think you mean, "get rid of DocBook as source format", not altogether?
>> I'm yet to be convinved we could rely on Asciidoctor's native formats.
> 
> What I mean is that, right now, I see only two alternatives for the
> media uAPI documentation:
>   1) keep using DocBook;
>   2) AsciiDoc/Asciidoctor.
> 
> Sphinx doesn't have what's needed to support the complexity of the
> media books, specially since cell span seems to be possible only
> by using asciiArt formats. Writing a big table using asciiArt is
> something that is a *real pain*. Also, as tested, if the table is
> too big, it fails to parse such asciiArt tables. So, while Sphinx
> doesn't have a decent way to describe tables, we can't use it.


Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
(mostly) you have more then one choice .. e.g. import csv tables .. 
but this should be discussed by example ...


> If it starts implementing it, then we can check if the other
> features used by the media documentation are also supported.
> Probably, multi-part books would be another pain with Sphinx.
> We have actually 4 books inside a common body. A few chapters
> (like book licensing, bibliography, error codes) are shared
> by all 4 documents.
> 
> But, so far, I can't see any way to port media books without
> lots of lot of work to develop new features at the Sphinx code.


may I can help you ...


>> The toolchain gets faster, easier to debug and simplified a lot with
>> DocBook out of the equation completely. Sphinx itself is stable, widely
>> available, and well documented. IMO there's sufficient native output
>> format support. There are plenty of really nice extensions
>> available. There's a possibility of doing kernel-doc as an extension in
>> the future (either by calling current kernel-doc from the extension or
>> by rewriting it).
> 
> Well, if we go to Sphinx for kernel-doc, that means that we'll need
> 2 different tools for the documentation:
>   - Sphinx for kernel-doc
>   - either DocBook or Asciidoctor/AsciiDoc for media.
> 
> IMHO, this is the worse scenario, as we'll keep depending on
> DocBook plus requiring Sphinx, but it is up to Jon to decide.
> 

The migration of kernel-doc is a long term project, not a
one shot job. The scope of documents to migrate is not limited
to the files with DocBook markup in, most documents have not
a real markup.

Please take a look at my thoughts and efforts about migration.

* https://sphkerneldoc.readthedocs.org

* https://github.com/return42/sphkerneldoc.git

sphkerneldoc.git is a small project started this weekend, within
this project I show you, how migration could be done and
we can discuss concerns like 

Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab :

> Em Thu, 10 Mar 2016 12:25:58 +0200
> Jani Nikula  escreveu:
> 
>> TL;DR? Skip to the last paragraph.
>> 
>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
>>> I guess the conversion to asciidoc format is now in good shape,
>>> at least to demonstrate that it is possible to use this format for the
>>> media docbook. Still, there are lots of broken references.  
>> 
>> Getting references right with asciidoc is a big problem in the
>> kernel-doc side. As I wrote before, the proofs of concept only worked
>> because everything was processed as one big file (via includes). The
>> Asciidoctor inter-document references won't help, because we won't know
>> the target document name while processing kernel-doc.
> 
> I was able to produce chunked htmls here with:
> 
>   asciidoctor -b docbook45 media_api.adoc
>   xmlto -o html-dir html media_api.xml
> 
> The results are at:
>   
> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
> 
> But yeah, all references seem to be broken there. It could be due to some
> conversion issue (I didn't actually tried to check what's wrong there),
> but I think that there's something not ok with docbook45
> output for multi-part documents (on both AsciiDoc and Asciidoctor).
> 
>> Sphinx is massively better at handling cross references for
>> kernel-doc. We can use domains (C language) and roles (e.g. functions,
>> types, etc.) for the references, which provide kind of
>> namespaces. Sphinx warns for referencing non-existing targets, but
>> doesn't generate broken links in the result like Asciidoctor does.
>> 
>> For example, in the documentation for a function that has struct foo as
>> parameter or return type, a cross reference to struct foo is added
>> automagically, but only if documentation for struct foo actually
>> exists. In Asciidoctor, we would have to blindly generate the references
>> ourselves, and try to resolve broken links ourselves by somehow
>> post-processing the result.
>> 
>>> Yet, from my side, if we're willing to get rid of DocBook, then
>>> Asciidoctor seems to be the *only* alternative so far to parse the
>>> complex media documents.  
>> 
>> I think you mean, "get rid of DocBook as source format", not altogether?
>> I'm yet to be convinved we could rely on Asciidoctor's native formats.
> 
> What I mean is that, right now, I see only two alternatives for the
> media uAPI documentation:
>   1) keep using DocBook;
>   2) AsciiDoc/Asciidoctor.
> 
> Sphinx doesn't have what's needed to support the complexity of the
> media books, specially since cell span seems to be possible only
> by using asciiArt formats. Writing a big table using asciiArt is
> something that is a *real pain*. Also, as tested, if the table is
> too big, it fails to parse such asciiArt tables. So, while Sphinx
> doesn't have a decent way to describe tables, we can't use it.


Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
(mostly) you have more then one choice .. e.g. import csv tables .. 
but this should be discussed by example ...


> If it starts implementing it, then we can check if the other
> features used by the media documentation are also supported.
> Probably, multi-part books would be another pain with Sphinx.
> We have actually 4 books inside a common body. A few chapters
> (like book licensing, bibliography, error codes) are shared
> by all 4 documents.
> 
> But, so far, I can't see any way to port media books without
> lots of lot of work to develop new features at the Sphinx code.


may I can help you ...


>> The toolchain gets faster, easier to debug and simplified a lot with
>> DocBook out of the equation completely. Sphinx itself is stable, widely
>> available, and well documented. IMO there's sufficient native output
>> format support. There are plenty of really nice extensions
>> available. There's a possibility of doing kernel-doc as an extension in
>> the future (either by calling current kernel-doc from the extension or
>> by rewriting it).
> 
> Well, if we go to Sphinx for kernel-doc, that means that we'll need
> 2 different tools for the documentation:
>   - Sphinx for kernel-doc
>   - either DocBook or Asciidoctor/AsciiDoc for media.
> 
> IMHO, this is the worse scenario, as we'll keep depending on
> DocBook plus requiring Sphinx, but it is up to Jon to decide.
> 

The migration of kernel-doc is a long term project, not a
one shot job. The scope of documents to migrate is not limited
to the files with DocBook markup in, most documents have not
a real markup.

Please take a look at my thoughts and efforts about migration.

* https://sphkerneldoc.readthedocs.org

* https://github.com/return42/sphkerneldoc.git

sphkerneldoc.git is a small project started this weekend, within
this project I show you, how migration could be done and
we can discuss concerns like "tables and cell-spans" by example. 

Believe me, most concerns 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Thu, 10 Mar 2016 12:25:58 +0200
Jani Nikula  escreveu:

> TL;DR? Skip to the last paragraph.
> 
> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
> > I guess the conversion to asciidoc format is now in good shape,
> > at least to demonstrate that it is possible to use this format for the
> > media docbook. Still, there are lots of broken references.  
> 
> Getting references right with asciidoc is a big problem in the
> kernel-doc side. As I wrote before, the proofs of concept only worked
> because everything was processed as one big file (via includes). The
> Asciidoctor inter-document references won't help, because we won't know
> the target document name while processing kernel-doc.

I was able to produce chunked htmls here with:

asciidoctor -b docbook45 media_api.adoc
xmlto -o html-dir html media_api.xml

The results are at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/

But yeah, all references seem to be broken there. It could be due to some
conversion issue (I didn't actually tried to check what's wrong there),
but I think that there's something not ok with docbook45
output for multi-part documents (on both AsciiDoc and Asciidoctor).

> Sphinx is massively better at handling cross references for
> kernel-doc. We can use domains (C language) and roles (e.g. functions,
> types, etc.) for the references, which provide kind of
> namespaces. Sphinx warns for referencing non-existing targets, but
> doesn't generate broken links in the result like Asciidoctor does.
> 
> For example, in the documentation for a function that has struct foo as
> parameter or return type, a cross reference to struct foo is added
> automagically, but only if documentation for struct foo actually
> exists. In Asciidoctor, we would have to blindly generate the references
> ourselves, and try to resolve broken links ourselves by somehow
> post-processing the result.
> 
> > Yet, from my side, if we're willing to get rid of DocBook, then
> > Asciidoctor seems to be the *only* alternative so far to parse the
> > complex media documents.  
> 
> I think you mean, "get rid of DocBook as source format", not altogether?
> I'm yet to be convinved we could rely on Asciidoctor's native formats.

What I mean is that, right now, I see only two alternatives for the
media uAPI documentation:
1) keep using DocBook;
2) AsciiDoc/Asciidoctor.

Sphinx doesn't have what's needed to support the complexity of the
media books, specially since cell span seems to be possible only
by using asciiArt formats. Writing a big table using asciiArt is
something that is a *real pain*. Also, as tested, if the table is
too big, it fails to parse such asciiArt tables. So, while Sphinx
doesn't have a decent way to describe tables, we can't use it.

If it starts implementing it, then we can check if the other
features used by the media documentation are also supported.
Probably, multi-part books would be another pain with Sphinx.
We have actually 4 books inside a common body. A few chapters
(like book licensing, bibliography, error codes) are shared
by all 4 documents.

But, so far, I can't see any way to port media books without
lots of lot of work to develop new features at the Sphinx code.

> ---
> 
> Mauro, I truly appreciate your efforts at evaluating both
> alternatives. I also appreciate Dan's inputs on Asciidoctor.
> 
> Despite your evaluation that Asciidoctor is the only alternative for
> media documents, it is my opinion that we should go with Sphinx.
> 
> It's an opinion, it's subjective, it's from my perspective, especially
> from the kernel-doc POV, so please don't take it as a slap in the face
> after all the work you've done. With that out of the way, here's why.
> 
> For starters, Jon's Sphinx proof-of-concept at
> http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and
> usable. Cross references work, there are no broken links (I hacked a bit
> more on kernel-doc and it gets even better). There's embedded search
> (and if this gets exported to https://readthedocs.org/ the search is
> even better). The API documentation is sensible and the headings aren't
> mixed up with other headings. It's all there. It's what we've been
> looking for.
> 
> The toolchain gets faster, easier to debug and simplified a lot with
> DocBook out of the equation completely. Sphinx itself is stable, widely
> available, and well documented. IMO there's sufficient native output
> format support. There are plenty of really nice extensions
> available. There's a possibility of doing kernel-doc as an extension in
> the future (either by calling current kernel-doc from the extension or
> by rewriting it).

Well, if we go to Sphinx for kernel-doc, that means that we'll need
2 different tools for the documentation:
- Sphinx for kernel-doc
- either DocBook or Asciidoctor/AsciiDoc for media.

IMHO, this is the worse scenario, as 

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Thu, 10 Mar 2016 12:25:58 +0200
Jani Nikula  escreveu:

> TL;DR? Skip to the last paragraph.
> 
> On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
> > I guess the conversion to asciidoc format is now in good shape,
> > at least to demonstrate that it is possible to use this format for the
> > media docbook. Still, there are lots of broken references.  
> 
> Getting references right with asciidoc is a big problem in the
> kernel-doc side. As I wrote before, the proofs of concept only worked
> because everything was processed as one big file (via includes). The
> Asciidoctor inter-document references won't help, because we won't know
> the target document name while processing kernel-doc.

I was able to produce chunked htmls here with:

asciidoctor -b docbook45 media_api.adoc
xmlto -o html-dir html media_api.xml

The results are at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/

But yeah, all references seem to be broken there. It could be due to some
conversion issue (I didn't actually tried to check what's wrong there),
but I think that there's something not ok with docbook45
output for multi-part documents (on both AsciiDoc and Asciidoctor).

> Sphinx is massively better at handling cross references for
> kernel-doc. We can use domains (C language) and roles (e.g. functions,
> types, etc.) for the references, which provide kind of
> namespaces. Sphinx warns for referencing non-existing targets, but
> doesn't generate broken links in the result like Asciidoctor does.
> 
> For example, in the documentation for a function that has struct foo as
> parameter or return type, a cross reference to struct foo is added
> automagically, but only if documentation for struct foo actually
> exists. In Asciidoctor, we would have to blindly generate the references
> ourselves, and try to resolve broken links ourselves by somehow
> post-processing the result.
> 
> > Yet, from my side, if we're willing to get rid of DocBook, then
> > Asciidoctor seems to be the *only* alternative so far to parse the
> > complex media documents.  
> 
> I think you mean, "get rid of DocBook as source format", not altogether?
> I'm yet to be convinved we could rely on Asciidoctor's native formats.

What I mean is that, right now, I see only two alternatives for the
media uAPI documentation:
1) keep using DocBook;
2) AsciiDoc/Asciidoctor.

Sphinx doesn't have what's needed to support the complexity of the
media books, specially since cell span seems to be possible only
by using asciiArt formats. Writing a big table using asciiArt is
something that is a *real pain*. Also, as tested, if the table is
too big, it fails to parse such asciiArt tables. So, while Sphinx
doesn't have a decent way to describe tables, we can't use it.

If it starts implementing it, then we can check if the other
features used by the media documentation are also supported.
Probably, multi-part books would be another pain with Sphinx.
We have actually 4 books inside a common body. A few chapters
(like book licensing, bibliography, error codes) are shared
by all 4 documents.

But, so far, I can't see any way to port media books without
lots of lot of work to develop new features at the Sphinx code.

> ---
> 
> Mauro, I truly appreciate your efforts at evaluating both
> alternatives. I also appreciate Dan's inputs on Asciidoctor.
> 
> Despite your evaluation that Asciidoctor is the only alternative for
> media documents, it is my opinion that we should go with Sphinx.
> 
> It's an opinion, it's subjective, it's from my perspective, especially
> from the kernel-doc POV, so please don't take it as a slap in the face
> after all the work you've done. With that out of the way, here's why.
> 
> For starters, Jon's Sphinx proof-of-concept at
> http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and
> usable. Cross references work, there are no broken links (I hacked a bit
> more on kernel-doc and it gets even better). There's embedded search
> (and if this gets exported to https://readthedocs.org/ the search is
> even better). The API documentation is sensible and the headings aren't
> mixed up with other headings. It's all there. It's what we've been
> looking for.
> 
> The toolchain gets faster, easier to debug and simplified a lot with
> DocBook out of the equation completely. Sphinx itself is stable, widely
> available, and well documented. IMO there's sufficient native output
> format support. There are plenty of really nice extensions
> available. There's a possibility of doing kernel-doc as an extension in
> the future (either by calling current kernel-doc from the extension or
> by rewriting it).

Well, if we go to Sphinx for kernel-doc, that means that we'll need
2 different tools for the documentation:
- Sphinx for kernel-doc
- either DocBook or Asciidoctor/AsciiDoc for media.

IMHO, this is the worse scenario, as we'll keep depending on
DocBook plus requiring 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

TL;DR? Skip to the last paragraph.

On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
> I guess the conversion to asciidoc format is now in good shape,
> at least to demonstrate that it is possible to use this format for the
> media docbook. Still, there are lots of broken references.

Getting references right with asciidoc is a big problem in the
kernel-doc side. As I wrote before, the proofs of concept only worked
because everything was processed as one big file (via includes). The
Asciidoctor inter-document references won't help, because we won't know
the target document name while processing kernel-doc.

Sphinx is massively better at handling cross references for
kernel-doc. We can use domains (C language) and roles (e.g. functions,
types, etc.) for the references, which provide kind of
namespaces. Sphinx warns for referencing non-existing targets, but
doesn't generate broken links in the result like Asciidoctor does.

For example, in the documentation for a function that has struct foo as
parameter or return type, a cross reference to struct foo is added
automagically, but only if documentation for struct foo actually
exists. In Asciidoctor, we would have to blindly generate the references
ourselves, and try to resolve broken links ourselves by somehow
post-processing the result.

> Yet, from my side, if we're willing to get rid of DocBook, then
> Asciidoctor seems to be the *only* alternative so far to parse the
> complex media documents.

I think you mean, "get rid of DocBook as source format", not altogether?
I'm yet to be convinved we could rely on Asciidoctor's native formats.

---

Mauro, I truly appreciate your efforts at evaluating both
alternatives. I also appreciate Dan's inputs on Asciidoctor.

Despite your evaluation that Asciidoctor is the only alternative for
media documents, it is my opinion that we should go with Sphinx.

It's an opinion, it's subjective, it's from my perspective, especially
from the kernel-doc POV, so please don't take it as a slap in the face
after all the work you've done. With that out of the way, here's why.

For starters, Jon's Sphinx proof-of-concept at
http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and
usable. Cross references work, there are no broken links (I hacked a bit
more on kernel-doc and it gets even better). There's embedded search
(and if this gets exported to https://readthedocs.org/ the search is
even better). The API documentation is sensible and the headings aren't
mixed up with other headings. It's all there. It's what we've been
looking for.

The toolchain gets faster, easier to debug and simplified a lot with
DocBook out of the equation completely. Sphinx itself is stable, widely
available, and well documented. IMO there's sufficient native output
format support. There are plenty of really nice extensions
available. There's a possibility of doing kernel-doc as an extension in
the future (either by calling current kernel-doc from the extension or
by rewriting it).

Dan keeps bringing up the active community in Asciidoctor, and how
they're fixing things up as we speak... which is great, but Sphinx is
here now, packaged and shipping in distros ready to use. It seems that
of the two, an Asciidoctor based toolchain is currently more in need of
hacking and extending to meet our needs. Which brings us to the
implementation language, Python vs. Ruby.

I won't make the mistake of comparing the relative merits of the
languages, but I'll boldly claim the set of kernel developers who know
Python is likely larger than the set of kernel developers who know Ruby
[citation needed]. AFAICT there are no Ruby tools in the kernel tree,
but there is a bunch of Python. My own very limited and subjective
experience with other tools around the kernel is that Python is much
more popular than Ruby. So my claim here is that we're in a better
position to hack on Sphinx extensions ourselves than Asciidoctor.

My conclusion is that Sphinx covers the vast majority of the needs of
our documentation producers and consumers, in an amazing way, out of the
box, better than Asciidoctor.

Which brings us to the minority and the parts where Sphinx falls short,
media documentation in particular. It's complex documentation, with very
specific requirements on the output, especially that many things remain
exactly as they are now. It also feels like the target is more to have
standalone media documentation, and not so much to be aligned with and
be part of the rest of the kernel documentation.

I want to question the need to have all kernel documentation use tools
that meet the strict requirements of the outlier, when there's a better
alternative for the vast majority of the documentation. Especially when
Asciidoctor isn't a ready solution for media documentation either.

In summary, my proposal is to go with Sphinx, leave media docs as
DocBook for now, and see if and how they can be converted to
Sphinx/reStructuredText later on when we 

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

TL;DR? Skip to the last paragraph.

On Wed, 09 Mar 2016, Mauro Carvalho Chehab  wrote:
> I guess the conversion to asciidoc format is now in good shape,
> at least to demonstrate that it is possible to use this format for the
> media docbook. Still, there are lots of broken references.

Getting references right with asciidoc is a big problem in the
kernel-doc side. As I wrote before, the proofs of concept only worked
because everything was processed as one big file (via includes). The
Asciidoctor inter-document references won't help, because we won't know
the target document name while processing kernel-doc.

Sphinx is massively better at handling cross references for
kernel-doc. We can use domains (C language) and roles (e.g. functions,
types, etc.) for the references, which provide kind of
namespaces. Sphinx warns for referencing non-existing targets, but
doesn't generate broken links in the result like Asciidoctor does.

For example, in the documentation for a function that has struct foo as
parameter or return type, a cross reference to struct foo is added
automagically, but only if documentation for struct foo actually
exists. In Asciidoctor, we would have to blindly generate the references
ourselves, and try to resolve broken links ourselves by somehow
post-processing the result.

> Yet, from my side, if we're willing to get rid of DocBook, then
> Asciidoctor seems to be the *only* alternative so far to parse the
> complex media documents.

I think you mean, "get rid of DocBook as source format", not altogether?
I'm yet to be convinved we could rely on Asciidoctor's native formats.

---

Mauro, I truly appreciate your efforts at evaluating both
alternatives. I also appreciate Dan's inputs on Asciidoctor.

Despite your evaluation that Asciidoctor is the only alternative for
media documents, it is my opinion that we should go with Sphinx.

It's an opinion, it's subjective, it's from my perspective, especially
from the kernel-doc POV, so please don't take it as a slap in the face
after all the work you've done. With that out of the way, here's why.

For starters, Jon's Sphinx proof-of-concept at
http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and
usable. Cross references work, there are no broken links (I hacked a bit
more on kernel-doc and it gets even better). There's embedded search
(and if this gets exported to https://readthedocs.org/ the search is
even better). The API documentation is sensible and the headings aren't
mixed up with other headings. It's all there. It's what we've been
looking for.

The toolchain gets faster, easier to debug and simplified a lot with
DocBook out of the equation completely. Sphinx itself is stable, widely
available, and well documented. IMO there's sufficient native output
format support. There are plenty of really nice extensions
available. There's a possibility of doing kernel-doc as an extension in
the future (either by calling current kernel-doc from the extension or
by rewriting it).

Dan keeps bringing up the active community in Asciidoctor, and how
they're fixing things up as we speak... which is great, but Sphinx is
here now, packaged and shipping in distros ready to use. It seems that
of the two, an Asciidoctor based toolchain is currently more in need of
hacking and extending to meet our needs. Which brings us to the
implementation language, Python vs. Ruby.

I won't make the mistake of comparing the relative merits of the
languages, but I'll boldly claim the set of kernel developers who know
Python is likely larger than the set of kernel developers who know Ruby
[citation needed]. AFAICT there are no Ruby tools in the kernel tree,
but there is a bunch of Python. My own very limited and subjective
experience with other tools around the kernel is that Python is much
more popular than Ruby. So my claim here is that we're in a better
position to hack on Sphinx extensions ourselves than Asciidoctor.

My conclusion is that Sphinx covers the vast majority of the needs of
our documentation producers and consumers, in an amazing way, out of the
box, better than Asciidoctor.

Which brings us to the minority and the parts where Sphinx falls short,
media documentation in particular. It's complex documentation, with very
specific requirements on the output, especially that many things remain
exactly as they are now. It also feels like the target is more to have
standalone media documentation, and not so much to be aligned with and
be part of the rest of the kernel documentation.

I want to question the need to have all kernel documentation use tools
that meet the strict requirements of the outlier, when there's a better
alternative for the vast majority of the documentation. Especially when
Asciidoctor isn't a ready solution for media documentation either.

In summary, my proposal is to go with Sphinx, leave media docs as
DocBook for now, and see if and how they can be converted to
Sphinx/reStructuredText later on when we have everything else in

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 12:39:21 -0300
Mauro Carvalho Chehab  escreveu:

> Pandoc failed to fully convert it, but at least it left all the texts,
> with prevented rewriting it from scratch. This is the manual fix
> I applied to it:
>   
> https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc
> 
> FYI, we have 133 xml files at the media uAPI doc with refmeta.

I used pandoc to convert from the html files and manually edited it.
I also fixed lots of other issues with the conversion.

I guess the conversion to asciidoc format is now in good shape,
at least to demonstrate that it is possible to use this format for the
media docbook. Still, there are lots of broken references.

The proof of concept html file is at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

I also added the ascii doc files there, at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/   

And I'm keeping the git tree, with helps to identify the work that was
needed to make it work:
https://git.linuxtv.org/mchehab/asciidoc-poc.git

In summary, AsciiDoc, formatted via AsciiDoctor worked fine to produce
an html file.

PROBLEMS


1)

I was not able to produce outputs on any other format.

For example, when trying to generate docbook45 output, it sems that
part of the trouble was due to pandoc conversion. It produces
links like:

link:#ftn.id-1.4.11.43.5.11.2.7.2.6.2[^[a]^]

Which causes errors with DocBook parsers, like xmllint:

media_api.xml:32300: parser error : Opening and ending tag mismatch: 
superscript line 32300 and ulink
[a]
 ^

I suspect that this is fixable. I may try to fix it later.

2) It seems that Asciidoctor doesn't allow annexes per document part.
It numberates them as chapters, instead of using A, B, C, ...

3) Even producing the html without troubles, it produces an error:
asciidoctor: ERROR: media_api.adoc: line 57: invalid part, must have at 
least one section (e.g., chapter, appendix, etc.)

4) There are some things that got lost during the conversion, like
copyright notes and revision notes. This could be simply a problem
with pandoc conversion. Nothing serious, I guess, as we could insert
the lost data manually. Yet, it means that, to move from the PoC to
the Kernel, there are still lots of work to do.

I was unable do discover why, nor to suppress this error message.

Yet, from my side, if we're willing to get rid of DocBook, then
Asciidoctor seems to be the *only* alternative so far to parse the
complex media documents.

Regards,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 12:39:21 -0300
Mauro Carvalho Chehab  escreveu:

> Pandoc failed to fully convert it, but at least it left all the texts,
> with prevented rewriting it from scratch. This is the manual fix
> I applied to it:
>   
> https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc
> 
> FYI, we have 133 xml files at the media uAPI doc with refmeta.

I used pandoc to convert from the html files and manually edited it.
I also fixed lots of other issues with the conversion.

I guess the conversion to asciidoc format is now in good shape,
at least to demonstrate that it is possible to use this format for the
media docbook. Still, there are lots of broken references.

The proof of concept html file is at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

I also added the ascii doc files there, at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/   

And I'm keeping the git tree, with helps to identify the work that was
needed to make it work:
https://git.linuxtv.org/mchehab/asciidoc-poc.git

In summary, AsciiDoc, formatted via AsciiDoctor worked fine to produce
an html file.

PROBLEMS


1)

I was not able to produce outputs on any other format.

For example, when trying to generate docbook45 output, it sems that
part of the trouble was due to pandoc conversion. It produces
links like:

link:#ftn.id-1.4.11.43.5.11.2.7.2.6.2[^[a]^]

Which causes errors with DocBook parsers, like xmllint:

media_api.xml:32300: parser error : Opening and ending tag mismatch: 
superscript line 32300 and ulink
[a]
 ^

I suspect that this is fixable. I may try to fix it later.

2) It seems that Asciidoctor doesn't allow annexes per document part.
It numberates them as chapters, instead of using A, B, C, ...

3) Even producing the html without troubles, it produces an error:
asciidoctor: ERROR: media_api.adoc: line 57: invalid part, must have at 
least one section (e.g., chapter, appendix, etc.)

4) There are some things that got lost during the conversion, like
copyright notes and revision notes. This could be simply a problem
with pandoc conversion. Nothing serious, I guess, as we could insert
the lost data manually. Yet, it means that, to move from the PoC to
the Kernel, there are still lots of work to do.

I was unable do discover why, nor to suppress this error message.

Yet, from my side, if we're willing to get rid of DocBook, then
Asciidoctor seems to be the *only* alternative so far to parse the
complex media documents.

Regards,
Mauro

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 09 Mar 2016, Dan Allen  wrote:
> On Tue, Mar 8, 2016 at 6:58 AM, Jani Nikula  wrote:
>
>> I need to look into this again. Is there a specific option or directive
>> to produce split output for includes? When I tried this, the result was
>> just one big output file. (And indeed we'd need both. Some includes we
>> want embedded, some includes should produce separate outputs.)
>>
>
> Nope. What I'm saying is that you run Asciidoctor on each sub-master
> include file (an include that manages a part or chapter). That gives you
> your individual part/chapter files. Then you need to make an index page,
> probably by using the Asciidoctor API to itemize all the chapters as a list
> or something.

Bummer.

Getting the inter-document cross references right may become
tricky. We'll be generating plenty of snippets of lightweight markup
from source code documentation comments. At the time of processing, we
won't know where e.g. a specific function to be cross referenced is
documented, if at all. We can't require the documentation comment
writers to figure that out either; it's too burdensome, too ugly in the
code, and they'll bitrot quickly.

Cross referencing in the asciidoc proofs of concept have worked because
they've all done the processing as a single single unit, with
includes. These hacks have also ignored any broken links, and there have
been

> Yes, it does require some thinking about cross references. There is a lot
> more we can do out of the box, but all those references can be fixed with a
> little bit of post-processing in the meantime.

It seems to me Sphinx provides much better support regarding cross
references, out of the box, within documents and to external documents
(intersphinx), with target roles and domains, including validation and
not creating broken links in the output.

Looking at the current hacks we have for post-processing references, I'm
really not thrilled about the prospect of keeping or redoing that.

See how this works in Jon's Sphinx test [1]. At the time of generating
the markup from source comments, there is no idea if and where
gem_init_hw() and intel_guc_ucode_init() are documented. Indeed,
documentation for the former does not exist, but there's no broken link.

BR,
Jani.


[1] http://static.lwn.net/kerneldoc/gpu.html#c.intel_guc_ucode_load


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Wed, 09 Mar 2016, Dan Allen  wrote:
> On Tue, Mar 8, 2016 at 6:58 AM, Jani Nikula  wrote:
>
>> I need to look into this again. Is there a specific option or directive
>> to produce split output for includes? When I tried this, the result was
>> just one big output file. (And indeed we'd need both. Some includes we
>> want embedded, some includes should produce separate outputs.)
>>
>
> Nope. What I'm saying is that you run Asciidoctor on each sub-master
> include file (an include that manages a part or chapter). That gives you
> your individual part/chapter files. Then you need to make an index page,
> probably by using the Asciidoctor API to itemize all the chapters as a list
> or something.

Bummer.

Getting the inter-document cross references right may become
tricky. We'll be generating plenty of snippets of lightweight markup
from source code documentation comments. At the time of processing, we
won't know where e.g. a specific function to be cross referenced is
documented, if at all. We can't require the documentation comment
writers to figure that out either; it's too burdensome, too ugly in the
code, and they'll bitrot quickly.

Cross referencing in the asciidoc proofs of concept have worked because
they've all done the processing as a single single unit, with
includes. These hacks have also ignored any broken links, and there have
been

> Yes, it does require some thinking about cross references. There is a lot
> more we can do out of the box, but all those references can be fixed with a
> little bit of post-processing in the meantime.

It seems to me Sphinx provides much better support regarding cross
references, out of the box, within documents and to external documents
(intersphinx), with target roles and domains, including validation and
not creating broken links in the output.

Looking at the current hacks we have for post-processing references, I'm
really not thrilled about the prospect of keeping or redoing that.

See how this works in Jon's Sphinx test [1]. At the time of generating
the markup from source comments, there is no idea if and where
gem_init_hw() and intel_guc_ucode_init() are documented. Indeed,
documentation for the former does not exist, but there's no broken link.

BR,
Jani.


[1] http://static.lwn.net/kerneldoc/gpu.html#c.intel_guc_ucode_load


-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 10:39:22 -0300
Mauro Carvalho Chehab  escreveu:

> Em Tue, 08 Mar 2016 05:13:13 -0700
> Dan Allen  escreveu:
> 
> > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <  
> > mche...@osg.samsung.com> wrote:
> >   
> > > pandoc did a really crap job on the conversion. To convert this
> > > into something useful, we'll need to spend a lot of time, as it lost
> > > most of the cross-references, as they were defined via DocBook macros.
> > >
> > 
> > I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> > the works called DocBookRx.
> > 
> > https://github.com/opendevise/docbookrx
> > 
> > It has converted several very serious DocBook documents and we're
> > continuing to improve it. It's also a lot easier to hack than pandoc.  
> 
> Didn't work:
> 
> $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml 
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.

I tried to use docbookrx for the bits that were not properly converted,
like the manpage-like pages:

$  ../docbookrx/bin/docbookrx Documentation/DocBook/media/v4l/func-ioctl.xml
No visitor defined for ! Skipping.

Dan, if you want to take a look on what's going wrong here, 
the XML I'm trying to convert is:


https://git.linuxtv.org/media_tree.git/tree/Documentation/DocBook/media/v4l/func-ioctl.xml

If this would work, it should be generating something like:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/tree/func-ioctl.adoc

Pandoc failed to fully convert it, but at least it left all the texts,
with prevented rewriting it from scratch. This is the manual fix
I applied to it:

https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc

FYI, we have 133 xml files at the media uAPI doc with refmeta.

-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 10:39:22 -0300
Mauro Carvalho Chehab  escreveu:

> Em Tue, 08 Mar 2016 05:13:13 -0700
> Dan Allen  escreveu:
> 
> > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <  
> > mche...@osg.samsung.com> wrote:
> >   
> > > pandoc did a really crap job on the conversion. To convert this
> > > into something useful, we'll need to spend a lot of time, as it lost
> > > most of the cross-references, as they were defined via DocBook macros.
> > >
> > 
> > I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> > the works called DocBookRx.
> > 
> > https://github.com/opendevise/docbookrx
> > 
> > It has converted several very serious DocBook documents and we're
> > continuing to improve it. It's also a lot easier to hack than pandoc.  
> 
> Didn't work:
> 
> $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml 
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.
> No visitor defined for ! Skipping.

I tried to use docbookrx for the bits that were not properly converted,
like the manpage-like pages:

$  ../docbookrx/bin/docbookrx Documentation/DocBook/media/v4l/func-ioctl.xml
No visitor defined for ! Skipping.

Dan, if you want to take a look on what's going wrong here, 
the XML I'm trying to convert is:


https://git.linuxtv.org/media_tree.git/tree/Documentation/DocBook/media/v4l/func-ioctl.xml

If this would work, it should be generating something like:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/tree/func-ioctl.adoc

Pandoc failed to fully convert it, but at least it left all the texts,
with prevented rewriting it from scratch. This is the manual fix
I applied to it:

https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc

FYI, we have 133 xml files at the media uAPI doc with refmeta.

-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Tue, 08 Mar 2016, Dan Allen  wrote:
> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

I need to look into this again. Is there a specific option or directive
to produce split output for includes? When I tried this, the result was
just one big output file. (And indeed we'd need both. Some includes we
want embedded, some includes should produce separate outputs.)

>> That actually makes choosing asciidoc harder, because
>> requiring another language environment complicates, not simplifies, the
>> toolchain. I'd really like to lower the bar for building the
>> documentation, for everyone, so much so that it becomes part of the
>> normal checks for patch inclusion.
>
> Pardon my bluntness here, but I don't buy that argument. This is Linux.
> Installing software couldn't be simpler, and we're talking about an
> extremely well supported language (Ruby).

Granted, that part works for me. I'm not so sensitive to the
dependencies; others may disagree.

> I think it's a huge exaggeration to say that Asciidoctor is any harder to
> install than AsciiDoc Python. It's also a heck of a lot smaller in size
> since AsciiDoc Python pulls in hundreds of MB of LaTeX packages.

For me, the comparison is really between Sphinx and Asciidoctor, not so
much doc vs. doctor. The native output format and extension support in
Sphinx is appealing; I am not yet convinced we could manage with
Asciidoctor but without DocBook. The extension offering seems better in
Sphinx.

> Whatever you decide, I wish you all the best with your documentation
> efforts!

Thanks!

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Jani Nikula]

On Tue, 08 Mar 2016, Dan Allen  wrote:
> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

I need to look into this again. Is there a specific option or directive
to produce split output for includes? When I tried this, the result was
just one big output file. (And indeed we'd need both. Some includes we
want embedded, some includes should produce separate outputs.)

>> That actually makes choosing asciidoc harder, because
>> requiring another language environment complicates, not simplifies, the
>> toolchain. I'd really like to lower the bar for building the
>> documentation, for everyone, so much so that it becomes part of the
>> normal checks for patch inclusion.
>
> Pardon my bluntness here, but I don't buy that argument. This is Linux.
> Installing software couldn't be simpler, and we're talking about an
> extremely well supported language (Ruby).

Granted, that part works for me. I'm not so sensitive to the
dependencies; others may disagree.

> I think it's a huge exaggeration to say that Asciidoctor is any harder to
> install than AsciiDoc Python. It's also a heck of a lot smaller in size
> since AsciiDoc Python pulls in hundreds of MB of LaTeX packages.

For me, the comparison is really between Sphinx and Asciidoctor, not so
much doc vs. doctor. The native output format and extension support in
Sphinx is appealing; I am not yet convinced we could manage with
Asciidoctor but without DocBook. The extension offering seems better in
Sphinx.

> Whatever you decide, I wish you all the best with your documentation
> efforts!

Thanks!

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:13:13 -0700
Dan Allen  escreveu:

> On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <
> mche...@osg.samsung.com> wrote:  
> 
> > pandoc did a really crap job on the conversion. To convert this
> > into something useful, we'll need to spend a lot of time, as it lost
> > most of the cross-references, as they were defined via DocBook macros.
> >  
> 
> I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> the works called DocBookRx.
> 
> https://github.com/opendevise/docbookrx
> 
> It has converted several very serious DocBook documents and we're
> continuing to improve it. It's also a lot easier to hack than pandoc.

Didn't work:

$ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml 
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.


> 
> -Dan
> 
> 


-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:13:13 -0700
Dan Allen  escreveu:

> On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <
> mche...@osg.samsung.com> wrote:  
> 
> > pandoc did a really crap job on the conversion. To convert this
> > into something useful, we'll need to spend a lot of time, as it lost
> > most of the cross-references, as they were defined via DocBook macros.
> >  
> 
> I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> the works called DocBookRx.
> 
> https://github.com/opendevise/docbookrx
> 
> It has converted several very serious DocBook documents and we're
> continuing to improve it. It's also a lot easier to hack than pandoc.

Didn't work:

$ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml 
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.
No visitor defined for ! Skipping.


> 
> -Dan
> 
> 


-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:09:40 -0700
Dan Allen  escreveu:

> Jani wrote:
> 
> > there was no support for chunked, or split
> > to chapters, HTML, and the single page result was simply way too big.
> >  
> 
> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

The problem on pre-splitting the documents and process them in separate
is that this will break cross-references. At the media uAPI Docbook,
we use a lot of cross references.

Btw, we use a lot of includes. Currently, it has 187 separate files.

We even parse the header files looking for typedefs, structs, enums,
#defines and functions, in order to produce a document that will
cross-reference the documentation.

> Second, there is a custom converter in the works to split post-generate
> (which is really what we're talking about when we compare it to the DocBook
> toolchain).
> 
> https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb
> 
> It's just a prototype, but proves it is possible by design.

I didn't test it, but I saw some comments at the web that the part
that would handle cross-references between files is not ready.

> Personally, I don't like most chunked HTML approaches because they split
> arbitrarily. We are trying to find the right balance so that the output is
> actually sensible. There's still work to do, but there are options in the
> meantime.

Well, if it is capable of creating one chunk per include file, and do
cross-references between chunks, this would work for media UAPI book.

Yet, it would be good to have the multi-chunk extension packaged
on major distros, as I don't like the idea of installing it without
using my distro's package manager.

-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:09:40 -0700
Dan Allen  escreveu:

> Jani wrote:
> 
> > there was no support for chunked, or split
> > to chapters, HTML, and the single page result was simply way too big.
> >  
> 
> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

The problem on pre-splitting the documents and process them in separate
is that this will break cross-references. At the media uAPI Docbook,
we use a lot of cross references.

Btw, we use a lot of includes. Currently, it has 187 separate files.

We even parse the header files looking for typedefs, structs, enums,
#defines and functions, in order to produce a document that will
cross-reference the documentation.

> Second, there is a custom converter in the works to split post-generate
> (which is really what we're talking about when we compare it to the DocBook
> toolchain).
> 
> https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb
> 
> It's just a prototype, but proves it is possible by design.

I didn't test it, but I saw some comments at the web that the part
that would handle cross-references between files is not ready.

> Personally, I don't like most chunked HTML approaches because they split
> arbitrarily. We are trying to find the right balance so that the output is
> actually sensible. There's still work to do, but there are options in the
> meantime.

Well, if it is capable of creating one chunk per include file, and do
cross-references between chunks, this would work for media UAPI book.

Yet, it would be good to have the multi-chunk extension packaged
on major distros, as I don't like the idea of installing it without
using my distro's package manager.

-- 
Thanks,
Mauro

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 11:49:35 +0200
Jani Nikula  escreveu:

> On Tue, 08 Mar 2016, Dan Allen  wrote:
> > One of the key goals of the Asciidoctor project is to be able to directly
> > produce a wide variety of outputs from the same source (without DocBook).
> > We've added flexibility and best practices into the syntax and matured the
> > converter mechanism to bridge this (sometimes very wide) gap.  
> 
> I think our conclusion so far was that the native AsciiDoc (and
> Asciidoctor) outputs fell short of our needs, forcing us to use the
> DocBook pipeline. I, for one, was hoping we could eventually simplify
> the toolchain. For example, there was no support for chunked, or split
> to chapters, HTML, and the single page result was simply way too big.
> 
> > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers
> > acknowledge that (including the original creator).  
> 
> Thanks for the input. We've touched the topic of AsciiDoc
> vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of
> AsciiDoc. That actually makes choosing asciidoc harder, because
> requiring another language environment complicates, not simplifies, the
> toolchain. I'd really like to lower the bar for building the
> documentation, for everyone, so much so that it becomes part of the
> normal checks for patch inclusion.

As I failed to find a way to solve the issues with Sphinx/RST, I started
a PoC using Asciidoctor for the Media uAPI docbook conversion, in order
to see if this would work for us.

I'm not a big fan of using a language that I don't domain, like
Ruby (and the same applies to Python), but, as I said before,
provided that the toolchain works, can easily be installed on
Fedora and Debian, and provide the functionality I need, I'm
ok with that. 

On my tests, Asciidoctor is really fast. It takes a fraction of the
time require to build from DocBook, with is a good thing.

Yet, I suspect that it doesn't have the strict checks that xmllint
have. For example, I didn't see any warning about a missing cross-ref.
We use those warnings to discover if something is added at the code,
but were not documented. Its error reports are also crappy, as it
doesn't tell where the problem is. For example:

$ asciidoctor -n -b docbook media_api.adoc
asciidoctor: WARNING: tables must have at least one body row

On a document with 33793 lines (this is the size of the uAPI doc
when converted to AsciiDoc format), the above warning doesn't help.

So, I'm actually planning to use AsciiDoc/xmllint to check for
documentation troubles, if possible.

Even with AsciiDoc/Asciidoctor, there are several features at the
media documents that aren't well supported. For example, we widely
use the Docbook's tags to generate a manpage-like description:


  
V4L2 open()

  

  
v4l2-open
Open a V4L2 device
  

  

  #include fcntl.h
  
int open
const char *device_name
int flags
  

  

Asciidoctor doesn't produce anything like that, if the booktype is not
"manpage". Well, the media documentation is a multi-part book.

So, all those tags should be manually converted (actually, pandoc
made a mess with those tags - so manual work is required anyway,
no matter what markdown language we use). It will be a hard work,
but, at least, this is doable. 

Also, it doesn't accept images with PDF format. We have a few PDF
images, although I we have them also in GIF. So, not a big issue here.

I also did lots of table conversions, to see if our complex tables
would fit. The answer is: Yes: on all cases I converted, it worked
fine.

There are two ways of doing that:
- Nested tables
- Merging cells

Some tables we use won't work fine with nested tables, as they have
cells merging the entire line of such tables.

Also, converting some tables to nested tables would be incredible hard,
as we have really big tables there that would need to add dozens of nested
tables inside, like this one:


https://linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html#id-1.4.4.14.6.4

Using merging cells works fine, tough, and it is not hard to use.
The tables are easy to edit. The Asciidoctor syntax (not sure if AsciiDoc
also accepts this) is:

.MFC 5.1 Control IDs
[width="100%",cols="7%,40%,13%,40%",options="header",]
|===
|ID |Type 2+| Description

...

The "2+|" tells that the next content should be merged into two cells.

One of the good things is that we're not forced to use asciiart, with
would make it really hard to handle the tables.

For those wanting to see the results so far:

The html is at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

The testing tree is at:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/

PS.: pandoc did a really crap job on the conversion. To convert this

Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 11:49:35 +0200
Jani Nikula  escreveu:

> On Tue, 08 Mar 2016, Dan Allen  wrote:
> > One of the key goals of the Asciidoctor project is to be able to directly
> > produce a wide variety of outputs from the same source (without DocBook).
> > We've added flexibility and best practices into the syntax and matured the
> > converter mechanism to bridge this (sometimes very wide) gap.  
> 
> I think our conclusion so far was that the native AsciiDoc (and
> Asciidoctor) outputs fell short of our needs, forcing us to use the
> DocBook pipeline. I, for one, was hoping we could eventually simplify
> the toolchain. For example, there was no support for chunked, or split
> to chapters, HTML, and the single page result was simply way too big.
> 
> > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers
> > acknowledge that (including the original creator).  
> 
> Thanks for the input. We've touched the topic of AsciiDoc
> vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of
> AsciiDoc. That actually makes choosing asciidoc harder, because
> requiring another language environment complicates, not simplifies, the
> toolchain. I'd really like to lower the bar for building the
> documentation, for everyone, so much so that it becomes part of the
> normal checks for patch inclusion.

As I failed to find a way to solve the issues with Sphinx/RST, I started
a PoC using Asciidoctor for the Media uAPI docbook conversion, in order
to see if this would work for us.

I'm not a big fan of using a language that I don't domain, like
Ruby (and the same applies to Python), but, as I said before,
provided that the toolchain works, can easily be installed on
Fedora and Debian, and provide the functionality I need, I'm
ok with that. 

On my tests, Asciidoctor is really fast. It takes a fraction of the
time require to build from DocBook, with is a good thing.

Yet, I suspect that it doesn't have the strict checks that xmllint
have. For example, I didn't see any warning about a missing cross-ref.
We use those warnings to discover if something is added at the code,
but were not documented. Its error reports are also crappy, as it
doesn't tell where the problem is. For example:

$ asciidoctor -n -b docbook media_api.adoc
asciidoctor: WARNING: tables must have at least one body row

On a document with 33793 lines (this is the size of the uAPI doc
when converted to AsciiDoc format), the above warning doesn't help.

So, I'm actually planning to use AsciiDoc/xmllint to check for
documentation troubles, if possible.

Even with AsciiDoc/Asciidoctor, there are several features at the
media documents that aren't well supported. For example, we widely
use the Docbook's tags to generate a manpage-like description:


  
V4L2 open()

  

  
v4l2-open
Open a V4L2 device
  

  

  #include fcntl.h
  
int open
const char *device_name
int flags
  

  

Asciidoctor doesn't produce anything like that, if the booktype is not
"manpage". Well, the media documentation is a multi-part book.

So, all those tags should be manually converted (actually, pandoc
made a mess with those tags - so manual work is required anyway,
no matter what markdown language we use). It will be a hard work,
but, at least, this is doable. 

Also, it doesn't accept images with PDF format. We have a few PDF
images, although I we have them also in GIF. So, not a big issue here.

I also did lots of table conversions, to see if our complex tables
would fit. The answer is: Yes: on all cases I converted, it worked
fine.

There are two ways of doing that:
- Nested tables
- Merging cells

Some tables we use won't work fine with nested tables, as they have
cells merging the entire line of such tables.

Also, converting some tables to nested tables would be incredible hard,
as we have really big tables there that would need to add dozens of nested
tables inside, like this one:


https://linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html#id-1.4.4.14.6.4

Using merging cells works fine, tough, and it is not hard to use.
The tables are easy to edit. The Asciidoctor syntax (not sure if AsciiDoc
also accepts this) is:

.MFC 5.1 Control IDs
[width="100%",cols="7%,40%,13%,40%",options="header",]
|===
|ID |Type 2+| Description

...

The "2+|" tells that the next content should be merged into two cells.

One of the good things is that we're not forced to use asciiart, with
would make it really hard to handle the tables.

For those wanting to see the results so far:

The html is at:

https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

The testing tree is at:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/

PS.: pandoc did a really crap job on the conversion. To convert this
into something useful, we'll need to spend a